Spider
The 2018 dataset paper that defined the cross-domain text-to-SQL task and became the standard benchmark for natural language database querying.
Paper: Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
Authors: Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, Dragomir Radev · 2018
Read the paper
The Problem
Early text-to-SQL systems were evaluated on datasets like WikiSQL or ATIS. These datasets were flawed: they featured simple, flat tables, didn't require complex SQL operations (like JOIN, GROUP BY, or nested queries), and critically, they evaluated models on the same database domains used in training. A model could achieve high accuracy by memorizing schema patterns, but would immediately break if deployed on a new, unseen database.
The Idea
Yale researchers created Spider, a massive, complex dataset of 10,181 questions and 5,693 complex SQL queries over 200 databases. Spider enforced "cross-domain" evaluation: the databases in the test set did not appear in the training set. To succeed, a model couldn't memorize; it had to genuinely understand how to map natural language intent to relational schema structure (schema linking) and construct complex SQL syntax.
How It Works
Spider was constructed with specific constraints to push the field forward:
- Complexity: The queries required multiple JOINs, nested queries (INTERSECT, UNION), GROUP BY, and complex filtering. The databases had multiple tables linked by foreign keys.
- Cross-Domain Split: Training on a "flight booking" database, but testing on a "college administration" database.
- Exact Set Match Evaluation: Rather than executing the SQL and checking the result (which can yield false positives due to empty tables), Spider evaluated the predicted SQL tree against the gold SQL tree to ensure the model actually wrote the correct logic.
Why It Mattered
Spider completely reset the baseline for text-to-SQL. Accuracy dropped from ~80% on WikiSQL to ~12% on Spider for early models. It defined the modern research agenda for semantic parsing, proving that "schema linking" (identifying which words map to which tables/columns) was the core bottleneck to cross-domain generalization.
What Came After
Spider drove years of architecture innovation, leading to models like RAT-SQL and PICARD. With the rise of LLMs, Spider remained the standard benchmark for evaluating prompt engineering techniques for SQL generation (like DIN-SQL). It eventually spawned successors like BIRD-SQL, which focused on larger, messier, real-world database values and execution efficiency.