The Winograd Schema Challenge (WSC) is a test of artificial intelligence designed to measure a machine's ability to perform common-sense reasoning and resolve ambiguous pronoun references. Named after Terry Winograd's earlier work with the SHRDLU program, each schema consists of a sentence containing a pronoun whose referent is ambiguous to a purely syntactic parser but can be resolved by a human using world knowledge. The challenge was proposed by Hector Levesque in 2011 as an alternative to the Turing Test, emphasizing reasoning over conversation. It has since become a benchmark for natural language understanding, with variants such as the WSC in the SuperGLUE dataset and the Definite Pronoun Resolution (DPR) dataset.
1 Background
1.1 Origins in Winograd's SHRDLU
The concept of the Winograd Schema Challenge draws directly from the earlier SHRDLU program, developed by Terry Winograd at MIT in the late 1960s. SHRDLU was a natural‑language system that operated within a simple blocks world, allowing a user to issue commands and ask questions about the arrangement of blocks. In its operation, SHRDLU could resolve ambiguous pronoun references by leveraging its knowledge of the virtual environment—for example, understanding that “it” in “Pick up the blue block and put it on the red block” refers to the blue block. This ability to use contextual knowledge rather than purely syntactic rules inspired later efforts to design tests that isolate such reasoning.
1.2 Motivation as an AI benchmark
1.2.1 Limitations of the Turing Test
By the early 2010s, the Turing Test faced widespread criticism as a reliable measure of machine intelligence. Critics noted that conversational agents could simulate human‑like responses using tricks such as chitchat or evasive answers, without demonstrating genuine understanding. Moreover, the open‑ended nature of the test made it difficult to compare systems quantitatively. The Winograd Schema Challenge was proposed as a more focused, objective alternative that could be scored automatically.
1.2.2 Focus on commonsense reasoning
The challenge specifically targets commonsense reasoning—the kind of everyday knowledge that humans use effortlessly to interpret language. By requiring a system to choose between two possible referents for a pronoun, the test isolates a narrow but fundamental aspect of understanding. Success on the WSC implies that a system can apply world knowledge (e.g., size relationships, typical physical behaviors) that goes beyond statistical patterns in text.
1.3 Definition and structure
1.3.1 The "Winograd schema" pattern
A Winograd schema is a sentence that contains a pronoun with two grammatically plausible antecedents. The sentence is carefully constructed so that a human reader can resolve the ambiguity using common sense, while a system relying solely on syntactic rules would have no basis for a decision. The schema is presented as a binary choice: “Which noun does the pronoun refer to?”
1.3.2 Grammatical ambiguity vs. pragmatic resolution
The key property of a Winograd schema is grammatical ambiguity: both possible referents agree with the pronoun in number and gender, and the surrounding syntactic structure does not disambiguate. Resolution requires pragmatic reasoning—for instance, understanding that a suitcase is unlikely to be described as “too big” if the trophy cannot fit into it. This distinction separates the WSC from simpler pronoun resolution tasks.
2 Formal description
2.1 Core components
2.1.1 Sentence with binary-choice pronoun
Every schema consists of a single English sentence containing one pronoun (e.g., “it”, “they”, “she”) that is ambiguous in context. The sentence is presented without additional discourse or background.
2.1.2 Two possible referents
The sentence includes two noun phrases that could each serve as the antecedent of the pronoun. These noun phrases are typically different in meaning and are often the subject and object of the main clause.
2.1.3 Correct answer requires world knowledge
The correct referent cannot be determined by syntactic agreement or simple co‑occurrence statistics. A system must draw on knowledge about the real world—for example, that a trophy is smaller than a suitcase in typical use, or that a person can hold an object. This requirement makes the WSC a test of commonsense reasoning.
2.2 Example schemas
2.2.1 Classic example: "The trophy would not fit in the brown suitcase because it was too big."
In this widely cited schema, the pronoun “it” could refer to the trophy or to the suitcase. Common sense indicates that the object being described as “too big” is the trophy (the thing that fails to fit), not the suitcase. A human instantly makes this inference; a purely statistical system might guess incorrectly if it relies on surface patterns.
2.2.2 Modified variants and contrast sets
To control for spurious correlations, researchers often create contrast sets by swapping the two nouns or altering the verb. For example, “The trophy fit in the brown suitcase because it was too big” would change the interpretation, showing that the correct answer depends on the situational knowledge implied by “fit” vs. “would not fit”.
3 Evaluation criteria
3.1 Scoring methods
3.1.1 Strict accuracy
The standard metric is binary accuracy: for each schema, the system outputs a choice (noun A or noun B), and the answer is considered correct only if it matches the human‑annotated referent. No partial credit is given.
3.1.2 Comparison with human performance
Human performance on the original 60‑schema set is reported as near‑perfect (over 95% accuracy). A system’s score is thus measured against this human baseline. The challenge is considered “passed” when a system achieves human‑level accuracy, though no system has consistently done so on the original set without specialized training.
3.2 Dataset versions
3.2.1 Original Levesque collection (60 schemas)
Hector Levesque curated a set of 60 Winograd schemas, each with a single sentence and a binary choice. This set remains a standard for evaluating core commonsense reasoning, though its small size limits statistical reliability.
3.2.2 WSC in SuperGLUE (273 examples)
The SuperGLUE benchmark, introduced in 2019, includes a Winograd Schema Challenge task with 273 examples. These are drawn from various sources and are formatted as either single‑sentence or two‑sentence versions. The WSC in SuperGLUE is one of several tasks that together assess general‑purpose language understanding.
3.2.3 DPR dataset (1882 examples)
The Definite Pronoun Resolution (DPR) dataset, created by Google AI, contains 1882 examples of ambiguous pronoun reference, including both Winograd‑style and simpler cases. It is larger than the original set and is often used for training and evaluation.
3.2.4 WinoGrande (44k crowd-sourced schemas)
WinoGrande, introduced by the Allen Institute for AI, is a large‑scale dataset of 44,000 Winograd‑style schemas. To reduce dataset bias, the schemas were generated using a controlled crowd‑sourcing pipeline that explicitly creates contrast sets. WinoGrande has become a popular benchmark because of its size and the difficulty of shortcut learning.
4 Approaches and systems
4.1 Rule-based methods
4.1.1 Hand-crafted linguistic heuristics
Early approaches to the WSC relied on manually written rules that encoded patterns of pronoun resolution, such as preferring the subject as antecedent or applying selectional restrictions (e.g., “big” typically modifies objects that can be described by size). These methods performed poorly because the schemas are designed to defeat such heuristics.
4.1.2 Knowledge-base reasoning
Some rule‑based systems attempted to query external knowledge bases (e.g., WordNet, ConceptNet) to infer the plausibility of each referent. For example, a system could check whether a trophy is commonly described as “big” in the context of fitting into a suitcase. While more robust than simple heuristics, these methods still struggled with the nuanced reasoning required.
4.2 Machine learning approaches
4.2.1 Feature-based models
Before the advent of deep learning, researchers used supervised classifiers trained on hand‑engineered features such as part‑of‑speech tags, dependency relations, and semantic roles. Performance was often low because the datasets were small and the schemas required knowledge beyond local features.
4.2.2 Neural sequence models
Recurrent neural networks (RNNs) and Long Short‑Term Memory (LSTM) networks were applied to encode the sentence and predict the referent. These models learned some non‑local patterns but still lacked the capacity to capture the world knowledge implicit in the schemas.
4.2.3 Transformer-based models (e.g., BERT, GPT)
4.2.3.1 Fine-tuning on WSC
With the rise of transformer pre‑training, models such as BERT achieved strong results on the WSC when fine‑tuned on a dataset of Winograd schemas. BERT’s masked language modeling objective allowed it to predict the pronoun’s referent by “filling in the blank” for the two candidate nouns. Fine‑tuned BERT reached over 90% accuracy on the SuperGLUE WSC, demonstrating that large‑scale language models can capture much of the commonsense knowledge needed.
4.2.3.2 Prompt-based reasoning
Another approach uses prompted reasoning: a large language model (e.g., GPT‑3) is given the schema along with a prompt such as “Which noun does the pronoun refer to?” The model generates an answer in natural language. This method can achieve high accuracy without task‑specific fine‑tuning, but it is sensitive to prompt wording and may rely on memorized patterns.
4.3 Commonsense knowledge integration
4.3.1 External knowledge graphs
Some systems augment neural models with external knowledge graphs (e.g., ConceptNet, Atomic) to provide explicit facts about typical properties and relations. The graph is used to score the plausibility of each candidate referent. This approach can improve performance on schemas that require uncommon or specific knowledge, such as “The lawyer questioned the witness because he was dishonest” (where “he” likely refers to the witness).
4.3.2 Pre-trained language model internal knowledge
Large pre‑trained models like BERT and GPT implicitly store a vast amount of commonsense knowledge in their parameters. Studies have shown that these models can perform well on the WSC without any external knowledge, suggesting that the necessary reasoning is encoded during pre‑training on massive text corpora. However, the models’ reliance on superficial cues can also lead to failures on carefully constructed adversarial examples.
5 Criticisms and limitations
5.1 Dataset bias and shortcut learning
5.1.1 Lexical overlap issues
Many Winograd schemas in early datasets contained lexical cues that allowed models to bypass genuine reasoning. For example, a system could learn that “big” is statistically associated with the trophy in “The trophy would not fit… because it was too big” because the word “big” appears in training data with “trophy” more often than with “suitcase”. Such biases were documented in the WinoGrande paper, which showed that simple baselines using word overlap alone could achieve surprisingly high accuracy.
5.1.2 Performance without genuine reasoning
When control experiments removed the pronoun‑resolution requirement (e.g., replacing the pronoun with the correct noun), many models achieved nearly the same accuracy, indicating that they were not truly performing coreference resolution but rather exploiting surface correlations. This raises doubts about whether high WSC scores reflect commonsense reasoning.
5.2 Relation to broader AI understanding
Critics argue that the WSC, even if solved perfectly, would only test a narrow slice of intelligence. Commonsense reasoning encompasses not only pronoun resolution but also physical intuition, social pragmatics, causal reasoning, and more. A system that passes the WSC might still lack understanding of the world in any deep sense.
5.3 Alternative benchmarks
5.3.1 Physical commonsense tasks
To address the limitations of the WSC, researchers have developed benchmarks that require understanding of physical dynamics, such as the Physical Reasoning Challenge (PIQA) or the Everyday Physical Science task. These datasets test whether a machine can predict the outcome of simple physical interactions (e.g., “If you drop a glass, it will [break / bounce]”).
5.3.2 Abductive reasoning datasets
Abductive reasoning benchmarks (e.g., ART, Abductive Natural Language Inference) require a model to infer the most plausible explanation for an observed event. Unlike the WSC, these tasks involve generating or choosing among multiple causal accounts, testing a broader form of commonsense reasoning.
6 Extensions and variants
6.1 Winograd Schema Challenge 2.0
Proposed by R. Davis and others, Winograd Schema Challenge 2.0 expands the original concept by using longer contexts (two or three sentences) and including a greater variety of pronoun types (e.g., possessive pronouns). The goal is to increase difficulty and reduce the impact of dataset biases.
6.2 Visual Winograd schemas
Visual versions of the WSC pair a sentence with an image, requiring the system to resolve the pronoun by analyzing both textual and visual information. For example, an image shows a woman holding a book next to a man, and the sentence “She handed him the book but he dropped it” tests whether the AI can link the correct referent to the visual scene.
6.3 Multilingual WSC
Several datasets have been created for languages other than English, including Chinese, French, and German. These multilingual collections help evaluate whether models can perform commonsense reasoning across different linguistic structures and cultural contexts. The CMU Multilingual WSC covers five languages.
6.4 Adversarial and contrastive sets
To expose shortcut learning, researchers have generated adversarial examples where the sentence structure is slightly altered to mislead models that rely on spurious correlations. For instance, changing a single word (e.g., “big” to “small”) while keeping the pronoun ambiguous can cause high‑performing models to flip their answer incorrectly.
7 Significance in formal sciences
7.1 Role in AI evaluation
The Winograd Schema Challenge has become a standard component of modern AI benchmarks, serving as a litmus test for progress in natural language understanding. Its inclusion in SuperGLUE and other multi‑task evaluations demonstrates its perceived importance. The challenge has also spurred research into dataset construction, bias detection, and interpretability.
7.2 Connections to linguistics and pragmatics
7.2.1 Pronoun resolution theory
In linguistics, pronoun resolution is studied under the umbrella of anaphora resolution. The WSC highlights the need for pragmatic (context‑based) principles beyond syntactic binding theories, such as Centering Theory and Discourse Representation Theory. The challenge has stimulated computational models that operationalize these linguistic frameworks.
7.2.2 Discourse representation theory
Discourse Representation Theory (DRT) provides a formal framework for representing discourse meaning and resolving referring expressions. Winograd schemas serve as test cases for DRT‑based systems, since the resolution often requires the construction of a discourse model that includes knowledge about typical states (e.g., the size of a trophy vs. a suitcase). The WSC therefore bridges AI research and formal semantics.
8 See also
* Turing Test * Commonsense reasoning * Natural language understanding * Coreference resolution * SuperGLUE * SHRDLU * Hector Levesque
9 References
- Levesque, H. J. (2011). "The Winograd Schema Challenge". AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
- Winograd, T. (1972). "Understanding Natural Language". Cognitive Psychology.
- Rahman, A., & Ng, V. (2012). "Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge". EMNLP.
- Wang, A., et al. (2019). "SuperGLUE: A Stickier Benchmark for General‑Purpose Language Understanding Systems". NeurIPS.
- Sakaguchi, K., et al. (2020). "WinoGrande: An Adversarial Winograd Schema Challenge at Scale". AAAI.
- Davis, R. (2016). "A Second Winograd Schema Challenge: More and Bigger Schemas". AAAI Workshop.
- Levesque, H. J., Davis, E., & Morgenstern, L. (2012). "The Winograd Schema Challenge". Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.