Incident 21: Tougher Turing Test Exposes Chatbots’ Stupidity (migrated to Issue)
CSETv0 Taxonomy ClassificationsTaxonomy Details
The Winograd Schema Challenge in 2016 highlighted shortcomings of an artificially intelligent system's ability to understand context. The Challenge is designed to present ambiguous sentences and ask AI systems to decipher them. In the Winograd Scheme Challenge, the two winning entries were successful 48% of the time, while random chance was correct 45% of the time. Quan Liu of the University of Science and Technology of China (partnering with University of Toronto and National Research Council of Canada) and Nicos Isaak of the Open University of Cyprus presented the most successful systems. It is notable that Google and Facebook did not participate.
The 2016 Winograd Schema Challenge highlighted how even the most successful AI systems entered into the Challenge were only successful 3% more often than random chance.
AI System Description
Artificially intelligent systems meant to understand ambiguous English sentences.
Sector of Deployment
Professional, scientific and technical activities
Relevant AI functions
Perception, Cognition, Action
New York, NY
Winograd Schema Challenge, University of Science and Technology of China, Quan Liu, University of Toronto, National Research Council of Canada, Nicos Isaak, Open University of Cyprus
Quan Liu, Nicos Isaak
GMF Taxonomy ClassificationsTaxonomy Details
Known AI Goal
Known AI Technology
Language Modeling, Distributional Learning
Potential AI Technology
Potential AI Technical Failure
Generalization Failure, Dataset Imbalance, Underfitting, Context Misidentification
Description: The 2016 Winograd Schema Challenge highli…
Did our AI mess up? Flag the unrelated incidents