The Winograd Schema Challenge in 2016 highlighted shortcomings of an artificially intelligent system's ability to understand context. The Challenge is designed to present ambiguous sentences and ask AI systems to decipher them. In the Winograd Scheme Challenge, the two winning entries were successful 48% of the time, while random chance was correct 45% of the time. Quan Liu of the University of Science and Technology of China (partnering with University of Toronto and National Research Council of Canada) and Nicos Isaak of the Open University of Cyprus presented the most successful systems. It is notable that Google and Facebook did not participate.

The 2016 Winograd Schema Challenge highlighted how even the most successful AI systems entered into the Challenge were only successful 3% more often than random chance.

Unclear/unknown

Artificially intelligent systems meant to understand ambiguous English sentences.

Professional, scientific and technical activities

Perception, Cognition, Action

New York, NY

Winograd Schema Challenge, University of Science and Technology of China, Quan Liu, University of Toronto, National Research Council of Canada, Nicos Isaak, Open University of Cyprus

Quan Liu, Nicos Isaak

2016-01-01T00:00:00.000Z

2016-01-01T00:00:00.000Z

Unclear/unknown

Unclear

No

Question Answering

Language Modeling, Distributional Learning

Transformer

Generalization Failure, Dataset Imbalance, Underfitting, Context Misidentification