Incident 9: NY City School Teacher Evaluation Algorithm Contested

Description: An algorithm used to rate the effectiveness of school teachers in New York has resulted in thousands of disputes of its results.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History
Alleged: New York city Dept. of Education developed and deployed an AI system, which harmed Teachers.

Incident Stats

Incident ID
9
Report Count
7
Incident Date
2012-02-25
Editors
Sean McGregor

CSETv0 Taxonomy Classifications

Taxonomy Details

Full Description

A value-added measurement based algorithm used to calculate the effectiveness of school teachers is being challenged for its lack of apparent accuracy and real-world relevance. In Rochester, New York approximately 600 teachers are disputing the results, and in Syracuse between 400-500 are disputing the results. The VAM score can be used to give raises to teachers deemed "effective" or "highly effective", but can also be used to fire teachers who are given two "ineffective" ratings in a row. Teachers criticize the algorithm for including only Math and English in the evaluation (even for teachers of other subjects as the algorithm only covers those two subjects), using school averages to calculate a single student's expected average, and high-grade-earning students being predicted to grow at literally impossible rates (to score grades higher than 100% on tests).

Short Description

An algorithm used to rate the effectiveness of school teachers in New York has resulted in thousands of disputes of its results.

Severity

Negligible

Harm Distribution Basis

Other:School Teachers

Harm Type

Financial harm

AI System Description

value-added analysis algorithm used for evaluating a school teacher's effectiveness of teaching

System Developer

New York city Dept. of Education

Sector of Deployment

Education

Relevant AI functions

Unclear

AI Techniques

value-added mesaurements

AI Applications

data processing, data prediction

Location

New York, United States of America

Named Entities

New York Department of Education, United Federation of Teachers, Sheri Lederman, Common Core, Governor Andrew Cuomo

Technology Purveyor

New York Department of Education

Beginning Date

2007-01-01T00:00:00.000Z

Ending Date

2010-01-01T00:00:00.000Z

Near Miss

Harm caused

Intent

Accident

Lives Lost

No

Data Inputs

School grades, student grades, predicted grades

CSETv1 Taxonomy Classifications

Taxonomy Details

Harm Distribution Basis

none

Sector of Deployment

Education

parentadvocates.org · 2004

Stories & Grievances

A NYC Math Teacher Fights Back After Receiving an Unfair 'Unsatisfactory' Rating from a Principal

Edmond Farrell uses the Freedom of Information Law and Department of Education/Teacher regulations in his fight to change…

New York City Teacher Ratings: Teacher Data Reports Publicly Released Amid Controversy
huffingtonpost.com · 2012

The New York City Department of Education released today a list of individual ratings of thousands of the city's schoolteachers, a move that concludes a lengthy legal battle waged by the local teachers' union and media.

The Teacher Data Rep…

shankerinstitute.org · 2012

Late last week and over the weekend, New York City newspapers, including the New York Times and Wall Street Journal, published the value-added scores (teacher data reports) for thousands of the city’s teachers. Prior to this release, I and …

Analyzing Released NYC Value-Added Data Part 2
garyrubinstein.teachforus.org · 2012

In part 1 I demonstrated there was little correlation between how a teacher was rated in 2009 to how that same teacher was rated in 2010. So what can be more crazy than a teacher being rated highly effective one year and then highly ineffec…

Teachers plan widespread appeals of 'unfair' evaluations
politico.com · 2013

Teachers plan widespread appeals of 'unfair' evaluations

ALBANY—Hundreds of teachers in urban school districts plan to appeal performance evaluations that could be used as grounds for termination under a new statewide system for evaluating …

Master teacher suing New York state over ‘ineffective’ rating is going to court
washingtonpost.com · 2015

Former New York state education commissioner John King, who is being sued by a N.Y. teacher over the state’s educator evaluation system. (Mike Groll/AP)

A veteran teacher suing New York state education officials over the controversial metho…

New data show more than half of NYC teachers judged, in part, by test scores they don’t directly affect
cityandstateny.com · 2017

Just over half of New York City teachers were evaluated in the 2015–16 school year, in part, by tests in subjects or of students they didn’t teach, according to data obtained by Chalkbeat through a public records request.

At 53 percent of c…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents