Citation record for Incident 37

Suggested citation format

Anonymous. (2016-08-10) Incident Number 37. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
Report Count
Incident Date
Editors
37
31
2016-08-10
Sean McGregor

CSET Taxonomy Classifications

Taxonomy Details

Full Description

In 2015, Amazon scrapped an internal recruiting algorithm developed by its Edinburgh office that would down-rank resumes when it included the word "women's", and two women's colleges. The algorithm ranked an applicant out of five stars, and it would give preference to resumes that contained what Reuters called "masculine language," or strong verbs like "executed" or "captured". These patterns occured because the engineered who made the algorithm trained it with past candidates' resumes submitted over the previous ten years, and the past candidates in the industry were male-dominated.

Short Description

Amazon shuts down internal AI recruiting tool that would down-rank female applicants.

Severity

Negligible

Harm Distribution Basis

Sex

Harm Type

Psychological harm, Financial harm

AI System Description

Resume screening tool developed by Amazon to scan resumes and raise strong job applicants for consideration

System Developer

Amazon

Sector of Deployment

Professional, scientific and technical activities

Relevant AI functions

Perception, Cognition

AI Techniques

Natural language processing

AI Applications

Natural language processing

Location

Edinburgh, Scotland

Named Entities

Amazon, Edinburgh

Technology Purveyor

Amazon

Beginning Date

2014-01-01

Ending Date

2015-01-01

Near Miss

Near miss

Intent

Accident

Lives Lost

No

Data Inputs

Resumes

Incidents Reports

Amazon has scrapped a “sexist” tool that used artificial intelligence to decide the best candidates to hire for jobs.

Members of the team working on the system said it effectively taught itself that male candidates were preferable.

The artificial intelligence software was created by a team at Amazon’s Edinburgh office in 2014 as a way to automatically sort through CVs and select the most talented applicants.

We’ll tell you what’s true. You can form your own view. From 15p €0.18 $0.18 USD 0.27 a day, more exclusives, analysis and extras.

But the algorithm rapidly taught itself to favour male candidates over female ones, according to members of the team who spoke to Reuters.

They realised it was penalising CVs that included the word “women’s,” such as “women’s chess club captain.” It also reportedly downgraded graduates of two all-women’s colleges.

The problem arose from the fact the system was trained on data submitted by applicants over a 10-year period – much of which was said to have come from men.

Five members of the team who developed the machine learning tool - none of whom wanted to be named publicly - said the system was intended to review job applications and give applicants a score ranging from one to five stars.

Some of the team members pointed to the fact this mirrored the way shoppers rate products on Amazon.

Shape Created with Sketch. World news in pictures Show all 50 left Created with Sketch. right Created with Sketch. Shape Created with Sketch. World news in pictures 1/50 5 April 2019 A refugee father and son lie on railway tracks to prevent a train from leaving a station during a protest in Athens, Greece. Dozens of migrants staged a protest in Athens central train station disrupting all railway services in the hope they will be transported to the Greek border and join other refugees attempting to follow a 2016 migration route towards northern Europe Getty 2/50 4 April 2019 Security agents and police officers hold back migrants during the evacuation of a makeshift camp at Porte de la Chapelle, in the north of Paris. More than 300 migrants and refugees were evacuated on early April 4 from a makeshift camp to accomodation structures AFP/Getty 3/50 3 April 2019 An Alexandra township resident gestures and they part is clashes with the Johannesburg Metro Police, South Africa during a total shutdown of the township due to protest against the lack of service delivery or basic necessities such as access to water and electricity, housing difficulties and lack of public road maintenance. AFP/Getty 4/50 2 April 2019 Children eat next to the debris of damaged homes at Purainiya village in Nepal's southern Bara district near Birgunj, following a rare spring storm. The freak storm tore down houses and overturned cars and trucks as it swept across southern Nepal killing at least 27 people and leaving more than 600 injured AFP/Getty 5/50 1 April 2019 A forensic expert works next to the remains of a small plane that crashed near Erzhausen, Germany. Natalia Fileva, chairwoman and co-owner of Russia's second largest airline S7, died when a private jet she was in crashed near Frankfurt on Sunday, the company said Reuters 6/50 31 March 2019 Ukrainian comic actor and presidential candidate Volodymyr Zelenskiy delivers a speech following the announcement of the first exit poll in a presidential election at his campaign headquarters in Kiev, Ukraine Reuters 7/50 30 March 2019 Catalan pro-independence protesters throw rocks during a counter-demonstration against a protest called by Spanish far-right party Vox against the Catalan independence push in Barcelona. Polls suggest Vox, which campaigns against illegal immigration and "radical" feminism, will become the first far-right party to win seats in the Spanish parliament since the late 70s and could emerge as a kingmaker in Spain's increasingly fragmented political landscape AFP/Getty 8/50 29 March 2019 Protests against President Abdelaziz Bouteflika continue in Algeria despite the announcement on 11 March that he will not run for a fifth Presidential term and postponement of presidential elections previously scheduled for 18 April 2019 until further notice EPA 9/50 28 March 2019 Firefighters on ladders work to extinguish a blaze in an office building in Dhaka after a huge fire tore through it, killing at least five people with many others feared trapped in the latest major fire to hit the Bangladesh capital AFP/Getty 10/50 27 March 2019 A Palestinian protester moves a burning tire during clashes with Israeli troops near the Jewish settlement of Beit El, in the Israeli-occupied West Bank Reuters 11/50 26 March 2019 Palestinians sisters girls look at a destroyed Hamas site close their family's destroyed house after an Israeli air strike in Gaza City. According to reports, Israel continued to launch air strikes on the Gaza Strip overnight after a rocket allegedly fired hit a house near Tel Aviv in central Israel injuring at least seven people EPA 12/50 25 March 2019

Amazon scraps 'sexist AI' recruitment tool

Amazon discovered a problem with using artificial intelligence to hire: their AI was biased against women.

The Seattle-based company developed computer programs designed to filter through hundreds of resumes and surface the best candidates, Reuters reports. Employees had programmed the tool in 2014 using resumes submitted to Amazon over a 10-year period, the majority of which came from male candidates. Based on that information, the tool assumed male candidates were preferable and downgraded resumes from women. In addition to the gender bias, the tool also failed to suggest strong candidates, an Amazon spokesperson told Inc. The company decided to scrap the project early last year.

"They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those," a person familiar with the matter told Reuters. Amazon used the AI only in a trial phase and never relied solely on the recommendations, the spokesperson said.

How Amazon Accidentally Invented a Sexist Hiring Algorithm

Machine learning, one of the core techniques in the field of artificial intelligence, involves teaching automated systems to devise new ways of doing things, by feeding them reams of data about the subject at hand. One of the big fears here is that biases in that data will simply be reinforced in the AI systems—and Amazon seems to have just provided an excellent example of that phenomenon.

According to a new Reuters report, Amazon spent years working on a system for automating the recruitment process. The idea was for this AI-powered system to be able to look at a collection of resumes and name the top candidates. To achieve this, Amazon fed the system a decade’s worth of resumes from people applying for jobs at Amazon.

The tech industry is famously male-dominated and, accordingly, most of those resumes came from men. So, trained on that selection of information, the recruitment system began to favor men over women.

According to Reuters’ sources, Amazon’s system taught itself to downgrade resumes with the word “women’s” in them, and to assign lower scores to graduates of two women-only colleges. Meanwhile, it decided that words such as “executed” and “captured,” which are apparently deployed more often in the resumes of male engineers, suggested the candidate should be ranked more highly.

The team tried to stop the system from taking such factors into account, but ultimately decided that it was impossible to stop it from finding new ways to discriminate against female candidates. There were apparently also issues with the underlying data that led the system to spit out rather random recommendations.

And so, Amazon reportedly killed the project at the start of 2017.

“This was never used by Amazon recruiters to evaluate candidates,” Amazon said in a statement.

Amazon isn’t the only company to be alert to the problem of algorithmic bias. Earlier this year, Facebook said it was testing a tool called Fairness Flow, for spotting racial, gender or age biases in machine-learning algorithms. And what was the first target for Facebook’s tests of the new tool? Its algorithm for matching job-seekers with companies advertising positions.

This article was updated to include Amazon’s statement.

Amazon Killed Its AI Recruitment System For Bias Against Women-Report

Image copyright Getty Images Image caption The algorithm repeated bias towards men, reflected in the technology industry

An algorithm that was being tested as a recruitment tool by online giant Amazon was sexist and had to be scrapped, according to a Reuters report.

The artificial intelligence system was trained on data submitted by applicants over a 10-year period, much of which came from men, it claimed.

Reuters was told by members of the team working on it that the system effectively taught itself that male candidates were preferable.

Amazon has not responded to the claims.

Reuters spoke to five members of the team who developed the machine learning tool in 2014, none of whom wanted to be publicly named.

They told Reuters that the system was intended to review job applications and give candidates a score ranging from one to five stars.

"They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those," said one of the engineers who spoke to Reuters.

'Women' penalised

By 2015, it was clear that the system was not rating candidates in a gender-neutral way because it was built on data accumulated from CVs submitted to the firm mostly from males, Reuters claimed.

The system started to penalise CVs which included the word "women". The program was edited to make it neutral to the term but it became clear that the system could not be relied upon, Reuters was told.

The project was abandoned, although Reuters said that it was used for a period by recruiters who looked at the recommendations generated by the tool but never relied solely on it.

According to Amazon, its current global workforce is split 60:40 in favour of males.

About 55% of US human resources managers said that AI would play a role in recruitment within the next five years, according to a survey by software firm CareerBuilder.

It is not the first time doubts have been raised about how reliable algorithms trained on potentially biased data will be.

Image copyright MIT Image caption An MIT AI system, dubbed Norman, had a dark view of the world as a result of the data it was trained on

An experiment at the Massachusetts Institute of Technology, which trained an AI on images and videos of murder and death, found it interpreted neutral inkblots in a negative way.

And in May last year, a report claimed that an AI-generated computer program used by a US court was biased against black people, flagging them as twice as likely to reoffend as white people.

Predictive policing algorithms were spotted to be similarly biased, because the crime data they were trained on showed more arrests or police stops for black people.

Amazon scrapped 'sexist AI' tool

Amazon has scrapped a "sexist" internal tool that used artificial intelligence to sort through job applications.

The AI was created by a team at Amazon's Edinburgh office in 2014 as a way to automatically sort through CVs and pick out the most promising candidates.

However, it quickly taught itself to prefer male candidates over female ones, according to members of the team who spoke to Reuters.

They noticed that it was penalising CVs that included the word "women's," such as "women's chess club captain." It also reportedly downgraded graduates of two all-women's colleges.

The problem stemmed from the fact that the system was trained on data submitted by people over a 10-year period, most of which came from men.

The AI was tweaked in an attempt to fix the bias. However, last year, Amazon lost faith in its ability to be neutral and abandoned the project altogether.

Amazon recruiters are believed to have used the system to look at the recommendations when hiring, but didn't rely on the rankings. Currently, women make up 40pc of Amazon's workforce.

Stevie Buckley, the co-founder of UK job website Honest Work, which is used by companies such as Snapchat to recruit for technology roles, said that “the basic premise of expecting a machine to identify strong job applicants based on historic hiring practices at your company is a surefire method to rapidly scale inherent bias and discriminatory recruitment practices.”

Amazon scraps 'sexist AI' recruiting tool that showed bias against women

What is artificial intelligence (AI)? We look at the progress of AI and automation in Australia compared to the rest of the world and how the Australian workforce may be affected by this movement.

Will the rise of AI take away our jobs? 0:57

AMAZON was forced to abandon a secret artificial intelligence recruiting tool after discovering it was discriminating against women.

According to a report in Reuters, since 2014 Amazon engineers have been building a computer program to review resumes with the goal of automating the talent search process.

The tool would give job candidates a score from one to five stars.

“Everyone wanted this holy grail,” one source told the news agency. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

After a year, however, Amazon realised its system was favouring male candidates for software developer and other technical roles, because it was observing patterns in resumes submitted over a 10-year period — most of which came from men.

It also penalised resumes that included the word “women’s”, according to Reuters, such as in the phrase “women’s chess club captain” and all-women’s colleges.

Even though the program was edited to make it neutral to those terms, the programmers couldn’t guarantee the AI would not teach itself to sort candidates in other discriminatory ways, the report said.

The project was eventually scrapped altogether in early 2017.

It’s understood the project was only ever used in a developmental phase, never independently, and never rolled out to a larger group.

It was abandoned for many reasons — it never returned strong candidates for the roles — and not because of the bias issue.

An Amazon spokeswoman said, “This was never used by Amazon recruiters to evaluate candidates.”

frank.chung@news.com.au

Amazon scraps ‘sexist’ AI hiring tool

London | Amazon has scrapped a "sexist" internal tool that used artificial intelligence to sort through job applications.

The program was created by a team at Amazon's Edinburgh office in 2014 as a way to sort through CVs and pick out the most promising candidates. However, it taught itself to prefer male candidates over female ones, members of the team told Reuters.

They noticed that it was penalising CVs that included the word "women's", such as "women's chess club captain". It also reportedly downgraded graduates of two all-women's colleges.

Amazon CEO Jeff Bezos. Women make up 40 per cent of Amazon's workforce. AP

The problem stemmed from the fact that the system was trained on data submitted by people over a 10-year period, most of which came from men.

The AI was tweaked in an attempt to fix the bias. However, last year, Amazon lost faith in its ability to be neutral and abandoned the project. Amazon recruiters are believed to have used the system to look at the recommendations when hiring, but did not rely on the rankings. Currently, women make up 40 per cent of Amazon's workforce.

Amazon ditches AI recruitment tool that 'learnt to be sexist'

Amazon sign, with dude. David Ryder/Getty Images

Thanks to Amazon, the world has a nifty new cautionary tale about the perils of teaching computers to make human decisions.

According to a Reuters report published Wednesday, the tech giant decided last year to abandon an “experimental hiring tool” that used artificial intelligence to rate job candidates, in part because it discriminated against women. Recruiters reportedly looked at the recommendations the program spat out while searching for talent, “but never relied solely on those rankings.”

The misadventure began in 2014, when a group of Amazon engineers in Scotland set out to mechanize the company’s head-hunting process, by creating a program that would scour the Internet for worthwhile job candidates (and presumably save Amazon’s HR staff some soul crushing hours clicking around LinkedIn). “Everyone wanted this holy grail,” a source told Reuters. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

It didn’t pan out that way. In 2015, the team realized that its creation was biased in favor of men when it came to hiring technical talent, like software developers. The problem was that they trained their machine learning algorithms to look for prospects by recognizing terms that had popped up on the resumes of past job applicants—and because of the tech world’s well-known gender imbalance, those past hopefuls tended to be men.

“In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word ‘women’s,’ as in ‘women’s chess club captain.’ And it downgraded graduates of two all-women’s colleges,” Reuters reported. The program also decided that basic tech skills, like the ability to write code, which popped up on all sorts of resumes, weren’t all that important, but grew to like candidates who littered their resumes with macho verbs such as “executed” and “captured.”

After years of trying to fix the project, Amazon brass reportedly “lost hope“ and shuttered the effort in 2017.

All of this is a remarkably clear-cut illustration of why many tech experts are worried that, rather than remove human biases from important decisions, artificial intelligence will simply automate them. An investigation by ProPublica, for instance, found that algorithms judges use in criminal sentencing may dole out harsher penalties to black defendants than white ones. Google Translate famously introduced gender biases into its translations. The issue is that these programs learn to spot patterns and make decisions by analyzing massive data sets, which themselves are often a reflection of social discrimination. Programmers can try to tweak the A.I. to avoid those undesirable results, but they may not think to, or be successful even if they try.

Amazon deserves some credit for realizing its tool had a problem, trying to fix it, and eventually moving on (assuming it didn’t have a serious impact on the company’s recruiting over the last few years). But, at a time when lots of companies are embracing artificial intelligence for things like hiring, what happened at Amazon really highlights that using such technology without unintended consequences is hard. And if a company like Amazon can’t pull it off without problems, it’s difficult to imagine that less sophisticated companies can.

Amazon's AI hiring tool discriminated against women.

Algorithms are often pitched as being superior to human judgement, taking the guesswork out of decisions ranging from driving to writing an email. But they're still programmed by humans and trained on the data that humans create, which means they are tied to us for better or worse. Amazon found this out the hard way when the company's AI recruitment software, trained to review job applications, turned out to discriminate against women applicants.

In place since 2014, the software was built to find the top talent by digging through mountains of applications. The AI would rate applicants on a scale of 1 to 5 stars, like you might rate a product on Amazon.

“Everyone wanted this holy grail,” a person involved with the algorithm tells Reuters. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

The model was trained to look at Amazon hiring patterns for software developer jobs and technical position over the last decade. While on the surface this makes sense—in the last 10 years Amazon has grown tremendously, a good sign that it has hired the right people—in practice it only reproduced the sexist biases already in place. Most of the hires over the last 10 years had, in fact, been men, and the algorithm began taking this into account.

It began to penalize resumes that included the word "women," meaning phrases like "volunteered with Women Who Code" would be marked against the applicant. It specifically targeted two all-women's colleges, although sources would not tell Reuters which ones.

The company was able to edit the algorithm to eliminate these two particular biases. But a larger question arose—what other biases was the AI reinforcing that weren't quite so obvious? There was no way to be sure. After several attempts to correct the program, Amazon executives eventually lost interest in 2017. The algorithm was abandoned.

The incident shows that because humans are imperfect, their imperfections can get baked into the algorithms built in hopes of avoiding such problems. AIs can do things we might never dream of doing ourselves, but we can never ignore a dangerous and unavoidable truth: They have to learn from us.

UPDATE, Oct 11: Amazon reached out through a spokesperson to PopMech with a statement, saying that “This was never used by Amazon recruiters to evaluate candidates.”

Source: Reuters

Amazon Fired Its Resume-Reading AI for Sexism

Machine learning technology is becoming increasingly common across various industries, from policing to recruiting. But reports have shown that many of these systems have long-standing problems regarding discrimination. To avoid amplifying bias, companies need to actively teach their technology to be inclusive.

There are several ways corporations can improve upon their machine learning tools. Quartz suggests assessing the wider impacts of new AI systems before implementation, as well as establishing internal codes of conduct and incentive models to enhance adherence to non-discriminatory practices. The publication also states that inclusivity and diversity should be made priorities early on, starting from the development of the design teams through the final product.

It's also important for companies to be transparent about the impact of their technology and to constantly evaluate its effectiveness, from refining algorithms to evaluating and reporting its behavior. By taking these proactive steps, there's potential for forward-thinking companies to create revolutionary AI systems without posing a risk to human rights.

Amazon Accidentally Created A 'Sexist' Recruitment Tool, Then Shut It Down

Why Global Citizens Should Care

Gender discrimination in the workplace prevents women from achieving to their full potential. Eliminating gender inequality in the workforce would greatly increase economic activity. When half of the population is held back, we’re all held back. You can join us by taking action here to take a stand for true gender equality.

Tech giant Amazon has abandoned an artificial intelligence (AI) tool it had been building for three years after determining that the system was discriminating against women, reports Reuters.

The AI tool, intended to help with recruitment by trawling for candidates online, reportedly downgraded résumés containing the word "women's" and filtered out potential hires who had attended two women-only colleges, noted Business Insider.

Take Action: Sign the petition calling on influential companies to support women-owned businesses.

“Everyone wanted this holy grail,” one source told Reuters. "They literally wanted it to be an engine where I'm going to give you 100 résumés, it will spit out the top five, and we'll hire those."

The AI tool was built using past résumés submitted to Amazon over a 10-year period as a reference point for hiring, Business Insider reported. Because these résumés were predominantly submitted by male applicants, the tool perpetuated this pattern and developed a bias against female hires, presuming male candidates were preferable.

Read More: This Trailblazing South African Pilot Is Now Working to Get Girls Into Science

While engineers attempted to tweak the system, glitches remained and executives lost faith in pursuing the project by early 2017, according to the Reuters report.

The case study sets a dismal precedent for other companies hoping to harness similar technology in the near future. According to a 2017 survey by talent software firm CareerBuilder, approximately 55% of US human resources managers said AI would be a regular part of their work within the next five years.

“How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable — that’s still quite far off,” said Nihar Shah, a computer scientist who teaches machine learning at Carnegie Mellon University, in an interview with Reuters.

John Jersin, vice president of LinkedIn Talent Solutions, also told Reuters that he didn’t see the service as a replacement for traditional recruiters.

Read More: Your Wedding Could Help End Child Marriage

“I certainly would not trust any AI system today to make a hiring decision on its own,” he said. “The technology is just not ready yet.”

Amazon is reportedly now testing a new version of the automated employment screening, focused on diversity.

Amazon Shuts Down AI Hiring Tool for Being Sexist

SAN FRANCISCO (Reuters) - Amazon.com Inc’s (AMZN.O) machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.

The team had been building computer programs since 2014 to review job applicants’ resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters.

Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions. The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars - much like shoppers rate products on Amazon, some of the people said.

“Everyone wanted this holy grail,” one of the people said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.

That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.

In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.

Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.

The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity. Amazon’s recruiters looked at the recommendations generated by the tool when searching for new hires, but never relied solely on those rankings, they said.

Amazon declined to comment on the technology’s challenges, but said the tool “was never used by Amazon recruiters to evaluate candidates.” The company did not elaborate further. It did not dispute that recruiters looked at the recommendations generated by the recruiting engine.

The company’s experiment, which Reuters is first to report, offers a case study in the limitations of machine learning. It also serves as a lesson to the growing list of large companies including Hilton Worldwide Holdings Inc (HLT.N) and Goldman Sachs Group Inc (GS.N) that are looking to automate portions of the hiring process.

Some 55 percent of U.S. human resources managers said artificial intelligence, or AI, would be a regular part of their work within the next five years, according to a 2017 survey by talent software firm CareerBuilder.

Employers have long dreamed of harnessing technology to widen the hiring net and reduce reliance on subjective opinions of human recruiters. But computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.

“How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that’s still quite far off,” he said.

FILE PHOTO: Brochures are available for potential job applicants at "Amazon Jobs Day," a job fair at the Amazon.com Fulfillment Center in Fall River, Massachusetts, U.S., August 2, 2017. REUTERS/Brian Snyder/File Photo

MASCULINE LANGUAGE

Amazon’s experiment began at a pivotal moment for the world’s largest online retailer. Machine learning was gaining traction in the technology world, thanks to a surge in low-cost computing power. And Amazon’s Human Resources department was about to embark on a hiring spree: Since June 2015, the company’s global headcount has more than tripled to 575,700 workers, regulatory filings show.

So it set up a team in Amazon’s Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.

The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates’ resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes, the people said.

Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said.

Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology retur

Amazon scraps secret AI recruiting tool that showed bias against women

David Ryder/Getty Images Amazon CEO Jeff Bezos.

Amazon tried building an artificial-intelligence tool to help with recruiting, but it showed a bias against women,Reuters reports.

Engineers reportedly found the AI was unfavorable toward female candidates because it had combed through male-dominated résumés to accrue its data.

Amazon reportedly abandoned the project at the beginning of 2017.

Amazon worked on building an artificial-intelligence tool to help with hiring, but the plans backfired when the company discovered the system discriminated against women, Reuters reports.

Citing five sources, Reuters said Amazon set up an engineering team in Edinburgh, Scotland, in 2014 to find a way to automate its recruitment.

The company created 500 computer models to trawl through past candidates’ résumés and pick up on about 50,000 key terms. The system would crawl the web to recommend candidates.

“They literally wanted it to be an engine where I’m going to give you 100 résumés, it will spit out the top five, and we’ll hire those,” one source told Reuters.

A year later, however, the engineers reportedly noticed something troubling about their engine – it didn’t like women. This was apparently because the AI combed through predominantly male résumés submitted to Amazon over a 10-year period to accrue data about whom to hire.

Consequently, the AI concluded that men were preferable. It reportedly downgraded résumés containing the words “women’s” and filtered out candidates who had attended two women-only colleges.

Amazon’s engineers apparently tweaked the system to remedy these particular forms of bias but couldn’t be sure the AI wouldn’t find new ways to unfairly discriminate against candidates.

Gender bias was not the only problem, Reuters’ sources said. The computer programs also spat out candidates who were unqualified for the position.

Remedying algorithmic bias is a thorny issue, as algorithms can pick up on subconscious human bias. In 2016, ProPublica found that risk-assessment software used to forecast which criminals were most likely to reoffend exhibited racial bias against black people. Overreliance on AI for things like recruitment, credit-scoring, and parole judgments have also created issues in the past.

Amazon reportedly abandoned the AI recruitment project by the beginning of last year after executives lost faith in it. Reuters’ sources said Amazon recruiters looked at recommendations generated by the AI but never relied solely on its judgment.

Amazon told Business Insider but declined to comment further.

An Amazon spokesperson told Business Insider, “This was never used by Amazon recruiters to evaluate candidates” and that the company was committed to workplace diversity and equality

Business Insider Emails & Alerts Site highlights each day to your inbox. Email Address Join

Follow Business Insider Australia on Facebook, Twitter, LinkedIn, and Instagram.

Amazon built an AI tool to hire people but had to shut it down because it was discriminating against women

information-age.com · 2018

Amazon ditches sexist AI

It’s not news to learn that AI can be something of a bigot.

Amazon scrapped an algorithm designed to become a recruitment tool because it was too sexist.

Did you hear the one about my wife — well, she… is a really nice person, actually.

We know that people suffer from bias. Alas, a growing pile of evidence suggests AI can be too.

Now it seems that Amazon has found this out the hard way — after investing in an AI recruitment tool.

Ethical AI – the answer is clear Being transparent with ethical AI is vital to engaging with the public in a responsible manner.

The idea was for the AI engine to scan job applications and give hopeful recruits a score between one and five. Reuters quoted one engineer saying: “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

Alas, it started weeding out CVs that included a certain five letter word. The ‘W’ word — women, there said it.

This was back in 2015, let’s face it, as far as AI is concerned, 2015 is ancient history.

See also: Regulating robots: keeping an eye on AI

It’s not news to learn that AI can be something of a bigot.

In 2016, it emerged that US risk assessment algorithms — used by courtrooms throughout the country to decide the fates and freedoms of those on trial – are racially biased, frequently sentencing Caucasians more leniently than African Americans despite no difference in the type of crime committed. How could this happen within a system that’s supposed to be neutral?

AI researcher Professor Joanna Bryson, said at the time: “If the underlying data reflects stereotypes, or if you train AI from human culture, you will find bias.”

See also: Augmented intelligence: why the human element can’t be forgotten

This brings us to the issue of diversity. Scot E Page is an expert on diversity and complex systems. Areas where he is most well-known include ‘collective wisdom’.

He is famous for saying “progress depends as much on our collective differences as it does on our individual IQ scores.”

And: “If we can understand how to leverage diversity to achieve better performance and greater robustness, we might anticipate and prevent collapses.”

AI, however, because of the way it learns from data, can reflect the biases in society.

“The fact that Amazon’s system taught itself that male candidates were preferable, penalising resumes that included the word ‘women’s’, is hardly surprising when you consider 89% of the engineering workforce is male,” observed Charlotte Morrison, General Manager of global branding and design agency, Landor.

She added: “Brands need to be careful that when creating and using technology it does not backfire by highlighting society’s own imperfections and prejudices. The long-term solution is of course getting more diverse candidates into STEM education and careers – until then, brands need to be alert to the dangers of brand and reputational damage from biased, sexist, and even racist technology.”

See also: Augmented intelligence: predicting the best customer moments

Amazon ditches sexist AI

Amazon had to scrap its AI hiring tool because it was ‘sexist’ and discriminated against female applicants, a report from Reuters has found.

Amazon’s hopes for creating the perfect AI hiring tool were dashed when it realised that the algorithm contained one huge, glaring error: it was “sexist”, according to a report released by Reuters yesterday (10 October).

Reuters reporter Jeffrey Dastin spoke to five machine-learning specialists, all of whom elected to remain anonymous. In 2014, these programmers were all working on hiring algorithms to sift through job applications for use in a recruitment tool. The tool used AI technology such as machine learning to rate each applicant between one and five stars, much in the same way Amazon products are rated.

Amazon had hoped the tool would be a “holy grail”, according to one of the programmers. “They literally wanted it to be an engine where I’m going to give you 100 résumés, it will spit out the top five and we’ll hire those.”

However, by 2015, it became apparent that the algorithm was discriminating based on gender. The computer models were trained to rate candidates by analysing patterns in résumés submitted to Amazon over the past 10-year period. Given that the tech industry has historically been, and continues to be, male-dominated, most of the applications the computer observed came from men.

The system then taught itself that male candidates were preferred, and penalised applications that included the word ‘women’s’, such as in ‘women’s basketball team’ or ‘women’s chess club’. The programmers interviewed by Reuters said that the AI downgraded graduates of two all-women colleges. The programmers did not specify which colleges these applicants came from.

While the company did attempt to edit the programs to make these terms appear neutral, it couldn’t guarantee that the tool wouldn’t continue to discriminate against women. The team was disbanded in 2015 after executives “lost hope” for the project, Reuters reported.

Amazon said that the tool “was never used by Amazon recruiters to evaluate candidates” but stated that recruiters did look at the recommendations generated by the tool while hiring for positions at the company. The company declined to comment further on the matter.

The HR community is becoming increasingly interested in the potential of AI to help speed up the recruitment process. In early 2018, a report on emerging HR trends released by LinkedIn found that 76pc of HR professionals acknowledged that AI will be at least somewhat significant to recruitment work in coming years. AIs were rated as most helpful at sourcing, screening and scheduling candidates.

This recent development, however, may deflate AI enthusiasts hoping to roll out these technologies in their enterprises in the near future. As AIs are only as good as the sum of the data they are given, they will likely continue to reflect bias in a way that mirrors the diversity issues that exist in the real working world.

Yet some would still argue that AI is the solution to human-generated biases. A HR study released by IBM found that recruiters spend as little as six seconds looking at a résumé that comes across their desk. IBM argues that when making decisions based purely on instant impressions, HR professionals are more inclined to fall back on their own implicit bias, something an AI could correct by helping to expedite the initial hiring stages.

In any case, many of those working in HR harbouring fears that an AI may replace their job will likely breathe a sigh of relief knowing that the technology still has a long way to go before it totally supplants human workers.

Amazon building in Santa Clara, California. Image: wolterke/Depositphotos

It turns out Amazon’s AI hiring tool discriminated against women

Amazon has been forced to scrap its AI recruitment system after it was discovered to be biased against female applicants.

The AI was developed in 2014 by Amazon as a way of filtering out most candidates to provide the firm with the top five people for a position. In 2015 it was found that it wasn't rating applicants in a gender-neutral way, which is a big problem and goes against Amazon's attempts to level the playing field by having an objective AI do the early decision making.

As it turns out, the problem lay in how the system was trained. As with everything in AI, a lack of diversity in the industry led it to be trained almost entirely upon male CVs. This meant that, as it was learning to detect the patterns in recruiting over a 10-year period, it was also learning to devalue the CVs of women.

READ NEXT: What can we do about tech's diversity problem?

According to Reuters, the system taught itself that male candidates were preferable to women. It downgraded CVs if found words such as “women’s” and penalised graduates of all-female colleges.

While Amazon recoded the software to make the AI neutral to these terms, it realised that this did not guarantee that the technology would find other methods of being discriminatory against women, the report said.

The team, set up in Amazon’s Edinburgh engineering hub, created 500 models concentrated on detailed job functions and locations. The system was also taught to recognise around 50,000 terms that showed up on past candidates’ CVs.

The technology learned to assign little importance to skills common across IT applicants, favouring terms more commonly found on male engineers’ resumes, such as “executed” and “captured,” the report said.

READ NEXT: Do diversity quotas help or hinder women in tech?

The model used in the AI system had other problems that led to unqualified candidates being recommended for a variety of unsuitable jobs.

This eventually led to Amazon pulling the plug on the team as executives “lost hope” over the project, anonymous sources told Amazon. The tool could not be solely relied upon to sort candidates.

The firm now uses a “much-watered down version” of the recruiting engine to carry out “rudimentary chores”.

Amazon's AI recruitment tool scrapped for being sexist

channels.theinnovationenterprise.com · 2018

Amazon decided to scrap a machine learning (ML) algorithm it was creating to help automate the recruitment process because the model kept favoring male candidates, Reuters revealed. The discrimination against female candidates has been put down to the largely male-dominated data sets it had been trained with.

The project, which was scrapped in 2017 was meant to be able to review job applications and assign a score to each candidate between one and five stars. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those," claimed one of the five team members who had worked on the tool that Reuters spoke to.

Visit Innovation Enterprise's Machine Learning Innovation Summit in New York on December 12–13, 2018

The team had worked on the recruitment algorithm since 2014, training it on resume's that covered a 10-year period. However, because the tech industry notoriously male-dominated, most of the resumes it was trained on came from men. This led the AI to begin favoring male candidates in its assessment simply by virtue of them being male, penalizing CV's simply for featuring the word "women".

Concerns around the impact biased data sets are having on AI training is becoming more and more of an issue as AI research continues to accelerate. Earlier this year, MIT researchers attempted to illustrate the impact datasets can have by creating the world's first psychopath AI . This incident with Amazon shows how easy it is to inadvertently pass on biases to the tech they are training for the explicit purpose of being impartial.

Amazon has so far declined to comment on the Reuters report.

Amazon abandoned sexist AI recruitment tool

Specialists had been building computer programs since 2014 to review résumés in an effort to automate the search process

This article is more than 5 months old

This article is more than 5 months old

Amazon’s machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.

The team had been building computer programs since 2014 to review job applicants’ résumés, with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters.

Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions. The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars – much as shoppers rate products on Amazon, some of the people said.

“Everyone wanted this holy grail,” one of the people said. “They literally wanted it to be an engine where I’m going to give you 100 résumés, it will spit out the top five, and we’ll hire those.”

But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.

Automation could destroy millions of jobs. We have to deal with it now | Yvette Cooper Read more

That is because Amazon’s computer models were trained to vet applicants by observing patterns in résumés submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.

In effect, Amazon’s system taught itself that male candidates were preferable. It penalized résumés that included the word “women’s”, as in “women’s chess club captain”. And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter.

Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.

The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity. Amazon’s recruiters looked at the recommendations generated by the tool when searching for new hires, but never relied solely on those rankings, they said.

Amazon declined to comment on the recruiting engine or its challenges, but the company says it is committed to workplace diversity and equality.

The company’s experiment, which Reuters is first to report, offers a case study in the limitations of machine learning. It also serves as a lesson to the growing list of large companies including Hilton Worldwide Holdings and Goldman Sachs that are looking to automate portions of the hiring process.

Franken-algorithms: the deadly consequences of unpredictable code Read more

Some 55% of US human resources managers said artificial intelligence, or AI, would be a regular part of their work within the next five years, according to a 2017 survey by talent software firm CareerBuilder.

Masculine language

Amazon’s experiment began at a pivotal moment for the world’s largest online retailer. Machine learning was gaining traction in the technology world, thanks to a surge in low-cost computing power. And Amazon’s Human Resources department was about to embark on a hiring spree; since June 2015, the company’s global headcount has more than tripled to 575,700 workers, regulatory filings show.

So it set up a team in Amazon’s Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.

The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that were found on past candidates’ résumés. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes, the people said.

Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured”, one person said.

Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

The problem or the cure?

Other companies are forging ahead, underscoring the eagerness of employers to harness AI for hiring.

Kevin Parker, chief executive of HireVue, a startup near Salt Lake City, said automation is helping companies look beyond the same recruiting networks upon which they have long relied. His firm analyzes candidates’ speech and facial expressions in video interviews to reduce reliance on résumés.

“You weren’t going back to the same old places; yo

Amazon ditched AI recruiting tool that favored men for technical jobs

AI may have sexist tendencies. But, sorry, the problem is still us humans.

Amazon recently scrapped an employee recruiting algorithm plagued with problems, according to a report from Reuters. Ultimately, the applicant screening algorithm did not return relevant candidates, so Amazon canned the program. But in 2015, Amazon had a more worrisome issue with this AI: it was down-ranking women.

The algorithm was only ever used in trials, and engineers manually corrected for the problems with bias. However, the way the algorithm functioned, and the existence of the product itself, speaks to real problems about gender disparity in tech and non-tech roles, and the devaluation of perceived female work.

Amazon created its recruiting AI to automatically return the best candidates out of a pool of applicant resumes. It discovered that the algorithm would down-rank resumes when it included the word "women's," and even two women's colleges. It would also give preference to resumes that contained what Reuters called "masculine language," or strong verbs like "executed" or "captured."

These patterns began to appear because the engineers trained their algorithm with past candidates' resumes submitted over the previous ten years. And lo and behold, most of the most attractive candidates were men. Essentially, the algorithm found evidence of gender disparity in technical roles, and optimized for it; it neutrally replicated a societal and endemic preference for men wrought from an educational system and cultural bias that encourages men and discourages women in the pursuit of STEM roles.

For clues on why there are so few women in tech, watch a recruiting session for college seniors. Few women engineers, even fewer speak, men talking over them https://t.co/8CmfAB9J3Q via @jessiwrites — Scott Thurm (@ScottThurm) March 1, 2018

As #womeninSTEM & #womeninPhysics, we are a minority & in my experience this has had a massive impact on my mental health over time.https://t.co/KPMcCMbt7X

I feel like an impostor. I often feel isolated. I regularly fear discrimination & even harassment.#WorldMentalHealthDay — Niamh Kavanagh (@NiamhTalking90) October 10, 2018

Amazon emphasized in an email to Mashable that it scrapped the program because it was ultimately not returning relevant candidates; it dealt with the sexism problem early on, but the AI as a whole just didn't work that well.

However, the creation of hiring algorithms themselves — not just at Amazon, but across many companies — still speaks to another sort of gender bias: the devaluing of female-dominated Human Resources roles and skills.

According to the U.S. Department of Labor (via the workforce analytics provider company Visier), women occupy nearly three fourths of H.R. managerial roles. This is great news for overall female representation in the workplace. But the disparity exists thanks to another sort of gender bias.

There is a perception that H.R. jobs are feminine roles. The Globe and Mail writes in its investigation of sexism and gender disparity in HR:

The perception of HR as a woman's profession persists. This image that it is people-based, soft and empathetic, and all about helping employees work through issues leaves it largely populated by women as the stereotypical nurturer. Even today, these "softer" skills are seen as less appealing – or intuitive – to men who may gravitate to perceived strategic, analytical roles, and away from employee relations.

Amazon and other companies that pursued AI integrations in hiring wanted to streamline the process, yes. But automating a people-based process shows a disregard for people-based skills that are less easy to mechanically reproduce, like intuition or rapport. Reuters reported that Amazon's AI identified attractive applicants through a five-star rating system, "much like shoppers rate products on Amazon"; who needs empathy when you've got five stars?

In Reuters' report, these companies suggest hiring AI as a compliment or supplement to more traditional methods, not an outright replacement. But the drive in the first place to automate a process by a female-dominated division shows the other side of the coin of the algorithm's preference for "male language"; where "executed" and "captured" verbs are subconsciously favored, "listened" or "provided" are shrugged off as inefficient.

The AI explosion is underway. That's easy to see in every evangelical smart phone or smart home presentation of just how much your robot can do for you, including Amazon's. But that means that society is opening itself up to create an even less inclusive world. A.I. can double down on discriminatory tendencies in the name of optimization, as we see with Amazon's recruiting A.I. (and others). And because A.I. is both built and led by humans (and often, mostly male humans) who may unintentionally transfer their unconscious sexist biases into business decisions, and the robots themselves.

So as our computers get smarter and permeate more areas of life and w

Amazon's sexist recruiting algorithm reflects a larger gender bias

Amazon trained a sexism-fighting, resume-screening AI with sexist hiring data, so the bot became sexist

Some parts of machine learning are incredibly esoteric and hard to grasp, surprising even seasoned computer science pros; other parts of it are just the same problems that programmers have contended with since the earliest days of computation. The problem Amazon had with its machine-learning-based system for screening job applicants was the latter.

Amazon understood that it had a discriminatory hiring process: the unconscious biases of its technical leads resulted in the company passing on qualified woman applicants. This isn't just unfair, it's also a major business risk, because qualified developers are the most scarce element of modern businesses.

So they trained a machine-learning system to evaluate incoming resumes, hoping it would overcome the biases of the existing hiring system.

Of course, they trained it with the resumes of Amazon's existing stable of successful job applicants -- that is, the predominantly male workforce that had been hired under the discriminatory system they hoped to correct.

The computer science aphorism to explain this is "garbage in, garbage out," or GIGO. It is pretty self-explanatory, but just in case, GIGO is the phenomenon in which bad data put through a good system produces bad conclusions.

Amazon built the system in 2014 and scrapped it in 2017, after concluding that it was unsalvagable -- sources told Reuters that it rejected applicants from all-woman colleges, and downranked resume's that included the word "women's" as in "women's chess club captain." Amazon says it never relied on the system.

There is a "machine learning is hard" angle to this: while the flawed outcomes from the flawed training data was totally predictable, the system's self-generated discriminatory criteria were surprising and unpredictable. No one told it to downrank resumes containing "women's" -- it arrived at that conclusion on its own, by noticing that this was a word that rarely appeared on the resumes of previous Amazon hires.

The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates’ resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes, the people said. Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said. Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

Amazon scraps secret AI recruiting tool that showed bias against women [Jeffrey Dastin/Reuters]

(Image: Cryteria, CC-BY)

Amazon trained a sexism-fighting, resume-screening AI with sexist hiring data, so the bot became sexist

Artificial intelligence (AI) human resourcing tools are all the rage at the moment and becoming increasingly popular. The systems can speed up, simplify and even decrease the cost of the hiring process becoming every recruiter's dream come true.

But as we have witnessed before, AI-powered systems can also, at times, exhibit potentially dangerous biased tendencies. Last July, non-profit watchdog the American Civil Liberties Union of Northern California was shocked to find flaws in Amazon's facial recognition technology called Rekognition that could possibly lead to racially-based false identifications.

The discovery led to public outcry regarding the system. However, Amazon defended its program and said the deployment of new tech could not be thwarted because some "could choose to abuse" it, implying any issues with Rekognition were related to user misuse.

Sexist AI scrapped

Now, it seems another AI tool, this time a human resources one, has somehow taught itself to be sexist. According to a report by Reuters, a secret internal project by Amazon, that was trying to use AI to vet job applications, had to be scrapped after it was found to be downgrading female candidates.

AI Politicians Angry After Amazon Facial Recognition AI Falsely Matches 28 Congress Members to Criminals

Amazon's machine learning experts had been working on computer programs since 2014 that would be able to review and sort out applicants’ resumes. The system worked by assigning potential candidates scores ranging from one to five stars.

“Everyone wanted this holy grail,” one of the people working on the AI project told Reuters. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

However, by 2015, the specialists found the system was making problematic gender-related candidate choices when it came to software developers and other technical positions. The problem, believe the experts, arose because the computer models vetting process was derived from past hiring patterns achieved over a 10-year period.

Teaching itself sexism

And since that period saw the tech industry be particularly male-dominated, the model inadvertently trained itself to prefer male candidates over female ones. It essentially trained itself to be sexist.

The program has reportedly since been scrapped, eliminating any potential associated negative consequences. But the story illustrates the dangers of relying on past data to create new models.

Advertisement

"Every human on the planet is biased. We all carry an incurable virus called our unconscious bias. I firmly believe that AI is the only way we can reduce human bias with recruiting. Unfortunately, naively using AI for recruiting is a guaranteed recipe for disaster. Racist/Sexist AI isn’t a risk, it is a guarantee if used by inexperienced teams. AI will naturally learn our bias and amplify it," explained to IE chief AI officer at Ziff Ben Taylor.

Still, one should not throw out the baby with the bathwater. There are ways to get the benefits of AI without the biases. "There are plenty of companies outside the media spotlight using AI responsibly, where they have spent millions in adverse impact protections/research to prevent something like this from happening," further added Taylor.

Advertisement

From detecting cancers to making treatments less toxic, the technology seems to be finding increasingly new and very helpful uses. And with organizations seeking to curb and tame any potentially unpleasant outcomes, it is a good guess that, if embraced properly, AI can be one of humanity's biggest assets.

We just have to remain vigilant and aware. Judging by the swift disposal of Amazon's misogynistic recruiting tool, we seem to be on track to keeping our AI mankind-friendly.

Amazon Shuts Down Secret AI Recruiting Tool That Taught Itself to be Sexist

Amazon's has scraped its artificial intelligence hiring tool after it was found to be sexist.

Photo: © 2014, Ken Wolter

A team of specialists familiar with the project told Reuters that they had been building computer programmes since 2014 to review job applicants' resumes, using artificial intelligence to give job candidates scores ranging from one to five stars.

However, by 2015 the company realised its new system was not rating candidates for software developer jobs and technical posts in a gender-neutral way.

The computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period, most of which came from men, a reflection of male dominance across the tech industry.

The system penalised resumes that include the world "women's" as in "women's chess club captain" and it downgraded graduates of two all-women's colleges, according to people familiar with the matter.

Amazon's experiment began at a pivotal moment for the world's largest online retailer. Machine learning was gaining traction in the technology world, thanks to a surge in low-cost computing power and Amazon's Human Resources department was about to embark on a hiring spree.

Since June 2015, the company's global headcount has more than tripled to 575,700 workers, regulatory filings show.

So it set up a team in Amazon's Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.

The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates' resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes, the people said.

Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers' resumes, such as "executed" and "captured," one person said.

Gender bias was not the only issue. Problems with the data that underpinned the models' judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.

Amazon declined to comment on the technology's challenges, but said the tool "was never used by Amazon recruiters to evaluate candidates." The company did not elaborate further. It did not dispute that recruiters looked at the recommendations generated by the recruiting engine.

The company's experiment, which Reuters is first to report, offers a case study in the limitations of machine learning. It also serves as a lesson to the growing list of large companies that are looking to automate portions of the hiring process.

Some 55 percent of U.S. human resources managers said artificial intelligence, or AI, would be a regular part of their work within the next five years, according to a 2017 survey by talent software firm CareerBuilder.

Employers have long dreamed of harnessing technology to widen the hiring net and reduce reliance on subjective opinions of human recruiters. But computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.

"How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable - that's still quite far off," he said.

Amazon scraps sexist AI recruiting tool

insights.tmpw.co.uk · 2018

So AI may be the future in hiring and recruitment but it certainly isn't there yet it seems.

If you're basing it's learning on history which quite possibly may have been biased towards men, then it is likely that it will discriminate against females. Well that is what Amazon have now found out.

The technology which was trained on data from the previous 10 years, much of which came from men, started to filter out female CVs.

Even when they recoded the software to try and make the AI neutral, this didn't guarantee that the technology would not find other ways to discriminate. Such as looking for more male used terms such as 'executed' and 'captured'.

Amazon AI sexist tool scrapped

Amazon’s AI gurus scrapped a new machine-learning recruiting engine earlier this month. Why? It transpired that the AI behind it was sexist. What does this mean as we race to produce ever-better artificial intelligence, and how can we understand the risks of machines learning the worst of our own traits?

Trained on the past decade’s worth of data about job applicants, the Amazon model began to penalize CVs that included the word “women.” The incident calls to mind another experiment in AI bias, Microsoft’s “Tay” project, which the company pulled from the web after the bot learned racism from users it was chatting with on GroupMe and Kik. In Amazon’s case, however, rogue users weren’t to blame. The AI was learning from the historical data of tech’s largest global company.

An artificial intelligence that dislikes women or people of color sounds like a concept straight out of a Twilight Zone episode. But sadly, it’s reality.

How did we get to this situation? And is it possible to build an AI that won’t reflect the bone-deep prejudices that are – knowingly or unknowingly – built into our social systems? To answer that second question, it’s crucial to address the first one.

How a Sexist AI Happens

Okay, the first point to make is that sexist or racist AI doesn’t emerge from nowhere. Instead, it reflects the prejudices already deeply held within both society at large, and the tech industry specifically.

Don’t believe us about sexism in tech? One study from earlier this year found that 57 out of 58 major U.S. cites paid women in tech less than men. Last year, two female tech cofounders demonstrated tech sexism at work by proving they could make better connections once they invented a fictional male cofounder.

And as long as tech companies continue overlooking sexism, they’ll keep perpetuating a system that prioritizes male applicants and promotes male staff.

Sexist AIs Start With a Blinkered Industry…

The tech world loves rapid growth above all else. But this year, it’s finally begun to come to terms with the impact that its culture can make, and a sense of responsibility is finally taking root.

Few sum it up better than former Reddit product head Dan McComas, whose recent New York Magazine interview (titled ‘I Fundamentally Believe That My Time at Reddit Made the World a Worse Place’) includes this insight:

“The incentive structure is simply growth at all costs. There was never, in any board meeting that I have ever attended, a conversation about the users, about things that were going on that were bad, about potential dangers, about decisions that might affect potential dangers. There was never a conversation about that stuff.”

…And Machine Learning Perpetuates Them

It’s this attitude that’s at the core of prejudiced AI, which perpetuates the system just as clearly, if a little more mechanically. As Lin Classon, director of public cloud strategy at Ensono, puts it, the process of machine learning is the issue.

“Currently, the most common application of AI is based on feeding the machine lots of data and teaching it to recognize a pattern. Because of this, the results are as good as the data used to train the algorithms,” she tells me.

Ben Dolmar, director of software development at the Nerdery, backs her up.

“Almost all of the significant commercial activity in Artificial Intelligence is happening in the field of Machine Learning,” explains Ben. “It was machine learning that drove Alpha Go and it’s machine learning that is driving the leaps that we’re making in natural language processing, computer vision and a lot of recommendations engines.”

Machine learning begins by providing a model with a core data set. The model trains on this before producing its own outputs. Any historical issues in the core data are then reproduced. Translation? Sexist data turns into sexist outputs.

“It’s not unlike painting with watercolors, where the brush has to be clean or it taints the colors,” says Classon. And, in modern society, sexism turns up everywhere, Classon says, whether it’s in “recruiting, loan applications or stock photos.” Or even in the emptiness of women’s restrooms at major tech conferences, as Classon has pointed out to Tech.Co before.

How to Combat AI Prejudice

How do we solve a problem like AI prejudice? Classon boils it down to a key guiding principle: conscientious and collective vigilance. And that begins with ensuring the community behind the AI developments is equipped to spot the issues. Which takes us right back to the core problem of ensuring that a diverse developer community is in place, to find issues faster and address them more quickly.

Practically speaking, Classon has further suggestions:

Increased Transparency

Right now, machine learning algorithms function like black boxes: Data goes in, trained models come out.

“DARPA has recognized [that this leaves users unaware of how the system came to any decision] and is working on Explainable Artificial Intelligence so future AI will be able to explain

Is Tech Doomed To Reflect The Worst In All Of Us?

The tech giant canned their experimental recruitment system riddled with problems, according to Reuters.

Amazon, back in 2014, set up the recruiting system in place, hoping to mechanize the entire hiring process. It used artificial intelligence to give candidates scores ranging from one to five stars. The system would then spit out the top 5 candidates, with the highest rating and qualifications.

But the machine-learning specialists found out a huge problem with this almost perfect system: It was sexist. The models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. A majority of them being men.

Thus, Amazon’s AI learnt that only male candidates would be suitable, and started to penalize resumes that included the word “women’s”. For example, a resume that may have contained, “the captain of the women’s basketball team”, was pushed back further behind on the list of most suitable candidates.

They did make changes to the program and claimed they fixed the issue, but people were not convinced as they believed the AI might find other discriminatory ways to target women.

Amazon defended the company by claiming the recruiters looked at the resumes put forth by the system but never completely relied on it. Furthermore, emphasized on its commitment to workplace diversity and equality.

The reports of the flawed system only supported the claims of women about the growing gender disparity in the tech industry.

Amazon Shuts Down It’s Problematic Sexist AI Recruitment System

wellesley.edu · 2018

Amazon recently scrapped an experimental artificial intelligence (AI) recruiting tool that was found to be biased against women. At this point, I hope you might have a few questions, such as: What is an AI recruiting tool and how does it work? Why was it biased against women? I’ll try to answer them for you in the following.

The AI Recruiting Tool

You have certainly heard of human recruiters. They are matchmakers between employers and potential employees. They travel, send cold emails, and “network” at conferences and job fairs. When recruiters make a successful match, they get paid, sometimes by one party, sometimes by both. As you can see, this matchmaking dance is often expensive and time-consuming. Surely technology can help, right? A human recruiter can review at most a few dozen applicants per day, before she gets tired. In contrast, artificial intelligence can “read” thousands of applications in seconds and rank them based on desired criteria, showing the most promising candidates at the top. Understandably then, compared to a human recruiter, an AI recruiter would be more time and cost efficient. And now that the human recruiter doesn’t need to sift through and rank candidates, she can devote her time to reaching out to the best candidates and wooing them to accept an offer. What nice team-work between the human and AI recruiters!

Unfortunately, things are never so simple. How can we ensure that the AI recruiter is being fair to all candidates? Can it offer explanations for why it didn’t suggest any women for a certain job opening? To answer these new questions, we need to understand how the AI tool “learns” to do its job.

It all starts with a big “training” set of job applications. For years, companies have been requiring job applicants to submit all their materials online. For example, if you have been on the academic job market, you were probably asked to upload your resume, cover letter, and letters of recommendation in a website like AcademicJobsOnline.org. Big corporations like Amazon and Google, unlike universities, run their own job application sites. Therefore, over time, they have amassed thousands and thousands of application materials, all in electronic form. Additionally, they have recorded which applicants were successful in their job hunts. Thus, they have examples of the materials submitted by applicants who were hired and by applicants who were rejected. This information is then given to the AI tool to “learn” the characteristics that reflect a successful candidate. In the case of Amazon’s tool, the AI “learned” that words like “executed” and “captured” in a resume correlate with success. Meanwhile, it also “learned” that the presence of a phrase like “women’s” (as in “women’s chess captain”) correlates with rejection, and so the corresponding resume was downgraded.

Artificial intelligence, despite all the hype (it will save the planet) and all the fear (it will kill mankind), is not, actually, intelligent. It has no idea what a word like “women’s” means and how it corresponds to entities in the real world. This kind of AI is only good at detecting patterns and finding relationships in the data we give it. So the data we provide to the AI, and what we tell it to do with it, is what matters the most.

Why was the AI tool biased against women?

The Amazon employees who talked to Reuters anonymously said that the AI tool downgraded applications of graduates from two women’s colleges, without specifying which colleges. This detail is what compelled me to write about the tool.

I am a woman computer science professor who teaches Artificial Intelligence at Wellesley College, which is a women’s college. As is typical at a liberal arts college, my students not only take computer science and mathematics courses for their major, but also courses in social sciences, arts, and humanities, courses with titles such as “Introduction to Women’s and Gender Studies,” “Almost Touching the Sky: Women’s Coming of Age Stories,” or “From Mumbet to Michelle Obama: Black Women’s History.” They are more likely than many other students to have the phrase “women’s” in their job application materials. Some of these students might have even been in the pool of applicants deemed as “not worthy to be recruited” by Amazon’s AI tool.

Every day, I stand in front of classrooms full of intelligent women, eager to learn about the beauty and power of algorithms. It pains me to find out that a major player like Amazon created and used algorithms that, ultimately, could have been used to crush their dreams of making their mark in the world by denying them the opportunity to join the teams of engineers who are designing and building our present and future technologies.

Why did the AI tool downgrade women’s resumes? Two reasons: data and values. The jobs for which women were not being recommended by the AI tool were in software development. Software development is studied in computer science, a discipline whose enrollments have

Is AI Sexist?

Amazon has scrapped a "sexist" internal tool that used artificial intelligence to sort through job applications.

The AI was created by a team at Amazon's Edinburgh office in 2014 as a way to automatically sort through CVs and pick out the most promising candidates.

However, it quickly taught itself to prefer male candidates over female ones, according to members of the team who spoke to Reuters.

They noticed that it was penalising CVs that included the word "women's," such as "women's chess club captain." It also reportedly downgraded graduates of two all-women's colleges.

Download the all-new Microsoft News app to receive up-to-the minute news from the world’s best sources – available now on iOS and Android

The problem stemmed from the fact that the system was trained on data submitted by people over a 10-year period, most of which came from men.

© AP Jeff Bezos, Amazon founder and CEO The AI was tweaked in an attempt to fix the bias. However, last year, Amazon lost faith in its ability to be neutral and abandoned the project altogether.

Amazon recruiters are believed to have used the system to look at the recommendations when hiring, but didn't rely on the rankings. Currently, women make up 40pc of Amazon's workforce.

Stevie Buckley, the co-founder of UK job website Honest Work, which is used by companies such as Snapchat to recruit for technology roles, said that “the basic premise of expecting a machine to identify strong job applicants based on historic hiring practices at your company is a surefire method to rapidly scale inherent bias and discriminatory recruitment practices.”

The danger of inherent bias in the use of algorithms is a common problem in the technology industry. Algorithms are not told to be biased, but can become unfair through the data they use.

© Getty Amazon logo Jessica Rose, a technical manager at education start-up FutureLearn and technology speaker, said that "the value of AI as it's used in recruitment today is limited by human bias."

"Developers and AI specialists carry the same biases as talent professionals, but we're often not asked to interrogate or test for these during the development process," she said.

Google had to remove the ability to search for photos of gorillas in its Google Photos app after the service began to suggest that photographs of people of colour were actually photographs of gorillas.

Amazon’s failed recruitment software and the issues with Google Photos illustrate one of the largest weaknesses of machine learning, where computers teach themselves to perform tasks by analysing data.

Gallery: 24 facts you may not know about Amazon founder Jeff Bezos (Photos)

Last month, IBM launched a tool which is designed to detect bias in AI. The Fairness 360 Kit allows developers to see clearly how their algorithms work and which pieces of data are used to make decisions.

“Considering Amazon's exhaustive resources and their exceptionally talented team of engineers,” Mr Buckley said, “the fact that their AI recruiting tool failed miserably suggests that we should maintain a default scepticism towards any organisation that claims to have produced an effective AI tool for recruitment.”

Amazon scraps 'sexist AI' recruiting tool that showed bias against women

However, bias also appears for other unrelated reasons. A recent study into how an algorithm delivered ads promoting STEM jobs showed that men were more likely to be shown the ad, not because men were more likely to click on it, but because women are more expensive to advertise to. Since companies price ads targeting women at a higher rate (women drive 70% to 80% of all consumer purchases), the algorithm chose to deliver ads more to men than to women because it was designed to optimise ad delivery while keeping costs low.

But if an algorithm only reflects patterns in the data we give it, what its users like, and the economic behaviours that occur in its market, isn’t it unfair to blame it for perpetuating our worst attributes? We automatically expect an algorithm to make decisions without any discrimination when this is rarely the case with humans. Even if an algorithm is biased, it may be an improvement over the current status quo.

To fully benefit from using AI, it’s important to investigate what would happen if we allowed AI to make decisions without human intervention. A 2018 study explored this scenario with bail decisions using an algorithm trained on historical criminal data to predict the likelihood of criminals re-offending. In one projection, the authors were able to reduce crime rates by 25% while reducing instances of discrimination in jailed inmates.

Yet the gains highlighted in this research would only occur if the algorithm was actually making every decision. This would be unlikely to happen in the real world as judges would probably prefer to choose whether or not to follow the algorithm’s recommendations. Even if an algorithm is well designed, it becomes redundant if people choose not to rely on it.

Many of us already rely on algorithms for many of our daily decisions, from what to watch on Netflix or buy from Amazon. But research shows that people lose confidence in algorithms faster than humans when they see them make a mistake, even when the algorithm performs better overall.

For example, if your GPS suggests you use an alternative route to avoid traffic that ends up taking longer than predicted, you’re likely to stop relying on your GPS in the future. But if taking the alternate route was your decision, it’s unlikely you will stop trusting your own judgement. A follow-up study on overcoming algorithm aversion even showed that people were more likely to use an algorithm and accept its errors if given the opportunity to modify the algorithm themselves, even if it meant making it perform imperfectly.

While humans might quickly lose trust in flawed algorithms, many of us tend to trust machines more if they have human features. According to research on self-driving cars, humans were more likely to trust the car and believed it would perform better if the vehicle’s augmented system had a name, a specified gender, and a human-sounding voice. However, if machines become very human-like, but not quite, people often find them creepy, which could affect their trust in them.

Even though we don’t necessarily appreciate the image that algorithms may reflect of our society, it seems that we are still keen to live with them and make them look and act like us. And if that’s the case, surely algorithms can make mistakes too?

Maude Lavanchy is Research Associate at IMD.

This article was first published by The Conversation.

Why Amazon's sexist AI recruiting tool is better than a human.

Last December Synced compiled its first “Artificial Intelligence Failures” recap of AI gaffes from the previous year. AI has achieved remarkable progress, and many scientists dream of creating the Master Algorithm proposed by Pedro Domingos — which can solve all problems envisioned by humans. It’s unavoidable however that researchers, fledgling technologies and biased data will also produce blunders not envisioned by humans.

That’s why a review of AI failures is necessary and meaningful: The aim of the article is not to downplay or mock research and development results, but to take a look at what went wrong with the hope we can do better next time.

Synced 10 AI failures of 2018.

Chinese billionaire’s face identified as jaywalker

Traffic police in major Chinese cities are using AI to address jaywalking. They deploy smart cameras using facial recognition techniques at intersections to detect and identify jaywalkers, whose partially obscured names and faces then show up on a public display screen.

The AI system in the southern port city of Ningbo however recently embarrassed itself when it falsely “recognized” a photo of Chinese billionaire Mingzhu Dong on an ad on a passing bus as a jaywalker. The mistake went viral on Chinese social media and Ningbo police apologized. Dong was unfazed, posting on Weibo: “This is a trivial matter. Safe travel is more important.”

CloudWalk Deep Learning Researcher Xiang Zhou told Synced the algorithm’s lack of live detection was the likely problem. “Live detection at this distance is challenging, recognizing an image as a real person is pretty common now.”

Chinese billionaire Mingzhu Dong’s face on a public display screen.

Uber self-driving car kills a pedestrian

In the first known autonomous vehicle-related pedestrian death on a public road, an Uber self-driving SUV struck and killed a female pedestrian on March 28 in Tempe, Arizona. The Uber vehicle was in autonomous mode, with a human safety driver at the wheel.

So what happened? Uber discovered that its self-driving software decided not to take any actions after the car’s sensors detected the pedestrian. Uber’s autonomous mode disables Volvo’s factory-installed automatic emergency braking system, according to US National Transportation Safety Board preliminary report on the accident.

In the wake of the tragedy Uber suspended self-driving testing in North American cities, and Nvidia and Toyota also stopped their self-driving road tests in the US. Eight months after the accident Uber announced plans to resume self-driving road tests in Pittsburgh, although the company’s self-driving future remains uncertain.

ABC 15 screenshot of deadly Uber accident.

IBM Watson comes up short in healthcare

“This product is a piece of shit” wrote a doctor at Florida’s Jupiter Hospital regarding IBM’s flagship AI program Watson, according to internal documents obtained by Stat. Originally a question-answering machine, IBM has been exploring Watson’s AI capabilities across a broad range of applications and processes, including healthcare. In 2013 IBM developed Watson’s first commercial application for cancer treatment recommendation, and the company has secured a number of key partnerships with hospitals and research centers over the past five years. But Watson AI Health has not impressed doctors. Some complained it gave wrong recommendations on cancer treatments that could cause severe and even fatal consequences.

After spending years on the project without significant advancements, IBM is reportedly downsizing Watson Health and laying off more than half the division’s staff.

Amazon AI recruiting tool is gender biased

Amazon HR reportedly used an AI-enabled recruiting software between 2014 and 2017 to help review resumes and make recommendations. The software was however found to be more favorable to male applicants because its model was trained on resumes submitted to Amazon over the past decade, when many more male candidates were hired.

The software reportedly downgraded resumes that contain the word “women” or implied the applicant was female, for example because they had attended a women’s college. Amazon has since abandoned the software. The company did not deny using the tool to produce recommendations, but said it was never used to evaluate candidates.

DeepFakes reveals AI’s unseemly side

Last December several porn videos appeared on Reddit “featuring” top international female celebrities. User “DeepFakes” employed generative adversarial networks to swap celebrities’ faces with those of the porn stars. While face-swapping technology has been under development for years, DeepFakes’ method showed that anyone with enough facial images could now produce their own highly convincing fake videos.

Realistic-looking fake videos of well-known people flooded the Internet through 2018. While the method is not technically a “failure,” its potential dangers are serious and far-reaching: if video evidence is no longer credible, this could further enc

2018 in Review: 10 AI Failures

It was supposed to make finding the right person for the job easier. However, an AI tool developed by Amazon to sift through potential hires has been dropped by the firm after developers found it was biased against picking women.

From pricing items to warehouse coordination, automation has been a key part of Amazon’s rise to e-commerce domination. And since 2014, its developers have been creating hiring programs aimed at making the selection of top talent as easy and as automated as possible.

Read more

“Everyone wanted this holy grail,” one of the anonymous sources told Reuters about the ambitions for the software.

“They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”

However, a leak by several of those familiar with the program give an insight into some of the mishaps in the AI-based hiring software’s development, and how it taught itself to penalize women… for being women.

It was in 2015 that human recruiters first noticed discrepancies with the tool, when it seemingly marked down female candidates for roles in the male-dominated spheres of software development and other technical roles at the firm.

When the engine came across words like “women’s” on a resume, or if a candidate graduated from an all-women’s college, it unfairly penalized female candidates from selection, the sources said.

Investigations into the cause of the gender imbalance found that the data which fed the algorithm was based on ten years of resumes sent to the company. The vast majority of which were submitted by men.

The algorithm in turn learned to dismiss female candidates as a negative leading to its sexist scoring system.

Edits were made by programmers to make the engine neutral to these particular terms, however, there was no certainty that it wouldn't develop other ways to discriminate in future.

READ MORE: Racist & sexist AI bots could deny you job, insurance & loans – tech experts

Dejected executives eventually scrapped the team in 2017 after losing hope in the project. An Amazon spokesperson told RT that the project never made it out of the trial phase. In addition to its apparent bias, the software "never returned storng candidates for the roles." Now, a “much-watered down version” is instead used for minor HR tasks such as sorting out duplicate applicants from its databases.

Amazon’s sexist algorithms isn’t the first time AI has landed tech firms in hot water. Last month Facebook got flack after it was discovered that women users were prevented from seeing job advertisements in traditionally male-dominated industries.

In May 2016, a report found that a US court that used automated software to provide risk assessments was biased against black prisoners, recording them as twice as likely to reoffend as their white counterparts.

Think your friends would be interested? Share this story!

Sexist AI: Amazon ditches recruitment tool that turned out to be anti-women

I’m at home playing a video game on my computer. My job is to pump up one balloon at a time and earn as much money as possible. Every time I click “Pump,” the balloon expands and I receive five virtual cents. But if the balloon pops before I press “Collect,” all my digital earnings disappear.

After filling 39 balloons, I’ve earned $14.40. A message appears on the screen: “You stick to a consistent approach in high-risk situations. Trait measured: Risk.”

This game is one of a series made by a company called Pymetrics, which many large US firms hire to screen job applicants. If you apply to McDonald’s, Boston Consulting Group, Kraft Heinz, or Colgate-Palmolive, you might be asked to play Pymetrics’s games.

While I play, an artificial-intelligence system measures traits including generosity, fairness, and attention. If I were actually applying for a position, the system would compare my scores with those of employees already working in that job. If my personality profile reflected the traits most specific to people who are successful in the role, I’d advance to the next hiring stage.

More and more companies are using AI-based hiring tools like these to manage the flood of applications they receive—especially now that there are roughly twice as many jobless workers in the US as before the pandemic. A survey of over 7,300 human-resources managers worldwide by Mercer, an asset management firm, found that the proportion who said their department uses predictive analytics jumped from 10% in 2016 to 39% in 2020.

As with other AI applications, though, researchers have found that some hiring tools produce biased results—inadvertently favoring men or people from certain socioeconomic backgrounds, for instance. Many are now advocating for greater transparency and more regulation. One solution in particular is proposed again and again: AI audits.

Last year, Pymetrics paid a team of computer scientists from Northeastern University to audit its hiring algorithm. It was one of the first times such a company had requested a third-party audit of its own tool. CEO Frida Polli told me she thought the experience could be a model for compliance with a proposed law requiring such audits for companies in New York City, where Pymetrics is based.

“What Pymetrics is doing, which is bringing in a neutral third party to audit, is a really good direction in which to be moving,” says Pauline Kim, a law professor at Washington University in St. Louis, who has expertise in employment law and artificial intelligence. “If they can push the industry to be more transparent, that’s a really positive step forward.”

For all the attention that AI audits have received, though, their ability to actually detect and protect against bias remains unproven. The term “AI audit” can mean many different things, which makes it hard to trust the results of audits in general. The most rigorous audits can still be limited in scope. And even with unfettered access to the innards of an algorithm, it can be surprisingly tough to say with certainty whether it treats applicants fairly. At best, audits give an incomplete picture, and at worst, they could help companies hide problematic or controversial practices behind an auditor’s stamp of approval.

Inside an AI audit

Many kinds of AI hiring tools are already in use today. They include software that analyzes a candidate’s facial expressions, tone, and language during video interviews as well as programs that scan résumés, predict personality, or investigate an applicant’s social media activity.

Regardless of what kind of tool they’re selling, AI hiring vendors generally promise that these technologies will find better-qualified and more diverse candidates at lower cost and in less time than traditional HR departments. However, there’s very little evidence that they do, and in any case that’s not what the AI audit of Pymetrics’s algorithm tested for. Instead, it aimed to determine whether a particular hiring tool grossly discriminates against candidates on the basis of race or gender.

Christo Wilson at Northeastern had scrutinized algorithms before, including those that drive Uber’s surge pricing and Google’s search engine. But until Pymetrics called, he had never worked directly with a company he was investigating.

Wilson’s team, which included his colleague Alan Mislove and two graduate students, relied on data from Pymetrics and had access to the company’s data scientists. The auditors were editorially independent but agreed to notify Pymetrics of any negative findings before publication. The company paid Northeastern $104,465 via a grant, including $64,813 that went toward salaries for Wilson and his team.

Pymetrics’s core product is a suite of 12 games that it says are mostly based on cognitive science experiments. The games aren’t meant to be won or lost; they’re designed to discern an applicant’s cognitive, social, and emotional attributes, including risk tolerance and learning ability. Pymetrics markets its software as “entirely bias free.” Pymetrics and Wilson decided that the auditors would focus narrowly on one specific question: Are the company’s models fair?

They based the definition of fairness on what’s colloquially known as the four-fifths rule, which has become an informal hiring standard in the United States. The Equal Employment Opportunity Commission (EEOC) released guidelines in 1978 stating that hiring procedures should select roughly the same proportion of men and women, and of people from different racial groups. Under the four-fifths rule, Kim explains, “if men were passing 100% of the time to the next step in the hiring process, women need to pass at least 80% of the time.”

If a company’s hiring tools violate the four-fifths rule, the EEOC might take a closer look at its practices. “For an employer, it’s not a bad check,” Kim says. “If employers make sure these tools are not grossly discriminatory, in all likelihood they will not draw the attention of federal regulators.”

To figure out whether Pymetrics’s software cleared this bar, the Northeastern team first had to try to understand how the tool works.

When a new client signs up with Pymetrics, it must select at least 50 employees who have been successful in the role it wants to fill. These employees play Pymetrics’s games to generate training data. Next, Pymetrics’s system compares the data from those 50 employees with game data from more than 10,000 people randomly selected from over two million. The system then builds a model that identifies and ranks the skills most specific to the client’s successful employees.

To check for bias, Pymetrics runs this model against another data set of about 12,000 people (randomly selected from over 500,000) who have not only played the games but also disclosed their demographics in a survey. The idea is to determine whether the model would pass the four-fifths test if it evaluated these 12,000 people.

If the system detects any bias, it builds and tests more models until it finds one that both predicts success and produces roughly the same passing rates for men and women and for members of all racial groups. In theory, then, even if most of a client’s successful employees are white men, Pymetrics can correct for bias by comparing the game data from those men with data from women and people from other racial groups. What it’s looking for are data points predicting traits that don’t correlate with race or gender but do distinguish successful employees.

Wilson and his team of auditors wanted to figure out whether Pymetrics’s anti-bias mechanism does in fact prevent bias and whether it can be fooled. To do that, they basically tried to game the system by, for example, duplicating game data from the same white man many times and trying to use it to build a model. The outcome was always the same: “The way their code is sort of laid out and the way the data scientists use the tool, there was no obvious way to trick them essentially into producing something that was biased and get that cleared,” says Wilson.

Last fall, the auditors shared their findings with the company: Pymetrics’s system satisfies the four-fifths rule. The Northeastern team recently published the study of the algorithm online and will present a report on the work in March at the algorithmic accountability conference FAccT.

“The big takeaway is that Pymetrics is actually doing a really good job,” says Wilson.

An imperfect solution

But though Pymetrics’s software meets the four-fifths rule, the audit didn’t prove that the tool is free of any bias whatsoever, nor that it actually picks the most qualified candidates for any job.

“It effectively felt like the question being asked was more ‘Is Pymetrics doing what they say they do?’ as opposed to ‘Are they doing the correct or right thing?’” says Manish Raghavan, a PhD student in computer science at Cornell University, who has published extensively on artificial intelligence and hiring.

For example, the four-fifths rule only requires people from different genders and racial groups to pass to the next round of the hiring process at roughly the same rates. An AI hiring tool could satisfy that requirement and still be wildly inconsistent at predicting how well people from different groups actually succeed in the job once they’re hired. And if a tool predicts success more accurately for men than women, for example, that would mean it isn’t actually identifying the best qualified women, so the women who are hired “may not be as successful on the job,” says Kim.

Another issue that neither the four-fifths rule nor Pymetrics’s audit addresses is intersectionality. The rule compares men with women and one racial group with another to see if they pass at the same rates, but it doesn’t compare, say, white men with Asian men or Black women. “You could have something that satisfied the four-fifths rule [for] men versus women, Blacks versus whites, but it might disguise a bias against Black women,” Kim says.

Pymetrics is not the only company having its AI audited. HireVue, another large vendor of AI hiring software, had a company called O’Neil Risk Consulting and Algorithmic Auditing (ORCAA) evaluate one of its algorithms. That firm is owned by Cathy O’Neil, a data scientist and the author of Weapons of Math Destruction, one of the seminal popular books on AI bias, who has advocated for AI audits for years.

ORCAA and HireVue focused their audit on one product: HireVue’s hiring assessments, which many companies use to evaluate recent college graduates. In this case, ORCAA didn’t evaluate the technical design of the tool itself. Instead, the company interviewed stakeholders (including a job applicant, an AI ethicist, and several nonprofits) about potential problems with the tools and gave HireVue recommendations for improving them. The final report is published on HireVue’s website but can only be read after signing a nondisclosure agreement.

Alex Engler, a fellow at the Brookings Institution who has studied AI hiring tools and who is familiar with both audits, believes Pymetrics’s is the better one: “There’s a big difference in the depths of the analysis that was enabled,” he says. But once again, neither audit addressed whether the products really help companies make better hiring choices. And both were funded by the companies being audited, which creates “a little bit of a risk of the auditor being influenced by the fact that this is a client,” says Kim.

For these reasons, critics say, voluntary audits aren’t enough. Data scientists and accountability experts are now pushing for broader regulation of AI hiring tools, as well as standards for auditing them.

Filling the gaps

Some of these measures are starting to pop up in the US. Back in 2019, Senators Cory Booker and Ron Wyden and Representative Yvette Clarke introduced the Algorithmic Accountability Act to make bias audits mandatory for any large companies using AI, though the bill has not been ratified.

Meanwhile, there’s some movement at the state level. The AI Video Interview Act in Illinois, which went into effect in January 2020, requires companies to tell candidates when they use AI in video interviews. Cities are taking action too—in Los Angeles, city council member Joe Buscaino proposed a fair hiring motion for automated systems in November.

The New York City bill in particular could serve as a model for cities and states nationwide. It would make annual audits mandatory for vendors of automated hiring tools. It would also require companies that use the tools to tell applicants which characteristics their system used to make a decision.

But the question of what those annual audits would actually look like remains open. For many experts, an audit along the lines of what Pymetrics did wouldn’t go very far in determining whether these systems discriminate, since that audit didn’t check for intersectionality or evaluate the tool’s ability to accurately measure the traits it claims to measure for people of different races and genders.

And many critics would like to see auditing done by the government instead of private companies, to avoid conflicts of interest. “There should be a preemptive regulation so that before you use any of these systems, the Equal Employment Opportunity Commission should need to review it and then license it,” says Frank Pasquale, a professor at Brooklyn Law School and an expert in algorithmic accountability. He has in mind a preapproval process for algorithmic hiring tools similar to what the Food and Drug Administration uses with drugs.

So far, the EEOC hasn’t even issued clear guidelines concerning hiring algorithms that are already in use. But things might start to change soon. In December, 10 senators sent a letter to the EEOC asking if it has the authority to start policing AI hiring systems to prevent discrimination against people of color, who have already been disproportionally affected by job losses during the pandemic.

Auditors are testing hiring algorithms for bias, but there’s no easy fix