Incident 183: Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Description: Airbnb allegedly considered publicly available data on users to gauge their trustworthiness via algorithmic assessment of personality and behavioral traits, resulting in unexplained bans and discriminatory bans against sex workers.
Alleged: Airbnb and Trooly developed an AI system deployed by Airbnb, which harmed sex workers and Airbnb users.

Suggested citation format

AIAAIC. (2017-07-01) Incident Number 183. in McGregor, S. (ed.) Artificial Intelligence Incident Database. Responsible AI Collaborative.

Incident Stats

Incident ID
183
Report Count
6
Incident Date
2017-07-01
Editors
Sean McGregor, Khoa Lam

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscover

Incident Reports

EPIC has filed a complaint with the FTC, alleging that Airbnb has committed unfair and deceptive practices in violation of the FTC Act and the Fair Credit Reporting Act. Airbnb secretly rates customers “trustworthiness" based on a patent that considers such factors as “authoring online content with negative language.” The company’s opaque, proprietary algorithm also considers "posts on the person’s social network account" as well the individual's relationships with others, and adjusts the "trustworthiness" score based on the scores of those associations. EPIC said the company failed to comply with "established public policies" for AI decision-making, such as the OECD AI Principles and the Universal Guidelines for AI. EPIC has recently brought complaints to the FTC about the employment screening firm HireVue and the Universal Tennis Rating secret scoring technique. EPIC has also petitioned the FTC to conduct a rulemaking for "the use of artificial intelligence in commerce." The EPIC AI Policy Sourcebook includes the OECD AI Principles, the Universal Guidelines for AI, and other AI policy frameworks.

EPIC Files Complaint with FTC about Airbnb’s Secret “Trustworthiness” Scores

Airbnb may be using automated decision-making to leave some users out in the cold.

If you've ever been in the market for a holiday rental property in the past few years, chances are you've been on Airbnb.

The US-based platform has become a global behemoth in the short-term rental market, with 5.6 million properties listed on its website around the world in 2021 alone. More than 800 million people worldwide have stayed at an Airbnb property.

But while the company has become synonymous with holidaying, some users in Australia are finding themselves blocked and banned from the platform, and are asking why.

Airbnb acquires 'trustworthiness' algorithm

In 2017, Airbnb acquired a tech startup called Trooly which specialised in background checks and had earlier patented an algorithm that gathered publicly and privately available data on users to give them a 'trustworthiness' score.

Kate Bower, consumer data advocate at CHOICE, says the data the algorithm uses to assess your 'trustworthiness' includes social media accounts of yourself and your friends, what you do for a living, your education level and any other online data it can find.

Assessing personality traits

The patented algorithm is claimed to assess people's personality traits, such as narcissism or conscientiousness, along with behavioural traits, such as use of drugs or alcohol or involvement in civil litigation and other behaviour, and combine them to create a holistic score that judges a person's trustworthiness.

Airbnb has updated the patent, which it owns directly, several times since 2017. This suggests it's using and developing the algorithm, not just letting it languish on some back shelf.

Airbnb's privacy policy includes a broad statement that indicates the algorithm may be in use in Australia. It states, "we may conduct profiling … [and that] automated processes, which analyse your account and activities on the Airbnb platform as well as information in relation to activities on and off the Airbnb platform that can be associated with you, could restrict or suspend access to the Airbnb Platform".

What does Airbnb know about you?

The short-term rental platform Airbnb might be using a hidden algorithm to give you a trustworthiness score based on your online information. This score could be used to exclude you from the service.

Comparison of information collected by hotels and Airbnb:

Hotels view and collect your identification documents, payment method, and contact details.

Airbnb views and collects your identification documents, payment method, contact details, IP address, device identifiers, location data, plus potentially social media connections, employment history and education history.

Customers report being blocked for no reason

Renae Macheda describes herself and her husband as "clean, boring people". She works in real estate and was baffled when she tried to book accommodation in Sydney in October – only to find herself banned from Airbnb and her account closed.

The company gave her no reason for the ban. When she queried the decision, she got a brief email that was no more illuminating.

It read: "After reviewing all the information available to us, we've determined that your account will be removed from the Airbnb platform. Removal means that your account will no longer be accessible, and you won't be able to create another one. We want to assure you that we reviewed your case thoroughly before reaching this conclusion. As such, we won't be able to offer you additional support on this matter at this time."

'It's not really good enough'

Renae tells CHOICE she's never had a dispute with an Airbnb host and has had nothing but good reviews from her many holidays. She says she's "devastated" by the decision and feels let down by the company.

"I think they should give you an explanation," she says. "To give nothing at all and no options to try and remedy whatever it is, it's not really good enough."

Technology experts say: 'Be more consumer-friendly'

Ellen Broad from the Australian National University (ANU) College of Engineering & Computer Science says companies like Airbnb, which operate similarly to public utilities, ought to be bound by a similar set of responsibilities.

"They have a responsibility to give people meaningful avenues to understand why a ban occurs and meaningful avenues to appeal then, but they're not doing that," she tells CHOICE.

Julian Thomas, from RMIT and director the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), also expresses concerns about the algorithm's potential use in Australia.

"We need to be looking at how we can regulate these systems better to ensure that they are much more consumer-friendly, that they do respect people's rights, that they are more responsible," he says.

Sex workers banned

Rick Andrews (not his real name) is a Melbourne-based erotic masseur who found himself banned from Airbnb last year when he tried to book a holiday in Sydney.

He says he had nothing but good reviews from the dozens of local and international holidays he's booked through the platform.

Airbnb sent him an email saying his account had been flagged during a "standard security review".

The email read: "It turned out that your account was linked to activity that goes against our Terms of Service, specifically it was linked to online ads for adult services, which can include escort activity and commercial pornography."

Rick says, "I was outraged, this is unbelievable, what have I done? Murdered someone?"

Jane (not her real name), a Melbourne-based sex worker, says her account was blocked in 2018. She believes it was due to it being linked to online ads for sex work, and says "there was no explanation given to me at all".

Ban could be discriminatory

Matthew Roberts from Sex Work Law Reform Victoria says that banning someone from a platform based on their occupation is discriminatory and should be protected against.

"I think we should all be aware of what Airbnb is doing before we sign up to Airbnb, that's not being disclosed or made transparent," he tells CHOICE.

Gala Vanting of Scarlet Alliance, the national representative body for sex workers, says they've also seen a large number of cases of sex workers being banned from Airbnb.

"For many sex workers removed from Airbnb, there's no process, no violation of their terms, no story to explain their loss of access – and no appeals process to get an explanation or reclaim an account," says Vanting. "Airbnb's silence around these cases is deafening."

Airbnb vague on algorithm use in Australia

We sent a list of detailed questions to Airbnb Australia, hoping to find out more about the way its algorithm is being used in this country.

But the company's response was extremely vague. It said: "The safety, security and privacy of our community is one of our top priorities. Our platform security and safety measures are designed to help ensure stays are safe and positive experiences for Hosts, guests and the wider community – while also protecting users' information, including their personal information."

'Lack of transparency'

Bower says that this response isn't good enough and that Airbnb has a duty to inform Australian customers how it's gathering and using their data.

"There is a worrying lack of transparency of potentially harmful and invasive practices," she says. "Airbnb claims it offers users an opportunity to have automated decisions reviewed but in reality people are left in the dark about why they have been removed from the platform."

Use of algorithms on the rise

Kimberlee Weatherall is professor of law at the University of Sydney, and a chief investigator with ARC Centre of Excellence for ADM+S. She says there are several concerning things about Airbnb's algorithm and its use of data unrelated to people's Airbnb rental history to judge how they'll behave in a property.

"Using what they did at university two years ago to decide whether they get holiday accommodation tomorrow, that crossing of contexts, I think is a real problem," she tells CHOICE. "These algorithms either punish people for the behaviour of others, or punish people for their past behaviour."

She likens this "crossing of contexts" in data use to China's controversial social credit scoring system, and says we should be concerned.

"This idea that you get surveilled across all different kinds of contexts and bad behaviour on one metric can lead to punishment is concerning," she says.

"In the China example, you didn't pay your fines, so you're not allowed to get on a train. If we move to a world where we say that it is OK for Airbnb to surveil you across multiple contexts then people will have to start censoring themselves online and what they post."

Dangers of 'social scoring'

Bower agrees with Weatherall, saying the "algorithm amounts to a form of social scoring reminiscent of a Black Mirror episode" (Black Mirror is a British TV sci-fi series that focuses on the dystopian use of technology, often in the near future). Social scoring, sometimes called 'social credit', is when an automated system assesses a person's trustworthiness or likely future behaviour.

Just as credit reporting aims to judge a person's ability to repay a debt from their past financial behaviour, so social scoring algorithms attempt to predict how likely a person is to behave in a certain way based on their past behaviour, as gleaned from their personal data.

Social scoring can reproduce societal biases and lead to real-life discrimination. "Most Australians would be shocked that profit-driven businesses like Airbnb have appointed themselves moral arbiters of their behaviour," says Bower.

Regulation needed

Automated decision-making is a computerised process that can either replace or assist human decision-making, often using advanced data analytics and machine learning.

Its use is growing around the world, including in Australia. Bower says, "It is already widely used in credit reporting, insurance and financial services and we are likely to see it increasing in other sectors including housing, healthcare, retail and many others. It has great potential to improve services for consumers but it is also potentially harmful."

The EU model

Thomas from the ARC Centre of Excellence for ADM+S says Australia needs to look at regulating the way companies use cross-contextual data for automated decision-making. He says Australia could follow the line of the European Union (EU), which is currently addressing concerns around automated decision-making and social scoring.

"We [Australia] are a small market, so it makes sense for us to follow the lead of the EU on this," he says. "All global companies will have to interact with the European market in some way and their policies are often global. If we think the EU framework does provide strong consumer safeguards, I think we should work with those."

CHOICE pushing for better protections

Our submission to the Privacy Act review supports the adoption of the EU framework, which proposes an outright ban on social scoring algorithms, along with regulation of other uses of automated decision-making and artificial intelligence that pose a high risk to consumers.

According to Bower, Airbnb's potential use of its algorithm in an unregulated space is highly worrying.

"CHOICE is concerned that businesses are implementing automated decision-making widely without informing consumers of the risks, offering avenues to opt out or opportunity to review decisions," she says.

"Automated decision-making has the potential to make inaccurate or biased decisions and lead to discrimination and unfair practices."

Do you think it's unfair that Airbnb can use your personal data in automated decision-making to restrict you from the platform? Answer our poll.

Is Airbnb using an algorithm to ban users from the platform?

Airbnb may be using automated decision making to boot users from the short-term rental platform, based on factors like social media, employment history and your IP address.

Consumer advocacy group Choice called out Airbnb in a report questioning the lack of transparency around its use of a “secretive algorithm” that judges if users are ‘trustworthy’.

According to Choice, Airbnb bought background-check startup Trooly in 2017.

Since then, it has reportedly updated the patent several times, suggesting it is in use.

As far as the fine print goes, Airbnb “may conduct profiling” using your interactions with the platform, as well as information obtained by third parties.

Its privacy policy says automated processes, which analyse users’ activities on and off Airbnb, could restrict or suspend access to the platform.

“Choice is concerned that businesses are implementing automated decision making widely without informing consumers of the risks, offering avenues to opt out or opportunity to review decisions,” said Kate Bower, consumer data advocate at Choice.

Can you trust the ‘trustworthy’?

Generally speaking, Dr Marc Cheong, senior lecturer in information systems (digital ethics) at the University of Melbourne, said such automated decision making poses two concerns.

“The main issue is those affected by this algorithm may not have an opportunity to appeal or seek recourse about the actions taken by the AI,” Dr Cheong told The New Daily.

The second concern is whether or not the system is even collecting accurate data.

If you have a machine trawling through everything about you online, how do you make sure it’s correct?

Can you trust the ‘trustworthiness’ algorithm?

Choice spoke to Australians who have been booted from Airbnb, despite receiving good reviews.

Renae Macheda, who described herself and her husband as “clean, boring people”, said she received no real explanation for the ban.

“To give nothing at all and no options to try and remedy whatever it is, it’s not really good enough,” she told Choice.

‘Guilty by association’

What about pictures or comments you are tagged in online?

Dr Cheong said the affected user would then be “guilty by association”, as it is the activities of the people in your social network that affect you, instead of them.

Professor of Law Jeannie Paterson, director of the Centre for AI and Digital Ethics at the University of Melbourne, said algorithmic decision making is not only difficult to understand, it can entrench or amplify existing biases and lead to discrimination.

“The idea that you can determine ‘trustworthiness’ from someone’s social media presence has got to be ‘junk science’, to quote the previous human rights commissioner Ed Santow,” Professor Paterson told TND.

The case for law reform

As Professor Paterson explained, Airbnb is a private company.

“It can decide who does and does not use its properties.”

Unfortunately, as the law exists now, people who get thrown off the platform have very few options.

“You’re wrongly accused of being disrupted and you respond, ‘I’m not’, but you’ve got no way of challenging that decision,” Professor Paterson said.

“That decision is made by an algorithm and Airbnb doesn’t appear to provide a process to appeal that, unlike Instagram or TikTok.”

Although a customer could argue, albeit with difficulty, that their removal was unfair, Professor Paterson said the strongest option is championing law reform.

“So shining a light on these practices is really important because at the end of the day, businesses like Airbnb thrive on reputation and if there is a reputation rift in using this unfair process, it might be a prompt for them to clean up their act.”

Professor Paterson said social media and the platform economy are part of our lives now, so it’s important that businesses – and it isn’t just Airbnb using automated decision making – recognise they need good governance.

Somewhat optimistic, Dr Cheong said responsible use of AI needs to have, at a minimum, human oversight and auditing frameworks to restore confidence.

There must be a clear and transparent process for contesting decisions that affect people.

Airbnb did not respond to The New Daily‘s request for comment in time for publication.

Airbnb blasted for using algorithm that judges if users are ‘trustworthy’

Airbnb may be digging through users’ old social media posts to keep people it deems untrustworthy off the site.

If the algorithm it uses doesn't like what it sees, it seems users can be rejected without explanation.

That’s what happened to Renae Macheda.

Macheda, who works in real estate, described herself and her husband as "clean, boring people" and was shocked when she tried to book an Airbnb last year and discovered she had been kicked off the platform.

When she started asking questions, she was not given a reason for the ban and was told the platform “reviewed your case thoroughly before reaching this conclusion”.

She was also told she would not be able to challenge the decision further.

"I think they should give you an explanation," Macheda said.

“To give nothing at all and no options to try and remedy whatever it is, it's not really good enough."

Information may be wrong

CHOICE consumer data advocate Kate Bower said it was very concerning that people didn’t have the opportunity to check the accuracy of the information gathered by these companies.

It’s not hard to imagine situations where the algorithm can be wrong.

For example, Bower said she parked in a liquor store car park every weekend to take her kids to an adjacent cafe.

To an algorithm, that might look like she goes to buy alcohol every Sunday at 9:00am.

What else does it screen for?

It’s hard to know exactly what the algorithm is looking for but available information on the software shows someone with a PhD may be considered more trustworthy than someone with a bachelor’s degree, for example.

“And we don't really know if that makes someone more trustworthy or not,” Bower said.

Other signs the algorithm may be looking for could be if people have used fake names online, have been involved with certain types of social media groups or have appeared on news sites the platform considers unsavoury.

CHOICE also heard from people who had been banned because they were sex workers or connected to the industry in some way.

Matthew Roberts from Sex Work Law Reform Victoria said banning someone from a platform based on their occupation was discriminatory and better protections were needed.

So what can I do?

According to Bower, it’s very difficult to stop companies snooping around unless you are prepared to go completely offline.

She said there was already a huge amount of data on consumers, and that algorithms and machine-learning technologies were only getting better at making sense of these data points.

She recommended reading privacy policies, although they were often vague, and not connecting to online platforms using your Facebook account.

Bower wants to see stronger regulation to “protect people because we're not able to protect ourselves”.

“Airbnb’s behaviour highlights a significant regulatory gap, which allows companies to collect and use data without any transparency or accountability,” Bower said.

“The Federal Government needs to strengthen protections against companies collecting, storing and using sensitive personal data like this.”

Banned from Airbnb with no explanation? Here’s why

An investigation by the Australian consumer rights organisation, Choice, has found that the online accommodation marketplace Airbnb is secretly collecting users’ personal data to assess whether they are trustworthy enough to make a booking.

It’s found that Airbnb may be using an algorithm which trawls users’ social media accounts and those of their friends, as well as other available data such as a person’s level of education and job.

The same privacy policy applies to both Airbnb Australia and Airbnb New Zealand, which indicates the algorithm could also be at use here.

The policy features vague statements about conducting profiling on users and their activities both on and off the platform, using information obtained from third parties. It claims that these processes are in place to detect potential safety risks to the Airbnb community.

Airbnb also has an anti-discrimination policy which talks about people being welcomed and respected no matter their background, which is at odds with the profiling methods in their privacy policy. Its website reads: "We are all committed to doing everything we can to help eliminate all forms of unlawful bias, discrimination and intolerance from our platform."

Choice has spoken to Australians who have been banned from Airbnb without good reason.

Rick Andrews (not his real name) is a Melbourne-based erotic masseur who found himself banned from Airbnb last year when he tried to book a holiday in Sydney. 

He said he had nothing but good reviews from the dozens of local and international holidays he's booked through the platform.

Airbnb sent him an email saying his account had been flagged during a "standard security review".

"I was outraged," said Rick "This is unbelievable, what have I done? Murdered someone?"

Another user, Renae Macheda, describes herself and her husband as "clean, boring people". She works in real estate and was baffled when she tried to book accommodation in Sydney in October – only to find herself banned from Airbnb and her account closed. The company gave her no reason for the ban. 

Renae said she's never had a dispute with an Airbnb host and has had nothing but good reviews from her many holidays.

What’s your trustworthiness score?

Airbnb’s algorithm gathers publicly and privately available data on users to give them a trustworthiness score.

The patented algorithm is claimed to assess people's personality traits, such as narcissism or conscientiousness, along with behavioural traits, such as use of drugs or alcohol or involvement in civil litigation and other behaviour, and combine them to create a holistic score that judges a person's trustworthiness.

Kate Bower, consumer data advocate at Choice, said: "Automated decision-making has the potential to make inaccurate or biased decisions and lead to discrimination and unfair practices … It is already widely used in credit reporting, insurance and financial services and we are likely to see it increasing in other sectors including housing, healthcare, retail and many others. It has great potential to improve services for consumers but it is also potentially harmful."

We reached out to Airbnb Australia and New Zealand, but they did not respond. However, Choice made contact with Airbnb Australia with a list of detailed questions about the way their algorithm is being used. The company's response was extremely vague.

Derek Nolan, Airbnb's Head of Public Policy, Australia and New Zealand said: "The safety, security and privacy of our community is one of our top priorities. Our platform security and safety measures are designed to help ensure stays are safe and positive experiences for Hosts, guests and the wider community – while also protecting users' information, including their personal information."

Bower says that this response isn't good enough and that Airbnb has a duty to inform customers how it's gathering and using their data.

If you’ve been denied an Airbnb rental or found yourself mysteriously kicked off the site, we want to hear from you! Contact: info@consumer.org.nz with ‘Airbnb’ in the subject line.

Could you be banned from Airbnb for your Instagram posts?

Did I just… get a lifetime @Airbnb booking ban for a 9 year old possession charge?!? 🤯

Tweet: @bethanyhallam