Incident 189: Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities
Suggested citation format
The UK government uses invasive data gathering techniques to try and prove that people claiming benefits are actually living lavish lifestyles.
Social media monitoring, covert physical surveillance, and data gathering from airlines, PayPal accounts, and bingo clubs, are among the tactics that the UK government has been using to monitor welfare claimants.
The Department for Work and Pensions (DWP), which is responsible for the UK’s social security system and benefits support for some of the country’s most vulnerable, deploys excessive surveillance methods, according to a new report by Privacy International. Cracking down on benefit fraud has been a target of successive UK governments, despite it making up a tiny proportion of benefits administered.
VICE World News spoke to Rachel Fletcher, a solicitor at the Manchester firm Slater Heelis who represents Ellen, a benefit claimant who has been subject to DWP surveillance. Ellen, whose name has been changed to protect her anonymity, lives with a chronic illness. The surveillance was carried out after the DWP began to suspect she wasn’t entitled to disability support.
Undercover DWP investigators trailed Ellen and filmed her at a fundraising event – one which she was attending to raise money for a charity for her illness. “She couldn’t actually take part in the event because of her illness and she left in tears,” says Fletcher. “But the DWP doesn't show that, they just show her arriving.”
According to a 995-page, heavily reacted DWP employee handbook analysed by Privacy International, surveillance of benefit claimants is permitted as long as it remains “covert, but not intrusive”. But heavy-handed tactics suggest that the social security system has become the site of potentially illegal surveillance against those receiving welfare and disability payments.
Ellen had been receiving Personal Independence Payments (PIP), a benefit that’s awarded to help cover everyday costs like transport and assisted living for those with disabilities and long-term illnesses. In 2018, a few months after being awarded the benefit, she started a new health treatment that gave her intermittent relief – some days she could move around easily, and other days she wasn’t able to get out of bed.
Still, she contacted the DWP to let them know that her condition had changed, and was told that she didn’t have to worry about any changes in her PIP until her benefit reassessment date, which was scheduled to take place in several years’ time. A few months later, the Department called her in for an “Interview Under Caution” and accused her of faking her disability.
At the interview, the DWP presented data obtained from an airline and a social club that she attended. Fletcher, who attended the interview alongside Ellen, says that the evidence was meant to portray her as an affluent spender that had no need for disability assistance. “They start playing all the evidence they'd been gathering over a period of about six weeks [...] Evidence of her going about her daily life, unloading her car, leaving coffee shops and exercising.” All of it was meant to show that she was defrauding the benefit system.
“They looked at things like her flights and [said] that she'd upgraded her flights. But they didn't appreciate that the reason she upgraded was that she needed more legroom because of her illness,” says Fletcher. Ellen had been advised by her GP to stay active, but the DWP fraud investigators filmed her exercising and presented it to her at the interview as proof that she should have her benefit withdrawn and face criminal charges.
They’d also approached her employer and other people that she knew to obtain information about her. “At this stage, they’re supposed to be just getting an account from you,” explains Fletcher. “And yet you turn up to an interview to find out they've already spoken to your associates and your employer, who may not have told you about it. It's embarrassing. You're supposed to be innocent until proven guilty, but people think there's no smoke without fire, so your reputation is tarnished. Even before you’re charged with an offence.”
According to the DWP’s employee handbook, benefit fraud cases are fed from the Department to journalists as part of a media strategy to increase visibility around these cases. “There is a procedure in place that says when you catch someone, you go and tell the tabloids,” explains Eva Blum-Dumontet, a senior researcher with Privacy International. Although details of the DWP’s coordination with media outlets is redacted, the systematic release of information about benefit fraud cases point to the Department’s role in perpetuating the myth of “benefit cheats” who take lavish vacations using taxpayer funds.
The Privacy International report suggests that this kind of DWP evidence-gathering may be in violation of the Regulation of Investigatory Powers Act 2000 (RIPA), which dictates how the government can surveil and monitor UK residents. The DWP is only partially authorised to use investigation and data-gathering tactics, and is not legally permitted to use information obtained from those who know and have a relationship with the person under investigation – known as Covert Human Intelligence Sources (CHIS) – who essentially act as informants. The DWP handbook acknowledges the legal limits around CHIS, but also offers 8 pages of guidance explaining how to gather information from them.
“Under RIPA, they’re not allowed to use Covert Human Intelligence Sources,” says Blum-Dumontet. “And yet the guide is full of: ‘Oh, you're not allowed to use this. But hey, if you have a Covert Human Intelligence Source that suddenly comes at you, here is how you should handle that.’ It's very cheeky.”
Fletcher says that from her experience working with claimants, DWP investigations are aggressive and often disproportionately severe. “The police don’t often use those powers that early in an investigation – whereas the DWP will.”
Many benefit cases Fletcher works on include instances of social media monitoring, which the DWP justifies on the grounds that the personal data they use is considered open source. “Just because you've got mobility issues doesn't mean you're not entitled to public life or a social life. They’ll print off photographs that you have on there [social media] and put them to you in an interview.” The tactic often has the effect of being intimidating and invasive for claimants being scrutinised by the DWP.
The investigations and breaches of privacy often take a heavy toll on claimants like Ellen. “She's quite frightened to leave the house,” says Fletcher. “She thinks that she's been watched all the time. She's scared to even go for a walk... it’s affected her family… It's affected her very badly.”
Blum-Dumontet says that the DWP’s use of data from third-party sources, as well as the Department’s use of data-matching algorithms, is a particular cause for concern. “Who wouldn’t feel paranoid? Who wouldn’t feel paranoid about the idea that when you buy something online the company is going to be sharing their data with the state that's giving you money.” Along with financial institutions and airlines such as EasyJet, the DWP can also source information from eBay, supermarkets, leisure centres and gyms, the BBC, and certain Sky subscription services.
The DWP’s staff guide notes: "PayPal, a subsidiary of eBay, are willing to provide information in response to a standard DPA [Data Protection Act] letter...the standard DPA letter contains a suitable form of words to provide them with the assurances that the information is required for law enforcement purposes.” Paypal ceased to be a subsidiary of eBay in July 2015.
“It's not just privacy, it also goes against a very fundamental right to dignity,” says Blum-Dumontet, who says that Privacy International is still trying to uncover details about how the DWP flags claimants for investigation.
Originally the DWP claimed that investigations were carried out when members of the public reported claimants through the DWP’s National Benefit Fraud Hotline. And in Ellen’s case, Fletcher says that she and Ellen suspect the investigation was triggered by a report to the DWP Hotline.
But frontline workers and those familiar with the benefit system have suggested that the tip line – which allows neighbours and family members to report on those they believe are committing benefit fraud – is highly ineffective. An investigation by the Independent found that 87 percent of fraud reported through the tip line were dead ends that resulted in no action on the part of the Department.
Interestingly, the DWP has suggested that it uses other methods to carry out fraud investigations as well. “When you go through the annual report of the DWP, there is quite a lot that's made about how they're developing this fancy artificial intelligence to cut down on fraud,” says Blum-Dumontet.
The DWP’s 2018-19 annual report noted that the Department was “using increasingly sophisticated data and analytical tools” to weed out fraud. Last August a representative for the DWP’s “Intelligent Automation Garage”, a unit of the Department that works closely with private AI companies like UiPath, noted that the facility was working on fraud detection technology and that there was, “[s]ome really exciting stuff on the cards.”
While an FOI request filed by Privacy International confirmed that the DWP was using data matching – what is essentially a simple algorithmic process – to flag benefit claimants for investigation, the DWP has been cagey about details. It’s unclear what behavioural patterns are triggering investigations. The DWP claims that revealing information about the fraud detection processes could potentially be used by criminals to carry out fraud.
But Blum-Dumontet says the DWP needs to be more transparent about how it uses algorithms. “We are at a turning point in society where we’re waking up to the fact that we need transparency on these issues – that you can’t use AI without clear safeguards and a clear understanding of what data is being used. How is this system going to work? And the fact that at this very early stage [the DWP] is already refusing to comply with any requests of transparency should really worry us.”
Labour Shadow Secretary for Work and Pensions Jonathan Reynolds expressed concern at the findings of the Privacy International report. “Labour believes in a compassionate social security system that treats people with the dignity they have every right to expect. Instead, we have a punitive Universal Credit system that is failing thousands of families who are unable to afford the essentials they need in this crisis.”
Ellen’s fraud case is ongoing, but Fletcher says that they’re working to have the case dismissed. “It has long-lasting psychological effects on claimants. I think it makes them very wary of asking for help when they often need it the most. I think it's unnecessarily intrusive.”
In a written statement a DWP spokesperson said: “Privacy International’s report grossly mischaracterises the use, and extent, of DWP powers, which are subject to independent scrutiny.
“The limited powers that the Department does possess are used to prevent and detect potential crime, with surveillance conducted only when the Department is investigating potential fraud, and even then only in cases where all other relevant lines of enquiry have been exhausted.”
The DWP also stated that “companies are not under any legal obligation to provide the requested data.”
EasyJet declined to comment on specific data sharing practices but a representative for the company told VICE World News that “as a principle we only provide data in line with our legal obligations, including data protection.”
A spokesperson for PayPal said: “PayPal only releases information to law enforcement authorities where there is a legal obligation to do so. This includes making sure that the request is correct, legitimate and lawful.”
Blum-Dumontet questions whether the DWP is making it clear to companies that there is no legal obligation to share customer data with them. "The very fact that [the DWP’s] guide written for their own staff list those companies with the type of data they can request suggests they know there is a fair chance that the data does get handed over."
When questioned on this point by VICE World News, the DWP declined to comment further.
Update: This article was amended on 8th March to reflect that Paypal ceased to be a subsidiary of eBay in July 2015.
Manchester group launches action after people with disabilities report high number of stressful checks for potential scams
Disabled people are being subjected to stressful checks and months of frustrating bureaucracy after being identified as potential benefit fraudsters by an algorithm the government is refusing to disclose, according to a new legal challenge.
A group in Manchester has launched the action after mounting testimony from disabled people in the area that they were being disproportionately targeted for benefit fraud investigations. Some said they were living in “fear of the brown envelope” showing their case was being investigated. Others said they had received a phone call, without explanation as to why they had been flagged.
The Department for Work and Pensions (DWP) has previously conceded that it uses “cutting-edge artificial intelligence” to track possible fraud but has so far rebuffed attempts to explain how the algorithm behind the system was compiled. Campaigners say that once flagged, those being examined can face an invasive and humiliating investigation lasting up to a year.
A legal letter has been sent to the DWP demanding details of the automated process that triggers the investigations. Greater Manchester Coalition of Disabled People (GMCDP), which sent the letter with the help of campaign group Foxglove, said a “huge percentage” of the group has been affected by the system. The group believes the government has a legal duty to be transparent about how the algorithm works.
Concerns were raised by the charity Privacy International, which first found references in a DWP report to its use of “cutting-edge artificial intelligence to crack down on organised criminal gangs committing large-scale benefit fraud”. A 2019 UN report into the “digital welfare state” said algorithms were “highly likely” to repeat biases reflected in existing data and make them even worse. It added: “Inbuilt forms of discrimination can fatally undermine the right to social protection for key groups and individuals.”
The government has until Friday to respond to the legal letter.
“We’re tired of the fear of the brown envelope and tired of being repeatedly forced by DWP officials just to justify who we are,” said Rick Burgess of the GMCDP. “It’s time for the DWP to come clean about how this algorithm works and explain why so many disabled people are flagged for investigation. Disabled people need support – not being ground down by a brutal system that assumes we are fraudulent until proven innocent.”
Foxglove said: “The DWP algorithm seems to send poor and disabled people to the call centre from hell. The DWP needs to explain exactly how its algorithm works and what is done to eliminate bias – or be taken to court.”
Officials stressed that specialist staff were involved in any cases of suspected fraud and error and support was offered to those who needed it. A DWP spokesperson said: “The DWP will be responding to the letter written by the representatives of the GMCDP in due course.”
The Department for Work and Pensions (DWP) has been questioned by MPs over its use of a secret set of instructions that are used by its computers to identify potential benefit fraudsters, and which appear to disproportionately target disabled people.
Two of the department’s most senior officials were quizzed yesterday (Wednesday) about the secret algorithm being used by DWP.
Earlier this week, Greater Manchester Coalition of Disabled People (GMCDP) announced that it had sent a legal letter to DWP to demand details of how the algorithm works, and requesting proof that it does not target any particular group.
It believes that disabled people are being unfairly and wrongly targeted by DWP for lengthy and stressful fraud investigations.
Labour MP Debbie Abrahams asked the two DWP officials, permanent secretary Peter Schofield and director general for change Neil Couling, about the algorithm when they appeared before the Commons work and pensions committee yesterday.
She pointed to a 2019 United Nations report (PDF) which warned of the risks of a “digital welfare dystopia” and, she said, concluded that the use of such algorithms in social security systems have “inherently built-in biases that are discriminatory”.
Abrahams, whose Oldham East and Saddleworth constituency is in Greater Manchester, said that some of the cases she had heard about through GMCDP of disabled people being targeted by DWP for fraud investigations were “very concerning”.
Schofield did not deny that DWP used an algorithm, but he insisted that any decision on action being taken by DWP “always comes down to an individual”.
He said: “The automation can help us guide our work but ultimately it’s a decision made by individuals.”
Couling (pictured) claimed that it “was not an algorithm in the way that people might understand an algorithm”, and he insisted that what DWP was doing was “data matching”.
He said: “What we’re doing is matching data sets to see whether there’s consistency or not in what’s being reported.”
He said it was “not a machine telling us what to do and making decisions on benefit cases.
“It is data matching that is identifying questions to be answered and then we put those questions to the claimants themselves and give them the right to say, well, what is going on here.”
Couling told Abrahams that he did not know what proportion of those claimants being investigated for fraud were disabled people.
But he added: “I don’t know the answer to that but given that a significant proportion of the benefit caseload are disabled you would expect to see some disabled people will come up in the data matching.”
Rick Burgess, from GMCDP, told Disability News Service after the meeting: “Data matching would imply some process that is operating as an algorithm, I suspect, so they are splitting hairs over terminology.
“As for a human decision-maker being involved, we are keen to find out about that, and are those decisions reflective of good judgment or simply nodding through machine-picked candidates.”
He said it was noticeable how Couling had not answered Abrahams’ question about the proportion of disabled people being investigated, as this was a “key” issue.
GMCDP has been working with campaign group Foxglove, which said this week: “This system seems to unfairly target disabled people for investigation but the DWP won’t reveal how it works, or what they’ve done – if anything – to make it unbiased.
“Once flagged, you are forced into a bureaucratic hamster wheel of endless call centre queues, frustrating discussions with unhelpful staff and long forms that ask the same questions over and over – seemingly designed to trip people up.
“This can last for up to a year.”
Burgess had told The Observer earlier this week: “We’re tired of the fear of the brown envelope and tired of being repeatedly forced by DWP officials just to justify who we are.
“It’s time for the DWP to come clean about how this algorithm works and explain why so many disabled people are flagged for investigation.”
The government is ‘forcing disabled people into gruelling and invasive benefit fraud investigations', a disability group has claimed.
A Manchester disabled people’s group has issued the government’s Department for Work and Pensions with a legal letter challenging how it singles applicants out for investigations to prevent benefits fraud, using an algorithm.
A spokesperson for the DWP explained a specialist member of staff analyses and reviews all potential cases suspected of fraud and error.
The group, the Greater Manchester Coalition of Disabled People (GMCDP), said people are now living ‘in fear of the brown envelope’ they could receive that could ‘throw their life into turmoil’.
Members said they have also received phone calls querying their status, without an explanation as to why they were being flagged for investigation.
Once flagged, the system ‘forces targeted people to repeatedly explain why they need payments in an aggressive and humiliating process that can last up to a year’, the GMCDP said.
The GMCDP said a ‘huge percentage’ of the group has been ‘targeted’ by the DWP system.
Some members said they have had to make long and frustrating calls to call centres, deal with confusing phone menus and unhelpful operators with no training to assist disabled and vulnerable people.
Others said they have been asked to fill in forms of over 80 pages that ask the same questions again and again, tripping them up.
The group has now sent a legal letter, supported by tech justice non-profit group Foxglove, demanding the government explain how the DWP’s algorithm works and what, if anything, it has done to eliminate bias.
The DWP has until December 15 to respond to that letter.
Charity Privacy International found a report from the DWP’s stating it was using ‘cutting-edge artificial intelligence to crack down on organised criminal gangs committing large-scale benefit fraud’ and it’s this algorithm that the GMCDP wants to challenge.
The group wants the DWP to demonstrate whether or not people singled out for investigation are disproportionately disabled, women, black, Asian or from minority ethnic groups and is calling for transparency so people have the opportunity to tell whether or not they are being treated unfairly.
Rick Burgess of the GMCDP said: “I suspect I have been flagged by this algorithm in the past. A huge percentage of disabled people have gone through it too. This causes us to be dragged through an invasive and highly stressful investigation, and frankly, we’ve had enough.
“We’re tired of the fear of the brown envelope and tired of being repeatedly forced by DWP officials just to justify who we are.
“It’s time for the DWP to come clean about how this algorithm works and explain why so many disabled people keep being flagged for investigation. Disabled people need support – not being ground down by a brutal system that assumes we are fraudulent until proven innocent.”
A Foxglove spokesperson said: “The DWP’s algorithm seems to send poor and disabled people to the call centre from hell. The DWP needs to explain exactly how its algorithm works and what it’s done to eliminate bias – or be taken to court.
“A machine in Whitehall spits out your name. It accuses you of lying about being disabled and shoves you into a bureaucratic hamster wheel of delays, doom loop phone menus and endless forms designed to trip you up. And it says nothing about why this is happening to you.
“This seems to be another case where government has created an algorithm that wields terrifying power over people’s lives with no input from the people it affects. The result is vulnerable people being trapped, humiliated and hurt. We’re proud to be working with GMCDP to make this right.”
A DWP spokesperson said: “The DWP will be responding to the letter written by the representatives of the Greater Manchester Coalition of Disabled People in due course.”
Campaigners say disabled people are being unfairly targeted and living in 'fear of the brown envelope' thanks to mysterious computer processes that the government has never fully explained.
The Department for Work and Pensions is facing fresh legal action to reveal what “algorithms” it uses to flag claims of benefit fraud.
The Greater Manchester Coalition of Disabled People has sent a “letter before action” demanding ministers “come clean” about techniques it claims is throwing disabled claimants into crisis.
The campaign group says disabled people have had questions raised over their benefit claims through a phone call, e-mail or “brown envelope” in the post without knowing why.
Sometimes they have essential cash “cut off” at the beginning of probes that can last over a year, the group said.
GMCDP spokesperson Rick Burgess said benefit claimants of all stripes are singled out, but campaigners believe the algorithm is “over-picking disabled people for investigations of fraud”.
He told a meeting tonight: “You have no idea how it came about that you’re being investigated - none whatsoever.
“It’s extremely upsetting when people get this.”
He added: “We are living in a shadow penal system that treats us like criminals for having to meet the same needs as everyone else. The fear of the brown envelope makes disabled people feel terrorised, in a constant state of anxiety and on edge.”
It is the second legal letter sent by the same group on the subject, with the first going out before Christmas.
The DWP responded in December, but the group alleges the government has still not answered key questions about what technology it uses to flag possible fraud.
The fresh legal letter - to which the DWP must respond in 28 days - challenges the “use of AI, algorithmic technology and other forms of automation” in investigation of benefit fraud.
It says the DWP uses “data matching” and “machine learning”, but other techniques may be being used and the DWP is not being clear about what they are.
The letter says there is a “risk of bias” to automated processing.
It demands the DWP set out “detailed information” on its use of computer processes and sets up “adequate safeguards”.
Barrister Will Perry, who is representing the legal action supported by the “tech justice” group Foxglove and Privacy International, said: “We know there is a data matching algorithm being used.
“The DWP seem to be experimenting with a wide range of different technologies.”
DWP Permanent Secretary Peter Schofield has previously insisted computer systems “guide our work” but “we do not take any action against anyone based on a machine.”
He added: “It always comes down to an individual.”
Disabled campaigners have sent a legal letter to the Department for Work and Pensions (DWP) to question how its use of a secret algorithm could be discriminating against disabled people in the way it selects claimants to face benefit fraud investigations.
Greater Manchester Coalition of Disabled People (GMCDP), which is being supported by the tech justice campaign group Foxglove, believes the algorithm is “over-picking” disabled people for its benefit fraud investigations.
It has now sent a formal letter to DWP to ask for information about how the algorithm is being used and for evidence that it does not discriminate against disabled people, and to warn of potential further legal action.
So far, DWP is refusing to say how people are being targeted, how it is ensuring disabled people’s rights are protected, and what checks and balances are in place to ensure benefit claimants are not being unfairly and unlawfully harassed.
Rick Burgess (pictured), from GMCDP, told an online event last week that algorithms “tend to reproduce the biases and discriminations in wider society and culture”.
He said GMCDP and Foxglove suspect that DWP’s algorithm is “over-picking disabled people for investigations of fraud”, resulting in extreme distress for the claimants who have been picked.
Burgess said this distress can be particularly severe if the claimant already experiences paranoia or anxiety.
He said: “It can be extremely debilitating. I know people have withdrawn from life because of it. I know there have been suicides because of it. It is extremely harmful to people.
“For those investigations to occur I think there needs to be an extremely high level of proof and suspicion before you can even start that process, because the risk to people is potentially lethal.”
He said the use of the algorithm by DWP had become a new barrier for disabled people.
He said: “Disabled people are consistently always having to be tested, always having to prove ourselves, always having to fill in forms, always having to pass tests, you’re always being questioned about your identity.”
GMCDP has started a crowdfunder to raise money for any legal costs it might face in the case against DWP, and is nearly halfway to its target of £15,000.
Burgess said the way fraud investigations are carried out by DWP is “absolutely, classically Kafkaesque”.
He said: “You don’t know who your accuser is. You don’t know why you’ve suddenly been picked out.
“Typically, it’s either an envelope, a phone call, or email. It’s extremely upsetting when people get this.”
He said that the “weird pattern” of disabled people being suddenly investigated for no apparent reason suggested a system that was “kind of making up its own rules”.
He added: “In an investigation, before anything has been proven, your benefits can be stopped so you are treated as guilty until proven innocent.”
Burgess said the idea that disability benefit fraud was widespread was a “masterpiece of propaganda” as the actual fraud rate for some disability benefits was as low as 0.5 per cent, but DWP “still treat everybody as a suspect”.
Laura Lazaro Cabrera, a legal officer with Privacy International, which campaigns against companies and governments exploiting data technology, said they originally uncovered evidence of the use of algorithms in 2019 in a DWP anti-fraud staff training manual and in DWP’s annual report and accounts for 2017-18.
The annual report said DWP was developing “cutting-edge artificial intelligence to crack down on organised criminal gangs committing large-scale benefit fraud”.
DWP later admitted to using “data matching and data analysis to help identify people who may not have declared their circumstances correctly”.
But it has so far refused to release any information about its use of algorithms to Privacy International, arguing that to do so would “prejudice the prevention and detection of fraud and crime”.
Will Perry, a barrister with Monckton Chambers, who is working on the GMCDP legal case, said DWP appeared to be “experimenting with a wide range of different technologies”.
He said the case focused on DWP’s lack of transparency, and was arguing potential breaches of rights under the European Convention on Human Rights and the General Data Protection Regulation (GDPR), as well as DWP’s obligations under the Equality Act’s public sector equality duty.
He said they expected a formal response to their legal letter next month, but he warned that it was “uncharted territory” for lawyers as the few previous UK cases have been settled before reaching court.
Rosa Curling, director of Foxglove, said they were hoping the legal action would show that algorithms such as those being used by DWP could be challenged in court, and that it would encourage other groups to consider taking action through the courts.
She said algorithms appear to be being used secretly across local and national government, “without proper scrutiny and accountability, and… are making decisions that are affecting everybody’s lives in an absolutely profound way”.
A DWP spokesperson said: “The DWP will be responding to the letter written by the representatives of the Greater Manchester Coalition of Disabled People in due course.”