Report 6136
A cash-strapped council is using AI to help determine whether elderly residents should go into care.
Derby city council has built an algorithm trained on adult services data to produce recommendations on the care and assistance it should offer. A human will then either approve these suggestions or make adjustments.
The automated recommendations will also include decisions on supported living applications, care reviews, placements and blue badges for disabled drivers. A similar AI-powered service handles special needs payments to the families of disabled children.
Charities have urged caution when using AI because the people reliant on such services are among the most vulnerable in society.
Dennis Reed, the director of Silver Voices, a campaign group, said: "We are running before we can walk with AI, and older and vulnerable people are being used as a test bed with no safeguards in place. A bot assessing care needs -- whatever next?"
Derby city council's adult services were graded as "requires improvement" by the Care Quality Commission (CQC), and officials turned to AI because budgetary constraints meant there was no money to improve staffing levels.
The project is part of a set of AI tools the council has commissioned from ICS.AI, a tech company, as part of a £7m contract.
Documents obtained by The Telegraph through freedom of information requests show that the authority is hoping to make £6m in savings in adult services alone as a result of AI adoption.
The authority is pioneering the use of AI for public services and was the first to launch a generative AI phone-answering voice agent, called Darcie.
Data show that it is able to answer residents' questions about council tax, bin collection and similar topics correctly about half the time, but has struggled with the local accent and slang. A range of councils are already using ICS.AI agents, with some understood to be considering launching Darcie-like generative updates.
Derby city council, like many local authorities, is running a threadbare service after years of budget cuts. It is facing a £4m overspend this financial year.
Andy Appleyard, the director of adult social care services, said the council had significant backlogs and was also seeing a rise in demand for services.
"More people are coming to our front door, and the level of complexity for those individuals is greater than it's been in the past," he said in a video conference call with ICS.AI, posted on YouTube.
"The problem that we had was how do you start to look at doing something differently when you know that from a budgetary point of view that you can't start to just bring in lots more staff and do things in a traditional way of fixing some of our issues. That's really where AI came in for us.
"Taking out the financial side of things, our issue was that we had rising demand, we had significant backlogs, and we couldn't bring in more resources. So we needed to find a different way of doing things."
He added that the products Derby hoped to launch soon made recommendations based on a person's file and what AI had learnt from the council's database of adult services.
"Initially we were looking at a product that was going to support us to make recommendations about reviews," said Mr Appleyard. "The proposal was that it could search through our system and it could find suitable recommendations about how care and support could be delivered.
"The products that will be deployed in Derby very soon specifically will make recommendations about what care and support can look like.
"We'll never have AI making decisions about how care and support should be delivered here in Derby, but what the proposal was was that it could search through our system and it could find suitable recommendations about how care and support could be delivered.
"AI will not be making decisions. It will be staff looking at the recommendations of AI and saying: 'I agree with that, that's the approach that I want to take.'"
Hardyal Dhindsa, a councillor, said Derby turned to "wholesale AI digital innovation across the council" to cope with budget cuts from central Government, which had taken services "down to the bone".
He added: "We went for a roots and all, hearts and minds approach to delivering innovative AI solutions to the challenges that local authorities face."
'Everyone should be careful'
ICS.AI guarantees councils £5m a year in savings using the technology, and Derby said it had made £7.5m in savings so far. The authority is being watched closely by other councils as they cautiously embark on their own AI adoption.
More strained councils struggling to prop up services with dwindling budgets and rising demand could soon follow Derby's example, but Caroline Abrahams, the charity director at Age UK, called for restraint.
She said: "AI is here to stay -- and used in the right way, it should benefit us all in time, as well as helping hard-pressed public bodies to manage within tight budgets.
"However, it is still very early days with this new technology, so while we should not be afraid of it -- and should indeed embrace it -- we are still finding out what it can do really well and what it can't.
"People in need of social care are by definition vulnerable, and so everyone should be careful to ensure that their best interests are always the first consideration when new approaches are trialled, and that any resultant risks are properly managed."