AI and Gender Bias in Care: Why Transparency and Oversight Matter
A recent study by the London School of Economics (LSE), reported in The Guardian, has raised urgent concerns about gender bias in artificial intelligence tools used by English councils for adult social care. The research found that AI-generated summaries of case notes often downplay women’s physical and mental health needs compared to men’s — a disparity that could directly affect the quality and quantity of care women receive.
The findings are stark: language such as “complex” or “unable” appeared far more frequently in descriptions of men’s needs, while similar cases for women were framed in more positive — but ultimately dismissive — terms, such as “independent” or “able to manage daily activities.” This suggests that women may be at risk of being provided with less support, even when their health needs are comparable.
Among the models tested, Google’s “Gemma” tool showed the most pronounced disparities. Meta’s “Llama 3” did not demonstrate the same gender differences, underscoring the need for transparency around which AI systems are being deployed in sensitive public sector contexts.
Dr. Sam Rickman, lead author of the study, highlighted the wider implications: “These models are already being used very widely and what’s concerning is that we found very meaningful differences… This could result in women receiving less care if biased models are used in practice.”
At the World Health Innovation Summit (WHIS), we believe that innovation must go hand-in-hand with equity. Artificial intelligence has the potential to transform care, but only if it is built and deployed responsibly.
Gareth Presch, CEO of WHIS, said: “The LSE research is a wake-up call. AI should never amplify inequality. If these tools are being used to make decisions about care, they must be transparent, accountable, and rigorously tested for bias. Innovation must support fairness, particularly in women’s health, where inequalities have persisted for far too long.”
The report concludes that regulators must mandate the measurement of bias in large language models used in long-term care. Without robust oversight, the promise of AI could inadvertently deepen the very inequalities health systems are working to close.
At WHIS, we continue to champion responsible innovation that puts people and fairness at its core