Ethical AI for Vulnerable Populations: Why the System Creates the Vulnerability

Vulnerability to an AI system is created by the relationship between a person and the system, not fixed in the person. What a genuine duty of care to vulnerable users requires.

The phrase 'vulnerable populations' is used often in conversations about AI ethics, and it is usually used in a way that quietly does harm. It is used as a label for a fixed list of groups, children, older adults, people with disabilities, the poor, treated as inherently fragile, as though vulnerability were a permanent property they carry into every situation. That framing is inaccurate, and the inaccuracy matters. It locates the problem inside the person. It implies the condition is fixed. And it obscures something essential: the same person is vulnerable to some AI systems and not to others, and a person who is not vulnerable today can become so tomorrow through illness, crisis, or simple change of circumstance. Vulnerability, in the context of an AI system, is not a fact about a person. It is a fact about a relationship. This pillar sets out the Mobiloitte Foundation's position on ethical AI for vulnerable populations. It is the Foundation's cross-cutting pillar, the Foundation has examined two specific populations in depth already, children in the pillar on digital safety in the AI age, and older adults in the pillar on aging well with technology. This pillar steps up a level. It asks the general questions those two pillars are both instances of: what actually makes a population vulnerable to an AI system, what failure pattern recurs across every such population, and what a genuine duty of care toward vulnerable users would require of the people who design, deploy, and govern these systems. The Foundation's central argument is simple to state and demanding to act on. Vulnerability to an AI system is created by the system as much as by the person. And because it is created by the system, the obligation to address it falls on the people who can change the system, not on the people least equipped to protect themselves from it. What 'vulnerable' actually means here It is worth being precise, because the loose use of the word is part of the problem. In the context of a specific AI system, a person is vulnerable when two conditions hold together. The first condition is consequence. The system's failures, and every system fails sometimes, would carry serious consequences for this person. A wrong decision, a misclassification, a denied service, an inappropriate output matters more for them than it would for someone else, because of what is at stake or how little margin they have to absorb it. The second condition is recourse. This person has limited ability to detect that the system has failed, to contest the failure, to avoid the system in the first place, or to recover afterwards. They may not know the decision was automated, may not understand it well enough to challenge it, may have no realistic alternative to using the system, or may lack the time, money, knowledge, or standing to appeal. Both conditions are about the relationship between the person and the system. Neither is a fixed deficiency in the person. And that is the whole point: change the system, make its failures less consequential, make its workings clearer, give it a genuine appeal route, design it without assuming a narrow typical user, and the vulnerability shrinks or disappears, with nothing whatever having changed about the person. The Foundation uses the term algorithmic vulnerability for exactly this reason: to keep the definition relational, and to keep the responsibility where it can actually be acted on. The pattern that recurs across every population The Foundation has looked closely at AI and children, and at AI and older adults. The striking finding is not how different those two cases are. It is how similar the underlying failure is, and how the same failure recurs for populations as different from each other as it is possible to be. The pattern has four parts, and they compound. Built around a presumed typical user An AI system is designed, by default, around an implicit picture of the person who will use it, their language, literacy, device, connectivity, abilities, circumstances, and needs. This typical user is rarely stated and rarely examined, but they are built into a thousand small decisions. People who differ from the presumed typical user, and vulnerable populations differ from it in some specific, consequential way almost by definition, are served by a system that was not shaped with them in mind. Underrepresented in the data AI systems learn from data, and vulnerable populations are typically underrepresented in that data, because they are harder to reach, because historical data already undercounts them, or because they were never considered when the data was assembled. A system trained on data that thinly represents a group will perform worse for that group. So the people who can least afford a system that works poorly are precisely the people the system works most poorly for. Underrepresented in the building and the testing Vulnerable populations are also typically underrepresented among the people who build AI systems and among the people the systems are tested with before launch. A failure mode that would be obvious to a member of an affected group, or to someone who designed the test with that group in mind, simply goes unnoticed. The system ships with the failure intact, and the failure is discovered later, in the world, by the people harmed by it. Least able to access recourse Finally, when the system does fail a vulnerable user, that user is typically the least able to do anything about it. Recourse, knowing a decision was automated, understanding it, finding the appeal route, having the time and standing to use it, is hardest to access for the people most likely to need it. The original harm is then compounded by the inability to correct it. These four parts form a single mechanism, and it is the same mechanism whether the affected population is children or refugees or people with a visual impairment or people in deep poverty. The specifics differ enormously. The structure does not. That is why the Foundation treats this as one problem with many faces rather than many separate problems, and why a position on it can be cross-cutting rather than population-by-population. Who is vulnerable, and why the list is not the point It is reasonable to ask which populations this concerns. A non-exhaustive list would include children and young people; older adults; people with disabilities of every kind; people living in poverty or financial precarity; people with limited literacy, including digital and financial literacy; people with limited digital access or connectivity; linguistic minorities and speakers of under-resourced languages; ethnic and racial minorities; people with serious physical or mental illness; refugees, migrants, and displaced people; and people in situations of dependence or institutional care, where someone else mediates their access to systems. But the Foundation's position is that the list, while useful, is not the point, and treating it as the point reintroduces the very error this pillar is arguing against. Three reasons. First, the list is relational: every group on it is vulnerable to some systems and not others, so the list only means anything alongside a specific system. Second, the list is incomplete by nature, because new systems create new vulnerabilities and no fixed list anticipates them. Third, and most importantly, vulnerability is often situational rather than permanent. Serious illness, a financial shock, bereavement, displacement, a mental health crisis, or simply a moment of distraction or distress can place almost anyone into a vulnerable relationship with a system, temporarily. The person who confidently navigates a system on an ordinary day may be unable to on the day a crisis lands. This is why designing only for a fixed list of named groups is insufficient. The more robust approach designs for the relationship, for clarity, for recourse, for the absence of a narrow presumed user, which protects the named groups and also the much larger number of people who are situationally vulnerable and would never appear on any list. The duty of care runs the other way Here is where the Foundation's position becomes a claim about obligation, not just description. The default arrangement in much of the AI industry places the burden of protection on the user. The system presents terms, a consent screen, a set of controls, a privacy policy; the user is presumed to read, understand, and decide; and if something goes wrong, the user is presumed to have agreed. This arrangement is questionable for any user. For a vulnerable user it is indefensible, because the entire premise, that the user can read, understand, and meaningfully decide, is precisely what their vulnerability calls into doubt. Consent obtained through an interface a person cannot realistically understand is not meaningful consent. A system that relies on it has not obtained agreement; it has shifted a risk it created onto the person least able to carry it. The Foundation's position is that the obligation runs the other way. A system used by vulnerable people should carry a duty of care toward them, a duty proportionate to two things: the severity of the harm the system can do, and the limited recourse its users have. The greater the potential harm and the weaker the user's ability to detect, contest, or recover from it, the stronger the duty. This is not a radical principle. It is the principle already applied in most other fields where products and services meet people who cannot fully protect themselves, and AI is overdue to be held to it. A duty of care, crucially, is not a disclaimer and not a consent screen. It is an obligation that shapes what gets built. It is discharged through design, evaluation, deployment, and governance, not through a paragraph that transfers liability. What a duty of care actually requires In design Designing under a duty of care means refusing the unexamined typical user. It means stating explicitly who the system is for, including the people at the edges of its intended use, and designing for that full range, clarity of communication, accessibility as a baseline rather than an add-on, interfaces that do not assume a particular literacy, language, device, or ability. It means designing the system so that its failures, when they come, are less consequential, softer failure modes, conservative behaviour at high stakes, human involvement where a wrong automated decision would seriously harm someone. In evaluation A system cannot be known to serve vulnerable users unless it is evaluated for them specifically. That means evaluation that disaggregates, that measures how the system performs for particular populations rather than only in aggregate, because strong average performance routinely conceals serious failure for a minority. It means involving members of affected populations in the testing, because the failure modes that matter to them are the ones outsiders most reliably miss. And it means evaluating not only accuracy but the things vulnerability is made of: whether users understand the system, whether they can tell when it has failed, whether recourse actually works. In deployment A duty of care extends past launch. It means clear communication, in the deployment, about what the system does and what it cannot do. It means a genuine, accessible, usable route to question and appeal an automated decision, not a route that nominally exists but is unreachable for the people most likely to need it. It means monitoring the system in the world for the harms that disaggregated evaluation was designed to catch, because deployment conditions reveal failures that testing does not. And it means a way for harm to be reported, heard, and acted on by someone with the power to change the system. In governance Finally, the duty of care has to be held somewhere. It must be a named, accountable responsibility within the organisation, not a value diffused across everyone and therefore owned by no one. Governance means the duty is written into how decisions about the system are made, that someone is answerable for it, and that the system can be questioned, corrected, paused, or withdrawn when it is failing the people it owes the most. The commercial discipline of AI governance, registry, risk-classification, review, monitoring, is the machinery; the duty of care toward vulnerable users is one of the things that machinery exists to protect. Inclusive design is not a trade-off A common objection is that all of this is a cost, that serving vulnerable users well means accepting a worse or more expensive product for everyone else. The evidence does not support the objection, and the Foundation considers it important to say so plainly. The things a duty of care requires, clarity of communication, accessibility, robust performance across varied conditions, genuine recourse, conservative behaviour at high stakes, are not narrow accommodations. They are general improvements. Clear communication helps every user, not only the one with low literacy. Accessibility features are used constantly by people who would never describe themselves as disabled. A genuine appeal route improves trust for everyone. A system robust enough to perform well for an underrepresented group is, simply, a more robust system. The recurring finding across accessibility and inclusive-design research is that designing for the edge improves the centre. There are real costs, the effort of disaggregated evaluation, of involving affected communities, of building genuine recourse. But they are the costs of building the system properly, and they are repaid in robustness, trust, and reach. The framing of inclusion as a trade-off against quality is, in most cases, simply mistaken. The test of a real AI ethics practice The Foundation will close with the argument it considers most important. Many organisations now have an AI ethics practice, principles, a committee, a set of commitments. Some are substantive and some are decorative, and from the outside the two can be very hard to tell apart. How an AI system treats its most vulnerable users is the clearest available test. A decorative ethics practice produces principles that read well and a system that still quietly fails the people with the least power to object, because failing them is cheap, they are unlikely to be heard, and the principles were never connected to the system's actual design. A genuine ethics practice shows up precisely here: in disaggregated evaluation that looks for harm to specific populations, in recourse that actually works for the people most likely to need it, in a duty of care that shaped the system before any harm appeared. Ethical AI for vulnerable populations is not a sub-topic of AI ethics, to be handled after the main work. It is the test that reveals whether the main work was ever real. The Foundation's role The Mobiloitte Foundation works on this in the ways a foundation can. It examines specific populations in depth, children and older adults so far, with the same rigour to be extended to others. It publishes evidence-led, plain-language analysis intended to be useful to the people who design, deploy, regulate, and report on AI systems. It advocates for the principle this pillar sets out: that vulnerability to an AI system is relational, that the duty of care runs toward the user, and that how a system treats its most vulnerable users is the measure of whether its ethics are real. The Foundation does not claim this is solved or simple. It claims that the framing matters, that locating vulnerability in the relationship rather than the person changes who is responsible and what they must do, and that this framing should be the starting point for anyone building, governing, or regulating AI that will be used by people who cannot fully protect themselves from it. Which, given how situational vulnerability is, means very nearly everyone, eventually. Frequently Asked Questions What does 'vulnerable populations' mean in the context of AI? In the context of a specific AI system, a vulnerable population is any group for whom the system's failures would carry serious consequences and who have limited ability to detect, contest, avoid, or recover from those failures. The Foundation stresses that this vulnerability is relational, it arises from the relationship between the person and the system, not from a fixed deficiency in the person, and uses the term algorithmic vulnerability to keep that meaning clear. Why does the Foundation say vulnerability is created by the system? Because the conditions that make a person vulnerable to a system, consequential failures and limited recourse, can all be changed by changing the system. Clearer communication, design that does not assume a narrow typical user, a genuine appeal route, human review at high-stakes points: each of these reduces vulnerability with nothing changing about the person. If the system can create or remove the vulnerability, the system, and the people who build it, are where the responsibility sits. Which groups count as algorithmically vulnerable? A non-exhaustive list includes children, older adults, people with disabilities, people in poverty, people with limited literacy or digital access, linguistic minorities, ethnic and racial minorities, the seriously ill, refugees and displaced people, and people in dependent or institutional care. But the Foundation cautions against treating any fixed list as the point: vulnerability is relational and situational, and a crisis, illness, or shock can place almost anyone into a vulnerable relationship with a system temporarily. Why are AI systems worse for vulnerable populations? Because the same four-part pattern recurs: systems are built around a presumed typical user that vulnerable people differ from, those people are underrepresented in the training data so the system performs worse for them, they are underrepresented in the teams and testing so failure modes affecting them go unnoticed before launch, and they have the least access to recourse when the system does fail them. The four parts compound into systems that serve worst the people who can least afford it. Isn't consent enough to protect vulnerable users? No. Consent obtained through an interface a person cannot realistically understand is not meaningful consent. For a vulnerable user, the very premise of the consent model, that the user can read, understand, and meaningfully decide, is what their vulnerability calls into question. A system that relies on consent screens to protect vulnerable users has shifted a risk it created onto the people least able to carry it, rather than discharging a genuine obligation. What is a duty of care in this context? It is an obligation, held by the people who build and deploy an AI system, toward the people who use it, proportionate to the severity of harm the system can cause and the limited recourse its users have. It is not a disclaimer or a consent screen. It is discharged through design, evaluation, deployment, and governance: by building the system so it serves people at the edges, evaluating whether it actually does, deploying it with genuine recourse, and governing it so someone is accountable for the people it owes the most. Does designing for vulnerable users make AI worse for everyone else? Generally the opposite. The things a duty of care requires, clear communication, accessibility, robustness across varied conditions, genuine recourse, conservative behaviour at high stakes, are general improvements that help all users. Accessibility features are used widely by people who would not describe themselves as disabled; clear communication helps everyone; a system robust enough for an underrepresented group is simply more robust. Inclusive design is rarely a trade-off against quality. How is this pillar different from the Foundation's pillars on children and older adults? The pillars on children's digital safety and on aging well with technology each examine one population in depth. This pillar steps up a level to the cross-cutting question, what makes any population vulnerable to an AI system, and what failure pattern they all share. Children and older adults appear here as two instances of a general pattern rather than as the subject. The three pillars are designed to be read together: this one for the principle, the other two for the depth on those specific populations. What can technology builders do about this? Refuse the unexamined typical user and state explicitly who the system is for, including people at the edges of its use. Build accessibility and clarity in as a baseline. Evaluate in a disaggregated way, measure performance for specific populations, not only in aggregate, and involve affected communities in testing. Build genuine, usable recourse. Design failures to be less consequential, with human review where a wrong decision would seriously harm someone. And make the duty of care a named, accountable responsibility rather than a diffuse value. Why does the Foundation call this the test of a real AI ethics practice? Because a decorative ethics practice and a genuine one are hard to tell apart from the outside, both produce principles and committees. How a system treats its most vulnerable users is where the difference becomes visible. Failing those users is cheap and rarely contested, so a decorative practice tends to fail them quietly while its principles still read well. A genuine practice shows up in disaggregated evaluation, working recourse, and a duty of care that shaped the system before any harm appeared. The treatment of the most vulnerable users reveals whether the ethics were ever connected to the system at all. Key Facts ● Vulnerability to an AI system is relational, it arises from the relationship between a person and a specific system in a specific context, not from a fixed deficiency in the person. ● The same person can be vulnerable to one AI system and not to another; vulnerability is not a permanent or total status. ● A person is algorithmically vulnerable to a system when two conditions hold together: the system's failures would carry serious consequences for them, and they have limited ability to detect, contest, avoid, or recover from those failures. ● Across very different groups, the same failure pattern recurs: AI systems are built around a presumed typical user, and people who differ from that presumed user are served worse, harmed more, and heard less. ● Vulnerable populations are typically underrepresented in the data AI systems are trained on, so systems perform worse for exactly the people who can least afford poor performance. ● Vulnerable populations are typically underrepresented in the teams that build AI systems and in the testing those systems undergo, so failure modes that affect them are less likely to be noticed before deployment. ● Recourse, the ability to question, appeal, or correct an automated decision, is hardest to access for the people most likely to be harmed by a wrong one, which compounds the original harm. ● Many people are situationally vulnerable rather than permanently so, illness, crisis, financial shock, displacement, or bereavement can place anyone into a vulnerable relationship with a system temporarily. ● A system that works well for vulnerable users tends to work well for everyone, because the clarity, recourse, and robustness it requires are general improvements, inclusive design is rarely a trade-off against quality. ● A duty of care toward vulnerable users should be proportionate to the severity of harm the system can cause and the limited recourse its users have, and should shape design, evaluation, deployment, and governance from the start. ● Consent obtained through interfaces vulnerable users cannot realistically understand is not meaningful consent, and a system that depends on it has shifted a risk it created onto the people least able to carry it. ● How an AI system treats its most vulnerable users is t