The phrase 'duty of care' can be absorbed into an organisation's language without changing anything the organisation does. It can become another value on a page. This article is about what a duty of care toward vulnerable AI users actually requires in practice, and, first, about what it explicitly is not.
What a duty of care is not
A duty of care is not a disclaimer. A disclaimer attempts to transfer responsibility away from the system and onto the user; a duty of care accepts responsibility. The two move in opposite directions.
A duty of care is not a consent screen. Consent, for a vulnerable user, is precisely the mechanism in question, if a person cannot realistically understand what they are agreeing to, their agreement does not discharge anyone's obligation to them. A system that points to a consent screen as evidence it has met its duty has misunderstood the duty.
And a duty of care is not a statement of values. A published principle that the organisation cares about vulnerable users is not a duty of care; it is an aspiration. The duty exists only where it has been turned into things actually done. The test, throughout, is simple: a duty of care is discharged through what gets built and how it is run, not through what gets written about it.
Proportionality: how much is owed
A duty of care is not uniform. It is proportionate to two things. The first is the severity of harm the system can do, a system that can affect someone's access to money, healthcare, housing, legal standing, or safety carries a heavier duty than one whose worst failure is a minor inconvenience. The second is the limited recourse of the user, the less able users are to detect, contest, avoid, or recover from a failure, the heavier the duty. High potential harm combined with weak recourse produces the strongest duty; low harm and strong recourse, the lightest. Proportionality is what makes the duty workable rather than absolute.
In design
Discharging the duty in design begins by refusing the unexamined typical user. It means stating explicitly who the system is for, including the people at the edges of its intended use, and designing across that whole range. It means accessibility and clarity as baselines, not later additions. And it means designing the system's failures to be less consequential: softer failure modes, conservative behaviour when stakes are high, and human involvement at the points where a wrong automated decision would seriously harm someone.
In evaluation
The duty requires evaluating whether the system actually serves vulnerable users, which aggregate metrics cannot show, because a strong average routinely hides serious failure for a minority. So evaluation must disaggregate, measuring performance for specific populations. It must involve members of affected populations, because they identify the failure modes outsiders miss. And it must measure the components of vulnerability directly: whether users understand the system, can tell when it has failed, and can actually use its recourse.
In deployment
After launch, the duty requires clear communication about what the system does and does not do. It requires genuine recourse, an accessible, usable route to question and appeal an automated decision, reachable by the people most likely to need it, not merely existing on paper. It requires monitoring in real conditions, which surface failures testing did not. And it requires a route by which harm can be reported, heard, and acted on by someone empowered to change the system.
In governance
Finally, the duty must be held. It has to be a named, accountable responsibility, not a value diffused across an organisation and therefore owned by no one. Governance writes the duty into how decisions about the system are made, makes someone answerable for it, and ensures the system can be questioned, corrected, paused, or withdrawn when it is failing the people it owes the most.
Why the distinction matters
The reason for insisting that a duty of care is design, evaluation, deployment, and governance, and not disclaimers, consent screens, or values, is that the decorative version is actively dangerous. An organisation that believes its consent screen has discharged its duty will stop looking. It will not run the disaggregated evaluation, will not build the genuine recourse, will not catch the harm. The decorative duty of care does not merely fail to help; it produces a false confidence that prevents the real work. A duty of care is only real in what it causes to be built.
