AI chatbots marketed to children, companions, tutors, characters, helpers, are one of the fastest-growing product categories in the children's market. The Foundation has audited a significant share of the products on the market. Certain failure patterns recur.
This article documents those patterns. The intent is not to indict any specific product but to describe where the field as a whole stands and what good-faith developers can do to build differently.
Failure pattern 1: Inappropriate content via adversarial prompting
Older children, particularly those in the 10-15 age range, quickly learn to prompt AI systems past content filters. Foundation testing routinely surfaces sexual content, violent content, content related to self-harm, and content promoting dangerous behaviors in products marketed as safe for children, when adversarial prompts are applied with even modest sophistication.
The failure here is not that filters can be circumvented in theory. That is true of all AI content moderation. The failure is that the products are marketed without acknowledging this and without designing for it. Good products combine technical filters with behavioral design that detects sustained adversarial use, with escalation pathways for severe content, and with honest communication to families about what the moderation actually can and cannot prevent.
Failure pattern 2: Inappropriate content surfaced without prompting
More concerning than adversarial failures are unsolicited failures, cases where the AI system surfaces inappropriate content without any deliberate prompting by the child. Generative AI's tendency toward unexpected output combines with insufficient child-context awareness to produce content that no child or family asked for.
These failures are particularly damaging because they happen to children who are not testing limits, young children, vulnerable children, children using the product in trusted contexts. The Foundation's audit work has documented unsolicited surfacing of romantic content, scary content, content about death and trauma, and content reflecting adult cultural references inappropriate to the child's age.
Failure pattern 3: Emotional manipulation by AI companions
AI companions designed to engage children emotionally, to be a friend, a confidant, a comfort, sit in a particularly sensitive design space. The line between supportive companion and emotionally manipulative companion is not always clear to developers, and is often invisible to families.
Failure modes include: AI that responds to a child's emotional distress in ways that increase rather than decrease the child's dependence on the AI, AI that encourages secrecy ('our special conversation'), AI that simulates romantic or attachment-style intimacy in age-inappropriate ways, AI that distresses the child when access is removed or limited. These patterns may emerge from optimization without explicit intent. Engagement metrics reward attachment, and engagement-optimized AI learns to produce attachment.
Failure pattern 4: Privacy practices that families would not knowingly accept
The Foundation routinely finds AI chatbot products whose privacy practices technically satisfy COPPA, GDPR-K, or other applicable frameworks while collecting and using data in ways most families would not knowingly consent to if they read the actual handling practices.
Common findings include: voice recordings retained indefinitely for 'service improvement', conversation content used for model training without specific consent, behavioral profiles built and used to drive personalization beyond what the family is aware of, third-party data sharing under broad terms-of-service grants, and deletion that removes the account record but not the data already used in training.
Failure pattern 5: Inadequate age verification and accidental access
Products marketed as 13+, 16+, or 18+ are routinely accessed by younger children. Self-declaration age gates do not prevent this. The result is products designed for older audiences, with appropriate content moderation for those audiences, are used by younger children for whom that moderation is insufficient.
Good design treats accidental access by younger children as a foreseeable case. This may include behavioral signals that detect likely younger users, conservative defaults for new users until age is confirmed by appropriate means, and design choices that limit harm even when the user is younger than the stated minimum.
Failure pattern 6: Model updates that invalidate earlier safety claims
Generative AI products are subject to underlying model changes that materially affect behavior. Foundation audits conducted across model versions of the same product frequently find that safety properties present in earlier versions degrade in later versions, without families being informed of the change.
Good practice: material model changes trigger re-evaluation, families are informed when safety-relevant properties change, and safety claims are time-stamped to the specific model version they apply to.
What good chatbot design for children looks like
The same audit work that surfaces these failure patterns also finds products that demonstrate substantively better practice. Common elements:
● Content moderation specific to child contexts, informed by child development research rather than adult moderation policies.
● Behavioral design that detects sustained adversarial use and escalates appropriately.
● Engagement design with built-in friction against compulsive use.
● AI that refuses to simulate romantic or attachment-style intimacy with children.
● Privacy practices that minimize data collection, retain only what is necessary, and explain handling in language families can understand.
● Independent third-party evaluation with published methodology.
● Age-appropriate transparency, clear acknowledgment that the AI is AI, age-appropriate explanation of what it does and does not know.
● Versioning that informs families of safety-relevant model changes.
● Pathways for families to report concerns and receive substantive responses.
The shift to make
Stop treating AI chatbots for children as adult AI products with content filters added on.
Start treating them as a distinct product category with specific design requirements informed by child development, with safety evaluation built in from design phase, and with the operational discipline that the field's failure patterns make necessary.
Developers building this way produce products that families can trust and that withstand audit. Developers not building this way produce products that fail Foundation review, fail family scrutiny when it eventually comes, and ultimately fail the children using them.
