Children's AI products are governed by an evolving patchwork of privacy frameworks, COPPA in the US, GDPR-K provisions in the EU, the UK Age-Appropriate Design Code, India's DPDP child-specific provisions, California's AB-2273, and emerging frameworks across other jurisdictions.
Most current products in the market satisfy the technical requirements of these frameworks. Most fall short of what privacy by design, taken seriously, would actually require. This article walks through the gap, with specific attention to AI-specific complications that traditional children's privacy frameworks were not designed to address.
The frameworks at a glance
COPPA (US)
Verifiable parental consent for collection of personal information from children under 13, with specific operational requirements. COPPA 2.0 efforts and the FTC's COPPA rule updates are extending this.
GDPR Article 8 (EU)
Specific rules for processing of children's data based on consent, with age thresholds set by member state between 13 and 16. The Better Internet for Kids strategy and DSA layer additional considerations.
UK Age-Appropriate Design Code (Children's Code)
15 standards for design of online services likely to be accessed by children, enforced by the ICO. Goes substantially beyond consent-based frameworks.
DPDP (India)
Specific provisions on processing of children's data, with verifiable parental consent and restrictions on tracking, behavioral monitoring, and targeted advertising directed at children.
California AB-2273
Age-Appropriate Design Code for California, mirroring much of the UK framework with state-specific provisions.
Each framework has specific provisions. The frameworks converge on similar broad principles: minimize data collection from children, restrict secondary use, prohibit profiling for advertising, require meaningful consent, and design for the child's interest rather than purely the commercial interest.
Where current AI products fall short of these principles
The Foundation's audit work consistently finds gaps in seven areas:
1. Data minimization that isn't
Products collect substantially more data than the service requires, voice recordings retained for product improvement, full conversation logs retained for analytics, behavioral data captured for personalization. Technically allowed under broad terms of service; not aligned with the principle of minimum necessary data.
2. Retention that doesn't expire
Privacy policies state that data is retained for 'as long as necessary'. In practice, the necessary period extends indefinitely. Children's voice recordings, conversation content, and behavioral profiles from early childhood remain in vendor systems through adolescence and beyond.
3. Training data use without specific consent
Conversation content, voice samples, and behavioral data are used to train or improve models. Often this is permitted under broad consent terms. Rarely is it consented to specifically. The child whose conversations train the next model version had no meaningful opportunity to consent.
4. Profiling for personalization that extends to advertising
Behavioral profiles built for personalization within the product are reused for advertising, targeting, or sharing with third parties. Even where direct advertising to children is restricted, indirect uses of the profile, to target parents based on inferred child characteristics, to drive purchase recommendations, to inform other commercial activity, happen routinely.
5. Third-party data sharing under broad grants
Terms of service grant broad rights to share data with 'service providers', 'affiliates', and 'partners'. The categories are broad enough to encompass essentially any data sharing the developer chooses to engage in. Families have no practical way to evaluate what this means for their child's data.
6. Deletion that doesn't delete
Account deletion removes the account record. Data already used in training stays in trained models. Data shared with third parties stays with those third parties. Aggregated and anonymized derivatives stay in analytics systems. Families requesting deletion rarely receive what they think they are requesting.
7. Consent flows children cannot meaningfully consent to
Even where consent is captured, it is captured in forms children cannot meaningfully evaluate. Parental consent, where required, is often captured through self-declaration of parental status with no verification. Both forms produce consent records that satisfy frameworks technically without producing meaningful informed consent in substance.
AI-specific complications
AI in children's products creates specific complications that traditional children's privacy frameworks were not designed to address.
● Inference at scale, AI can infer information about a child, mood, family situation, learning struggles, mental health concerns, that the child never disclosed and the family never consented to having inferred
● Voice and biometric data, voice samples used to train voice AI are biometric data; the implications for the child throughout their life are significant
● Long-term retention against an unknown future, data collected now may be used in ways not yet imagined when the child becomes an adult; consent today cannot meaningfully cover unknown future uses
● Cross-product profiles, AI systems trained on one product's data are deployed across other products, propagating profile-like patterns across contexts the family did not intend
● The training-data question, children whose conversations train models become, in effect, contributors to commercial AI capability without recognition, compensation, or meaningful consent
What privacy by design for children's AI actually looks like
● Data minimization treated as a design constraint, only what the service requires, with explicit justification for each data element collected
● Retention with default deletion, short retention windows by default, with longer retention requiring explicit purpose and family approval
● Training data use as a separate consent, children's conversations not used for model training without specific consent for that purpose
● Profiling limited to in-product personalization, no extension of behavioral profiles to advertising, targeting, or third-party use
● Third-party data sharing limited to specific named purposes, not blanket terms-of-service grants
● Deletion that actually deletes, including from training data, third-party systems, and derivative analytics where practicable
● Consent flows designed for the actual user, families need to understand what they are consenting to in language and structure they can actually parse
● Independent privacy review with published methodology, not internal compliance assessment only
● Transparency about model training, inference, and use, families informed in plain language what AI is doing with their child's data
The shift to make
Stop treating children's privacy as a compliance ceiling, what is the most we can do while satisfying the framework?
Start treating it as a design principle, what is the minimum we need to collect and use to deliver the service well? What information should we deliberately not collect, even though collecting it would be technically allowed? What rights to deletion and control should families have, even where the framework doesn't require them?
Privacy-by-design products earn family trust, withstand audit, and produce a sustainable relationship between developer and family. Compliance-ceiling products satisfy frameworks today and lose trust as families learn what their children's data is actually being used for.
