Children are growing up with AI in ways no generation before them has.
By the time a child born in 2026 is ten years old, they will have spoken to more AI systems than to many of their human teachers. They will have AI tutors that adapt to their learning. AI companions that listen when they are lonely. AI toys that converse, remember, and respond. AI image generators that turn their drawings into worlds. AI in classrooms, in cars, in homes, in healthcare, in everything else.
Most of these products are being built by commercial actors with primarily commercial goals. Most are not subjected to independent safety review before reaching children. Most rely on the marketing claim that 'this is safe for kids' rather than on demonstrated, audited safety. And most of the safety failures when they happen are discovered by journalists, families, or regulators well after children have already been exposed.
The Mobiloitte Foundation exists, in part, because this gap is not closing on its own.
Independent, audit-led, technically credible work on children's digital safety in the AI age is one of the most under-served and most consequential areas of digital ethics work globally. The Foundation publishes findings, evaluates products, advocates for policy, and provides plain-language guidance to families not because no one else is trying, but because the field as a whole is still significantly behind what children need.
This pillar sets out the Foundation's position what the actual risks are, why most current products fail, what good looks like, what regulators are doing, and what families, researchers, and developers can do now. It is a position document, not a vendor pitch. The Foundation does not sell to families and does not endorse commercial products. The findings are published independently and updated as the field evolves.
Why this matters more now than five years ago
Children's digital safety has been a concern for decades. Online predators, inappropriate content, addictive design, and data exploitation are not new. What is new is the rate at which generative AI changes the threat surface and the rate at which children are being exposed to it without adequate safety review.
Four shifts have happened in 24 months.
Generative AI in children's hands at scale
Free or low-cost generative AI tools chatbots, image generators, voice synthesizers, video tools are now in most children's hands, either directly through apps or indirectly through embedded features in tools they already use. The capability of these systems has improved dramatically. The safety controls have improved less.
AI companions for children
A category of product that did not meaningfully exist five years ago AI companions designed to be a child's friend, listener, confidante is now a meaningful market. Some products are well-designed. Many are not. The line between supportive companion and emotionally manipulative companion is often unclear to families purchasing them, and not always clear to the developers building them.
AI-powered toys
Connected toys with embedded AI talking dolls, conversational stuffed animals, interactive characters, learning robots are widely available. Many connect to cloud services, capture audio and video, transmit data to vendor servers, and operate with privacy controls that few parents understand. Independent safety review across the category is rare.
AI in education at scale
AI tutors, AI grading, AI feedback systems, AI study companions are being adopted across schools and home learning environments. The promise is genuine. The risks bias in feedback, age-inappropriate content generation, data handling, over-reliance are also genuine. Most education systems are deploying these tools faster than they are evaluating them.
Children are being exposed to AI systems faster than the field is producing the independent evaluation work needed to understand what those systems are actually doing to them. Closing that gap is what this pillar and the Foundation's broader work on children's digital safety is about.
The categories of risk that matter
Children's digital safety in the AI age covers several distinct risk categories. Conflating them produces confused policy and confused product design. The Foundation tracks five categories explicitly.
1. Content safety
AI systems can generate or surface content inappropriate for the child using them sexual content, violent content, content promoting self-harm or disordered eating, content normalizing dangerous behaviors. Content safety failures happen through deliberate adversarial prompting by older children, through accidental triggering by younger children, and through model behaviors that surface inappropriate content without any prompt at all.
2. Manipulation and exploitation safety
AI systems can be designed sometimes deliberately, sometimes through optimization without explicit intent to maximize engagement at the expense of the child's wellbeing. AI companions that foster excessive emotional dependence. AI tutors that reward extended use over actual learning. AI characters that subtly nudge purchasing decisions. Manipulation safety failures often appear benign in any single interaction and surface only in pattern, which makes them difficult for families to detect.
3. Privacy and data safety
Children's personal data including biometric information, voice recordings, behavioral patterns, conversation content, and inferred psychological profiles is being collected at scale by AI products. Much of this collection is opaque to families and not adequately governed by current data protection frameworks. Long-term retention, secondary use, and the implications for the child as they grow into adulthood are rarely addressed by product design.
4. Developmental appropriateness
AI systems designed without specific consideration of child development can interact with children in ways that are technically safe but developmentally inappropriate encouraging reliance on AI for cognitive tasks that children should be developing themselves, displacing human relationship development with AI relationship simulation, or modeling interactional patterns that don't serve the child's growth.
5. Equity and access
AI products often work better for some children than for others. Speech recognition that struggles with children's voices, particularly accented or non-Western ones. AI tutors trained predominantly on data from majority populations. Image recognition that misidentifies children with darker skin tones. Equity failures in AI for children compound children for whom the AI works less well receive less educational benefit, less developmental support, less of whatever the AI is offering.
Why most current AI products for children fail safety review
When the Foundation audits AI products marketed to children, certain failure patterns recur. Naming them is not an indictment of any one product it is a description of where the field as a whole stands.
● Safety claims based on training data filtering only, with no behavioral evaluation of how the deployed system actually responds to children's inputs
● Adult-grade content moderation applied to child contexts, missing the developmental specificity that child-appropriate moderation requires
● Privacy policies that satisfy COPPA or GDPR-K technically while collecting data the family would not knowingly consent to if they read the actual handling practices
● Age verification handled through self-declaration only, with no meaningful technical or design controls
● Engagement design optimized for the same metrics as adult products session length, return rate, daily active use without specific guardrails for children
● No independent evaluation, with safety claims relying entirely on internal testing by the product team
● Updates to the underlying model that materially change behavior without re-evaluation against the original safety claims
● Failure modes that affect minority children accented voices, less-common languages, darker skin treated as edge cases rather than first-class concerns
● Bolt-on parental controls that families cannot effectively configure or monitor
● Marketing language ('safe for kids', 'family-friendly') that does not correspond to any defined safety standard
These are not theoretical failures. Each appears in the Foundation's audit work with regularity. The combination a market full of products with marketing claims that exceed their actual safety posture is the gap the Foundation's work addresses.
What good AI for children actually looks like
Good AI products for children are not products with AI removed. They are products built with specific design principles that address the risk categories named above. The Foundation tracks the following indicators.
● Behavioral evaluation across adversarial, accidental, and unsolicited paths into inappropriate content with results published or independently auditable
● Content moderation specific to child contexts informed by developmental research, not just adult moderation policies
● Privacy by design minimum necessary data collection, clear retention limits, no secondary use without specific consent, deletion that actually deletes
● Age-appropriate interaction patterns including limits on emotional intensity for AI companions, encouragement of human relationships, and refusal of inappropriate intimacy from the AI side
● Engagement design that prioritizes child wellbeing over usage metrics including built-in breaks, encouragement to put the device down, and refusal to provide content optimized for compulsive return
● Independent third-party safety evaluation with published methodology
● Transparent model behavior clear acknowledgment to the child that they are interacting with AI, age-appropriate explanation of what the AI does and does not know
● Equity testing across diverse children accents, languages, skin tones, abilities, contexts with documented performance
● Meaningful family controls configurable, monitorable, defaulting to safer settings, and accessible to families with varying technical sophistication
● Versioning and re-evaluation when the underlying model changes, with families informed of material safety-relevant changes
None of these is theoretical. Each appears in the better products on the market today. The point of naming them is to make the difference between marketing claim and actual safety posture legible to families, regulators, and other developers.
What regulators are actually doing
Children's digital safety in the AI age is moving from voluntary best-practice space into active regulation across major jurisdictions. The picture is uneven some frameworks are mature, some are nascent, some are in active enforcement, some are still being drafted but the overall trajectory is unmistakable.
United Kingdom Online Safety Act and the AI Security Institute
The UK's Online Safety Act establishes duties on platforms and services to protect children from harmful content, with Ofcom as the enforcing regulator. Codes of practice address content, design, and risk assessment. The Age-Appropriate Design Code (the 'Children's Code') sets specific standards for the design of online services likely to be accessed by children. The AI Security Institute (formerly the AI Safety Institute) is doing technical evaluation work that informs the broader regulatory environment for AI affecting children.
European Union AI Act and child-specific provisions
The EU AI Act prohibits certain AI practices specifically when they target children including subliminal manipulation and exploitation of vulnerabilities and classifies other practices as high-risk. The Digital Services Act, GDPR-K provisions, and the Better Internet for Kids strategy layer additional protections. For products marketed to or accessed by children in the EU, the compliance surface is substantial and tightening.
United States KOSA, COPPA 2.0, and state laws
The Kids Online Safety Act (KOSA) federal effort has been in active legislative consideration. COPPA 2.0 updates the framework that has governed children's online privacy since 1998. State laws California's Age-Appropriate Design Code (AB-2273), state attorney general actions, and emerging state AI laws are creating an active US regulatory environment for AI affecting children, even before comprehensive federal AI regulation arrives.
India and other emerging frameworks
India's DPDP Act includes specific provisions on processing of children's personal data, with explicit consent requirements and restrictions on profiling. India is also developing broader child online safety frameworks. Other major jurisdictions Australia, Canada, Singapore, the UAE, South Africa each have child-specific provisions emerging through their respective data protection and digital safety frameworks.
For developers, the practical implication building to the strictest applicable framework and applying that standard everywhere is operationally simpler than maintaining country-by-country variants. The strictest applicable frameworks for children are converging on the same broad principles, even where specific provisions differ.
The Foundation's annual AI Toys and Apps Safety Index
Once each year, the Foundation publishes an independent audit of leading AI-powered children's products toys, apps, learning tools, AI companions across the five risk categories named above. The Index covers products that families are actually buying, not only the products developers want evaluated.
The methodology is published. Findings are public. Products are scored on observed behavior, not on marketing claims or self-reported compliance. Where products perform well, they are credited. Where products fail, the failures are documented. The Index is updated when models or products change materially.
The Index serves three audiences. Families by providing accessible scoring that helps them make informed decisions. Regulators by providing technical findings that inform policy and enforcement. Developers by providing a reference standard that good-faith developers can build toward.
The Index does not endorse products. The Foundation does not accept payment from developers in exchange for evaluation. Methodology and findings are independent of the Foundation's commercial parent organization. Where the Foundation's commercial parent has any relationship with an evaluated developer, the relationship is disclosed and the evaluation is conducted under specific independence protocols.
What families can do now
Children's digital safety in the AI age is too important to wait for perfect regulation or perfect products. Families can take specific steps now.
● Ask developers directly for evidence of independent safety evaluation, not marketing claims. The absence of evidence is itself information
● Read privacy policies for AI products children use. Where the language is opaque, that is itself information well-designed products explain their data handling clearly
● Configure parental controls intentionally. Defaults are designed for the average user, not for your specific child
● Talk with children about what AI is and what it is not including that AI does not know them, does not care about them in a human sense, and does not have their interests at heart in the way a parent does
● Notice when use of an AI product is changing the child's behavior, mood, or relationships. Increased reliance, withdrawal from human connection, distress when the product is unavailable, secrecy around use these are signals worth attending to
● Use the Foundation's annual Safety Index and similar independent resources when making purchasing decisions
● Where products fall short, complain to the developer, to relevant regulators, and publicly. Developer behavior responds to family voice when family voice is organized and persistent
What developers can do now
Good-faith developers building AI products for or accessible by children can take specific steps. The Foundation's audit work consistently distinguishes developers who treat safety as a design constraint from developers who treat safety as a post-deployment patching activity.
● Build safety evaluation in from the design phase, not the launch phase. By the time a product is in pre-launch testing, fundamental design choices that affect safety are already locked
● Engage independent third-party evaluation. Internal testing alone does not produce safety claims that survive serious scrutiny
● Apply child-specific moderation, not adult-default moderation. The line between safe and unsafe for a 7-year-old is different from the line for a 17-year-old, both of which differ from the line for adults
● Design for the actual user, not the user the marketing imagines. Children are not always the user the developer expects siblings, friends, classroom contexts, and home environments all shape actual use
● Test with diverse children across ages, languages, accents, skin tones, abilities, socioeconomic contexts. Children whose use is invisible at the design stage are children the product serves badly
● Be honest about what the AI does and does not do. Marketing claims that exceed actual capability create trust failures the field as a whole pays for
● Re-evaluate when the underlying model changes. Generative AI products in particular are subject to material behavioral changes through model updates that may invalidate earlier safety claims
● Engage with regulatory frameworks proactively. The frameworks emerging are not all perfect, but they are not going away engagement now produces better outcomes than reaction later
What this work is and is not
The Foundation's work on children's digital safety in the AI age is technical, evidence-led, and oriented toward outcomes that improve life for actual children.
It is not anti-AI advocacy. AI is in children's lives and will be more so over the next decade. The question is whether the AI children encounter is safe, fair, developmentally appropriate, and privacy-respecting not whether AI should exist.
It is not commercial product certification. The Foundation does not sell certifications, does not accept payment for evaluation, and does not provide endorsements that developers can use in marketing. The Index and the Foundation's other work are reference resources, not commercial endorsements.
It is not a substitute for parental judgment. Families know their specific child, their specific context, and their specific values better than any external audit can. The Foundation provides information that supports parental decision-making, not directives that replace it.
It is not a critique of any specific developer. Where the Foundation publishes findings on specific products, the findings are factual descriptions of observed behavior against published criteria. Many developers receiving critical findings have responded constructively and improved subsequent versions of their products. The Foundation's interest is in product improvement and field improvement, not in reputation damage.
Why the Foundation is positioned to do this work
Children's digital safety in the AI age sits at the intersection of three disciplines child development, AI/IoT engineering, and policy and ethics. Most organizations operating in this space have depth in one or two of these but not all three.
The Mobiloitte Foundation operates with technical AI and IoT depth informed by the commercial parent organization's engineering experience, with child-development and policy expertise built into the Foundation's own programmes, and with the independence and transparency required for credible audit-led work.
The Foundation does not claim to be the only voice in this space. 5Rights Foundation, the Internet Safety Coalition, Common Sense Media, the AI Now Institute's child-focused work, academic researchers across multiple universities, and others are all doing valuable work. The field benefits from multiple credible voices with overlapping but distinct approaches. The Foundation's specific contribution technical audit-led work on AI products specifically, with published findings and family-accessible guidance is one part of a broader ecosystem.
How to engage with the Foundation's work
The Foundation's children's digital safety work is open to engagement from multiple audiences.
Researchers can collaborate on audit methodology, jointly conduct evaluations, and contribute to the published findings. The Foundation publishes methodology openly and welcomes peer review and refinement.
Families can use the Foundation's published Safety Index, the AI Toy Safety Checker, and other family-facing resources when making decisions. Families with specific experiences or concerns can contact the Foundation to share observations that inform ongoing audit work.
Developers building AI products for children can engage with the Foundation's evaluation criteria, request information about audit methodology, and where willing submit products for review. Constructive engagement is welcomed; the Foundation distinguishes good-faith developers from defensive ones in how it documents findings.
Regulators and policymakers can engage with the Foundation's research findings, policy briefs, and submissions to consultations. The Foundation participates in policy consultations across multiple jurisdictions and contributes technical findings where they inform regulatory design.
Funders and partners foundations, grant-makers, individual donors, and aligned organizations can support the Foundation's children's safety work through funding, partnership, or in-kind contribution. The Foundation's governance, transparency, and impact measurement frameworks are documented and available.
Frequently asked questions
What does the Foundation mean by 'AI' in this context?
Broadly any system using machine learning, generative models, large language models, or other AI techniques to interact with or about children. This includes products marketed as 'AI' (chatbots, AI companions, AI tutors), products embedding AI without marketing it as such (voice assistants, recommendation systems, content moderation), and products where AI is part of the underlying infrastructure even when not visible to the child or family (recommendation feeds, search, content filtering). All of these affect children, regardless of how prominently AI features in the marketing.
Why focus on children specifically?
Three reasons. Children are developmentally distinct their cognitive, emotional, and social capacities are not the same as adults, and AI interactions that are appropriate for adults may not be appropriate for children. Children have less capacity to evaluate the AI systems they encounter, which makes the responsibility of developers and oversight bodies correspondingly higher. And children's exposure now will shape the adults they become the design of the AI they encounter today is, in some sense, the design of their future relationship with technology.
Is the Foundation against AI for children?
No. AI has the potential to expand educational access, support children with specific learning needs, provide age-appropriate stimulation and engagement, and serve children who lack access to traditional educational and developmental resources. The Foundation's position is that this potential is real and worth pursuing and that pursuing it well requires honest evaluation of what current products actually do, what risks they create, and how those risks can be addressed. Critique of specific products or practices is in service of better products and practices, not opposition to AI as such.
How is the AI Toys and Apps Safety Index funded?
Through the Foundation's general operating funding, which includes grants, donations, and the commercial parent organization's contributions to the Foundation. The Index does not accept payment from developers in exchange for evaluation. The Foundation's funding sources and governance are disclosed in its annual reporting.
What happens when the Foundation finds a serious safety issue with a product?
The Foundation follows a documented responsible disclosure process. Where a serious safety issue is identified, the developer is notified and given a reasonable window to respond and remediate before public disclosure. In cases of severe or imminent risk to children, the Foundation may engage regulators or other appropriate authorities before public disclosure. Findings are eventually published with documentation of the issue, the developer response, and the Foundation's assessment of remediation.
How does the Foundation handle conflicts of interest with the commercial parent organization?
The Foundation operates with documented independence from the commercial parent organization in its research and audit work. Where the commercial parent has any relationship with an evaluated developer or product, the relationship is disclosed. Evaluation methodology is established independently of commercial considerations. The Foundation's governance structure includes independent voices specifically to ensure that audit work is not influenced by commercial interests.
How can my organization support this work?
Several pathways. Direct funding to the Foundation. Research partnership on specific audit work or methodology development. Engagement of the Foundation in policy consultations within your jurisdiction or area of work. Sharing the Foundation's published resources with audiences who can benefit from them families, schools, regulators, other researchers. Specific partnership inquiries can be directed to the Foundation through standard contact channels.
