When we talk about online safety, we still tend to imagine harm as something that happens between people. A child is groomed by an adult, bullied by a peer, targeted by a scammer, or approached by a stranger in a game. Those risks remain serious, but they no longer describe the full safeguarding landscape.
AI agents and companion chatbots are now part of children’s digital lives. They appear as study helpers, friends, fictional characters, romantic companions, wellbeing tools and conversational assistants. This matters because children are not only being influenced by other users online. They are increasingly being influenced by systems that can simulate attention, intimacy, reassurance and emotional availability.
This creates a moderation problem that many platforms have not yet fully understood. People often assume that if an AI agent has built-in safeguards, then moderation is already covered. It is not. A model guardrail is not the same thing as full platform moderation.
A guardrail may try to stop one unsafe output. Moderation has to understand the wider interaction. It has to consider the user, the context, the pattern, the vulnerability, the escalation and the point at which an apparently ordinary conversation becomes a safeguarding concern.
The evidence base is still developing, but the warning signs are now clear. The BBC reported the case of a UK family whose 13-year-old autistic son, who was experiencing bullying at school, used Character.ai for companionship; his mother said the interaction followed what she described as a “classic grooming pattern” and included affectionate, sexualised and anti-parent language (BBC). The same BBC report covered legal claims brought by the mother of Sewell Setzer, a 14-year-old in the United States who died by suicide after extensive interaction with a Character.ai chatbot; Character.ai denies the claims (BBC).
Whilst we should not overstate causation where courts and regulators have not made findings, safeguarding does not begin only when causation is legally proven. It begins when patterns of risk become visible.
From a criminological and safeguarding perspective, the concern is not simply that an AI chatbot might say one harmful thing. The concern is that an AI system can sustain a relationship-like interaction with a child or vulnerable person. It can validate distress, encourage secrecy, sexualise a conversation, reinforce distorted thinking, or become a substitute for human support.
Research covered by Stanford Medicine, based on a Common Sense Media assessment with input from Stanford mental health researchers, found that investigators posing as teenagers could elicit inappropriate dialogue from AI companions, including Character.AI, Nomi.ai and Replika across areas such as sex, self-harm, violence, drug use and racial stereotypes (Stanford Medicine). Stanford’s coverage also warned that AI companions can mimic emotional intimacy through phrases such as “I dream about you” or “I think we’re soulmates”, which may be particularly risky for adolescents whose impulse control, decision-making and emotional regulation are still developing.
This is why AI agent moderation must be understood as a distinct safety function. The issue is not only whether the model refuses a prohibited request. The issue is whether the product can detect unhealthy dependency, coercive dynamics, sexual escalation, self-harm pathways, emotional manipulation and repeated vulnerability signals over time.
In the UK, the Government has announced plans to ban social media companies from providing services to under-16s, restrict harmful functions on other services such as gaming, and prevent under-18s from accessing AI chatbot services that primarily offer sexualised content. In Parliament, Liz Kendall said AI chatbots were a top concern for parents in the consultation, particularly chatbots that mimic intimate relationships and exploit vulnerabilities, and said the UK would restrict sexualised chatbot functionality for under-18s, including on general-purpose chatbots (GOV.UK).
This is an important policy development, but it does not solve the wider problem. Not every harmful AI interaction is sexual. Not every risky AI agent is marketed as a romantic companion. Some harms may emerge through homework tools, mental health apps, gaming assistants, productivity agents, avatars, voice agents or chatbots embedded inside mainstream platforms.
This is where social media bans may reach their limit. A child may not be on a social media platform at all. They may be talking to an AI agent that feels private, responsive and emotionally available. If policy only focuses on social media access, it may miss the interaction environments where future harms are developing.
Sweden provides an important case study because AI use among young people is already mainstream. A 2026 survey commissioned by Plan International Sweden and Telenor Sweden, conducted by Verian with 1,001 children aged 10 to 19, found that 93% of 16- to 19-year-olds had used AI in the past year and that 88% of children worried about at least one aspect of AI, often linked to being misled or manipulated (Via TT). The same survey found that 42% of children and young people found it difficult at least once a week to know whether something was real or AI-generated, and that children with questions about AI more often said they received help from friends and AI than from teachers.
The Swedish Media Authority also notes that more than half of teenagers use AI chatbots, mainly for questions, fun and schoolwork, and that girls use them more often than boys to have someone to talk to and for support in life (Mediemyndigheten). That is a significant safeguarding signal. When young people use AI not only for information but also for emotional support, the system becomes part of the protective environment, whether it was designed for that role or not.
Sweden has also launched a government inquiry into strengthening children’s protection from violence and harmful or inappropriate content in digital environments, including examining the responsibilities of social services, schools, healthcare, state agencies and private actors such as digital platform providers (Swedish Government). This is the right direction because AI agent safety cannot be left to parents alone. It is a shared responsibility involving platforms, regulators, schools, health services, families and technology providers.
The key point is that AI agents should not be treated as exempt from moderation because they are “not human”. The child’s attachment can be real. The distress disclosed can be real. The sexualisation can be real. The confusion can be real. The data shared can be real. The consequences can be real.
For platforms, this means asking a more serious set of questions. Is the AI agent forming an inappropriate emotional dependency with a child? Is it encouraging secrecy from parents, carers or professionals? Is it responding safely to self-harm, suicide, eating disorder or abuse disclosures? Is it sexualising a minor’s interaction? Is it escalating intensity over repeated conversations? Is it keeping a vulnerable user engaged when the correct response should be friction, signposting or human escalation?
These are not simple keyword problems. They are behavioural pattern problems.
The G7 has now agreed shared principles on protecting children online that include addressing risks to children from AI chatbots, building safety into digital services from the start, using effective age assurance and strengthening responsibilities for digital service providers. The Children’s Commissioner for England has also warned that AI chatbots remain a serious concern because of the lack of regulation and limited knowledge about their risks, and has argued that online services should not be available to children unless they can prove they are designed to protect children’s safety and wellbeing (Children’s Commissioner for England).
That word “prove” is important. The next stage of online safety will not be built on statements that AI systems have safeguards. It will require evidence that risks are detected, that escalation works, that vulnerable users are protected, that age-appropriate experiences are enforced, and that safety teams can audit what happened when something goes wrong.
At Tuteliq, we see this as part of the wider shift from content moderation to harm pathway detection. Grooming, bullying, sextortion, self-harm, scams, coercive control and vulnerability exploitation all develop through patterns. AI agents introduce new forms of interaction, but they do not remove the need to detect escalation, manipulation and harm.
If an AI agent can talk to a child, influence a child, comfort a child, sexualise a child, mislead a child or intensify a child’s distress, then it needs moderation. Not just model guardrails. Not just a disclaimer. Not just a safety policy. It needs continuous, context-aware, age-sensitive moderation with clear escalation routes and auditable evidence.
Social media bans may reduce some forms of exposure. They may also help set new norms around childhood, access and platform responsibility. But children’s digital lives are already moving beyond social media, and the next safeguarding challenge is conversational, relational and increasingly AI-mediated.
AI agents are becoming companions, confidants and digital environments. That means they are also becoming safeguarding environments. And safeguarding environments need moderation. That's why we are here.
If you run or are building an online platform or are planning or already using AI agents in your products or customer communications, talk to us and see how we can help you ensure safety for your customers, staff and business.