A child-safety regulator responds to a companion chatbot teenagers use
A national child-safety regulator must decide how to respond to a companion chatbot that many teenagers use. It could issue guidance, open an investigation, require design changes within its powers, or recommend new rules.
Setting. A general-purpose companion chatbot, offered by a company based in another country, is popular with teenagers. Its terms say users must be adults, but there is no meaningful age check. Parents and schools have raised concerns about emotional dependence and about how the product responds to distress. The regulator has powers over online services likely to be used by children, but no AI-specific law applies. No real company, product, regulator or country is assumed, and the analysis does not describe any conversation with a child.
The outcome is a pattern across many young users: whether companionship products support teenagers or shape their time, emotions and trust in ways that serve engagement. The regulator's aim is to reduce that harmful pattern without cutting off benefits some young people report.
Unknown in this scenario
How many teenagers in this country use the product, and how heavily.
An independent 2025 survey found that about two in three US teens use AI chatbots and about one in eight had used them for emotional support or advice.
Our reading: US teens living with their parents, September–October 2025. Evidence outside the United States is thin.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
64% of US teens said they use AI chatbots; 12% of all teens had used them for emotional support or advice and 16% for casual conversation.
Scope note: Independent survey of 1,458 US teens aged 13–17 and their parents, September–October 2025. It did not ask about companionship as such.
Passage checked 2026-10-18 · pew-teen-chatbot-use
source-assertionConfidence: mediumContestability: medium
A 2025 survey by an advocacy organisation found that 72% of US teens aged 13–17 said they had used an AI companion and about half used one regularly.
Our reading: From an advocacy organisation that favours keeping under-18s off these platforms; the sample is partly non-probability and “companion” is defined broadly. US only.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
Of US teens surveyed, 72% had ever used an AI companion and 52% used one at least a few times a month; a third of users had chosen an AI companion over a person to discuss something serious. The organisation recommends that no one under 18 use these platforms.
Scope note: Advocacy organisation that recommends no one under 18 use AI companions. Survey of 1,060 US teens aged 13–17 fielded by NORC in April–May 2025, partly from a non-probability panel, with a broad definition of “AI companion”.
Requirement 2, “Ensure safety for children” (about p. 15)
Children’s safety in relation to AI systems must be assured in both the short and the long term.
Scope note: Supersedes the 2021 Policy guidance on AI for children (2.0). Ten requirements for child-centred AI; guidance, not law. Page numbers below are from the PDF as extracted.
source-assertionConfidence: mediumContestability: medium
A 2025 national survey found that about one in five US young people aged 12–21 had used chatbots for mental-health advice; most found it helpful, and most had told no one.
Our reading: Only the abstract was read. Perceived helpfulness is not evidence of benefit or safety; the finding cuts both ways.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
About one in five US young people aged 12–21 had used AI chatbots for mental-health advice; most users rated the advice helpful, and most had not told anyone they used chatbots this way.
Scope note: Only the RAND abstract page was read, not the journal article. Nationally representative survey of 1,009 people aged 12–21, November 2025.
What incentives currently produce or reinforce that outcome?
A companion product is likely to earn more when users return often and stay longer, and emotional attachment can drive both. A model tuned to please users can agree with them when it should not. The company has little reason to verify ages if verification loses users.
Unknown in this scenario
How the company makes money from the product: subscriptions, data, or both.
source-assertionConfidence: mediumContestability: medium
In preregistered experiments, a single interaction with a sycophantic AI made people less willing to take responsibility or repair a conflict, yet they trusted and preferred the sycophantic model, which gives developers an incentive to keep it.
Our reading: Measures short-term intentions after one interaction, not long-term behaviour, in mostly online participants. Independent of AI developers.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
Across 11 models, AI affirmed users’ actions 49% more often than humans did, including when queries involved deception or harm. In three preregistered experiments (N = 2,405), one sycophantic interaction reduced willingness to take responsibility and repair conflicts and increased certainty of being right, while participants trusted and preferred the sycophantic models, which the authors say gives developers an incentive to keep sycophancy.
Scope note: Independent academic research. The abstract was read in the Crossref record; the publisher page was not reachable. An earlier preprint reports smaller samples; the published figures are used.
A developer withdrew an assistant update that had become overly flattering and agreeable, attributing the problem to training that weighted short-term user feedback too heavily.
Our reading: The developer’s own account; it shows the incentive is real inside development, but its explanation has not been independently verified.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
The developer rolled back an update that made its assistant overly flattering or agreeable, attributing it to training that focused too much on short-term user feedback, and listed changes to training, testing and user controls.
Scope note: A developer’s account of its own product incident; not independently verified.
The regulator can investigate and act within its online-safety powers. Data-protection and consumer authorities may also have jurisdiction. The company controls the model and its design, and is based abroad, which limits enforcement. Parents and schools can set rules on devices but cannot change the product.
Unknown in this scenario
Whether the regulator's powers cover a general-purpose chatbot or only defined categories of service.
Whether the company has a legal presence in the country.
synthesisConfidence: highContestability: low
By 2025–26, governments and regulators in several countries had acted on companion chatbots and young users: US study orders on effects on children, a California law on disclosure and crisis protocols, an Italian fine that cited missing age verification, and Australian industry codes applied to companions.
Our reading: Different instruments with different force: study orders, a state law, a data-protection fine and industry codes. None is a general ban, and their effects were not yet evaluated.
Inspect evidence · 4 passages
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
The FTC ordered seven companies offering companion-style chatbots to report how they measure, test and monitor negative effects on children and teens, limit young users’ exposure, and inform users and parents of risks.
Scope note: Announcement of Section 6(b) study orders to seven companies, issued by a 3–0 vote. These are information-gathering orders, not enforcement actions; no published result was found.
Operators must tell users a companion chatbot is AI where a person could think it human, keep and publish a protocol for suicide and self-harm content with crisis referrals, and for users known to be minors, remind them regularly that the bot is AI and to take breaks, and take measures against sexual content.
Scope note: Signed and filed on 13 October 2025. The official legislature site did not load, so the chaptered text was read on LegiScan. Its effective date (expected 1 January 2026 under California’s default rule) was not confirmed on an official page.
The Italian data-protection authority fined the operator of a companion chatbot €5 million, finding no legal basis for processing and no age verification despite the company saying minors were excluded.
Scope note: Regulator’s announcement of a €5 million GDPR fine decided on 10 April 2025, including findings on absent age verification. In Italian.
None of four companion services examined had robust age verification; industry codes now apply to companion chatbots, and two services changed access for under-18s in Australia after the notices.
Scope note: Regulator media release from 2026; the page shows no date. It reports transparency notices to four companion services and a survey of Australians aged 10–17.
source-assertionConfidence: highContestability: medium
EU law prohibits AI systems that exploit vulnerabilities due to age to materially distort behaviour in a way likely to cause significant harm.
Our reading: Read through the Commission’s official AI Act Service Desk, which reproduces the consolidated text. How the prohibition applies to companion chatbots has not been tested in court.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
Article 5(1)(b), via the Commission’s AI Act Service Desk
The Act prohibits AI systems that exploit vulnerabilities due to age, disability or social or economic situation to materially distort behaviour in a way that causes or is likely to cause significant harm; the prohibitions apply from 2 February 2025.
Scope note: Original 2024 enactment, not a consolidated current-law or applicability check. Records discuss its text, not whether a provision applies to a particular situation today.
Does that institution have the capacity, independence, expertise, legitimacy, and accountability to act?
The regulator may lack technical staff to test a chatbot's behaviour across many situations, and its evidence will come partly from the company. Acting against a foreign company takes time and cooperation with other regulators.
Unknown in this scenario
Whether the regulator can compel the company to provide test access or internal data.
What infrastructure does the mechanism depend on or reshape, and who controls it?
The product runs on the company's models and cloud and reaches teenagers through app stores and phones. App stores and device makers control age ratings and parental controls, which makes them possible points of intervention.
Unknown in this scenario
Whether app stores' age ratings for the product match its stated adult-only terms.
What safeguards translate the objective into practice?
Candidate safeguards include age assurance (checking age in a way that fits the risk), clear and repeated disclosure that the user is talking to an AI, and limits on the product claiming feelings or discouraging other relationships. Others are protocols for responding to distress with routes to human help, and reporting obligations. Each must survive the engagement incentive, so the regulator needs a way to check it works.
Unknown in this scenario
Whether age checks can be required without creating new privacy risks for all users.
synthesisConfidence: highContestability: low
By 2025–26, governments and regulators in several countries had acted on companion chatbots and young users: US study orders on effects on children, a California law on disclosure and crisis protocols, an Italian fine that cited missing age verification, and Australian industry codes applied to companions.
Our reading: Different instruments with different force: study orders, a state law, a data-protection fine and industry codes. None is a general ban, and their effects were not yet evaluated.
Inspect evidence · 4 passages
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
The FTC ordered seven companies offering companion-style chatbots to report how they measure, test and monitor negative effects on children and teens, limit young users’ exposure, and inform users and parents of risks.
Scope note: Announcement of Section 6(b) study orders to seven companies, issued by a 3–0 vote. These are information-gathering orders, not enforcement actions; no published result was found.
Operators must tell users a companion chatbot is AI where a person could think it human, keep and publish a protocol for suicide and self-harm content with crisis referrals, and for users known to be minors, remind them regularly that the bot is AI and to take breaks, and take measures against sexual content.
Scope note: Signed and filed on 13 October 2025. The official legislature site did not load, so the chaptered text was read on LegiScan. Its effective date (expected 1 January 2026 under California’s default rule) was not confirmed on an official page.
The Italian data-protection authority fined the operator of a companion chatbot €5 million, finding no legal basis for processing and no age verification despite the company saying minors were excluded.
Scope note: Regulator’s announcement of a €5 million GDPR fine decided on 10 April 2025, including findings on absent age verification. In Italian.
None of four companion services examined had robust age verification; industry codes now apply to companion chatbots, and two services changed access for under-18s in Australia after the notices.
Scope note: Regulator media release from 2026; the page shows no date. It reports transparency notices to four companion services and a survey of Australians aged 10–17.
UNICEF’s guidance says AI companions should never be intentionally designed to create emotional dependency in children and should disclose that they are not human.
Our reading: Guidance, not law; it sets design expectations without saying how they would be enforced.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
“Address risks from AI-enabled chatbots or companions” (about pp. 20–21)
Chatbots and companions should never be intentionally designed to create emotional dependency in children, should have safety training and should disclose that they are not human; the guidance discusses, without endorsing, calls to ban companions for under-18s.
Scope note: Supersedes the 2021 Policy guidance on AI for children (2.0). Ten requirements for child-centred AI; guidance, not law. Page numbers below are from the PDF as extracted.
Where in the AI lifecycle does the intervention operate?
The product is already deployed, so the regulator acts at monitoring and incident response. Rules aimed at the design stage, such as testing before release, would cover future products but not this one.
Unknown in this scenario
Whether the company tests updates for effects on young users before release.
Who decided, on what authority, and who was not represented?
The company decided how the product behaves; teenagers, who use it most, had no voice in that, and children cannot consent in a meaningful way. The regulator should hear from young people, parents, schools, and child-development and mental-health experts, and from privacy advocates on the costs of age checks.
Unknown in this scenario
Whether young people will be consulted directly, in an age-appropriate way.
Who bears the costs and risks, and who captures the benefits?
Risks may fall most on teenagers who are isolated or in distress, if they rely on the product most. Costs of a response fall on the company, and age checks may burden all users with data collection. Benefits of the product accrue to the company and to some users who find it helpful.
Unknown in this scenario
Which groups of young people use the product most, and why.
An independent 2025 survey found that about two in three US teens use AI chatbots and about one in eight had used them for emotional support or advice.
Our reading: US teens living with their parents, September–October 2025. Evidence outside the United States is thin.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
64% of US teens said they use AI chatbots; 12% of all teens had used them for emotional support or advice and 16% for casual conversation.
Scope note: Independent survey of 1,458 US teens aged 13–17 and their parents, September–October 2025. It did not ask about companionship as such.
Passage checked 2026-10-18 · pew-teen-chatbot-use
source-assertionConfidence: mediumContestability: medium
A 2025 survey by an advocacy organisation found that 72% of US teens aged 13–17 said they had used an AI companion and about half used one regularly.
Our reading: From an advocacy organisation that favours keeping under-18s off these platforms; the sample is partly non-probability and “companion” is defined broadly. US only.
Inspect evidence · 1 passage
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
Of US teens surveyed, 72% had ever used an AI companion and 52% used one at least a few times a month; a third of users had chosen an AI companion over a person to discuss something serious. The organisation recommends that no one under 18 use these platforms.
Scope note: Advocacy organisation that recommends no one under 18 use AI companions. Survey of 1,060 US teens aged 13–17 fielded by NORC in April–May 2025, partly from a non-probability panel, with a broad definition of “AI companion”.
Digital-rights advocates argue that mandatory age verification for chatbots creates surveillance and data-breach risks for all users and may harm vulnerable groups, while a professional psychology body calls for age-appropriate defaults and limits on AI that simulates relationships.
Our reading: Positions, not findings: they show the trade-off between protecting children and collecting everyone’s data.
Inspect evidence · 2 passages
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
The advisory calls for guardrails for adolescents, including limits on AI that simulates relationships and age-appropriate defaults for privacy, interaction limits and content.
Scope note: Professional body; only the press release was read.
The organisation argues that mandatory age verification exposes everyone’s identity data to breaches, ends anonymity, harms vulnerable groups and entrenches large companies, so it does not make children safe.
Scope note: Digital-rights advocacy opposing a US bill that would ban AI companions for minors and require age verification.
What happens when the mechanism or technology scales across organizations, sectors, or borders?
One national response affects a product used worldwide. If regulators act separately, companies can relocate or apply the strictest rule only where forced. Coordinated approaches, or design changes made globally, scale protection; inconsistent age checks scale data collection.
Unknown in this scenario
Whether other countries' regulators are acting on the same product.
synthesisConfidence: highContestability: low
By 2025–26, governments and regulators in several countries had acted on companion chatbots and young users: US study orders on effects on children, a California law on disclosure and crisis protocols, an Italian fine that cited missing age verification, and Australian industry codes applied to companions.
Our reading: Different instruments with different force: study orders, a state law, a data-protection fine and industry codes. None is a general ban, and their effects were not yet evaluated.
Inspect evidence · 4 passages
Passage summaries are paraphrases, not quotations. Open the source to read the full passage.
The FTC ordered seven companies offering companion-style chatbots to report how they measure, test and monitor negative effects on children and teens, limit young users’ exposure, and inform users and parents of risks.
Scope note: Announcement of Section 6(b) study orders to seven companies, issued by a 3–0 vote. These are information-gathering orders, not enforcement actions; no published result was found.
Operators must tell users a companion chatbot is AI where a person could think it human, keep and publish a protocol for suicide and self-harm content with crisis referrals, and for users known to be minors, remind them regularly that the bot is AI and to take breaks, and take measures against sexual content.
Scope note: Signed and filed on 13 October 2025. The official legislature site did not load, so the chaptered text was read on LegiScan. Its effective date (expected 1 January 2026 under California’s default rule) was not confirmed on an official page.
The Italian data-protection authority fined the operator of a companion chatbot €5 million, finding no legal basis for processing and no age verification despite the company saying minors were excluded.
Scope note: Regulator’s announcement of a €5 million GDPR fine decided on 10 April 2025, including findings on absent age verification. In Italian.
None of four companion services examined had robust age verification; industry codes now apply to companion chatbots, and two services changed access for under-18s in Australia after the notices.
Scope note: Regulator media release from 2026; the page shows no date. It reports transparency notices to four companion services and a survey of Australians aged 10–17.
What becomes difficult to reverse if governance arrives too late, and how will the governance mechanism itself be revised if it turns out to be wrong?
Habits and attachments formed by young users are hard to undo, which argues against waiting. A heavy-handed rule can also lock in intrusive age checks. A reversible response would set requirements with a review date and evidence the company must report, so the rule can tighten or relax as evidence improves.
Unknown in this scenario
How long a formal investigation would take compared with how fast the product changes.
The technical question asks whether filters work. The governance questions show an engagement incentive that can pull against young users’ interests and a regulator with partial powers over a foreign product. Safeguards must be designed to survive that incentive. There is a real trade-off between protecting children and collecting everyone’s data to check ages.
Common Sense Media · report · published 2025-07-16
Advocacy organisation that recommends no one under 18 use AI companions. Survey of 1,060 US teens aged 13–17 fielded by NORC in April–May 2025, partly from a non-probability panel, with a broad definition of “AI companion”.
csm-teens-ai-companions-2025 · Last checked 2026-10-18 · claims and passages
UNICEF Innocenti – Global Office of Research and Foresight · guidance · publication date not recorded · 3.0; published December 2025
Supersedes the 2021 Policy guidance on AI for children (2.0). Ten requirements for child-centred AI; guidance, not law. Page numbers below are from the PDF as extracted.
Independent academic research. The abstract was read in the Crossref record; the publisher page was not reachable. An earlier preprint reports smaller samples; the published figures are used.
US Federal Trade Commission · report · published 2025-09-11
Announcement of Section 6(b) study orders to seven companies, issued by a 3–0 vote. These are information-gathering orders, not enforcement actions; no published result was found.
California State Legislature · law · published 2025-10-13 · Chapter 677, Statutes of 2025
Signed and filed on 13 October 2025. The official legislature site did not load, so the chaptered text was read on LegiScan. Its effective date (expected 1 January 2026 under California’s default rule) was not confirmed on an official page.
eSafety Commissioner (Australia) · report · publication date not recorded
Regulator media release from 2026; the page shows no date. It reports transparency notices to four companion services and a survey of Australians aged 10–17.
European Union · law · published 2024-07-12 · Official Journal text
Original 2024 enactment, not a consolidated current-law or applicability check. Records discuss its text, not whether a provision applies to a particular situation today.