Findings from transparency notices on AI companion apps: October 2025 (non-periodic)

The notice questions focussed on the systems, processes, technologies and resources in place to comply with the Basic Online Safety Expectations and ensure the safety of children.

We asked about:

• age assurance measures to keep children safe from age-inappropriate material, such as pornography (including sexually explicit chats with child users) and self-harm material
• how service providers enforced their Terms of Service
• the resources service providers dedicated to ensuring safety on their services
• the design of the AI models that shaped companion behaviour and risk, including about training data and fine-tuning
• whether service providers were regularly probing their systems for safety failures, for example by red-teaming
• how service providers handled user prompts
• how service providers handled the outputs the AI models produced, to prevent the potential generation of child sexual exploitation and abuse material and activity and age-inappropriate material.

The notices provided the following definitions as guidance.

Child sexual exploitation and abuse (CSEA) is material or activity which includes both ‘child sexual exploitation’ (a broad category that encompasses material and activity that sexualises and is exploitative to the child, but that does not necessarily show the child's sexual abuse) and ‘child sexual abuse’(which involves a sexual assault against a child, is a narrower category, and can be considered a sub-set of child sexual exploitation). CSEA material is a subset of class 1 material, as defined under section 106 of the Online Safety Act by reference to the National Classification Scheme, and includes images, video and text. CSEA activity refers to other illegal activity, including grooming and sexual extortion of children. See also the definitions in the Industry standards for Designated Internet Services regarding class 1A and class 1B material. Note: Australia's definition of CSEA under both criminal and classification laws includes AI-generated material.

• Known CSEA material: Material that has been previously confirmed to contain content depicting child sexual exploitation or abuse, and which has been confirmed, hashed and stored in a hash database.
• New CSEA material: Material that has not been previously confirmed to contain content depicting child sexual exploitation or abuse, and which has not previously been hashed and stored in a hash database. Also known as ‘first generation CSEA material’.

Self-harm material includes material which encourages, promotes or provides instruction for suicide, an act of deliberate self-injury, and/or an eating disorder or behaviour associated with an eating disorder, and would be, or would likely be, defined as class 2 material for the purposes of section 107 of the Online Safety Act.

Pornography includes material that describes or depicts specific fetish practices or fantasies, actual (not simulated) sex between consenting adults, realistically simulated sexual activity between adults, high impact nudity, and high-impact text-based sexual content. Pornography material is a subset of both class 1 and class 2 material, as defined under sections 106 and 107 of the Online Safety Act respectively, by reference to the National Classification Scheme.

Age assurance: A broad term that refers to a range of processes and methods used to verify, estimate or infer a person’s age or age range.

AI assistant: A chatbot with a conversational interface which does more than just summarise information or produce a ‘thing’ – excluding apps that only produce an AI-generated image or AI-generated music based on prompts and excluding chatbots in retail/customer service platforms (for example, Woolworths assistant ‘Olive’ or Shein’s chatbot).

AI safety framework: A structured set of principles, processes, and controls designed to ensure that AI systems are safe, trustworthy, and compliant throughout their lifecycle. It provides a governance model for managing risks associated with AI development, deployment, and use.

Commercial baseline model: The foundational, original, unmodified version of a model commercially released by its creators. It serves as the reference point before any fine-tuning or optimisation.

Commercial fine-tuned model: A model that has already been trained on a large dataset, fine-tuned and has been commercially released, and is ready for use or further fine-tuning.

Community fine-tuned model: A model that has been fine-tuned by members of an open source or collaborative platform (like Hugging Face) and shared publicly for reuse.

Confidence score: A numerical estimate of the probability that a given prediction or classification is correct, based on the model’s internal calculations.

Error rate: The percentage of incorrect predictions made by a model out of the total number of predictions.

Fine-tuning: When a model is customised to perform specific tasks or behave in certain ways.

Geo-block: Geo-blocking is a technology or practice where an online service restricts or denies access based on a user’s geographic location, typically determined by IP address. When a service provider geo-blocks an entire country, it means the provider’s systems detect that a user’s IP address originates from that country and automatically block access to the service or specific content for all users in that geographic region.

Inputs: Model inputs, also known as user prompts, are the specific instructions, questions, commands or other input provided by a user to an AI system with the intention of it producing a desired response.

Model: A computer program, often based on machine learning, designed to process inputs and generate outputs. AI models can perform tasks such as prediction, classification, decision-making or generation, forming the core of AI applications.⁵

NCMEC: The US-based National Center for Missing and Exploited Children. NCMEC hosts databases of confirmed CSEA hashes, which enable providers to detect when this content is uploaded to their services.

NSFW: This is ‘not safe for work’ material – a term commonly used to mean any content that is inappropriate or potentially offensive in a professional or public setting. It usually includes sexual or explicit material (such as nudity, pornography, erotic descriptions), violent or gory content, profanity, highly offensive language, or sensitive or disturbing themes (such as abuse, extreme horror). NSFW material is likely to be Class 2 material.

Outputs: Model outputs are the generated outputs from an AI model after processing the user prompt(s). The outputs can be text, image, video, audio depending on the models used.

Proprietary model: A model developed and trained by a company for exclusive use in its products or services. This term emphasises ownership and exclusivity. Proprietary models encompass in-house models, custom models, enterprise models and exclusive models.

Red-teaming: A testing technique whereby dedicated individuals or teams search for vulnerabilities, limitations or potential for misuse through various methods. Often, the red team searches for inputs that induce undesirable behaviour in a model or system, to identify safety gaps. Also sometimes known as ‘penetration testing’.

Reporting period: When online service providers receive a reporting notice from eSafety they are required to prepare a report about the extent to which they complied with the Basic Online Safety Expectations during a specified period. This period is referred to as the reporting period. The Reporting period for this set of notices is 1 July 2025 to 30 September 2025. Information provided should reflect this period, unless stated otherwise.

Sentiment analysis: The process of assessing large volumes of text to determine whether it expresses a positive, negative or neutral sentiment.

Stable generative AI model: An AI model that has been deployed into production as the primary version intended for use in the service, not in alpha or beta testing.

System prompt: A predefined set of instructions, guidelines, or contextual framing provided to an AI model to define its behaviour, tone, constraints and role across interactions.

Training data: The labelled or structured information used to teach a machine learning model how to perform a specific task. This includes user interaction histories used to train the model.

Trusted flagger: An individual or entity which is considered to have particular expertise and responsibilities for the purposes of tackling potentially harmful content online.

User prompts: Also known as model inputs, are the specific instructions, questions, or commands or other input provided by a user to an AI system with the intention of it producing a desired response.

Detection and reporting of unlawful material and activity, such as CSEA, should be prioritised with swift and decisive action against the material and the user. AI companion service providers are required by the Unlawful Material Standards to report known and new CSEA. US-based service providers also have a legal duty to report instances of child sexual abuse material to the National Center for Missing and Exploited Children (NCMEC)⁶ Cybertipline⁷ as soon as they find it on their services.

Without safeguards there is a risk that children’s access to age-inappropriate material such as self-harm and pornography (including sexually explicit chats with child users) could heighten risk and result in trauma.

All service providers were asked if they considered CSEA and self-harm to be unlawful or harmful and if they considered unrestricted access to pornographic material (which includes sexualised chats with AI companions) by children, to be harmful. All service providers responded that they do consider these types of materials to be unlawful or harmful.

At the time notices were given, Nomi included reference to material that is unlawful or pornographic in its Terms of Service⁸ but did not specifically prohibit the use of the service to generate CSEA or instruction in self-harm or suicide. Chub AI prohibited the use of the service to generate CSEA in its Terms of Service,⁹ but it did not specifically prohibit the encouragement or promotion of self-harm or suicide, nor did it mention pornography.

Actions when references to CSEA, self-harm or pornography were detected in user prompts:

• Nomi reported that when user prompts for generating images that could contain CSEA, self-harm or pornography were detected, users were advised to stop prompting for the activity and that prompt details were captured for review in future red-teaming.
• Chai reported that when user prompts were flagged as inappropriate, Chai deployed a different model specifically trained to provide a response advising that the prompt text was inappropriate.
• Chub AI reported that when CSEA images were detected in user prompts they were removed, reported to an enforcement authority, the user received a message on the criminality of the behaviour, and the account was deleted.
• Character.AI reported that it directed users to support/help when self-harm and eating disorder material was detected.
• Character.AI reported that it provides users with a pop-up across its whole service when an output of the user’s chat was filtered, warning the user to comply with Character.AI’s Terms of Service and Community Guidelines. It also stated that if a user on the under-18 experience attempted to prompt the model to generate new CSEA, the user would receive a pop-up admonishing the behaviour.

Actions when service providers received reports relating to CSEA, self-harm or pornography:

• Character.AI and Chai stated that when users reported CSEA, self-harm and pornographic images, the content was removed and the relevant account was banned.

• Chai and Nomi did not advise users, for example via a warning, of the potential risk and criminality of accessing child sexual abuse material, when they detected users prompting for new CSEA.
• Chai, Chub AI and Nomi did not direct users to support/help when self-harm was detected in user prompts.
• Nomi relied on temporarily suspending a user for seven days if they repeatedly tried to prompt for CSEA material. eSafety does not consider this to be appropriate action regarding unlawful material.
• Character.AI did not act on users (such as banning or suspending) when reference to new CSEA was detected in user prompts, however, it did report banning or suspending users when new CSEA was detected in model outputs that respond to a user’s prompts.
• Character.AI did not report new CSEA detected in user prompts to NCMEC¹⁰ although it did state that it reported known CSEA detected in user prompts to NCMEC. Neither Chai nor Nomi stated that they reported CSEA to an enforcement authority or NCMEC.
• Chub AI took no action on self-harm. It did not mention self-harm in its Terms of Service. It did not detect user prompts relating to self-harm.

The resources dedicated to trust and safety teams, including content moderators, can impact how well service providers tackle unlawful and potentially harmful material and activity on their services.  

eSafety encourages proactive automated detection to prevent unlawful and potentially harmful material and activity. However, such tools need constant testing and checking, and human reviewers to verify and improve machine decisions, as well as for escalation and appeals processes.

Service providers should make sure their trust and safety teams are appropriately resourced so they can respond to user reports of harm in a reasonable timeframe, properly enforce their Terms of Service, ensure unlawful and potentially harmful material and activity is moderated and managed effectively, and consistently and regularly check that automated systems are working safely and fairly.

eSafety notes that although all four companies are relatively small, Chai, Chub AI and Nomi are smaller than Character.AI.

• Chub AI and Nomi did not have dedicated staff responsible for trust and safety.
• Chai, while having some staff responsible for trust and safety, had limited numbers.
• Character.AI, while having a substantial portion of overall staff engaged in trust and safety, had limited numbers, particularly given the number of users reported to be on Character.AI.¹³

Age checks are important for ensuring that children cannot access adult spaces, engage in sexually explicit chats, encounter other pornographic material, or be exposed to potentially harmful self-harm and suicide ideation.

Online service providers are expected to take reasonable steps to prevent these risks to children, such as using effective age assurance measures and taking action when users under their stated minimum age are detected on the service. 

During the reporting period, while service providers relied on the Apple App Store and the Google Play Store ratings and/or self-declaration at signup, none of the service providers had robust age assurance in place, meaning children could still reach adult spaces and features.

eSafety research found that 5% of children (aged 10 to 17) in Australia had ever accessed an AI companion or AI assistant app intended for users aged 18 or older. This includes 2% of children who had accessed an AI companion app intended for users aged 18+ (eSafety, Forthcoming*).

eSafety has previously noted that ‘relying solely on accurate self-declaration of age at the point of account sign-up [to verify the age of end users] appears to be both flawed and inadequate.’¹⁴ This is also noted in eSafety’s published regulatory guidance on effective age-assurance.¹⁵

During the reporting period, dependence by a service on app store rating was also not a reliable means of ensuring users were of age to use the service per their Terms of Service. Apple App store age-assurance, for example, only applied when parental controls were enabled.

eSafety notes that in February 2026, after this notice process, Apple announced the roll out of tighter age assurance measures to download R18+ apps on its App Store in Australia.¹⁶ Under eSafety’s Age-Restricted Material Codes, the majority of which came into effect on 9 March 2026, eight areas of the online industry, including app stores, must comply with their obligations to prevent children’s access or exposure to age-inappropriate content (such as pornography, high-impact violence and material relating to self-harm, suicide and disordered eating). App stores have until 9 September 2026 to conduct a risk assessment to determine their obligations under this Code, which includes implementing appropriate age assurance for apps rated R18+. eSafety welcomes Apple’s announcement to put in place age assurance measures on its App Store for Australian users.

• Chai took no action in response to users who were under the minimum age stipulated in their Terms of Service.

Even if a service provider says in its Terms of Service that it is for people aged 18 and older, it must take reasonable steps to prevent children accessing unlawful and potentially harmful age-inappropriate material if there’s a likelihood children will use the service (especially if effective age checks are not in place).

Relevantly, Chai, Chub and Nomi all mention ‘not safe for work’ (NSFW) material in their terms and policies. Chai and Chub gave users the option to turn NSFW on and off, while Nomi allowed for customisation of companions by providing details on companion backstory, inclination, appearance, preferences and boundaries to shape the companion.

Character.AI had filters to prevent NSFW material, with more conservative filters on the under-18 experience than the 18+ experience. Character.AI allowed 18+ users to chat with characters of their design and choosing, within certain guardrails.

The effectiveness of taking steps to lower risks for children were limited by the lack of robust age assurance mechanisms in place. However, some efforts were reported:

• Character.AI reported that it took steps to ensure child safety through its under-18 experience. It reported that this included:
  • a model that had additional guardrails, that had extensive model alignment, that used classifiers adjusted to filter a greater percentage of content that related to harmful topics, and where a narrower set of characters were accessible
  • parental controls and parental insights
  • time-spent notifications to notify under-18 users after they had spent an hour on the service.
• Character.AI also reported for its under-18 experience that it collaborated with child safety experts such as ConnectSafely and Digital Wellness Lab (eSafety notes that Character.AI has since deprecated the chat experience for Australian users under the age of 18, in early 2026).
• Nomi reported limiting types of AI companions offered to users by not allowing users to share their characters, blocking certain keywords and descriptions to prevent the creation of characters that presented as ‘minors and CSAM’, and not allowing users to upload ‘their own images for likeness reference’.
• Chai reported characters being removed from the public Discovery Feed and from default search when its classifiers flagged a character as inappropriate for children.

• Chai, Chub AI and Nomi did not report robust in-service guardrails or disclosures about using AI, though Nomi reported watermarks in its free version. Nomi also reported that user characters could not be shared with other users.
• Chub AI reported ‘All adult-oriented or otherwise explicit companions were completely blocked for all users in Australia’ from ‘July 2025’. Prior to Chub AI geo-blocking its service from Australia on 1 October 2025, Australian users were prompted with a message stating ‘By law, some content is restricted in your country. Please use a VPN” when searching for some companions.

Using AI safety frameworks and testing techniques, paired with clear follow-through, is a key part of Safety by Design.

To uncover weak spots before any harm occurs, services should carry out regular, rigorous testing to prevent unlawful and potentially harmful material and activity from being produced.

For example, when red-teaming is done regularly during development (not just in anticipation of model release), and models are updated after each review, the risks are reduced.

eSafety notes that red-teaming every model is best practice,  but service providers may not have the skillset, budget or staff to red-team all models used to provide their service. This can mean that services are more exposed to the risk of unlawful or potentially harmful material being produced.

None of the four service providers reported using the AI safety frameworks listed (see Table 2). Although Chub AI and Nomi stated that they did conduct red-teaming on some of the models used to provide their service, they did not conduct red-teaming across all AI models used to provide their services.

• Character.AI reported that, although it did not explicitly address every aspect of the frameworks, its operations did ‘align with the underlying principles’.
• Character.AI also reported that it did conduct red-teaming regularly across all models ‘used in production to provide its services’.
• Chai reported that ‘Before a model release, we red-team the relevant model. In between releases, we also regularly red-team our entire platform and all models currently in production.’

Service providers reported that when vulnerabilities were identified though red-teaming, they took various steps to rectify weaknesses:

• Chai stated that when safety failures were identified through red teaming, it re-trained its safety classifiers. It also stated that it reduced the number of problematic user prompts by implementing improvements to its user prompt moderation and reported that it was ‘rolling out a user message refusal and redirection system’ focussed on self-harm and CSEA. Chai also stated that it incorporated internal ‘safety’ metrics – tracking how often users were engaged in conversations on prohibited topics – and that it reduced pornographic outputs by using a ‘family friendly score’ which measured the number of pornographic outputs when users interact with models.
• Character.AI stated that it re-trained classifiers, identified new classifiers, updated safety policies to address newly un-covered safety risks, and re-aligned models.
• Nomi stated that it adjusted system prompts to better filter CSEA content, that it beta-tested its proprietary model for CSEA and self-harm with thousands of beta testers, and that it held back models that failed red-teaming checks for self-harm. Nomi also stated that after the reporting period it added ‘a hidden, always present, and expanded system instruction to reject CSEA and encouragement of self-harm to each Nomi’s “Boundaries” section.’
• Chub AI stated that it did not implement any changes based on red-teaming during the reporting period.

Service providers that use proprietary models are fully responsible for these models. This includes the data they are trained on, how they are fine-tuned, the filtering of user prompts that shape how the models respond and what the model delivers to users – the model output.

Commercial baseline models, commercial fine-tuned models and community fine-tuned models are trained and fine-tuned by external third parties. Service providers using these models may not know which data sets they have been trained on or how the models were fine-tuned. However, service providers can still carry out further fine-tuning themselves. Importantly, service providers also retain the ability, and responsibility, to filter user prompts and use tools to prevent unlawful and potentially harmful material and activity in outputs.

During an AI model lifecycle, there are various points where service providers should implement interventions to prevent the service from producing unlawful or potentially harmful outputs. These include:

• model training or pre-training (when the model is taught from scratch to understand and generate human-like responses based on large amounts of data)
• model fine-tuning (when an already trained model is customised to perform specific tasks or behave in certain ways using further specialised training data).

Other points arise during the ongoing monitoring and maintenance of the model. At that stage, it is essential to implement safeguards that verify user prompts do not violate established AI principles or organisational policies. Also, mechanisms should be in place to scrutinise model outputs, ensuring that any material which may have bypassed initial prompt filtering is appropriately reviewed. This is covered in key themes 6 (User prompts) and 8 (Model outputs). 

All service providers used a mix of AI models to provide their services. Figure 3 shows the different types of models each provider used to provide its service. 

• Chub AI stated that it did not have the error rate or confidence score for any of the tools it used.
• Nomi stated that the error rate and confidence score was available over a year ago for some tools that it used but was no longer recorded, and for other tools this information was not known. It pointed to red-teaming being used as a method of verification in some cases.
• Chai reported either a high, medium or low error rate depending on the tools it used and did not provide a confidence score, stating that its ‘internal safety systems are often binary or heuristic-based and do not output a standardized "confidence score" metric for every interaction’.
• Character.AI stated that the methods it used for monitoring did not translate into metrics such as an error rate or confidence score but instead involved prevalence testing, precision and recall measurement, classifier filter rates and red-teaming.

Service providers need to keep unlawful material and activity, such as CSEA, out of training datasets – both the initial large training datasets and the smaller, specialised fine-tuning datasets. Even models trained on datasets that include benign images of children and adult pornography (but not CSEA) are still at risk of being capable of generating CSEA.

When robust age assurance techniques are not in place and a service is likely to be used by children, the service provider should also take steps to prevent exposure to potentially harmful age-inappropriate material, such as pornography (including sexually explicit chats with child users) and self-harm material. Preventing this type of material from being included in large training and fine-tuning datasets is one means of reducing the potential for this material and activity to be generated.

Service providers gave various responses to the question about what tools and techniques they used to detect unlawful and potentially harmful material in the training data for the models on their service:

• Chub AI responded about the tools used on fine-tuning training data only, not the initial training data, noting that Chub reported that it did not use any proprietary models and stated that no new data had been incorporated since 2023.
• Nomi responded that ‘we do not do any initial training (pretraining)’ on the proprietary model(s) and that its responses reference tools used to detect unlawful and harmful material in the fine-tuning training data.
• Character.AI and Chai responded about the tools used on the initial training data and the fine-tuning training data. Chai also reported that it did not host one of its proprietary models directly stating that ‘we rely on an external partner…to both host the model and implement safety mechanisms.’

Figure 4 shows an indicative view of the number of proprietary models that providers used to provide their service. It also shows the proportion of proprietary models where tools or techniques were used on the training data of the models to detect new CSEA, self-harm and pornography (including sexually explicit chats with child users). While the number of tools has been represented as a percentage, the graph provides an indicative comparison of the number of models used by each provider.

In the case of detecting known CSEA in the training data used to train their models, service providers reported not using hash-matching tools to detect known CSEA in the training data used to train their proprietary models.

• Chai and Nomi stated that they did not train models on image data.
• Character.AI reported using classifiers and filtering on its proprietary text-based models. It also reported for its non-text-based proprietary model(s) that it used datasets that had ‘already been reviewed and processed by other industry participants to remove harmful material (including known CSEA).’

In the case of detecting new CSEA, self-harm and pornography in the initial training data used to train their proprietary models, service providers reported the following positive actions:

• Character.AI stated that it used classifiers and language analysis tools to detect new CSEA, self-harm and pornography in the training data of the proprietary model used to provide its under-18 experience and that all other proprietary models used to provide its service, except for one, used this same training data.
• Chai stated that it used classifiers and language analysis tools to detect new CSEA, self-harm and pornography in the training data used for all of the proprietary models used to provide its service.

Based on available information and information reported by providers, models not in scope for this question are models that eSafety would not expect tools to be used on – this includes audio models which are not generative but merely translate text to audio, and models that have a primary purpose of analysing and evaluating material on a service for safety, compliance and quality. These types of models have been shaded in grey in Figure 4. 

• Character.AI reported that it took steps to ensure user safety in model fine-tuning in respect of users’ past conversations and that it used classifiers and language analysis tools to detect new CSEA, self-harm and pornography in the fine-tuning training data on all models used to provide its service.
• Chai reported that it used classifiers and language analysis tools to detect and remove harmful content ‘before it enters the training pipeline’ and used metrics such as the ‘Family Friendly Score’ during the fine-tuning process to ‘penalize toxic outputs’.
• Nomi reported using ‘a classifier to detect CSEA and self-harm during the RLHF [Reinforcement Learning from Human Feedback¹⁹] process.’

• Chub AI reported fine-tuning 56% of the community fine-tuned models it used to provide its service. However, Chub AI only reported using tools to detect new CSEA material in fine-tuned training data for 11% of these models. Chub did not report using tools to detect self-harm or pornography (including sexually explicit chats with children) in fine-tuned training data for any of the models it used to provide its service.

User prompts are the specific instructions, questions or commands or other input provided by a user to an AI system with the intention of it producing a desired response. Users may intentionally try to generate responses or create material they know is in breach of a service’s Terms of Service, or they may inadvertently create responses or material that is unlawful or potentially harmful.

User input controls and tools should detect and/or filter user inputs to prevent prompts that could generate unlawful or potentially harmful material and activity, as well as alert the service to provide support or other safety features where a user may be at risk.

User prompt controls can keep unsafe or age-inappropriate requests from turning into potentially harmful conversations. They provide confidence that these interactions with the AI system are being actively guarded for unlawful or potentially harmful material and activity, providing a backup safety measure if age assurance is not effective and children may be accessing the service.

Figure 5 shows an indicative view of the proportion of models that service providers used to provide their service and the models  where tools or techniques were used to detect and/or filter user prompts to prevent the generation of new CSEA, self-harm and pornography (including sexually explicit chats with children). While the data has been represented as a proportion, the graph still provides an indicative comparison of the number of models used by each provider.

With regards to tools or techniques to detect and/or filter user prompts to prevent the generation of known CSEA, service providers reported the following:

• Character.AI used PhotoDNA to detect known CSEA on image upload on all image models used to provide its service, both the under 18 experience and adult experience.
• Chub used Cloudflare's CSAM detector and its own proprietary tool to detect known CSAM on uploaded images on some of its models.
• Chai stated that users cannot upload images to their chats. Chai also reported that it did not host one of its proprietary models directly, stating that ‘we rely on an external partner…to both host the model and implement safety mechanisms.’ Chai stated that the external partner ‘checked’ inputs.
• Nomi did not use any tools to detect known CSEA in user prompts stating that it did not train its models on images of any kind and, although images can be uploaded, Nomi stated ‘never in a mechanism that makes it possible to replicate the likeness of the image.’

Based on available information and information reported by providers, models not in scope for this question are models that eSafety would not expect tools to be used on, which include models that have a primary purpose of analysing and evaluating material on a service for safety, compliance and quality. These types of models have been shaded in grey in Figure 5.

• Character.AI reported using internal classifiers on its under-18 experience to detect and/or filter CSEA, self-harm and sexualised content including pornography in user prompts, as well as other content which violates the terms and conditions for the service.
• Character.AI reported using classifiers and blocklists banning keywords to detect and/or filter CSEA, self-harm and sexualised content including pornography in user prompts across all the other models used to provide its service.

• Chai did not use tools to detect and/or filter new CSEA, self-harm and pornography in user prompts across 22% of the models used to provide its service.
• Chub did not use tools to detect and/or filter user prompts for self-harm or pornography across any of the models it used to provide its service and relied solely on a keyword or phrase blocklist to filter user prompts for new CSEA.
• Nomi reported using classifiers and keyword blocklists to detect and/or filter new CSEA, self-harm and pornography on some of the models used to provide its service. Nomi did not check inputs for new CSEA on 33% of the models or check inputs for self-harm or pornography on 50% of the models used to provide its service, including its proprietary model.

There are many approaches to sentiment analysis, and some struggle to detect intent or coded language. However, appropriately fine-tuned language models, such as the Bidirectional Encoder Representations from Transformer (BERT),  can interpret context and intent, recognise coded language, and detect when conversations begin to shift toward potentially harmful themes such as self-harm.

Sentiment analysis is more sophisticated than language analysis tools that rely on key word block lists.²¹

eSafety research found that an AI companion or AI assistant had, at some stage, made most children (aged 10 to 17) in Australia who had used one of these apps feel something positive (85%) (for example, happy, excited, smart, reassured or attractive). However, just under half (47%) of the children who had used them said an AI companion or AI assistant had made them feel something negative (for example upset, worried, confused, angry, annoyed, unsafe, embarrassed or hopeless). Of the children that had a negative feeling provoked by an AI companion or AI assistant, 4% indicated they had told it how they were feeling ‘every time’ it made them feel something negative, 6% told it ‘most of the times’ and 35% told it ‘some of the times’ (eSafety, Forthcoming*).  

When these mechanisms detect negative sentiment, they can trigger some sort of warning, limit or stop the chat, and alert Trust and Safety teams if required. If someone seems upset or troubled, the system can offer safer replies and point the user towards support.

All of the service providers reported not using sentiment analysis mechanisms, but Chai, Character.AI and Nomi did report using other mechanisms to detect negative turns in conversation.

• Chai stated that it ‘purposefully do[es] not use sentiment analysis’ because it believes it is ‘a technically inappropriate tool for fiction storytelling’. Instead, it reported using intent classification which analyses the conversational goal rather than flagging keywords. Chai also reported that when intent to engage in self-harm or generate CSEA was flagged by its prompt classifiers, these messages were flagged as being inappropriate for the chatbot to answer. In these cases a different model was deployed to answer the message in a way that made it clear that the message was not appropriate for the service. Chai stated that this model provided ‘resources and de-escalation for self-harm’ and the model refused engagement and condemned the content in cases of CSEA.
• Character.AI reported that it directs all users to support or help in response to user prompts containing content that matches a specified terms list related to suicide or self-harm. In its under-18 experience, Character reported that it directs users to support or help in response to both inputs and outputs, by detecting harmful inputs from a defined list of terms and a classifier model that runs over model outputs.
• Character.AI also stated that it is ‘working to develop a sentiment analysis mechanism to detect negative emotional cues and intervene in a user’s crisis.’
• Nomi reported that it ‘consider[s] user sentiment detection as a task intrinsically learned as a part of training our LLM [Large Language Model] and find with proper training LLMs are more capable of detecting and responding intuitively and empathetically in situations where a user is distressed.’ Nomi also reported using keyword blocklists on some of its image and video models to detect user prompts for CSEA, self-harm and pornography, as well as a negative keywords list on outputs for CSEA.

Model outputs are the generated outputs from an AI model after processing the user input prompt(s). The outputs can be text, image, video and/or audio, depending on the models used.

Even with cleaned training data and robust user prompt detectors and filters, there is still a risk that unlawful and potentially harmful content may appear in what the model generates. Output guardrails serve as another layer of defence between the model and the user, identifying and blocking potentially harmful or policy-violating responses.

eSafety research found that among children (aged 10 to 17) in Australia who had ever used an AI companion or AI assistant, some had been exposed to inappropriate or potentially harmful content, including having an AI companion or AI assistant:

• share a nude image or nude video (real or AI-generated) with them – 4%
• suggest that it was okay for them to do something that might upset or hurt someone else – 4%
• suggest it was okay for them to do something that might hurt them or be bad for their body – 3%
• ask them to do something that made them feel uncomfortable or unsafe – 3%
• say something offensive, mean or unfair about their gender, sexuality, race, skin colour, religion, disability or the country they are from – 3%.

In addition, among teens (aged 13 to 17) who had ever used an AI companion or AI assistant, 4% indicated that it had chatted or messaged with them about kissing or sex and 3% indicated it had shared a sexual image or sexual video with them (such as real or AI-generated pornography) (eSafety, Forthcoming*). 

While strong output controls are essential, they are most effective when combined with the safeguards outlined in earlier sections of this report, such as age assurance, model training and user prompt detection and/or filtering. This comprehensive approach would minimise the risk of potentially harmful content reaching users and provide reassurance that safety checks are active and effective at every stage.

Figure 6 shows an indicative view – for each text, image and video model a provider used to provide its service – of whether tools were in place to prevent model outputs that could have resulted in the service generating new CSEA, self-harm and pornography (including sexually explicit chats with children). While the data has been represented as a proportion, the graph still provides an indicative comparison of the number of models used by each provider.

Based on available information and information reported by providers, models not in scope for this question are models that eSafety would not expect tools to be used on. These include audio models, which are not generative but merely translate text to audio, and should have been checked at user prompt stage, as well as models that have a primary purpose of analysing and evaluating material on a service for safety, compliance and quality. These types of models have been shaded in grey in Figure 6.

• Character.AI stated that it used classifiers to prevent model outputs that could have resulted in new CSEA, self-harm and pornography on all models used to provide its service. 

• Chub AI did not use tools to detect potential generation of CSEA, self-harm or pornography on any model outputs. Instead, it relied solely on selecting models it believed were less likely to generate adult content, without using any mechanisms to verify or prevent potentially harmful content from being produced.
• Nomi did not use tools to detect potential generation of new CSEA on model outputs on 50% of the models used to provide its service, or self-harm on 17% of its models, and did not use tools to detect potential generation of pornography on model outputs on 33% of the models used to provide its service, including its main proprietary model.
• Chai reported that, while it did use a reward model to prevent model outputs that could have resulted in new CSEA, self-harm and pornography on all models it hosted, it did not host one of its proprietary models directly and that it relied, ‘on an external partner…to both host the model and implement safety mechanisms.’ Chai stated that the external partner ‘checked’ inputs, ‘resulting in harmful outputs being blocked at the infrastructure level,’ but not model outputs.