The word "moderation" suggests one mechanism. Inside companion apps it is at least four separate ones, moving at different speeds and involving different people.
The four layers
The further down, the less content and the closer the look.
1. Live filters. Software classifiers can inspect what you send and what the model answers before either is displayed. They look for prohibited topics, including minors, some kinds of violence and self-harm cues, and respond by blocking, softening the reply or displaying a notice. Refusals and crisis cards come from here, and content filters covers why they sometimes trigger in the middle of a scene.
2. Conversation flagging. Other systems may rate full chats or accounts on patterns such as persistent filter-dodging, harassment, or signs that a minor has joined an adult app. Being flagged is not a punishment; it only means a human might look.
3. People. Trust and safety teams, frequently employed by outside contractors, go through some flagged chats, user reports and appeals. A few firms also pull ordinary chats to check quality or train models. Policies tend to describe this stage as "improving our services" or "enforcing our terms."
4. Legal demands. Courts, police and regulators can require data through formal orders, and each company responds according to its own policy and the law it is subject to.
What tends to bring a person in
| Event | Why it matters |
|---|---|
| Anything involving minors | Legally required in most countries; often reported further |
| Self-harm or suicide cues | Several US states now mandate crisis protocols |
| Threats to actual people | Risk of real-world harm |
| A report or support ticket from you | You asked someone to look |
| Persistent filter evasion | Enforcement of the terms of service |
| Random sampling for quality | Product improvement, if the policy allows |
Police and courts in Canada
Canadian privacy law restricts what a company may hand over. PIPEDA lets a private organization disclose personal information to a government body that asks only when that body names its lawful authority. In R. v. Spencer (2014), the Supreme Court of Canada ruled that subscriber information tied to online activity carries a reasonable expectation of privacy, so police normally need a warrant or production order; an informal request does not suffice.
Keep two limits in mind. First, that protection applies in Canadian proceedings, yet many companion apps operate from other countries and follow local law there, so their own policies tell you what they really do. Second, the practical lesson is unchanged: whatever is stored can eventually be demanded.
What the new laws add
In the US, companion chatbot statutes, starting with New York in 2025 and California in 2026, oblige apps to spot talk of suicide and self-harm and point users to crisis services; California further requires companies to report on their protocols. Expect more automated watching of the most sensitive chats as a result, and not less. See AI companion laws in the US.
Canada has nothing equivalent in force. In June 2026 Ottawa tabled Bill C-34, the Safe Social Media Act. It would extend to AI chatbot services alongside social media, requiring them to limit harmful content and to report when a user intends to hurt themselves or someone else, overseen by a new Digital Safety Commission. It remains a bill and may be amended or lapse.
Reading the fine print
Scan the privacy policy and terms for these words:
- "review," "moderate," "human" reveal whether people read chats, and for what purpose.
- "contractors" or "service providers" show if the reviewers work for a third party.
- "improve" and "train" point to sampling of ordinary conversations.
- "law enforcement," "legal process," "court order" explain when data is released and whether a court order is needed.
The better policies list the triggers and confine reviewer access to what is necessary. A loose one granting staff "all content for any business purpose" is informative too. The rest of the document is covered by our four privacy checks.
Practical takeaways
- Type as though a reviewer could see it, since occasionally one does.
- Omit details that identify other people, above all in explicit scenes; whoever examines a flagged chat reads those details too.
- When a filter trips during a story, a brief out-of-character remark usually clears it, while arguing in character can raise further flags.
- If privacy from everyone is your top priority, only a companion hosted on your own computer guarantees it. Running an AI companion locally describes the trade-offs.
This is oversight, not surveillance. In nearly all conversations, nobody and nothing looks past the automatic filter. The exceptions, though, are set by the policy and not by how friendly the app feels, so learn them ahead of time.