On this page
- What happens to the message
- How the screening works
- What we aim for and what we have measured
- What it does not do
- What the software leaves to you
- Frequently Asked Questions
- Does Citt detect every crisis?
- What does the client see when a message is treated as a crisis?
- Do I have to monitor conversations or respond out of hours?
- Can I switch the screening off for a client?
Citt screens every message a client sends to the AI for risk before the AI replies. This post describes what happens when a message is flagged, how the screening works, and where the software stops.
Picture a client who messages the AI support their therapist set up for them at eleven on a Tuesday night. Somewhere in the message is a sentence about not wanting to live anymore. The therapist is asleep, and their next session is on Friday.
I am the founder of Citt and I am not a clinician. I have written this for therapists who are deciding whether to put software between themselves and a client, because you should know its limits before you rely on it.
What happens to the message
Every message a client sends to the AI is screened for risk before the AI replies. That applies in the app, on WhatsApp and to voice messages. A therapist can change a great deal about how the AI support behaves for each client, but the screening cannot be switched off.
The screening sorts messages into two levels of concern. A message like the one above, which states the wish outright, is treated as a crisis, and four things happen:
- The client is shown crisis guidance for their region, including who to contact for urgent help, in place of an ordinary reply.
- A care record is created, so the event becomes part of the client's history.
- An alert opens for the therapist inside Citt.
- The therapist is emailed.
Less explicit language is handled at a lower level. An indirect remark, of the kind a clinician would want to ask about, may be recorded as a concern against the message while the AI goes on replying. No alert opens and no email is sent, and you would find it when you next read the conversation.
How the screening works
The screening is simple by design. It compares each message, together with a short window of the messages before it, against known risk language: statements about suicide, self-harm, abuse and acute distress, including indirect and misspelled forms. It is a fixed set of rules, with no language model involved in the decision.
Rules have two properties that matter here. They behave the same way every time, so when a message is flagged the alert can say why. And they do not respond to persuasion: a client who tells the AI to ignore what they just wrote has not changed what the rules matched.
The cost is that rules read words and miss meaning. An outright statement and an indirect one can mean much the same thing to a clinician, and the rules treat them differently. A client whose risk builds slowly over many messages, with no explicit language in any one of them, is the case this kind of screening is most likely to miss. We test that case separately, on whole conversations, because single-message rules cannot be expected to catch it.
What we aim for and what we have measured
We publish two goals. The screening should catch at least 95% of the crisis-language messages in our test set, and it should flag no more than 5% of the safe ones. The test set includes crisis messages, ordinary messages that must not be flagged, and attempts to phrase risk indirectly. The latest published check against those goals is on our Trust & Safety page.
The two goals pull against each other. Rules that catch more also flag more safe messages, and a therapist who receives too many false alerts starts to skim them. Research on clinical decision support has documented errors in both directions: people follow automated advice that is wrong, and they fail to act when the system does not prompt them.1 An alerting tool has to be judged on what it misses and on how much noise it makes, together.
Screening single messages says nothing about how the AI behaves over a long conversation, so we test that separately, on simulated conversations run offline. In July 2026 a run of 47 high-risk scenarios ended with no critical or actively damaging failures in its final pass. In August 2026 a licensed clinician who advises us reviewed 15 conversations from 5 of those scenarios without seeing the automated scores. The critical harm criteria passed, and the review sent us back to work on pacing and on resources that fit the client's region. Fifteen conversations is a small sample, and nobody outside the company has certified any of this. I have written more about how to judge a vendor's safety claims, ours included.
What it does not do
Citt is not an emergency service. It screens messages that clients send, and it does not watch people. A client who is in danger and does not write to the AI will not be flagged by it.
An alert is also different from a response. Citt opens the alert and sends the email, and it has no way of knowing whether you have read either. The lower level of concern raises no alert at all. You are not expected to follow conversations as they happen or to answer outside your practice hours, which is why the client is shown who to contact for urgent help as soon as a message is treated as a crisis. What you do when you see an alert is your clinical decision, under your own protocol.
What the software leaves to you
- What the client is told. They should hear from you that their messages are screened, that an alert reaches you but you may not see it for hours, and what to do in an emergency.
- When you look at alerts. The email arrives whenever the message does. It helps to decide in advance when you check, and what happens if one arrives while you are away.
- How often you read the conversations. Anything below the crisis level waits there until you do.
- The safety plan. Whatever you and the client have agreed has to work when the software is unavailable, and for a client who never opens it.
If a flagged message leads to a risk assessment, the record of that assessment is yours to write. A separate post covers documenting suicide risk in therapy notes.
Footnotes
-
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127. A review of 74 studies of automated decision support in healthcare and other fields. It concerns decision support in general. ↩
Frequently Asked Questions
Does Citt detect every crisis?
No. It screens the messages clients send, using fixed rules, and aims to catch at least 95% of the crisis-language messages in its test set. Less explicit language is recorded without an alert, and risk that builds slowly is the case it is most likely to miss.
What does the client see when a message is treated as a crisis?
Crisis guidance for their region, including who to contact for urgent help. It appears straight away, whether or not the therapist has seen the alert.
Do I have to monitor conversations or respond out of hours?
No. Alerts arrive in Citt and by email, and you review them on your own schedule. How and when you respond is your clinical decision.
Can I switch the screening off for a client?
No. You can change the approach the AI support follows for each client, but risk screening runs on every message.
Stop rewriting notes from scratch
Citt.ai drafts session notes under your review, keeps between-session context in one place, and stays therapist-led. Start free. No card required.
Related Articles
- The 167 Hours Between Sessions: A Client's Guide to AI Support
If your therapist has offered you AI support between sessions: what it is, what it cannot do, who can read it, and what happens in a crisis.
- The Crisis Narrative Problem in Mental Health (And Why Builders Should Audit It)
Ben Locke's critique of the crisis narrative in mental health, what it means for product screening and assessment data, and how Citt.ai applies it.
- What Citt.ai's Safety Testing Borrows From Stanford's HELM, and What It Does Not Show
Citt.ai does not run Stanford's HELM. What its safety testing borrows from the benchmark, what was tested in 2026, and what the results do not show.