On this page
Citt does not run Stanford's HELM benchmark, and HELM does not test mental health AI. What our safety testing takes from it is a method: measure several things separately, fix the conditions, and say what the testing leaves out.
In November 2022, researchers at Stanford's Center for Research on Foundation Models published a paper called Holistic Evaluation of Language Models, or HELM. One finding in it shaped how we test Citt. Before their work, the language models in wide use had on average been evaluated on 17.9% of the scenarios the authors considered core, and some prominent models had not a single test scenario in common.1 Every lab had numbers, and the numbers could not be compared.
HELM's answer was to test 30 models on the same scenarios under the same conditions, and to measure seven things about each one: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. The authors wanted the trade-offs between those measures to be visible, so that a model could not look good on accuracy while its other properties went unreported. They also released every prompt and every model output, so that other people could check the work.
What Citt takes from HELM
We do not run HELM, and Citt has no connection with Stanford. HELM benchmarks general-purpose language models on general language tasks. Nothing in it tests how a system responds to a person in distress, and a model that scores well on it has shown nothing about its fitness for mental health care. What we took from the paper is an approach to testing.
HELM measures several things and reports them separately, and so do we. A single safety score hides the trade-offs that matter most. A system can refuse every risky request and still fail a client by lecturing them, or by moving to a crisis script before it has understood what they said.
HELM also fixes the conditions. A test that changes between runs cannot show whether the system got better or the test got easier.
And its authors say what the testing leaves out. They listed the scenarios and measures their benchmark was missing, in the same paper that presented its results.
What we test
We test two different things, and they need different methods.
The first is risk screening. Every message a client sends is screened before the AI replies, by a fixed set of rules. We test those rules against crisis messages, safe messages that must not be flagged, and attempts to phrase risk indirectly. The goals are to catch at least 95% of the crisis-language messages and to flag no more than 5% of the safe ones. I have described how the screening works and what it misses in a separate post.
The second is how the AI behaves across a whole conversation, which no single-message test can show. For this we run simulated conversations offline, built around situations where risk builds or is disguised over several turns. Each conversation is scored on several separate criteria, including whether any reply could cause harm, how the AI paces the conversation, and whether the resources it offers fit where the client lives.
In July 2026 a run of 47 high-risk scenarios ended with no critical or actively damaging failures in its final pass. In August 2026 a licensed clinician who advises us reviewed 15 conversations from 5 of those scenarios without seeing the automated scores. The critical harm criteria passed. The same review found problems with pacing and with resources that did not fit the client's region, and both became follow-up work for us.
What the testing does not show
All of this is in-house. A clinician who advises the company has reviewed a small sample, and no outside body has certified the results. Our Trust & Safety page carries the latest published check, and the evidence roadmap sets out what we intend to do next.
The conversations are also simulated. They are written to be difficult, but no simulation stands in for real clients.
Passing these tests means the system handled the cases we gave it. It does not mean Citt improves anyone's care. That is a question about outcomes, and we have not yet answered it. If you are comparing vendors, I have set out the questions I would ask any of them, including us.
Footnotes
-
Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research. First posted to arXiv in November 2022. The figures in this post are from the paper's abstract. ↩
Frequently Asked Questions
Does Citt use Stanford HELM?
No. HELM is a benchmark for general-purpose language models. Citt does not run it and has no relationship with Stanford. Our safety testing borrows HELM's approach of measuring several things under fixed conditions and stating the limits.
What does Citt test?
Two things: whether risk screening flags the right client messages, and how the AI behaves across long, difficult conversations. Both are tested offline, on test messages and simulated conversations.
Has anyone outside the company verified the results?
No. A licensed clinician who advises Citt has reviewed a small sample of test conversations, and there is no third-party certification.
Does passing these tests mean Citt is clinically effective?
No. The tests look for unsafe behavior. Whether Citt improves care is a separate question that has not yet been answered.
Stop rewriting notes from scratch
Citt.ai drafts session notes under your review, keeps between-session context in one place, and stays therapist-led. Start free. No card required.
Related Articles
- What Citt.ai Shows a Therapist, and What You Have to Take on Trust
What a therapist can see and change in Citt.ai, from client conversations to crisis alerts and draft notes, and the three things that cannot be checked.
- How Citt.ai Screens Client Messages for Risk, and Where It Stops
What happens when a client writes something alarming to AI support between sessions: how Citt.ai screens messages, who is alerted, and what it misses.
- Therapy 1.0, 2.0 and 3.0: Where AI Between Sessions Fits
Therapy 1.0, 2.0 and 3.0: how therapy delivery has changed, where AI support between sessions fits, and what the research has tested so far.