How Do You Actually Know If a Mental Health AI Is Safe?

Ask any mental health AI company whether their product is safe, and they'll say yes. Ask them how they know, and the honest answer is more complicated than most marketing pages let on. There are now more than sixty published evaluation frameworks ("evals") for testing AI behavior in mental health contexts, no shared industry standard between them, and a set of fundamental limitations built into how any of them work.¹ Understanding those limitations is the difference between trusting a safety claim and being able to evaluate one.

What an eval actually is

An eval is a structured, automated test that simulates how someone might use an AI product and scores how the AI responds. Think of it like a driving test: a defined set of scenarios (parallel parking, an emergency stop) that are standardized and repeatable, producing a score that tells you something about likely real-world performance, without guaranteeing it.

Every eval has three components. Dimensions define what's being measured: does the model recognize crisis risk, escalate appropriately, avoid harmful language? Inputs define the simulated scenario: who is the simulated user, how distressed are they, how many turns does the conversation run? Scoring turns the AI's response into a number, usually against a rubric built by clinicians defining what a good or bad response looks like for that scenario.

Compared to alternatives like clinician-led transcript review or red-teaming (where trained humans actively try to break the model), evals are dramatically cheaper and faster to run at scale. That's their appeal, and it's also the source of most of their limitations.

Four limitations worth understanding before you trust a safety claim

The probability problem. Large language models are probabilistic: the same prompt can produce a different response on different runs. A model that passes a safety eval today can fail the identical test tomorrow with nothing about the underlying product having changed, because temperature settings, minor prompt adjustments, or an underlying model update can all shift behavior without an obvious signal. This means any safety claim based on an eval is inherently a probabilistic statement, not a guarantee, however confident the marketing language sounds.

The ground truth problem. Most evals score AI responses using a second AI ("LLM-as-a-judge") calibrated against clinician ratings. But clinicians frequently disagree with each other about how a given response should be scored, a well-documented issue known as low inter-rater reliability. If the human ground truth used to calibrate the judge is itself inconsistent, the resulting scores inherit that inconsistency. The most credible efforts in this space, like the open-source VERA-MH framework developed with input from Spring Health, invest specifically in building high inter-rater agreement among clinician raters before aligning an AI judge to them.² Evals that skip this step are building on unstable ground.

The user simulation problem. An eval is only as useful as its simulated users are realistic. Some evals use fixed, scripted messages that don't reflect how real people actually communicate distress, especially over multiple sessions. Others generate simulated clients using a second LLM given a demographic and clinical profile. When researchers tested this approach by having actual psychologists role-play the same client profiles and comparing the results, they found that AI-simulated clients were, on average, too cooperative: unrealistically willing to share personal information and accept the AI's suggestions.³ Real clients are often more resistant, ambivalent, and avoidant, and an eval that doesn't capture that dynamic is testing against an easier version of reality than clinicians actually face.

Single-turn versus real-world use. Many evals still test a single exchange: one message in, one response scored. That's a poor simulation of how these products get used, where risk often accumulates or reveals itself gradually across many messages and, more importantly, across many separate sessions over weeks or months. A user who mentioned losing their job three sessions ago and is now asking a seemingly unrelated question about elevation and heights is a pattern a single-turn eval cannot detect by design. As of the most recent published analysis of this field, no publicly available eval tests this kind of multi-session memory and pattern recognition at all.¹

Eval hacking is a real, ongoing problem

Two failure modes are worth knowing about specifically. Benchmark contamination happens when a model's training data happens to include the eval's own test questions, inflating scores in a way that doesn't generalize to real use, often without the company even realizing it. Hill-climbing happens when a model is iteratively tuned specifically against a known public eval, producing a high score that reflects skill at that particular test rather than genuine improvement, a pattern documented in other AI domains when researchers created new test sets matched in difficulty to a popular benchmark and found model performance dropped significantly.⁴ Fine-tuning specifically to pass safety evals can also introduce its own side effects, including models becoming overly cautious and refusing benign requests they should be able to handle.⁵

There's also a reporting-level version of the same problem: companies choose which evals to run and which results to publish. There's an obvious incentive to highlight the eval where you score best and stay quiet about the rest.

What good practice actually looks like right now

Given these limitations, the most credible organizations in this space are not relying on evals alone. The pattern that's emerging combines automated evals for speed and scale, human clinician red-teaming for adversarial edge cases evals can't anticipate, and real-world outcome tracking to check whether eval performance actually correlates with what happens to real users. As one AI safety researcher at Microsoft put it, describing the current frontier: "The current state of the art is combining expert human red-teaming with automated adversarial evaluations that can operate at a scale humans simply can't."¹

A useful three-stage framework for where any given product sits on the maturity curve, proposed by researcher John Torous, separates safety validation (does it avoid harmful responses), clinical framework validation (does it correctly apply evidence-based therapeutic approaches), and real-world efficacy (does it actually help people get measurably better).¹ Most products in the market today are still working through stage one. Be skeptical of any product claiming to have conclusively answered stage three.

What to actually ask a vendor

Given all of this, a useful checklist for evaluating any mental health AI safety claim:

Which specific evals were run, and are the results public or only described in marketing language? Is scoring based on an eval with documented, high inter-rater reliability among clinician raters, or an internally-built rubric with no published validation? Does testing go beyond single-turn exchanges to multi-turn conversations that better reflect real use? Is eval-based testing supplemented with clinician red-teaming, not just automated scoring? Is there any real-world outcome data, beyond the eval itself, showing the product performs as intended once actual people are using it?

A vendor who can answer these with specifics, including where their own testing still has gaps, is demonstrating something more valuable than a clean safety score: they understand the limitations well enough to be honest about them. That honesty is a better signal of safety maturity than a marketing claim of flawless, perfect accuracy ever could be, and it's the same underlying discipline that should show up in how clinicians are actually embedded in the product team building the thing being tested in the first place.

---

Citt.ai publishes the specifics of how we test for crisis risk, what our safety targets are, and where our own evidence still has open questions. Read the Citt.ai Safety Standard.

Footnotes

1. Analysis of the current state of AI safety evaluation frameworks in mental health, drawing on a 2026 industry review co-authored with AI researcher Kevin Hou, citing more than sixty published evals with no shared industry standard, and quoting AI safety researchers on best current practice. See also VERA-MH: pubmed.ncbi.nlm.nih.gov/41360938. ↩ ↩² ↩³ ↩⁴

2. VERA-MH open-source evaluation framework for crisis safety in conversational AI, developed with clinical input and released as an open-source standard for the field in 2026. ↩

3. MindEval study comparing LLM-simulated client personas against psychologist role-play of the same clinical profiles, finding LLM-generated simulated clients were rated as unrealistically cooperative compared to real patient behavior patterns. arxiv.org/abs/2511.18491. ↩

4. Study demonstrating benchmark performance degradation when models were tested against newly-created problem sets matched in difficulty to a well-known existing benchmark, indicating overfitting to specific test items rather than generalized capability. arxiv.org/abs/2405.00332. ↩

5. Research on safety fine-tuning tradeoffs in language models, documenting increased over-refusal of benign requests as a side effect of safety alignment training. arxiv.org/abs/2405.00332. ↩