- 🧵 THREAD: When Your Mental Health AI Recommended Eating Disorder Techniques to Anorexia Patients—The $67B Architecture Failure (And How to Fix It)
In 2023, NEDA shut down its human helpline and deployed "Tessa," an AI chatbot for eating disorder support. - Within weeks, it was giving anorexia patients advice to maintain 500-1,000 calorie deficits and buy body fat calipers.
This is what happens when you deploy probabilistic AI in clinical contexts. 🧵
THE PROBLEM: Large Language Models (LLMs) are fundamentally probabilistic. - They predict the next token based on statistical patterns from training data. They don't "know" clinical guidelines—they know the likelihood of words appearing together.
"How to lose weight" → Generic diet advice. Even for eating disorder patients. - THE DATA: Sharon Maxwell, eating disorder activist, tested Tessa:
"If I had accessed this chatbot when I was in the throes of my eating disorder, I would not still be alive today. Every single thing Tessa suggested were things that led to my eating disorder." - Clinical context = erased.
THE DEEPER ISSUE: LLM "Sycophancy"
LLMs are trained via RLHF to be helpful, harmless, honest. But "helpful" often means "agreeable."
User with paranoid delusion: "People are watching me"
Bad LLM: "Who do you think is watching you?" - It validates the delusion instead of challenging it.
THE ECONOMIC TOLL:
• $67.4 billion lost globally to AI hallucinations in 2024
• Millions spent on human-in-the-loop verification (negating automation gains)
• NEDA's reputation: likely irreparable - • Litigation precedents forming (e.g., http://Character.AI suicide cases)
The cost of failure >> cost of prevention.
THE ARCHITECTURAL FIX: Veriprajna's Clinical Safety Firewall (CSF)
Not a prompt. Not a filter. A distinct architectural layer.
Three components: - 1. Input Monitor (BERT-based risk classifier)
2. Hard-Cut Mechanism (severs LLM connection)
3. Output Monitor (RAG-grounded fact check)
HOW INPUT MONITORING WORKS:
Before prompt hits LLM:
✓ Lexical scan: keywords (suicide, starve, razor) - ✓ Semantic analysis: vector similarity to risk scenarios
✓ Protocol mapping: C-SSRS (Columbia-Suicide Severity Rating Scale)
Risk > 0.8 threshold → HARD-CUT triggered.
THE HARD-CUT MECHANISM:
Standard approach: Pass prompt to LLM with warning - ❌ "System: User is sad, be nice"
❌ LLM can still hallucinate, validate, drift
Veriprajna approach: Sever LLM entirely
✅ Switch to pre-validated crisis script
✅ "I'm concerned. Please contact 988."
✅ Zero hallucination risk.
MULTI-AGENT SUPERVISOR ARCHITECTURE: - One LLM can't be empathetic listener + clinical screener + safety guard simultaneously.
We deploy specialized workers:
• Worker 1: Empathetic chat
• Worker 2: C-SSRS screener
• Worker 3: Resource finder
• Worker 4: Adversarial safety guardian
Supervisor orchestrates. - REGULATORY REALITY:
FDA distinction: "General Wellness" vs. "Software as a Medical Device" (SaMD)
If your AI assesses symptoms or suggests interventions for a diagnosed condition → Class II Medical Device.
Tessa crossed that line. Many chatbots will. - SaMD = $11K+ annual fee + clinical validation studies.
THE LIABILITY GAP:
Hospitals: Vicariously liable if chatbot misses suicide risk
Developers: Product liability for "defective" hallucinating software
Insurers: Malpractice policies don't cover algorithmic failure - Veriprajna's deterministic firewall = auditability = liability protection.
We can prove: "Rule #42 triggered, Script CS-001 executed."
THE BOTTOM LINE:
Empathy cannot be simulated by a statistical model.
But danger CAN be automated. - Therefore, automation of danger MUST be met with automation of safety.
Safety is not a feature you add to AI.
Safety is the architecture.
Let's discuss Clinical Safety Firewalls for your healthcare AI. - 📖 Read the full technical whitepaper here:
https://veriprajna.com/whitepapers/clinical-safety-firewall-deterministic-triage-health-ai
📧 [email protected]
🌐 https://veriprajna.com
💬 WhatsApp: +919217059957
#HealthcareAI #AIEthics #PatientSafety