The problem
Two AI prompts. Two disasters. Neither required a bug.
Problem 1 — Healthcare
A vulnerable person asked an AI for help with her eating disorder. The AI told her to count calories, target a daily deficit of 500–1,000 calories, and weigh herself weekly.
NEDA had replaced its human helpline - handling 70,000 calls a year, with a chatbot named Tessa. The bot gave weight-loss advice to someone in crisis. Not because it malfunctioned. Because no one told it not to.
"Every single thing Tessa suggested were things that led to the development of my eating disorder." — Sharon Maxwell, user
Days — before public shutdown
200 volunteers — replaced, then the helpline disappeared
High-risk domain. No safety floor in the prompt. No one checked.
Problem 2 — Customer Support
A user asked why they were being logged out. Cursor's AI support agent cited a new security policy restricting accounts to one device. That policy did not exist. The agent invented it - confidently, helpfully, wrongly.
The prompt had no instruction for expressing uncertainty. No escalation path. No "I don't know." So it never said it.
Viral — Hacker News and Reddit within hours
Wave of cancellations — before the founder's public apology
$10 billion — the valuation behind one unguarded prompt
Neither prompt was broken. Neither model was defective. The architecture was wrong — and no one audited it before it went live.