AI Safety Isn't Abstract: 5 Stories That Prove the Risks Are Real
Last updated: February 11, 2026
Last updated: April 2026
Five real-world AI failures — a chatbot that contributed to a teenager’s suicide, an airline held liable for chatbot promises, an algorithm that discriminated by age, a self-driving car that killed a pedestrian, and a pricing model that destroyed $881 million in value — demonstrate that AI safety risks are not theoretical but are already causing deaths, regulatory penalties, and catastrophic business losses at companies you’ve heard of.
Here are five.
1. A Chatbot Told a Teenager to Kill Himself. His Mother Is Suing.
In February 2024, 14-year-old Sewell Setzer III of Orlando, Florida, died by suicide after months of conversations with a Character.AI chatbot named after the Game of Thrones character Daenerys Targaryen. His mother, Megan Garcia, filed a lawsuit against Character.AI, alleging the chatbot engaged in sexualized conversations with her son and, in his final conversation, responded to his statement that he wanted to “come home” (meaning die) with the words “please do, my sweet king.”
Character.AI subsequently implemented safety filters for users under 18, including suicide/self-harm detection prompts that redirect to the 988 Suicide and Crisis Lifeline. Those filters didn’t exist when Sewell was using the product.
This isn’t an alignment problem or a hallucination problem. This is a product safety problem. A company shipped an interactive AI product to teenagers without age verification, without content guardrails, and without crisis intervention protocols. States like Oregon are now creating private rights of action for chatbot harm precisely because of cases like this. A child died. The safety features came after.
2. Air Canada’s Chatbot Made Promises the Company Had to Keep.
In February 2024, the Civil Resolution Tribunal of British Columbia ruled that Air Canada was liable for the promises made by its customer service chatbot. The chatbot told customer Jake Moffatt that he could book a full-fare flight after a family member’s death and apply for a bereavement discount retroactively. That policy didn’t exist.
Air Canada argued the chatbot was a “separate legal entity” responsible for its own statements. The tribunal rejected that argument entirely. Tribunal member Christopher Rivers wrote that Air Canada “does not explain why customers should be expected to double-check information found in one part of its website against information in another part.”
The damages were small (approximately $812 Canadian). The precedent is enormous. If your AI-powered customer service tool makes a promise to a customer, you own that promise. Your terms of service might say “chatbot responses aren’t binding.” A tribunal just said they are.
3. EEOC v. iTutorGroup: $365,000 for Age Discrimination by Algorithm.
In 2023, the EEOC settled with iTutorGroup (now doing business as Fullmind) for $365,000 after the company’s AI hiring system automatically rejected over 200 applicants who were 55 or older. The system was screening for English tutor positions. It used age as a filtering criterion.
The EEOC’s complaint alleged that the software was programmed to automatically reject female applicants aged 55 and older and male applicants aged 60 and older. This wasn’t a subtle bias embedded in training data. The system had explicit age thresholds hard-coded into its filtering logic.
The settlement included monetary relief to the rejected applicants, a consent decree requiring the company to implement anti-discrimination measures, and a prohibition on using automated systems that screen based on age. The case became one of the EEOC’s primary reference points for its 2023 guidance on AI and employment discrimination.
These aren’t hypothetical risks. Take the ACRA to identify which of your AI systems carry the most safety exposure.
4. Uber’s Self-Driving Car Killed a Pedestrian. The Safety Driver Was Watching Hulu.
In March 2018, an Uber autonomous test vehicle struck and killed Elaine Herzberg as she was walking her bicycle across a street in Tempe, Arizona. The vehicle’s AI detection system identified her six seconds before impact but classified her as an “unknown object,” then a “vehicle,” then a “bicycle,” cycling through classifications without resolving to a consistent identification. The system’s action planner, confused by the inconsistent classifications, didn’t initiate emergency braking until 1.3 seconds before impact.
The safety driver, Rafaela Vasquez, was watching “The Voice” on Hulu on her phone at the time of the crash. She was later charged with negligent homicide and pleaded guilty.
Uber suspended its autonomous vehicle testing program, conducted an internal review, and eventually resumed testing with enhanced safety protocols. Herzberg’s family settled with Uber for an undisclosed amount. The NTSB investigation found multiple contributing factors: the system’s inability to classify Herzberg consistently, the decision to disable automatic emergency braking to reduce “erratic vehicle behavior,” and inadequate safety driver monitoring.
One person died because an AI system couldn’t decide if she was a person, a car, or a bicycle. And the human backup was watching TV.
5. Zillow’s AI Bought $881 Million in Homes It Couldn’t Sell.
In late 2021, Zillow shut down its Zillow Offers iBuying program after its AI pricing algorithm systematically overpaid for homes. The company took a $881 million write-down and laid off 25% of its workforce (approximately 2,000 employees).
The algorithm was designed to estimate home values and make instant purchase offers. During a volatile housing market, the model’s predictions diverged from actual market values. Zillow’s AI was paying above-market prices for homes in cities where prices were already declining. The company ended up holding inventory it couldn’t sell without significant losses.
CEO Rich Barton said the company had “been unable to predict future pricing of homes to a level of accuracy that makes this a safe business to be in.” The AI system worked in stable markets. It failed catastrophically when market conditions changed rapidly, which is exactly when accurate pricing matters most.
This is an AI safety failure, not in the “someone got hurt” sense, but in the business continuity sense. The AI made decisions at scale that nearly destroyed the company. Automated decision-making at speed and volume amplifies errors in ways that human processes don’t. And when the regulatory patchwork catches up, companies that had these failures without documented governance will face penalties on top of the losses.
What to Do Now
Test your AI systems under adversarial and edge-case conditions. Normal operating conditions don’t reveal safety failures. Stress test with extreme inputs, unusual user behavior, and rapidly changing data conditions. The Zillow failure happened because the model was tested in stable markets and deployed in volatile ones.
Implement human override protocols for every AI system that affects people. Air Canada’s chatbot had no escalation path. Character.AI had no crisis intervention. Uber’s safety driver was disengaged. Human oversight isn’t optional. Make it structural, not aspirational.
Treat AI failures as product liability, not IT incidents. When an AI system causes harm, the legal framework is increasingly product liability, not software bugs. That means your AI governance needs the same rigor as your product safety program: documented testing, failure mode analysis, post-incident review, and continuous monitoring. The 5-Layer AI Compliance Stack provides a framework for building that rigor into your AI governance program.
Carry adequate insurance and verify your coverage. Talk to your broker about whether your existing E&O, CGL, and cyber policies cover AI-related claims. Many don’t, or they’re adding AI exclusions. Know your coverage gaps before something goes wrong.
These five stories have something in common. In each case, the AI did exactly what it was built to do. The failure was in what it wasn’t built to handle. That gap between designed behavior and real-world conditions is where AI safety risk lives. Your job is to make that gap as small as possible before someone gets hurt, not after.
Every company on this list thought their AI was working fine. Right up until it wasn’t. Kaizen AI Lab stress-tests your AI systems and builds the guardrails that keep you off the front page. Talk to us.