AI incident response: Why most incidents are process failures

McDonald’s spent three years working with IBM to build AI-powered drive-thru ordering. The system was supposed to simplify orders and improve customer experience. Instead, viral TikTok videos showed customers pleading with the AI to stop adding Chicken McNuggets to their order - eventually reaching 260 pieces. McDonald’s shut down the entire pilot in June 2024.

Here’s what their incident response probably looked like: technical teams frantically debugging the speech recognition model, data scientists analyzing training data, engineers tweaking parameters. The real problem? They never built proper processes for handling edge cases, customer escalation, or graceful degradation when the AI confused sports terms with food orders.

Most AI incidents follow this pattern. We focus on the technical failure - the model, the data, the algorithm. But RAND research shows that more than 80% of AI project failures stem from organizational and process issues, not technical ones. This echoes the fragmentation problem we see with AI readiness assessments - traditional incident response misses this entirely.

The process failure pattern

The same pattern repeats across organizations: the most damaging incidents happen when processes break down, not when models fail.

The AI Incident Database reached its 1000th incident milestone in early 2025, with over sixty new incidents added in just two months. Even more telling: GenAI was involved in 70% of incidents, but agentic AI caused the most dangerous failures. But here’s what the data doesn’t show: most of these incidents were preventable through better processes, not better algorithms.

Consider Air Canada’s chatbot incident. Their customer service AI erroneously granted a refund based on outdated rules it had “learned.” When challenged, Air Canada argued they weren’t responsible for their chatbot’s promises. Small claims court disagreed. The technical failure was straightforward - stale training data. The process failure was catastrophic - no oversight around what the AI could promise customers.

Or take the McDonald’s McHire platform breach in 2025. Security researchers discovered the AI-powered hiring platform was accessible through default credentials “123456/123456” with no multi-factor authentication, exposing data linked to 64 million job application records. The AI worked fine - the process around securing it didn’t exist.

The pattern is consistent:

• Poor change management: Teams deploy AI updates without proper testing procedures
• Inadequate oversight: No clear authority structure for AI decision-making
• Missing escalation paths: No human backup when AI systems hit edge cases
• Weak monitoring: Focus on uptime numbers instead of quality degradation

IBM’s 2025 Cost of a Data Breach Report found that 13% of organizations reported breaches of AI models or applications, with 97% of those lacking proper AI access controls. Shadow AI is even worse - one in five organizations reported breaches due to unauthorized AI, costing $670,000 more and taking 59 days longer to contain than other incidents.

AI incident classification needs rethinking

Traditional incident response categorizes by technical severity - P1 for service down, P2 for degraded performance. AI incidents don’t fit this model.

Consider these real scenarios from my consulting work:

• Quality drift: Model accuracy slowly degrades from 94% to 87% over six months
• Bias amplification: AI recruiting tool systematically filters out qualified candidates
• Context confusion: Customer service AI provides confidently wrong answers
• Hallucination cascade: AI-generated content includes false information that spreads

These issues often stem from poor prompt design - something that proper prompt engineering practices can help prevent.

None of these register as “outages” in traditional monitoring. But they can damage business reputation and customer trust more than a complete system failure.

The OWASP Top 10 for LLM Applications 2025 introduced three new threat categories that didn’t exist in 2023, including System Prompt Leakage, Vector and Embedding Weaknesses, and Misinformation. Modern AI incident response systems now classify incidents by business impact rather than just technical severity:

• Type A: Immediate safety risk (autonomous systems, medical AI)
• Type B: Financial or legal exposure (decision-making AI, regulatory systems)
• Type C: Brand or reputation risk (customer-facing AI)
• Type D: Work efficiency impact (internal process AI)

Each type requires different response procedures and stakeholder involvement. A slowly degrading recommendation system (Type D) needs different handling than a chatbot giving legal advice (Type B).

Response procedures that actually work

NIST’s latest incident response guide emphasizes that effective response depends more on preparation and process than technical expertise. The new Cyber AI Profile (NIST IR 8596), released in December 2025, specifically addresses AI-related risks aligned with NIST’s Cybersecurity Framework 2.0. For AI systems, this preparation is even more critical.

The 15-minute rule

You have about 15 minutes to make the critical decision: contain or continue. Unlike traditional systems where the choice is obvious (broken = shut down), AI systems often limp along providing “mostly correct” output.

AI incident response frameworks recommend immediate isolation of affected systems to prevent further damage, but this requires pre-defined triggers:

• Quality threshold breach: Accuracy drops below acceptable levels
• Output anomaly detection: Unusual patterns in AI responses
• User feedback spikes: Complaints about AI behavior
• External notification: Media coverage or regulatory inquiry

Communication that builds trust

Research shows that open, swift communication during incidents helps organizations recover faster and maintain customer confidence. But AI incidents require different messaging than system outages - and as we’ve discussed in communicating AI changes effectively, the messaging needs to focus on human impact, not technical features.

Instead of “We’re experiencing technical difficulties,” try:

• “We’ve temporarily paused our AI recommendations while we investigate quality concerns”
• “Our customer service team is handling inquiries while we improve our AI responses”
• “We’re reviewing our AI decision-making process to ensure fair outcomes”

The key difference: acknowledge the AI component explicitly. Customers understand system outages. They don’t understand why AI gave them wrong information.

Multi-team coordination

AI incidents typically require coordination across teams that don’t usually work together:

• Technical teams: Model debugging, data analysis, system recovery
• Business teams: Customer impact assessment, team communication
• Legal/regulatory: Rule implications, liability assessment
• Communications: Public statements, media handling

Effective incident response requires clear escalation procedures and decision-making authority distributed across these functions. The worst AI incidents happen when technical teams make business decisions or business teams make technical ones.

Investigation techniques for AI failures

Root cause analysis for AI systems requires different approaches than traditional software. The problem isn’t just “what broke” but “why did we design it to break this way?”

The five-layer analysis

1. Immediate cause: What triggered the incident?
2. Technical cause: Why did the AI system behave unexpectedly?
3. Data cause: What in the training or input data contributed?
4. Process cause: Which procedures failed or were missing?
5. Organizational cause: What cultural or structural factors enabled this?

Analysis of AI failures shows that most root causes exist at layers 4 and 5 - process and organizational issues rather than technical problems. But traditional incident response focuses almost exclusively on layers 1-3.

Documentation standards

AI incident investigation requires documenting both technical facts and human decisions:

• Technical timeline: What happened to the system when
• Decision timeline: Who made which choices and why
• Data provenance: What training data or inputs were involved
• Model behavior: How the AI system responded to different scenarios
• Process gaps: Where existing procedures didn’t cover the situation

Harvard Business Review research indicates that organizations with detailed AI incident documentation recover 40% faster from subsequent similar incidents.

Recovery strategies that stick

Getting AI systems back online is only half the challenge. The other half is rebuilding trust - with customers, regulators, and your own team.

Phased restoration

Never restore full AI functionality immediately after an incident. Use staged rollouts:

1. Manual mode: Human handling with AI assist
2. Limited automation: AI handling simple cases only
3. Monitored automation: Full AI with enhanced human oversight
4. Normal operations: Standard monitoring resumed

Organizations using phased AI restoration report 60% fewer repeat incidents within six months.

User confidence rebuilding

AI incidents damage user trust differently than system outages. When your website crashes, users understand. When your AI gives wrong answers, they question your competence.

Confidence rebuilding requires:

• Visible improvements: Show users what you’ve changed
• Open monitoring: Share quality numbers publicly when appropriate
• Human backup: Ensure customers can easily reach humans when AI fails
• Feedback loops: Make it simple to report AI problems

Building incident response capabilities

63% of breached organizations either don’t have an AI governance policy or are still developing one. Of those with policies, only 34% perform regular audits for unsanctioned AI. The governance gap is widening: only 35% of organizations have established AI governance frameworks, and just 8% of leaders feel equipped to manage AI-related risks.

Training that works

Run quarterly tabletop exercises specifically for AI incidents. Use realistic scenarios:

• Gradual quality degradation over weeks
• Bias discovery in hiring AI
• Hallucination in customer communications
• Regulatory inquiry about AI decisions

Companies that conduct regular AI incident simulations resolve real incidents 50% faster than those that don’t.

The capability stack

Build incident response capabilities in this order:

1. Detection: Monitoring that catches quality issues, not just outages
2. Assessment: Rapid business impact evaluation systems
3. Communication: Templates and approval processes for AI incidents
4. Technical response: Containment and recovery procedures
5. Investigation: Root cause analysis that includes process factors
6. Learning: Post-incident improvement that prevents similar failures

(Consider using Tallyfy’s process documentation to standardize and automate your incident response workflows.)

The post-incident reality

ISACA’s analysis of 2025 incidents confirms that the biggest AI failures were organizational, not technical - weak controls, unclear ownership, and misplaced trust. Organizations using security AI and automation save an average of $2.2 million per breach compared to those with limited or no AI deployment. For AI systems, post-incident learning is even more critical because the failure patterns are still evolving.

Creating organizational memory

After every AI incident, document:

• What we learned about AI system behavior
• Which processes need updating or creating
• How we’ll detect similar problems earlier
• What authority structures worked or failed

Share these lessons across teams. The AI incident you prevent is worth more than the one you handle perfectly.

Continuous improvement

AI systems change faster than traditional software. Your incident response capabilities need to keep up. A practical incident-response framework for generative AI systems published in 2026 identifies six recurrent incident archetypes and formalizes structured playbooks aligned with NIST SP 800-61r3, NIST AI 600-1, MITRE ATLAS, and OWASP LLM Top-10. Organizations that adopt these structured approaches achieve significantly better AI system reliability.

Schedule regular reviews:

• Monthly: Detection capability assessment
• Quarterly: Response procedure updates
• Annually: Full incident response system review

The goal isn’t just handling incidents better. It’s building AI systems that fail gracefully and recover quickly.

Most AI incidents are process failures masquerading as technical problems. The organizations that recognize this - and build incident response around human factors, not just model monitoring - will have more reliable AI systems and more satisfied customers.

Start with your processes. The technology will follow.