Identifying AI Hazards and Responsibility Gaps
Card Grid View — Paper by Cummings (2025)
1. Swiss Cheese Model Limitations
- Why is Swiss Cheese model insufficient for AI?
- Focuses mainly on human/organizational failures
- Does not capture AI as a causal factor in accidents
- Cannot address failures in design, testing, maintenance, oversight
- Assumes accidents come from aligning human errors
- What is missing?
- AI-specific failure modes not covered
- Non-deterministic AI behavior not accounted for
- Responsibility gaps in AI systems ignored
2. TAIHA Framework
- What is TAIHA?
- Taxonomy for AI Hazard Analysis
- Adapts Swiss Cheese model for AI systems
- Identifies AI-specific layers of failure
- 4 TAIHA Layers
- Inadequate AI Oversight
- Inadequate AI Design
- Inadequate AI Maintenance
- Inadequate AI Testing
- Purpose
- Decompose design, maintenance, testing shortcomings
- Identify technical and responsibility gaps
3. AI vs Human Reasoning (SRKE)
- SRKE Taxonomy
- Skill-based reasoning (lowest uncertainty)
- Rule-based reasoning
- Knowledge-based reasoning
- Expert-based reasoning (highest uncertainty)
- Key difference
- Humans use both bottom-up AND top-down reasoning
- Neural networks use only bottom-up (pattern recognition)
- AI lacks contextual understanding and causal reasoning
- AI struggles with novel/high-uncertainty situations
4. Responsibility Gaps
- Definition
- Situation where it is unclear who should be held accountable after AI-related accidents
- Responsibility spread across design, oversight, testing, maintenance, regulation
- Companies may downplay shortcomings or shift blame
- Why do gaps occur?
- Many human actors involved across AI lifecycle
- AI exhibits autonomous behaviors
- Lack of clear regulations and responsibility frameworks
- Humans deliberately or unintentionally avoid accountability
5. Case Studies
- 2018 Uber Self-Driving Car
- Computer vision system struggled to classify pedestrian
- Relied on vision estimations instead of LIDAR
- Failed in different TAIHA layers than other cases
- Other cases: TuSimple, Cruise
- Each failed in different TAIHA layers
- Common pattern: insufficient controlled testing before public deployment
- Organizational influences contributed
6. Solutions & Recommendations
- Who should be accountable?
- Senior lead test engineers as new "first-line actors"
- Engineers must certify system safety before deployment
- Increased regulatory involvement needed
- Key recommendations
- Strong testing, verification and validation before deployment
- Identify humans accountable across the chain
- AI systems must be updated and retrained when conditions change
- Public trust in self-driving cars is decreasing
7. Model Drift
- What is model drift?
- Relationship between input data and outputs changes over time
- AI models cannot generalize to new environments
- Performance degrades when conditions change
- Why maintenance matters
- Continuous updating and retraining on new data
- Environments and conditions evolve constantly
- Inadequate maintenance is a key TAIHA layer