When Justice Is Blind to Algorithms: Multilayered Blackboxing of Algorithmic Decision-Making in the Public Sector
Card Grid View — Paper by Kronblad et al. (2024)
1. Case Study: Gothenburg Schools
- What happened?
- School placement algorithm placed children far from home
- Some children assigned to schools very hard to reach
- Algorithm was simple rule-based, not complex AI
- The problem
- Specific technical choice led to unfair placements
- System ran for years without correction
- Even after audit confirmed algorithm was illegal
- Court outcome
- Parents could not win the lawsuit
- Algorithm was too opaque to challenge legally
2. Multilayered Blackboxing
- Three layers of blackboxing
- 1. Technical Blackboxing
- Algorithm internals are opaque, hard to understand
- Even simple algorithms become blackboxed
- 2. Intra-Organizational Blackboxing
- Knowledge silos within the organization
- No one person understands the full system
- 3. Extra-Organizational Blackboxing
- External parties (courts, public) cannot access or understand
- Legal system unable to scrutinize effectively
3. Three Ignoring Practices
- 1. Obscuring
- Actively hiding or confusing how system works
- Reframing problems as technical rather than social
- 2. Avoiding
- Refraining from addressing early warnings
- Delaying action, hoping problems resolve
- 3. Denying
- Refusing to acknowledge systemic failures
- Claiming algorithm is neutral/objective
- These practices drive and reinforce blackboxing
4. Legal vs Social Justice
- Legal Justice
- Formal legal process, court decisions
- Limited by what can be proven in court
- Social Justice
- Broader fairness and equity outcomes
- Algorithmic harm affects vulnerable groups
- The gap
- Algorithm can be technically legal but socially unjust
- Legal system failed to deliver social justice
- Court could not assess algorithm adequately
5. Why Blackboxing Persists
- Organizational factors
- Public institutions lack incentives to scrutinize
- No one held personally responsible
- Responsibility diffused across many actors
- Systemic issues
- ADM systems create opacity by design
- Institutional blindness to algorithmic errors
- Misrepresentation: vendors oversimplify capabilities
6. Effects of ADM in Public Sector
- Negative impacts observed in
- Consumers, job applicants, patients
- (Environment is the exception)
- Why public sector is different
- Citizens cannot choose alternative provider
- Due process and transparency are legal requirements
- Algorithmic errors have direct human impact
- Examples of harm
- Welfare benefits wrongly denied
- School placements unfair
- Discrimination in hiring
7. Solutions & Recommendations
- What can help?
- Ensure relevant actors pay full attention to ADM errors
- Increase transparency in algorithmic decisions
- Assign clear responsibility for outcomes
- Practical mechanism proposed
- Help vulnerable groups contest algorithmic decisions
- Independent auditing of ADM systems
- Guide blackboxing layers toward explainability
- Key message
- Blackboxing is not just technical — it is organizational and institutional
- Solutions must address all three layers