Framework
Every feature fails in specific ways. Identify them upfront:
| Failure Mode | Severity | Likelihood | Detection | Mitigation |
|---|---|---|---|---|
| AI recommends inappropriate product | High | Medium | User reports, filtering | Add content moderation layer |
| Recommendations don't load (API down) | Medium | Low | Error monitoring | Show cached recommendations |
| Cold-start: no recommendations for new user | Medium | High | User feedback | Show popular items as fallback |
| Stale data: recommendations outdated | Low | Medium | Compare to real-time | Refresh cached recs every 6hr |
For each: Severity × Likelihood = Priority
Actionable Steps
1. List All Failure Modes
Think like an adversary. What could go wrong?
2. Score by Severity × Likelihood
Only high-priority failures get mitigations.
3. Plan Detection + Rollback
How will you know it's happening? How do you disable it?
Key Takeaways
- Failure modes are inevitable. Having a plan beats being surprised.
- Priority = Severity × Likelihood. Don't over-mitigate low-priority failures.