Show Your Work: Building Explainability into AI-Assisted People Decisions

When an algorithm touches a promotion, a performance rating, or a flight-risk flag, "the model said so" is not an answer an employee will accept — and in a growing number of jurisdictions, it's not a legally sufficient one either. Explainability means giving people a plain-language reason tied to their own data, distinct from interpretability (how the model works internally) and contestability (whether they can challenge the outcome).

Quick Answer: Model interpretability, user-facing explainability, and contestability are three different design problems. PMs need reason codes tied to real feature contributions, a clear boundary between "AI decides" and "AI recommends, human decides," and a working appeal path — not just a disclaimer.

Interpretability, Explainability, and Contestability Are Three Different Jobs

Interpretability is a model property; explainability is a product experience; contestability is a process guarantee — and conflating them is the single most common design mistake in people-analytics tools. A model can be interpretable to a data scientist and still produce an explanation no employee can act on. Building all three requires three separate design decisions, not one "add an explanation" checkbox.

Interpretability asks whether a human can understand the model's mechanics well enough to audit them: is it a decision tree someone can trace, or a gradient-boosted ensemble that needs a post-hoc approximation like SHAP or LIME to summarize? Researcher Cynthia Rudin has argued, notably in her widely cited Nature Machine Intelligence work, that for high-stakes decisions affecting people's lives, teams should default to inherently interpretable models rather than explaining black boxes after the fact — because post-hoc explanations can be plausible without being faithful to what the model actually did.

Explainability, by contrast, is what the employee sees: a reason, in their language, tied to their own record. It doesn't require exposing model internals — it requires translating them. A flight-risk score's SHAP values might say tenure_gap, manager_change_count, and comp_percentile are the top three contributors; the explainability layer turns that into "no promotion in 30 months, two manager changes in the last year, and pay below your peer band are driving this score."

Contestability is the process question: can the employee actually do something with that explanation? This is where most tools stop short. A reason code with no appeal path is transparency theater — it satisfies an audit checkbox without giving the person any recourse. The three layers compose: interpretability makes an honest explanation possible, explainability makes it legible, and contestability makes it matter.

The Right to an Explanation Is Already Showing Up in Regulation

Regulators increasingly require that automated decisions affecting employment be explainable and challengeable, not just accurate. The EU's GDPR Article 22 gives individuals the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects, plus a right to obtain human intervention and contest the decision. The EU AI Act goes further by classifying most HR and employment-decision AI systems as "high-risk," triggering documentation, human-oversight, and transparency obligations before deployment.

In the US, there's no single federal statute mirroring GDPR Article 22, but the pressure is converging from multiple directions:

1. EEOC guidance has clarified that employers remain liable for discriminatory outcomes produced by AI hiring or evaluation tools, regardless of whether a vendor built the model.
2. State and local laws — New York City's Local Law 144 being the most cited example — require bias audits of automated employment-decision tools and disclosure to candidates.
3. NIST's AI Risk Management Framework treats explainability and "human alongside the loop" oversight as core trustworthiness characteristics, not optional add-ons, for consequential AI systems.

None of this requires a PM to become a lawyer. It does mean explainability and contestability belong in the requirements doc next to accuracy and latency, not in a post-launch compliance retrofit. Teams building AI-hiring or screening tools should also read up on the broader fairness landscape — our guide to AI hiring fairness and bias regulation covers the adjacent obligations around disparate impact testing that often travel with explainability requirements.

The pattern across every regulation worth tracking is the same: automated influence on employment is fine; automated finality without recourse is the thing being restricted.

A Framework: "AI Decides" vs. "AI Recommends, Human Decides"

The single highest-leverage design decision a PM makes on any people-analytics feature is drawing the line between AI-decides and AI-recommends-human-decides — and defaulting to the latter for anything with material employment consequences. This isn't a compliance nicety; it changes what UI, logging, and appeal infrastructure you need to build.

AI Decides means the system's output is the decision with no required human review — appropriate for low-stakes, high-volume, easily reversible actions. Auto-flagging a scheduling conflict, suggesting a training module, or ranking a candidate list for a recruiter's own review (where the recruiter still chooses) can live here.

AI Recommends, Human Decides means a human accountable person reviews the AI's output, has visibility into the reasoning, and has explicit authority to deviate — with that deviation logged and treated as legitimate, not as an error to be corrected next release. Promotion eligibility, performance ratings, flight-risk-triggered retention interventions, and anything touching pay or termination should default here.

The test for which bucket a feature belongs in is reversibility times consequence. Ask: if the model is wrong, how hard is it to undo, and how much does the person lose while it's wrong? A course suggestion costs nothing if ignored. A flight-risk flag that triggers a manager conversation, or worse, informs a stack-rank, can shape someone's career trajectory even if later reversed. When in doubt, push the feature down a row — the cost of over-involving a human is friction; the cost of under-involving one is a decision nobody can defend. This is the same logic that should inform how you design for both manager and employee needs on the same feature: the manager needs the reasoning to act responsibly, and the employee needs it to trust the outcome.

Designing Reason Codes That Actually Explain

A good reason code names the top 2-4 factors that moved the score, in the employee's own data and language, ranked by contribution — not a generic list of "factors considered." This is the single artifact that turns a black-box number into something a manager can discuss and an employee can respond to.

Three properties separate a reason code that works from one that's decorative:

• Specific, not categorical. "Compensation" is categorical; "your pay is 8 percentile points below your peer band" is specific. Specificity is what makes a reason code actionable — the employee (or their manager) knows exactly what changed would move the score.
• Ranked by actual contribution, not alphabetical or arbitrary order. If the underlying model supports SHAP, LIME, or comparable attribution methods, the reason code's ordering should reflect the model's real feature weights for that individual, not a static global importance ranking that ignores their specific case.
• Stable under small, irrelevant changes. If correcting a typo in someone's job title flips their top reason code, the explanation layer is unreliable — a known failure mode of some post-hoc attribution methods that researchers including Rudin have flagged as a reason to prefer inherently interpretable models for consequential decisions wherever performance allows it.

A Flight-Risk Score Example

Consider a flight-risk model that outputs a 0-100 score per employee, feeding a manager dashboard. A bad implementation just shows the number. A good one surfaces:

1. The score and its trend — "72/100, up from 54 three months ago."
2. Top three drivers, in plain language — "No lateral or upward move in 28 months (highest weight); two skip-level manager changes in the past year; team's voluntary attrition is 40% above the org average."
3. What the model does not know — an explicit disclosure that the score doesn't see recent private conversations, health situations, or informal retention commitments, so the manager knows its blind spots.
4. A confidence indicator — flagging when the score is based on thin data (a recent hire, a role with little historical comparison data) so the manager weighs it accordingly.

This is also where the "AI recommends" framing earns its keep: the dashboard should make it structurally obvious that the score is an input to a conversation, not a verdict — through UI hierarchy (the score is not the biggest, boldest element on the screen) and through workflow (the manager must record an action or a rationale for inaction, closing the loop rather than letting the flag silently expire).

Building the Appeal and Override Flow

A contestability flow needs three things to be real rather than symbolic: a channel the employee actually knows exists, a reviewer with genuine authority to change the outcome, and a bounded response time. Absent any one of the three, "you can appeal" is a line in a privacy policy, not a functioning process.

A minimally credible flow looks like this:

• Disclosure at the moment of impact — the employee is told a score or flag influenced a specific action (a PIP, a passed-over promotion, a retention outreach), not buried in an annual privacy notice they never read.
• A named path to ask "why," and to ask for reconsideration — routed to a person with actual discretion (typically HR business partner or the manager, not a support queue), not a chatbot that restates the reason code.
• A record of the override, including who made it and why, so the pattern of overrides itself becomes a feedback signal — if managers are overriding the same reason code repeatedly, that's a data point that the model or its explanation is missing something real.
• A time bound — a promotion-readiness dispute reviewed six months later, after the promotion cycle has closed, isn't a functioning remedy regardless of how thorough it eventually is.

The override log deserves particular product attention: it's simultaneously a fairness safeguard and a source of ground truth for improving the model, since human overrides that cluster around a specific reason code or demographic pattern are an early warning that the explanation — or the model behind it — is missing context. Treat override rate and override reason as first-class product metrics, tracked the same way you'd track adoption or accuracy, and reviewed on the same cadence as any other trust metric — our piece on employee trust as an HRTech metric covers how to instrument trust signals like this so they don't stay anecdotal.

Making Explainability an Acceptance Criterion, Not an Afterthought

The reason explainability so often ships late or thin is that it's treated as a UI polish pass after the model and workflow are already locked, instead of a requirement negotiated before engineering starts building. By the time a team notices the reason codes are generic or the override path doesn't exist, the data pipeline may not even be capturing the feature attributions needed to fix it cheaply.

The fix is procedural: write explainability and contestability into the spec as testable acceptance criteria, the same way you'd write a latency budget or an accuracy threshold. "Each flight-risk score must surface its top 3 contributing factors by weight, refresh reason codes when underlying data changes, and route every override through a logged review" is a criterion an engineer can build against and a reviewer can verify — "make it explainable" is not.

This is the specific gap Prodinja's Spec Studio is built to close for teams sketching out this kind of feature: its readiness gates let you flag explainability and contestability as required acceptance criteria on a PRD before it's handed to engineering, alongside the living document's PR-style diffs, so a reviewer can see exactly when a reason-code requirement or an appeal-flow requirement was added, weakened, or dropped — rather than discovering the gap in a post-launch audit. It doesn't generate the explanations for you; it's designed to make sure the requirement doesn't quietly disappear between the whiteboard and the sprint board.

Whether or not you use a specific tool for it, the underlying discipline transfers: treat contestability the way you'd treat any other non-functional requirement a customer will eventually test you on. Framing this well often benefits from thinking in jobs-to-be-done terms — what job is the employee actually hiring the appeal flow to do — which our complete guide to Jobs to Be Done walks through in more depth, and from mapping where explanation and trust moments sit along the employee journey rather than treating them as an isolated feature.

Key Takeaways

• Interpretability, explainability, and contestability are separate design problems — a model can be interpretable without being explainable to an employee, and explainable without being contestable.
• Default to "AI recommends, human decides" for anything touching pay, promotion, or termination; reserve full automation for low-stakes, easily reversible actions.
• Reason codes need specificity and honest ranking — name the top 2-4 factors by actual contribution, in the employee's own data, and disclose what the model doesn't see.
• A contestability flow only counts if it has a known channel, a reviewer with real authority, and a bounded response time — anything less is transparency theater.
• Override logs are a fairness signal, not just an escape hatch — clustering overrides around a reason code or group is an early warning worth tracking as its own metric.
• Regulation is already converging on this pattern — GDPR Article 22, the EU AI Act's high-risk classification, NYC Local Law 144, and EEOC guidance all point toward the same requirement: automated influence is fine, automated finality without recourse is not.
• Write explainability and contestability into the spec as acceptance criteria before engineering starts, not as a post-launch fix, so the requirement is testable rather than aspirational.