Decision Journals: How Great Leaders Calibrate Their Own Judgment

Great leaders calibrate judgment by keeping a decision journal: before any outcome is known, they log the decision, their confidence as a number, and the reasoning behind it. Weeks or months later, they score the call against reality using a Brier-score mindset. Over dozens of entries, this converts scattered hindsight into a measurable, improvable skill.

Quick answer: A decision journal captures the call, your confidence percentage, and your reasoning at the moment of decision — not after. Review entries later, score them the way a forecaster does, and you build calibrated judgment instead of outcome-driven guesswork.

Most leadership advice treats decision-making as a personality trait: you're either "decisive" or you're not. That framing is useless because it gives you nothing to practice. Calibration is a skill, not a trait — and like any skill, it needs a record to improve against. That's the entire case for a decision journal.

Why Judging Decisions by Their Outcomes Wrecks Your Judgment

Leaders instinctively grade decisions by what happened afterward, but a good decision can produce a bad outcome, and a reckless one can get lucky. Poker player and author Annie Duke calls this the resulting fallacy: confusing the quality of a decision with the quality of its result. Untangling the two is the entire point of a decision journal.

Duke's background is instructive. In poker, a player can shove all-in with the statistically correct hand and still lose to a bad beat — the decision was right, the card was wrong. Product leadership works the same way, except the feedback is slower and the noise is louder:

• Good decision, good outcome — the easy case. Don't let it inflate your confidence more than the process actually deserves.
• Good decision, bad outcome — variance. Log it as a good decision anyway. Resisting the urge to rewrite history here is the whole discipline.
• Bad decision, good outcome — the dangerous quadrant. A lucky break that teaches you exactly the wrong lesson if you don't catch it.
• Bad decision, bad outcome — at least the record matches the process. Usually the easiest one to learn from.

The reason leaders default to outcome-grading isn't stupidity — it's hindsight bias. Once you know how something turned out, your brain quietly rewrites how "obvious" it was all along. Daniel Kahneman, along with Olivier Sibony and Cass Sunstein in their book Noise, documents how this same distortion shows up across professional judgment: doctors, judges, and underwriters all convince themselves, after the fact, that the result was foreseeable. A decision journal is the only defense, because it timestamps what you actually believed before you knew the ending.

This matters more the more senior you get. Directors and VPs make fewer decisions than individual PMs, but each one carries more weight and gets scrutinized more heavily after the fact — which means the incentive to quietly revise your own memory of what you believed only grows. If you're building out a broader leadership practice around this kind of self-awareness, the pm-leadership-complete-guide is a useful map of where calibration fits alongside the rest of the craft.

The Superforecasting Method: Predict, Log, Score, Repeat

Philip Tetlock's forecasting research, including the IARPA-funded Good Judgment Project, found that the best forecasters improve mainly by stating a numeric probability, writing down their reasoning, and later scoring the forecast against what actually happened. That predict-log-score loop, run over enough decisions, is what separates calibrated judgment from confident guessing.

The headline finding from that multi-year tournament, described in Tetlock and Dan Gardner's Superforecasting, is directional but striking: the top forecasters — ordinary volunteers, not intelligence professionals — outperformed the average participant by a wide margin, and reportedly beat intelligence analysts working with classified information by something on the order of 30 percent over the life of the tournament. The gap wasn't raw intelligence. It was method: probabilistic thinking, frequent updating, and relentless score-keeping.

You don't need a forecasting tournament to borrow the method. In practice, it's a four-step loop:

1. State the decision and a probability. Not "I feel good about this" — a number, like 70%.
2. Write one sentence of reasoning and one sentence of disconfirming evidence. Force yourself to name the strongest fact against your own call.
3. Set a review date. A prediction with no deadline never gets scored.
4. When the date arrives, score it — right, partly right, or wrong — and file it by the confidence you originally stated.

That last step is where most people quietly stop, because it's the one that can bruise the ego. It's also the only step that actually produces calibration.

Where the Brier score comes in

Meteorologist Glenn Brier introduced the Brier score in 1950 as a way to grade probabilistic weather forecasts, and it's still used by forecasting shops, including national weather services, to hold forecasters accountable to their stated confidence rather than just whether they were "right." The intuition matters more than the formula:

• If you say 90% confident and you're right, the gap between your stated confidence and reality is small — a small penalty.
• If you say 90% confident and you're wrong, the gap is huge — a large penalty, because you were emphatically, not just slightly, off.
• If you say 50-50 and either outcome occurs, the penalty sits in the middle either way.

The practical lesson for a leader: hedging at 50-50 on everything protects your ego but never earns you credit, while stating high confidence only pays off if you're actually right that often. A decision journal is what lets you find out whether your 90% calls really do land 90% of the time.

The Decision Journal Template: Fields That Force Real Calibration

A useful decision journal has a small, fixed set of fields you fill in at the moment of decision, before you know how it turns out: the call, the stakes, reversibility, your confidence as a percentage, your reasoning, disconfirming evidence you're ignoring, and a review date. Skip any of these and the entry stops teaching you anything.

The reversibility field borrows directly from Jeff Bezos's well-known distinction, laid out in Amazon's 2015 shareholder letter, between one-way doors (decisions that are costly or impossible to reverse) and two-way doors (decisions you can walk back cheaply). It's worth logging every entry, high stakes or low, but it's the one-way-door entries where calibration errors are most expensive — which is exactly why regulated or capital-intensive fields treat this discipline as non-negotiable. Multi-season planting-cycle bets in the kind of decisions covered in our agritech complete guide, or trial-gate calls of the sort discussed in our biotech and pharma product guide, can cost a full year if the confidence behind them was never actually tested.

The disconfirming evidence field is the one people skip, and it's the one that matters most. Kahneman, Sibony, and Sunstein's decision-hygiene research recommends actively arguing against your own conclusion before committing — sometimes called a premortem — precisely because unopposed reasoning tends to just restate your priors in more confident language.

A single prompt catches more bad calls than almost anything else in this template: what would have to be true for this decision to be wrong — and is any of it already happening?

Scoring Calibration Without a Statistics Degree

You don't need a Brier score formula to see whether you're calibrated — you need buckets. Group your resolved decisions by the confidence you originally stated (50-60%, 70-80%, and so on), then check what fraction actually turned out right in each bucket. If your 90%-confidence calls are only right 60% of the time, you're overconfident, not unlucky.

Here's what that looks like with a modest set of resolved entries, the kind a director might accumulate over two or three quarters:

Read that table honestly and a pattern jumps out immediately: the leader's top-confidence bucket is the least reliable one. That's an extremely common finding in calibration research generally, and it's the single most useful thing a decision journal can show you about yourself, because it's nearly impossible to see from memory alone.

Two caveats keep this exercise honest:

• Small samples lie. Eight decisions in a bucket isn't enough to draw a firm conclusion — treat early calibration tables as a hypothesis, not a verdict, and let the pattern build over dozens of entries.
• "Partly right" needs a rule. Decide in advance whether a partial outcome counts as half a point or a judgment call at review time — otherwise you'll unconsciously round in your own favor.

The goal was never a perfect score. It's a trend line: is the gap between your stated confidence and your actual hit rate shrinking quarter over quarter? That trend is what "tracking decision confidence" actually buys you — not a grade, a feedback loop.

Turning Entries Into a Compounding Judgment Engine

A decision journal only compounds if you actually reopen it — a monthly or quarterly review where you resolve open entries, score them, and look for repeat patterns across categories, stakeholders, or decision types. Without that ritual, the journal is just an expensive diary; with it, it becomes a feedback loop your instincts alone can't provide.

Ray Dalio's Principles describes a related habit at Bridgewater: tracking people's (including your own) track record on specific types of calls, then weighting future input by that demonstrated record rather than by seniority or confidence of delivery. You can run a personal version of this by tagging journal entries by category — hiring, pricing, competitive timing, scope trade-offs — and reviewing each category's calibration separately instead of lumping everything into one score.

Patterns worth watching for on review day:

1. Category bias — you might be well-calibrated on scope decisions but consistently overconfident about competitor timing.
2. Stakeholder-driven bias — decisions made under pressure from a specific stakeholder relationship tend to skew in a predictable direction; our stakeholder politics guide covers how political pressure distorts judgment in the moment, which is exactly what a dated, pre-outcome journal entry can catch after the fact.
3. Style-driven bias — a leader operating in a more directive or pace-setting mode, in Daniel Goleman's terms, tends to make faster, higher-confidence calls than one operating in a coaching or affiliative mode; our breakdown of Goleman's six leadership styles is worth pairing with your journal review if your calibration seems to shift with your mood or team context.

The review ritual itself is a specific, structured instance of a broader habit worth building deliberately — see our guide to the PM craft of reflection for how to build the review cadence into a sustainable practice rather than a New Year's resolution that fades by March.

Where Software Can Help You Keep the Habit

A decision journal is a discipline, not a piece of software, but a dedicated tool removes the friction that kills most journaling habits — the extra tab, the forgotten spreadsheet, the entry you never revisit. What matters is that logging confidence and reasoning takes under a minute, and reviewing it later is just as fast.

Prodinja's Decision Journal is built for exactly this: it lets you log a decision, your confidence level, and your rationale so you can revisit calibration once reality lands. It's part of Prodinja's Leadership Suite, currently shipping as an interactive prototype, and it walks through the same fields covered above — stakes, reversibility (one-way vs. two-way door), category, and a review date — then lets you resolve each entry as right, partly right, or wrong. When you do, it plots your stated confidence against your actual hit rate side by side, so the gap between the two is something you can see rather than something you have to remember to calculate.

The point isn't that software makes you a better forecaster. It's that a calibration habit only survives if logging and reviewing are both fast enough to actually happen — which is the design intent behind building it as a dedicated, lightweight journal rather than another field on a status report.

Key Takeaways

• A good decision can still produce a bad outcome, and a bad decision can still get lucky — score the process, not just the result, to avoid Annie Duke's resulting fallacy.
• Log the stakes, reversibility, confidence, reasoning, and disconfirming evidence before you know the outcome, not after — hindsight quietly rewrites your memory of what you believed.
• State confidence as a number, not a feeling. "70%" is falsifiable later; "pretty confident" never is.
• Group resolved decisions into confidence buckets and compare stated confidence to actual hit rate — the gap is your calibration signal, in the spirit of the Brier score.
• Review the journal on a fixed monthly or quarterly cadence; entries you never reopen don't teach you anything.
• Track calibration separately by category (hiring, pricing, competitive timing) — most leaders are well-calibrated in some areas and badly miscalibrated in others, and averaging hides the difference.
• Treat one-way-door decisions as the highest-priority entries; that's where a miscalibrated confidence number is most expensive to discover late.