Logs, Metrics, and Traces: The Three Ways to Know Why Production Broke

Logs tell you what happened in granular, timestamped detail; metrics tell you how much or how often something is happening in aggregate; traces tell you where in the system a request slowed down or failed. During an incident, each answers a different question — and knowing which one to ask for is often the fastest way a PM adds real value in the room.

Quick Answer: Logs = what happened (event detail). Metrics = how much/how often (trends and thresholds). Traces = the path a request took across services (where it broke). Ask for metrics first to confirm scope, logs to understand the specific failure, and traces when the failure crosses service boundaries.

Why "the three pillars" became the standard PM-adjacent vocabulary

Observability engineering converged on logs, metrics, and traces as the three foundational signal types because each captures a different shape of data, and none substitutes for the other two. The framing was popularized by distributed-tracing pioneers and cemented in Google's Site Reliability Engineering (SRE) book and the OpenTelemetry project's own documentation.

The core idea PMs need to internalize: you can't improve reliability you can't observe. A system can be failing right now in a way that no dashboard you've ever looked at would surface, simply because nobody instrumented that path. This isn't a hypothetical — it's the default state of most systems until someone deliberately builds the observability in.

• Logs are discrete, timestamped records of events — a request came in, a payment failed, a user logged out.
• Metrics are numeric measurements aggregated over time — requests per second, error rate, p99 latency.
• Traces stitch together the individual steps (spans) a single request took as it moved through multiple services.

None of these three is "the monitoring tool." They're complementary lenses, and most incidents require moving between all three in sequence, not picking one and stopping.

Where this fits your broader technical vocabulary

If observability is new territory, it sits inside the same "just enough engineering fluency to ask good questions" skill set covered in a complete guide to technical foundations for product managers and the mental model for how the web actually works. Observability is what happens after a request leaves the browser and travels through the services those two pieces describe.

Logs: the "what happened" layer, one event at a time

Logs answer "what specific event occurred, and what was its exact context?" — they're the closest thing to a system's diary, recording individual occurrences with timestamps, severity levels, and often a stack trace or error message. When an incident is localized to one service, logs are usually where the root cause is actually visible.

Logs come in a few recognizable flavors:

1. Application logs — code-level events: exceptions, warnings, custom debug statements.
2. Access/request logs — every HTTP request hitting a server, with status code and latency.
3. System logs — OS or infrastructure-level events (disk full, process restarted, out-of-memory kill).
4. Audit logs — who did what, when — often the ones compliance and security teams care about most.

The PM-level question during an incident: "What does the error message actually say, and does it point at a specific dependency or bad input?" Logs are verbose and unaggregated by design — searching them (via tools like Elasticsearch, Splunk, or Datadog Log Management) is how an engineer goes from "something is wrong" to "this specific null pointer exception fired 4,000 times starting at 2:14pm."

A useful mental model: logs are evidence, not verdicts. A single alarming log line rarely tells you scope or impact — that's the metrics layer's job.

Metrics: the "how much, how often" layer that tells you if it's actually bad

Metrics answer "is this happening a lot, is it getting worse, and has it crossed a threshold that matters?" — they're numeric time-series data, cheap to store and query at scale, which is exactly why they power the dashboards and alerts a PM sees first when something breaks.

Three metric categories cover almost every incident conversation:

The percentile distinction matters more than most PMs realize. Average latency can look fine while p99 latency is catastrophic — the average is dragged down by the 99% of requests going smoothly, hiding that 1% of your users are having a terrible time. This is why SRE practice (as Google's SRE book popularized) leans heavily on percentiles and Service Level Objectives (SLOs) rather than averages.

The PM question that separates a blip from a crisis

Ask: "What's the trend over the last hour compared to the last week, and what threshold did we cross?" Metrics are what let a team distinguish normal noise from a genuine regression — and they're the layer where "healthy" gets defined before anything breaks, which is the connective thread to the next section.

Traces: the "where in the system" layer for anything crossing service boundaries

Traces answer "which service, in a chain of services, is actually where the time went or the failure occurred?" — a trace follows one request end-to-end across every microservice, database call, and external API it touched, breaking the total time into individual spans.

Modern systems are rarely a single monolith; a single checkout request might touch an auth service, an inventory service, a payments API, and a notifications queue. Without tracing, "checkout is slow" is a mystery. With it (via OpenTelemetry, Jaeger, or Datadog APM), you can see that 1.8 seconds of a 2-second request was spent waiting on a third-party payments call.

The PM-level question during an incident: "Is this contained to one service, or is it a cascading failure across several?" Traces are what let engineers answer that in minutes instead of hours of manual log correlation across teams.

How the three pillars work together in a real incident

A realistic sequence, not a hypothetical:

1. A metric alert fires: error rate crossed 5% on the checkout endpoint.
2. The team checks traces to see whether the errors are isolated to one downstream service or spread across several.
3. Traces point to the payments service; the team pulls logs from that specific service in that time window.
4. Logs reveal the exact exception — a timeout connecting to a third-party payment processor.

Each layer narrowed the search. Skipping any one of them — jumping straight to logs without knowing scope, or reading metrics without ever confirming where the failure lived — tends to waste the most expensive resource in an incident: engineering attention under time pressure.

Defining "healthy" before launch is the actual PM job here

The PM's highest-leverage observability work happens before an incident, not during one: defining what "healthy" means for a feature so metrics and alerts have something meaningful to compare against. Without a pre-agreed baseline, every incident conversation starts with an argument about whether the numbers are even bad.

This is where Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets — concepts formalized across Google's SRE literature — become PM tools, not just an SRE's:

• SLI — the specific metric you're measuring (e.g., "percentage of requests under 300ms").
• SLO — the target for that metric (e.g., "99.5% of requests under 300ms over 30 days").
• Error budget — the acceptable amount of failure implied by the SLO, which becomes a shared currency between product velocity and reliability work.

A PM who shows up to a launch review and asks "what's our SLO for this, and who gets paged if we breach it?" is doing observability work — long before any dashboard exists. This is directly analogous to defining the acceptance criteria and hand-off conditions covered when scoping any feature spec, and it's a natural extension of understanding what an API contract actually promises, as laid out in what an API actually is, explained for product managers — an SLO is essentially a reliability contract the same way an API is a functional one.

A quick comparison table for the incident room

Use this table as a triage script: metrics first to confirm it's real, traces to localize it, logs to explain it. The order matters — starting with logs on a system-wide outage means reading thousands of irrelevant lines before finding the one that matters.

What deferred instrumentation costs a roadmap later

Skipping observability instrumentation at build time is a specific, common flavor of technical debt — it doesn't break anything today, but it means every future incident on that feature takes longer to diagnose, and every future reliability conversation runs on guesswork instead of data. This is the same tradeoff pattern explored in technical debt explained for your CEO: the cost doesn't disappear, it moves downstream and compounds with interest.

Treat "add basic logging, a health metric, and trace context" as a launch-readiness checklist item, not an optional nice-to-have engineers add if they have time. It's cheaper by an order of magnitude to add during the initial build than to retrofit onto a live system mid-incident, when nobody has the luxury of a calm deploy.

Turning an incident into a product decision, not just a postmortem

The hardest part of any incident isn't diagnosing it — it's making sure the lesson actually changes what the team builds next, rather than evaporating once the fire is out. Prodinja's Journals are designed to make that capture frictionless: after an incident review, you can log a Reflection or Assumption entry — even by voice — directly tied to the situation, so the "we should have had an SLO for this" insight feeds back into product decisions instead of living only in a Slack thread nobody revisits.

That habit of capturing the lesson while it's fresh connects observability work to the broader discipline of tracking assumptions across a product's customer journey and the underlying jobs the product is hired to do — reliability incidents are, after all, moments where the job the customer hired the product for visibly failed.

Key Takeaways

• Logs, metrics, and traces answer different questions — what happened, how much/how often, and where in the system — and an incident usually requires moving through all three, not picking one.
• Metrics scope an incident fastest; start there to confirm severity and trend before digging into logs or traces.
• Percentiles beat averages for latency and error data — a healthy-looking average can hide a genuinely broken p99 experience.
• Traces are essential once a failure crosses service boundaries — without them, cross-service debugging becomes manual log correlation across teams.
• "Healthy" needs a pre-agreed definition (an SLI/SLO/error budget) before launch, or every incident starts with a debate about whether the numbers are even bad.
• Skipping instrumentation is technical debt — it costs nothing today and compounds every time a future incident on that feature takes longer to diagnose.
• Capturing the lesson matters as much as fixing the bug — a reflection logged right after an incident is far more likely to change the next roadmap decision than one recalled weeks later from memory.