Your dashboard number is late or wrong more often than a raw SELECT COUNT(*) should ever be, because between the click and the chart sits a supply chain: capture, transport, transformation, and load — each step with its own failure modes. Understanding that supply chain is how you tell your data team "the signup number looks off" instead of just feeling it.
Quick Answer: A dashboard metric is the output of a pipeline — event capture → transport → transformation (ETL/ELT) → warehouse → BI tool — and most "wrong number" incidents trace to a break in one of those links, not a bug in the chart itself.
What Actually Happens Between a Click and a KPI
A single user action generates a raw event — a timestamped record like signup_completed with a user ID and properties — that travels through several systems before it becomes a number on your screen. Each hop can introduce delay, duplication, or reinterpretation. Treat the number as a manufactured product, not a fact.
The canonical path looks like this:
- Capture — a client (web/app) or server emits an event via a tracking library or webhook.
- Ingestion — an event collector or queue (like Segment, Kafka, or a cloud pub/sub) receives and buffers it.
- Transformation — raw events are cleaned, joined, and reshaped into analysis-ready tables.
- Load/storage — the transformed data lands in a warehouse (Snowflake, BigQuery, Redshift).
- Serving — a BI tool (Looker, Tableau, Metabase) queries the warehouse and renders your chart.
If you've internalized how requests move across the internet in general, the web mental model for PMs is a useful prerequisite — pipelines are just a specialized, batch-and-stream version of that same client-server dance.
Why "The Data Is Wrong" Is Rarely One Bug
When a number looks off, the instinct is to blame "the dashboard." In practice, the defect almost always lives upstream — in a schema change, a join, or a timing assumption — and the dashboard is just faithfully rendering bad input. Knowing the pipeline stages lets you ask a sharper first question: "Which stage?" rather than "Why is this broken?"
ETL vs. ELT: The Order of Operations Matters
ETL (Extract, Transform, Load) transforms data before it reaches the warehouse; ELT (Extract, Load, Transform) loads raw data first and transforms it inside the warehouse using SQL or a tool like dbt. Most modern stacks default to ELT because cloud warehouses are now cheap and fast enough to do the heavy lifting themselves.
| Dimension | ETL (transform before load) | ELT (transform after load) |
|---|---|---|
| Where transformation happens | Separate processing layer (e.g., Spark, custom scripts) | Inside the warehouse (SQL, dbt) |
| Raw data retained? | Often discarded or archived separately | Yes — raw layer always queryable |
| Debuggability | Harder — must reproduce the transform job | Easier — can re-run SQL against raw rows |
| Typical latency | Batch, often hourly/daily | Can be near-real-time with streaming loads |
| Best fit | Legacy on-prem systems, strict compliance transforms | Cloud-native teams, fast-iterating product analytics |
For a PM, the practical takeaway is: ELT means you can usually ask "show me the raw event" and get an answer, which is invaluable when a metric is disputed. In pure ETL setups, the raw event may already be gone by the time you notice a discrepancy.
Streaming vs. Batch
Some pipelines process events continuously (streaming, via tools like Kafka or Kinesis); others run on a schedule (batch, hourly or nightly). Streaming gets you near-live dashboards but is harder to reconcile and re-run. Batch is simpler to reason about but means your "live" metric might be several hours stale — worth knowing before you page someone about a flat line.
Your Metrics Are a Product With a Supply Chain
Every KPI on your dashboard has an implicit bill of materials: a definition of what counts as an event, a set of transformation rules, and a delivery schedule. If any ingredient changes upstream without notice, the finished product — your chart — silently changes shape.
This reframe matters because PMs already know how to manage a supply chain: define the spec, watch the handoffs, and instrument for defects. The same discipline applies to a metrics pipeline. Three failure points account for most "why is this number wrong" incidents:
Schema Drift
Schema drift happens when an upstream event's structure changes — a field renamed, a type changed from string to integer, a new required property added — and downstream transformations silently break or misinterpret it. Engineering ships a mobile app update, renames plan_type to plan_tier, and every dashboard segmenting by plan quietly goes to null.
- Drift is rarely announced; it's discovered when a metric drops to zero or a
NULLbucket balloons. - Contracts between event producers and consumers (schema registries, versioned event specs) reduce this, but few product orgs enforce them rigorously.
- This is exactly the problem entity modeling is meant to prevent — more on that below.
Late-Arriving Data
Late data is any event that arrives at the warehouse after the reporting window it belongs to has already been calculated and shown on a dashboard. A user's mobile app queues events offline and syncs them two days later; your "yesterday's signups" number was already final by the time they land.
This is why "today's" numbers on almost any real-time dashboard should be read as provisional, not final, until at least one full reconciliation cycle has passed.
Common causes: offline mobile clients, retried failed API calls, cross-timezone batch jobs, and third-party webhook delays (payment processors, ad platforms). The fix isn't eliminating lateness — it's rarely possible — but building dashboards that visibly flag "still updating" windows.
Duplicate Events and the Case for Idempotency
Deduplication failures happen when the same real-world action generates more than one event record, inflating counts. This is the single most common cause of an inflated signup, purchase, or activation number — and it is entirely preventable with the right engineering pattern.
Worked Example: The Double-Counted Signup
Imagine your signup funnel fires a signup_completed event from the client the moment the confirmation screen renders. A user on a flaky connection taps "Create Account," the request appears to hang, they tap again, and the server actually completes both requests — because there was no protection against it. Two rows land in the events table for one human being.
Without deduplication logic, your dashboard reports 1,240 signups when only 1,180 people actually signed up — a 5% overstatement that quietly inflates every downstream metric: activation rate, CAC, and conversion all look better than reality.
The fix engineers reach for is idempotency: designing an operation so that performing it multiple times has the same effect as performing it once. In practice:
- The client generates a unique
idempotency_key(often a UUID) at the moment the user initiates the action, not when it succeeds. - The server checks whether it has already processed that key before creating a new signup record or emitting a new event.
- Duplicate requests with the same key are safely ignored or return the original result — no second row, no second event.
| Approach | What it catches | What it misses |
|---|---|---|
| Client-side "disable button after click" | Accidental double-taps on the same session | Retries after network timeout, app crash/restart |
| Server-side idempotency key | True duplicate requests regardless of client state | Duplicate events from genuinely separate systems (e.g., web + mobile both firing) |
Warehouse-layer dedup (dbt model on event ID) | Any duplicate that made it into raw storage | Nothing upstream — this is a safety net, not a prevention |
| Definition-level dedup (one signup per user ID, ever) | Multiple signup attempts from the same person | Legitimate re-signups after account deletion, if not modeled explicitly |
The deeper lesson: the definition of "a signup" has to be decided once, explicitly, and enforced everywhere — not re-derived ad hoc in every dashboard query. This is a naming and modeling problem as much as an engineering one, and it's exactly the kind of ambiguity that festers when nobody owns the event schema.
Common Failure Points and Who Should Own the Fix
Different pipeline failures need different owners, and knowing which is which saves you from filing the wrong ticket.
| Failure point | Symptom on your dashboard | Typical owner |
|---|---|---|
| Schema drift | Metric drops to zero or spikes in "unknown" bucket | Whoever owns the event schema (often product + data eng jointly) |
| Late-arriving data | Today's number keeps changing after the day "ends" | Data engineering (reconciliation/backfill jobs) |
| Duplicate events | Numbers run higher than manual spot-checks | Backend engineering (idempotency) + data eng (dedup logic) |
| Broken join/transformation | A specific segment or breakdown looks wrong, totals look fine | Analytics engineering (dbt model owner) |
| Timezone mismatch | Off-by-one-day discrepancies between two reports | Whoever wrote the reporting query |
| Sampling in the tracking tool | Numbers roughly right but never exactly reproducible | The analytics vendor's sampling settings, not a "bug" |
When you report a suspicious number, naming the likely failure point (from this table) rather than just "this looks wrong" gets you a faster, more precise answer from engineering — the same way naming an API endpoint speeds up a bug report, as covered in what an API actually is for product managers.
Where Definitions Get Decided: Data Modeling
Most of the failure points above trace back to one root cause: nobody defined the entity — "what is a signup," "what is an active user," "what is a completed purchase" — clearly enough, early enough, for everyone downstream to agree on it. Data modeling is the discipline of defining those entities, their attributes, and their relationships before a single transformation is written.
Get the model right and schema drift has less surface area to bite, duplicate events have an obvious primary key to dedup against, and late data has a clear reconciliation rule. Get it wrong, and every downstream dbt model or dashboard query re-invents its own slightly-different definition — which is how two teams end up with two different "active user" counts in the same board deck.
Key Takeaways
- A dashboard number is the last stop in a pipeline, not a standalone fact — capture, transport, transformation, and load each add risk.
- ETL transforms before loading; ELT loads raw data first and transforms it inside the warehouse, which is now the more common — and more debuggable — pattern.
- Schema drift, late-arriving data, and duplicate events are the three most common reasons a metric looks wrong, and each has a different owner.
- Idempotency keys are the standard engineering fix for duplicate events like the double-counted-signup scenario — designing an action so repeating it has no extra effect.
- Treat "today's" real-time numbers as provisional until a full reconciliation window has passed; don't over-react to intraday swings.
- Clean entity definitions, decided once and modeled explicitly, prevent most of the downstream ambiguity that produces conflicting metrics.
- When a number looks wrong, naming the likely failure point speeds up the fix far more than reporting "the dashboard is broken."
Frequently Asked Questions
Why does my dashboard show a different number today than it did yesterday for the same date?
This is almost always late-arriving data: events from that date are still trickling in and being reconciled. Real-time and next-day numbers for the same date will rarely match exactly; treat the same-day figure as provisional and the next-day figure as closer to final.
What's the difference between ETL and ELT, in plain terms?
ETL cleans and reshapes data before it reaches the warehouse, while ELT loads raw data first and transforms it afterward using warehouse SQL. ELT is more common today because it keeps the raw event available for debugging, which ETL pipelines often don't.
Why would the same metric show different numbers in two different tools?
Two tools (say, a product analytics tool and your data warehouse) often apply different definitions, different sampling, or different timezones to "the same" metric. This is a definitions problem, not necessarily a bug — the fix is agreeing on one canonical definition and one source of truth.
How do I know if a data problem is a schema issue versus a duplicate-event issue?
Schema drift typically shows up as a metric dropping toward zero or values shifting into an "unknown" or null bucket. Duplicate events typically show up as a metric running consistently higher than a manual spot-check or a competing source would suggest.
Should product managers actually understand pipeline internals, or is this an engineering problem?
You don't need to write the dbt models yourself, but understanding the stages — capture, transport, transformation, load — lets you ask precise questions and read metric anomalies as pipeline symptoms rather than mysteries. It's the same value understanding technical debt gives you, as explored in technical debt explained for your CEO, or understanding customer behavior data gives you when mapping a complete customer journey.