Data downtime is any stretch where a data product is wrong, missing, or stale — a broken pipeline, a silent schema change, a dashboard nobody flagged — and it usually stays invisible until a downstream user trusts a bad number. Observability means continuously monitoring freshness, volume, schema, and distribution, then running a defined incident response the moment one breaks.
Quick Answer: Data downtime is the data-team equivalent of an outage: pipelines silently break, freshness and schema drift, and nobody notices until a report looks wrong. Fix it with the five pillars of data observability — freshness, volume, schema, distribution, and lineage — plus a severity-tiered incident runbook with named owners.
What "Data Downtime" Really Means
Data downtime describes any period a dataset is inaccurate, incomplete, or unavailable — the direct data-team analog of application downtime in software reliability. The term reframes data quality from an occasional cleanup chore into an operational discipline with the same urgency, tooling, and ownership expectations as infrastructure uptime.
Barr Moses, co-founder of Monte Carlo Data, popularized the phrase around 2019 to name a problem every data team already lived with but rarely measured. Before the term existed, there was no shared vocabulary for "the pipeline ran but the numbers are wrong." That gap is exactly why detection lagged so far behind software monitoring for a decade.
The "CEO Noticed" Failure Mode
The most common way data downtime gets discovered is the worst way: a stakeholder in a board meeting spots a number that doesn't look right, asks why, and someone scrambles to find out that a join broke three weeks ago. By the time a VP notices, the damage compounds — decisions were made, forecasts were shared, and trust in the entire dashboard is now suspect, not just the one broken metric.
This reactive pattern persists because most data teams still treat monitoring as a debugging tool, not a product requirement. Once a table feeds a dashboard someone checks weekly, that person is a user of your data product whether you designed for them or not — a mindset shift our guide to why data products have users covers in more depth. Users don't file tickets when something feels off; they just stop trusting you.
The cost of staying reactive isn't hypothetical. Gartner has cited poor data quality as costing organizations an average of roughly $12–13 million a year, and the analyst firm's broader research consistently ranks trust and reliability among the top blockers to getting more value out of analytics investments. That figure covers everything from bad marketing spend decisions to re-work, but a meaningful share of it traces back to exactly this pattern: a broken pipeline nobody caught until a decision was already made on top of it.
Borrowing the SRE Playbook for Data
Google's Site Reliability Engineering book, popularly known as the SRE handbook, established that reliability work needs explicit service-level objectives (SLOs), error budgets, and blameless postmortems rather than heroic firefighting. Data teams are roughly a decade behind on adopting the equivalent discipline for pipelines.
The translation is direct: an application's uptime SLO becomes a pipeline's freshness SLA; an error rate becomes a row-count anomaly rate; an on-call rotation for services becomes an on-call rotation for data products. If you're building out the broader data PM function, our data PM role complete guide walks through where this reliability ownership sits alongside roadmap and stakeholder work.
The Five Pillars of Data Observability
Data observability rests on five signal categories — freshness, volume, schema, distribution, and lineage — that together answer whether a dataset is trustworthy right now, not just whether the job that produced it exited with a success code. Monitoring all five closes the gap between "the pipeline ran" and "the data is correct."
This framing, widely credited to Monte Carlo and echoed across the data engineering community, gives teams a checklist instead of a vague mandate to "watch the data."
| Pillar | What it answers | Typical detection signal |
|---|---|---|
| Freshness | Is the data as recent as it should be? | Time since last successful load vs. expected cadence |
| Volume | Did the expected amount of data arrive? | Row count vs. historical baseline, ±threshold |
| Schema | Did the structure change unexpectedly? | Column added/removed/retyped without a migration |
| Distribution | Do the values still look statistically normal? | Null rate, mean/variance drift, category-cardinality shift |
| Lineage | What upstream/downstream tables does a failure touch? | Dependency graph mapping producers to consumers |
- Freshness monitoring flags a dbt job that silently stopped running, not just one that errored — a stalled cron job still "succeeds" if nothing downstream checks the timestamp.
- Volume monitoring catches the classic partial-load bug: a source API returns 200 OK with half the expected rows and no error is ever thrown.
- Schema monitoring is the one most teams skip, and it's the one that causes the worst incidents — a renamed or retyped column upstream breaks every downstream join without a single failed job.
- Distribution monitoring catches the subtlest failures: a currency field that quietly starts reporting cents instead of dollars still has the right row count and the right schema.
- Lineage isn't a monitoring signal by itself, but it's what turns a detected anomaly into an actionable alert — you can't triage a schema break if you don't know which six dashboards depend on that column.
Treating these five pillars as a checklist — rather than ad hoc dbt test assertions scattered across a repo — is what our companion piece on data quality as a feature argues for: quality monitoring belongs in the product spec, not as an afterthought bolted on post-incident.
Where DAMA's Quality Dimensions Fit
The DAMA DMBOK (Data Management Body of Knowledge), the closest thing the data management field has to an industry standard, defines a longer list of quality dimensions — accuracy, completeness, consistency, timeliness, validity, uniqueness. The five observability pillars aren't a replacement for that framework; they're the operational, always-on subset of it that's cheap enough to automate and check on every pipeline run.
Timeliness maps to freshness. Completeness maps to volume. Validity and consistency map to schema and distribution. What DAMA doesn't give you is lineage, because lineage is a data-engineering concern DAMA's data-management lens was never built to cover — which is exactly why it's the fifth pillar layered on top, not a rename of an existing one.
Building the Monitoring Stack: From Manual Checks to Proactive Detection
A proactive monitoring stack combines rules-based tests for known failure modes with statistical anomaly detection for the ones you haven't imagined yet, wired into alerting that reaches the right owner before a stakeholder does. Neither approach alone is sufficient — rules catch what you predicted; anomaly detection catches what you didn't.
Rules-Based Tests vs. Statistical Detection
Rules-based checks — dbt test, Great Expectations assertions, or a Soda scan — are cheap, explainable, and catch exactly the failure you wrote the rule for. Tools like Monte Carlo and Bigeye add ML-based anomaly detection on top, learning a baseline for freshness and volume per table and flagging deviations without a human writing a threshold for every column.
- Rules-based is best for known invariants: "order_id is never null," "revenue is never negative," "this table must land by 6am."
- Statistical/ML-based is best for the long tail: a slow drift in null rate, a seasonal pattern breaking, a category distribution shifting after an upstream vendor change.
- Neither replaces lineage-aware alert routing — an anomaly detected in a raw source table needs to notify every downstream owner, not just the table's own maintainer.
Setting Data SLAs and SLOs
A data SLA is the commitment you make to consumers ("the revenue table refreshes by 7am daily, 99% of business days"); the SLO is the internal target you monitor against before you breach that SLA. Borrow this distinction directly from SRE practice rather than reinventing it — the vocabulary already exists.
Set SLAs at the table level for anything with a named downstream consumer, not for every table in the warehouse — over-instrumenting creates alert fatigue that trains people to ignore pages. A metrics layer that consumers actually rely on, the subject of our piece on building a metrics layer the company can trust, only stays trustworthy if the tables feeding it carry real SLAs, not aspirational ones nobody enforces.
Severity Levels, Ownership, and the Incident Runbook
A data incident runbook needs a severity scale that maps directly to response time and communication requirements, plus a named owner for every stage from detection through postmortem. Without both, "everyone" ends up responsible, which in practice means no one is.
A Severity Framework Data Teams Can Adopt
| Severity | Definition | Response time | Who's paged |
|---|---|---|---|
| Sev1 | Exec-facing metric or revenue-critical pipeline is wrong or down | Immediate, 24/7 | On-call data engineer + data PM |
| Sev2 | A team-level dashboard or model input is broken | Within business hours | On-call data engineer |
| Sev3 | A non-critical table has quality issues, no active consumer impact yet | Next business day | Table owner |
| Sev4 | Cosmetic or low-confidence anomaly, needs investigation not action | Weekly triage | Data quality backlog owner |
Calibrate these tiers against your own dependency graph, not a generic template — a table feeding a finance close process is Sev1 in a way an internal experimentation dashboard never will be.
The Runbook: Six Stages, Named Owners
- Detect. An automated alert fires from a freshness, volume, schema, or distribution check — never a Slack message from a confused stakeholder. If detection is coming from a human, that's the gap to close first.
- Triage. The on-call engineer confirms the alert is real (not a flaky check), assigns a severity from the table above, and opens an incident channel.
- Communicate. For Sev1/Sev2, the data PM or on-call lead posts a status update to affected stakeholders within an agreed window — don't let downstream teams find out from a bad dashboard number.
- Mitigate. Stop the bleeding: freeze the downstream dashboard, roll back the offending deploy, or flag the table as unreliable in the catalog, even before root cause is known.
- Resolve. Fix the underlying pipeline, backfill affected data, and confirm downstream tables recompute correctly.
- Postmortem. A blameless writeup covering what broke, why detection was slow (or fast), and what monitoring gap gets closed — the SRE tradition Google's handbook made standard, applied to data.
Assigning Clear Ownership
Every table with a named consumer needs a documented owner responsible for its SLAs, not a rotating "whoever's on-call this week" default — on-call handles response, an owner handles prevention. Thinking about the on-call engineer's actual job to be done during an incident — diagnose fast, communicate honestly, prevent recurrence — is the same lens our jobs-to-be-done complete guide applies to product work generally; a runbook is really a JTBD map for the three-in-the-morning version of your team.
Designing Detection Where Failures Actually Propagate
Effective data incident response depends on understanding causal propagation, not just the point where a failure was detected: a silent schema change rarely stays contained, it cascades through joins, feeds a model, and surfaces as a wrong number in a dashboard someone trusts. Mapping that chain in advance is what turns "we got paged" into "we knew exactly where to look."
The Cascade Nobody Draws Until After the Incident
A renamed column doesn't just break one query. It silently nulls out a join, which quietly shrinks a feature used in a churn model, which nudges a forecast, which a VP repeats in an all-hands — each hop invisible individually, catastrophic in aggregate. Most postmortems draw this chain for the first time after the incident, when it's too late to have monitored the right node.
Consider a source system renaming customer_id to cust_id during a routine vendor upgrade, with no heads-up to the data team. The nightly ingestion job still runs and still reports success, because nothing in it errors — it just silently writes nulls into what used to be a populated join key. A retention model downstream trains on a feature that's now mostly empty, quietly degrading its accuracy for two weeks before anyone notices the forecast looks off. By the time someone traces it back, the schema change is ancient history and the actual damage — a mispriced retention campaign — has already shipped.
This is a reinforcing loop, not a single point of failure: each silent hop makes the eventual discovery more expensive and the root cause harder to find, because the failure and its symptom are now separated by three systems and two weeks. A monitoring stack that only watches the ingestion job's exit code will never catch it; one that watches schema and distribution on the join key would have flagged it on day one.
The trust erosion that follows an incident is itself a journey worth mapping — the same emotional-curve thinking our customer journey complete guide applies to product experiences applies just as well to a stakeholder's confidence in your data, from first doubt to full distrust of an entire dashboard.
Where Prodinja Fits
This is precisely the gap Prodinja's Systems Engineering tool is built for: it lets you map the causal loops behind an incident — how a silent schema change cascades through a join, into a model feature, into a dashboard number a VP repeats — so you can see where reinforcing loops amplify a small failure into a full trust crisis. Drawing that loop diagram before an incident, rather than reconstructing it during a postmortem, is what lets you place a freshness or schema check at the node where failures actually propagate, not just at the node where they happened to surface. It's a genuinely useful way to reason about where instrumentation earns its keep, not a promise that a model will find your next incident for you.
Key Takeaways
- Data downtime is any period a dataset is wrong, missing, or stale — the data-team analog of application downtime, and it deserves the same monitoring discipline.
- The five pillars — freshness, volume, schema, distribution, lineage — cover the failure modes that a "job succeeded" status check will always miss.
- Reactive discovery ("the CEO noticed") is a symptom of missing monitoring, not bad luck; proactive detection means an alert fires before a stakeholder complains.
- Combine rules-based tests with statistical anomaly detection — rules catch what you predicted, anomaly detection catches the long tail you didn't.
- Set data SLAs only on tables with named consumers, borrowing SLA/SLO vocabulary directly from SRE practice rather than instrumenting everything and drowning in alerts.
- A severity-tiered runbook with named owners at every stage — detect, triage, communicate, mitigate, resolve, postmortem — turns incident response from improvisation into a repeatable process.
- Map the causal chain before an incident, not during the postmortem, so detection sits at the node where a failure actually propagates, not just where it happened to surface.
Frequently Asked Questions
What is the difference between data quality and data observability?
Data quality describes whether data meets defined standards (accuracy, completeness, validity) at a point in time, while data observability is the continuous monitoring practice that detects when quality degrades. Quality is the target state; observability is the instrumentation that tells you when you've drifted from it.
How do you calculate a data SLA for a pipeline?
Start from the downstream consumer's actual requirement — when do they check the dashboard, and how stale can the number be before a decision goes wrong — then set a freshness and volume threshold with margin above your pipeline's typical runtime. Review and tighten the SLA quarterly as usage patterns change, rather than setting it once and forgetting it.
Who should own data incident response, the data team or the business stakeholders?
The data team (data engineering plus a data PM) owns detection, triage, and technical resolution, while the affected business stakeholders own confirming impact and prioritizing what gets fixed first when multiple tables break at once. Ownership should be documented per table, not assumed by whoever happens to be online.
What tools do data teams actually use for data observability?
Common categories include rules-based testing frameworks like dbt test, Great Expectations, and Soda, alongside dedicated observability platforms like Monte Carlo and Bigeye that add ML-based anomaly detection and lineage mapping. Most mature teams combine at least one tool from each category rather than relying on a single approach.
How is a data incident's severity level determined?
Severity is set by business impact, not technical complexity — a broken revenue table is Sev1 even if the fix is trivial, while a complex but low-visibility pipeline issue can be Sev3. Base the tiering on which dashboards, models, or reports depend on the affected table, mapped in advance rather than guessed during the incident.