Insights / Practice · 2023-10-03 · 5 min read

Lineage you can actually use during an incident

Lineage diagrams look impressive in a slide. The test is whether, at 2am with a wrong dashboard, you can answer one question fast: what feeds this, and what does this feed?

Data lineage — the map of what flows into what — is usually sold as a governance nicety. Its real value is operational and shows up under pressure. A number looks wrong; you need to know upstream what could have caused it, and downstream what else is now suspect. Lineage that can't answer those two questions quickly is decoration.

Two questions, fast

Upstream: if this table is wrong, which sources and transformations could be responsible? That narrows the search from "everything" to a handful of candidates. Downstream: if this table is wrong, which dashboards and models inherited the error? That tells you what to caveat before someone makes a decision on it.

Lineage earns its keep in the ten minutes after a number looks wrong, or it doesn't earn its keep at all.

Derive it, don't draw it

Hand-drawn lineage is wrong the moment someone ships a change. Lineage parsed automatically from the actual transformation code stays current because it's generated from the thing it describes. If your tooling can produce column-level lineage from your models, that's the version that survives contact with a fast-moving pipeline.

Good enough beats complete

You don't need lineage for all four thousand tables to benefit. Coverage of the critical path — the tables feeding decisions that matter — answers the 2am questions for the incidents that actually hurt. Start there.


← All insights