Metadata, lineage, and documentation

“Where does this number come from?” and “what happens if I change this column?” are the two questions that come up most often, and answering them costs hours when it…

Guide≈ 7 min readCheckedReading

On this page

Why this theme
What this theme measures
The five levels
Milestones by level
Detailed reading by level
Partial achievement and rating
Move to the next level
See also

Level: 🟢→🔴 · Theme code: MET

Why this theme#

“Where does this number come from?” and “what happens if I change this column?” are the two questions that come up most often, and answering them costs hours when there is no tooling. Metadata is data that describes data: definitions, types, owner, classification, translations. Lineage (traceability, ancestry) is the path of a number from source to report, at table or column level. Documentation is the readable, narrative part. Without them, knowledge is in the author's head: a departure destroys it, an impact analysis becomes an investigation, a compliance request (SEC) becomes impossible to satisfy. The opposite risk also exists: manually written documentation becomes outdated faster than it is read. So maturity means bringing metadata closer to code and extracting it rather than copying it.

What this theme measures#

Axis: from tacit knowledge to active metadata, extracted from code and used by the platform.
Key indicator: share of critical objects documented, with up-to-date lineage, and time to answer “where does this number come from / who is impacted?”
Unit: % of documented objects; average documentation age relative to code; response time for an impact analysis (minutes, hours, or days).

The five levels#

Level	Short name	Promise	Typical indicator (orders of magnitude)
1	Tacit	Knowledge is in the author's head; answering a question requires finding that person.	Impact analysis: days
2	Written	Main objects have a readable description and an overall diagram exists.	≥ 50% of critical objects documented
3	Referenced	A data dictionary and reference lineage exist for critical products, maintained with changes.	≥ 80% documented; impact in hours
4	Extracted	Metadata and lineage are automatically extracted from code and database, so always up to date.	Doc/code gap ≈ 0; impact in minutes
5	Active	Metadata drives the platform: documentation generation, translations, controls, permissions, impact detection before deployment.	Impact calculated before any release

Milestones by level#

ID	Milestone	Verifiable evidence
PD-MET-1.1	Datamart objects have names understandable by the business.	Review of a sample of ten objects by a business representative.
PD-MET-1.2	An overall diagram (source flows → datamart → reports) exists.	Dated diagram.
PD-MET-1.3	Scripts include a header: object, author, role.	Sample of scripts with header.
PD-MET-1.4	The list of published reports and their owners is maintained.	Report inventory.
PD-MET-2.1	Critical tables have a description (role, grain, source).	Description on ≥ 50% of critical tables.
PD-MET-2.2	Critical columns have a label and a definition.	Partial but usable dictionary.
PD-MET-2.3	The meaning of indicators is written and accessible to users (ADO).	Glossary accessible from the report.
PD-MET-2.4	A newcomer can independently find the source of a simple indicator in less than one day.	Test with a newcomer or an external person.
PD-MET-3.1	A complete data dictionary covers tables, columns, types, rules, owner.	Dictionary covering ≥ 80% of critical products.
PD-MET-3.2	Lineage at table or column level exists for critical indicators.	Lineage sheet for ten indicators.
PD-MET-3.3	Documentation is updated with every change (task completion criterion).	Release rule applied on a sample of ten changes.
PD-MET-3.4	Multilingual metadata (labels, descriptions) is managed in a single location.	Single source for translations (BI — Accessibility and Multilingual).
PD-MET-4.1	Metadata is stored with code (extended properties, versioned files) and deployed with it.	Metadata in the repository; deployment with no manual entry.
PD-MET-4.2	Lineage is automatically extracted (dependency analysis, semantic model, pipelines).	Reproducible generation; comparison with reality shows no gap.
PD-MET-4.3	An impact analysis answers in minutes (“who consumes this column?”).	Timing on three cases.
PD-MET-4.4	An automatic check flags objects without description or owner (GOV).	Automated coverage report.
PD-MET-5.1	Documentation is generated from metadata (site, dictionary, diagrams).	Scheduled generation; publication with each version.
PD-MET-5.2	Impact is calculated before deployment and blocks unannounced breaking changes (CTR).	Impact analysis pipeline step.
PD-MET-5.3	A single catalog links technical metadata, business metadata, quality, classification, and usage.	Catalog queried daily by multiple profiles.
PD-MET-5.4	Metadata quality is itself measured and improved.	Completeness and accuracy indicators tracked.

Detailed reading by level#

🟢 Level 1 — Tacit#

Signs: “ask Marie”; impact analysis is done by opening every report; documentation is an old email.
Gain: zero writing effort.
Does not guarantee: continuity when the author leaves.
Difficulty: 1/5 · Effort: 1/5.

🟢 Level 2 — Written#

Signs: descriptions on main tables; overall diagram; glossary in the report.
Gain: a newcomer can self-orient on simple cases.
Does not guarantee: long-term accuracy (manual docs become outdated).
Difficulty: 2/5 · Effort: 2/5.

🟡 Level 3 — Referenced#

Signs: complete dictionary; lineage for critical indicators; updates required with every change.
Gain: impact analysis in hours; audit answers are possible.
Does not guarantee: absence of drift (manual discipline erodes).
Difficulty: 3/5 · Effort: 3/5.

🧪 Example

“Can we rename the PRD_PILIER column?” At level 1: you search through 25 reports, half a day, with a risk of missing something. At level 4: one query on extracted dependencies (database system view of dependencies, semantic model, and report definitions) lists consumers in two minutes: three views, two measures, four reports. The rename becomes a decision, not an investigation (see also the expand/contract principle in MOD).

🟡 Level 4 — Extracted#

Signs: metadata in extended properties or versioned files; generated lineage; impact analysis queried in minutes; automatic coverage checks.
Gain: documentation can no longer become outdated faster than code.
Does not guarantee: meaning: perfect lineage of a poorly defined indicator does not tell you what it means.
Difficulty: 4/5 · Effort: 4/5.

-- Base-level impact analysis: who depends on the column? (T-SQL, referenced dependencies)
SELECT DISTINCT OBJECT_SCHEMA_NAME(d.referencing_id) AS SchemaConsommateur,
       OBJECT_NAME(d.referencing_id)               AS Consommateur,
       o.type_desc
FROM   sys.sql_expression_dependencies AS d
JOIN   sys.objects                     AS o ON o.object_id = d.referencing_id
WHERE  d.referenced_id = OBJECT_ID('[dim].[Produit]')
  AND  d.referenced_minor_id = COLUMNPROPERTY(OBJECT_ID('[dim].[Produit]'), 'PRD_PILIER', 'ColumnId');
-- Limit: column level is tracked only for schema-bound objects (sys.dm_sql_referenced_entities for others); dynamic dependencies (SQL built as strings) and reports are not visible.

🔴 Level 5 — Active#

Signs: generated documentation and translations; impact analysis calculated before release; unified catalog; measured metadata quality.
Gain: the platform documents and controls itself.
Does not guarantee: user adoption (comprehensive documentation nobody reads); revisit ADO.
Difficulty: 5/5 · Effort: 5/5.

Partial achievement and rating#

Evaluate on ten randomly selected critical objects (tables, indicators, reports). A coverage milestone is worth 1 if ≥ 9 objects out of 10 satisfy it, 0.5 if 5 to 8, otherwise 0. “In minutes” milestones are timed.

A “partial 5”: datamart documentation is generated with every version (5.1 at 1), impact analysis still misses reports (5.2 at 0.5), there is no unified catalog (5.3 at 0), metadata quality is not tracked (5.4 at 0).

⚠️ Pitfall

Manual documentation that becomes outdated: a 200-page document where no one knows what is still true. Symptom: docs say one thing, code says another. Remedy: everything that can be extracted is extracted; manual docs keep only what code does not say (the why, decisions, context).

⚠️ Pitfall

Reassuring partial lineage: lineage that ignores dynamic SQL, manual extracts, and reports gives false confidence in impact analysis. Remedy: state what lineage does not cover, and complement with a non-regression test (REC).

💡 Expert tip

Put metadata where the developer works: SQL extended properties, versioned semantic model definition files, repository annotations. Metadata far from code gets filled in once, then never again.

Move to the next level#

Possible means (optional)

Transition	Possible levers
1 → 2	First document the 20 most consumed tables and 20 most consumed indicators; add a script header.
2 → 3	Make documentation updates a task completion criterion (definition of “done” — see Scrum in Practice); map lineage for critical indicators.
3 → 4	Store metadata with code; extract dependencies; check coverage in the pipeline (DOP).
4 → 5	Generate documentation and translations; integrate impact analysis into the pipeline; adopt a catalog (state of offerings as of 2026-10 to be verified).

Cross-cutting pitfalls: (1) False positive — 100% coverage with descriptions like “Customer column”; check text quality on a sample. (2) Goodhart — aiming for a high rate of described columns produces meaningless descriptions. (3) Tool without practice — installing a catalog with no content owner results in an empty catalog.

🤖 AI

It excels at proposing column descriptions from names and content, summarizing lineage, translating labels, and answering “where does this number come from” from a catalog. Risk: plausible but false descriptions that gain written authority. Every generated description is flagged as such and validated by the owner (GOV).