i

Description

Une mêlée de rugby sur un terrain brumeux, vue d'ensemble

Source

auteur non indiqué · original page

Licence

PD Public domain

Original photo, unretouched. The background is a version of it stylised by the site.

Scrum Applied to Data, BI, and Datamarts

You will know how to adapt Scrum to a data / BI / datamart project: write data stories, plan exploration spikes, define an Increment across a source → ETL → model →…

In short

• You will know how to adapt Scrum to a data / BI / datamart project: write data stories, plan exploration spikes, define an Increment across a source → ETL → model → report chain.
• You will know how to build a data Definition of Done (quality, acceptance testing on figures, performance, security, documentation, replayability).
• You will know how to balance data sprints and visualization sprints, manage data debt, source dependency, and test datasets.
• You will know how to frame the data Product Owner role and organize business acceptance testing on figures.
• You will get a strengths/weaknesses table and /5 fit ratings of Scrum for data, with an AI section.

Illustrative running example (anonymous). A SQL Server datamart fed by a field sales force application (SFA), with multilingual Power BI reports, versioned in TFVC and delivered as installation kits. The figures provided are illustrative orders of magnitude.

On this page

1. 🟢 Novice — Why Scrum and data rub against each other
2. 🟢 Novice — What is an Increment in BI?
   1. Increments at different levels
3. 🟢 Novice — The multidisciplinary data team
4. 🟡 Practice — Writing data stories
   1. Data story template
   2. Five types of data stories
   3. The “technical” story in BI: when is it legitimate?
5. 🟡 Practice — Data exploration spikes
   1. Typical data spikes
   2. Rules for a good spike
6. 🟡 Practice — A Sprint across the source → ETL → model → report chain
   1. The slicing challenge
   2. A typical two-week Sprint
   3. Long-running processing and feedback loop
7. 🟡 Practice — Data Sprints or visualization Sprints?
   1. When separation is justified
8. 🟡 Practice — The Data Definition of Done
9. 🟡 Practice — Test Data Sets
   1. Rules
10. 🟡 Practice — Dependency on Sources
    1. Treating dependency as an item
    2. Data contracts
11. 🟡 Practice — The data Product Owner
    1. Where can you find this PO?
12. 🟡 Practice — Business acceptance testing on numbers
    1. The five levels of control
    2. Process
13. 🔴 Expert — Data debt
    1. Managing debt in Scrum
14. 🔴 Expert — Replayability, idempotence, and history tracking in the Sprint
    1. Idempotence
    2. History tracking
    3. Planning for “replayable” in Scrum
15. 🔴 Expert — Migrations, model versions, and kits
    1. Product version vs. Sprint
16. 🔴 Expert — DataOps coordination and governance
17. 🔴 Expert — When is Scrum the wrong choice for data?
18. Strengths and weaknesses
19. Metrics: fit ratings /5
    1. Fit by type of data work
20. AI contribution
21. See also

Level: 🟢→🔴

🟢 Novice — Why Scrum and data rub against each other#

Scrum was born from software development, where you can deliver a small functional increment in two weeks: one feature, one screen. Data has three differences that create friction (see Project Management Fundamentals: data-specific characteristics):

This friction does not make Scrum inapplicable. It requires adapting slicing, DoD, planning, and the PO role — that's the purpose of this file. Alternatives and complements (CRISP-DM, DataOps, Kimball for modeling) are in Methods Specific to Data Projects; the overall comparison is in Comparison of Project Management Methods.

🟢 Novice — What is an Increment in BI?#

The Guide requires a usable Increment (Scrum, the framework (Scrum Guide 2020)). In BI, “usable” means: a user can make a decision based on a reliable figure. That requires a minimal vertical slice through the entire chain.

Increments at different levels#

🟢 Novice — The multidisciplinary data team#

An autonomous BI team across the full chain brings together the following skills (one person may cover several):

Scrum uses Developers in the broad sense: this includes data engineers and BI developers. When a skill exists only outside the team (central DBA, security team, source owner), every dependent item creates a bottleneck: this is a risk factor to address explicitly (service agreements, reserved time slots, a “liaison” member), or a reason to broaden the team.

🟡 Practice — Writing data stories#

User stories (Scrum in Practice) apply, with an adaptation: the “I want to…” is often a decision-making capability, and the “so that…” carries the value. You need to add data-specific elements.

Data story template#

Five types of data stories#

The “technical” story in BI: when is it legitimate?#

Pure Scrum avoids stories with no value. In BI, some work is prerequisite and indivisible (creating a shared dimension). Common-sense rule:

1. Pull technical work from a consumer story (“see revenue by customer” pulls the customer dimension).
2. If the work supports several future stories (shared dimension, calendar), carry it explicitly with a justification (“prerequisite for 4 Q2 stories”) and limit it to the strict minimum.
3. Do not let the share of technical stories climb without discussing it in retro (no universal threshold: the team sets its own and recalibrates it).

🟡 Practice — Data exploration spikes#

A spike is timeboxed research work whose deliverable is knowledge (a decision, an estimate, a proof of feasibility), not production code. It is the preferred tool against data uncertainty.

Typical data spikes#

Rules for a good spike#

• Explicit question at the start; exit criterion (“we will know whether…”).
• Strict timebox (hours or days); if inconclusive, the remaining uncertainty is itself the result.
• Written deliverable (one page: question, method, findings, recommended decision) archived near the backlog.
• Disposable code by default: if the spike produced good code, it is rebuilt with the DoD.

-- Spike : profil d'un champ candidat (exemple générique)
SELECT
    COUNT(*)                                             AS nb_lignes,
    SUM(CASE WHEN segment IS NULL OR segment = '' THEN 1 ELSE 0 END) AS nb_vides,
    COUNT(DISTINCT segment)                              AS nb_valeurs,
    CAST(100.0 * SUM(CASE WHEN segment IS NULL OR segment = '' THEN 1 ELSE 0 END)
         / NULLIF(COUNT(*), 0) AS DECIMAL(5,2))          AS pct_vides
FROM stg.client;

-- Unicité de la clé annoncée
SELECT code_client, COUNT(*) AS nb
FROM stg.client
GROUP BY code_client
HAVING COUNT(*) > 1;

🟡 Practice — A Sprint across the source → ETL → model → report chain#

The slicing challenge#

Vertical slicing in BI is difficult because the layers are interdependent: no report without a measure, no measure without a model, no model without data. The solution is to reduce the size of each slice, not the depth: little data, few measures, one page — but the entire chain.

A typical two-week Sprint#

Each day combines one building block and its test, never “build everything then test everything.” Number tests are written with or before the building block (TDD mindset applied to data), as described in Testing and Reliability of Numbers.

Long-running processing and feedback loop#

When a consolidation or full reload takes hours, the “change → test” loop gets longer. Countermeasures:

These points are detailed on the delivery side in CI/CD for data: foundations, datamarts, pipelines, and reports. For Sprint cadence: if the full test cycle takes more than one day, a two-week Sprint is a reasonable minimum; one week becomes unrealistic unless partial loading is under control.

🟡 Practice — Data Sprints or visualization Sprints?#

A common temptation is to separate Sprints: first “data Sprints” (prepare all data), then “visualization Sprints” (build reports). This is mini-waterfall, where value arrives only at the end and the business side discovers too late that the number is not what it expected.

When separation is justified#

• Truly indivisible heavy foundations (e.g., platform migration), to be bounded and tied to a user milestone.
• Rare specialties split across teams: in that case, synchronize Sprints and share one backlog.
• Distinct visual exploration phase (UX), upstream, with prototypes on mock data, provided you then validate on real data.

The semantic model / reports split, however, has real technical value: multiple reports consume the same model. In that case, the model is a product with its own backlog and its own versioning (see Semantic Model and Versioning and Industrialization of Reports).

🟡 Practice — The Data Definition of Done#

The DoD drives the quality of a BI project. It must cover much more than “the report displays.” Template to adapt (each row is binary):

-- Monthly revenue reconciliation between source and fact table (sample: pilot region)
WITH src AS (
    SELECT EOMONTH(f.date_facture) AS mois, SUM(f.montant_ht) AS ca
    FROM   src.facture AS f
    JOIN   src.client  AS c ON c.code_client = f.code_client
    WHERE  c.region = 'PILOTE'
    GROUP  BY EOMONTH(f.date_facture)
), dwh AS (
    SELECT d.fin_de_mois AS mois, SUM(v.ca_ht) AS ca
    FROM   dwh.fact_ventes AS v
    JOIN   dwh.dim_date    AS d ON d.id_date = v.id_date
    JOIN   dwh.dim_client  AS c ON c.id_client = v.id_client
    WHERE  c.region = 'PILOTE'
    GROUP  BY d.fin_de_mois
)
SELECT COALESCE(s.mois, w.mois) AS mois,
       s.ca AS ca_source, w.ca AS ca_dwh,
       COALESCE(w.ca,0) - COALESCE(s.ca,0) AS ecart
FROM   src s FULL OUTER JOIN dwh w ON w.mois = s.mois
WHERE  COALESCE(w.ca,0) <> COALESCE(s.ca,0);   -- expected: 0 rows

Related themes in the matrices: Data Quality, Acceptance testing and business validation, Security, Privacy, Compliance, Metadata, Lineage, and Documentation, RLS Security and Access.

🟡 Practice — Test Data Sets#

Without representative and stable test data, you cannot test, iterate quickly, or reproduce a defect. Strategy by level:

Rules#

1. Versioned reference set with its expected values (customer X's revenue in March equals Y); these values are the figure non-regression test.
2. Anonymization / pseudonymization before any use outside production; do not move personal data to a test environment without legal basis and access control.
3. Edge cases documented and covered: merged customer, credit note, currency, time zone, language, extreme value, truncated history.
4. Dedicated environments: acceptance testing is not done on an environment that keeps changing under your feet.
5. Evolution: when the source changes, the reference set is consciously refreshed, with visible diff and validation.

🟡 Practice — Dependency on Sources#

The source (here, the SFA application) is a third party: its schema, quality, and schedule do not depend on the team. Consequences for Scrum:

Treating dependency as an item#

A dependency is not “what blocks an item”: it is a backlog item with its own lifecycle, and one that must be ordered (“get access” must be ready before selecting items that depend on it). Map: source, owner, access mode, change frequency, contact, SLA.

Data contracts#

Formalize a contract with the source owner: guaranteed schema, field semantics, freshness, expected quality, change procedure (notice period), versions. Even lightweight (one page + automated tests), it turns a dependency relationship into a managed relationship. Details: Data Sources and Contracts.

🟡 Practice — The data Product Owner#

The data PO has the PO responsibilities (Scrum, the framework (Scrum Guide 2020)), with additional requirements. This is generally the hardest person to find, because it requires a rare profile: close to the business, comfortable with numbers, able to arbitrate definitions.

Where can you find this PO?#

🟡 Practice — Business acceptance testing on numbers#

Acceptance testing of a BI report is not a screen check: it is verification that the numbers are correct and understood.

The five levels of control#

Process#

1. Prepare: frozen reference dataset; scope; expected outcome (named source of truth: source system, accounting statement, validated Excel file).
2. Run automated checks (reconciliation SQL); record gaps.
3. Explain each gap: bug, different rule, dirty data, calendar difference (invoice date vs shipping date), rounding, time zone.
4. Decide with the PO: fix, accept (with traceability), or document an intentional difference (“our revenue excludes credit notes of type X”).
5. Sign off: acceptance testing sheet (date, scope, version, results, reservations, signatory). This protects the team and builds trust.

Acceptance testing is effectively a Sprint event, even if the Guide does not define it: include it in the DoD, plan it within the Sprint (days 8–9 in the example above), and do not push it to a final “acceptance Sprint.” The matrices cover this point in Acceptance testing and business validation and Quality and UAT (QUA).

🔴 Expert — Data debt#

Technical debt (Ward Cunningham) refers to design shortcuts that speed things up today and slow things down tomorrow. In data, a related form, data debt, deserves its own tracking.

Managing debt in Scrum#

1. Make it visible: “debt” items in the Product Backlog, labeled, with cost of delay (what it costs us each month).
2. Reserve capacity (a fixed share of the Sprint, calibrated based on debt magnitude; there is no established standard); do not depend on a hypothetical “cleanup Sprint.”
3. Intentional debt: a voluntary and dated trade-off (“we accept full import for the MVP, payback planned in Sprint N”) is better than hidden debt. The PO signs off.
4. Non-worsening rule: any change in a fragile area improves at least one point (boy scout rule).
5. Repay first the debt that blocks a priority objective.
6. Measure: age, number of items, time spent on workarounds.

See the matrices topic Technical Debt and Maintainability.

🔴 Expert — Replayability, idempotence, and history tracking in the Sprint#

These properties are not “tested” during acceptance testing: they are designed, so they must be planned as requirements in the Sprint.

Idempotence#

A process is idempotent if running it multiple times with the same inputs produces the same final state as a single run. This is the condition for a replayable load after an incident, and for peace of mind in production.

-- Chargement idempotent d'une dimension : MERGE sur clé naturelle
MERGE dwh.dim_client AS cible
USING stg.client_propre AS source
   ON cible.code_client = source.code_client
WHEN MATCHED AND (cible.libelle <> source.libelle OR cible.region <> source.region) THEN
     UPDATE SET libelle = source.libelle, region = source.region, date_maj = SYSUTCDATETIME()
WHEN NOT MATCHED BY TARGET THEN
     INSERT (code_client, libelle, region, date_maj)
     VALUES (source.code_client, source.libelle, source.region, SYSUTCDATETIME());
-- Rejouer deux fois : mêmes lignes, pas de doublon.

For a fact table: use a “delete window then reinsert” pattern (by period) within a single transaction, or partition swap, rather than “append unconditionally.”

History tracking#

Attributes that change (a customer's region) raise the question of dimension type (SCD type 1: overwrite; type 2: version). Choose early, because late change is costly. The DoD test must verify expected behavior when a change occurs (dataset with two versions of the same customer).

Planning for “replayable” in Scrum#

• Systematic acceptance criterion: “a double load does not change the result.”
• Recovery story: “after a mid-load failure, restart leaves no partial data.”
• Test dataset with late-arriving data events (data arriving after the period).
• Batch logging (identifier, time, row count) for diagnostics.

🔴 Expert — Migrations, model versions, and kits#

A datamart delivered to multiple clients, in successive versions, is a product: each new version must be able to install on a fresh database and migrate an existing database without data loss. Scrum supports this provided these aspects are treated as Increment requirements, not as “post-processing.”

Product version vs. Sprint#

A Sprint produces internal Increments; a product version (kit) groups several Sprints and goes through a heavier delivery DoD. Do not confuse them or pace versions by Sprint. For version governance and delivery, see CI/CD for data: foundations, datamarts, pipelines, and reports, Choosing Your Method and Hybridizing and, on the tools side, Branching Strategies and Source Control for Data: SQL and Power BI.

🔴 Expert — DataOps coordination and governance#

Scrum answers “how to organize team work.” By itself, it does not cover:

Touchpoints:

• Governance ↔ DoD: governance requirements (classification, retention, access) become DoD criteria.
• DataOps ↔ Sprint: the more automated the checks, the less costly the DoD and the shorter the Sprint.
• Architecture ↔ Product Backlog: architecture decisions (platform choice) are explicit items, sometimes spikes, with an owner.

🔴 Expert — When is Scrum the wrong choice for data?#

This typology is extended in Choosing Your Method and Hybridizing and Comparison of Project Management Methods.

Strengths and weaknesses#

Metrics: fit ratings /5#

Expert assessments, contextual (see Rating). Context: team of 4 to 8 people working on an evolving enterprise datamart (application source, Power BI reports), with version-based deliveries. Recalibrate for a data science project or a one-person team.

Fit by type of data work#

These ratings are discussion benchmarks, not measurements: they reflect one practitioner's view of a typical context and do not replace your own self-assessment (Maturity Matrix — Data Project Management (PD)).

AI contribution#

Example prompts:

• “Here is the schema of my fact table and customer dimension (no data). Write T-SQL validation queries: key uniqueness, foreign key orphans, negative values, out-of-range dates. Explain each check.”
• “Here are three acceptance criteria. For each one, propose an SQL reconciliation test and an edge case I did not anticipate.”

What AI does not replace: value definition and business rules (PO), signed-off business acceptance testing, accountability for numbers, decisions about accepted technical debt, access governance. Data-specific risks: syntactically correct but semantically wrong queries (bad join multiplying rows, lost filter) producing a plausible number; data leakage if real data is pasted into an unauthorized service; hidden debt from generated code the team does not understand. Rule: every AI-generated test must itself be tested (on a case where the result is known). See AI for Project Management (General Then Data) and AI in Planning and Version Control Tools.

See also#

• Framework and practice: Scrum, the framework (Scrum Guide 2020), Scrum in Practice, Project Management Fundamentals.
• Dedicated methods and comparison: Methods Specific to Data Projects, Agile and Lean Methods (Manifesto, Kanban, XP, Lean, Shape Up, DSDM, SAFe, DevOps…), Comparison of Project Management Methods, Choosing Your Method and Hybridizing.
• Delivery: CI/CD: Continuous Integration and Continuous Delivery, CI/CD for data: foundations, datamarts, pipelines, and reports, Pitfalls and good practices in project management (cross-functional summary).
• Tools: VCS and SQL/BI data, Azure DevOps / TFS, Project Metrics and Reporting.
• Matrices: Maturity Matrix — Data Project Management (PD), Lead Time and Industrialization (DataOps), Maturity Matrix — BI Tool/Application Development.
• Power BI reporting: Cross-Cutting Power BI Good Practices, Power BI Reporting Project Checklists.
• Glossary: Glossary.

✻ Written by Claude Sonnet 5.5