- Description
- Turboréacteur monté sur un banc d'essai dans une soufflerie, un technicien à genoux réglant ses organes.
- Source
- NASA (Lewis Flight Propulsion Laboratory) · original page
- Licence
- PD Public domain
Testing
This theme measures the confidence you can place in a change before delivering it. The commitments focus on outcomes: how long it takes to know nothing was broken, what…
In brief
This theme measures the confidence you can place in a change before delivering it. The commitments focus on outcomes: how long it takes to know nothing was broken, what share of defects is caught before production, and how reliable the tests themselves are (no “flaky” tests).
Level: 🟢→🔴 · Theme code: TST (DL matrix)
Why this theme#
Without reliable automated tests, every delivery is a gamble: teams compensate with lengthy manual acceptance testing, code freezes, and fear. Delivery frequency collapses, batches get larger, so risk per delivery increases — a vicious circle. With good tests, a change is validated in minutes, and the fear of making changes disappears.
But poorly designed tests are expensive: an unstable (flaky) test teaches the team to ignore a red CI; a test coupled to implementation breaks at every refactoring; high coverage with empty assertions provides false reassurance. So this theme measures detection capability and suite reliability, not volume.
Datamart case: testing SQL is possible and cost-effective (procedure tests with a reference dataset, expected-result comparisons, idempotence tests: running twice yields the same state). Acceptance testing on business figures belongs to the matrix data theme UAT and Business Validation; here we are talking about automated tests.
What the theme measures#
- Axis: from “we test manually before delivery” to “the test suite is a calibrated, fast, reliable detector.”
- Key indicators: fast-suite feedback time (minutes); unstable test rate (%); mutation score (share of mutants killed); production defect rate detectable by an existing test.
- Unit: minutes; %; escaped defects per release.
Five-level table#
| Level | Short name | Commitment | Typical indicator (order of magnitude) |
|---|---|---|---|
| 1 | Manual | You know a feature works because someone tried it. | No automated suite; acceptance testing takes several days |
| 2 | Targeted | Critical cases are repeatable identically in a few hours. | Automated tests on critical paths; suite runs in hours |
| 3 | Systematic | A change is validated in under 15 minutes, without human intervention. | Fast suite < 10-15 min; < 2% unstable tests |
| 4 | Calibrated | The suite actually detects regressions: what breaks is seen, what does not break does not fail it. | Measured mutation (≥ 60-70% on core) ; pyramid respected |
| 5 | Predictive | Production defects become rare and each one produces a test that would have caught it; the suite guides delivery decisions. | > 90% of escaped defects have a non-regression test; controlled testing in production |
Milestones by level#
| ID | Milestone | Verifiable evidence |
|---|---|---|
| DL-TST-1.1 | A manual test procedure exists for critical user journeys | Document or replayable case list |
| DL-TST-1.2 | Found issues are logged and linked to a version | Issue → version traceability |
| DL-TST-1.3 | A test environment separate from production exists | Environment identified and accessible |
| DL-TST-2.1 | Critical paths have replayable automated tests | Successful run on a fresh workstation |
| DL-TST-2.2 | Tests use controlled reference data (versioned dataset, rebuildable database) | Proven identical rebuild |
| DL-TST-2.3 | Each fixed defect gets a non-regression test | Review of 10 recent fixes |
| DL-TST-2.4 | Test results are archived by version | Browsable history |
| DL-TST-3.1 | The fast suite (unit + light integration) runs on every change in < 15 minutes | Duration measured in CI |
| DL-TST-3.2 | An explicit test pyramid: mostly fast tests, few end-to-end tests | Numeric distribution by layer |
| DL-TST-3.3 | Unstable test rate is < 2% and each unstable test is quarantined within 48h | Tracking of new instabilities |
| DL-TST-3.4 | Tests depend on neither time, nor execution order, nor shared data | Successful random-order and parallel execution |
| DL-TST-3.5 | Idempotence tests exist for data processing (double run) | Test replaying the same batch twice |
| DL-TST-4.1 | Coverage is measured by branch on the core and tracked over time | Coverage report by module |
| DL-TST-4.2 | Mutation testing is run regularly on the core; its score is tracked | Score ≥ target set by the team |
| DL-TST-4.3 | Contract tests protect consumed/produced interfaces | Contract tests in CI on producer and consumer sides |
| DL-TST-4.4 | Property-based or generated-data tests cover critical calculation rules | Present on ≥ 3 business rules |
| DL-TST-4.5 | Performance non-regression tests bound critical operations | Versioned thresholds |
| DL-TST-5.1 | Every escaped defect is analyzed: why did no test catch it? a test is added | ≥ 90% of escaped defects have an associated test |
| DL-TST-5.2 | The suite is selective: only impacted tests run on each change, full suite continuously | Median feedback time < 5 min |
| DL-TST-5.3 | Controlled tests in production (progressive rollout, switchovers, synthetic monitoring) complement preproduction | Active synthetic journeys |
| DL-TST-5.4 | Trends in test maintenance cost are measured and managed | Share of time spent fixing tests < 10% |
Detailed reading by level#
🟢 Level 1 — Manual#
- Signs: acceptance testing takes several days; “we'll test in preprod”; regressions reappear.
- Gain: zero initial investment.
- Does not guarantee: reproducibility, coverage, speed.
- Difficulty: 1/5 · Effort: 1/5.
🟢 Level 2 — Targeted#
- Signs: a few automated tests on paths that already broke; a reference dataset exists.
- Gain: fear decreases on covered areas.
- Does not guarantee: that the suite is fast or reliable.
- Difficulty: 2/5 · Effort: 2/5.
🟡 Level 3 — Systematic#
- Signs: a change does not go out without suite feedback; slow tests are separated from fast ones; unstable tests are tracked like defects. The pyramid (many unit tests, fewer integration tests, few end-to-end tests) is an economic choice: test cost grows with scope.
- Gain: short feedback loop, more frequent deliveries (05).
- Does not guarantee: that tests really detect issues — you can have 90% coverage and 0 useful assertions.
- Difficulty: 3/5 · Effort: 3/5.
🔴 Level 4 — Calibrated#
- Signs: detection quality is measured: mutation testing introduces small code alterations (condition inversion, shifted boundary) and checks that a test fails; a surviving mutant reveals a missing assertion. Contracts between components are tested on both sides.
- Gain: a truly protective suite, fearless refactoring.
- Does not guarantee: absence of specification defects (you test what you understood, not necessarily what was needed).
- Difficulty: 4/5 · Effort: 4/5.
🔴 Level 5 — Predictive#
- Signs: each escaped defect feeds the suite; impact-based test selection keeps feedback under a few minutes; production itself is observed like a test (synthetic journeys, progressive rollout).
- Gain: confidence comes from evidence, not habit.
- Does not guarantee: exhaustiveness — absence of defects is never proven.
- Difficulty: 5/5 · Effort: 5/5.
Partial attainment and rating#
Apply the rating rule.
- 0.5 on DL-TST-3.3: unstable test rate is measured (5%) but quarantine is not systematic.
- 0.5 on DL-TST-4.2: mutation run once, as a one-off, without tracking.
- A partial 5: each escaped defect gets a test, but impact-based selection does not exist: the suite runs in 25 minutes.
Reassuring coverage. Symptom: 85% line coverage, yet a fix breaks a calculation. Cause: coverage measures what is executed, not what is verified. Remedy: branch coverage + mutation on the core; never set coverage as an individual objective.
Flaky tests. Symptom: “rerun it, it'll pass.” Cause: clock, order, shared data, fixed waits. Remedy: isolate, make deterministic, quarantine within 48h, track them.
Measure mean feedback time and CI rerun rate: this is the most reliable indirect indicator of suite health.
Moving to the next level#
Possible means (optional)
- 1 → 2: start with already seen defects; first automate the journeys that are most expensive to replay manually.
- 2 → 3: separate fast/slow, rebuildable dataset (database rebuilt from scripts), parallel and random execution to uncover order dependencies.
- 3 → 4: mutation tools one module at a time; contract tests; property-based tests for financial calculations/rounding.
- 4 → 5: systematic analysis of every escape; impact-based selection; production synthetic monitoring.
Datamart consolidation: the idempotence test replays the same batch twice and compares the fingerprint (counts + checksum per fact table). In 20 seconds, it finds a duplicate that manual acceptance testing only spotted after a month. Without this test, the defect survived because each isolated run looked correct.
Generating test cases and data: gain 4/5, risk 3/5 (tests that validate current behavior, including wrong behavior). Guardrail: review assertions, validate via mutation that generated tests actually detect something.
Rewarding test count produces redundant tests; rewarding coverage produces tests without assertions. Reward reduction in escaped defects.
See also#
- Neighbors: 02 Code Quality · 05 CI · 06 Deployment · 09 Technical Debt.
- Other matrices: BI — Testing and Metrics Reliability · Data — Data Quality · Project — Quality and Acceptance (QUA).
- Practice: CI/CD: Continuous Integration and Delivery · CI/CD for Data: Databases, Datamarts, Pipelines, and Reports · Project Management Pitfalls and Best Practices (Cross-Cutting Summary).
- Glossary · Rating: How to Read Scores, Ratings, and Legend.
✻ Written by Claude Sonnet 5.5
