- Description
- Un cadran gradué éclairé, dont l'aiguille se lit sur l'échelle
- Source
- freestocks.org · original page
- Licence
- CC0 Public domain
Rating: How to Read Ratings, Scores, and the Legend
This section uses three measurement tools: criterion-by-criterion /5 ratings (method, tool, and architecture comparisons), maturity scores (five-level matrices), and…
In brief
This section uses three measurement tools: criterion-by-criterion /5 ratings (method, tool, and architecture comparisons), maturity scores (five-level matrices), and reading-level markers (🟢🟡🔴). None of these is a physical measurement. This file explains what each one means, how to read it without misinterpretation, and how to rate your own organization honestly.
Level: 🟢→🔴
🟢 Novice — /5 Ratings in Comparisons#
In comparisons (for example Comparison of Project Management Methods,
Comparison of Planning Tools, reporting architectures),
each solution receives a 1-to-5 rating per criterion, written 4/5 or ★★★★☆ 4/5.
| Rating | Meaning | How to read it |
|---|---|---|
| 1/5 | Very weak on this criterion | A blocking point or nearly so, unless expensive workarounds are used |
| 2/5 | Weak | Possible but painful, or with major tradeoffs |
| 3/5 | Adequate, average | Gets the job done, no more |
| 4/5 | Good | A clear strength |
| 5/5 | Excellent | Best-in-class on this criterion in the given context |
Three reading rules:
- A rating only has meaning in its context. Each table states the context (team size, project type). A 5/5 for a 5-person team may be a 3/5 for 200 people.
- A 3/5 is not criticism: it's the middle of the scale. Tables are designed to discriminate; if they were all 4s, they would teach nothing.
- Do not compute blind averages. A solution rated 5/5 on ten criteria and 1/5 on “fit for your case” is still bad for you. Read the one-sentence justification, and weight according to your priorities.
Two options rated on three criteria: A = {cost 5, freshness 2, governance 3}, B = {cost 3, freshness 4, governance 3}. Averages: 3.3 each. If your need is an operational dashboard refreshed every 15 minutes, freshness carries significant weight (for example ×3): A = 2.8 · B = 3.6 — B wins. The math is trivial; what matters is making weights explicit before looking at ratings.
🟢 Novice — The 🟢🟡🔴 Scale and Callout Legend#
Level markers. Placed in headings or at the start of a section:
| Marker | Audience | Content |
|---|---|---|
| 🟢 Novice | Discovery | Definitions, vocabulary, principles, simple examples |
| 🟡 Practitioner | Day-to-day practice | Implementation, tradeoffs, tooling, concrete cases |
| 🔴 Expert | Mastery | Internal mechanisms, edge cases, scaling, fine-grained tradeoffs |
The meta line Level: 🟢→🔴 in a file indicates the covered range. You can stop reading at the end of your level without losing the thread.
Recurring callouts.
A proven practice, with the reason it works.
A frequent error, with its symptom, cause, and remedy.
A detail you generally only learn through experience.
A concrete case, often with numbers, based on the running example of a SQL Server datamart fed by a field sales force automation (SFA) application.
What AI brings at this point, and its limits.
🟡 Practitioner — Aspects and Their Subjectivity#
“Aspects” are the comparison criteria. To stay consistent within a dossier, this section groups them by family:
| Family | Typical aspects |
|---|---|
| Project methods | predictability · adaptation to change · risk control · lightness · fit for data projects · ease of adoption · scalability · visibility and governance · deliverable quality |
| Version control system | learning curve · performance on large repositories · branching and merging · traceability · permissions granularity · large files · CI/CD integration · SQL/BI fit · long-term viability |
| Planning | flexibility · reporting · code/CI integration · cost · adoption · governance · scalability · customization · extensibility |
| Reporting architectures | freshness · performance · cost · complexity · governance and security · scalability · required skills · technology maturity · lock-in |
Why subjectivity is irreducible.
- The choice of aspects already steers the conclusion: adding “cost” or removing “lock-in” changes the ranking.
- The rating depends on the evaluator's experience, the state of the offering, and team context.
- 1-point gaps are often within the margin of error; a 2-point or greater gap is a signal.
- AI-generated writeups have not used your tools: the ratings summarize public knowledge and a typical expert judgment, not a measurement.
Redo the table with your own weights and ratings in a 90-minute workshop. The gap between your version and the site's version is more instructive than the table itself.
Quoting “4/5” out of context in a committee (“this section says X is worth 4/5”). Symptom: the decision relies on a number without criterion. Cause: losing the one-sentence justification. Remedy: quote the full row with context.
🟡 Practitioner — Maturity Matrices#
Maturity Matrices — Principle, Anatomy, and Comparison measure a practice. A matrix is defined over a domain, themes, and for each theme, five levels. Each level has:
- a promise: the observable result achieved (never the means);
- milestones: three to five precise, verifiable criteria, with a stable identifier
<CODE>-<THEME>-<level>.<n>(for examplePI-RSK-3.2).
Matrix codes: PI = IT project management, DL = software development, PD = data project management, BI = BI development.
“Open a door.” Performance: level 1 = under one hour, level 5 = under one second. Noise: level 1 = very noisy, level 5 = silent. Locking: level 1 = none, level 3 = locked, level 5 = restored to initial state. We describe outcomes, not the lock.
By design, level 5 is very difficult to reach fully. It serves as a promise to drive progress, even if its milestones are only partially met. We assess substance only (practices, outcomes), not a real interface.
🔴 Expert — The Matrix Rating Rule#
Common rule, identical in each matrix README:
Each milestone has progress 0 / 0.5 / 1. A level N is achieved if the average progress of all milestones from levels 1 to N is ≥ 0.8 and no milestone is at 0. Theme score = N_achieved + average progress of milestones at level N_achieved+1 × 0.99 (e.g., 2.4). Matrix score = weighted average of themes (default weight 1, editable). A full 5.0 score is exceptional and may never be reached: that is not failure.
Step-by-step reading.
- Rate each milestone: 0 = absent, 0.5 = partial or uneven, 1 = present and verifiable with evidence.
- Find the highest level N such that the average of milestones from levels 1 to N is ≥ 0.8 and none of those milestones is 0.
- Add the average of milestones at level N+1 multiplied by 0.99 (the cap prevents reaching the next level by rounding).
- Weight themes according to your stakes.
Theme with 4 milestones per level. Level 1: 1, 1, 1, 1 (average 1). Level 2: 1, 1, 0.5, 1; average of levels 1-2 = (4 + 3.5) / 8 = 0.94 → level 2 achieved. Level 3: 1, 0.5, 0, 0.5; average of levels 1-3 = (4 + 3.5 + 2) / 12 = 0.79 < 0.8 and one milestone at 0 → level 3 not achieved. Score = 2 + (2 / 4) × 0.99 = 2.495, i.e., 2.5.
What a “partial 5” means. A theme may display 4.6: level 4 is achieved, and more than half of level 5 milestones are already partially visible. That's a very good result: level 5 measures the horizon, not an obligation.
How to read matrix scores.
| Range | What the score says |
|---|---|
| 0–1 | Practices are absent or individual; outcomes are unpredictable |
| 1–2 | Practices exist but are uneven across people or projects |
| 2–3 | Practices are defined and applied; outcomes are generally predictable |
| 3–4 | Practices are measured; decisions are based on indicators |
| 4–5 | Tool-supported continuous improvement; the organization corrects its own gaps |
No range is a moral judgment: a score of 2 is suitable for a small internal project, and higher requirements there could be counterproductive.
🔴 Expert — Gaming a Measure: Goodhart's Law#
“When a measure becomes a target, it ceases to be a good measure.” A poorly used maturity matrix becomes a team ranking, and milestones get checked without changing outcomes.
| Drift | Symptom | Countermeasure |
|---|---|---|
| Milestone checked with no evidence | Score rises, outcomes stagnate | Require the listed verifiable evidence for each milestone |
| Score compared across teams | Polishing, information hoarding | Scores reserved for the team that produces them |
| Aiming for 5 everywhere | Overinvestment, cost without benefit | Set a target per theme based on stakes |
| Assessment by management alone | Gap from field reality | Cross self-assessment, see Self-Assessment with Maturity Matrices |
An AI can prefill a grid from documents (tickets, repository history), which speeds up evidence collection. It cannot judge whether a practice is truly lived. Treat its suggestions as hypotheses to confirm by sampling.
See also#
- README Read This First · Reading Paths · Glossary
- Maturity Matrices — Principle, Anatomy, and Comparison · Self-Assessment with Maturity Matrices
- Project Metrics and Reporting · Choosing Your Method and Hybridizing
✻ Written by Claude Sonnet 5.5
