Matrix: Advanced analytics

How the product supports advanced analytics (prediction, anomaly detection, recommendation): scoping, data, experimentation, validation, production release, monitoring, accountability, integration.

≈ 11 min readReading

In brief

8 dimensions, levels 0 to 5, 112 criteria.
Tool: run an assessment by choosing "Advanced analytics".
Levels and criteria: original synthesis, references cited at the bottom of the page.

On this page

Overview
Scoping and preparation
Validation and production release
Trust and decision-making
1. Explainability and responsible use
2. Integration into decision-making
Sources and verification

Scope

The analytical capabilities beyond reporting built into the product: forecasts, anomaly detection, classification, recommendations, assistants. Scoping of use cases, data preparation, experimentation, validation, industrialization of models, monitoring, explanation and responsible use, integration into decision-making. Descriptive reporting belongs to the "Reporting" theme.

8 dimensions, six levels (0 to 5) and 112 observable criteria. To place your product: open the assessment for this theme. The scoring rules are here.

Overview#

Each cell is the level's signature for the dimension.

Dimension	0 · Handcrafted	1 · Locally tooled	2 · Shared	3 · Industrialized	4 · Integrated and governed	5 · Autonomous
Use case scoping	Analyses launched out of curiosity or because of a trend.	Ideas noted.	Use cases described.	Use cases prioritized.	Value tracked.	Portfolio steered.
Preparation of analysis data	Data extracted and cleaned by hand for each trial.	Datasets frozen.	Preparation scripted.	Features managed.	Preparation controlled.	Features shared.
Experimentation and reproducibility	Trials in personal notebooks, with no record of parameters.	Trials kept.	Trials tracked.	Reproducible environments.	Training automated.	Experimentation industrialized.
Validation and performance measurement	A model is judged good because the result looks plausible.	Simple measurement.	Structured validation.	Independent validation.	Validation in real conditions.	Continuous validation.
Industrialization of models	The model is a file sent by hand, used by its author.	Model handed to the application team.	Reproducible training.	Automated deployment.	Automated operation.	Full lifecycle.
Monitoring and drift	A model is assumed good as long as nobody complains.	Manual monitoring.	Regular monitoring.	Drift detected.	Response organized.	Automatic response.
Explainability and responsible use	The model is a black box used unconditionally.	Operation described.	Results explained.	Use framed.	Fairness examined.	Accountability verified.
Integration into decision-making	Results stay in study documents.	Results communicated.	Results in the reports.	Results in the process.	Feedback loop.	Assisted decision-making.

Link with the overall model

Dimensions of the overall model that are most relevant: Usage (level 5, assisted experience), Business processes and Industrialization and deployment. For analytics at the organization level, see the TDWI matrix.

Scoping and preparation#

Use case scoping#

Are advanced analyses chosen on the basis of expected value and feasibility?

Level	Signature	Criteria to meet
0 · Handcrafted	Analyses launched out of curiosity or because of a trend.	Starting point: no criteria.
1 · Locally tooled	Ideas noted.	Ideas for advanced analyses are collected in a list. Each idea states the business question it answers.
2 · Shared	Use cases described.	Each selected use case describes the targeted decision, the data needed and the user concerned. A success indicator is set before starting. A simpler solution (rule, threshold, cross-tab) is examined before a model.
3 · Industrialized	Use cases prioritized.	Use cases are ranked by value, cost and data availability. Feasibility is checked on a sample before committing to development. A business owner commits to using the result.
4 · Integrated and governed	Value tracked.	The value obtained from each use case in production is measured. A use case with no demonstrated value is stopped or reworked. Lessons learned from past use cases guide the choice of the next ones.
5 · Autonomous	Portfolio steered.	The portfolio of use cases is reviewed at regular intervals with the business. Reusable components (feature sets, base models) are pooled across use cases. Analysis needs emerge from the observed usage of the product.

Effect on the user

The product carries only analyses whose contribution is known and whose success can be recognized.

Preparation of analysis data#

Is the data used by models prepared, versioned and reproducible?

Level	Signature	Criteria to meet
0 · Handcrafted	Data extracted and cleaned by hand for each trial.	Starting point: no criteria.
1 · Locally tooled	Datasets frozen.	Study datasets are kept with their extraction date. Cleaning rules are written down.
2 · Shared	Preparation scripted.	Data preparation is a script, replayable on new data. Study data is drawn from the product's validated layer, not from copies. Training and test periods are separated, with no leakage of information from one to the other.
3 · Industrialized	Features managed.	Derived variables (calculated indicators, aggregates) are defined once and reused. The same preparation applies to training and to use in production. Training datasets are versioned.
4 · Integrated and governed	Preparation controlled.	The quality of input data is checked automatically before each training run. Personal data used is justified by a purpose and minimized. The provenance of each variable is traced back to the source.
5 · Autonomous	Features shared.	A shared feature store serves training and production with the same values. The freshness and health of features are monitored. A feature that has become useless or degraded is detected automatically.

Effect on the user

It is known what data a model was built with, and that data can be rebuilt identically.

Experimentation and reproducibility#

Are trials tracked, comparable and reproducible?

Level	Signature	Criteria to meet
0 · Handcrafted	Trials in personal notebooks, with no record of parameters.	Starting point: no criteria.
1 · Locally tooled	Trials kept.	The code of the trials is kept in a shared repository. The main results are noted.
2 · Shared	Trials tracked.	Each trial records its parameters, its data, its code and its results. Trials are comparable with one another using the same measures. A trial can be redone by another person.
3 · Industrialized	Reproducible environments.	Execution environments (library versions) are described and rebuilt identically. Random seeds and data versions are recorded. The move from an experiment notebook to tested code follows a defined rule.
4 · Integrated and governed	Training automated.	Training runs through an automatic pipeline, triggered on demand or on a fixed schedule. Models and their metadata are kept in a registry. A model can be rebuilt from its description, with no manual intervention.
5 · Autonomous	Experimentation industrialized.	Parameter search and model comparison are automated. Compute resources are allocated on demand and released after use. Every published result is linked to the trial that produced it.

Effect on the user

A result can be redone, and the best model is chosen on facts, not on someone's memory.

Validation and production release#

Validation and performance measurement#

Is it known how to measure that a model is good, and for whom?

Level	Signature	Criteria to meet
0 · Handcrafted	A model is judged good because the result looks plausible.	Starting point: no criteria.
1 · Locally tooled	Simple measurement.	A performance measure (error, precision) is calculated on data not used for training. The result is compared with a simple baseline (previous value, average).
2 · Shared	Structured validation.	The choice of measures follows from the targeted decision and the cost of errors. Performance is examined by segment (period, region, customer type), not only overall. The known limits of the model are written down.
3 · Industrialized	Independent validation.	Validation is done by someone other than the person who built the model. The model is tested with business users on concrete cases before go-live. A minimum performance threshold is a condition for production release.
4 · Integrated and governed	Validation in real conditions.	A new model is compared with the old one in parallel before replacing it. The performance observed in production is compared with that measured at validation. Decisions made with the model are matched with their actual outcomes.
5 · Autonomous	Continuous validation.	Validation is replayed automatically with each new training run. Comparative trials in real conditions (for example two versions in parallel) are run regularly. Performance gaps by segment trigger an alert.

Effect on the user

It is possible to say how reliably the model gets things wrong, and in which cases it should not be trusted.

Industrialization of models#

Does a trial become a model that is operated reliably, versioned and automated?

Level	Signature	Criteria to meet
0 · Handcrafted	The model is a file sent by hand, used by its author.	Starting point: no criteria.
1 · Locally tooled	Model handed to the application team.	The model is handed over with its description (inputs, outputs, version). The code that uses the model is versioned.
2 · Shared	Reproducible training.	Training is automated and reproducible. Training performance is tracked centrally. Models are managed in a registry with their version.
3 · Industrialized	Automated deployment.	Deploying a model follows an automatic pipeline, with unit and integration tests. Moving from one environment to another is automated. Traceability of the model in production goes back to the source data.
4 · Integrated and governed	Automated operation.	A model can be withdrawn or replaced by its previous version in a single operation. Production metrics trigger automatic retraining. Promoting a model is subject to a policy that is applied automatically.
5 · Autonomous	Full lifecycle.	The whole cycle (data, training, validation, deployment, monitoring) is automated and supervised. New models are added without infrastructure development. Downtime caused by model renewal is close to zero.

Effect on the user

A model in production is a component like any other: versioned, tested, deployed and supervised.

Monitoring and drift#

Does a model in production remain good when the data changes?

Level	Signature	Criteria to meet
0 · Handcrafted	A model is assumed good as long as nobody complains.	Starting point: no criteria.
1 · Locally tooled	Manual monitoring.	The model's performance is looked at from time to time. Glaring errors are corrected.
2 · Shared	Regular monitoring.	The model's inputs and outputs are recorded. Performance is recalculated at regular intervals when actual results become known. A degradation threshold is defined.
3 · Industrialized	Drift detected.	Input data drift (distribution, missing values) is measured. Drift in the relationship between inputs and outcome is measured as soon as outcomes are known. A threshold breach triggers an alert with an assigned owner.
4 · Integrated and governed	Response organized.	A procedure describes the response to drift: retraining, fallback, shutdown. A fallback to a simple rule exists when the model is no longer reliable. Users are notified when a model is suspended.
5 · Autonomous	Automatic response.	A drift or regression signal automatically triggers retraining. The replacement model is validated automatically before going live. The health of all models is visible on a common dashboard.

Effect on the user

A model that degrades is detected and dealt with before it distorts decisions.

Trust and decision-making#

Explainability and responsible use#

Can a result be explained, and is the use of the model framed?

Level	Signature	Criteria to meet
0 · Handcrafted	The model is a black box used unconditionally.	Starting point: no criteria.
1 · Locally tooled	Operation described.	The principle of the model is described in a document accessible to users. The data used is listed.
2 · Shared	Results explained.	Each result comes with the factors that influenced it most. A model card states its purpose, its limits and the cases where it should not be used. A result is presented with its degree of uncertainty.
3 · Industrialized	Use framed.	Important decisions made with a model remain validated by a person. Users can contest or correct a result, and the correction is recorded. Sensitive data or discriminatory criteria are excluded or justified.
4 · Integrated and governed	Fairness examined.	Differences in results between groups of people are measured. Each model has an owner responsible for its use and its effects. An impact review precedes the go-live of models that affect people.
5 · Autonomous	Accountability verified.	Fairness and explainability checks are automated with each new version. Model registries state their permitted uses and their regulatory obligations. Feedback from the people concerned is analyzed and taken into account.

Effect on the user

The user understands why the model suggests something, and knows when not to follow it.

Integration into decision-making#

Do advanced results reach the point of decision, and are they used?

Level	Signature	Criteria to meet
0 · Handcrafted	Results stay in study documents.	Starting point: no criteria.
1 · Locally tooled	Results communicated.	Results are communicated to the business as a report or a file. A business user is identified for each result.
2 · Shared	Results in the reports.	Results (forecasts, scores) are available in the reports, alongside actual figures. Modeled results are flagged as such, so they are not mistaken for facts. The date of the last calculation is displayed.
3 · Industrialized	Results in the process.	Results trigger alerts or tasks in the work tools. Alert thresholds are set with the business. The use of results is measured: viewed, acted on, ignored.
4 · Integrated and governed	Feedback loop.	Decisions taken after a result are recorded with their outcome. This feedback is used to improve the model and the thresholds. Results that are never acted on are identified and reworked or withdrawn.
5 · Autonomous	Assisted decision-making.	Users query the data and the models in plain language, with sourced answers. Routine decisions are automated within a defined framework, with handover to a person in case of doubt. The value of each automation is measured and reviewed.

Effect on the user

A forecast or an alert really changes a decision; otherwise it is not worth its cost.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and criteria are an original synthesis, not a reproduction of a framework. They are offered for discussion and adaptation, not as authoritative.

Sources and verification#

Microsoft Learn — MLOps Maturity Model (Azure Architecture Center) — five levels: no MLOps, DevOps but no MLOps, automated training, automated model deployment, fully automated operations; a level 4 where drift triggers retraining
dbt — Add data tests to your DAG — automatic data tests on inputs
Microsoft Learn — Fabric adoption roadmap: maturity levels — advanced analytics within the adoption path
GDPR, Article 5(1)(c) and (e) — text on the CNIL website (in French) — data minimization and storage limitation, applicable to training data