clement13.net
Original photo: A warehouse
Description
A warehouse
Source
Casey Deshong · original page
Licence
PD Public domain
Original photo, unretouched. The background is a version of it stylised by the site.

Matrix: Storage

Where and how the product’s data is stored, modeled, historized, protected, retained and then removed.

≈ 12 min readReading
In brief
  • 8 dimensions, levels 0 to 5, 110 criteria.
  • Tool: run an assessment by choosing “Storage”.
  • Levels and criteria: original synthesis, references cited at the bottom of the page.
On this page
  1. Overview
  2. Organization and model
    1. Layered organization
    2. Model and integrity
  3. Data lifecycle
    1. Historization and traceability
    2. Retention and lifecycle
  4. Protection
    1. Backup and restore
    2. Storage security
  5. Performance and cost
    1. Physical performance
    2. Capacity and cost
  6. Sources and verification
Scope

What concerns the retention of the product’s data: locations and layers, table model, history, backup, retention period, storage security, physical performance and cost. The flows that feed the storage belong to the “Ingestion” theme, those that transform it to the “Processing” theme.

8 dimensions, six levels (0 to 5) and 110 observable criteria. To place your product: open the assessment for this theme. The calculation rules are here.

Overview#

Each cell is the level’s signature for the dimension.

Dimension0 · Handcrafted1 · Locally tooled2 · Shared3 · Industrialized4 · Integrated and governed5 · Autonomous
Layered organizationData dumped in bulk into files or a single database, with no distinction between raw and usable.One location per source.Raw separated from usable.Formalized layers: raw, validated, ready to use.Layers by business domain, under the responsibility of owners.Layers driven by versioned definitions and monitored automatically.
Model and integrityTables copied from the source systems, with no model of their own.Tables identified and named.Facts separated from analysis axes.Relationships declared and integrity checked.Model documented, changes reviewed.Model evolving without breakage, blocking checks.
Historization and traceabilityOnly the latest value is kept; the previous one is overwritten.Dated copies from time to time.Dates and source carried by the rows.Dimension history and atomic loads.Replay and provenance.History queryable and explained automatically.
Retention and lifecycleEverything is kept indefinitely, or deleted at random.Useless data identified.Periods written down by data family.Periods justified and purges traced.Archiving, minimization and erasure propagated.Policy applied and checked automatically.
Backup and restoreNo backup of its own: you rely on the storage itself.Full backup, manual or scheduled.Automatic, monitored backups, restore described.Recovery objectives set and restore tested.Point-in-time restore, protected copies.Recovery rehearsed and automated.
Storage securityShared accounts or uncontrolled access to storage locations.Named accounts.Rights by group, encryption at rest, secrets protected.Rights reviewed, access logged, network restricted.Data classified, protected according to class.Automatic detection and control.
Physical performanceTables with no indexes and no thought given to physical layout.Indexes on keys.Indexes chosen from actual queries.Storage adapted to usage, maintenance scheduled.Aggregates and load tests.Optimization guided by measured usage.
Capacity and costStorage size unknown; you find out when it is full.Free space checked by hand.Volume measured, alert before saturation.Growth forecast, cost known, storage tiers.Costs allocated, cold data moved.Automatic sizing and cost anomalies.
Link with the global model

The dimensions of the global model most concerned are Data sources and flows, Data quality, Security and compliance and Availability and performance; this theme details their storage side.

Organization and model#

Layered organization#

Are raw, validated and ready-to-use data distinguished, and who can read what?

LevelSignatureCriteria to meet
0 · HandcraftedData dumped in bulk into files or a single database, with no distinction between raw and usable.Starting point: no criteria.
1 · Locally tooledOne location per source.
  • Each data source has a dedicated, named location.
  • Data received is kept as is before any transformation.
2 · SharedRaw separated from usable.
  • A raw zone is distinct from the zone used by reports.
  • Raw data is not modified after it is deposited.
  • Reports read only the usable zone.
3 · IndustrializedFormalized layers: raw, validated, ready to use.
  • Three layers, or their equivalent, are named and documented.
  • A written rule says what is allowed in each layer (for example: no cleansing in the raw layer).
  • The layers have separate spaces and access rights.
4 · Integrated and governedLayers by business domain, under the responsibility of owners.
  • Each business domain has its own ready-to-use layer.
  • Each dataset has an owner and a contract: expected structure, freshness, scope.
  • Any data in the validated layer can be rebuilt from the raw layer.
5 · AutonomousLayers driven by versioned definitions and monitored automatically.
  • The creation and evolution of layers and schemas are automated from versioned definitions.
  • Differences between layers (volumes, structure) are detected automatically.
  • Datasets that are no longer used are identified and proposed for removal.
Effect on the user

Reports read only data whose level of trust is known; a source error does not contaminate the published figures.

Model and integrity#

Do the tables follow a model designed for analysis, and is integrity verified?

LevelSignatureCriteria to meet
0 · HandcraftedTables copied from the source systems, with no model of their own.Starting point: no criteria.
1 · Locally tooledTables identified and named.
  • Each table has an identified key.
  • Table and column names follow a written convention.
2 · SharedFacts separated from analysis axes.
  • Analysis data is separated into fact tables (measures) and dimensions (axes).
  • The grain of each fact table, that is, what a row represents, is defined in writing.
  • Join keys are stable: surrogate keys or codes guaranteed to be unique.
3 · IndustrializedRelationships declared and integrity checked.
  • Relationships between tables are declared (foreign keys or equivalent), and their activation is justified.
  • Common dimensions are shared across fact tables.
  • Integrity anomalies (orphan rows, duplicate keys) are detected on every load.
4 · Integrated and governedModel documented, changes reviewed.
  • The model is documented (schema, column descriptions) from the database metadata.
  • Any model change goes through a review and an impact check on consumers.
  • The same concept has the same type and the same nullability in all tables.
5 · AutonomousModel evolving without breakage, blocking checks.
  • Model changes are versioned, and their compatibility with consumers is tested.
  • A failing integrity check prevents the dataset from being published.
  • Model differences between environments are detected automatically.
Effect on the user

The same figures come out of all reports, and a join does not silently change a total.

Data lifecycle#

Historization and traceability#

What happens to a value that changes, and can you tell where a row comes from?

LevelSignatureCriteria to meet
0 · HandcraftedOnly the latest value is kept; the previous one is overwritten.Starting point: no criteria.
1 · Locally tooledDated copies from time to time.
  • A dated copy of the data is kept at irregular intervals.
  • Copies are named with their date.
2 · SharedDates and source carried by the rows.
  • Each row carries its load date and its source.
  • Fact history is kept by appending, without overwriting.
  • The historization rule for each dimension is chosen and written down: overwrite, add a row, or add a column.
3 · IndustrializedDimension history and atomic loads.
  • Significant changes in dimensions are historized with their validity dates.
  • The state of the data at a past date can be reproduced.
  • An interrupted load does not leave partial data (transaction or atomic replacement).
4 · Integrated and governedReplay and provenance.
  • A period can be reloaded from the raw layer without creating duplicates.
  • Retroactive corrections in the source are taken into account and traced.
  • The provenance of each row (source, batch, rule version) can be queried.
5 · AutonomousHistory queryable and explained automatically.
  • The history of each table can be queried at a given date or version.
  • The difference between an old output and the current one is explained automatically (batch, modified rule).
  • Historization rules are covered by automated tests.
Effect on the user

You can answer “what was the figure on that date?” and explain why a figure has changed.

Retention and lifecycle#

How long is each piece of data kept, why, and how is it removed?

LevelSignatureCriteria to meet
0 · HandcraftedEverything is kept indefinitely, or deleted at random.Starting point: no criteria.
1 · Locally tooledUseless data identified.
  • Data that is no longer used is identified.
  • An identified person decides what is deleted.
2 · SharedPeriods written down by data family.
  • A retention period is written down for each major family of data.
  • Personal data is identified in the storage.
3 · IndustrializedPeriods justified and purges traced.
  • Each period is justified by a purpose or an obligation (storage limitation principle).
  • A purge or an archive is run at regular intervals.
  • Each purge is traced: what was removed, when, in what quantity.
4 · Integrated and governedArchiving, minimization and erasure propagated.
  • Rarely consulted data is archived on cheaper storage, and remains retrievable.
  • Personal columns that are not needed for analyses are not kept in the usable zones (minimization).
  • An erasure or rectification request is propagated to all layers, according to a written rule for backups.
5 · AutonomousPolicy applied and checked automatically.
  • Periods are enforced by automatic lifecycle rules.
  • Deviations from the retention policy are detected automatically.
  • Periods are reviewed at fixed intervals with their owners.
Effect on the user

You keep neither too much (risk and cost) nor too little (loss of useful history); erasure requested by a person is possible.

Protection#

Backup and restore#

If data is lost or corrupted, how quickly and how completely can it be recovered?

LevelSignatureCriteria to meet
0 · HandcraftedNo backup of its own: you rely on the storage itself.Starting point: no criteria.
1 · Locally tooledFull backup, manual or scheduled.
  • A full backup exists.
  • It is kept in a different location from the original data.
2 · SharedAutomatic, monitored backups, restore described.
  • Backups are scheduled automatically and their success is monitored.
  • The frequency takes into account the acceptable data loss.
  • A restore procedure is written down.
3 · IndustrializedRecovery objectives set and restore tested.
  • A recovery time objective (RTO) and an acceptable data loss (RPO) are set for each dataset according to its criticality.
  • Restore is tested regularly, and the dated result is kept.
  • A copy is kept outside the original site or region, with credentials separate from those of production.
4 · Integrated and governedPoint-in-time restore, protected copies.
  • Data can be restored to a precise point in time.
  • Copies are protected against deletion and modification (immutable or write-once).
  • The restore order takes into account the dependencies between data stores.
5 · AutonomousRecovery rehearsed and automated.
  • Recovery exercises are run at regular intervals, measuring the actual RTO and RPO.
  • Restore is automated and can be triggered by script.
  • The recovery plan is reviewed with every architecture change and at least twice a year.
Effect on the user

An outage or a handling error costs a planned recovery, not the loss of the product.

Storage security#

Who can read, modify or take away the stored data, and how do you know?

LevelSignatureCriteria to meet
0 · HandcraftedShared accounts or uncontrolled access to storage locations.Starting point: no criteria.
1 · Locally tooledNamed accounts.
  • Access to storage locations goes through named accounts.
  • No storage location is publicly accessible.
2 · SharedRights by group, encryption at rest, secrets protected.
  • Access rights are granted by group on each location.
  • Encryption at rest is enabled.
  • Secrets (keys, connection strings) do not appear in clear text in scripts or settings files.
3 · IndustrializedRights reviewed, access logged, network restricted.
  • Access rights are reviewed at regular intervals.
  • Access to sensitive data is logged.
  • The storage is accessible only through a private network or a list of authorized addresses.
4 · Integrated and governedData classified, protected according to class.
  • Data is classified by sensitivity, and a storage rule matches each class (masking, pseudonymization).
  • Encryption keys are managed: rotation, separation of roles.
  • Test environments do not contain real, unmasked data.
5 · AutonomousAutomatic detection and control.
  • Abnormal access and misplaced sensitive data are detected automatically.
  • Rights are granted and withdrawn automatically, including when a person leaves.
  • Compliance of storage settings is verified continuously, with exportable evidence.
Effect on the user

Sensitive data stays within its perimeter, and abnormal access leaves a trace.

Performance and cost#

Physical performance#

Does the physical layout of the data (indexes, partitions, formats) serve the actual queries?

LevelSignatureCriteria to meet
0 · HandcraftedTables with no indexes and no thought given to physical layout.Starting point: no criteria.
1 · Locally tooledIndexes on keys.
  • The keys of each table are indexed.
  • Someone can list the largest tables.
2 · SharedIndexes chosen from actual queries.
  • The slowest queries are identified and measured.
  • Indexes are chosen from the queries actually executed.
  • The statistics used by the engine are kept up to date.
3 · IndustrializedStorage adapted to usage, maintenance scheduled.
  • The storage format is adapted to usage (columnar and compression for analysis, row-based for frequent updates).
  • Large time-based tables are partitioned or split.
  • Index maintenance is scheduled.
4 · Integrated and governedAggregates and load tests.
  • The most consulted calculations are precomputed (aggregates, materialized views).
  • Load tests are run on a realistic volume.
  • A degradation in query times is detected after every change.
5 · AutonomousOptimization guided by measured usage.
  • Measured usage triggers index or aggregate recommendations.
  • Compute capacity adjusts automatically to load.
  • A performance threshold is part of the automatic acceptance criteria of a delivery.
Effect on the user

Reports display quickly and consistently, even as the data grows.

Capacity and cost#

Do you know how much space you use, how much it costs, and when it will run out?

LevelSignatureCriteria to meet
0 · HandcraftedStorage size unknown; you find out when it is full.Starting point: no criteria.
1 · Locally tooledFree space checked by hand.
  • Free space is checked at regular intervals.
  • The size of the largest datasets is known.
2 · SharedVolume measured, alert before saturation.
  • The volume of each dataset is measured regularly.
  • An alert is triggered before saturation.
3 · IndustrializedGrowth forecast, cost known, storage tiers.
  • Growth is forecast over twelve months.
  • The storage cost is known by environment and by major dataset.
  • Distinct storage tiers exist (frequently read data, rarely read data).
4 · Integrated and governedCosts allocated, cold data moved.
  • Each storage cost is attached to an owner.
  • Rarely read data is moved to a cheaper tier according to a rule.
  • Non-production environments are sized to need or shut down when not in use.
5 · AutonomousAutomatic sizing and cost anomalies.
  • Storage sizing adjusts automatically.
  • A cost anomaly is detected automatically.
  • The trade-off between cost and performance is reviewed from measurements.
Effect on the user

Data growth is a budgeted decision, not a surprise.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and the criteria are an original synthesis, not a reproduction of a framework. They are offered to be discussed and adapted, not as authority.

Sources and verification#