Lesson 3.58: Validating the metrics you report
Boards decide on the numbers they're given, so a wrong KRI is worse than none: it gives false comfort. Metrics go wrong in quiet ways: the definition narrows over time (only laptops, only open status), exceptions are excluded without saying so, the time window shifts. Validation means recomputing a reported number from raw data with the stated definition, and comparing. In the lab you recompute two KRIs from last quarter's board pack, find that one silently excluded inconvenient items and the other measured a much smaller population than its name suggests, and restate both.
19 min
You will learn to
- Recompute a reported metric from raw data
- Find silent exclusions
- Find definition drift
- Tell scope decisions from cherry-picking
- Restate metrics transparently
Key terms
- Recomputation
- Calculating a reported number again from raw data using its stated definition.
- Definition drift
- When what is measured quietly changes from what the metric's name implies.
- Silent exclusion
- Leaving items out of a metric without disclosing it.
- Cherry-picking
- Choosing data or scope that makes a result look better.
- Restatement
- Correcting a previously reported number and explaining why.
← Continuous control monitoring · All lessons · The remediation programme after a major breach →