A compliance team improves its inspection completion dashboard from 70 to 96 percent. Leadership celebrates—until an auditor discovers that the increase came mainly from newly recorded inspections, not from additional inspections performed on time. Another organization launches an easier incident-reporting channel and temporarily sees higher incident counts. The two stories illustrate why data-collection quality and process performance need separate measures. One describes whether observations are captured accurately, completely and promptly. The other describes whether the underlying activity achieved its intended outcome. Treating one as proof of the other can reward the wrong behavior.
Collecting more complete data is not the same as improving actual process performance
Ask whether apparent improvement comes from more faithful observation or from changed process outputs. Compare capture coverage with independently verified work before drawing a conclusion.
01Define two different control objectives
Data collection aims to capture faithful, timely, authorized representations of events. Process performance concerns delivery reliability, control effectiveness, customer outcomes or other operational results. Neither can be inferred automatically from the other. A complete data table may contain systematically wrong values; a genuinely excellent process may have poor evidence because frontline recording is unreliable.
For an inspection process, measure whether inspections were due, performed and effective separately from whether the evidence reached the system with the correct ID and timestamp. Link both sets of indicators to the same transaction identifier. This permits diagnosis without inventing a new business outcome from a clerical status change.
Specify ownership at the start. The process owner accounts for execution and outcomes, while the data steward maintains capture, transformation and validation controls. Both are responsible for resolving cross-boundary anomalies rather than redirecting every dispute to IT.
Technical context: ISO 9001:2026 — Quality management systems — Requirements is one verified source for the concepts discussed here.
02Choose distinct indicators for data quality
Typical collection-quality dimensions include coverage, validity, timeliness, uniqueness, consistency and traceability. Their relevance depends on the operating context; not every measure needs every dimension. For a daily temperature record, timeliness and identity may be critical. For a monthly employee training survey, appropriate coverage and stable question wording could matter more.
Define each indicator with its own denominator. “Complete records” could mean all mandatory fields are populated, but that says nothing about whether all required events were created. A coverage indicator uses expected events as its population, while a field-completeness indicator uses received records. Confusing those denominators creates apparent improvement with little additional assurance.
ISO/IEC 25012:2008 provides a model of structured-data quality characteristics. It is a useful reference for selecting dimensions, but the organization must still determine controls that fit its process and obligations.
03Measure process outcomes with independent evidence
Process indicators should reflect what the operation promised or was expected to control. For a repair process, useful evidence may include customer acceptance, repeat failure rate or restoration within an agreed period. A record-submission rate measures neither repair success nor failure prevention. Where possible, challenge reported performance with an independent output or sampled physical verification.
Avoid designing a process measure that simply rewards documentation. If supervisors are paid for a low error rate, staff may record fewer problems instead of solving them. Review exceptions, rejected transactions and the ease of adverse-event reporting before attributing an improved percentage to better control.
When the data source is also a behavior affected by incentives, audit its completeness and look for independent traces such as customer complaints, equipment histories or dispatch records.
04Use a two-axis management view
A practical dashboard can present data confidence alongside operational outcome, creating four broad conditions. Strong process performance with strong data quality supports a confident conclusion. Weak performance with strong data supports corrective action. Weak data alongside apparently strong results calls for verification before celebrating. Weak data with weak results requires parallel process containment and collection repair.
These conditions are diagnostic, not a universal classification mandated by ISO. Determine materiality and action rules based on consequence. For high-risk activities, weak evidence may itself demand immediate assurance regardless of the apparent process score. For low-risk functions, a short sampling exercise may be a reasonable first response.
Use the view to allocate attention rather than to create another composite score. Do not average the two dimensions: a highly complete database cannot compensate for serious operational failures, and a good output cannot erase inadequate mandatory records.
05Investigate apparent improvement after new technology
Following a mobile-form rollout, recorded defects may increase because operators can finally submit issues promptly. Conversely, a migration may reduce visible exceptions when a field is dropped. Distinguish observation changes from process changes with a short parallel run or retrospective source-record sample. Annotate the date of system deployment and any revised definition.
Compare the same process population across old and new capture channels where feasible. Identify fields whose meaning changed, defaults that can produce artificial zeros and entries that can no longer be reconciled to source events. Use a transition period to learn whether increased reporting reflects broader detection, genuine deterioration or both.
Avoid imposing a performance penalty based solely on the first post-migration trend. Require investigation, but protect staff who report more faithfully under the improved system.
06Assign corrective actions to the correct mechanism
If transactions are late because staff cannot save forms offline, redesign capture or synchronization. If transactions are timely but actual work is late, investigate capacity, workflow or competence. If both are weak, sequence containment of critical operational risk alongside evidence restoration; do not postpone safety action while analysts reconcile the database.
Use distinct action owners, acceptance evidence and deadlines for collection and process improvement. A successful collection fix might be fewer unmatched event IDs; a successful operational fix might be sustained compliant outputs confirmed by samples. Record whether the two changes interact, so one project does not claim credit for an effect achieved by the other.
Monitor both dimensions for several periods after corrective action. A single clean month can reflect backlog clearance, selective reporting or changed population rather than lasting improvement.
07Design the management review narrative
When reporting a process KPI, disclose whether its data passed the quality threshold required for that decision. The report should not merely append a tiny data-quality footnote; a material coverage deficit can change the headline conclusion. Use specific explanations such as “capture coverage improved; operating performance has not yet been independently reassessed.”
Keep supporting evidence proportional and easy to reproduce. A transaction sample, an expected-event count and a clear definition of validity may be enough to demonstrate that the analytical conclusion is grounded. Require process and data owners to agree on the interpretation before it reaches leadership.
The distinction creates incentives to tell the truth. Better observation should be recognized as better observation, even when it initially makes operational outcomes appear less favorable.
Practical Review Questions
- Does this KPI describe capture or operational output?
- Do coverage and field completeness use different denominators?
- Is the result challenged by independent process evidence?
- Could improved reporting explain the apparent trend?
Launch a paired-indicator review
Select a recurring operational KPI where management has recently debated missing or inconsistent records. Give the data steward responsibility for completeness, validity and traceability, and the process owner responsibility for actual outcome.
Publish both results for a trial period and identify at least one disagreement. Trace it to source events, then decide whether the corrective action belongs in capture, execution or both. Keep the conclusions separate on the management review page. Build a paired process-and-data test for an activity where form completion is strongly incentivized. Sample scheduled tasks from a source independent of the completed forms; verify whether the tasks occurred and met the intended acceptance criterion. Separately inspect whether each performed task generated a timely, complete and traceable record. This produces two different findings: operational execution and recording reliability. If the first is strong and the second weak, repair capture design without unfairly alleging the work was absent. If records are excellent but task execution is poor, avoid assigning a data-cleanup action as the remedy. Present these findings as two columns in management review so the next decision targets the correct mechanism.
Closing Perspective
Good data capture is an enabler of trustworthy process improvement, not proof of it. Maintain separate ownership, indicators and corrective actions for the measurement system and the operation being measured.
References & Further Reading
Official ISO publications informing the measurement and data-quality method discussed above. Some ISO publications set requirements while others offer guidance or reference models. Select the applicable edition and the evidence appropriate to the organization.