Appendix C
Appendix C: Metric definitions
Every metric with its definition, the stage that owns it, its unit, and how it is aggregated.
| Metric | Definition | Unit and aggregation | Stage | Kind |
|---|---|---|---|---|
| intake_to_score_days | Days between a request entering the backlog and receiving its first score. | days, median | intake | KPI |
| backlog_age_days | How long scored requests have been waiting with no one starting the hypothesis. | days, median | intake | KPI |
| readiness_at_hypothesis | Of the requests that reach hypothesis, the share whose data is already collected in full and kept 30 days or more (readiness 3 or higher). | percent, ratio | hypothesis | KPI |
| research_hours_per_rule | Hours logged in research for one detection. | hours, median | research | KPI |
| percent_with_tests | Share of detections past the build stage that have both a test event to match and one to ignore. | percent, ratio | build | KPI |
| first_pass_validation_rate | Share of pull requests that pass every validation check on the first run. | percent, ratio | validate | KPI |
| review_turnaround_hours | Hours from a pull request being marked ready to a second engineer approving it. | hours, median | validate | KPI |
| deploy_lead_time_days | Days from the merge that sets a detection's stage to deploy to the merge that moves it to tune and switches its alerting on. | days, median | deploy | KPI |
| post_deploy_replay_failures | Promotion checks in the quarter that failed, because the first 7 days live returned more alerts than the threshold or none at all when research counted some. | count, sum per quarter | deploy | KPI |
| detection_latency_median | Minutes from the timestamp on the event to the alert arriving in the queue. It is the floor under every response time you report, because nobody responds before the alert arrives. Scheduled searches inherit their interval, so a detection running hourly cannot beat 30 minutes on average. | minutes, median per detection, 30 days | deploy | KRI, red over 60 minutes |
| precision_30d | Of a detection's alerts in the last 30 days that an analyst closed with an outcome, the share that were true positives. A benign true positive, behavior that happened and was allowed, counts against precision. An alert closed as undetermined counts on neither side. | ratio, per detection; program median | tune | KPI |
| triage_minutes_median | Typical analyst minutes spent on one alert from the detection. | minutes, median | tune | KPI |
| minutes_per_true_positive | Analyst minutes spent for each true positive the detection produces: triage minutes divided by precision. | minutes, per detection; program median | tune | KPI |
| upkeep_minutes_per_detection | Engineering minutes spent in a month keeping live detections working (precision checks, tuning pull requests, query fixes after a field or schema change), divided by the number of live detections. It replaces the 20-minute default in the maintenance ceiling once a quarter has been measured. | minutes, monthly, across the live set | tune | KPI |
| rule_debt | Share of live detections that fired at least 10 times in 180 days and were never right. It counts the same detections the second retirement trigger names, so it is the share of the live set that should already be retiring. | percent, ratio | retire | KRI, red over 20% |
| rules_retired_per_quarter | Detections set to deprecated in the quarter. | count, sum | retire | KPI |
| robustness_weighted_coverage | Per ATT&CK tactic, each live detection counts as its robustness divided by 5. A level-5 detection counts as one, a level-1 detection as one fifth. Shown beside the plain detection count. A tactic covered only by easily evaded detections needs five of them to count as much as one level-5 detection. It is a sum, so it grows with every detection added. The coverage app's weighted share is the per-technique version: each technique scores only its strongest live detection, so it tops out at 100%. | weighted count, per tactic | program | KPI |
| incident_detection_rate | Share of confirmed incidents in the period that a live detection alerted on first. The rest were found by a user report, a third party, an audit, or luck. Each incident this misses goes to intake as a candidate, with the incident as its evidence. | percent, per quarter | program | KPI |
| silent_source_hours | Hours in the quarter that a log source in the inventory sent nothing for longer than its silence threshold. Report the sources behind live detections first. | hours, per quarter, per source | program | KRI, red over 24 hours |
| priority_technique_gaps | Share of the program's priority techniques with no live detection. The quarterly review keeps the priority list: the techniques used by the threat groups the team tracks, or the top of the threat-led backlog. | percent, per quarter | program | KRI, red over 40% |
| misses_by_cause | Confirmed incidents and exercises in the quarter that no live detection caught first, counted by the miss code the detection review recorded. | count, per quarter, by miss code | program | KRI, red the same code in two quarters running |
| open_detection_fixes_age | Days the oldest fix from a detection review has waited without passing validate, counted from the day the incident was confirmed. | days, oldest open, at quarter end | program | KRI, red over 60 days |
| false_close_rate | Share of alerts later confirmed malicious that triage had closed as benign, by a person or an agent. Section 19's ai_false_close_rate is the same count for an agent alone. | percent, per quarter | program | KRI, red a wrong close on a detection rated impact 4 or 5 |
| time_to_scope_hours | Hours from triage escalating an alert to the case recording the incident's scope: which hosts, accounts, and data. | hours, median per quarter | program | KPI |
| time_to_contain_minutes | Minutes from an incident being confirmed to the first containment action completing. | minutes, median per quarter | program | KPI |
| notifications_on_time | Share of required notifications, to leadership, customers, or a regulator, sent within their deadline. | percent, per quarter | program | KRI, red any regulator or customer deadline missed |
| playbook_coverage | Share of alerts in the quarter whose detection has a live playbook at tier 1 or higher. | percent, per quarter | program | KPI |
| playbook_minutes_saved_per_alert | The manual baseline minutes on the playbook's record, minus the analyst minutes an alert takes with the playbook running, per alert class. | minutes, median per quarter | program | KPI |
| playbook_actions_unattended | Actions a playbook completed with no person in the loop, counted by ladder tier. A tier 4 count above zero names the expendable asset the record allows. | count, per month, by tier | program | KPI |
| playbook_actions_reverted | Playbook actions a person refused in propose mode or undid after they ran. Each one is a tuning pull request. | count, per quarter | program | KRI, red a reverted tier 4 action, or a refused close on an impact 4 or 5 alert |
| incidents_without_followup | Confirmed incidents that produced no intake candidate, no tuning pull request, and no log source request within 30 days of confirmation. | count, per quarter | program | KRI, red the same responder twice, or any incident in two quarters running |
DETECT by Habibullah Tora, https://detect.htora.dev, licensed CC BY-NC 4.0. Version 0.136.0.