IdeaIntake, Hypothesis, Research
Section 6
Prioritization
A backlog score decides what earns engineering time next. It blocks candidates that cannot be built, penalizes ones that cost many build hours or analyst minutes, and discounts ideas that duplicate live coverage.
Do it in the browser Score a backlog
Remember
- Every number in a score needs an evidence line a reviewer can check.
- Never type a score by hand. The scorer writes it, and CI fails any score that disagrees with the formula.
- A telemetry readiness of 0 blocks the candidate, and you fix that with a log source request.
- Mandates due within 90 days go first, in order of due date.
- Scores in the same 0.2 band count as tied, and the cheaper detection goes first.
- A candidate the live set already covers scores half, unless it catches a variant the live detection misses.
- Publish the weights, so everyone can see why the backlog is in its order.
On this page
6.1The inputs
| Input | Scale | Question |
|---|---|---|
| threat_relevance (T) | 1 to 5 | How recently, and how close to you, has this technique been used? |
| impact (I) | 1 to 5 | What does the attacker reach if the technique succeeds? |
| robustness (B) | 1 to 5 | How hard is it for an attacker to slip past this detection using the data you have? Score it with the five Summiting the Pyramid levels: 1 matches something the attacker can change in seconds, 5 matches something the technique cannot work without. |
| telemetry_readiness (R) | 0 to 5 | Is the data this detection needs already in the SIEM, and in good enough shape to use? |
| build_effort (E) | 1 to 5 | How many hours of work from writing the hypothesis to passing validation? |
| run_cost (C) | 1 to 5 | How many analyst minutes a day will this detection consume? Expected alerts per day multiplied by the typical minutes to triage one. For a detection that runs as scheduled SQL, add the warehouse compute it burns, converted the same way your finance team converts it. |
Appendix B has the wording for every level of every input.
Each input comes with one line saying where its number came from: an incident id, a report section, a query, or a ticket. CI rejects a record missing any of them.
6.2The formula
value = wT·T + wI·I + wB·B
cost = wE·E + wC·C
score = (value / cost) × (R / 5)^k × M
A readiness of 0 sets the score to 0 and marks the detection blocked. M is the marginal coverage factor from section 6.3. Scores round to two decimals. With every weight at 1 and k = 1, scores run from 0.06 to 7.50. The shape comes from Weighted Shortest Job First in SAFe, which divides value by size. I added readiness as a factor because you can't build anything on missing telemetry, and the marginal factor so a detection is scored on the coverage it contributes. The 0.8 and 0.5 behind that factor are defaults I chose so a stronger replacement still outranks most new work and a duplicate falls below it. Nothing derives them. If your program disputes them, change them in score.py and say why in the pull request.
Six one-to-five judgments carry noise, so the order sorts scores into fixed bands 0.2 wide (2.40 to 2.59, 2.60 to 2.79, and so on). The width is a chosen default until a team measures how far its scorers disagree. Scores in one band count as tied, and the one with the lower run cost goes first. The band is the rule, so two scores on either side of a band edge are ordered even when they differ by less than 0.2.
6.3Additions to an existing program
Scored one at a time, the eighth detection for a covered technique scores the same as the first for an uncovered one. The marginal coverage factor fixes that by scoring a detection on what it adds to what already fires.
| It comes back | Factor | When | What to do about it |
|---|---|---|---|
| first | ×1.0 | No live detection names this technique. | Full value. This is coverage the program lacks. |
| stronger | ×0.8 | The technique has a live detection, and this candidate scores higher on robustness than the best one live. | Most of the value. It replaces something brittle, so plan the retirement of what it supersedes in the same change. |
| equal | ×0.5 | A live detection already covers the technique at the same robustness level or higher. | Half. Build it only if it catches a variant the live one misses, and say which variant in the hypothesis. |
The pipeline computes it from the live set, comparing ATT&CK technique and robustness level. It is written into the record like the score; a hand-typed value fails the build. With no live set to compare against, every candidate counts as first.
A candidate that comes back stronger ships in the same change as the retirement of what it supersedes, because leaving both live doubles the alerts for one technique. A candidate that comes back equal is a duplicate until somebody writes down the variant it catches that the live one misses. If the hypothesis can't name that variant, close the candidate.
6.4Presets
| Preset | Change | Use when | Where it goes wrong |
|---|---|---|---|
| balanced | none | The default. One score for the whole backlog. | Only as good as the six numbers, so every number needs its evidence line. |
| threat_led | wT=2 | You have your own incident history or strong reporting about your sector. | Ranks by what attackers do, so it can pull effort toward techniques your logs can't see yet. |
| asset_led | wI=2 | You know which systems matter most and how an attacker would reach them. | Only works with a current asset inventory. |
| telemetry_led | k=2 | A new program that needs early wins from the data already in the SIEM. | Favors what your logs already show and can leave the biggest threats uncovered. |
A detection required by a regulation or an audit carries a mandate with the framework, the requirement number, and a due date. If the due date is within 90 days, it goes to the front of the queue, ordered by date. Everything else is ordered by score, highest first; ties go to the detection with the lower run cost. A mandated candidate with telemetry readiness 0 is still blocked; the mandate moves its log source request to the front of that list instead. Mandated detections come with budgets and deadlines of their own.
6.5A worked example
DET-0001, Kerberoasting (T1558.003), scores 4, 4, 4, 4, 2, 2 on the balanced preset: value 12, cost 4, scaled by 4/5, score 2.40. DET-0002, a scheduled task created by a non-admin user, scores 3, 2, 3, 3, 2, 4: value 8, cost 6, scaled by 3/5, score 0.80. The gap comes from run cost and readiness. Under threat_led the two score 3.20 and 1.10; under telemetry_led, 1.92 and 0.48. The order doesn't change under any of the three.
6.6The floor, and when a detection has to go
A detection's floor is its typical triage minutes divided by the program cap on analyst minutes per true positive. The cap is 120 by default. A detection that takes 30 minutes to triage must be right at least 25% of the time; one that takes 6 minutes, 5%.
Any one of these triggers proposes retirement, and the lead makes the call:
- Its true-positive share stayed below its floor for two 30-day periods in a row.
- It fired at least 10 times in 180 days and was never right.
- The log source it needs has been missing for 30 days straight, so its readiness fell to 0. Suspend it with alerting off and raise a log source request, and retire it only if the source owner declines to restore the source.
- The technique or the system it protects is gone, for example because the platform was decommissioned.
- Another live detection covers the same technique at an equal or higher robustness level and catches everything this one catches. Retire the weaker one in the same change that ships the stronger one.
A detection that hasn't fired at all in 365 days gets a review before anyone retires it, because either its data stopped or its hypothesis was wrong.