DETECT hunt Run and measure

Handling objections

Bring this into a working program and people will push back, usually in the first meeting. These are the arguments I'd expect, with my answer to each. Some of them are at least partly right, and for those my answer is to agree.

For the rolloutFramework v0.136.0. 10 objections.

Remember

  • Concede early on the four objections that are partly right: the record cost on a small team, the score's precision, the size of a move to code, and fixing telemetry first.
  • Answer with your own program's numbers where you have them.
  • Keep the vendor rules and score them like any other detection.

Objections that are partly or wholly right: 4 of 10

On this page

"Our SIEM ships hundreds of rules. Why write our own?"

Usually from: The platform owner, usually in month one

Keep them. Nobody is asking you to rewrite a vendor's work out of pride. Score the vendor pack like anything else, enable it in batches you can measure against the alert budget, and retire what doesn't clear its floor. Switch on a whole pack at once and you've made a coverage claim nobody can check and signed your analysts up for alert volume nobody agreed to.

"Writing a record for every detection is overhead we cannot afford."

Usually from: The engineer who is already behind

For detections already live, start with four of the 11 sections: goal, categorization, blind spots, and response. That takes 30 minutes a detection, and most of it is remembering what you already know. Lint wants every section written from validate on, so keep these short records outside the kit's detections folder until the other seven are written. For new detections, you write the rest as they move through the stages. Skipping the record only moves the cost. The next person re-derives the intent from the query, and the responder at 3 a.m. has no response steps and a strong opinion about whoever wrote the rule.

What it gets right. That's true for a program of 10 detections that one person owns and will keep owning. It stops being true the first time that person takes a vacation.

"Our MSSP or vendor manages detections for us."

Usually from: The manager who signed the contract

Then the definitions of done and the metrics become contract terms. Ask for the record, the fixtures, the precision per detection, and the retirement list. A provider who can't produce precision per detection is selling you alert volume with no measure of quality. Your analysts still spend the alert budget, whoever wrote the rules.

"We already report ATT&CK coverage. This duplicates it."

Usually from: Whoever built the heatmap

Keep reporting it and add the weighted number beside it. If the two are close, the heatmap was honest and you've lost nothing. If they're far apart, the gap is the finding, and it's much nicer to find it yourself than to have a post-incident review find it for you.

"Scoring is false precision. Six numbers do not capture the judgment."

Usually from: The most experienced engineer in the room, and they have a point

The score records the judgment, so somebody other than its author can inspect it. The value is in the evidence line under each number, and that's what a reviewer argues with. The order treats differences under 0.2 as ties for the same reason. I picked that width as a default, because nobody has measured how far two scorers disagree, me included.

What it gets right. The objection is right about precision. A score of 2.40 and a score of 2.52 are the same score, which the band rule states and the sort implements.

"We cannot retire detections. Audit will ask why coverage dropped."

Usually from: The compliance lead

A retirement with a note is a stronger audit position than a detection nobody has looked at in a year. The note says which trigger fired, what you measured, and what replaced it. If an auditor is going to object to something, it should be counting a detection that fired for 180 days and was never right as coverage.

"This is too heavy for a team of two."

Usually from: The team of two

Then take the four practices section 3 gives a team of two to four: the scored backlog, the definitions of done, the detection record, and the retirement triggers. Defer the full metric set, but keep precision and triage minutes per detection, because the retirement triggers need them. Keep the review too. One of you builds and the other reviews, so neither of you approves your own detection.

"Our detections are in the SIEM console. Moving to code is a six-month project."

Usually from: Anyone who has tried it

Section 3 puts detection as code last for teams under five engineers. Records, scores, definitions of done, and retirement all work with the console. What code adds is a build that enforces the definitions of done, including on the day a deadline makes skipping them look reasonable.

What it gets right. Migrating is a project with its own plan and budget. Do it when hand-converting queries becomes the bottleneck, around 200 live detections.

"Nobody will attend four more meetings."

Usually from: Everybody

I don't like meetings either. The four take about three hours and 40 minutes a month for a small team: three standing ones, plus 45 minutes after each incident. The three standing ones replace something you're already doing badly: an ad-hoc request queue, a tuning conversation in chat, a quarterly slide nobody can source. The post-incident review is the only new one, and it produces the intake items with the strongest evidence.

"Our telemetry is a mess. We should fix that first."

Usually from: The person who is right

I agree, and the framework makes that answer visible. Every candidate gets a readiness score, a zero blocks the build, and the backlog then shows which log source unblocks the most value. That list is the telemetry business case, and you get it as a side effect of doing the detection work.

What it gets right. Telemetry work outranks detection work whenever telemetry readiness blocks more than about a third of the scored backlog.

Concede the four marked above first, then answer the rest.

Next step Checks against outside sources