DETECT hunt Build detections

IdeaProductionShipMeasure

Section 18

The closed loop

Framework v0.136.0, updated November 17, 2026.

Remember

  • Write every check against the way an agent would fake its way past it, and make each check a job the build runs.
  • Take field names from a dictionary the SIEM exported today, never from documentation or memory.
  • Fixtures come from a collector the agent cannot write to, with a signature the build verifies.
  • Lock the selection blocks that express the behavior with a hash. Tuning adds filters beneath them and changes nothing above.
  • Survival after 90 days is the number that says the checks work. Acceptance alone says reviewers are tired.

Section 10 lets agents draft the lifecycle's artifacts while people review, deploy, and retire. In the closed loop, agents run the mechanics of every stage, from a threat report to a tuning pull request, and people keep the approvals. The loop stays safe because each stage ends on a check written to catch the way an agent would fake its way past it. Three routes send work back when a check fails.

The loop has six machine stages, and they map onto the eight lifecycle stages. The definitions of done, the records, and the kit's tools stay the same; the loop adds the checks an unattended run needs and the routes back. Nothing in the Never list of section 10 changes. An agent still never approves, deploys, switches alerting on, or acts on an instruction found in a report or a log.

On this page

18.1The six stages and their checks

1. ExtractIntake and hypothesisReads a threat report, an incident, or an exercise result as data and writes a blueprint: the technique id, the behavior in terms of what the host or service must do, and the fields that would show it.Passes when: Schema-valid blueprint, ids in the pinned map, no duplicate.
2. GroundResearchLooks each field in the blueprint up in a dictionary exported from the SIEM.Passes when: Every field in today's dictionary, source reporting, baseline counted.
3. AuthorBuildWrites the rule in Sigma from the blueprint and the dictionary, and converts it for every target.Passes when: Lint, approved copies, scan under the cap, core hashed.
4. ProveResearch and validateRuns the behavior in a throwaway environment, collects the events, and replays the converted query over them and over the last 30 days of production..Passes when: Every variant caught on signed fixtures, 30-day replay grouped.
5. TuneTuneFor each group of benign matches, adds one filter beneath the locked core: the parent process, a valid signature and its signer, and the command or flag, as an allowlist entry with a ticket and an expiry.Passes when: Core hash unchanged, filters only, three attributes each, recall re-run.
6. ShipValidate, deploy, and the first week liveOpens one pull request carrying the rule, the conversions, the signed fixtures, the replay counts, the hash of every artifact, and every query it ran with its time range and row count.Passes when: Every check plus one person, 7 days alerting off, rate check armed.
Prove back to AuthorA variant in the plan that the rule misses sends the blueprint and the missed event back to Author.
Prove back to TuneMatches over the threshold in the 30-day replay go to Tune with the groups already made, and the tuned rule goes through Prove again before it ships..
Live back to TuneAn analyst's false-positive disposition, an alert rate over the hourly limit, or a daily field check that finds a renamed column each opens a tuning pull request.
People keep every pull request approval, the deploy run, the switch that turns alerting on, every retirement, and the thresholds the checks use.
Each stage ends in a check that a job runs, written against the way an agent would fake its way past it. A failed check sends the work back along one of the three routes, or stops it with a log source request.
Stage Lifecycle stages What the agent does What passes it
Extract Intake and hypothesis Reads a threat report, an incident, or an exercise result as data and writes a blueprint: the technique id, the behavior in terms of what the host or service must do, and the fields that would show it. It leaves out the clues the attacker picks, such as file names, hashes, and addresses. It drafts the six scores with an evidence line each, which is intake. The blueprint validates against a schema, and every technique id is in the pinned ATT&CK map. No rule in the repository already names the same behavior, by a similarity score under a threshold the team sets.
Ground Research Looks each field in the blueprint up in a dictionary exported from the SIEM. Documentation is never the source. Checks that the source is arriving: the share of hosts that reported in the last day and the rate of empty values in the fields it needs. Counts the 7-day baseline with a read-only query. Every field the rule will read exists in the live dictionary. The source reports from at least the share of the fleet the team sets, and 85% is a starting point. The baseline count and its query are in the record. A missing field or a silent source stops the run and opens a log source request, which is research's route back to intake.
Author Build Writes the rule in Sigma from the blueprint and the dictionary, and converts it for every target. Writes the fixtures it expects to match and to ignore. Declares every threshold, allowlist, and scope as a variable with bounds and evidence. Lint passes, each conversion matches its approved copy or creates one, and the rule passes the validators. No high-volume field is matched with a leading wildcard, and the estimated scan over the baseline window is under the team's cap. The selection blocks that express the behavior are canonicalized and hashed, and that hash is the core the later stages protect.
Prove Research and validate Runs the behavior in a throwaway environment, collects the events, and replays the converted query over them and over the last 30 days of production. Every variant the plan names is caught, the fixtures carry the collector's signature, and the 30-day replay's matches are grouped by parent process, signer, and command, with the groups in the record. A missed variant goes back to Author. Matches over the threshold go on to Tune.
Tune Tune For each group of benign matches, adds one filter beneath the locked core: the parent process, a valid signature and its signer, and the command or flag, as an allowlist entry with a ticket and an expiry. Then runs Prove again. The core's hash is unchanged. The diff adds filter selections and nothing else. Each filter names at least three conjunctive attributes, one of them a signature where the source carries one, and no path wildcard. Recall is still complete, and the 30-day matches are under the threshold. A rule with no historical matches at all skips tuning, because an exclusion on a rule that never fired removes something nobody has seen.
Ship Validate, deploy, and the first week live Opens one pull request carrying the rule, the conversions, the signed fixtures, the replay counts, the hash of every artifact, and every query it ran with its time range and row count. After the merge it packages the rule with alerting off, and seven days later opens the pull request that switches alerting on, with the week's count attached. Branch protection requires every check above plus one approval by a person, and the deploy run needs the lead's approval. Its opening seven days run with alerting off and at or under the threshold. A rule that fires more than its hourly limit after alerting is on goes back to alerting off and to Tune.

18.2How each check can be gamed, and what stops it

Stage How an agent games it What stops it Tools
Extract The report is the attack surface. Text planted in a page, a ticket, or a log can tell the agent to skip a check or weaken a rule. The extractor produces structure and nothing else. It runs with no tool but the schema validator, hidden text and markup are stripped before it reads, and the hostile-input rules of section 10 apply to everything it sees. A model call with a JSON schema for its output; the pinned ATT&CK extract for ids; an embedding or tag match against the catalog for duplicates.
Ground An agent invents a field name that reads well, or picks a source that is configured but not arriving, and the rule converts cleanly and matches nothing. The dictionary comes from a scheduled job the agent does not run, exported daily from the SIEM's own schema. The agent's query credential is read-only and scoped to that export and the baseline count. Splunk tstats and fieldsummary over the data models, Microsoft Sentinel's table schema and getschema, Elasticsearch's field capabilities, OCSF for the neutral names; the framework's inventory.csv for the sources and their silence alert.
Author A rule that matches the blueprint's ephemeral clues, the file name or the hash, passes every syntax check and is worthless a week later. The robustness draft scores the rule from its observables, the way the library's sources page describes. A draft at level 1 fails the build unless the record says why that is enough. pySigma and its validators, sigma-cli check, the backends in the kit's target registry; the SIEM's job statistics for bytes scanned; the kit's lint, convert, and golden diff.
Prove The agent writes the fixtures it needs, or picks the replay window that returns the count it wants. The sandbox and its collector run under a credential the agent does not have and sign what they emit. The replay window is fixed by the job. The fixtures are read from the signed archive, never from the pull request. Atomic Red Team through Invoke-Atomic, Stratus Red Team for cloud accounts, and Caldera for chains. Throwaway hosts from Terraform or Ludus, with Sysmon and nxlog or Winlogbeat. OTRF's Security-Datasets when there is no lab. The SIEM's own search over 30 days, or a lakehouse query over OCSF, and Chainsaw or Hayabusa for EVTX offline.
Tune The cheapest way to zero false alarms is to silence the rule: widen a filter, drop a selection, or exclude a directory. The lock, the additive-only diff, and the re-run of recall. The exclusion policy is a lint rule the build runs. The kit's declared variables and allowlist entries; a lint that diffs the parsed rule between base and head; the same replay as Prove.
Ship A reviewer who trusts the bundle approves in a minute, and the bundle is the agent's own account of itself. The bundle links to artifacts that other jobs produced, under hashes the reviewer can check, and the metrics below count how many approvals still look right after 90 days. That number says whether review is still review. GitHub Actions job summaries and branch protection, the kit's promotion check, and the rule APIs of Sentinel, Splunk ACS, and Elastic. A scheduled alert-rate query that switches alerting off. The case tool's dispositions, such as Sentinel's incident classification or a Splunk ES notable status, as the webhook that drafts a tuning pull request.

18.3The routes back

Route When it is taken
Prove to Author A variant in the plan that the rule misses sends the blueprint and the missed event back to Author. The next draft is written against the event the rule did not match.
Prove to Tune Matches over the threshold in the 30-day replay go to Tune with the groups already made, and the tuned rule goes through Prove again before it ships.
Live to Tune An analyst's false-positive disposition, an alert rate over the hourly limit, or a daily field check that finds a renamed column each opens a tuning pull request. The detection stays live until it merges, unless the rate already switched alerting off.

18.4What people keep

  • Approve every pull request. Branch protection enforces it, so it never depends on habit.
  • Approve the deploy run and the pull request that switches alerting on.
  • Decide every retirement, as in section 6.
  • Set the thresholds the checks use: the share of the fleet a source must report from, the scan cap, the exclusion policy, and the hourly alert limit. Write each one down with the reason.
  • Read the 90-day survival number every month, and sample the approvals when review time per pull request falls under a few minutes.

18.5What the kit has and what it lacks

Lint on the record, the fixtures, and the stage moves. Conversion with an approved copy per target, and replay on five engines. Declared variables, and the promotion check that reads a detection's live date from git. The core lock and the exclusion policy, in tools/corelock.py. The daily field check in tools/fields.py fails when a live rule's log source sends nothing or a field the rule reads is empty in every event. A person opens the tuning pull request from the failed run. The daily running check in tools/running.py fails when a live rule's saved search in Splunk differs from what the deploy job packaged, is disabled, or alerts at the wrong stage. AGENTS.md as the policy an agent runner reads, the hostile-input rules, and the four research lines the record must carry.

The extractor, and the daily export of the whole field dictionary for Ground to draft from. The scan estimate, and the robustness draft as a build check. The sandbox runner and its signing collector, and the 30-day replay job. The alert-rate check that switches alerting off, and the disposition webhook. Each is one job in the pipeline, and the rollout below adds them in the order that pays first.

18.6The pieces, specified

Each check above is a job. These are the six pieces the loop rests on, specified closely enough to build. The kit carries two of them, the core lock and the exclusion policy; the other four are the team's to build.

Piece What it is
The field dictionary One table a scheduled job writes daily: source, table, field, type, the share of events in the last 24 hours in which the field was non-empty, and the share of hosts that reported. Ground reads this and nothing else for field names. A field under the team's non-empty share is treated as absent.
The core lock Every selection in the detection block whose name does not start with filter, plus the condition with its 'and not filter' clauses removed. The two are canonicalized, with keys and value lists sorted and values stripped, and hashed with SHA-256. Tuning must leave the hash unchanged. The kit's tools/corelock.py computes it and fails a pull request that changes it on a live rule.
The exclusion policy A filter is one map of at least three fields that must all be true, such as the parent process, the signer or original file name, and the flag or path. No value may replace a whole directory with a wildcard. Removing a filter widens the rule and goes back through validate. Each filter cites a ticket and an expiry in the record's allowlist variable.
The signed fixture The collector in the sandbox writes each run's events to an archive with a manifest listing every file's SHA-256, and signs the manifest with a key the agent never sees. Prove reads fixtures from that archive only. The build verifies the signature before it counts a match as recall.
The pull request bundle The rule, the approved copy for every target, the signed fixture archive's hash, and the recall result per variant. With them, the replay window and its counts grouped by parent and signer, the core hash, the scan estimate, and every query the agent ran with its time range and row count. Branch protection requires the jobs that produced each item.
The alert-rate check A scheduled query every few minutes counts each live rule's alerts in the last hour against the hourly limit on its record. Over the limit, the job switches the rule's alerting off, records the hour's count, and opens a tuning pull request with the matches grouped. Alerting comes back on only through the promotion pull request, as for a new rule.

18.7What to measure

Target How to measure it
First-time pass rate per check The monthly share of runs that pass each check on the first try. A check that always passes checks nothing, and one that never passes is set wrong.
Trips around the loop per rule Median count of returns to Author and to Tune before a rule merges. A rising number points at the extractor or the dictionary before it points at the model.
Hours from intake to review-ready The intake_to_review_ready_hours metric of section 10, which the loop should move from days to hours. Report the median; the best run flatters.
Acceptance and survival agent_draft_acceptance_rate and agent_detection_survival_90d from section 10. Survival is the number that tells you the checks work; acceptance alone tells you reviewers are tired.
Review minutes per pull request Measured from the pull request's timeline. When it falls below a few minutes, sample a tenth of the approvals and re-review them.
Alerting switched off by the rate check Count per month, with the rule and the cause. Each one is a tuning pull request the loop should have opened earlier.
Renames the field check caught Count per month. Each one is a detection that would have gone silent.

18.8Rolling it out

Phase Add Done when
Days 1 to 30 The repository, lint, conversion with approved copies, the daily dictionary export, and the scan cap. No model yet. No rule reaches a pull request with a field the dictionary lacks, and no query over the scan cap merges.
Days 31 to 60 The extractor, then Prove: the sandbox or the recordings, the signing collector, and the 30-day replay. Every pull request carries signed fixtures, a recall result, and a replay count.
Days 61 to 90 Tune with the core lock and the exclusion policy, the promotion check, the alert-rate check, and the disposition webhook. A false-positive disposition becomes a tuning pull request without a person starting it, and the core hash of every live rule matches its approved copy.

18.9One pass, by hand

No agent has run the loop. To see whether its checks work on real events, I walked the six stages by hand. The tools were the kit's, and the events were the 107 recorded attacks the library already runs its rules against. The procedure was LSASS memory dumped through comsvcs.dll, T1003.001, the example the reference architecture behind this section was built around. The table shows what each stage did and what came back.

Stage What was done What came back
Extract Read the procedure. rundll32.exe loads comsvcs.dll and calls its MiniDump export against the process id of lsass.exe. The library name and the export are durable; the attacker can also name the export by its ordinal, 24. The dump file's path and name are the attacker's choice, so they were left out. One behavior and two candidate observables: the command line of the rundll32 process, and the call trace of its access to lsass.exe. Both map to T1003.001, which the offline ATT&CK copy confirms is a real id.
Ground Counted, in the recordings, the two event types the candidates read, and how many of each had its field filled. Sysmon event 1, process creation: 1,515 events in 95 of the 107 recordings, with CommandLine filled in every one. Sysmon event 10, process access: 249,519 events in 103 recordings, with CallTrace filled in every one. Both fields are safe to build on.
Author Wrote the rule as Sigma: process creation where CommandLine contains comsvcs and one of MiniDump, #24, #+24, or '24 followed by a space. Computed its core hash and its robustness draft, and converted it to SQL with the kit's SQLite backend. Core hash b251dfc760654ed9. The same rule with its keys and its list of values in another order gives the same hash. The robustness draft is level 2, text from the attacker's own tool, because the attacker types the command line. SigmaHQ's rule for the same procedure, a49fa4d5, reads the call trace of the process access, and it drafts at level 2 as well.
Prove Ran both rules over all 776,654 events in the corpus, grouped by recording. The recording of this procedure is SDWIN-201018195009, Lsass Memory Dump via Comsvcs.dll, 184 events. The candidate fired on 1 event, in that recording, and on nothing else in the corpus. SigmaHQ's rule fired on 2 events, in the same recording, and on nothing else. The match is the fixture: rundll32.exe started by powershell.exe as WORKSTATION5\wardog, with the MiniDump command line quoted below. Recall against the one recording of this procedure is 1 of 1.
Tune Looked for noise to exclude. There was nothing to exclude. The corpus has no benign recording, so zero false positives here says nothing about a real fleet. The exclusion policy was tested another way. The kit self-test adds a one-field filter, a filter that wildcards a directory, and a change to a live rule's selections, and the core lock refuses each.
Ship Nothing. No SIEM received the rule, no pull request was opened, and no seven days ran. Most of the bundle the pull request would carry exists: the rule, the SQL conversion, the one fixture, the replay counts per recording, the core hash, and the queries below with their row counts. The signature on the fixture does not, because no collector outside the author's reach produced it.

The one event the candidate matched, quoted the way a record quotes log content, as data in a fenced block:

"C:\Windows\System32\rundll32.exe" C:\windows\System32\comsvcs.dll MiniDump 756 C:\Users\wardog\AppData\Local\Temp\lsass-comsvcs.dmp full

The queries the pass ran, listed the way the pull request template asks for them:

Query Over Rows
Sysmon event 1 with CommandLine filled, by recording all 107 recordings 1,515 in 95 recordings
Sysmon event 10 with CallTrace filled, by recording all 107 recordings 249,519 in 103 recordings
The candidate rule, converted to SQL all 776,654 events 1, in SDWIN-201018195009
SigmaHQ a49fa4d5, converted to SQL all 776,654 events 2, in SDWIN-201018195009
Any command line containing comsvcs all 776,654 events 2, both in SDWIN-201018195009

The other four T1003.001 recordings, Mimikatz twice, Task Manager, and direct syscalls, each set off 8 to 10 library rules, and none of those rules was a comsvcs rule. One rule covers one procedure, and a technique's recall is the sum over its procedures. Extract should emit one candidate per procedure a report describes, and a coverage count by technique overstates what one rule covers.

The pass shows that ground, author, prove, and the core lock work on real events with the kit's tools, and that the replay has a corpus to run on today. It does not show that a model can do extract or author unassisted, or how the exclusion policy fares against a tuning agent. It says nothing about the rule's alert rate on a real fleet. The outside checks page lists the pass and these gaps.

The loop multiplies what section 10 already says about the review bottleneck. Cap open agent pull requests at what reviewers clear in a week, and let the scored backlog decide what gets drafted.

A 30-day replay over every candidate is the loop's largest bill. Run it over a sample of hosts first, and over the whole fleet only for a rule that passed.

One pass has been measured, by hand and against recordings, in 18.9. The targets above are what a team should expect to see, and the outside checks page lists them as untested.