DETECT hunt Build detections

IdeaProductionShipMeasure

Ship a detection, step by step

A checklist for the engineer doing the work, stage by stage: what to do, the command to run, and how you know the stage is done.

ChecklistFramework v0.136.0. The purposes, owners, definitions of done, and common failures come from framework.json.

Remember

  • Work from the top of the scored backlog, in the order the lead confirmed at the weekly intake meeting.
  • Run lint, the conversion check, and the SQLite replay locally before you open or update a pull request.
  • Never edit a query in the SIEM's own interface. Every change reaches production through a pull request.
On this page
  1. 1

    Intake

    Owner: Detection lead

    Write down a request for a new detection with enough context that it can be scored.

    Do

    • Open an intake ticket with the fields from the intake template.
    • Score the six inputs with the wording in Appendix B: 1 to 5 for five of them and 0 to 5 for telemetry readiness. Write one evidence line for each.
    • Run the scorer, or paste the numbers into the backlog scorer to see where the candidate lands.

    Run

    python tools/score.py detections/*/score.yml

    Example

    DET-0001 scored 4, 4, 4, 4, 2, 2 on the balanced weights, for a score of 2.40.

    Done when

    6 scores, each with an evidence line, and 1 computed score.

    Watch for

    Requests arrive as nothing more than an alert name, with no note about the threat behind it, so you build them in the order they showed up.

    If it can't be met

    A candidate whose scores can't be evidenced stays at intake, and the missing evidence becomes a question for the next intake meeting. A candidate with telemetry readiness 0 is blocked, and its log source request goes to the platform owner.

  2. 2

    Hypothesis

    Owner: Detection engineer

    Write one sentence saying what the detection will catch, which log source it needs, and what it will miss.

    Do

    • Write one sentence: when an attacker does X, log source Y records Z. It opens the Strategy Abstract section of ads.md.
    • Name the log source the detection will read. It goes in the first sentence of Strategy Abstract with the hypothesis.
    • Name one thing the detection misses, in the Blind Spots and Assumptions section.
    • Write the Goal and Categorization sections, so ads.md carries all four hypothesis sections.
    • Set the stage in score.yml to hypothesis, then run lint. It checks for a Hypothesis: line and a Log source: line in Strategy Abstract, and for something written under Blind Spots and Assumptions.

    Run

    python tools/lint.py DET-NNNN

    Example

    When an attacker requests a service ticket to crack offline, the domain controller's Security log records event 4769 with encryption type 0x17.

    Done when

    1 hypothesis sentence, 1 named log source, 1 named blind spot.

    Watch for

    The hypothesis describes one tool's fingerprint. When that tool changes, the detection goes quiet.

    If it can't be met

    If no log source you collect records the behavior, the candidate goes back to intake with telemetry readiness 0 and a log source request. If the only hypothesis anyone can write names one tool's fingerprint, rescore its robustness at intake before going on.

  3. 3

    Research

    Owner: Detection engineer

    Produce the events the rule will be written against, and confirm they exist in your logs. Then count how often they appear during normal business.

    Do

    • Run the behavior in a lab. An Atomic Red Team test covers many endpoint techniques; cloud and identity techniques need a planned emulation.
    • Save the events as fixtures: one the detection must match and one it must ignore. Give each a fixture_source naming the run tag or query it came from, and name the same tag or query in the Technical Context.
    • Read every field name you'll use from a real event.
    • Write the Technical Context section of ads.md from what the run showed: why the behavior looks like this and what normal traffic in it looks like. Add the four lines lint reads: Plan approved by, Run tag, Baseline (the query, its time range, and its count), and Teardown verified.
    • Count how often the behavior happens normally over 7 days of production data. That count sets the alert volume threshold on the record: 1.5 x 7 x the daily count, never under 3.
    • Tear the lab down and confirm nothing is left behind.
    • Set the stage in score.yml to research, then run lint.

    Run

    python tools/lint.py DET-NNNN

    Done when

    1 emulation run, fixtures saved, every field checked against a real event, 7 days of baseline counts, lab torn down.

    Watch for

    Field names come from documentation or memory, so the detection is almost right and never fires. The other common failure is skipping the 7-day count, which spends the first week in production on noise the count would have shown.

    If it can't be met

    If the events never reach the SIEM, or a field the rule needs arrives unparsed, the candidate goes back to intake with its telemetry readiness lowered and a log source request. If the 7-day count shows the behavior is routine here, rewrite the hypothesis before building anything.

  4. 4

    Build

    Owner: Detection engineer

    Write the detection in Sigma and the tests that prove it.

    Do

    • Write rule.yml in Sigma.
    • Complete ads.md: all 10 ADS headings plus an empty Retirement record, because lint fails on a missing heading. Hypothesis and research wrote five sections. Add False Positives, Validation, Priority, Response, and Additional Resources, and fill every section before review.
    • Add the fixtures to tests/.
    • Write the converted queries with python tools/convert.py --update-golden DET-NNNN. The conversion check compares against them from now on, and the reviewer approves them at validate.
    • Set the stage in score.yml to build and the status in rule.yml to experimental, then run lint and the conversion check.

    Run

    python tools/convert.py --update-golden DET-NNNN
    python tools/lint.py DET-NNNN && python tools/convert.py --check

    Done when

    Lint passes, 1 conversion per target, 1 event it must match, 1 event it must ignore.

    Watch for

    You write the detection straight in the SIEM's own query language. Then it runs on one product only, you can't compare its changes against an approved version, and it can't move when the SIEM changes.

    If it can't be met

    If a target can't express the rule, mark that target convert-only in targets.yml and say so in the record. If Sigma can't express it at all, as with a correlation across events, write it as a query file beside the same record and fixtures, and review that file directly.

  5. 5

    Validate

    Owner: Reviewer, a second engineer

    Prove the detection fires on what it should catch, stays silent on what it should ignore, and still matches the query a reviewer approved.

    Do

    • Set the stage in score.yml to validate and the status in rule.yml to test. Lint now wants every section but the retirement record written, and numbered steps under Response.
    • If the rule changed since build, run python tools/convert.py --update-golden DET-NNNN again, so the pull request carries the queries the reviewer approves.
    • Replay the fixtures on every replay engine in targets.yml.
    • Open a pull request with the review checklist.
    • Ask a second engineer to review the record first, then the query, and approve.

    Run

    python tools/replay.py --fixtures --target sqlite
    python tools/replay.py --fixtures --all-replay-targets

    Done when

    Fixtures pass on every replay engine, every converted query approved (matching its approved copy on a re-validation), 1 approval by someone other than the author (one-person team: the author after 24 hours).

    Watch for

    Whenever the conversion output changes, someone overwrites the approved copy to match it. After that the comparison can't fail, so it catches no changes.

    If it can't be met

    If a fixture fails or the reviewer asks for changes, the pull request goes back to build and stays open. If the reviewer rejects the hypothesis itself, the detection goes back to hypothesis.

  6. 6

    Deploy

    Owner: Detection lead

    Put the detection into production with alerting switched off, leave it running for 7 days, and switch alerting on only if the alert count stayed at or under its threshold.

    Do

    • Merge the pull request that sets the stage to deploy. The lead approves the deploy run, which packages the detection with alerting off. The kit's vetting and install steps are placeholders, so fill them in for your Splunk stack before the first deploy.
    • Leave it running for 7 days and watch the alert count against the threshold on the record. If the count goes over, it stays at deploy with alerting off; fix it in a pull request that goes back through validate, and the 7 days start again from that merge.
    • After 7 days at or under the threshold, open a pull request that moves the stage to tune. Its promotion check counts the 7 days again, and merging it switches alerting on.

    Run

    python tools/history.py DET-NNNN
    python tools/replay.py --production

    Done when

    7 days live with alerting off, alert volume at or under its threshold, then alerting on.

    Watch for

    Alerting goes on before the 7-day check, and the SOC pays for the noise in triage time.

    If it can't be met

    If the first 7 days run over the threshold, alerting stays off and the detection stays at deploy. The fix is a pull request that goes back through validate, and the 7 days start again from its merge. If they return nothing though research counted the behavior, check the fields and the index against a real event.

  7. 7

    Tune

    Owner: Detection engineer with the SOC or triage lead

    Keep the share of true alerts above the agreed floor as the environment changes.

    Do

    • Check precision every 30 days against the detection's floor.
    • Tune in a pull request that adds the exclusion as a filter_ selection in rule.yml, which the core lock allows, and names it. Record the value in variables.yml in the same pull request, because no tool substitutes it yet.
    • Send every tuning change back through validate. A change to the query is counted over the last 7 days before it merges.
    • When an incident or an exercise shows the detection missed something, add the missed events as a positive fixture and fix the rule in the same pull request. The fixture's source is the incident or exercise id, which the Validation section names.

    Run

    python tools/lint.py DET-NNNN && python tools/convert.py --check
    python tools/replay.py --fixtures --target sqlite

    Done when

    Precision checked every 30 days, running query matches the approved copy at each check.

    Watch for

    Someone tunes the detection by adding exclusions in the SIEM's own interface, so the rule in the repo no longer matches the rule running in production.

    If it can't be met

    If precision stays under the floor for two 30-day periods in a row, propose retirement. If an incident or an exercise shows the detection missed what it claims to catch, fix this detection. The missed events become a positive fixture, and the change goes back through validate and its 7-day count.

  8. 8

    Retire

    Owner: Detection lead

    Remove detections whose alerts cost more analyst time than they are worth.

    Do

    • Retire when a retirement trigger fires, and record the numbers behind it.
    • Set the stage in score.yml to retire and the status in rule.yml to deprecated, and keep the record. The lead switches alerting off.
    • Write the Retirement record section, and add retired.date to score.yml. If the technique still matters, open its replacement at intake and name it in retired.replacement_id.

    Done when

    Status deprecated, alerting off, record kept, retirement note written, replacement linked if one exists.

    Watch for

    Nothing is ever retired, so analysts keep triaging alerts from detections that no one owns or understands anymore.

    If it can't be met

    A retirement is not undone. If the technique still matters, the replacement is a new detection at intake with a link to the retirement note.

Copy it into a ticket

The same steps as a Markdown checklist, ready for a ticket or a pull request description.

# DET-NNNN: <name>

## 1. Intake
- [ ] Open an intake ticket with the fields from the intake template.
- [ ] Score the six inputs with the wording in Appendix B: 1 to 5 for five of them and 0 to 5 for telemetry readiness. Write one evidence line for each.
- [ ] Run the scorer, or paste the numbers into the backlog scorer to see where the candidate lands.
- Done when: 6 scores, each with an evidence line, and 1 computed score
- If it can't be met: A candidate whose scores can't be evidenced stays at intake, and the missing evidence becomes a question for the next intake meeting. A candidate with telemetry readiness 0 is blocked, and its log source request goes to the platform owner.

## 2. Hypothesis
- [ ] Write one sentence: when an attacker does X, log source Y records Z. It opens the Strategy Abstract section of ads.md.
- [ ] Name the log source the detection will read. It goes in the first sentence of Strategy Abstract with the hypothesis.
- [ ] Name one thing the detection misses, in the Blind Spots and Assumptions section.
- [ ] Write the Goal and Categorization sections, so ads.md carries all four hypothesis sections.
- [ ] Set the stage in score.yml to hypothesis, then run lint. It checks for a Hypothesis: line and a Log source: line in Strategy Abstract, and for something written under Blind Spots and Assumptions.
- Done when: 1 hypothesis sentence, 1 named log source, 1 named blind spot
- If it can't be met: If no log source you collect records the behavior, the candidate goes back to intake with telemetry readiness 0 and a log source request. If the only hypothesis anyone can write names one tool's fingerprint, rescore its robustness at intake before going on.

## 3. Research
- [ ] Run the behavior in a lab. An Atomic Red Team test covers many endpoint techniques; cloud and identity techniques need a planned emulation.
- [ ] Save the events as fixtures: one the detection must match and one it must ignore. Give each a fixture_source naming the run tag or query it came from, and name the same tag or query in the Technical Context.
- [ ] Read every field name you'll use from a real event.
- [ ] Write the Technical Context section of ads.md from what the run showed: why the behavior looks like this and what normal traffic in it looks like. Add the four lines lint reads: Plan approved by, Run tag, Baseline (the query, its time range, and its count), and Teardown verified.
- [ ] Count how often the behavior happens normally over 7 days of production data. That count sets the alert volume threshold on the record: 1.5 x 7 x the daily count, never under 3.
- [ ] Tear the lab down and confirm nothing is left behind.
- [ ] Set the stage in score.yml to research, then run lint.
- Done when: 1 emulation run, fixtures saved, every field checked against a real event, 7 days of baseline counts, lab torn down
- If it can't be met: If the events never reach the SIEM, or a field the rule needs arrives unparsed, the candidate goes back to intake with its telemetry readiness lowered and a log source request. If the 7-day count shows the behavior is routine here, rewrite the hypothesis before building anything.

## 4. Build
- [ ] Write rule.yml in Sigma.
- [ ] Complete ads.md: all 10 ADS headings plus an empty Retirement record, because lint fails on a missing heading. Hypothesis and research wrote five sections. Add False Positives, Validation, Priority, Response, and Additional Resources, and fill every section before review.
- [ ] Add the fixtures to tests/.
- [ ] Write the converted queries with python tools/convert.py --update-golden DET-NNNN. The conversion check compares against them from now on, and the reviewer approves them at validate.
- [ ] Set the stage in score.yml to build and the status in rule.yml to experimental, then run lint and the conversion check.
- Done when: lint passes, 1 conversion per target, 1 event it must match, 1 event it must ignore
- If it can't be met: If a target can't express the rule, mark that target convert-only in targets.yml and say so in the record. If Sigma can't express it at all, as with a correlation across events, write it as a query file beside the same record and fixtures, and review that file directly.

## 5. Validate
- [ ] Set the stage in score.yml to validate and the status in rule.yml to test. Lint now wants every section but the retirement record written, and numbered steps under Response.
- [ ] If the rule changed since build, run python tools/convert.py --update-golden DET-NNNN again, so the pull request carries the queries the reviewer approves.
- [ ] Replay the fixtures on every replay engine in targets.yml.
- [ ] Open a pull request with the review checklist.
- [ ] Ask a second engineer to review the record first, then the query, and approve.
- Done when: fixtures pass on every replay engine, every converted query approved (matching its approved copy on a re-validation), 1 approval by someone other than the author (one-person team: the author after 24 hours)
- If it can't be met: If a fixture fails or the reviewer asks for changes, the pull request goes back to build and stays open. If the reviewer rejects the hypothesis itself, the detection goes back to hypothesis.

## 6. Deploy
- [ ] Merge the pull request that sets the stage to deploy. The lead approves the deploy run, which packages the detection with alerting off. The kit's vetting and install steps are placeholders, so fill them in for your Splunk stack before the first deploy.
- [ ] Leave it running for 7 days and watch the alert count against the threshold on the record. If the count goes over, it stays at deploy with alerting off; fix it in a pull request that goes back through validate, and the 7 days start again from that merge.
- [ ] After 7 days at or under the threshold, open a pull request that moves the stage to tune. Its promotion check counts the 7 days again, and merging it switches alerting on.
- Done when: 7 days live with alerting off, alert volume at or under its threshold, then alerting on
- If it can't be met: If the first 7 days run over the threshold, alerting stays off and the detection stays at deploy. The fix is a pull request that goes back through validate, and the 7 days start again from its merge. If they return nothing though research counted the behavior, check the fields and the index against a real event.

## 7. Tune
- [ ] Check precision every 30 days against the detection's floor.
- [ ] Tune in a pull request that adds the exclusion as a filter_ selection in rule.yml, which the core lock allows, and names it. Record the value in variables.yml in the same pull request, because no tool substitutes it yet.
- [ ] Send every tuning change back through validate. A change to the query is counted over the last 7 days before it merges.
- [ ] When an incident or an exercise shows the detection missed something, add the missed events as a positive fixture and fix the rule in the same pull request. The fixture's source is the incident or exercise id, which the Validation section names.
- Done when: precision checked every 30 days, running query matches the approved copy at each check
- If it can't be met: If precision stays under the floor for two 30-day periods in a row, propose retirement. If an incident or an exercise shows the detection missed what it claims to catch, fix this detection. The missed events become a positive fixture, and the change goes back through validate and its 7-day count.

## 8. Retire
- [ ] Retire when a retirement trigger fires, and record the numbers behind it.
- [ ] Set the stage in score.yml to retire and the status in rule.yml to deprecated, and keep the record. The lead switches alerting off.
- [ ] Write the Retirement record section, and add retired.date to score.yml. If the technique still matters, open its replacement at intake and name it in retired.replacement_id.
- Done when: status deprecated, alerting off, record kept, retirement note written, replacement linked if one exists
- If it can't be met: A retirement is not undone. If the technique still matters, the replacement is a new detection at intake with a link to the retirement note.