Section 17
The reference pipeline
| Stage | What the definition of done counts | The job that fails when it is skipped |
|---|---|---|
| Intake | 6 scores, each with an evidence line, and 1 computed score | score |
| Hypothesis | 1 hypothesis sentence, 1 named log source, 1 named blind spot | lint |
| Research | 1 emulation run, fixtures saved, every field checked against a real event, 7 days of baseline counts, lab torn down | lint |
| Build | lint passes, 1 conversion per target, 1 event it must match, 1 event it must ignore | lint convert |
| Validate | fixtures pass on every replay engine, every converted query approved (matching its approved copy on a re-validation), 1 approval by someone other than the author (one-person team: the author after 24 hours) | lint convert replay (one leg per engine) golden-diff review |
| Deploy | 7 days live with alerting off, alert volume at or under its threshold, then alerting on | deploy promote-check field-check (scheduled) running-check (scheduled) |
| Tune | precision checked every 30 days, running query matches the approved copy at each check | lint (core lock, on pull requests) field-check (scheduled) running-check (scheduled) precision-check (scheduled, planned) |
| Retire | status deprecated, alerting off, record kept, retirement note written, replacement linked if one exists | deploy (the next package leaves it out) |
Remember
- Copy the repository layout and replace the example detection with your own.
- The scorer writes every score, and CI fails any score that disagrees with the formula.
- Convert once, commit the approved copies, then review every difference after that.
- Lint also fails when AGENTS.md no longer matches framework.json.
- The Splunk replay leg needs only its container's password and HEC token. The promotion check needs a read-only Splunk account in the siem-read environment, and the two daily checks need the same account in siem-read-daily. The deploy job needs the deploy credential in the production environment.
If you're using the repository as a template, replace the example and point the secrets at a Splunk Cloud stack. Each detection is a directory. It carries rule.yml (Sigma), ads.md (the record), score.yml (the inputs, an evidence line for each, and a CI-written score), the fixtures, and the approved copies of the converted queries.
| Stage | Sigma status | CI jobs | Artifact |
|---|---|---|---|
| 1. Intake | no rule file yet | score | score.yml with stage set to intake |
| 2. Hypothesis | no rule file yet | lint | The Goal, Categorization, Strategy Abstract, and Blind Spots and Assumptions sections of ads.md |
| 3. Research | no rule file yet | lint | The Technical Context section of ads.md; the emulation plan and run report; tests/events.jsonl |
| 4. Build | experimental | lint, convert | rule.yml, tests/events.jsonl, tests/expected.yml |
| 5. Validate | test | lint, convert, replay (one leg per engine), golden-diff, review | Passing CI run; approved pull request |
| 6. Deploy | stable | deploy, promote-check, field-check (scheduled), running-check (scheduled) | The pull request that moves the detection from deploy to tune, carrying the promotion check's count for its first 7 days live |
| 7. Tune | stable | lint (core lock, on pull requests), field-check (scheduled), running-check (scheduled), precision-check (scheduled, planned) | A tuning pull request that cites the alert ids that prompted it |
| 8. Retire | deprecated | deploy (the next package leaves it out) | A retirement note in ads.md: reason, date, and the id of the replacement detection |
The definitions of done call these tools in the repository:
| Tool | Jobs it backs | What it does |
|---|---|---|
| lint | lint | Checks the layout and parses the detection. It confirms the id and tag agree, the Sigma status matches the stage, the ADS sections are present, and every named fixture exists. It also validates the score record, rejects any ATT&CK id outside the pinned map, and fails when AGENTS.md no longer matches framework.json. |
| convert | convert, golden-diff | Turns each Sigma rule into all 13 targets in targets.yml, and fails when any output differs from its approved copy. |
| score | score | Computes each record's score and sorts the backlog. |
| metrics | score | Adds to the score job's metrics the numbers only the file layout, incidents.csv, and notifications.csv can tell. |
| package | deploy | Builds a Splunk app from the approved queries. A detection at deploy runs with alerting off, one at tune runs with it on, and a retired one is left out, which takes it out of production. |
| replay | replay, promote-check | Loads the fixtures into a real instance of each replay engine (SQLite, Splunk, Kusto, Elasticsearch, ES|QL, OpenSearch). On the pull request that moves a detection to tune, it counts the first 7 days live against the threshold. |
| history | promote-check | Reads from git when each detection changed stage and when it went live, so no record carries a date by hand. |
| fields | field-check | Once a day, counts the events from each live rule's log source and how many carry each field the rule reads. It fails a source that sent nothing and a field that was empty in every event. |
| running | running-check | Once a day, compares each live rule's saved search in Splunk with what the package writes from main. It fails a changed query, schedule, or alert condition, a disabled search, alerting at the wrong stage, and a search in the app with no live detection behind it. |
| next | none; run by hand | Runs the checks a detection's current stage needs, in one command, and prints what is still missing, the count the stage must reach, and the step that moves the detection on. |
| intake | none; run by hand | Prints the weekly intake meeting's agenda from the records and git: new candidates, the ones blocked on telemetry, the queue in scored order, the work in flight, and what has not moved in a month. |
The precision-check job is planned and not yet written. The other seven targets (EQL, Loki, LogScale, NetWitness, Panther, Microsoft Defender XDR advanced hunting, Google SecOps) have no public test instance. Their conversions stay drafts until someone runs them on a real one. Separate upkeep tools reshape fixtures for each engine, refresh the pinned ATT&CK map, and regenerate the agent instructions.
I cap each query backend below its next major version in requirements.txt. A major release can change the query a backend emits, and the golden check catches that. When pysigma-backend-sqlite 2.0.0 renamed the table placeholder, it failed the check on DET-0001. To move a cap, you open a pull request that updates the approved copies in the same change.
These have run on the example: lint, convert with the golden check on all 13 targets, score, metrics, packaging with one detection at deploy and one at tune, and the promotion check against a stubbed search. The two daily checks ran against a real Splunk in a container. The field check ran on the fixtures with fields renamed, blanked, and removed, and the running check on the packaged app after edits made through the console's own API. Every replay adapter has run end to end against a real engine in a container: SQLite, Splunk, the Kusto emulator, Elasticsearch, and OpenSearch each returned the two fixtures DET-0001 must match and none of the three it must ignore. The site's own build also copies the kit and breaks it on purpose, one case per check: a rule only some targets can convert, a regular expression, a silent re-approval, each stage move the lifecycle refuses, and each count the promotion check fails on. The build stops if any tool misses its break. These haven't run anywhere yet and stay placeholders until they do: the container services inside the GitHub Actions workflow, and the AppInspect and ACS steps.