DETECT hunt Reference

Checks against outside sources

The site makes claims about ATT&CK and D3FEND, about 4,211 Windows Sigma rules in its library, and about the queries its kit converts. This page says which of those claims something other than the author has checked, with the numbers, and which remain opinion.

EvidenceFramework v0.136.0. The four generated checks rerun with the data they test: the ATT&CK check on every ATT&CK release, the D3FEND check on every D3FEND release, the draft check on every build, and the recordings on the weekly library job.

On this page

The ATT&CK extract, against MITRE's own library

The coverage map and the library read ATT&CK 20.1 through a parser this site wrote. On 2026-11-06, tools/attack_crosscheck.py read the same STIX bundle with mitreattack-python 5.4.1, the library MITRE maintains, and compared every count the site shows. They agree on all of them.

ComparedItems
techniques706
platforms706
analytics1,708
components706
used by706
revoked149
deprecated12
platform counts11

'Used by' counts direct users; MITRE's library would add 280 group-technique pairs credited through campaigns attributed to the group.

The D3FEND links, against the source's own API

The coverage map reads D3FEND 1.6.0 from the release's bulk mappings file, which lists each ATT&CK technique's links on its own. D3FEND's website counts a parent technique's links with its sub-techniques' and a sub-technique's with its parent's, and the map does the same fold. On 2026-10-20, tools/d3fend_extract.py --check asked D3FEND's API about 5 techniques and compared the count of defensive techniques, and of those that detect. They agree on every one.

TechniqueThe map (detect)D3FEND's API (detect)
T1003 OS Credential Dumping48 (19)48 (19)
T1003.001 LSASS Memory12 (3)12 (3)
T1558.003 Kerberoasting21 (10)21 (10)
T1059.001 PowerShell15 (4)15 (4)
T1021.001 Remote Desktop Protocol11 (9)11 (9)

The robustness draft, against hand scores from MITRE's CTID

The library's robustness draft reads a rule and assigns Summiting the Pyramid level 1, 2, or 3. MITRE's Center for Threat-Informed Defense scored 92 public analytics by hand and published the table; 72 of them are SigmaHQ rules the draft can read. On 2026-11-03 the draft matched the hand score on 55 of the 72, stayed at or under it on 66, and was within one level on 66. The draft is meant as a floor a person raises, so the second number is the one that matters, and the 6 rules it overshoots are listed in the check's output. The rules and the table are kept in the repository under library/ctid/, so the check reruns on every build.

CTID's levelThe draft's levelRules
1130
124
132
213
2217
314
338
434

The library's Windows rules, against recorded attacks

OTRF's Security-Datasets are Windows event logs captured while one technique ran in a lab, each tagged with the techniques it exercised. On 2026-11-09 the library loaded 107 recordings, 776,654 events covering 56 techniques, and ran every Windows Sigma rule in the catalog against them. Each rule was converted with pySigma's Sysmon and Windows pipelines, so it reads only its own channel and event id. 1,695 of the 4,211 rules have a recording of a technique they name.

Of those 1,695 rulesRules
fired on a recording of its technique139
stayed silent, though the recording carries events of the type it reads1,307
could not fire: the recording lacks a field the rule reads45
could not fire: the recording has no event of the type it reads119
could not run: a keyword rule, or one the SQLite backend cannot convert85

A rule that fired is confirmed on real events of its technique; each one says so on its library page, and the library filter Fired on a recorded attack lists them. A rule that stayed silent is not shown wrong: a recording exercises one procedure, and a rule for the same technique may watch another. The two "could not fire" rows are facts about the recordings as much as the rules. No benign recording exists in the corpus, so nothing here measures false positives. The weekly library job reruns this with library/recordings.py and commits the result.

Other runs and reviews

The kit's replay, in real engines

2026-10-15

DET-0001's converted queries ran in containers of Splunk (splunk/splunk:latest), the Kusto emulator (kustainer), Elasticsearch 8.15.0, and OpenSearch 2.16.0, and in SQLite, each loaded with the five fixture events.

Every engine returned the two events the rule must match and none of the three it must ignore. The dry runs had hidden two faults. Elasticsearch rejected every document, because the index mapping tried to map nested objects as keywords. The Splunk image now refuses to start without the terms flag.

The converted queries mean the same thing in five engines, on these five events.

The production count, against a real Splunk search endpoint

2026-10-15

In a copy of the kit with git history that set DET-0001 to deploy three days earlier, tools/replay.py --production ran against the container's REST API.

It reported 3.0 of 7 days live and a count of 0, and no error. The count was 0 because the admin role's default search scope leaves out the fixture index, so a detection that fires could have passed as quiet. Since v0.85.0 the count searches the indexes named in SPLUNK_INDEXES, and the promotion check fails a count of 0 when that variable is unset.

The search request and the window from git history work end to end, and a count needs its indexes named. The promotion check itself has run only against a stubbed search, in the kit self-test.

The kit's field check, in a real Splunk

2026-10-29

tools/fields.py checked DET-0001 against Splunk 10.6 (splunk/splunk:latest) in a container, over indexes loaded with its five fixture events: as recorded, with ServiceName renamed, with ServiceName blank, with TicketEncryptionType removed, under another source name, and under another event id. A review of the first version, on 2026-10-29, found two faults. It took a selection's name as the sign of an exclusion, where the rule's condition is the sign. It also counted every event from the source, so a missing event type read as a renamed field.

It passed the events as recorded and failed the other five. Each failure named the field or the source and what it breaks: without TicketEncryptionType the rule cannot match, and without ServiceName its two exclusions never apply. The count now covers only the event the rule fixes, 4769, and a source that flows without it fails as a stopped event. Splunk refused every search on the first run because its disk had less than 5 GB free, and the check now prints Splunk's reason.

The check catches a renamed, blank, or missing field, a source that arrives under another name, and an event that stopped being logged, on these events. It has not run against a production Splunk.

The kit's running check, in a real Splunk

2026-10-29

tools/package_app.py built the Splunk app with DET-0001 at tune, and the app was installed in Splunk 10.6 (splunk/splunk:latest) in a container. The check, tools/running.py, then ran after each change made through Splunk's REST API, the interface its console uses.

It passed the app as built. It failed an exclusion added to the query, naming the words added, and failed the search disabled, the alert threshold raised, alerting switched off, and a second search added to the app. It also failed alerting left on after the record moved back to deploy, a live detection with no saved search, and a missing app. A review on 2026-10-29 found three console edits the first version passed, because it compared only the settings the package writes: throttling, a custom trigger condition, and a schedule window. It fails each of them now. Splunk refused to delete the packaged search, so a console user can disable one but not remove it.

The check sees the console edits Splunk allows on a packaged search, on one detection. It has not run against a production Splunk, and it reads only Splunk.

The kit self-test

every build

tools/kit_selftest.py copies the kit, breaks it 61 ways, and runs the tool that should notice each break. Eight of the cases test the core lock. It must refuse a changed selection, a removed filter, a one-field filter, and a filter that wildcards a directory. It must accept a three-field filter, the same rule reordered, and any change to a rule not yet live.

Every case passes on this kit. Against the v0.61 kit, 32 of the 33 cases that existed then failed.

Each check in the kit catches the fault it exists for.

One pass of the closed loop, by hand

2026-10-16

The author walked section 18's six stages against the 107 recorded attacks, on LSASS memory dumped through comsvcs.dll. The pass counted the two fields the candidates read, wrote the rule, and computed its core hash with tools/corelock.py. It then replayed that rule and SigmaHQ's rule for the same procedure over every event in the corpus.

Both fields were filled in every event of their type. The candidate fired on 1 event and SigmaHQ's rule on 2, all in the one recording of the procedure, and neither fired elsewhere. The core hash survived a reordered file. No model ran any stage, no SIEM received the rule, and no pull request was opened.

Ground, author, prove, and the core lock work on real events with the kit's tools. Extract and tune by a model, the signed fixture, and the alert-rate check remain untested.

DET-0001's fixtures

2026-10-15

The five fixture events are real event 4769 records from OTRF Security-Datasets SDWIN-201022042947, a recording of PurpleSharp running T1558.003 in a lab domain.

The rule fires on the recording's 5 RC4 service-ticket requests and on nothing in the other 104 recordings with events. Two of the 107 recordings are empty.

The kit's example detection matches a real Kerberoasting run, and no other recorded technique.

The regulatory crosswalk, against the regulations' text

2026-10-21

An agent read every mapped control and clock in its source: NIST's CSF 2.0 PDF, EUR-Lex for DORA and NIS2, the SEC's Form 8-K, eCFR for HIPAA, and the NYDFS PDF. PCI DSS and SOC 2 came from secondary copies, and ISO from its published titles.

The 16 NIST outcomes were quoted word for word, and every base clock was right. Of the 59 paraphrases, 21 left out or narrowed part of what the control asks. One clock ran later than the law, since NIS2 gives trust service providers 24 hours, and DORA's late classification case was marked late though the law allows it. Each was corrected in v0.82.0.

What each control asks, read from its source. Whether a record meets the control stays the auditor's call.

Still opinion

Nothing outside the site has checked these. Where the site states them, it says they are the author's choices.

  • Section 8.9's run with Stratus Red Team. No technique has been detonated from this repository. The commands were run offline against a binary built from the tool's source, and the fields and marks come from that source and its published detonation logs.
  • Section 19's questions for AI products and its shadow-mode thresholds. No team has graded a product with them yet, and the product tables there come from each maker's documents.
  • The closed loop of section 18 as a loop. No team or agent has run it end to end; one pass by hand, in 18.9, exercised ground, author, prove, and the core lock. Its targets are expectations, and each one names the metric that would confirm it.
  • The score model: its six inputs, the anchors behind each, the presets, k, and the 0.8 and 0.5 of the marginal factor. No real backlog with incident outcomes has been scored, and no two people have scored the same candidates to measure agreement.
  • The lifecycle's numbers: 7 days with alerting off, 30-day precision windows, the 24-hour self-approval wait, the 120-minute cap per true positive. They are defaults the author chose.
  • The KRI tolerances of section 15.2, the miss codes of section 11.5, the OKR and roadmap rules of section 11.6, and the four response dimensions of section 13. They are defaults the author chose, and no team has run a quarter with them yet.
  • The regulatory crosswalk of section 11.7. Which records answer which controls is the author's reading, and no auditor, regulator, or counsel has reviewed it.
  • The maturity levels, the three ceilings, and the breakpoints by team size.
  • The coverage map's 'ready to build' rule and its roll-up from sub-techniques, beyond agreeing with ATT&CK's own analytics.
  • The five ATLAS mitigations the ATLAS and D3FEND page counts as detecting an attack in progress. The author chose them from each mitigation's description, and ATLAS itself does not sort its mitigations that way.
  • The metrics page's sample numbers, which are made up to show the page.
  • The query translator's claim that its outputs mean the same thing, beyond its own self-test and the five engines above on one rule.
  • Every library rule with no recording of its technique, and every rule that stayed silent on a recording with the right event type. A recording exercises one procedure, and a rule for the same technique may watch another, so silence there is not evidence either way.
  • False positives: no benign recording exists in the corpus, so nothing here measures how often a rule fires on normal activity.