DETECT hunt Build detections

IdeaProductionShipMeasure

Section 19

Choosing AI for detection and response

Framework v0.136.0, updated November 17, 2026.

Do it in the browser Grade an AI product

Remember

  • Adopt AI where a bad output costs one review, such as drafting into a pull request, before uses where it costs an incident.
  • Count an agent's benefit in analyst minutes saved and its price in alerts it wrongly closed, and report the two together.
  • Run an AI triage agent in shadow mode on the team's own alerts, and replay every confirmed incident through it before it closes anything.
  • Ask for all four counts on the team's own alerts, with the number of real attacks. A share of alerts closed hides every attack the agent missed.
  • Ask any product where its rules go, which events prove them, and who switches alerting on. Grade it by its weakest answer.
  • Treat a new model, prompt, or vendor release as a new agent that goes back through shadow mode.

An AI detection and response program pays for itself in analyst and engineering minutes. It fails when it misses an attack it was trusted to catch. Every choice in this section weighs the minutes an AI use saves against the evidence that it misses nothing. The same tests apply whether the AI ships inside a SIEM, comes from a separate vendor, or is assembled from open-source parts.

Section 18 says what a safe automated detection pipeline looks like. This section turns that design into a buying and building decision. It covers which uses to adopt first, which path fits a team, what to ask any product, and how to prove an AI triage agent before it closes an alert.

On this page

19.1Where AI earns its keep

Each use below saves a kind of minute the scaling ceilings of section 14 count, and each can break something the lifecycle exists to catch. The order is the order to adopt them in.

Use Minutes it saves What it can break What proves it When to adopt it
Drafting detections and records Engineering minutes spent writing a rule, its record, and its conversions. A rule keyed on one tool's fingerprint, a field the logs do not carry, or a test event the model wrote. The closed loop's checks in CI, and the share of drafts still live at 90 days. First. Every draft goes through a pull request, so a bad one costs one review.
Translating rules between platforms Engineering weeks in a SIEM migration. A translation that reads the same and matches different events, as with case sensitivity or a renamed field. The same fixtures replayed in the old engine and the new one, with identical matches. First, during any migration.
Enriching and summarizing an alert Analyst minutes per alert, which raises the triage ceiling. A summary that leaves out the deciding fact, or states one the data does not show. A weekly sample in which analysts mark each summary right, incomplete, or wrong. Second. It only reads, and the analyst still decides.
Triage verdicts that close alerts More analyst minutes than any other use. It closes a real attack, the costliest failure the program can have. Shadow mode on the team's own alerts, with every confirmed incident replayed through it, as 19.4 describes. Third, one alert class at a time, after shadow mode passes.
Tuning proposals Upkeep minutes spent on noisy rules. An exclusion broad enough to blind the rule. The core lock and the exclusion policy of section 18, with the true-positive fixtures still firing. Once the core lock runs in CI.
Response actions Minutes to contain an intrusion. A wrong action that causes an outage, or one an attacker steered with text planted in a log. A playbook the team wrote for each action, a log of every action taken, and the count of actions people reverted. Last. Without a person's approval, only actions that can be undone in one step.

19.2Build, buy, or assemble

Path It fits when Watch for Start with
Use the AI in the SIEM or XDR the team runs One platform carries most of the telemetry, and the team is small. Agent actions logged only in the vendor's console, verdicts that cannot be replayed, and per-unit pricing that grows with use. The investigation assistant, read-only. Then triage in shadow mode.
Buy an AI SOC product Alert volume is past the triage ceiling, and the telemetry spans several tools. Write credentials into the team's tools, accuracy claims with no method, and verdicts that cannot be exported. Shadow mode on 30 days of the team's own alerts with known outcomes.
Assemble from open models and MCP servers Logs cannot leave the team's tenant, or the team already runs detection as code. The team owns the evaluation, the defenses against planted instructions, and the on-call rota for the agent. A drafting agent that opens pull requests into the kit's CI and does nothing else.
Build the closed loop of section 18 The detection program is itself the product, as at a managed security provider or a team with its own platform group. The checks cost more than the model. Budget the field dictionary and the signed fixtures before the prompts. The 90-day rollout in 18.8.

A team can mix the paths by function. A common split takes triage from the SIEM vendor, builds drafting around the team's own repository, and keeps response with people until the action log has a quarter of history.

19.3What to ask any product

Each question comes from a check in section 18. A product that passes all of them behaves like the closed loop, whoever built it. The Grade an AI product tool records the answers and does the arithmetic.

Area Ask A passing answer A red flag
Extract What happens to instructions hidden in a threat report, a ticket, or a log line? Untrusted text reaches only a model with no tool access, and the vendor shows a test of it. The model has been trained to ignore them.
Ground Where do the field names in a generated rule come from? From the team's own data, queried read-only when the rule is drafted. From documentation, a schema the vendor ships, or the model's memory.
Author Where does a generated rule go when it is done? Into a pull request in the team's repository, through the team's CI. Straight into the SIEM, or into a vendor-side library the team cannot diff.
Prove Which events prove that a generated rule fires? Captured events from the team's environment or a named emulation run, stored beside the rule. Events the model wrote, or no test events at all.
Tune Can the agent change what a live rule catches? Only by adding a narrow, named exclusion that a person approves. The detection logic itself is locked. It adjusts live rules on its own to cut noise.
Ship Who switches alerting on for a new rule? A named person, after a period with alerting off and an alert count at or under its threshold. The agent, or anyone with access to the console.
Triage Which alerts can the agent close without a person, and who decided that? Classes the team chose after shadow mode, each with a measured false-close rate. Any alert above a confidence score the vendor set.
Investigate Can every query the agent ran be seen outside the vendor's console, with its time range and cost? Yes. The SIEM's own audit log records them under the agent's identity, or the product exports them. Only a written summary of the investigation.
Respond What can the agent do without a person's approval? A short list of actions that can be undone, each in a playbook the team wrote. Anything an analyst could do.
Access What credentials does the agent use? Its own identity, read-only and scoped, with write access only for the listed actions. An administrator's API key, or an analyst's account.
Data and model Where do the team's logs go, and can the model be pinned? A stated region, no training on customer data, and a pinned model version whose changes are announced. The model changes without notice.
Accuracy What were the four counts in the trial, real attacks escalated and closed and benign alerts escalated and closed? All four counts for each alert class, from the team's own labeled alerts during the trial, with the number of real attacks stated. A share of alerts closed, an agreement rate, or a percentage from an unnamed benchmark or other customers.

The grade is the weakest answer among the questions that count. Any failed question keeps that function with people until it passes.

19.4Proving an AI triage agent before it closes anything

An agent that closes alerts goes through the steps a detection takes before its alerting goes on. It runs where it can do no harm, its record is counted, and a person promotes it.

  1. Run the agent on every alert for 30 days with no power to close one. It records a verdict, and analysts decide as before without seeing it until they have.
  2. Replay the alerts of every confirmed incident and purple-team exercise from the last year through the agent. Each one must come back escalated. The number replayed sets the smallest miss rate the trial can show, as 19.7 explains.
  3. Count the agent's four cells for each alert class: real attacks it escalated or would have closed, and benign alerts it escalated or would have closed. Count analyst minutes per alert with and without its summary too.
  4. Promote one class to auto-close when it shows no wrong closes over at least 100 alerts and every replayed incident came back escalated. The team can raise either number.
  5. After promotion, analysts review a random 5% of each class's auto-closed alerts every week.
  6. One confirmed miss sends the class back to shadow mode. So does a new model, prompt, or vendor release, because each makes it a different agent.

19.5Metrics for AI in detection and response

Metric Unit Aggregation Definition
ai_false_close_rate percent per month Share of alerts later confirmed malicious, by an incident, an exercise, or the weekly sample, that the agent closed or would have closed.
ai_verdict_agreement percent per month Share of shadow or sampled alerts on which the agent's verdict matched the analyst's.
ai_minutes_saved_per_alert minutes median per month Analyst minutes per alert without the agent's summary, minus minutes with it, on the same alert class.
ai_actions_reverted count per month Response actions the agent took or proposed that a person then undid or refused.
ai_cost_per_true_positive currency per month The agent's spend on licenses, model tokens, and query compute, divided by the true positives it worked on.

The triage ceiling of section 14 rises as minutes per alert fall, so minutes saved is the agent's benefit. The false-close rate is its price, and a report shows the two side by side.

19.6What ships today

These tables list what each maker documents, read on 2026-10-17 from product documentation, release notes, model cards, and repositories, with DefendBench read on 2026-10-27. Each row maps a product to the loop's stages and lists the controls its maker documents. Grading is left to the questions in 19.3 and the scorecard tool.

Read side by side, the documents show the following.

  • Every SIEM and XDR platform read for this table ships AI for triage or investigation, and most let a person approve each response action.
  • Google's detection engineering agent and Panther's builder test the rules they write against events or test cases a model generated. Prophet backtests against the customer's own history, which counts a rule's noise. None of the products that write rules documents a test against a recorded run of the attack, the event section 18's Prove stage asks for.
  • No AI SOC product or platform publishes all four counts of its triage record from production. Qevlar publishes a false-negative rate without saying what it is out of, and 19.7 compares what each maker publishes.
  • Several MCP servers can change the tenant as soon as they run, so the access question in 19.3 applies to the open-source path too.
  • The one public benchmark that scores generated rules covers Azure and KQL. DefendBench scores rules on six SIEMs and keeps its tasks private. No defense against planted instructions reports a test on log fields.

AI in the SIEM or XDR

Google's detection engineering agent and Panther's builder are the two platforms here that test the rules they write.

Product Stages and functions What it does Controls it documents Sources
Security Copilot with Sentinel and Defender XDR (Microsoft). The triage agent is generally available for user-reported phishing and in preview for some identity and cloud alerts. The Sentinel MCP server is in preview, with some tools generally available. Ground, author, triage, investigate, respond with a person The triage agent classifies Defender alerts and resolves the ones it judges false positives, leaving true positives open. A hunting assistant writes and runs KQL, and a migration tool maps Splunk and QRadar rules to Microsoft's own. The agent runs under its own Entra identity with scoped roles, and its actions go to Purview audit logs. Analyst feedback becomes lessons in its memory, which an administrator can review or reject. learn.microsoft.com (1), learn.microsoft.com (2), learn.microsoft.com (3), learn.microsoft.com (4), learn.microsoft.com (5)
Gemini in Security Operations (Google Cloud). The triage agent is generally available. The detection engineering agent and event simulation have been in preview since 2026-08-18. Extract, author, prove, ship, triage, investigate A triage agent investigates SIEM alerts and returns a verdict with a confidence and a summary. A detection engineering agent, run from MCP clients, drafts YARA-L where no existing rule matches the intelligence, and tests rules against simulated events. The user approves or rejects each draft rule. Simulated events are machine-generated, ingested live under a SIMULATION label the caller must add, and counted against quota. Agent calls go to Cloud Audit Logs. docs.cloud.google.com (1), docs.cloud.google.com (2), docs.cloud.google.com (3), docs.cloud.google.com (4), docs.cloud.google.com (5)
Charlotte AI (CrowdStrike). Detection triage has been available since 2025-02-13. The product documentation needs a customer login, so this row rests on public pages. Author, triage, investigate, respond Detection triage gives each endpoint detection a priority, a true or false positive call, and a recommended action. Named agents write correlation and YARA rules and translate Splunk queries into CrowdStrike's query language. It takes no automated response by default, and each action can run on its own or wait for approval, set per workflow. Responses respect the user's permissions. The public pages do not say where audit logs or data are kept. crowdstrike.com (1), crowdstrike.com (2), crowdstrike.com (3)
Cortex XSIAM with Cortex AgentiX (Palo Alto Networks). AgentiX has been available in XSIAM since 2025-10-28. Agentic response and AI data-model rules are in preview. Ground, author queries, triage, investigate, respond Agents plan and run playbooks, scripts, and commands on cases, write and run XQL, and summarize cases. An AI feature proposes field mappings for raw data from sampled logs. Sensitive actions need a user's explicit approval, and an audit dataset names the agent and the user behind each action. EU and US tenants pick the model per chat. Prompts stay in region and are not used for training. cortex-docs.paloaltonetworks.com (1), cortex-docs.paloaltonetworks.com (2), cortex-docs.paloaltonetworks.com (3), cortex-docs.paloaltonetworks.com (4)
Splunk Enterprise Security (Splunk, a Cisco company). Enterprise Security 8.7 was released 2026-09-02. The AI SOC analyst is in the Premier edition. Author, tune, triage, investigate, respond A detection builder drafts and tunes SPL detections in the editor, and it cannot save, test, or change a detection's status. An AI SOC analyst triages findings and runs the response actions it is allowed to run. Each builder suggestion needs Approve or Reject, and each response action is set to run automatically or wait for approval. The MCP server blocks REST writes, and its Enterprise Security tools only read. help.splunk.com (1), help.splunk.com (2), help.splunk.com (3), help.splunk.com (4)
Elastic Security (Elastic). Agent Builder has been generally available since 9.4, with AI rule creation in the Enterprise tier. Attack Discovery is generally available. Ground, author, triage, investigate, respond Drafts an ES|QL rule with an ATT&CK mapping from a description and previews it on live data. Automatic Migration translates Splunk, QRadar, and Sentinel rules, and Attack Discovery groups alerts into attacks. Rule drafts must be applied by hand, and migrated rules are installed for the user to enable. The customer picks the model connector and can anonymize fields. Everyone using one MCP API key acts with that key's permissions. elastic.co (1), elastic.co (2), elastic.co (3), elastic.co (4), elastic.co (5)
Purple AI (SentinelOne). Auto-triage has been generally available since April 2025. The product documentation needs a customer login, so this row rests on public pages. Triage, investigate, respond Investigates alerts without an analyst, collects evidence, and issues verdicts that can trigger automated responses. No detection-writing feature is documented publicly. Automated actions run only within policies the customer approved, and each verdict carries an evidence chain. Customer data is not used for training. sentinelone.com (1), sentinelone.com (2)
Panther AI (Panther Labs). The AI Detection Builder has been in open beta since version 1.118, and automatic closing of low-risk alerts is in closed beta. Author, prove, ship, triage, investigate The builder writes Python detections, log types, and test cases, and triage judges each alert a true or false positive. Write tools wait for a person to accept or reject them, and each decision goes to Panther's audit logs. Builder changes save only when a person clicks Deploy. Claude runs through Amazon Bedrock in the customer's single-tenant AWS account. docs.panther.com (1), docs.panther.com (2), docs.panther.com (3)
Cloud SIEM SOC analyst agent (Sumo Logic). The help page states no release stage. Triage, investigate Investigates each Cloud SIEM insight in the background and labels it malicious, suspicious, benign, or inconclusive, with its key findings. It takes no remediation or containment action, and its recommended severity applies only when an analyst accepts it. Every tool call is logged to Cloud SIEM. Models run on Amazon Bedrock, with no training on customer data. sumologic.com (1), sumologic.com (2)

AI SOC products

Products that triage and investigate alerts from the team's existing tools. Qevlar alone publishes a false-negative rate, from a test it ran, and Intezer alone publishes results with a sampling method. Section 19.7 compares what each publishes.

Product Stages and functions What it does Controls it documents Sources
Dropzone AI SOC Analyst (Dropzone AI). Triage, investigate, respond Investigates each alert it receives and writes a conclusion with its evidence. Containment runs when a user starts it, and customer-written Python actions can run without a click. Read-only access by default. A changelog and a system events log record activity. Models from Anthropic, Azure OpenAI, and others, in AWS's Oregon region, or in the EU on request, with no training on customer data. docs.dropzone.ai (1), docs.dropzone.ai (2), docs.dropzone.ai (3), dropzone.ai
Socrates (Torq). Triage, investigate, respond An agent inside Torq's automation platform that triages and investigates cases and runs response workflows. Asks a user to confirm sensitive actions and runs only the tools given to it. Torq's audit log records every action with the user who asked. Torq's model or the customer's own subscription, in US, EU, or Japan regions. kb.torq.io (1), kb.torq.io (2), kb.torq.io (3), kb.torq.io (4)
AI SOC Analyst (Prophet Security). Triage, investigate, respond Investigates every alert with a plan built for that alert, and sends its findings to case management. Takes only actions the customer approved, each previewed and backtested first. Documents every query and reasoning step. Single-tenant with a bring-your-own-key option, and setup asks for read-only access. prophetsecurity.ai (1), prophetsecurity.ai (2), prophetsecurity.ai (3)
Intezer AI SOC (Intezer). Triage, investigate, respond Triages every alert by combining forensic analysis, such as sandboxing and memory analysis, with AI models, and closes false positives automatically. Remediation can be automated with a person's explicit approval, and any verdict can be challenged. Experts review a random 5% of alerts each week, stated at 95% confidence with a margin under 2%. Models and residency are not documented. intezer.com (1), intezer.com (2)
CognitiveSOC (Conifers.ai). Author, prove, tune, ship, triage, investigate, respond Agents investigate alerts to a verdict and a remediation plan. Others write, deploy, and tune detections, testing rule changes in a sandbox. The customer picks a person in the loop, on the loop, or full autonomy. Each conclusion lists the queries run and the data touched. Runs in Azure in a chosen region, and reads from and writes to the customer's tools. conifers.ai (1), conifers.ai (2), conifers.ai (3)
Qevlar AI. Triage, investigate, respond when an analyst starts it An orchestrator plans the steps of an investigation and model agents carry them out. Each produces a verdict, a report, and a containment or tuning recommendation. Analysts review and start remediation. Every step and every queried observable is visible. Hosted in the EU on Google Cloud, with no training on customer inputs. qevlar.com (1), qevlar.com (2), qevlar.com (3)
Command Zero. Triage, investigate Runs investigations as sequences of curated questions that query the connected sources at the time, without ingesting their data. Connects through read-only APIs, logs every decision, and replays investigations. Analysts can take over any autonomous run. Model and residency are not documented. commandzero.ai

AI for detection engineering

Products that apply a model to writing, translating, testing, or tuning rules. None of them publishes an accuracy figure for the rules it writes.

Product Stages and functions What it does Controls it documents Sources
AI Detection Engineer (Prophet Security). Announced 2026-07-30 for its AI SOC Analyst customers. Ground in part, author, prove, tune, ship Maps coverage from the customer's own detections, drafts detections for the gaps, and tunes noisy rules in Splunk, Sumo Logic, or Microsoft Sentinel. Each change is backtested against the customer's recent history. Suggests by default, and the team approves version-controlled changes before they deploy. Autonomy can be raised per category of work. The model is not named. prophetsecurity.ai (1), prophetsecurity.ai (2)
Anvilogic. Author, tune, ship, triage, investigate A detection layer over Splunk, Snowflake, and Databricks. Its agents plan detections for coverage gaps, write the queries, and deploy them, and users accept or dismiss tuning suggestions. Approval steps, roles, and budgets limit what agents run, with version history and rollback. Its chat assistant uses OpenAI-hosted models, with no training on raw customer data. anvilogic.com, public-docs.anvilogic.com (1), public-docs.anvilogic.com (2)
Uncoder AI (SOC Prime). Extract, author, ship Writes Sigma rules or indicator queries from pasted threat-report text, translates Sigma into 48 query languages, checks syntax, and suggests ATT&CK tags. Open-weight models hosted by SOC Prime, an opt-in OpenAI option for some functions, and no training on user data. Its validation checks syntax only, and the user reviews the output. help.socprime.com (1), help.socprime.com (2), help.socprime.com (3), socprime.com
NARC (Tidal Cyber). Extract Turns threat reports, blogs, and incident documents into ATT&CK procedures linked to groups, software, and campaigns. Analysts review and correct the extractions. Single-tenant with a choice of region on the enterprise tier. The model is not described. tidalcyber.com (1), tidalcyber.com (2), tidalcyber.com (3)
Detection Rule Validation (Picus Security). Ground, prove Reads the live rule inventory from Splunk, Microsoft Sentinel, or QRadar, checks that each rule's log sources send events, and runs attack simulations to see which rules fire. Its AI maps unmapped rules to ATT&CK. Integrates with read-only access and does not change rules. The customer applies any fix. picussecurity.com

MCP servers for security tools

Several servers expose tools that change the tenant as soon as they run. Read each tool list, start a server read-only where it offers a flag, and give it an identity of its own, as section 10.3 asks.

Product Stages and functions What it does Controls it documents Sources
MCP Server for the Splunk platform (Splunk). A Splunkbase app, version 1.3. Ground, author queries, triage, investigate Runs SPL searches and saved searches, lists indexes and metadata, and generates or explains SPL. Its Enterprise Security tools list findings, investigations, and response plans. A search may be refused past 1 minute or 1,000 events, or when it uses a command the server deems unsafe. REST writes are blocked, the Enterprise Security tools only read, and each tool needs a capability an administrator grants. help.splunk.com (1), help.splunk.com (2), help.splunk.com (3)
mcp-security (Google). Apache 2.0 local servers, plus a managed remote server. Ground, prove, tune, ship, triage, investigate, respond Local servers cover the SIEM, SOAR, threat intelligence, and Security Command Center. Some tools change the tenant, such as creating rule exclusions, changing curated rule deployments, and ingesting events. The remote server needs an IAM role, writes to Cloud Audit Logs, and can screen tool calls with Model Armor. The local servers run under the caller's Google Cloud credentials. github.com, raw.githubusercontent.com (1), raw.githubusercontent.com (2)
Microsoft Sentinel MCP server (Microsoft). Hosted. In preview, with some tools generally available. Ground, triage, investigate Tool collections explore the data lake with KQL, list and fetch incidents and alerts, run Defender advanced hunting, and create and deploy Security Copilot agents. Uses the caller's Entra identity and roles. The triage and exploration tools only query data, and deploying an agent writes. learn.microsoft.com (1), learn.microsoft.com (2), learn.microsoft.com (3)
Agent Builder MCP endpoint (Elastic). Generally available since 9.3. Ground, author, prove, triage, investigate Serves Agent Builder's tools, among them ES|QL queries, index mappings, security alerts, a tool that writes an ES|QL detection rule from a description, and rule preview. An API key gives every caller one shared identity, so Elastic advises scoping it to the needed indices and rotating it. Per-user OAuth is available on Serverless only. elastic.co (1), elastic.co (2), elastic.co (3)
falcon-mcp (CrowdStrike). MIT. Pre-1.0, maintained by CrowdStrike, and described as not an official product. Ground, author, tune, triage, investigate, respond Covers 28 Falcon modules, including detections, hosts, SIEM search, intelligence, exclusions, correlation rules, and SOAR, with tools that create, update, and delete exclusions and run workflows. A --read-only flag registers only the tools that read. HTTP transports have no authentication unless an API key is set. github.com
purple-mcp (SentinelOne). MIT. Maintained by SentinelOne and described as not an official product. Ground, triage, investigate Asks Purple AI questions, runs PowerQuery over the data lake, and reads alerts, vulnerabilities, misconfigurations, and assets. Read-only, with one token scoped to an account or a site. It has no built-in authentication, so it belongs off shared networks or behind a proxy. github.com
mcp-panther (Panther Labs). Apache 2.0. Ground, triage, investigate Runs data-lake SQL, lists schemas and log sources, updates alerts, and lists or disables detections. Uses a scoped Panther API token, and the README advises starting with a read-only one. Disabling a detection is among its tools. github.com
Wazuh-MCP-Server (gensecaihq, a community project). MIT. Ground, triage, investigate, respond Exposes 55 tools for Wazuh, 41 that read and 14 that act, such as blocking an IP, isolating a host, or disabling a user, each with a rollback. Read-only by default. Write tools need a write scope and a confirmation in production, and each is audit-logged. github.com
Sumo Logic MCP Server (Sumo Logic). Hosted. Generally available since 2026-07-30. Ground, author, triage, investigate Searches logs and alerts, creates and updates dashboards, and works with Cloud SIEM insights and detection rules. Uses OAuth 2.0, and its tools respect existing permissions. The release note does not say which rule operations are allowed. sumologic.com

Open models trained for security

The model cards report knowledge tests such as CTIBench and CyberMetric. None reports a result for writing or translating rules.

Product Stages and functions What it does Controls it documents Sources
Foundation-Sec-8B and its variants (Cisco Foundation AI). The base model is Apache 2.0. The instruct and reasoning variants use the Llama 3.1 license, with Cisco's changes under Apache 2.0. Extract, triage, investigate Eight-billion-parameter Llama 3.1 models trained further on about 5.1 billion tokens of security text, with instruct, 64k-context, and reasoning variants. The cards report CTIBench scores. The reasoning card says the model is unsuited to autonomous security decisions. The cards recommend an extra safety filter and human oversight. huggingface.co (1), huggingface.co (2), huggingface.co (3)
Llama-Primus (Trend Micro AI Lab). MIT, plus the Llama 3.1 license. Extract, investigate Eight-billion-parameter Llama 3.1 models trained further on 2.77 billion tokens of security text, which Trend Micro publishes, then tuned on about 1,000 security tasks. The card reports small losses on general benchmarks and has no limitations section. huggingface.co, arxiv.org
SecureBERT 2.0 (Cisco AI). Apache 2.0. Extract A 150-million-parameter encoder trained on threat reports and code, with an entity model that tags indicators, malware, and vulnerabilities in text. It labels and embeds text and generates none. Its entity labels do not include ATT&CK techniques. huggingface.co (1), huggingface.co (2)

Open-source detection engineering projects

SigmaOptimizer is the one project here that tests a generated rule against captured logs and a benign baseline, the pattern section 18 asks for.

Product Stages and functions What it does Controls it documents Sources
SigmAIQ (AttackIQ). LGPL 2.1. Pre-release. Author Wraps pySigma with an optional LangChain toolkit for writing and translating Sigma rules with a model, and converts rules to 18 backends. The README warns that it is pre-release and may have bugs. github.com
SigmaOptimizer (Yusuke Nakajima). MIT. The original repository is unmaintained, and the author points to a UI version. Author, prove, tune Runs a command, captures the Windows and Sysmon logs, and has a model draft a Sigma rule from them. It tests the rule with Hayabusa against those logs and a benign baseline, then refines it. Windows only. The default benign baseline is one bundled set of Windows 10 client logs. github.com (1), github.com (2)
Security-Detections-MCP (MHaggis). Apache 2.0, per its README. Author Indexes about 8,200 detections from Sigma, Splunk, Elastic, KQL, Sublime, and CrowdStrike, so a model can search them by technique, CVE, or process, find coverage gaps, and open a draft pull request. Local mode reads the repositories the user points it at. A hosted mode offers read-only tools behind a token. github.com
TRAM (MITRE Center for Threat-Informed Defense). Apache 2.0. Extract Maps sentences in threat reports to ATT&CK techniques with fine-tuned SciBERT models, covering up to 50 common techniques, and lets users annotate and retrain. Publishes no accuracy figures, and targets ATT&CK v13. github.com
CTINexus (peng-gao-lab). MIT. Extract Uses a model to pull entities and relations out of threat-intelligence text, merge duplicates, and draw a knowledge graph, with OpenAI, Gemini, Bedrock, or a local Ollama model. Its README gives no evaluation figures. github.com, arxiv.org
Atomic Red Team MCP (cyberbuff, a community project). MIT. Prove Lets an agent search Red Canary's Atomic Red Team tests, check a test against the schema, and optionally run one to produce telemetry. Running tests stays off until an environment variable turns it on, and the README says to run them only in isolated machines. github.com

Benchmarks and recorded attack data

CTI-REALM is the one public benchmark found that scores rules an agent writes from threat reports, and it covers Azure and KQL. Spectrum's DefendBench scores writing, tuning, and repairing rules on six SIEMs, but its tasks are private. No public benchmark found scores tuning, and GUIDE's labels are the nearest public data for triage.

Product Stages and functions What it does Controls it documents Sources
CTI-REALM (Microsoft Security AI). Evaluation code in inspect_evals. The dataset states no license. Extract, ground, author, prove Agents read threat reports, explore the data, and write detections over emulated Linux, AKS, and Azure attacks in a Kusto emulator. Scores weigh both the steps taken and the detections' results. The authors say it centers on Azure and does not reproduce production-scale telemetry. The best of 16 models reached a reward of 0.637. arxiv.org, github.com
DefendBench (Spectrum). Announced 2026-10-27. The tasks stay private to keep them out of training data, and Spectrum plans to release the simulated environment. Extract, ground, author, tune Agents work 100 tasks in threat research, rule writing, tuning, rule health, and data engineering on Splunk, Sentinel, Elastic, Google SecOps, CrowdStrike, and Databricks. Each task runs in a simulated enterprise built from real environments and passes only if it meets all of its checks. An agent grades some of the checks. The best of 14 entries, each a model in a coding agent, passed 31 of 100. At 95% confidence that score fits a pass rate from about 23% to 41%, a range that takes in the top five entries. The post gives no pass rate for experts other than each task's author, and no counts of attacks kept or benign events removed on tuning tasks. spectrum.security (1), spectrum.security (2)
ExCyTIn-Bench (Microsoft). MIT code. The dataset states no license. Ground, investigate Agents answer 7,542 investigation questions by querying 57 log tables drawn from 8 incidents. Needs about 33 GB of disk, and the best model in the paper reached a reward of 0.606. github.com, arxiv.org
CyberSOCEval (Meta and CrowdStrike). MIT code, with data licensed per folder. Extract, investigate Questions on malware analysis and on reasoning over threat-intelligence reports, built from sandbox output and CrowdStrike reports. Its paper says current models are far from saturating it. github.com, arxiv.org
CTIBench (Rochester Institute of Technology). CC BY-NC-SA 4.0. Extract Five threat-intelligence tasks, among them mapping CVEs to CWEs, scoring CVSS, and extracting ATT&CK techniques from reports. The extraction and attribution sets are small, at 60 and 50 items. github.com, arxiv.org
GUIDE (Microsoft Security Research). CDLA 2.0, per its paper. Triage, tune About one million incidents from 6,100 organizations, each labeled true positive, benign positive, or false positive by the customer's analysts. Its labels are per incident, so it tests triage and leaves rules untested. Microsoft's guided-response paper publishes a full confusion matrix for one region's test split. arxiv.org, kaggle.com
SIR-Bench (Amazon Web Services researchers). Preprint, 2026-04-13. The authors plan to release the test cases, labels, and replay framework. Triage, investigate 794 cases built from 129 anonymized incident patterns, replayed in cloud environments for an incident response agent to triage and investigate. 475 are true positives and 319 false positives. Its results give all four cells for the authors' agent, which missed 14 of 475 true positives and raised 85 false alarms on 319 false positives. arxiv.org (1), arxiv.org (2)
ALERT-BENCH (Virginia Tech, Google, and North Carolina State University researchers). Preprint, 2026-10-07. No release of the benchmark is stated. Triage Replays the AIT-LDSv2 log dataset through a Wazuh SIEM, producing 225 alerts tied to a multi-stage attack and 1,022 benign ones for an agent to triage. Reports the false-negative rate beside the share of alerts escalated. Five approaches on GPT-4.1 each missed at least 40.4% of the attack alerts, and the authors' AIDA missed 7 of 225 while escalating 18.4% of all alerts. arxiv.org (1), arxiv.org (2)
CyberSecEval 4 (Meta). MIT. Any stage Measures a model's help with attacks, its false refusals, prompt injection, and code-interpreter abuse, among other risks. Its prompt-injection tests apply to any agent that reads untrusted text. github.com
Security-Datasets (OTRF). MIT. Ground, prove Recorded attack and benign telemetry from simulated techniques, mapped to ATT&CK. The library on this site replays its rules against these recordings. Each recording covers one procedure, so a rule's silence on one says little. github.com
attack_data (Splunk). Apache 2.0. Ground, prove Raw attack logs filed by ATT&CK technique, most produced with Attack Range, ready to replay into a SIEM. More than 9 GB through Git LFS, and a single dataset can be fetched alone. github.com
BOTSv3 (Splunk). CC0. Investigate, prove About 320 MB of indexed Splunk data across about 100 sourcetypes, built for a capture-the-flag in 2018. It needs the Splunk app versions it was built with. github.com

Defenses against planted instructions

Each is tested on chat and agent benchmarks. None of the pages read for this table reports a test on instructions planted in log fields, the attack section 10.3 assumes.

Product Stages and functions What it does Controls it documents Sources
Llama Prompt Guard 2 (Meta). Llama 4 license. Any stage that reads untrusted text A small classifier that flags text trying to override instructions. Its card reports recall of 97.5% at a 1% false-positive rate on English text. It reads 512 tokens at a time, so long inputs must be split, and the card says adaptive attacks can succeed. huggingface.co
LlamaFirewall (Meta). MIT. Any stage that reads untrusted text Combines Prompt Guard, a model that audits an agent's reasoning for goal hijacking, and checks on generated code. Its paper calls the reasoning audit experimental, and the framework needs a model to run. github.com, arxiv.org
Meta SecAlign (Meta). Llama 3.1 and 3.3 licenses for the models. Any stage that reads untrusted text Llama models fine-tuned to ignore instructions inside untrusted data placed in a separate input role. On the 8B model, attack success on InjecAgent fell from 15.1% to zero. It works only when untrusted text goes in the input role, and adaptive attacks still succeed at times. huggingface.co, arxiv.org
CaMeL (Google Research). Apache 2.0. Research code. Ground, triage, investigate, respond Takes control flow and data flow from the trusted request, so untrusted data cannot change what the program does, and enforces data-flow rules at each tool call. Its README calls it a research artifact that likely contains bugs and is unmaintained. github.com, arxiv.org
NeMo Guardrails (NVIDIA). Apache 2.0. Any stage that reads untrusted text Programmable checks on input, dialog, retrieval, tool calls, and output, including jailbreak and injection detection. The README says its built-in checks may not suit a given production use. github.com
Agent Scan (Snyk). Apache 2.0. Before an agent's tools are trusted Scans MCP server configurations, tool descriptions, and agent skills for injection, tool poisoning, destructive tools, and exposed secrets. Sends tool names and descriptions, with secrets redacted, to Snyk's API. Scanning can start local MCP servers, so it belongs in a sandbox. github.com
MCP Scanner (Cisco AI Defense). Apache 2.0. Before an agent's tools are trusted Scans MCP tools, prompts, resources, and server code with YARA rules, a model judge, and heuristics. Static scanning works offline, and the API engine needs a Cisco subscription. github.com

19.7Reading an accuracy claim

A triage agent's record has four cells. A true positive is a real attack it escalated, and a false positive is benign activity it escalated. A true negative is benign activity it closed, and a false negative is a real attack it closed. The false-close rate of 19.5 divides the false negatives by every real attack the agent saw.

Common figures hide the false negatives. A share of alerts closed automatically counts true and false negatives together, so every missed attack sits inside it. An agreement rate counts the alerts where agent and analyst made the same call, and both can miss the same attack. A sample of closed alerts measures the share of closes that were wrong, which stays small when benign alerts dominate. With a hundred benign alerts for each attack, an agent that closes half the attacks and nine in ten benign alerts makes about 0.6% of its closes wrong.

Only real attacks show how often an agent misses one. An agent that misses none of 30 real attacks may still miss up to 1 in 10, at 95% confidence. One that misses none of 100 may still miss up to 3 in 100. Eleven attacks with none missed leave room for a miss rate of 24%.

Ask any maker for all four counts for each alert class, from the team's own alerts during the trial, with the number of real attacks stated. The table lists what makers and the researchers studying their products publish, read on 2026-10-18, with the most complete first.

Publisher What it publishes What it leaves out Sources
SIR-Bench (Amazon Web Services researchers, 2026) Counts that give all four cells for the authors' own agent on 794 replayed cases. It caught 461 of 475 true positives and rejected 234 of 319 false positives. A commercial product and production alerts. The cases are not yet released. arxiv.org (1), arxiv.org (2)
Copilot Guided Response (Microsoft researchers, 2024) A full confusion matrix for one region's test split of GUIDE, real incidents graded true positive, benign positive, or false positive. Macro-F1 is 0.87, and misgrading a true positive as benign or false positive is rare, at 2.4%. Results from the triage agent Microsoft ships. The model in the paper is a research system. arxiv.org (1), arxiv.org (2)
ALERT-BENCH (Virginia Tech, Google, and North Carolina State University researchers, 2026) False-negative rates beside the share of alerts escalated, on 225 attack and 1,022 benign alerts. Every approach studied missed at least 40.4% of the attacks, and the authors' AIDA missed 7 while escalating 18.4% of alerts. A full matrix for every system, a second attack scenario, and a release. arxiv.org (1), arxiv.org (2)
Charlotte AI reasoning triage (CrowdStrike researchers, 2026) Precision and recall for each verdict on 42,686 real Windows endpoint detections kept out of training. Benign calls tuned to 98% precision on validation data were 90.8% precise on the test set, so about 1 in 11 was a real attack. Malicious calls were 98.9% precise, and overall accuracy is 82.6%. Counts and the share of detections that were malicious. The product claims more than 98% agreement with Falcon Complete's analysts, which leaves misses both made uncounted. arxiv.org, crowdstrike.com (1), crowdstrike.com (2)
Qevlar AI (2025 test) 99.8% accuracy, 0.3% false positives, and 0.1% false negatives on 5,000 email alerts that analysts at four organizations had reviewed in January 2025. The analysts themselves scored 97%, 4%, and 2%. What each percentage is out of and how many alerts were malicious. The three figures do not add up on any one basis. qevlar.com
Intezer (2024 review) Production rates for 2024 across over 5.4 million alerts in 500+ customer environments, from a sample of about 5%. Accuracy was 93.45% on the 3.81% of alerts it escalated and 97.68% on the 68.40% it closed, at 95% confidence with a margin under 2%. Counts, a definition of accuracy, and how many of the wrong closes were real attacks. intezer.com (1), intezer.com (2)
Phishing Triage Agent trial (Microsoft, 2025) The agent's precision and recall, both 100%, on 93 user-reported emails, 11 of them malicious. Enough attacks to bound the miss rate, which could still be 24%. arxiv.org (1), arxiv.org (2)
Exaforce (2025 review) 98% accuracy across 1.29 million alerts in 2025, defined as AI recommendations that matched outcomes confirmed by human review. How many alerts were reviewed. The same post says 98% of alerts were benign or false positives after review, so closing every alert would score about the same. exaforce.com
Prophet Security 99.8% agreement with the analysts of one Fortune 500 manufacturer across 12,000 investigations, and a 98.5% accuracy rate on its product page. How agreement and accuracy are measured, and any count of misses. prophetsecurity.ai (1), prophetsecurity.ai (2)
Conifers.ai Accuracy above 99% across roughly 500,000 investigations, checked against the ground truth of each customer's environment. A definition of accuracy, a period, and any split by verdict. conifers.ai
ReliaQuest GreyMatter 99.4% accuracy on routine tier 1 and tier 2 alerts. A definition, a sample, a method, a date, and any word on misses. reliaquest.com
Torq Auto Triage accuracy that rises from roughly 94% to 99% within weeks. A definition, a sample, or a method. torq.io
Elastic (agentic SOC benchmark) Scores from 1 to 10 for each model on eight intrusions and roughly 95 alerts, graded against an answer key Elastic built, with misses marked down. Precision, recall, and any result from customer alerts. elastic.co
Splunk (Cisco, .conf26) The triage agent's suggested dispositions at the .conf26 event: 44% true positive, 42% benign positive, 6% false positive, and 7% with no suggestion. Whether the suggestions were right. Cisco says to read the figures as observations from the event environment. blogs.cisco.com
Command Zero A 95 to 98% auto-triage rate. Any accuracy figure. A share handled automatically counts the misses with the correct closes. commandzero.ai
Google Security Operations Over 5 million alerts investigated by its triage agent, each in about 60 seconds. Any accuracy or miss figure. cloud.google.com
Sumo Logic That senior analysts regularly review a sample of the agent's verdicts. Any result from that review. sumologic.com
Dropzone AI (quality control post) Definitions of precision and recall, tracked as rolling averages and checked with ISO 2859-1 sampling. Any figure from that tracking. Simbian's ranking reports that Dropzone's CEO cited a false-negative rate below 1% in a press interview. dropzone.ai, simbian.ai

A precision tuned on one set of alerts can slip on the next. In CrowdStrike's paper, benign calls tuned to 98% on validation data reached 90.8% on the test set, which is why 19.4 keeps sampling closes after promotion.

Palo Alto Networks, SentinelOne, and Panther publish volume and speed figures for their triage agents, and this reading found no accuracy figure from them. It found none for 7AI, Radiant Security, Arctic Wolf, or Expel either.

Simbian's 2026 ranking of AI SOC vendors says almost no vendor publishes a measured false-negative rate, and that Simbian does not publish one either.

A benchmark's pass rate needs the same reading. Ask how people other than each task's author score, and how wide the interval is. The best score on DefendBench, 31 of 100 tasks, fits a true pass rate anywhere from about 23% to 41% at 95% confidence. A tuning score needs the four counts as well. Ask for the attack events a tuned rule still catches and those it lost, and the benign events it still matches and those it dropped.

The tables in 19.6 list what each maker documents on the date given. Products change monthly, so check a claim against its linked page before relying on it.

The author has not run any product in those tables. The questions in 19.3 and the steps in 19.4 are how a team finds out for itself.

The thresholds in 19.4 are defaults the author chose, and the outside checks page lists them as untested.