CTI-REALM tests whether AI agents can produce working detections

Microsoft introduced CTI-REALM on 20 March as an open-source benchmark for detection engineering. It asks agents to read threat reports, inspect telemetry, revise queries and produce detection logic, rather than only answer questions about attacker techniques. The published account describes tasks spanning Linux, Kubernetes and Azure environments.

The benchmark uses reports selected for reproduction in a controlled environment and scores outputs against reference behaviour. That design makes it possible to test a workflow, including whether a proposed detection finds the simulated activity. It is still a bounded evaluation. A strong score does not establish that an agent can safely operate an organisation’s live security desk, where telemetry, permissions and incident costs differ from the benchmark.