Unit tests and replay for detection rules

Unit tests check a rule against a handful of fixed events. Replay runs it over stored data to see volume and false positives. You need both, and platforms document them unevenly: Panther, Splunk and Elastic ship unit tests, Google and Sentinel lean on replay, and several vendors document neither.

A rule without a test is a guess with a severity. Two kinds of test answer two different questions. Unit tests ask whether the logic is right. Replay asks whether the rule is usable on your data. A rule can pass the first and fail the second, and the reverse is common too.

Unit tests prove the logic, so keep them small

A unit test is an input event and an expected result. The event should alert, or it should not. Write the positive case from a real attack sample. Write the negative case from the nearest benign thing you can find: the admin tool that looks like the attacker tool, the scheduled task with the same parent process.

Panther documents this pattern directly. Each detection carries test events with an expected alert outcome, and panther_analysis_tool test runs them, filterable by path, severity or log type. Panther sets no minimum number of tests, so the discipline is yours (Panther testing docs, read 2026-11-03). Elastic's detection-rules repository keeps Python unit tests for rules, described as a sub-component of its detections-as-code reference (Elastic DaC reference, read 2026-11-03).

Splunk's contentctl takes another route. contentctl test runs detections against Splunk Attack Data inside a real Splunk instance (contentctl testing guide, read 2026-11-03). That is closer to an integration test than a unit test, and it needs a running Splunk. The benefit is that the test exercises the real search engine. The cost is a heavier CI job.

Replay proves the behaviour, so run it before you enable

Replay runs a rule over stored events. You read two numbers: how many alerts, and whether a human would act on them. A rule that fires 4,000 times a day on a single noisy service account passes every unit test and still fails in the SOC.

Panther documents Data Replay (Beta) for testing rules against historical log data before enabling them. The docs describe a preview of the outcome on real data, and no saved report (Panther Data Replay, read 2026-11-03). Google Security Operations documents a test rule action over a time range and Retrohunt on historical data. Test results there are not saved and do not appear in the Rules dashboard (Google SecOps manage rules, read 2026-11-03). Elastic documents manual runs over a chosen past time range, tracked in a Manual runs table (Elastic manage detection rules, read 2026-11-03). Sentinel documents a manual rerun of analytics rules and a results simulation against current data (Sentinel rule integrity, read 2026-11-03).

Two lessons follow. First, replay in most of these products is an interactive action, not a CI step. If you want replay in a pipeline you script it through the API. Second, none of the pages read describe a saved backtest report with comparison against a prior version. You keep your own: record the alert count and a sample of results in the pull request.

Synthetic events fill the gap when you have no history

New log sources have no history. Attack simulation fills that gap by generating the behaviour and checking the rule fires. Splunk lists Attack Range, Attack Data and Atomic Red Team in its contentctl ecosystem table (contentctl README, read 2026-11-03). Google documents an event simulation feature that delivers realistic threat sequences into ingestion, marked pre-GA on the page (Google SecOps event simulation, read 2026-11-03). Detecteam describes generating attack behaviour data from an ATT&CK scenario library, a vendor claim with no format or replay mechanics published on its home page (read 2026-11-03).

Vendor claims about testing are not tests

Read the verbs. "Validated" and "tested and refined" are claims. A command, an input format and an output are documentation.

Hunters says detections are continuously tested and refined by its research team, which is a vendor claim, and documents no customer test tooling beyond a Test logic step on a custom detector over a chosen timeframe (Hunters custom detector docs, read 2026-11-03). Sekoia documents a rule testing view that evaluates a Sigma pattern against historical data, with results shown but not saved as a report (Sekoia rules catalog, read 2026-11-03). Mate Security publishes no testing method for the detections it writes on the pages read, and ranks eleventh of fourteen on the proposed scores. Its rule-health claims are vendor claims.

What a useful test suite looks like

Keep three layers.

The first layer is unit tests, a few per rule, running in seconds on every commit. The second layer is replay, run once when a rule is created or its logic changes, over at least the last 7 days of data you have. Seven days is a suggestion from practice, not a documented figure. It catches weekday and weekend patterns. The third layer is a scheduled attack simulation that checks the rules still fire after a parser or schema change.

Record three numbers in every pull request that touches logic: tests passing, alerts in the replay window, and the share of replay alerts you would triage. If the third number is under half, the rule is not ready.

Checklist

Every rule has one alerting event and one near miss stored in the repo.
Unit tests run on every commit and block the merge.
Replay runs before a new or changed rule is enabled.
The pull request records alert count and a reviewed sample from replay.
New rules start in a disabled or low severity state until replay is read.
A scheduled simulation checks that critical rules still fire.
Failing tests are fixed, not deleted.
You know which of your platforms has no unit test layer and you wrote one.