Security analytics
Deterministic vs probabilistic
Every security pipeline reduces volume. The question is what it drops. A probabilistic pipeline drops what scores low. A deterministic one drops only what is redundant. That distinction decides what you have left on the day an investigation needs it.
Why anything gets dropped at all
No SOC can read everything its tools emit. That has been true since the early 2000s, and the industry has spent twenty years trying to distil the firehose: pivot alerts to hosts and rank the top ones, group hosts and users by behaviour, wire playbooks together to enrich records. Each distilled a little. None solved the underlying problem.
Then storage and processing costs forced a blunter answer: throw away the signal that is probably not useful. An entire category of pipeline tooling grew up around that promise. The triage was always probabilistic — a good guess — and the guess was made months before anyone knew which incident would need the evidence.
That is the cost people underestimate. A probabilistic drop is irreversible and silent. The record is not in a cheaper tier or a slower index; it does not exist. When a responder reaches for it during an incident, there is nothing to reach for, and no way to find out what was there.
The difference, in practice
Both approaches shrink the haystack. Only one of them can tell you afterwards why a particular record is missing.
What it does with a signal it has not seen before
Probabilistic
Scores how likely it is to matter, and drops what scores low.
Deterministic
Fingerprints it, identifies what kind of event it is, and asks whether it changes anything already known.
Why a record was dropped
Probabilistic
Because a model or a rule judged it unlikely to be useful.
Deterministic
Because the exact combination of object, action and characteristic had already gone downstream. Nothing new was in it.
What you can say in an audit
Probabilistic
That the tool was configured to a threshold, and the threshold held.
Deterministic
Which records were dropped, and that each was a repeat of one already sent.
What happens to an investigation
Probabilistic
It proceeds on what survived triage. What was discarded is not available later, when it turns out to matter.
Deterministic
It proceeds on all of the evidence, because none of the evidence was novel and lost.
What AI is asked to do
Probabilistic
Fill the gaps left by sampling, and often deliver the verdict.
Deterministic
Read a complete graph and propose where to look. The verdict rests on the record.
How we tested for lossless reduction
The bar was set before the testing started, which is the only way a result like this means anything.
- The control
- The strongest SIEM and XDR tools we could put our hands on, fed unsampled data from known attacks.
- The bar for “lossless”
- The pipeline had to reproduce 100% of those true positives with 100% of the supporting evidence an investigator would need to close the case.
- The bar for efficiency
- The processing required had to drop by at least 90%.
- How many runs
- More than 4,000 experiments.
The first round with enterprise and MSSP partners came in north of 95% reduction, with no needles thrown away. The figure WitFoo publishes today, measured across deployments, is a 90 to 98% range — and the caveat travels with it: the real numbers depend entirely on the environment. Your number depends on how repetitive your particular tools are, which is not something we can know in advance.
“So the question reduced to something almost childishly simple to state: how do you shrink a haystack by 90% without losing a single needle?”
The mechanism is duller than the number suggests
There is no model doing the reduction. There are two deterministic steps, and the second one only works because of the first.
-
Adaptive Parsing establishes what a message is
Every signal is fingerprinted, with non-overlapping fingerprints, and each fingerprint maps to a semantic frame describing how to parse the message, why it fired, where it came from and what business impact it points at. The pipeline does not guess what a message is — it can prove what it is. Nothing downstream is trustworthy without this step.
-
ProtoGraph decides whether it adds anything
A signal does one job: it increments what is known about an object — a user, a host, a file, a service — and about how those objects relate. ProtoGraph keeps state on whether that exact combination of object, action and characteristic has already gone downstream. If it has, the signal changes nothing and is dropped. If it has not, it goes.
Why that removes so much: many security products are not stateful in what they emit, so they repeat the same combination thousands of times an hour. Not one of those repeats improves what you know. The reduction is large because the redundancy is large — not because anything clever decided what mattered.
Where AI belongs in this
Not opposed to it — we build with it. But there is a difference between using a guess to start an investigation and using a guess as the verdict.
The current pattern is to sample the evidence with heuristics and then point an agent at the gaps, asking it to infer what happened. That compounds two problems rather than solving either: the model is reasoning about data that was discarded on a guess, and its confidence is unrelated to whether the missing piece mattered. A hallucination and a correct inference read exactly the same.
Over a deterministic graph the job changes. The model is not filling gaps, because the evidence is intact; it is reading a complete record and proposing where to look. Its place is generating the hypotheses that warm up an investigation, not signing the verdict at the end of one. WitFoo's MCP server exposes that graph to an assistant — including a local model in an air-gapped network — and the figures it reports trace back to the artifacts they came from.
What this does not claim
A page arguing for defensible evidence should be careful about its own.
-
It is not a guarantee that no real detection is ever dropped. The 4,000 experiments measured a known-attack corpus, and reproducing 100% of those true positives is a result against that corpus — not a promise about every future attack. No pipeline can honestly make the second claim, and we are not going to imply it by phrasing the first one loosely.
-
The reduction range is not a spec. 90 to 98% is what has been measured across deployments; the real numbers depend entirely on the environment. If your tools are less repetitive than average, you will see less.
-
The research is ours, and it has not been independently audited. We publish the control, the success criteria and the count instead of asking you to take the figure on trust — but it is vendor research with its working shown, and you should read it as such.
-
Deterministic does not mean detection is solved. It means the evidence under a detection is complete and the drops are explainable. What you conclude from that evidence is still analysis.
Questions people actually ask
What is deterministic security analytics?
A pipeline is deterministic when every decision it makes about a record can be traced to a property of that record rather than to a score. WitFoo fingerprints each signal, identifies what kind of event it is, and then asks a single question before forwarding it: does this change anything we already know about the users, hosts, files and services involved? If the answer is no — because that exact combination has already gone downstream — it is a duplicate, and dropping it removes nothing an investigator could have used. That is a different operation from deciding a record is probably not important.
What is the difference between deterministic and probabilistic analytics?
A probabilistic pipeline estimates how likely a record is to matter and discards what falls below a threshold. A deterministic pipeline establishes what a record is and discards only what is redundant. Both reduce volume. Only one of them can tell you afterwards why a specific record was not kept. As Charles Herring puts it: a probabilistic heuristic "can make a compelling, often correct guess, but it cannot back the claim up."
How much data does deterministic deduplication actually remove?
Across WitFoo deployments the measured range is 90 to 98%, and the honest caveat from the research is that the real numbers depend entirely on the environment. The reduction comes from a specific and unglamorous cause: many security products are not stateful in what they emit, so they repeat the same combination of object, action and characteristic thousands of times an hour. None of those repeats adds anything. Your own number depends on how chatty your particular tools are.
Does WitFoo guarantee it never drops a real detection?
No, and we will not phrase it that way. What we can say is what was measured: in more than 4,000 controlled experiments, with unsampled data from known attacks through leading SIEM and XDR tools as the control, the pipeline reproduced 100% of the true positives with 100% of the supporting evidence. That is a result against a known corpus, not a guarantee about every future attack, and no pipeline can honestly make the second claim. The structural argument is stronger than a guarantee anyway: a record is only dropped when an identical one has already been forwarded, so there is no novel evidence for the drop to destroy.
Is this research independently audited?
No. The experiments are ours, run by WitFoo, and we publish the methodology rather than asking you to take the number on trust: the control, the definition of lossless set before testing, and the bar for efficiency are all stated. Treat it as vendor research with its working shown, which is what it is. The underlying mechanism is also inspectable in your own environment — deduplication rates are reported per deployment.
Where does AI fit if the pipeline is deterministic?
AI generates the hypotheses that start an investigation; it does not sign the verdict at the end of one. That ordering is the entire point. A model reasoning over a complete, deterministic evidence graph is working from the record; a model reasoning over sampled data is guessing at what was thrown away, and its confidence will not tell you which it is doing. WitFoo Analytics puts an assistant on top of the graph, including against a local model in an air-gapped network, and the figures it reports trace back to the artifacts they came from.
Reduce the haystack without guessing.
Conductor runs the deterministic pipeline in front of whatever you already own — SIEM, XDR or data lake — and forwards what changes the picture. Flat per appliance, unlimited data rates.