Security operations
What is a security data pipeline?
It is the layer between the tools that emit security telemetry and whatever stores or analyses it. It collects every record, parses it, enriches it, reduces the volume and routes what remains to a SIEM, a data lake or cheap retention storage.
Put less abstractly: it is the thing that decides which evidence still exists when an investigation goes looking for it.
What it sits between, and what it is not
A SIEM is a destination. It stores security data, runs detections over it and gives analysts somewhere to work. A pipeline is not an alternative to that — it is infrastructure in front of it, shaping what arrives. Most deployments add one to a SIEM that is already in place, usually because the SIEM meters what it receives and what arrives has quietly become a budget decision as well as a security one.
It is also a close relative of general ETL and observability pipelines, and differs from them in one assumption that matters. Those assume the meaning of the data is known in advance, and that a dropped row is an accounting problem. Security telemetry has to stay evidentiary: a record dropped today may be the one an investigation needs next quarter, and a record that cannot be identified should not be quietly discarded on that basis. That is a stricter contract than a general-purpose pipeline is built to keep.
The five things it does to every record
-
Collect
Take delivery from everything that emits security telemetry — firewalls, endpoints, identity providers, cloud control planes, servers, SaaS audit logs — over whatever each one is willing to speak. A pipeline is only ever as good as the sources it can actually accept.
-
Parse
Turn a vendor-shaped message into structured fields with known meanings. This is the step that quietly decides whether everything after it is trustworthy, and it is much harder than it sounds.
-
Enrich
Attach what the record does not carry about itself: which asset this address belonged to at that moment, which human owns that account, what the business does with that service, whether an indicator is already known.
-
Reduce
Send less onward than arrived. This is the step people buy pipelines for, the step every vendor quotes a percentage about, and — as the next section argues — four different operations wearing one name.
-
Route
Deliver to wherever each record needs to be: the SIEM for detection, cheap object storage for retention, a data lake for analysis, more than one of those at once, in a format the destination already understands.
Why parsing is the step that decides the rest
Vendors implement even well-specified formats inconsistently, and a pipeline that misreads a field does not fail loudly — it produces confident, wrong structure that everything downstream then trusts. Syslog severity is the standing example. RFC 5424 defines severity 0 as Emergency: system is unusable and severity 7 as Debug: debug-level messages, so lower numbers are more severe. Any component that assumes a higher number means a worse problem inverts the meaning of every event it touches, and the alerts still look perfectly well-formed.
This is why source coverage is not a checkbox. WitFoo maintains parsing for 158 products across 102 vendors rather than asking customers to write and maintain their own, and the approach it comes from is described under Empathetic Processing.
“Reduction” is four different operations
This is the part worth reading slowly, because it is where percentages come from. Every vendor in this category — us included — publishes a reduction figure. Those figures are not comparable, and the reason is not that anyone is lying. It is that one word is doing the work of four operations that destroy completely different things.
Filtering
- Drops
- Records a rule says you do not want.
- Destroys
- Whatever the rule was wrong about.
You can name the rule. You cannot name the records.
Sampling
- Drops
- Records at a rate, regardless of what is in them.
- Destroys
- Completeness.
You can state the rate. You cannot testify that what you kept is a full account of anything.
Trimming
- Drops
- Fields, from records you keep.
- Destroys
- Detail, inside records that still look present.
Hardest of the four to notice, because the number of events never changes. Only their contents get thinner.
Deduplication
- Drops
- Records identical in meaning to one already sent.
- Destroys
- Nothing novel.
The only one of the four that can answer “why is this specific record missing” after the fact — because the answer is always “an identical one went ahead of it”.
None of this makes the first three illegitimate. Filtering is exactly right when a source is genuinely out of scope — debug logging from a development cluster, a health check firing every second. Sampling is fine for capacity planning, where you want a shape rather than an account. The argument is not that three of the four should be banned. It is that they are not interchangeable, and quoting all four as the same percentage hides the only thing a buyer actually needs to know.
There is a fifth thing sold as reduction that is not reduction at all: storage-format compression. Writing to a columnar format with strong compression genuinely shrinks a storage bill, and removes nothing whatsoever from what arrived. It is a good idea. It simply answers a different question, and belongs in a different number.
Four questions that make any percentage mean something
Ask these of every vendor you evaluate, including us. They are not gotchas — a vendor who has measured carefully will enjoy answering them.
- What was reduced?
- Records, bytes, fields, or the invoice. These are four different quantities and they do not convert into one another. A volume cut and a cost cut can describe the same deployment and differ by tens of points, because metered pricing is tiered and committed rather than linear.
- Measured against what baseline?
- The whole estate, or one deliberately chosen high-volume source? Almost any environment contains one source that can be cut by 90%. That number tells you about the source, not about the pipeline.
- Which of the four operations produced it?
- Filtering, sampling, trimming and deduplication destroy different things. A figure built mostly from filtering is a statement about the confidence of a rule; a figure built from deduplication is a statement about how repetitive the sources were.
- What was “without losing visibility” tested to mean, and when was that bar set?
- This is the one that separates measurement from marketing. A definition of success fixed before the test can be argued with. One written afterwards, to describe the result that was obtained, cannot.
Two rules worth adopting whatever you buy
Neither of these requires a product. Both are cheap to adopt now and expensive to retrofit after an incident.
Never let an inference travel in the same field as an observation
A device reporting what it saw and a model reporting what it concluded are different kinds of statement, and they must not arrive in the same column. Once a scored guess is stored in a field that reads like a measurement, nothing downstream can separate them again — and the distinction is precisely what you will be asked about later. If you cannot tell the two apart in your own data model today, fix that before you buy anything.
Keep clock discipline
Record the time source for every contributing system, and any known offset, as deliberately as you record the events themselves. Sequence is how intent gets established — what happened before what — and it is the first thing challenged when an account is disputed. A pipeline that normalises timestamps without preserving where each one came from has thrown away the ability to defend its own ordering.
Both rules are drawn from WitFoo co-founder Charles Herring's writing on treating security operations as an evidence problem rather than a technology one, at charlesherring.com.
Where it sits, and whether you need one
Placement is a real decision with only three answers: at the edge, near the sources; in the middle, as a single central stage; or both, with collection at the edge and processing centrally. Teams whose actual difficulty is getting data out of many locations in the first place should start at unifying distributed collection, because that is a different problem from volume.
And a pipeline is genuinely not always the answer. If your estate is a handful of sources at a few tens of gigabytes a day, this adds a component to operate without solving anything. If your SIEM is licensed per user, per endpoint or per appliance rather than per gigabyte, then shaping volume does not move your invoice at all and the case has to rest on data quality alone — which how your SIEM is metered determines entirely. If you have one log source and no correlation problem, you do not have a pipeline problem.
The case gets strong when volume is coupled to cost, when the same event arrives from several tools at once, or when dropping sources to control spend has started to feel like a security decision rather than a finance one.
How WitFoo implements this
Briefly, because this page is about the category rather than our product. WitFoo Conductor identifies what each message is before doing anything else, then keeps state on whether that exact combination of object, action and characteristic has already gone downstream. If it has, the record is a repeat and is dropped. That is the fourth operation above, not the first three: no rule decides what is probably unimportant, and no sampling rate is applied.
Across WitFoo deployments the measured range is 90 to 98%, and the real numbers depend entirely on the environment. That is our own research and it is not independently audited — we defined what “lossless” had to mean before we measured, and the bar, the control and the method are set out under deterministic vs probabilistic analytics. If you are weighing this against a metered SIEM bill specifically, reducing SIEM costs works through that case, and our vendor comparisons take on the alternatives one at a time.
What this page does not claim
-
Our figure is our own research and has not been independently audited. Nor, as far as we can establish, has anyone else’s in this category.
-
We are not saying other vendors’ numbers are wrong. We are saying no two of them are comparable without a denominator — and that ours needs one just as much.
-
Deduplication destroying nothing novel is a property of the operation, not a promise about any future attack. A repeat is a repeat; that is arithmetic, not a guarantee.
-
Filtering and sampling are legitimate operations with legitimate uses. This page argues for labelling them accurately, not for banning them.
Questions people actually ask
What is a security data pipeline?
A security data pipeline is the layer between the tools that emit security telemetry and whatever stores or analyses it. It collects records from every source, parses them into structured fields, enriches them with context they do not carry themselves, reduces the volume, and routes what remains to a SIEM, a data lake, cheap retention storage, or several at once. Its practical job is deciding what evidence still exists by the time an investigation goes looking for it.
What is the difference between a security data pipeline and a SIEM?
A SIEM is a destination: it stores security data, runs detections over it and gives analysts somewhere to investigate. A pipeline is infrastructure that sits in front of destinations and shapes what reaches them. They are not alternatives — a pipeline has to deliver somewhere, and most deployments put one in front of a SIEM that is already in place. The reason to add one is usually that the SIEM meters what it receives, so what arrives has become a budget decision as well as a security one.
Is a security data pipeline the same thing as ETL or an observability pipeline?
Architecturally they are close relatives; in what they assume they differ in a way that matters. General ETL and observability pipelines assume the meaning of the data is known in advance and that a dropped row is an accounting problem. Security telemetry has to stay evidentiary: it arrives in formats the sender may implement inconsistently, and a record dropped today may be the one an investigation needs next quarter. A record that cannot be identified should not be silently discarded, which is a stricter contract than most general-purpose pipelines are built to keep.
What is log deduplication, and how is it different from filtering logs?
Filtering drops records that match a rule about what you do not want. Deduplication drops records that are identical in meaning to one already sent onward. The difference is what each destroys. A filter destroys whatever its rule was wrong about, and afterwards you can name the rule but not the records. Deduplication destroys nothing novel, because by definition an identical record already went ahead of it — which also makes it the only one of the common reduction operations that can explain, after the fact, why a specific record is missing. Deduplication tends to remove a lot, for an unglamorous reason: many security products are not stateful in what they emit and repeat the same observation many times an hour.
Why do vendors’ data-reduction figures vary so widely?
Mostly because they are not measuring the same thing. “Reduction” is used for at least four distinct operations — dropping records by rule, dropping records by rate, removing fields from records that are kept, and removing exact repeats — and it is also used for storage-format compression, which reduces nothing that arrived. On top of that, some figures count records, some count bytes, some count the bill, and some describe one high-volume source rather than a whole estate. Two honest vendors can publish very different numbers for identical behaviour. Ask what was reduced, against what baseline, by which operation, and what “without losing visibility” was tested to mean.
Do I need a security data pipeline?
Often not. If your estate is a handful of sources at a few tens of gigabytes a day, a pipeline adds a component to operate without solving a problem you have. If your SIEM is licensed per user, per endpoint or per appliance rather than per gigabyte, then shaping volume does not move your invoice at all, and the case has to be made on data quality alone. If you have one log source and no correlation problem, you do not have a pipeline problem. The case gets strong when volume is coupled to cost, when the same event arrives from several tools at once, or when dropping sources to control spend has already started to feel like a security decision.
Ask us the four questions.
We would rather be evaluated on what we measured and how than on the size of the number. Conductor runs in front of whatever you already own, licensed flat per appliance with unlimited data rates.