Research
Open cybersecurity datasets, labelled and licensed for reuse
114,421,340 labelled security events, captured live from five organisations between 26 July and 28 August 2024, published under Apache-2.0 on Hugging Face — with the collection window, the labelling method and the limitations stated on this page.
The corpus at a glance
The block a methods section copies.
| Repository | witfoo/precinct6-cybersecurity-100m |
|---|---|
| Version | 2.0.0, built 11 September 2026 |
| Live capture (signals) | 114,421,340 rows |
| Historical incident leads (incident_signals) | 238,511 rows |
| Capture window | 26 July 2024 11:10:20 UTC – 28 August 2024 00:36:03 UTC |
| Incident lead window | 30 May 2022 – 31 December 2024 |
| Contributing organisations | 5 |
| Incident provenance graphs | 60,575 GraphML files |
| Merged graph | 67,172 nodes, 101,338,165 edges |
| Formats | Parquet, gzipped NDJSON, GraphML, JSONL |
| Download size | 13.9 GB (current revision) |
| Licence | Apache-2.0 |
Where the records came from, and how they were labelled
These are live enterprise records, captured by WitFoo deployments running in production — not synthetic traffic and not a testbed. Five organisations contributed over 33 days. The distribution across them is very uneven: one contributed 99,556,527 rows, 87.0% of the live capture, and the smallest contributed 23. That skew matters more than the number five, which is why it is in this paragraph rather than a footnote.
Each signal carries one of three labels — benign, suspicious or malicious — derived by 261 lead detection rules across 158 security products. In the live capture the shares are 99.46% benign, 0.54% suspicious, and 7,728 rows malicious. The second table, 238,511 historical incident leads from 2022 to 2024, is malicious by construction.
The labels are machine-derived from detection-rule output. They are not analyst-adjudicated ground truth, and a label records what a rule concluded rather than what a human confirmed. That is a meaningful limitation and it is repeated below, because it is the one most likely to be missed.
What happened to version 1
The first public release was withdrawn rather than preserved at a tag. On re-verification, some values had escaped sanitization: device and account names survived inside JSON-escaped Windows event text and in one stream field. Rather than patch it, the corpus was fully regenerated as version 2.0.0.
Tokens are not comparable between the two releases, so any result computed against version 1 cannot be compared with version 2. If you are still holding version 1, discard it — and note that the April 2026 press coverage describes that release, not this one.
A smaller version, built deterministically
If 13.9 GB is more than you want, witfoo/precinct6-cybersecurity is 2,011,674 rows at 8.9 GB, drawn from four organisations. Its construction is published because that is the point of it: every malicious live row, plus every row within five minutes either side of an in-window lead on the same organisation, plus a uniform random fill at p=0.00371 with seed 42. The incidents, provenance graphs and attack reports are identical to the large set.
Licence, citation and reuse
The datasets are Apache-2.0: academic, commercial and government use, with attribution. That covers all five WitFoo datasets. It does not cover our published model — witq-1.0 carries the Llama 3.1 licence, not Apache-2.0, and we would rather say so here than let you find out later.
There is no DOI. Cite the repository URL and the version:
@misc{witfoo_precinct6_cybersecurity_100m,
title = {WitFoo Precinct6 Cybersecurity Dataset (large)},
author = {WitFoo, Inc.},
year = {2026},
howpublished = {\url{https://huggingface.co/datasets/witfoo/precinct6-cybersecurity-100m}},
note = {Version 2.0.0, Apache-2.0}
}What this dataset is not fit for
Read this before you design an experiment around it.
-
One organisation contributed 99,556,527 of the rows — 87.0% of the live capture — and the smallest contributed 23. Anything you conclude about “enterprise traffic” generalises to that one environment first.
-
The live capture is 33 days in mid-2024. Seasonal effects, campaign lifecycles and anything slower than a month are out of reach.
-
7,728 rows out of 114,421,340 are labelled malicious. That imbalance is realistic, and it makes the corpus unsuitable for anything needing a balanced split without resampling you do yourself.
-
Labels are machine-derived from detection-rule output, not analyst-adjudicated. A label records what a rule concluded, not what an analyst confirmed. Study detection behaviour with it; do not treat it as ground truth about attacker activity.
-
Most incident provenance graphs — 57,846 of them — have no matching lead among the live rows, because those incidents pre-date the capture or the triggering artifact was not retained.
-
The Hugging Face dataset viewer is disabled, so no third party has independently verified these counts. Every figure here is self-declared by our build. The signals/metadata.json and graph/metadata.json files ship beside the data for exactly this reason — check us against those rather than against this page.
-
It is not a benchmark. There is no published baseline, no task definition and no leaderboard. It is a corpus.
Research context
The corpus was produced in collaboration with the University of Canterbury (New Zealand) Computer Science and Software Engineering department, for two research projects: an AI cyber-security battle simulator, and intrusion detection based on provenance graphs.
The most useful evidence of that is the rebuild itself. Version 2.0.0 was regenerated on 11 September 2026 to address findings from the provenance-based intrusion detection evaluation: attacks and normal traffic now share a timeline, usernames are wired into the provenance graph, and mis-parsed timestamps are repaired. A dated list of defects that an evaluation surfaced is worth more than a description of the relationship.
Why some coverage says 100 million
The repository was named at its first release — the slug still reads 100m, and the declared size category is 100M to 1B rows — while the current build is 114,421,340 rows. April 2026 coverage reported the corpus at both 100 million and 114 million, and described version 1, whose label distribution differs from this one. Both articles are listed with our other press coverage. We would rather explain the discrepancy here than have you find it.
Checking the numbers yourself
Every figure on this page is self-declared by our build, because the Hugging Face dataset viewer is disabled for these repositories and no third party has recomputed them. We would rather hand you the means to check than ask for the benefit of the doubt.
The per-build metadata files — signals/metadata.json and graph/metadata.json — ship alongside the data, and the dataset card is rendered from them. The generator that produced the corpus is open source and Apache-2.0 at witfoo/dataset-from-precinct6. Check us against those rather than against this page.
The methodological argument behind all of this — why we think evidence has to be checkable rather than asserted — is set out under deterministic versus probabilistic analytics, and the processing paradigm the data comes from is Empathetic Processing.
Questions people actually ask
What is in the WitFoo Precinct6 dataset?
114,421,340 labelled security events captured live from five organisations between 26 July and 28 August 2024, plus a second table of 238,511 historical incident leads from 2022 to 2024, 60,575 per-incident provenance graphs, and a merged graph of 67,172 nodes and 101,338,165 edges. The formats are Parquet, gzipped NDJSON and GraphML, and the current revision is 13.9 GB.
Why does the repository say “100m” when the count is higher?
The repository was named at its first release and the name has not changed; its declared size category is 100M to 1B rows. The current build is 114,421,340 rows in the live capture. Press coverage in April 2026 reported the corpus at both 100 million and 114 million, and described the first release — which has since been withdrawn and replaced, with different label distributions.
What licence applies, and can I use it commercially?
Apache-2.0: academic, commercial and government use, with attribution. That covers all five WitFoo datasets. It does not cover our published model — witq-1.0 carries the Llama 3.1 licence, not Apache-2.0.
Is there a DOI?
No. There is no DOI for this dataset. A BibTeX block is on the dataset card and reproduced on this page; cite the repository URL and the version. We would rather say that plainly than register a placeholder identifier that resolves to nothing.
Has anyone independently verified the row counts?
No. The Hugging Face dataset viewer is disabled for these repositories, so the counts are self-declared by our build rather than computed by a third party. The per-build metadata files — signals/metadata.json and graph/metadata.json — are uploaded alongside the data so a sceptic can check the figures against the build output rather than against our description of it.
What happened to version 1 of the dataset?
It was withdrawn rather than kept at a tag. On re-verification some values had escaped sanitization — device and account names survived inside JSON-escaped Windows event text and in one stream field — so the release was pulled and fully regenerated as v2.0.0. Tokens are not comparable between the two releases, so any result computed against v1 cannot be compared with v2. If you are still holding v1, discard it.
Take it, cite it, argue with it.
Apache-2.0, no registration, no form. If you find something wrong with it, we would like to know — that is the point of publishing the limitations alongside the data.