cve-recall

Find the CVE behind a piece of text, and measure which search model finds it best.

Tip

New here? Getting started takes you from nothing to your first search, with one setup script. The documentation home lists every page.

Contents: Why | What it does | At a glance | Quick start | Results | Screenshots | Documentation | How it works | Limitations | Credits

Why this exists

When a new vulnerability is in the news, people describe it in their own words. An advisory might say "remote code execution in the file upload page", while the NVD record for the same bug talks about "unrestricted upload of a dangerous file type". The right CVE exists, but it is hard to find when the two texts use different words.

I built cve-recall to study this problem in a practical way. It answers two questions. Given a description, which CVEs match it? And which kind of search model finds the right CVE most often: keyword models, sentence-embedding models, or a fusion of both?

What it does

• Downloads public vulnerability data: NVD, CISA KEV, FIRST EPSS and OSV. It then keeps the CVEs of the date range you choose.
• Turns it into clean tables, with one English description per CVE as the search corpus.
• Builds five known-item test sets from KEV and OSV texts, and logs every question it drops and why.
• Indexes the corpus with four models: TF-IDF and BM25S (keyword) and MiniLM and MPNet (sentence embeddings).
• Evaluates every model and all 11 reciprocal rank fusion (RRF) combinations, with clustered confidence intervals and a fixed family of significance tests.
• Offers search from the command line, with CVSS scores, and in a local Streamlit app, with CVSS, EPSS and KEV shown as context.

Everything runs on your own machine. The repository publishes only aggregate results, never downloaded records.

At a glance

flowchart LR
    subgraph sources ["Public sources"]
        NVD[("NVD")]:::src
        KEV[("CISA KEV")]:::src
        EPSS[("FIRST EPSS")]:::src
        OSV[("OSV")]:::src
    end
    subgraph search ["Search: fetch, prepare, index"]
        T["Clean tables<br/>one description per CVE"]:::data
        I["Four indices<br/>TF-IDF, BM25S,<br/>MiniLM, MPNet"]:::data
    end
    subgraph bench ["Benchmark: bench, evaluate, report"]
        B["Five test sets"]:::bench
        E["Scores, intervals<br/>and 150 tests"]:::bench
        R["Tables and charts"]:::bench
    end
    U(["Search command<br/>and app"]):::use
    NVD & KEV & EPSS & OSV --> T --> I --> U
    T --> B --> E --> R
    I --> E
    classDef src fill:#FFF4D6,stroke:#E69F00,color:#1a1a1a
    classDef data fill:#DCEEFB,stroke:#0072B2,color:#1a1a1a
    classDef bench fill:#F3E5F5,stroke:#CC79A7,color:#1a1a1a
    classDef use fill:#CCEFE5,stroke:#009E73,color:#1a1a1a

Loading

The search path is all you need to search. The benchmark path is optional and produces the numbers below. Architecture shows how the code is organised.

Quick start

You need
A computer	Windows or Linux on a 64-bit x86 machine
Disk	about 10 GB free
Software	uv, which installs Python 3.14 and every package for you
GPU	optional: an NVIDIA GPU whose driver supports CUDA 13.0 builds of PyTorch, for MiniLM and MPNet. TF-IDF and BM25S run without one

1. Install uv, then clone or download this repository.
2. Run the one-time setup in the repository folder: double-click setup.cmd on Windows, or run ./setup.sh on Linux. It asks before it installs or downloads anything. It then installs the packages, creates config.toml, downloads about 3 GB of public data and builds the indices. Without an NVIDIA GPU, it offers the two keyword models only.
3. Start the app: double-click start-app.cmd, or run ./start-app.sh. Your browser opens http://localhost:8501. Stop it with stop-app.cmd or ./stop-app.sh.

Getting started walks through each step, with the setup questions and the time each step took in my test run.

Prefer to type the commands yourself?

Run these in the repository folder. On Windows, use copy instead of cp.

uv sync
cp config.example.toml config.toml
uv run cve-recall fetch
uv run cve-recall prepare
uv run cve-recall index
uv run cve-recall search "remote code execution through a crafted file upload in a web admin panel"
uv run cve-recall app --open

For the benchmark, add uv run cve-recall bench, then evaluate and report. uv run pytest -q runs the tests on small synthetic data. Two of them check the GPU path: they need CUDA and download MiniLM once. uv run pytest -q -m "not gpu" leaves those two out.

Results

Data up to 2026-10-11: NVD feeds downloaded 2026-10-11T01:23:34Z (385,312 descriptions), KEV catalog 2026.10.08 (1,739 entries), EPSS scores of 2026-10-10, OSV all.zip generation 1791679722927067 (downloaded 2026-10-11T01:28:36Z). Evaluated 2026-10-11T13:55:49Z at commit fb4430b; seed 20261011.

test set	system	Hits@10 [95% CI]	Hits@10 macro-average over sources	MRR@100	MRR@100 macro-average over sources	n	fusion vs this model (Hits@10)
kev_name	tfidf	0.382 [0.346, 0.419]	-	0.237	-	969	fusion better
kev_name	bm25s	0.541 [0.501, 0.581]	-	0.351	-	969	no clear difference
kev_name	minilm	0.513 [0.476, 0.550]	-	0.334	-	969	fusion better
kev_name	mpnet	0.524 [0.490, 0.559]	-	0.334	-	969	fusion better
kev_name	rrf(tfidf+bm25s+minilm+mpnet)	0.580 [0.543, 0.618]	-	0.365	-	969
kev_summary	tfidf	0.683 [0.641, 0.728]	-	0.572	-	1,022	fusion better
kev_summary	bm25s	0.770 [0.731, 0.811]	-	0.661	-	1,022	no clear difference
kev_summary	minilm	0.708 [0.662, 0.754]	-	0.556	-	1,022	fusion better
kev_summary	mpnet	0.704 [0.653, 0.754]	-	0.558	-	1,022	fusion better
kev_summary	rrf(tfidf+bm25s+minilm+mpnet)	0.786 [0.746, 0.827]	-	0.640	-	1,022
osv_t	tfidf	0.707 [0.691, 0.724]	0.733 [0.715, 0.751]	0.544	0.571	25,388	fusion better
osv_t	bm25s	0.825 [0.812, 0.837]	0.841 [0.825, 0.855]	0.694	0.722	25,388	fusion better
osv_t	minilm	0.821 [0.810, 0.832]	0.814 [0.797, 0.829]	0.681	0.691	25,388	fusion better
osv_t	mpnet	0.835 [0.824, 0.846]	0.832 [0.817, 0.846]	0.688	0.695	25,388	fusion better
osv_t	rrf(tfidf+bm25s+minilm+mpnet)	0.877 [0.867, 0.886]	0.882 [0.868, 0.894]	0.739	0.760	25,388

osv_t Hits@10 and MRR@100 are pooled, uncapped: every question counts once, so the largest sources weigh most. Source shares: GHSA-Maven 20.7%, GHSA-npm 20.2%, GHSA-PyPI 18.5%, GHSA-Packagist 14.8%, GO 8.7%, GHSA-Go 7.7%, GHSA-RubyGems 3.1%, GHSA-crates.io 2.4%, GHSA-NuGet 2.1%, RUSTSEC 1.1%, GHSA-Hex 0.4%, other 0.2%, GHSA-GitHub Actions 0.2%. A macro-average is the mean of the per-source values.

How to read the table: Hits@10 is the share of questions whose answer CVE is in a model's first ten results, and the brackets hold a 95% clustered-bootstrap interval. The last column compares the fusion of all four models with each model. "Fusion better" means the test is significant after the Holm correction over all 150 tests and the interval stays at least 2 points above zero. "Significant, below the 2-point margin" means the test is significant but the interval comes closer to zero than 2 points. "No clear difference" means the test is not significant, even when the point estimate is higher.

The commit in the stamp is the code version that produced these numbers; the git tag eval-run marks the same commit.

What this run shows (Hits@10 unless stated, snapshot above):

• The fusion of all four models is better than TF-IDF, MiniLM and MPNet on all three sets in the table.
• Against BM25S, the fusion is better on the two OSV sets: on osv_t (n = 25,388) by 5.2 points (95% CI 4.7 to 5.7), and on osv_d (n = 6,060, not in the table) by 11.1 points (95% CI 10.1 to 12.1). On the two KEV sets in the table, kev_name (n = 969) and kev_summary (n = 1,022), it does not meet the rule against BM25S.
• At Hits@10, BM25S, MiniLM and MPNet never pass the "better" rule against each other on these three sets. At other cut-offs BM25S does. At Hits@1 on kev_summary it is better than MiniLM by 13.0 points (95% CI 8.8 to 17.6) and than MPNet by 12.2 points (95% CI 8.1 to 16.5). At Hits@100 on kev_name it is better than MiniLM by 5.7 points (95% CI 2.7 to 8.7).
• Product names matter, most of all for the dense models. On the 838 CVEs shared by kev_summary and kev_summary_noname, removing vendor and product names lowers Hits@10 by these amounts (descriptive only): MiniLM 0.456 [0.395, 0.519], MPNet 0.462 [0.401, 0.526], BM25S 0.200 [0.161, 0.243].
• Fusion does not always help. On kev_summary_noname (n = 897), BM25S has a higher Hits@1 than the four-model fusion, by 11.5 points (95% CI 7.4 to 15.6), and this meets the "better" rule. TF-IDF does too, by 8.7 points (95% CI 4.0 to 13.4).
• Of the 150 pre-declared tests, 102 are Holm-significant and 91 meet the full "better" rule.

The results page has all five test sets, all 15 systems, the per-source OSV tables and the difficulty and exclusion tables. It also has the agreement and overlap analyses, the limitations, and 16 charts. Each chart is in docs/images as a PNG and as an SVG. The CSV files and the run manifest are in docs/results.

Closest prior work. HackerSignal (Ampel and Samtani, arXiv:2605.03158) is the closest work I know of. Its Task 1 also searches NVD descriptions with queries from KEV and other public sources, using BM25, MiniLM, MPNet, BGE, E5 and a hybrid, and reports point estimates. cve-recall adds three KEV sets with ceilings and exclusion logs, single-answer OSV sets and an automated copy filter. It also adds clustered confidence intervals, a test family fixed before the first run and fusion diagnostics, and it runs on any date range without training. benchmarks.md lists the differences.

Screenshots

The app in light and dark mode. It follows your system's setting, and the menu at the top right switches it.

Search. One fused list, plus one tab per model.	CVE record. CVSS, EPSS, KEV and OSV packages, dark mode.
Benchmark. Every system with its 95% interval.	Tests. Each pre-declared test with a plain verdict, dark mode.

Using the app explains every page.

Documentation

Page	What it covers
Docs home	a map of all pages, and reading paths for different readers
Getting started	install, the one-step setup, the first search
Using the app	the search command, every page of the app, starting and stopping
Architecture	how the code is organised, with diagrams of the data and the classes
Pipeline	the six steps, what each reads and writes, and the exit codes
Models	the four models and RRF fusion in plain words, and which to use
Benchmark guide	the benchmark from one question to a verdict, with worked examples
Test sets in detail	every rule that builds the five test sets
Evaluation	every metric, interval, test and chart
Results	all numbers of the published run
Configuration	the settings in config.toml
Data sources	the four sources, how each download is checked, the licences
Adding a model	how to add another model
Troubleshooting	common error messages and their fixes
Glossary	every term in one place

How it works

1. fetch downloads the sources and checks them: each NVD feed against the hash in its .meta file, the OSV zip against the MD5 the server reports, and the KEV and EPSS files (which publish no hash) by their format. A file that fails its check never replaces the copy from the last fetch. Output: data/raw/.
2. prepare builds the corpus (one English description per CVE, rejected CVEs left out) and the detail tables. Output: data/prepared/.
3. bench builds the five test sets and logs every exclusion. Output: data/bench/.
4. index builds one index per model. Output: data/indices/.
5. evaluate scores every question once per model, builds the fusions and computes the metrics, intervals and tests. Output: data/runs/ and results/.
6. report writes the tables, the charts and results.md to results/report/. It never recomputes a number. With docs_dir set, it also copies them into the docs folder.

fetch, prepare, bench and evaluate each write a manifest with what they read and wrote, and fingerprints of the settings they used. Later steps refuse inputs built from another corpus. Details: pipeline.md, benchmarks.md and evaluation.md.

All settings live in config.toml. The main keys are the date range ([dates]), the folders ([paths]), the enabled models ([models]), the fusion ([fusion]) and the sources ([sources]). A past end date only filters: CVEs by publication date, and the KEV test questions by the date CISA added them. NVD, KEV and OSV publish only their current state, so descriptions and scores are still today's; only EPSS is downloaded for that day. configuration.md explains the main keys, and config.example.toml lists them all.

To add another sentence-transformers model, add a [models.<name>] section with a pinned revision and run index, evaluate and report. Search, the app and the evaluation share one search service, so the new model appears in all three. See adding-a-model.md.

What the scores mean

• Model scores (TF-IDF, BM25S, cosine similarity) only order the results of one model. They are not comparable between models.
• The RRF score shows how strongly the models agree on a CVE's rank. It is not a probability.
• CVSS is a 0 to 10 severity score, from NIST, from the CVE's publisher (the CNA) or from CISA's enrichment program (CISA-ADP).
• EPSS is the estimated probability that a CVE is exploited in the next 30 days. "n/a" means no score, not 0.
• KEV means CISA knows the CVE is exploited. A CVE outside KEV may still be exploited.

Limitations

• The questions are known-item proxies written by CISA and advisory databases, not real analyst queries.
• Only English NVD descriptions are searched.
• A past end date still uses today's descriptions, CVSS scores and KEV texts. Only EPSS has dated files.
• KEV is vendor-heavy and OSV covers open-source packages, so neither is a random sample of CVEs.
• MiniLM and MPNet were trained by others on public text that may include some of these descriptions.
• CVSS, EPSS and KEV are shown as context. cve-recall does not tell you what to patch first, and a CVE missing from the results may still affect you.
• The numbers change with every download.

The results page gives more detail on the benchmark limits, with counts where they exist.

Future work

• Security-domain text encoders next to the general ones.
• A cross-encoder that reranks the top results.
• Relevance judged by people, with more than one right answer per question.
• Test sets in other languages.
• The same benchmark repeated over several snapshots, to see how results drift.

Data credits and licence

• NVD. This product uses data from the NVD but is not endorsed or certified by the NVD.
• CVE. CVE records are published by the CVE Program. Its terms of use allow reproduction when the copyright notice and the licence are kept.
• CISA KEV. The KEV catalog is published by CISA under the CC0 1.0 licence, with no use of the CISA logo and no implied endorsement.
• FIRST EPSS. EPSS scores are published freely by FIRST, which asks for attribution: FIRST, Exploit Prediction Scoring System (EPSS), https://www.first.org/epss.
• OSV. Each OSV advisory keeps the licence of the database it comes from, for example CC-BY 4.0 for GitHub advisories and CC0 for RustSec.
• Models and libraries. all-MiniLM-L6-v2, all-mpnet-base-v2 and sentence-transformers are Apache 2.0; bm25s is MIT.

Never load index folders from someone else: build your own with cve-recall index. The app listens on localhost only; to use it on a remote GPU machine, forward the port with ssh -L 8501:localhost:8501 you@host. See search-app.md.

The code is released under the MIT licence (see LICENSE).

---

Back to top | Docs home | Getting started