PRISMA-Local

A local LLM screening assistant for systematic and scoping reviews.

Open the demo → · no install, no sign-up, nothing leaves your browser

Drop in a RIS export (Scopus, IEEE Xplore, Web of Science, EndNote, Zotero, Rayyan, ASReview) or a CSV, and it ranks the corpus in the tab. Decisions already recorded in the file are picked up, and a RIS file comes back out as RIS — reordered, with every field your tool wrote left intact.

---

The problem

A systematic review begins with thousands of database records that two humans must screen by hand against the protocol. Recall must be near-total — a missed eligible study invalidates the review — while precision can be poor and the team still saves weeks. The useful output is therefore a ranking with a stopping rule, not a yes/no classifier. Records often can't leave the institution, so the tool has to run on local hardware.

The approach

Fine-tune small open models to score each title+abstract against a review protocol using a single generated token. The eligibility score is the raw logit(INCLUDE) − logit(EXCLUDE) margin, which turns a classifier into a ranker.

Data: SYNERGY — 73 systematic reviews. 67 reviews train, 6 are held out entirely, so the model is tested on topics it has never seen. Training draws 8,000 records with negatives downsampled to 9 per positive (10.5% positive); the test set is the 6 held-out reviews subsampled to ≤3,000 records each preserving each review's natural prevalence — 13,665 records, 632 includes, 4.6% overall, ranging from 0.53% (Walker_2018) to 23.8% (Rinne_2021).

Three techniques, three distinct jobs

QLoRA	4-bit NF4 fine-tune of Qwen2.5-7B — the only way a 7B trains on 16 GB
LoRA	bf16 fine-tune of Qwen2.5-3B, twice, for two independent screeners
vLLM	Batch scoring, both adapters on one base copy, prefix caching

The pipeline

Stage 1 — Windows, RTX 4080 Super. All training and HF scoring. No admin rights, no WSL. Stage 2 — Colab T4. All vLLM work. Stage 3 — Mac, CPU. Baselines, metrics, figures, README, demo app.

Only ~120 MB of adapters moved between machines. That portability is the practical argument for LoRA as a deployment format — and it's what made the split viable when WSL wasn't an option.

Results

Mean over the 6 held-out reviews. WSS@95% = (TN+FN)/N − 0.05 at 95% recall.

system	AUC	recall@10%	screened@95%	WSS@95%	NNS@95%	distinct scores
Zero-shot 3B	0.635	0.180	0.821	0.129	55.5	138 ⚠
LoRA 3B (A)	0.710	0.289	0.619	0.331	40.3	13,619
LoRA 3B (B)	0.657	0.109	0.704	0.246	48.3	36 ⚠
QLoRA 7B	0.725	0.247	0.675	0.275	45.2	59 ⚠
TF-IDF cross-topic	0.697	0.230	0.656	0.294	43.4	13,659
TF-IDF active learning	0.934	0.701	0.291	0.659	17.0	6,879

⚠ Read this table with the last column. Only LoRA 3B (A) was rescored in fp32. The other three LLM rows are the original bf16 runs, in which 13,665 records collapse onto a few dozen distinct values, and under pessimistic tie-breaking that costs them recall@10% specifically. The table therefore compares one corrected system against three uncorrected ones, and it favours row A. Rescoring the rest is the first item of unfinished work here, not a detail. The next section measures exactly how much it is worth.

Ties are handled pessimistically: within a group of equal scores, positives are ranked last. A model that assigns hundreds of records the same value gets no credit for the includes buried in that group, because a human working down the list cannot reach them first.

Measured

Records scored per screener	13,665
vLLM throughput (T4)	5.88 rec/s
HF throughput (RTX 4080 Super)	12.8 rec/s (3B) · 7.9 rec/s (7B)
Prefix caching	1.40× — 191.5 s → 137.2 s per 800 records
Score resolution after the fp32 fix	45 → 13,619 unique values in 13,665 records

At 5.9 rec/s an 8,000-record review takes 23 minutes on a free Colab T4, and about 10 minutes on the 4080 Super it was trained on.

Two findings worth writing up

A precision bug that AUC could not see

A decent AUC of 0.708 sat next to a recall@10% of 0.229. The cause was not the model. Taking the decision through a bf16 softmax collapsed 13,665 records onto 45 distinct scores; recomputing the same margin in fp32 and storing it raw gave 13,619. Same adapter, same records, same prompts — only the float path differs:

run	distinct scores	AUC	recall@10%	WSS@95%
A — bf16 softmax	45	0.708	0.229	0.312
A — vLLM fp16 margin	5,836	0.710	0.290	0.334
A — HF fp32 margin	13,619	0.710	0.289	0.331
B — bf16 softmax	36	0.657	0.109	0.246
B — vLLM fp16 margin	3,967	0.659	0.167	0.286

AUC moves by 0.002. recall@10% moves by 26% for screener A and 53% for screener B. AUC is a rank statistic that averages tied ranks, so a tie mountain barely dents it; top-k selection has to break those ties, and a reviewer working down the list gets whatever order the tie group happens to sit in. The metric that survived the bug is the one most screening papers lead with.

Score resolution is now logged for every run — unique_scores in results/tables/per_review.csv — and a run whose resolution collapses is treated as broken rather than as a weak model.

Prefix caching gave 1.40×, and the arithmetic nearly explains it

The shared prefix — system message plus protocol header, everything before the record's title — is 98 tokens of a 428-token mean prompt, 23%. Prefix caching cut wall-clock time by 28.4% (191.5 s → 137.2 s for 800 records).

The saving slightly exceeds the prefix's token share, which is the expected direction: only one token is generated, so the run is essentially all prefill, and attention over the cached prefix is quadratic in position, so the tokens skipped are cheaper to skip than their count suggests. The record body still dominates — the header is not where the win is.

Correction to an earlier draft. These figures were previously written as a 300–400 token header inside a ~900-token prompt, capping the saving at 35–40%. Measured against the tokenizer, no prompt in the test set reaches 900 tokens (mean 428, max 690), and the 1024-token limit never binds. The revised numbers above are the measured ones.

Framing

This is cross-topic transfer with zero in-review labels. Published WSS@95% figures come from within-review active learning, which starts from labelled seeds in the same review and retrains as screening proceeds — a substantially easier problem. Both baselines are computed here, so the gap is stated rather than discovered by a reviewer.

That gap is the contribution: how far a local fine-tuned model gets on a brand-new review before anyone has labelled a single record. On this evidence, not far enough to replace active learning — the fine-tuned 3B beats zero-shot decisively (WSS@95% 0.331 vs 0.129) but only draws level with a TF-IDF logistic regression trained on the same reviews (0.294), and active learning is twice as good (0.659). Reporting that plainly is the point of computing both.

Limitations

The models were never given eligibility criteria. The criteria field is empty for all 73 reviews in this build of the corpus, and review_title holds a citation key. The protocol block the model actually saw reads:

### REVIEW PROTOCOL
Topic: Butink 2023

Eligibility criteria:
See review topic.

So the model could learn only what an includable record looks like in general, not whether one matches this protocol. That is a fair description of the ceiling these numbers sit under, and supplying real criteria is the first experiment to run next.

Three of the four LLM rows are still tie-degenerate. zeroshot-3b, qlora-7b and hf-screener_b carry 138, 59 and 36 distinct scores across 13,665 records. Screener B can be fixed today from the stage-2 vLLM run that is already in results/scores/ — the other two need a GPU rerun. Until then the main table understates them, and the gap between LoRA 3B (A) and the rest is partly an artefact of which run got corrected.

Other limits: one epoch, rank-16 adapters, no hyperparameter search. Title and abstract only, no full-text screening. Throughput figures span two GPUs and are not a controlled comparison. Six held-out reviews is a small sample to average over, and the per-review spread is wide — LoRA 3B (A) ranges from AUC 0.39 on Adamo_2021 to 0.89 on Butink_2023, so the headline mean hides a system that fails outright on some topics.

Repository

notebooks/   stage 2 (vLLM on T4) and stage 3 (analysis) as executed
src/         prompt construction, training, both scorers, metrics, exporters
app/         Gradio app that talks to a local vLLM endpoint
adapters/    lora-3b-a, the deployable 3B screener (Git LFS)
results/     tables, figures, per-record scores, timings, environments
docs/        the GitHub Pages site and the in-browser demo

Run the metrics without a GPU:

pip install -r requirements.txt
python -c "import sys; sys.path.insert(0,'src'); import prisma_metrics as pm, pandas as pd; \
print(pm.per_review(pd.read_csv('results/scores/scores_screener_a-fp32.csv')))"

Serve the 3B screener and point the Gradio app at it:

vllm serve Qwen/Qwen2.5-3B-Instruct --enable-lora \
  --lora-modules screener=./adapters/lora-3b-a --max-lora-rank 16
python app/app.py

Reproducing

1. Stage 1 — training. src/train_lora.py and src/score_hf.py on a 16 GB CUDA GPU. The notebook that drove this stage was not kept; the scripts are, and they take the same arguments the notebooks passed.
2. Stage 2 — notebooks/PRISMA_Local_02_vLLM_Cloud.ipynb. vLLM on a free Colab or Kaggle T4, with outputs as executed.
3. Stage 3 — notebooks/PRISMA_Local_03_Mac_Analysis.ipynb. Baselines, metrics and figures. CPU only.

The SYNERGY records are not redistributed here beyond the two held-out reviews the demo ships. src/fetch_data.py rebuilds work/train.jsonl and work/test.jsonl from the public dataset using results/split.json, which pins the exact 67/6 split, the sampling seed, and the per-review evaluation counts.

pip install synergy-dataset
python src/fetch_data.py --check     # what is available, before downloading GBs
python src/fetch_data.py             # download if needed, then rebuild

Pin the release. split.json records which reviews were used but not which SYNERGY release they came from, and the releases are not supersets of one another — against the current default (v3.0), 28 of the 73 reviews in the split are absent. --check reports this before any work happens and the rebuild refuses to proceed, because quietly dropping reviews would change the model and every number in results/tables/. Use --version to pin.

Verified against real data: load_review returns 3,247 records and 158 includes for Burska_2023, and the title / abstract / label_included columns resolve as expected. The full 73-review rebuild has not been run end to end here, because the bulk download had not finished.

Licence

Code MIT, see LICENSE. The SYNERGY dataset is CC-BY-4.0 from the ASReview project; the two review corpora shipped under docs/assets/data/ carry that licence and its attribution.