OpenPaper

OpenPaper is a private, searchable paper library for selected AI, security, and computer architecture conferences and journals. It maintains a source-neutral canonical paper record: DBLP supplies finalized bibliographic metadata, while configured official prepublication pages can create papers before DBLP indexes them. Semantic Scholar, OpenAlex, official proceedings, publisher pages, and arXiv enrich abstracts and paper-level affiliations.

Run the local search interface

The interface is a local, single-user research tool. It provides separate keyword and author search modes plus an exact year and one or more publication source filters. Keyword search covers titles and abstracts. Author search treats the input as one name: a base name such as Tong Sun includes DBLP's numbered identities, while Tong Sun 0006 keeps that disambiguation suffix in the match. There is no account or login system.

The home page shows a different in-scope paper with an abstract on every refresh. The header reports the exact 已有摘要 / 总论文数 count. Open /settings to select the active year range and conferences or journals, configure automatic refresh intervals, and queue an immediate update. On the first visit of each calendar day, a compact reminder reports whether the latest metadata, missing-abstract, and author-affiliation jobs produced new data. The reminder uses the configured timezone and is remembered in that browser's local storage.

Install the Python and frontend dependencies once:

python -m pip install -r requirements.txt
npm --prefix frontend install

Build the frontend and create the SQLite FTS5 index:

npm --prefix frontend run build
python -m openpaper build-search-index --rebuild

Start the local application:

python -m openpaper serve

Then open http://127.0.0.1:8000. The server only binds to 127.0.0.1 by default, so it is not exposed to other computers.

For frontend development, keep the API running in one terminal and start the Vite development server in another:

npm --prefix frontend run dev

Open http://127.0.0.1:5173 for live frontend updates. The production server serves the static build from frontend/dist. Once the search index exists, database insert, update, and delete triggers keep it synchronized; use --rebuild after replacing the database or when a full consistency refresh is needed.

The main API endpoints are:

GET /api/facets: available years, sources, and exact coverage counts
GET /api/search: mode=keyword|author, query q, year, repeated venue, and pagination
GET /api/papers/random: one random in-scope paper with an abstract
GET /api/papers/{dblp_key}: one in-scope paper
GET and PUT /api/settings: read or update the active data scope and scheduler settings
GET /api/jobs: recent and queued background jobs
GET /api/daily-update: today's concise outcome for the three automatic update jobs
POST /api/jobs/{job_type}: queue metadata_sync, abstract_retry, or affiliation_sync

Publication source configuration

The source of truth is config/venues.json. It currently contains 12 venues:

AI/ML core: NeurIPS, ICML, ICLR
AI extension: AAAI
Security: S&P, CCS, USENIX Security, NDSS
Architecture: ISCA, MICRO, HPCA, ASPLOS

dblp_slug is explicit because some acronyms do not map mechanically to DBLP. For example, NeurIPS uses nips, S&P uses sp, and USENIX Security uses uss. Existing schema-version-1 entries default to a conference. A journal entry uses schema version 2 and adds:

{
  "acronym": "TDSC",
  "name": "IEEE Transactions on Dependable and Secure Computing",
  "area": "security",
  "tier": "core",
  "publication_type": "journal",
  "dblp_stream_type": "journals",
  "dblp_slug": "tdsc"
}

The metadata and author queries automatically use DBLP Article records and publication year for journals, and Inproceedings records and event year for conferences. After adding a source, restart the web and worker services; it will appear in Settings and the metadata worker will backfill every missing configured year.

An official source that publishes accepted or Online First papers before DBLP can be attached to the same venue:

{
  "prepublication_sources": [
    {
      "source_id": "usenix_prepublication",
      "adapter": "usenix_accepted_papers",
      "url_templates": [
        "https://www.usenix.org/conference/usenixsecurity{year_short}/cycle1-accepted-papers",
        "https://www.usenix.org/conference/usenixsecurity{year_short}/cycle2-accepted-papers"
      ],
      "start_year": 2026
    }
  ]
}

The ingestion framework is shared by conferences and journals, while each publisher page format is handled by an adapter that returns the same common record fields (source_id, title, year, URL, optional DOI, abstract, ordered authors, and affiliations). The default configuration enables official prepublication collection for all 12 sources:

NeurIPS and ICLR official proceedings indexes;
ICML through the matching PMLR volume;
AAAI OJS issues whose names exactly match AAAI-YY Technical Tracks N;
IEEE S&P, USENIX Security, and NDSS accepted-paper pages;
CCS's official accepted-paper JSON;
ISCA, MICRO, and ASPLOS main-conference program paper blocks;
HPCA's Main Conference track.

The parsers deliberately select paper containers rather than arbitrary page headings. Workshop/tutorial pages, keynotes, breaks, student abstracts, and AAAI's IAAI/EAAI issues are therefore not imported. A publisher that has not posted its accepted-paper list yet is reported as a source error and retried on the next scheduled metadata run; it does not stop the other sources. These creator sources start at 2026 in the default configuration because the 2020-2025 history is already finalized through DBLP. Lower start_year only when intentionally backfilling an older official list.

Adding a conference or journal before DBLP indexes it

Adding only acronym, dblp_slug, and the DBLP stream is sufficient when DBLP already contains the publication. It is not sufficient when the publisher announces accepted, Online First, or Early Access papers before DBLP. In that case the coding agent must update both the venue configuration and the official-source collection path.

Use this decision process:

Confirm the DBLP stream, record type, and year semantics. Conferences use Inproceedings and event year; journals use Article and publication year.
Find a public, official source that exposes finalized paper metadata. Good sources include an accepted-paper list, proceedings index, publisher API, OAI-PMH endpoint, or Online First issue. Do not collect confidential submission-system pages or infer acceptance from preprints.
If DBLP is already current, add only the normal venue entry. If the official source can lead DBLP, add prepublication_sources.
Reuse an adapter only when the new source has the same data format and paper-container semantics. A visually similar site with different HTML requires a new parser and tests.
If no public official source exists yet, leave prepublication collection unconfigured. DBLP polling will continue, and an adapter can be added when the venue or publisher exposes a suitable source.

The configured adapter names are validated at startup. Their current purposes are:

Adapter	Expected official source
`proceedings_index`	NeurIPS/ICLR-style index with paper links and ordered authors
`pmlr_icml`	ICML volume discovered from the PMLR volume index
`aaai_ojs`	AAAI OJS archive and Technical Tracks issues
`ieee_sp_accepted_papers`	IEEE S&P accepted-paper accordion
`ccs_accepted_papers`	CCS accepted-paper JSON
`usenix_accepted_papers`	USENIX accepted-paper pages
`ndss_accepted_papers`	NDSS accepted-paper cards
`architecture_program`	ISCA/MICRO/ASPLOS-style main-program paper blocks
`researchr_main_track`	Researchr rows restricted to the main conference track

url_templates may use {year}, {year_short}, and {micro_number}. A new edition-number convention must be implemented in PrepublicationSource.render_urls; it must not be approximated in the JSON configuration.

Every adapter must return records with this contract:

{
  "source_id": "stable publisher-side identifier",
  "title": "Final paper title",
  "year": 2026,
  "url": "https://official.example/paper/123",
  "doi": "10.xxxx/example",
  "abstract": "Optional official abstract",
  "pdf_url": "https://official.example/paper/123.pdf",
  "track": "Optional main-conference track",
  "authors": [
    {
      "name": "Alice Example",
      "affiliations": ["Example University"]
    }
  ]
}

Only source_id, title, year, and url are required. The remaining fields are nullable, but an adapter must preserve ordered authors and per-author affiliations whenever the official source provides them. Never invent an abstract or affiliation. A list-only record is valid: it is stored with a pending abstract and is retried by the normal recovery job.

source_id is an identity, not a display value. Prefer a publisher article ID, DOI, OAI identifier, program UUID, or stable official URL. Include the year when a site can reuse slugs between editions. If no official identifier exists, use the deterministic normalized-title helper; never use a Python process hash or a list position that can change when the page is reordered.

When a new page format needs an adapter, the coding agent must make all of these changes in the same pull request:

Add the adapter name to SUPPORTED_PREPUBLICATION_ADAPTERS in openpaper/venues.py.
Add a fetcher and a pure parser in openpaper/publications.py. The fetcher must turn temporary HTTP or parsing failures into source errors so one bad publisher cannot stop the other venues.
Register the adapter in _sync_one_source.
Add the venue or journal and its prepublication_sources entry to config/venues.json.
Add realistic parser fixtures to tests/test_publications.py and configuration validation to tests/test_venues.py.
Update this adapter table and document any non-obvious scope rule.

The parser must select publication-bearing containers, not every heading on the page. For conferences, explicitly exclude workshops, tutorials, keynotes, panels, breaks, calls for papers, artifact announcements, student abstracts, and other non-proceedings material. For journals, exclude editorials, covers, tables of contents, corrections, and news unless the user deliberately places those document types in scope. Placeholder or embargoed titles must remain skipped until the official title appears.

Before considering the source complete, verify all of the following:

a known paper is parsed with the correct title, year, URL, author order, and author-to-affiliation mapping;
obvious non-paper page elements are not parsed;
the parsed count is plausible against the official page;
two identical runs do not insert duplicates;
an existing provisional row is reconciled when the matching DBLP DOI or unique venue/year/title record arrives;
a 404, an unpublished future page, or a temporary provider error is reported without failing the whole metadata job.

Run the required checks:

python -m openpaper generate-dblp-links
python -m pytest -q
cd frontend
npm run build

Exercise the new source against a disposable database first. After review, run it against the persistent database. On PowerShell:

Copy-Item `
  data/papers/openpaper.sqlite3 `
  "$env:TEMP/openpaper-adapter-test.sqlite3"

python -m openpaper sync-prepublications `
  --venue "NEW VENUE" `
  --year 2026 `
  --database "$env:TEMP/openpaper-adapter-test.sqlite3"

Inspect the temporary database and generated summary before touching runtime data. The reviewed source can then be run against the persistent database:

python -m openpaper sync-prepublications `
  --venue "NEW VENUE" `
  --year 2026 `
  --database data/papers/openpaper.sqlite3

Rebuild Docker after changing code or bundled configuration:

docker compose up -d --build

The worker will then include the new source in scheduled metadata_sync. New provisional papers automatically enter search and the abstract-retry flow; later DBLP records reconcile onto the same stable paper identity.

Copy-paste task for a coding agent

Add <VENUE OR JOURNAL> to OpenPaper. First verify its DBLP stream, publication
type, year semantics, and main-publication scope. Then find a public official
accepted/Online First source that may lead DBLP. Reuse an existing
prepublication adapter only if the source format and paper-container semantics
really match; otherwise implement and register a new adapter. Return stable
source IDs, final titles, correct years, official URLs, and all available DOI,
abstract, PDF, ordered-author, and per-author-affiliation metadata. Exclude
workshops and other non-paper content. Add configuration, parser fixtures,
idempotence/reconciliation tests, README documentation, and run the full
Python tests plus the frontend production build. Do not modify the persistent
dataset while validating; use a disposable SQLite database first.

Generate the canonical DBLP stream and browse links:

python -m openpaper generate-dblp-links

The command validates the configuration and atomically writes data/dblp_venues.json.

Custom paths are supported:

python -m openpaper generate-dblp-links `
  --config config/venues.json `
  --output data/dblp_venues.json

Prepublication records and canonical identity

Run the configured official-source collectors directly:

python -m openpaper sync-prepublications `
  --venue "NDSS" `
  --year 2026

Official records without a finalized DBLP entry are stored with publication_status=provisional and shown with a 预收录 badge. Every paper has a stable paper_id; paper_external_ids stores its DBLP, publisher, or official-site identifiers and URLs. When DBLP later supplies the same paper, OpenPaper matches DOI first and then a unique normalized title within the same venue and year. It upgrades the existing row to published, keeps the stable paper_id, preserves collected abstracts and affiliations, and replaces the temporary storage key with the real DBLP key. Ambiguous matches are retained in metadata_merge_reviews for manual review rather than merged silently. Embargo placeholder titles are skipped and retried on the next metadata run.

The scheduled metadata_sync job runs both DBLP and every configured prepublication adapter. Consequently, official accepted-paper, Online First, and Early Access records use the same daily update notification and automatic abstract-retry flow. A list page may initially expose only titles, authors, affiliations, and links; those rows remain pending for the normal abstract retry job until the paper page, DOI provider, or publisher exposes an abstract.

Build the title dataset

Fetch all currently DBLP-indexed conference papers from 2020 through the current year:

python -m openpaper fetch-dblp-papers

The command writes:

data/papers/by_year/<year>.jsonl: restart-friendly yearly partitions
data/papers/dblp_papers_2020_<year>.jsonl: combined dataset
data/papers/openpaper.sqlite3: queryable dataset with nullable abstract columns reserved for the second collection phase
data/papers/dblp_papers_2020_<year>.summary.json: counts and metadata completeness

Each paper contains its title, DBLP key, venue and research area, publication and event years, DOI, DBLP/TOC/document URLs, and empty abstract fields. The requested range and yearly partitions use DBLP's conference event year; the potentially different publication year remains available as metadata. Re-running the DBLP import updates bibliographic metadata in SQLite while preserving any abstracts already added there.

DBLP venue streams sometimes include colocated workshops or associated proceedings. The main_toc_patterns entries in the venue configuration mark canonical main-conference records with is_in_scope=1; associated records are kept for audit with scope_reason=associated_proceedings but are excluded from the default dataset. Title-based classification also retains but excludes clearly non-research proceedings content: front matter and chair messages, workshop overview notices, keynotes, tutorials, doctoral-symposium overviews, and explicitly labelled abstract reprints. Posters, demos, and student abstracts remain included because they contain original research content. Preview and then apply classification to an existing database after changing these rules:

python -m openpaper classify-scope --dry-run
python -m openpaper classify-scope

The command writes data/papers/scope_classification.summary.json with aggregate and per-source counts. Workshop-overview detection uses structural event-title patterns across every configured conference rather than a venue-specific allow or deny list; a research title that merely discusses a workshop remains in scope.

Choose a fixed range when reproducibility matters:

python -m openpaper fetch-dblp-papers `
  --from-year 2020 `
  --to-year 2026 `
  --output-dir data/papers

DBLP is updated continuously. The current year's partition is therefore a snapshot and can be incomplete, especially for conferences whose proceedings have not yet been published.

Fetch ordered authors, DBLP person identifiers, and available ORCIDs:

python -m openpaper fetch-dblp-authors `
  --from-year 2020 `
  --to-year 2026

Authors are stored relationally in authors and paper_authors, and also as an ordered authors_json projection on each paper for fast export and future frontend search.

Enrich paper-level affiliations

Fetch author affiliations attached to each paper's OpenAlex authorship record:

python -m openpaper fetch-affiliations `
  --from-year 2020 `
  --to-year 2026 `
  --workers 4

Use --venue CCS, --limit 100, or both for a pilot run. The command only processes unresolved DOI papers by default; --refresh retries records that already have a terminal affiliation status.

Affiliations are stored at the paper-author level rather than directly on the author because researchers can change institutions over time:

institutions: normalized institution name, OpenAlex ID, ROR, country, and institution type;
paper_author_affiliations: exact paper, author position, institution, source string, matching method, and confidence;
affiliation_attempts: resumable source-level audit records.

The DBLP four-digit author disambiguation suffix remains unchanged. The frontend shows paper-specific affiliation references separately, for example Jiajun Wu 0001[1]. It never assigns a paper-level institution list to an author unless the source provides a reliable authorship mapping.

Enrich abstracts

Create a local .env from .env.example and fill in the API keys. The real .env file is ignored by Git.

Run a small DOI-matched sample first:

python -m openpaper fetch-abstracts `
  --year 2024 `
  --limit 100

Then run all unresolved DOI records for the year:

python -m openpaper fetch-abstracts --year 2024

The source priority defaults to Semantic Scholar followed by OpenAlex. Semantic Scholar is queried in batches of up to 500 DOI identifiers; OpenAlex is queried in batches of up to 100 DOI identifiers. Responses are only accepted when the DOI lookup also has a compatible title and publication year.

Results are stored in SQLite with provenance and confidence fields. Every source attempt is recorded in the abstract_attempts table. The command also writes:

data/papers/abstracts_<year>.summary.json: coverage and attempt report
data/papers/enriched/<year>.jsonl: enriched yearly dataset export

Provider-specific HTML, JATS, MathML wrappers, HTML entities, Markdown emphasis/links, and invalid extraction control characters are normalized before storage. Literal paths such as /r/Scams and LaTeX commands such as \nabla are preserved. Inline LaTeX is retained in $...$ form so the web frontend can render it with KaTeX and accessible MathML. To audit and migrate an older database:

python -m openpaper clean-abstracts `
  --database data/papers/openpaper.sqlite3 `
  --dry-run
python -m openpaper clean-abstracts `
  --database data/papers/openpaper.sqlite3

Papers without a DOI remain pending for venue-specific or reviewed title matching.

Official proceedings providers handle the major no-DOI collections:

python -m openpaper fetch-official-abstracts `
  --venue NeurIPS `
  --from-year 2020 `
  --to-year 2025 `
  --workers 16

The current abstract-recovery adapters support AAAI OJS, ICML through PMLR, NeurIPS official proceedings, ICLR 2020-2023 through OpenReview search, ICLR 2024 onward through official proceedings, USENIX Security, and NDSS. The remaining official lists still create searchable metadata immediately; their abstracts are filled when a DOI, publisher page, or preprint becomes available. A Semantic Scholar bulk venue/year fallback is available for still-unresolved unique title matches:

python -m openpaper fetch-title-abstracts `
  --venue ICLR `
  --from-year 2020 `
  --to-year 2025

Official records use exact official-URL, unique normalized-title, or a high-confidence unique fuzzy-title match. The title normalizer reconciles common LaTeX and Unicode math notation. In addition to the abstract, the collector stores the source URL, source paper identifier, match method and confidence, fetch time, official DOI, PDF URL, and conference track when available. Collection is committed in batches and can be safely resumed.

Candidates rejected by automatic validation remain in review state. A manually approved candidate is stored with match_method=manual_doi_review and an explicit source such as openalex_manual_review. Its DOI, title, year, content checks, reviewer, notes, and verification URL are retained in the manual_abstract_reviews table.

Retry missing abstracts

Previously unavailable abstracts can appear later as publisher metadata and preprint indexes are updated. Retry only papers whose most recent source attempt is at least seven days old:

python -m openpaper retry-missing-abstracts `
  --from-year 2020 `
  --to-year 2026 `
  --min-retry-days 7

The recovery order is:

DOI lookups through Semantic Scholar and OpenAlex;
configured official proceedings adapters;
per-paper Semantic Scholar closest-title matching;
ACM Digital Library pages for 10.1145/* DOIs;
exact-title arXiv matching.

The title fallbacks accept a publication year difference of at most one year, which covers a preprint followed by the conference version. They still require a unique high-confidence title match. A preprint DOI never replaces an existing publisher DOI.

Every run is stored in abstract_recovery_runs; every accepted or rejected paper match is stored in abstract_attempts. The latest machine-readable summary is written atomically to data/papers/missing_abstract_retry.summary.json. Existing abstracts are never overwritten.

Useful controls:

--min-retry-days 0: force an immediate retry;
--venue ASPLOS --from-year 2026 --to-year 2026: narrow the scope;
--limit 20: run a small batch;
--skip-doi, --skip-official, --skip-title-match, --skip-acm, or --skip-arxiv: disable individual stages.

The persistent worker replaces a platform-specific scheduled task:

python -m openpaper worker

It stores schedules, queued work, run summaries, failures, and a single-worker lock in SQLite. Metadata refresh checks the current and previous years on every scheduled run, checks configured official prepublication sources, and backfills any enabled source/year pair not yet present. New or newly reconciled DBLP papers receive finalized DBLP authors. New papers automatically queue abstract recovery; old missing abstracts remain eligible after the configured retry age. A temporary provider failure is recorded in the job summary but does not terminate the worker. Every metadata refresh creates a verified online SQLite backup under data/backups and retains the newest seven.

Run one job synchronously for diagnostics:

python -m openpaper run-job metadata_sync
python -m openpaper run-job abstract_retry
python -m openpaper run-job affiliation_sync

Publish and restore through GitHub

The Git repository contains application code and configuration, but it does not contain runtime data or API keys. In particular, .gitignore excludes:

.env;
the complete data/ directory, including data/papers/openpaper.sqlite3.

Never remove the secret exclusions or commit a real API key. Before the first push, verify the exclusions:

git check-ignore -v .env
git check-ignore -v data/papers/openpaper.sqlite3
git status

Push the code to a preferably private GitHub repository:

git add .
git commit -m "Initialize OpenPaper"
git branch -M main
git remote add origin git@github.com:YOUR_ACCOUNT/OpenPaper.git
git push -u origin main

Publish the seed database separately

The SQLite database is currently several hundred MiB and cannot be stored as a normal Git object. GitHub blocks regular repository files larger than 100 MiB. Keep the database out of Git history and publish a snapshot as a GitHub Release asset instead. GitHub currently permits an individual Release asset smaller than 2 GiB:

After installing and authenticating the GitHub CLI, create a versioned dataset release:

gh auth login
gh release create dataset-YYYY-MM-DD `
  "data/papers/openpaper.sqlite3#openpaper.sqlite3" `
  --repo YOUR_ACCOUNT/OpenPaper `
  --title "OpenPaper dataset YYYY-MM-DD" `
  --notes "OpenPaper SQLite seed database."

Use a new tag such as dataset-2026-08-01 for a later snapshot. Do not replace the database silently: versioned releases make deployments reproducible and allow rollback.

Restore a fresh clone in WSL

Clone into the WSL Linux filesystem rather than /mnt/c or /mnt/d:

cd /home/ucas
git clone https://github.com/KiRinXC/OpenPaper.git
cd OpenPaper

Download the current database Release asset into the exact path expected by Compose:

mkdir -p data/papers
curl -fL \
  "https://github.com/KiRinXC/OpenPaper/releases/download/dataset-2026-07-27/openpaper.sqlite3" \
  -o data/papers/openpaper.sqlite3
test -s data/papers/openpaper.sqlite3
ls -lh data/papers/openpaper.sqlite3

Optionally verify this snapshot:

echo "80024c77fbbbc08702695affb432dd7b785b8b6107aa09d582c1f56cff213f13  data/papers/openpaper.sqlite3" \
  | sha256sum --check

Create one ignored .env file from the committed template:

cp .env.example .env
chmod 600 .env
nano .env

Fill in:

S2_API_KEY=YOUR_SEMANTIC_SCHOLAR_KEY
OPENALEX_API_KEY=YOUR_OPENALEX_KEY

The worker receives these values as container environment variables. This is convenient for a private, single-user deployment, but container environment variables can be inspected by a user with Docker access. Never commit .env or share its contents.

Verify Docker and start every service:

docker version
docker compose version
docker compose config --quiet
docker compose up -d --build
docker compose ps -a
curl http://127.0.0.1:8000/api/health

The first build can take several minutes. A healthy deployment has the following lifecycle:

init: Exited (0) after seeding, normalizing abstracts, checking FTS, and assigning the live database to the unprivileged runtime user;
web: Up (healthy);
worker: Up.

The exited init one-shot service is expected and does not indicate a failure. Open http://127.0.0.1:8000 from Windows after the health endpoint returns {"status":"ok"}.

If initialization reports OpenPaper seed database is missing, verify data/papers/openpaper.sqlite3. If Compose reports that .env is missing, copy .env.example and fill in both keys. Diagnose other startup failures with:

docker compose logs --tail=200 init web worker

For later code-only upgrades, keep the live database volume and rebuild:

git pull
docker compose up -d --build

Use docker compose down to stop the stack without deleting data. Do not use docker compose down -v unless the live database and its backups should be deleted and the next start should reseed from the downloaded Release snapshot.

WSL and Docker deployment

Keep the deployed copy in the Linux filesystem, for example /home/ucas/OpenPaper, instead of running it from /mnt/d. This gives SQLite, Docker build contexts, and frontend assets native Linux filesystem behavior. The Compose stack contains:

a one-shot initializer that seeds the persistent database and verifies FTS;
a web service exposed only on Windows/WSL loopback port 8000;
a persistent worker for automatic metadata, abstract, and affiliation jobs;
a named volume for the live database and its backups.

Prepare the ignored environment file once:

cd /home/ucas/OpenPaper
cp .env.example .env
chmod 600 .env
nano .env

Fill in both variables from .env.example; the worker uses them for abstract and paper-affiliation enrichment.

Build and start:

docker compose up -d --build
docker compose ps
curl http://127.0.0.1:8000/api/health

Then open http://127.0.0.1:8000 from Windows. Override the host port with OPENPAPER_PORT=8080 docker compose up -d. The binding intentionally remains on 127.0.0.1; change it only if access from other machines is desired.

If Docker Hub is unreachable on the current network, the base image registry is configurable without editing the Dockerfile:

docker compose build --pull \
  --build-arg NODE_IMAGE=docker.m.daocloud.io/library/node:22-alpine \
  --build-arg PYTHON_IMAGE=docker.m.daocloud.io/library/python:3.12-slim
docker compose up -d --no-build

The seed database is copied into the named volume only when the volume has no database. On every initialization, abstract normalization is applied idempotently before the FTS index is checked. Normal upgrades preserve the live volume:

docker compose up -d --build

Stop without deleting data using docker compose down. Do not add -v unless you explicitly want to delete the live database and reseed it on the next start.

Build one venue/year dataset

The unified pipeline accepts a configured venue acronym (or DBLP slug) and one conference year:

python -m openpaper build-dataset `
  --venue ASPLOS `
  --year 2024

It performs five ordered stages:

refresh that venue/year from the DBLP stream;
fetch and store ordered DBLP author signatures;
run DOI-based abstract providers in priority order;
run the configured official proceedings provider;
export the scoped dataset, missing-abstract queue, and completeness report.

Outputs use a stable scope directory:

data/datasets/<dblp_slug>/<year>/papers.jsonl
data/datasets/<dblp_slug>/<year>/missing_abstracts.jsonl
data/datasets/<dblp_slug>/<year>/summary.json

Use already stored DBLP records without making a metadata request:

python -m openpaper build-dataset `
  --venue ICLR `
  --year 2024 `
  --use-local-metadata

Useful workflow controls:

--metadata-only: export titles and metadata without calling abstract APIs
--sources semantic_scholar,openalex: set provider priority
--require-complete: return exit code 2 if any abstract is still missing
--output-dir <path>: choose another dataset root

The pipeline reports complete: true only when the main-proceedings scope is non-empty and every paper has a non-empty title, ordered authors, and abstract.

Export the canonical dataset

After enrichment, generate frontend-ready JSONL plus explicit audit queues:

python -m openpaper export-dataset `
  --from-year 2020 `
  --to-year 2026 `
  --output-dir data/dataset

The export writes:

data/dataset/papers.jsonl: canonical main-proceedings papers;
data/dataset/missing_abstracts.jsonl: unresolved canonical papers;
data/dataset/associated_proceedings.jsonl: preserved excluded records (associated proceedings and front matter, distinguished by scope_reason);
data/dataset/summary.json: field coverage by venue/year and source.

data/papers/openpaper.sqlite3 remains the primary queryable database and contains the full provenance and normalized author tables.

Run the tests:

python -m pytest -q

Navigation Menu

Repository files navigation

OpenPaper

Run the local search interface

Publication source configuration

Adding a conference or journal before DBLP indexes it

Copy-paste task for a coding agent

Prepublication records and canonical identity

Build the title dataset

Enrich paper-level affiliations

Enrich abstracts

Retry missing abstracts

Publish and restore through GitHub

Publish the seed database separately

Restore a fresh clone in WSL

WSL and Docker deployment

Build one venue/year dataset

Export the canonical dataset

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
config		config
docs/superpowers		docs/superpowers
frontend		frontend
openpaper		openpaper
scripts		scripts
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
compose.yaml		compose.yaml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

OpenPaper

Run the local search interface

Publication source configuration

Adding a conference or journal before DBLP indexes it

Copy-paste task for a coding agent

Prepublication records and canonical identity

Build the title dataset

Enrich paper-level affiliations

Enrich abstracts

Retry missing abstracts

Publish and restore through GitHub

Publish the seed database separately

Restore a fresh clone in WSL

WSL and Docker deployment

Build one venue/year dataset

Export the canonical dataset

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages