Repository navigation
fix(nara): --objects downloads /medialz/ records; atomic downloads; honest sizes and help - #113
Merged
Merged
Conversation
…d help Found by an agent gathering test PDFs with the openfoia CLI: 19 of 27 NARA records serve objects under /medialz/, which --objects refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while using openfoia itself to gather NARA PDFs for the PDF-engine test corpus.
/medialz/refused. NARA serves objects under both/medialive/and/medialz/(19 of 27 records sampled use/medialz/, e.g. NAID 122681436). The allowlist from fix(nara): --objects downloads scans served from catalog.archives.gov/medialive #110 only had/medialive/, so--objectsstill skipped most records. Both are allowed (https only, no..).objectFileSize: 1234on many objects (live: listed 1,234, real 1,573,115). Resume compared the file to that number and re-downloaded.download_to_filenow streams to<dest>.partand renames when complete (all adapters), so an existing file is whole and resume trusts it.--list-objectsprints "size unknown" for the placeholder.records search --helplisted 2 of 11 sources and said no keys are needed; it now lists all 11 and the key each reads.Tests first (all failed before): medialz allowlist, placeholder-size resume, no partial file at dest mid-stream, size display, help lists every registered source. Full suite: 530 passed.
Live: NAID 122681436
--objects: 4 ingested; rerun: 4 already present; no .part left.🤖 Generated with Claude Code
https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y