This document captures notes of discussions between Ted and Josh in November 2024. The goal is to create a package that interfaces with the librarian.
Archive Creation
- The librarian knows about the archive idea and contains a background job that can go through and dump manifests on a cadence once they reach a specific size
- These manifests get dumped to a folder, that the archivist watches with a cron job or similar
- Once picked up, the manifests are dumped to tape
- The archivist then makes an https request to the librarian server to let it know this process is done
- That gets registered in the database as an 'official archive'
So we have one entrypoint that ingests manifests and creates "archives".
Archive Extraction
It is assumed that re-extracting archives will be done to a temporary location (e.g. scratch filesystem). So this means that on a given computing system, there will only be one "canonical" copy of each book (either it is on disk or it is in an archive). Is this correct? Or do we need two types of extraction (extract to untracked temp location and extract to official disk location)? The general steps for extraction are:
- An external process builds the list of books for extraction.
- Archivist tool queries librarian server to get the list of archives that span these books.
- Archivist tool invokes system-specific process to extract archives to disk.
General Notes
This package can be a simple, pure-python package. There should be base classes for the critical points where we need to implement system-specific operations to interface with the offline storage system. At the very least we need methods "storing an archive" and "extracting an archive".
This document captures notes of discussions between Ted and Josh in November 2024. The goal is to create a package that interfaces with the librarian.
Archive Creation
So we have one entrypoint that ingests manifests and creates "archives".
Archive Extraction
It is assumed that re-extracting archives will be done to a temporary location (e.g. scratch filesystem). So this means that on a given computing system, there will only be one "canonical" copy of each book (either it is on disk or it is in an archive). Is this correct? Or do we need two types of extraction (extract to untracked temp location and extract to official disk location)? The general steps for extraction are:
General Notes
This package can be a simple, pure-python package. There should be base classes for the critical points where we need to implement system-specific operations to interface with the offline storage system. At the very least we need methods "storing an archive" and "extracting an archive".