Repository navigation
Conversation
The repository format has been fixed at 5 since 1.00. It was a compile-time constant written into every info file and manifest, and any mismatch at load was a hard error. A version could only read the format it wrote, so introducing a format meant a flag day where every repository had to be recreated. 1.00 was that flag day and this branch is so the next format does not need one. A file is readable when its format is between REPOSITORY_FORMAT_MIN and REPOSITORY_FORMAT_MAX. A version can read a format it does not write and a repository can hold both while older backups and archives expire. The format is requested with repo-format, which is repo indexed and valid only for stanza-create and stanza-upgrade. It is command line only so a format left in the configuration cannot migrate a stanza as a side effect of an upgrade run for another reason, such as a PostgreSQL version upgrade. The default stays 5. stanza-upgrade --repo1-format=6 rewrites the two info files and nothing else, so a repository of any size migrates in the time it takes to write them. A downgrade is refused, since the info files are what stop a version from reading a format it does not support. The two files are saved separately, so an upgrade interrupted between them leaves a mismatch that running the upgrade again repairs and that every command reading both files now reports. A backup set adopts a new format only at a full backup. A prior backup is a candidate for diff or incr only if it is at the format new backups are written with, and a resumable backup at another format is discarded, so a backup set is never mixed and its format can be read once rather than per file. Migrating therefore costs one full backup per stanza. Once the info files are at format 6 a version that supports only format 5 cannot read the stanza at all, including the backups still at format 5. Such a version fails at info load with "expected format 5 but found 6" before touching any data.
Repositories written before format 6 derive the key from the passphrase with SHA-1 and continue to. Format 6 derives with SHA-256. The digest is chosen along with the pass rather than at each use, so it now travels with the pass in CipherSpec and defaults to SHA-256, and the places that must keep deriving with SHA-1 say so explicitly. A reader cannot learn which digest a file needs from the file itself, since the format is stored in the content the pass decrypts. Encrypted info files at format 6 therefore begin with a plaintext header, PGBR followed by the three-digit format and a byte held back for whatever the header turns out to need, in place of the OpenSSL Salted__ magic. A file that begins with the magic was written at format 5, the only format there was before the header. The header format is validated before anything is decrypted, since this version cannot know what a newer format expects, and is compared against the format read from the content afterward so a file assembled from parts of two files is rejected rather than half read.
# Conflicts: # test/src/module/common/cryptoTest.c
# Conflicts: # build/config.yaml # doc/xml/reference.xml # src/info/info.c # src/info/info.h # src/version.h # test/src/harness/info.c # test/src/harness/info.h # test/src/module/command/restoreTest.c # test/src/module/common/cryptoTest.c # test/src/module/info/infoArchiveTest.c # test/src/module/info/infoBackupTest.c # test/src/module/info/infoPgTest.c # test/src/module/info/infoTest.c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repositories written before format 6 derive the key from the passphrase with SHA-1 and continue to. Format 6 derives with SHA-256. The digest is chosen along with the pass rather than at each use, so it now travels with the pass in CipherSpec and defaults to SHA-256.
The write side does not add encryption itself: a file is written at a format and the cipher filter decides from that whether a header goes in front, so header is a parameter only a reader passes. verify checksums an info file as it is stored rather than as it decrypts, since the header has to be read before decryption can start, and a file and its copy are written from the same bytes, which is all that comparison needs.
A pass outlives the format of the file it is stored in, so the format cannot be what decides how that pass derives. An info file at format 6 stores cipher-digest next to cipher-pass, and a pass found without one derives with SHA-1, which is what every repository derived with before the digest could be stored. stanza-upgrade writes the digest the pass already had, so archives and backups written before the migration are still readable afterward. Only a pass generated at format 6 derives with SHA-256, i.e. one for a stanza created at that format or for the manifest of a full backup written at it. The info files themselves are re-encrypted whole by the migration, so they derive with the digest their own format calls for.
A reader cannot learn which digest a file needs from the file itself, since the format is stored in the content the pass decrypts. Encrypted info files at format 6 therefore begin with a plaintext header, PGBR followed by the three-digit format and a byte held back for whatever the header turns out to need, in place of the OpenSSL Salted__ magic. A file that begins with the magic was written at format 5, the only format there was before the header.
The header format is validated before anything is decrypted, since this version cannot know what a newer format expects, and is compared against the format read from the content afterward so a file assembled from parts of two files is rejected rather than half read. The format in the content is read as an unsigned int, so a value that is negative or too large to be one is rejected where it is read rather than narrowed into range first.
The format constants move from version.h to common/format, along with repoFormatValidate() and repoFormatDigest(). The cipher filter needs both as it reads a header and sits below info, so common is the only place they can be shared from. The filter turns the digest a format calls for into an openssl digest with the same lookup by name it already used for the digest a cipher spec contains.