OPENNLP-1909: General verified installer for user-supplied third-party resources - #1211
Open
krickert wants to merge 75 commits into
Open
OPENNLP-1909: General verified installer for user-supplied third-party resources#1211krickert wants to merge 75 commits into
krickert wants to merge 75 commits into
Conversation
…ctionaries A Viterbi decoder over word and connection costs segments languages written without spaces; the same engine serves Japanese and Korean because the language lives entirely in the dictionary. Unknown text is handled through the dictionary's character categories, and every span stays in original text coordinates. An installer fetches and unpacks a user-chosen dictionary archive at install time: nothing is bundled, no location is built in, and entry names are flattened so no archive path escapes the target directory. (cherry picked from commit a699c8a)
Common-prefix lookup walks a trie built at load time instead of probing substrings per length, terminating on the first missing prefix and allocating nothing per position. (cherry picked from commit e10ce4b)
A Viterbi search maximizing summed word log-probabilities segments Chinese and similar scripts from a plain word-count lexicon, with unlisted characters falling back to single-character words. The user supplies the lexicon and thereby accepts its license; nothing is bundled. (cherry picked from commit bff3f23)
…nigram segmenters, corrected javadoc
…s, add an EUC-JP loading example
…ng download helper
…e their category run, and validate context ids at load
…expressible dictionary values The lattice tokenizer rescanned the same-category run from every position, so a run of L characters cost on the order of L squared category lookups; a 16,000-character katakana run measured around half a second. One right-to-left pass per stretch now fixes every position's category and run end, and the same 16,000-character run tokenizes in about half a millisecond at 31 million characters per second. The character table holds Category instances instead of names, so the per-character path compares by identity with no name-map lookup, and a char.def mapping to an undefined category now fails at load naming the code point. The lexicon trie's children are sorted character arrays found by binary search, so a descent no longer boxes a Character per step. matrix.def loading rejects connection costs outside the 16-bit range instead of silently truncating them, and dimension products beyond the addressable array size fail at the header. The unigram segmenter's unknown-character fallback advances one code point, never one code unit, so an unknown supplementary character is stepped over whole and no span can split its surrogate halves.
…recoded labels The lexicon trie's per-node child lookup, a binary search over the node's fan-out, paid about a dozen comparisons at the root of a real dictionary; the classic base/check double-array makes every transition one array read and one comparison. Characters are recoded into dense labels ordered by descending frequency before the array is built, so the array stays compact although CJK surfaces draw on tens of thousands of distinct characters, and a character the lexicon never uses misses in the recode table before the array is consulted. On the IPADIC harness the prefix walk now matches the fastest previous implementation at 5.6M chars/s with strictly constant-time transitions, and building the array adds about a quarter second to the 392k-entry load.
…er-position lists The Viterbi lattice held one ArrayList per text position plus one fresh candidate list per position, pure allocation churn on long stretches. Nodes ending at a position now chain through their own link field behind a single head reference per position, and candidate gathering fills one scratch list reused across positions, so building the lattice allocates nothing besides the nodes themselves. IPADIC throughput on the 400k-character harness rises from 5.6M to 6.5M characters per second with identical output.
…example Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest.
…lexicon accessors
…aries A clean-room reader for .dic and .aff files with PFX/SFX rules, strip strings, character-class conditions matched by a single scan, cross products, and char, long, and num flag modes. No dictionary data is bundled: users point at their own files, so dictionary licenses never attach to the jar. Unsupported affix features fail closed, missing analyses rather than inventing them. (cherry picked from commit 0ecc39c)
…lasses Suffix rules now carry the continuation flags declared on their affix text, and analysis undoes a stacked pair when the inner rule's classes allow the outer one, so derived-then-inflected forms reduce to their dictionary word. (cherry picked from commit b543dea)
…spell engine, precise javadoc
…-preserving download helper
…nd strip-only rules, and read the parser through the whitespace seam
…, and stem nothing from nothing Loading the Spanish dictionary of the LibreOffice collection, the same collection this module's README recommends, exposed three gaps against real data. Flags under FLAG UTF-8 are now one code point each instead of one UTF-16 unit, since that dictionary names prefix rules with supplementary characters that would otherwise split into two flags and abort the load; a variation selector after a flag character selects presentation, not identity, and is dropped, which the same file also relies on. A numeric or long flag run ends at the first space or tabulator, the separators the word-list format defines, so trailing morphological text without a tag no longer aborts the load; the morphology cut itself now splits on exactly those two separators, the set the reference implementation's hashmgr.cxx uses, which the javadoc previously claimed while scanning wider whitespace. Stemming the empty word answers the empty word instead of letting a strip-only rule conjure a stem from nothing. All four downloaded dictionaries of the collection, English, Spanish, Hungarian, and German, now load and stem; new tests pin the escaped slash, the multi-word entry with trailing tags, and each corrected behavior.
…able The published Hungarian dictionary flags all of its entries as numeric references into an AF alias table, so without alias support every entry loaded flagless and stemming answered the surface form unchanged. The affix parser now reads the AF table, the first line as the declared count and every further line as one flag run with trailing comments discarded, and a purely numeric flag field in the word list resolves as a 1-based reference into it, failing loud with the line and table size when the reference is out of range. Without an AF table numeric fields keep their FLAG num meaning. The Hungarian dictionary of the LibreOffice collection now stems inflected forms; remaining gaps there are compound territory, which is tracked separately.
…heir boundary character Undoing a suffix requires the word to end with the rule's affix material, so only rules whose material ends in the word's last character can ever apply, and likewise for prefixes and the first character. The dictionary now buckets its rules by that boundary character at load, and every scan in the stem path, including the twofold and cross-product inner scans, walks the one bucket plus the strip-only rules instead of the whole inventory. Measured on the LibreOffice dictionaries at 4,000 words each: English 553k to 1,024k words per second, Spanish 9.6k to 28.9k, German 132k to 287k.
When the affix analysis finds nothing and the affix file declares compounding, a word now splits into two listed parts that the COMPOUNDFLAG or the positional COMPOUNDBEGIN and COMPOUNDEND flags allow in their positions, honoring COMPOUNDMIN, with the parts reported left to right. Affix analyses keep precedence, listed words never decompose, and unflagged parts block a split. Against the published Hungarian dictionary the unlisted kutyahaz decomposes into its two nouns while listed compounds and inflected forms keep their regular analyses. Longer chains, syllable rules, and the compound-only flags stay unimplemented and simply leave such words unanalyzed.
…itioning A NEEDAFFIX (or PSEUDOROOT) entry is a virtual stem that exists only to be affixed, an ONLYINCOMPOUND entry appears only inside compounds, and a FORBIDDENWORD entry is listed to be blocked; none of them is a standalone analysis anymore, per homonym flag set, and an affix carrying NEEDAFFIX among its continuation classes yields no single-removal analysis while its twofold and cross-product removals stand, the other affix being exactly the further one required. A cross-product now also requires both removed affixes' flags in the same homonym's flag set, and CIRCUMFIX binds marked prefix and suffix halves to one another, so neither half analyzes alone and a marked half never combines with an unmarked affix. Decomposition grows from two verbatim parts to the compound machinery the published German dictionary actually uses: any number of parts under the positional COMPOUNDBEGIN/COMPOUNDMIDDLE/COMPOUNDEND flags and COMPOUNDWORDMAX, parts standing on an entry plus one affix with COMPOUNDPERMITFLAG required at internal boundaries and COMPOUNDFORBIDFLAG barring marked forms, zero and dash linking suffixes included, an uppercased retry for capitalized entries spelled lowercase inside a compound, and the CHECKCOMPOUNDDUP, CHECKCOMPOUNDCASE, and CHECKCOMPOUNDTRIPLE junction guards, case judged against the original surface. A listed forbidden word never decomposes, and a fixed part-licensing budget keeps adversarial input bounded, missing analyses rather than stalling. Abbildungsverzeichnis, Haustuer, and Kinderzimmer now decompose against de_DE_frami at 137k words/s single-threaded. An opt-in test class checks everyday morphology against downloaded dictionaries under -Dopennlp.hunspell.dict.dir; nothing is bundled.
Extend docbkx/stemmer.xml with the hunspell affix-stemmer section, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and HunspellManualExampleTest asserting the load-and-stem values the chapter prints. Point the dictionary README at the new manual example.
…, split null contracts, thread-safety annotations
…e tokenizer overrides
The frequency lexicon was trimmed with String.trim(), which strips only ASCII
control characters and the space. A line starting with an ideographic space
(U+3000), ordinary in hand-edited CJK text files, therefore kept that space as
part of the word and pushed the count field one token to the right, so the load
failed as a malformed count. The lexicon reader now trims with
StringUtil.trimUnicodeWhitespace, matching the White_Space convention the rest
of the tokenizer already scans by, and a test pins the leading U+3000 case.
The mecab reader's line and numeric-field trims move to the same call so one
class does not mix two whitespace judgments; those fields are ASCII in valid
dictionaries, so the behavior there is unchanged.
Both tokenizer views also gain {@inheritdoc} and their null contract, and the
unknown-candidate helper drops a static modifier it did not need.
…fold fixture duplication - Document the private lattice helpers decode, relax, and candidates, and the installer's boundedStream, with the parameter, return, and exception contracts the review expects every method to carry. - Document the WordEntry and Category record components and the double-array builder's findBase and ensureCapacity helpers. - Record on analyze, tokenize, and tokenizePos that a unk.def without a DEFAULT template leaves the lattice disconnected and makes them throw IllegalStateException. - State on readLines that it never returns an empty list, which is what lets the matrix.def header be read before the emptiness check. - Rename the Tokenizer override parameter from s to text in LatticeTokenizer and UnigramSegmenter, so the javadoc names a parameter that exists. - Hoist the matrix.def, char.def, and unk.def file names, the DEFAULT category name, the 0x code point prefix, the .. range separator, and the flag value into named constants in MecabDictionary, and let LatticeTokenizer reach the DEFAULT name through MecabDictionary instead of repeating the literal. - Name the tar block size, header field offsets, and field lengths in the TarGzArchives test helper instead of writing 512, 124, and 148 inline. - Replace the boolean[1] capture in candidates with a check that the candidate list is still empty, which is the same signal without the array. - Track the best boundary total in decode instead of recomputing the incumbent's connection cost on every comparison. - Drop the categories map field from MecabDictionary, which nothing read once the constructor resolved the DEFAULT category out of it. - Match the char.def code point prefix once, case insensitively, rather than testing 0x and 0X separately, and cut the range at the separator's own length. - Trim the matrix.def header before parsing it and report an empty first line as an empty matrix.def, since readLines never yields the empty list the previous check was looking for. - Split the omnibus malformed-dictionary test into named cases that pin the messages for a missing definition file, a char.def without DEFAULT, a lexicon with no entries, and an empty matrix.def. - Parameterize the malformed char.def cases and the malformed unigram lexicon cases, which were repeated assertThrows calls over one fixture shape. - Add a Morpheme test pinning the null and empty argument rejections and the defensive copy of the feature list. - Extend the invalid-argument tests to the entry points that were uncovered: MecabDictionary.load with a null directory or charset, the installer's null target, UnigramSegmenter's path and stream overloads, and both tokenize methods of each tokenizer. - Fold the repeated Files.write fixture calls into one write helper and hoist the shared lexicon, matrix, char.def, and unk.def fixture text into constants. - Correct dev/README-mecab-dictionaries.md to say that dicrc is the configuration file the distributions ship alongside the csv and def files a MecabDictionary reads, rather than implying the dictionary reads dicrc itself.
…plete javadoc
- Fold the two per-kind bucketing loops in the HunspellDictionary constructor into a
single bucketByBoundary helper that takes the rule list, the kind, and the sink for
the rules with empty affix material.
- Fold collectSuffixedPartStem and collectPrefixedPartStem, which differed only in the
boundary they face, into one collectAffixedPartStem with a suffix marker and an
atEdge marker; document what atEdge means at each end.
- Extract a parseValue helper for the single-integer directives so COMPOUNDMIN and
COMPOUNDWORDMAX no longer share one case body that re-tests which directive it is.
- Extract PREFIX_TAG, SUFFIX_TAG, and NO_MATERIAL constants and use them at the affix
block header, the rule lines, and the strip and affix material checks.
- Give FORBIDDENWORD its own case in the flag directive switch instead of letting the
catch-all default assign it, and make that default throw for a directive listed on
the outer switch but not handled on the inner one.
- Add the missing javadoc on the AffixCondition and HunspellDictionary constructors,
the Affix record components, and the splitLines, splitOn, and split helpers.
- Convert the single-line accessor javadoc on the compounding and affix bucket getters
to the {@return ...} form, and replace the hand-written prose on
HunspellStemmerFactory.newStemmer with {@inheritdoc} plus the instancing note.
- Trim commentary that restates the code: the bucketing rationale duplicated in
HunspellStemmer, the LibreOffice Spanish anecdote on the code point flag reader, and
the sentence left dangling in testGermanCompoundsDecompose.
- Drop the defensive null and directory guards from the test helpers
writeAndLoadFixture and load, which no caller can trip, and document what the real
dictionary tests assert.
- Fold the repeated ByteArrayInputStream plumbing in HunspellStemmerTest into two load
overloads, one UTF-8 and one taking the charset the SET declaration test needs.
- Turn the four table-style stemming tests into parameterized tests over their word and
expected stem pairs, so a failing row names itself.
- Add testNullArgumentsAreRejected, pinning the exact IllegalArgumentException message
of every public entry point including the argument names the stream loader reports.
- Correct the stemmer manual: name the example files after the fixture the test loads
rather than en_US, and state that the printed stems are the fixture's, since which
stem a published dictionary yields is that dictionary's decision.
Place LatticeTokenizer, UnigramSegmenter, MecabDictionary, and Morpheme with the other resource-driven tokenizers. Keep MecabDictionaryInstaller in runtime. Lift MAX_ENTRIES into ResourceLimits so api loaders can share the bound, and reject incomplete or oversized matrix.def payloads.
Reject ICONV, OCONV, and COMPLEXPREFIXES at load time; keep skipping cosmetic tables such as REP. Copy lookup results defensively and document the compound search budget on HunspellStemmer.
The installer flattened every csv and def file in the archive into the target directory, so mecab-ko-dic's nested user-dic templates, whose numeric fields are empty because they are mecab-dict-index input, landed beside the real lexicon and failed the load. On a case-insensitive file system a template could even overwrite a real lexicon file of the same base name. Entries deeper than one leading directory are now skipped.
…d full-strip rules with failing tests COMPOUNDRULE, IGNORE, and KEEPCASE alter analyses when ignored, so the fail-closed loader policy requires them to fail at load time like ICONV, OCONV, and COMPLEXPREFIXES; the loader currently accepts them silently. A suffix rule whose strip string is the whole stem is applied without the FULLSTRIP declaration hunspell requires for it, inventing a stem for a surface form the dictionary does not license.
…ll-strip rules behind FULLSTRIP COMPOUNDRULE licenses pattern compounds, IGNORE drops characters before matching, and KEEPCASE forbids the capitalized variants this stemmer analyzes through lowercasing, so ignoring any of them would change stems with no signal; they now join ICONV, OCONV, and COMPLEXPREFIXES in the load-time rejection. An affix rule whose strip string consumes the whole stem is now undone only when the affix file declares FULLSTRIP; without the declaration the rule is skipped at match time, which is what hunspell itself does rather than rejecting the file. The manual and the dictionary README follow the loader.
Red evidence: mvn test-compile fails with "cannot find symbol: class Limits" and the missing install(URI, Path, String, Limits) and resolveRedirect seams; the SHA-512, staged-atomicity, and redirect policy tests pin behavior the current installer does not have. The new tests cover: SHA-512 digests selected by hex length next to SHA-256, malformed digests rejected as argument errors, staged installation that leaves the target untouched when a tar, zip, or truncated archive fails partway, download and expansion ceilings against oversized sources and small archives that expand into bombs, and a scripted loopback HTTP server exercising redirects (absolute, relative, capped chains, missing Location, non-http targets, https downgrade), error statuses, stalled responses against the read timeout, and bodies that exceed the download ceiling with or without a declared length.
Downloads and unpacking now run under Limits: http and https fetches get connection and read timeouts, follow at most a capped number of redirects, resolve relative Locations, and refuse redirect targets that leave the http and https schemes or downgrade https to http. A declared content length beyond the download ceiling fails before the body is read, and both the transferred and the expanded bytes are charged against their ceilings so lying servers and archive bombs abort within one buffer. Checksums accept SHA-512 next to SHA-256, selected by hex digest length, and malformed digests fail fast as argument errors. Installation is staged: content unpacks into a hidden staging directory on the target filesystem and is promoted by renames only after the download verified and every entry unpacked cleanly, so a failed installation leaves the target as it was. The red suite from the previous commit passes: 27 ResourceInstaller tests, 12 scripted local-server HTTP tests, 14 TarStream tests, and the full runtime module at 1700 tests.
The manual's resource-installer section now explains digest-length algorithm selection (64 hex characters SHA-256, 128 SHA-512), the staged installation guarantee that a failed install leaves the target untouched, and the bounded network behavior: timeouts, the capped redirect policy with its scheme and downgrade rules, and the download and expansion ceilings with their defaults. A limits listing is mirrored by ResourceInstallerTest#testInstallWithinCustomCeilingsSucceeds.
Both sides of each ceiling are now asserted: a download and an expansion exactly at the ceiling install, one byte of cumulative overrun across entries rejects, proving the expansion budget is shared rather than per entry. Limits.DEFAULT values are pinned, all five redirect statuses the code claims (301, 302, 303, 307, 308) are followed under a parameterized test, a zero redirect allowance refuses the first redirect, a malformed Location fails loud with the offending value, SHA-512 comparison ignores hex letter case like SHA-256, and reinstalling over the same target replaces the delivered files. The manual now cites the staged-installation and default-limit tests next to the guarantees they assert. 39 + 19 installer tests green.
Does not compile on its own: the tests call Limits.builder() and
createDownloadFile, both added in the following commit. Against the
current implementation, with those two added as stubs, the run is:
ResourceInstallerHttpTest
testSubMillisecondReadTimeoutStillTimesOut
timed out after 15 seconds
testTimeoutBeyondTheMillisecondRangeIsCapped
Arithmetic long overflow
ResourceInstallerTest
testPromotionRefusesToFollowASymlinkedDirectory
Expected java.io.IOException to be thrown, but nothing was thrown
testUnsupportedSourceSchemeIsRejected
expected IllegalArgumentException but was java.net.UnknownHostException
expected IllegalArgumentException but was java.net.UnknownServiceException
expected IllegalArgumentException but was java.nio.file.NoSuchFileException
expected "source scheme must be ..." but was "URI is not absolute"
testUnsupportedSourceSchemeIsRejectedBeforeCreatingTheTarget
expected IllegalArgumentException but was java.net.UnknownHostException
TarStreamTest
testHeaderWithWrongChecksumIsRejected
testRejectedPaxGlobalHeader (8 cases)
testSparseEntriesAreRejected (2 cases)
testBase256SizeFieldIsRead
testBase256SizeFieldCarriesLengthsBeyondTheOctalRange
testBase256SizeFieldAcceptsTheLargestRepresentableLength
testBase256SizeFieldBeyondTheLongRangeIsRejected
testNegativeBase256SizeFieldIsRejected
the base-256 size encoding is not read at all
testStartsWithHeaderRejectsUstarMagicWithoutAChecksum
expected false but was true
testPaxExtendedHeaderSuppliesTheEntryName
testGnuLongNameHeaderSuppliesTheEntryName
expected the full path, but was its first 100 bytes
testPaxExtendedHeaderSuppliesTheEntrySize
expected 10 but was 0
testGnuHeaderDoesNotReadItsAtimeAsANamePrefix
expected "./short.txt" but was "15237132225/./short.txt"
testUstarPrefixIsJoinedToTheName
testHeaderWithAnEmptyNameIsRejected
testEntryStreamRejectsInvalidReadRanges
testZeroLengthReadReturnsZero
expected 0 but was -1
The sub-millisecond read timeout hangs rather than failing an assertion:
zero milliseconds means no timeout to HttpURLConnection, so the tightest
setting a caller can express becomes the loosest. That is why the test
carries an explicit @timeout.
The tar fixtures follow archives written by GNU tar 1.35 rather than a
minimal shape, because the shape is the point. A pax archive carries an
extended header ahead of every entry, including entries needing no
override, so the metadata-only case is the common one. The entry header
after an extension header holds a truncated name, so the extension header
is the only place the real one appears. Verified against tar --format=pax,
--format=posix, --format=gnu, and --format=gnu --incremental.
The base-256 size fixture is synthetic, since producing a real one needs an
entry of 8 GiB or more. It was cross-checked the other way instead: GNU tar
1.35 lists a header written by TarArchives.base256Header as 8589934592
bytes, so the encoding the tests assert against is the one tar writes.
Two timeout tests replace an earlier pair that computed the expected
milliseconds with their own copy of the conversion. Those asserted the
test's arithmetic, not the installer's, and would stay green through any
regression. These drive HttpURLConnection instead.
testClassicHeaderWithoutUstarMagicIsRead passes before the change as well.
It guards the new checksum-based detection against dropping classic v7
archives, which the previous ustar-magic shortcut happened to accept.
Also consolidates the tar fixture that existed once per test package into
TarArchives, which now writes real ustar magic and header checksums, and
builds classic, GNU, and prefix headers plus pax records with correct
length prefixes.
ResourceInstaller Timeouts: a positive duration shorter than a millisecond rounded to zero, which HttpURLConnection reads as no timeout at all, and Duration.toMillis raises ArithmeticException on a duration too large for the long range, before the old Math.min could cap it. Conversion now clamps into [1, Integer.MAX_VALUE] and catches the overflow. Schemes: only http, https, and file are accepted, checked at the public boundary before the target directory is created. Anything else went to whichever URL handler the runtime had installed; those carry no connection or read timeout, so an unresponsive server blocked the caller forever. The byte ceilings did apply on that path, since they are charged as bytes arrive. file locations now read through Files.newInputStream. Download file: created on the target filesystem rather than in the system temporary directory, where the 1 GiB default ceiling could exhaust a small /tmp while the target had room. It is hidden, so a leaked one shows up as staging residue rather than as an installed file. Promotion: refuses to descend through a symbolic link that already exists below the target. Every entry name can be inside the staging directory and the content still land outside the target, because createDirectories followed such a link. This covers links present when the installation runs, not a tree modified while it runs. Limits: adds a builder seeded from DEFAULT, so a caller states only what differs instead of five positional arguments, two of them Duration and two of them long. The record stays immutable and the builder validates through the canonical constructor. TarStream Detection now verifies the header checksum instead of accepting the ustar magic outright, so arbitrary content carrying that magic at offset 257 is no longer read as an archive with a trusted size field. Both the unsigned and the signed sum are accepted, as historical writers differ. Long names are read from the extension header that carries them. A pax extended header supplies path and size for the entry after it; a GNU long-name header supplies its name. Both are everyday formats: bsdtar and tar --format=posix write an extended header ahead of every entry, and the entry header that follows holds only a truncated name, so neither ignoring nor refusing the extension header is workable. Keywords other than path and size are ignored, because this reader exposes only an entry's name, size, type, and content, and no other pax keyword changes those. Sparse entries are the exception and are refused, by type flag S and by the GNU.sparse.* records, because their archived bytes encode holes rather than the content. A global header is refused if it carries path or size, which would change every entry after it. The POSIX ustar name prefix is honored, but only for POSIX ustar. GNU writes "ustar" and two blanks where ustar writes "ustar" and a NUL, and puts atime at the offset ustar gives to the prefix, so reading the two alike delivered every entry of a GNU incremental archive under a directory named after an octal timestamp. A size field in the base-256 encoding, which GNU writes for entries beyond the eleven octal digits the field holds, is read, so an entry of 8 GiB or more states its length correctly. The sign sits in the bit below the encoding marker, not in the marker itself, so a negative value is refused rather than wrapped into an enormous positive length, and a value past the long range is refused rather than truncated into a short one, which would stop the reader inside the entry and leave it reading content as headers. A header with a valid checksum but no name is refused, matching what detection already required of the first byte. entryStream's read(byte[], int, int) validates its range before answering, so an invalid range is reported even when the entry is exhausted or the length is zero, and a zero-length read returns 0 rather than -1. This override raises the exceptions InputStream specifies rather than the IllegalArgumentException used elsewhere in the package.
States the accepted source schemes and why the others are refused: those handlers carry no connection or read timeout. The byte ceilings applied on that path either way, so the manual does not claim otherwise. Records which tar formats unpack under their real paths, namely bsdtar, tar --format=posix, and tar --format=gnu, and which entries are refused and why. Scopes the staged-installation guarantee to symbolic links that already exist below the target, which is what the check covers. Switches the Limits example to Limits.builder(), so it states only the limits that differ from the defaults instead of five positional arguments. The listing is mirrored by ResourceInstallerTest#testInstallWithinCustomCeilingsSucceeds.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 14, 2026
krickert
added a commit
that referenced
this pull request
Aug 15, 2026
…ing tests) Red evidence: all four new tests fail at this commit. An http source without a checksum installs instead of being rejected, a checksum-less https call reaches the network (UnknownHostException instead of the argument error), a reinstall silently replaces the existing file, and a colliding archive promotes its fresh entry beside the collision. The pinned contract, from the OPENNLP-1894 review threads: a checksum is required for http and https sources on every overload and rejected before any connection is opened or the target directory is created, while file sources may still omit it; promotion refuses to replace a file that already exists in the target, names the file, detects the collision before moving anything so a failed installation never leaves a mix of old and new files, and a reinstall succeeds once the operator removes the old file.
…existing files An http or https source now requires a checksum on every overload, rejected as an argument error before any connection is opened or the target directory is created; only file sources may skip verification. The checksum-less overload documents that it treats the source as trusted caller input and performs no cryptographic integrity verification. This restores the contract the OPENNLP-1894 download path settled on: no unverified remote fetch remains. Promotion no longer replaces existing target files. Every destination is checked for vacancy before the first move, so a colliding installation leaves the target exactly as it was instead of a mix of old and new files, and the collision message names the file. Moves drop REPLACE_EXISTING, so a file that appears between the check and the move fails the move instead of being clobbered. Refreshing a resource means removing its old files first. The http failure-path tests pass a well-formed placeholder digest, since they fail before verification runs; the reinstall test now pins refusal, the surviving content, and the remove-then-reinstall flow.
…ests) Red evidence: mvn test-compile fails with "cannot find symbol" on Limits.maxEntries(), the three ceiling property name constants, Limits.Builder.maxEntries, and the longProperty parser; none of them exist yet, and no entry-count bound exists that the byte ceilings would enforce, since an archive of countless tiny files stays under maxExpandedBytes while exhausting directory entries. The pinned contract, from the OPENNLP-1894 review: archive entry counts are budgeted like bytes, rejecting the entry beyond the ceiling on tar and zip alike while exactly-at-the-ceiling installs, with a 100000-entry default; and the three ceiling defaults (download bytes, expanded bytes, entries) read system properties once at class load, falling back to the built-in value when the property is absent, not a number, empty, or not positive.
…load Limits gains maxEntries, a budget on archive entries counted like the byte budgets: tar and zip unpacking charge every entry, including directories, and reject the entry beyond the ceiling, while an archive exactly at the ceiling installs. The default is 100000 entries. This closes the gap the byte ceilings leave open, where an archive of countless tiny files stays under maxExpandedBytes while exhausting directory entries. The three ceiling defaults now read system properties once at class load: opennlp.download.max.bytes, opennlp.install.max.total.bytes, and opennlp.install.max.entries, so an operator can raise a ceiling for a known large resource without a code change. A value that is absent, not a number, or not positive falls back to the built-in default. The property names are public constants pinned by tests, and the parser is tested directly since DEFAULT captures its values once. Existing Limits construction sites state the default entry ceiling explicitly; the invalid-value tables cover the new component.
… overrides The installer chapter now states that http and https sources require a checksum while file sources are trusted caller input, that promotion refuses to replace existing files and checks every destination before moving anything, and that archives are bounded by a 100000-entry default ceiling beside the byte ceilings, with the three startup property names that override the ceiling defaults.
…eiling prose The entry-ceiling message, the collision message prefix, and the parser test property name become test constants instead of repeated literals, and the inode-exhaustion rationale for maxEntries now lives once, in the Limits javadoc and the manual, instead of being repeated in a test javadoc.
…ller Stacks the resource installer on the hunspell stemmer work so both download paths can converge on ResourceInstaller.
…staller Stacks the resource installer on the CJK lattice work so both download paths can converge on ResourceInstaller.
…vior (failing tests) Red evidence on the merged, unconverged tree: - testPaxLongNamedEntryInstallsUnderItsRealName fails with "java.io.IOException: the archive contains no dictionary file": the installer's own tar reader cannot read a pax path record, so the long-named lexicon entry is invisible to it. - testReinstallOverAnExistingDictionaryIsRefused fails with "Expected java.io.IOException to be thrown, but nothing was thrown": the old extraction replaces existing files silently. - DictionaryCatalogTest#testInstallStoresTheEntryUnderItsSourceName does not compile: "cannot find symbol: method install(String, Path)" and "cannot find symbol: variable REMOTE_DOWNLOAD_PROPERTY" in DictionaryCatalog, pinning the converged API before it exists.
One hardened download-and-unpack path serves the hunspell and mecab dictionary features: - DownloadUtil returns to its pre-branch, model-only surface; the file-download extensions and their DownloadUtilFileTest are removed. - DictionaryCatalog owns the opennlp.download.remote opt-in gate and installs entries through ResourceInstaller. The redundant filename catalog key and Entry component are gone: an installed plain file is stored under its source name, which every shipped entry already used. - HunspellDictionaryDownload fetches the cataloged files through the catalog install; a file that already exists in the target is refused. - MecabDictionaryInstaller keeps only its payload selection (csv, def, dicrc at the archive root, flattened) and delegates fetch, verification, and unpacking. Its own tar reader, gzip ratio budget, and ceiling properties are gone; ResourceInstaller.Limits and the opennlp.download.max.bytes, opennlp.install.max.total.bytes, and opennlp.install.max.entries startup properties bound the install, and pax and GNU long names now unpack under their real paths. Entries flattening to the same base name are refused instead of silently overwriting each other. - TarGzArchives delegates tar layout to the TarArchives fixture, and InstallerTestSupport digests delegate to DigestTestUtil, so each test helper exists once. The per-installer budget tests moved with the budgets: the shared ceilings are pinned in ResourceInstallerTest.
The stemmer chapter states that catalog downloads go through the digest-verified ResourceInstaller path and that refreshing means removing old files first. The tokenizer chapter replaces the per-installer extraction ceilings with the shared ResourceInstaller defaults and their three startup properties.
krickert
marked this pull request as ready for review
August 18, 2026 21:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a general installer for user-supplied third-party resources: training corpora, dictionary archives, and lexicons that the project cannot bundle. The caller supplies the location and thereby accepts that resource's license; no locations are built in and no data ships with OpenNLP.
ResourceInstalleris a single hardened download-and-unpack path:http,https, andfilelocations are accepted, checked at the public boundary. Remote fetches carry connection and read timeouts, follow a bounded number of redirects, and refuse redirects that leave the http/https schemes or downgrade https to http.httpandhttpssources and optional forfilesources, which are trusted caller input; it is verified against the downloaded bytes before anything is unpacked: a 64-character hex digest selects SHA-256, a 128-character one SHA-512.Limitsvalue (download bytes, expanded bytes, entry count);Limits.DEFAULTapplies when none is given andLimits.builder()starts from it. The three default ceilings can be raised at JVM startup viaopennlp.download.max.bytes,opennlp.install.max.total.bytes, andopennlp.install.max.entries.*.binsources are stored packed, because OpenNLP model files are zip archives their consumers load packed.The tar reading is a small forward-only reader,
TarStream, rather than a dependency. It reads classic v7, POSIX ustar, GNU, and pax formats with a valid header checksum required in every case, honors the ustar name prefix field, reads paxx(path, size) and GNULlong-name extension headers, and reads GNU base-256 sizes for entries of 8 GiB or more. Sparse entries are refused because their archived bytes are not the file content, and a pax global header carryingpathorsizeis refused because it would silently rewrite every following entry. Metadata expansion (extension header sizes) is bounded like everything else.Every behavior above is pinned by tests, in failing-test-then-fix commit pairs.
This PR is now stacked on #1190 and #1191 and carries the convergence discussed there: the final commits delete the per-feature download code both PRs grew during review.
DownloadUtilreturns to its model-only surface,DictionaryCatalogowns theopennlp.download.remoteopt-in gate and installs throughResourceInstaller, andMecabDictionaryInstallerkeeps only its payload selection while delegating fetch, verification, and unpacking (gaining pax and GNU long-name support). Until #1190 and #1191 merge, the diff here includes their changes; the installer itself isResourceInstaller/TarStreamplus the last three commits. Suggested merge order: #1190, #1191, then this.