Opennlp 1897 term vectors - #1212
Draft
krickert wants to merge 24 commits into
Draft
Conversation
…yers over the original text Adds opennlp.tools.document to opennlp-api: Document (immutable, copy-on-add layer container over the original text), Annotation (a typed value on a Span), LayerKey (open, typed layer identity), and DocumentAnnotator (pipeline step declaring the layers it requires and provides). DocumentAnalyzer assembles annotators into a pipeline validated at build time. Standard keys in Layers cover sentences, tokens, part-of-speech tags, and entities, populated through thin adapters over the existing SentenceDetector, Tokenizer, POSTagger, and TokenNameFinder interfaces, which stay the primary API for single-task use and are unchanged. All spans refer to the text as supplied. No new dependencies.
…ntainer, javadoc precision pass
…rom missing layers, validate providers at build time
… adaptive data on failure The lemmatizer adapter now slices tokens and tags per sentence like its POS and name-finder siblings, so lemmatization decisions never cross a sentence boundary, and it declares the sentence layer as required. The POS adapter rejects a tagger that returns a wrong tag count. The name-finder adapter rejects mentions whose token indices lie outside their sentence instead of silently reading the next sentence's tokens, clears adaptive data even when annotation fails, and derives UNTYPED from NameSample.DEFAULT_TYPE instead of re-declaring the literal.
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…nt rule Adds the Document Annotation Container chapter to the manual, with every code example and every stated span and value mirroring the passing pipeline example test. The review pass aligns the branch with the project's conventions: layer key ids validate through StringUtil.isBlank, the annotator interface leaves thread safety implementation specific, the sentence and tokenizer adapters document annotate like their siblings, repeated rejection-message literals become per-class constants, and the name finder test's nine anonymous fixtures fold into one helper. Layers now states the key placement rule: core layer keys live there, capability layer keys on their providing annotator.
Every key the toolkit defines now carries the opennlp: id prefix (opennlp:sentences, opennlp:tokens, opennlp:pos, opennlp:entities, opennlp:lemmas, opennlp:stems). An extension defines its keys under its own prefix, and a bare id stays legal for an application-local layer, so ids from independent producers cannot collide. The rule is stated on Layers, LayerKey, and in the manual chapter.
A layer key now declares whether its layer is positional or document-scoped. A positional key, the default, guarantees a span on every annotation, so consumers never null-check one. A document-scoped key, created through LayerKey.document, carries whole-document values without spans, the home for a language id, a category distribution, or provenance. The scope is declared per key, never per annotation: the container rejects a span-less annotation under a positional key and a spanned annotation under a document-scoped key, naming the layer either way. Scope participates in key equality.
…on text The three invariants the contract tests already enforce are now stated on the Document interface and in the manual chapter: layers preserve insertion order and are never reordered, layers are immutable once added and detached from the caller's input list, and adding a layer is once-only with the rejection naming the key. Together they keep index-based references between layers valid for the lifetime of the document.
A corpus may carry a hand-annotated version of a layer beside a produced one. The convention is a gold: id prefix on the same key scheme, for example gold:opennlp:tokens beside opennlp:tokens. Because adding a layer is once-only, competing versions of a layer always live under distinct keys and never replace each other. Stated on Layers and in the manual chapter, with a contract test pinning the coexistence.
…ctories on Layers
…le test
Add {@inheritdoc} to the Document, LayerKey, and adapter overrides, and note in
the manual that DocumentPipelineExampleTest asserts the pipeline round-trip.
…ainer contract - Reject zero-length finder mentions in NameFinderAnnotator and pin the second-sentence case, which was previously mapped silently wrong, with a test - Add DocumentAnnotators with requireLayers and the per-sentence token walk, replacing three copies of the walk loop and four spellings of the missing-layer rejection; direct tests pin the helpers as public API - Capture the document text as a String at construction so ImmutableDocument's immutability and thread-safety claims hold for mutable CharSequence inputs - Move the copy-on-add and threading narrative from the Document interface Javadoc to ImmutableDocument; the interface now states that thread safety is implementation specific - Carry the entity type as the annotation value only; entity spans are untyped, and the Javadoc names the value as the single source of the type - Make all six adapter annotators final before the types freeze - Align the TokenLengthAnnotator example with the documented required-layer contract in both the manual and the example test via requireLayers - Housekeeping per review: docbook CDATA placement, imports over qualified names, a ParameterizedTest for the blank-input matrix, shared deterministic test components, static assertion imports, inheritDoc on the runtime adapters, Layers constructor comment, and the documented NPE of StringUtil.isBlank
…ll rejection - Fold the three verbatim copies of the "Ana runs. Bob sits." document into a single twoSentenceDocument() helper in NameFinderAnnotatorTest, so the sentence and token layers of the shared fixture are declared once instead of drifting between the over-long mention, zero-length mention, and per-sentence offset tests - Hoist the no-op TokenNameFinder out of the blank-input test into a NO_NAMES constant in DocumentAnalyzerTest, since a finder that returns no spans is pipeline plumbing rather than part of any one test case, and document what it is for - Add testAnnotatorAdaptersRejectNullDocuments to pin that all four adapters reject a null document with the same "document must not be null" message, whether they validate directly or through DocumentAnnotators.requireLayers; the shared message was previously unpinned and free to drift per adapter - Trim the stale "person-free" qualifier from the New York comment in testTokenIndexSpansBecomeCharacterSpans; the finder emits a location mention and the extra negation described a distinction the test no longer draws
…, pin blank and span edge cases - ImmutableDocument: wrap the layer map unmodifiable at construction and expose its cached key set; split the combined null check so the message names the offending argument - StringUtil.isBlank javadoc: state how it differs from isUnicodeBlank - Tests: parameterize the isBlank accept and reject sides, pin the null NPE, and pin char-indexed spans over a supplementary-plane character
…quency, offsets) A new opennlp.tools.termvector package aggregates the document token layer into a document-scoped layer of TermVector records for index consumers, without touching the opennlp.tools.document container. TermVector carries the term string, the occurrence count, and the occurrence spans in original text coordinates. It comes in two shapes: full (one span per occurrence) and scoring-only (counts only, no offset storage). TermVectorAnnotator implements DocumentAnnotator: it requires Layers.TOKENS and provides its own opennlp:term-vectors key. Term identity is delegated, never analyzed: without a normalizer the token's covered text groups as-is; with an OffsetAwareNormalizer the document text is normalized once and each token span is mapped through the alignment, so tokens differing only by a normalization fold (case, eszett expansion, collapsed whitespace) group together while every emitted occurrence span still points into the original text.
…gation by mode - Delegate the two convenience constructors of TermVectorAnnotator through this(...) instead of repeating the field assignments, so the no-arg form is defined as FULL mode and the normalizer-only form as normalizer plus FULL. - Drop the requireMode and requireNormalizer helpers and do the null checks inline in the two canonical constructors, matching the argument-validation style used elsewhere in the module and keeping the thrown message next to the parameter it guards. - Split annotate into fullVectors and countVectors, one per Mode, so the scoring-only path no longer carries a null span map as a mode sentinel and the per-term branch inside the emit loop disappears. Each helper is documented and returns the annotations in first-occurrence order, which is the ordering the tests pin. - Make termOf an instance method, since it is now only reached from the two mode helpers and no longer needs to be static to be shared. - Tighten the TermVector class javadoc: the two shapes are told apart by whether spans() is empty, stated once, without the redundant aside about a flag, and the closing sentence now names the invariant instead of repeating the shape list. - Tighten the TermVectorAnnotator class javadoc the same way, and fix the termOf parameter doc that read "or null likewise" to spell out the condition. - Remove the javadoc references to AlignmentTest and DocumentPipelineExampleTest from the test fixtures. Those tests are not part of the contract under test here and the pointers go stale as soon as either file moves. - Add a DigitDeletingNormalizer fixture and a pinning test for a token the normalizer deletes entirely. It groups under the empty term rather than being dropped, which the class javadoc promises but nothing exercised. - Parameterize the frequency rejection test over 0, -1 and Integer.MIN_VALUE instead of only 0, so the guard is pinned across the whole illegal range. - Fix the span assertion in TermVectorPipelineTest. It compared Span.getCoveredText against an equivalent subSequence of the same text, which holds for any span, so it now asserts that the covered text equals the vector's own term.
…mple Adds a term vectors section to the document container chapter, citing TermVectorPipelineTest#testTokenizerAndTermVectorPipeline as the pin for the programlisting and covering the scoring-only mode.
The documented normalization workflow could not be built with any commonly wanted normalizer: TextNormalizer.Builder.buildAligned() rejects caseFold, nfc, nfkc, and accent folding, so the only OffsetAwareNormalizer chains the annotator accepted were the per-code-point folds. The restriction is unnecessary here because the occurrence spans the annotator emits are always the token's own span in the original text; the normalized text is used only as the term key. Add TermVectorAnnotator(CharSequenceNormalizer) and TermVectorAnnotator(CharSequenceNormalizer, Mode) as the general path: the normalizer is applied to each token's covered text to produce the term, the span stays the token's original span, and any normalizer works (case fold, NFC, accent fold, stemmer-backed). The OffsetAwareNormalizer constructors keep their whole-document aligned behavior unchanged; their javadoc now points at the plain-normalizer constructors as the general path. Tokens that normalize to the empty string still collapse into one empty term on both paths, pinned by matching tests. Red evidence: the new tests cannot compile against the old API (no suitable constructor found for TermVectorAnnotator(CharSequenceNormalizer)), so the tests and the fix land together in this commit. The negative pin that builder().caseFold().buildAligned() throws IllegalStateException already exists in AlignedNormalizerPipelineTest and is not duplicated.
…ted example Extend the term vector section of the manual with the plain-normalizer path: a whitespace tokenizer plus a shipped case folder built by TextNormalizer.builder().caseFold().build(), folding "Word word WORD" into one term with three occurrence spans that stay the tokens' original spans. The example lives in opennlp-runtime because the shipped folds do, and TermVectorNormalizedExampleTest#testCaseFoldedTermsKeepOriginalSpans asserts the behavior shown in the listing.
…nnotator, share the space-split test fixture, pin supplementary-plane offsets
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 16, 2026
krickert
added a commit
that referenced
this pull request
Aug 16, 2026
… term vector layer An empty string is no term: it cannot be queried and its token stays accounted for in the token layer. Omission also keeps one semantic for term vectors across the library and its search consumers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a document-scoped term vector layer for index consumers (OPENNLP-1897).
TermVector: one entry per distinct term, in two shapes: full (one span per occurrence, spans always in original text coordinates) or scoring-only (counts without offset storage). The shape invariant is validated.TermVectorAnnotator: rolls the token layer up into theopennlp:term-vectorslayer. Term identity is pluggable: covered text as-is, a plain per-tokenCharSequenceNormalizer(case/NFC/accent folds stemmer-backed normalizers), or anOffsetAwareNormalizerapplied to the whole document through its alignment. Tokens normalized to nothing group under the empty term rather than being dropped.TermVectorPipelineTestandTermVectorNormalizedExampleTest.Stacks on #1182 (OPENNLP-1888 document container); the shared foundation commits will drop out when that lands. We'll keep this in draft until #1182 merges.
This functionality is available in the sandbox branch for the gRPC server.
Consumer story: this is the aggregation a BM25/lexical index asks of the analysis chain; the gRPC server work (OPENNLP-1833) will expose the layer.
Thank you for contributing to Apache OpenNLP.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.