feat(connectors): add OpenSearch sink connector - #3873
Open
mattp5657 wants to merge 2 commits into
Open
Conversation
|
Thanks for the PR. It is labeled Slash commands (own line, regular comment) move it around the queue:
See CONTRIBUTING.md for details. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3873 +/- ##
============================================
- Coverage 82.84% 81.98% -0.86%
Complexity 1299 1299
============================================
Files 1199 1200 +1
Lines 161885 163626 +1741
Branches 131360 133206 +1846
============================================
+ Hits 134120 134156 +36
- Misses 24225 25831 +1606
- Partials 3540 3639 +99
🚀 New features to boost your workflow:
|
Contributor
Author
|
Getting some errors with a 503 returned when they passed locally, for example for Typo. Will try running again later: Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2026-08-12 19:30:09 ERROR 503: Service Unavailable.Update: These seem to have resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR address?
Closes #3504
Rationale
Iggy connectors ship sinks for several external systems but not for OpenSearch, a widely used search/analytics backend. This adds one, modeled on the existing
elasticsearch_sinkshape but closing a retry gap that sink still has (see Known trade-offs).What changed?
Adds
core/connectors/sinks/opensearch_sink/, following the sink lifecycle end to end:open(): validates config (URL shape, credential pairing,document_id_fieldconstraints), thenretry_on_open-wraps a cluster health check, an index-exists check, and, whencreate_index_if_not_exists(defaulttrue), index creation with an optional custom mapping. Capped atmax_open_retrieswith exponential backoff and jitter (sharedretryhelpers).consume(): batches incoming messages intobatch_sizechunks, builds aniggy_*-enriched document per message (hashed or field-derived_id), and hands each chunk toindex_chunk.index_chunk(): POSTs a_bulkrequest and loops up tomax_retrieswith the same backoff._bulkanswers200even when individual documents fail, so each response is parsed per item rather than trusting the top-level status: permanent failures (4xx, e.g. a mapping conflict) are recorded immediately, while transient ones (429/5xx) shrink the pending set to just the rejected documents and get resent, so a partial rejection under load doesn't re-index or lose the rest of the chunk. Counts from earlier attempts are merged into the outcome so a later failure doesn't erase already-indexed documents from the tally.close(): drops the client, no special teardown.Integration tests (
core/integration/tests/connectors/opensearch/) run against a real container (testcontainers-modules, reused across tests viaReuseDirective::Always, per-test-unique index names for isolation), covering the happy path plus a static mapping conflict, a missing index with index-creation disabled, and confirming a failing chunk doesn't block chunks queued behind it.Credentials: HTTP Basic auth only (
username/password, both-or-neither validated at config time);passwordis aSecretString, never logged or serialized. AWS SigV4 (AWS-managed OpenSearch / Serverless) is not supported.Known trade-offs, deliberately out of scope here, verified against current
master:elasticsearch_sinkhas the same gap this PR fixes for OpenSearch:bulk_index_documents(elasticsearch_sink/src/lib.rs:205-219) tallies per-item_bulkfailures intoerrors_countbut never retries the transient subset (e.g. 429es_rejected_execution_exception). Worth a follow-up issue rather than folding into this PR.consume()return value:core/connectors/runtime/src/sink.rs:740-748invokes the FFIconsumecallback as a bare statement, never binding itsi32result, soprocess_messagesalways returnsOk. Combined with offsets auto-committing at poll time (sink.rs:522), a plugin-level failure never reaches connector status,last_error, or/stats, and the batch is never redelivered. Pre-existing, repo-wide, affects every sink.meilisearch_sink(lib.rs:451-454) andelasticsearch_sink(lib.rs:319-328) both silently dropiggy_headers/_iggy_headers:BTreeMap<HeaderKey, HeaderValue>can't serialize as a JSON object (serde_jsonrequires string keys), and both sinks swallow that error viaif let Ok(...)instead of surfacing it.core/commoneven ships aserialize_headersworkaround for this exact case that neither sink uses.elasticsearch_source, the client is built onSingleNodeConnectionPool(lib.rs:208) with no cluster sniffing or multi-node failover. A dead configured node fails every request rather than routing around it.Local Execution
AI Usage