Skip to content

feat(connectors): add OpenSearch sink connector - #3873

Open
mattp5657 wants to merge 2 commits into
apache:masterfrom
mattp5657:feat/connectors-opensearch-sink
Open

feat(connectors): add OpenSearch sink connector#3873
mattp5657 wants to merge 2 commits into
apache:masterfrom
mattp5657:feat/connectors-opensearch-sink

Conversation

@mattp5657

@mattp5657 mattp5657 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Closes #3504

Rationale

Iggy connectors ship sinks for several external systems but not for OpenSearch, a widely used search/analytics backend. This adds one, modeled on the existing elasticsearch_sink shape but closing a retry gap that sink still has (see Known trade-offs).

What changed?

Adds core/connectors/sinks/opensearch_sink/, following the sink lifecycle end to end:

  • open(): validates config (URL shape, credential pairing, document_id_field constraints), then retry_on_open-wraps a cluster health check, an index-exists check, and, when create_index_if_not_exists (default true), index creation with an optional custom mapping. Capped at max_open_retries with exponential backoff and jitter (shared retry helpers).
  • consume(): batches incoming messages into batch_size chunks, builds an iggy_*-enriched document per message (hashed or field-derived _id), and hands each chunk to index_chunk.
  • index_chunk(): POSTs a _bulk request and loops up to max_retries with the same backoff. _bulk answers 200 even when individual documents fail, so each response is parsed per item rather than trusting the top-level status: permanent failures (4xx, e.g. a mapping conflict) are recorded immediately, while transient ones (429/5xx) shrink the pending set to just the rejected documents and get resent, so a partial rejection under load doesn't re-index or lose the rest of the chunk. Counts from earlier attempts are merged into the outcome so a later failure doesn't erase already-indexed documents from the tally.
  • close(): drops the client, no special teardown.

Integration tests (core/integration/tests/connectors/opensearch/) run against a real container (testcontainers-modules, reused across tests via ReuseDirective::Always, per-test-unique index names for isolation), covering the happy path plus a static mapping conflict, a missing index with index-creation disabled, and confirming a failing chunk doesn't block chunks queued behind it.

Credentials: HTTP Basic auth only (username/password, both-or-neither validated at config time); password is a SecretString, never logged or serialized. AWS SigV4 (AWS-managed OpenSearch / Serverless) is not supported.

Known trade-offs, deliberately out of scope here, verified against current master:

  • elasticsearch_sink has the same gap this PR fixes for OpenSearch: bulk_index_documents (elasticsearch_sink/src/lib.rs:205-219) tallies per-item _bulk failures into errors_count but never retries the transient subset (e.g. 429 es_rejected_execution_exception). Worth a follow-up issue rather than folding into this PR.
  • The connectors runtime discards a sink's consume() return value: core/connectors/runtime/src/sink.rs:740-748 invokes the FFI consume callback as a bare statement, never binding its i32 result, so process_messages always returns Ok. Combined with offsets auto-committing at poll time (sink.rs:522), a plugin-level failure never reaches connector status, last_error, or /stats, and the batch is never redelivered. Pre-existing, repo-wide, affects every sink.
  • meilisearch_sink (lib.rs:451-454) and elasticsearch_sink (lib.rs:319-328) both silently drop iggy_headers/_iggy_headers: BTreeMap<HeaderKey, HeaderValue> can't serialize as a JSON object (serde_json requires string keys), and both sinks swallow that error via if let Ok(...) instead of surfacing it. core/common even ships a serialize_headers workaround for this exact case that neither sink uses.
  • Single-node transport: like elasticsearch_source, the client is built on SingleNodeConnectionPool (lib.rs:208) with no cluster sniffing or multi-node failover. A dead configured node fails every request rather than routing around it.

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

  1. Tools: Claude
  2. Use Used for scaffolding and iteration.
  3. Verification: Full local compilation, extensive integration testing against a live container.
  4. Reviewed and can explain every line if asked.

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 12, 2026
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.27972% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.98%. Comparing base (fb9307e) to head (6f078e2).

Files with missing lines Patch % Lines
core/connectors/sinks/opensearch_sink/src/lib.rs 93.27% 82 Missing and 35 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3873      +/-   ##
============================================
- Coverage     82.84%   81.98%   -0.86%     
  Complexity     1299     1299              
============================================
  Files          1199     1200       +1     
  Lines        161885   163626    +1741     
  Branches     131360   133206    +1846     
============================================
+ Hits         134120   134156      +36     
- Misses        24225    25831    +1606     
- Partials       3540     3639      +99     
Components Coverage Δ
Rust Core 83.42% <93.27%> (+0.06%) ⬆️
Java SDK 66.06% <ø> (ø)
C# SDK 56.95% <ø> (-19.02%) ⬇️
Python SDK 89.98% <ø> (ø)
PHP SDK 84.26% <ø> (ø)
Node SDK 96.21% <ø> (ø)
Go SDK 68.53% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sinks/opensearch_sink/src/lib.rs 93.27% <93.27%> (ø)

... and 94 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mattp5657

mattp5657 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Getting some errors with a 503 returned when they passed locally, for example for Typo. Will try running again later:

  Connecting to github.com (github.com)|140.82.112.4|:443... connected.
  HTTP request sent, awaiting response... 503 Service Unavailable
  2026-08-12 19:30:09 ERROR 503: Service Unavailable.

Update: These seem to have resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(connector): Add Opensearch sink connector

1 participant