Skip to content

ref(api): Rework how replicator resources are done#883

Merged
iambriccardo merged 7 commits into
mainfrom
improve-bigquery
Jul 8, 2026
Merged

ref(api): Rework how replicator resources are done#883
iambriccardo merged 7 commits into
mainfrom
improve-bigquery

Conversation

@iambriccardo

@iambriccardo iambriccardo commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Rework ETL API Kubernetes resource defaults so k8s.replicator_resources uses the clearer DefaultReplicatorResourcesConfig type and requires only baseline replicator CPU/memory requests.
  • Keep pipeline replicator_resources fully optional with ReplicatorResourcesConfig: pipelines can override CPU/memory requests and/or explicit CPU/memory limits, while omitted limits are derived from the final request values using the static ETL API multipliers.
  • Compute full Kubernetes resource settings through ReplicatorStatefulSetResourcesConfig, using API defaults plus optional pipeline overrides for the replicator container and static constants for the vector container; final requests are clamped to the API's minimum emitted CPU/memory values and limits are never below their paired requests.
  • Require source connection settings, including TLS settings, to be present in ETL API configuration instead of relying on implicit source defaults.
  • Validate default replicator resource requests immediately after the API config is loaded during ETL API startup, keeping application construction free of config validation plumbing.
  • Document the default-vs-pipeline override behavior in the ETL API README and align skip_serializing_if = "Option::is_none" usage across related ETL API config/response types.
  • Simplify BigQuery BIGNUMERIC scale validation by reading the parsed numeric scale directly instead of formatting the value as a string.

Validation

  • cargo x fmt
  • cargo check -p etl-api --all-targets --all-features
  • cargo nextest run -p etl-api --all-features -E 'test(replicator_stateful_set_resources) | test(pipeline_replicator_resources_are_persisted_and_used_on_start) | test(updating_a_running_pipeline_reapplies_replicator_resources) | test(updating_a_stopped_pipeline_only_persists_replicator_resources) | test(invalid_replicator_resources_are_rejected) | test(stored_pipeline_config_serialization) | test(stored_pipeline_config_deserializes_without_replicator_resources)'
  • cargo nextest run -p etl-api --all-features -E 'test(test_create_ducklake_replicator_stateful_set_json) | test(test_create_bq_replicator_stateful_set_json) | test(test_create_iceberg_replicator_stateful_set_json) | test(test_create_init_containers)'
  • cargo clippy -p etl-api --all-targets --all-features -- -D warnings

Comment thread crates/etl-destinations/src/bigquery/validation.rs
@iambriccardo iambriccardo marked this pull request as ready for review July 8, 2026 06:34
@iambriccardo iambriccardo requested a review from a team as a code owner July 8, 2026 06:34
@blacksmith-sh

This comment has been minimized.

@coveralls

coveralls commented Jul 8, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 72.775% (+0.05%) from 72.721% — improve-bigquery into main

@iambriccardo iambriccardo enabled auto-merge (squash) July 8, 2026 09:07
@iambriccardo iambriccardo merged commit 3bee6bf into main Jul 8, 2026
22 checks passed
@iambriccardo iambriccardo deleted the improve-bigquery branch July 8, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants