You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #5932 made the application tier's object storage configurable via a storage.s3 block, but left the two data-tier components still hardwired to the in-cluster MinIO:
LakeFS blockstore (dataset versioning) points at {{ .Release.Name }}-minio.
Lakekeeper Iceberg REST catalog warehouse is initialized against MinIO, and its init job unconditionally creates the bucket and emits an s3-compat storage profile (custom endpoint + path-style access).
There is also no single switch to drop the bundled MinIO entirely, and no worked example of an all-external-S3 deployment. As a result a fully external-S3 install (e.g. AWS S3 on EKS) is not yet possible with the chart. This is the LakeFS/Lakekeeper half of "make object storage pluggable" that #5932 explicitly deferred.
Proposed Solution or Design
Finish the external-S3 story so an operator can run Texera with no in-cluster MinIO at all, backing the app tier, LakeFS, and Lakekeeper by an external S3 store:
minio.enabled switch (values.yaml, default true) gating the bundled MinIO subchart (Chart.yaml dependency condition), its persistence template, and the MinIO ExternalName alias exposed to the computing-unit pool namespace. Default install renders identically.
LakeFS external blockstore: keep the in-cluster MinIO blockstore as the default; document overriding lakefsConfig to a region-only S3 blockstore with credentials injected via extraEnvVars for external S3.
Lakekeeper init job: route STORAGE_S3_* through the shared texera.s3.* helpers (added in feat(k8s): make app-tier object storage configurable via storage.s3 #5932); add lakekeeperInit.createBucket (skip mc mb for pre-existing external buckets whose IAM principal lacks CreateBucket); add warehouse.flavor (s3-compat vs native aws, the latter omitting endpoint + path-style access) and an optional warehouse.keyPrefix to isolate the warehouse within a shared bucket.
values-aws.yaml example overlay: a complete, copy-pasteable all-external-S3 configuration (MinIO off, storage.s3, LakeFS S3 blockstore, Lakekeeper flavor: aws + createBucket: false).
Non-breaking: with no overrides the chart renders the same set of resources as today.
Feature Summary
PR #5932 made the application tier's object storage configurable via a
storage.s3block, but left the two data-tier components still hardwired to the in-cluster MinIO:{{ .Release.Name }}-minio.s3-compatstorage profile (custom endpoint + path-style access).There is also no single switch to drop the bundled MinIO entirely, and no worked example of an all-external-S3 deployment. As a result a fully external-S3 install (e.g. AWS S3 on EKS) is not yet possible with the chart. This is the LakeFS/Lakekeeper half of "make object storage pluggable" that #5932 explicitly deferred.
Proposed Solution or Design
Finish the external-S3 story so an operator can run Texera with no in-cluster MinIO at all, backing the app tier, LakeFS, and Lakekeeper by an external S3 store:
minio.enabledswitch (values.yaml, defaulttrue) gating the bundled MinIO subchart (Chart.yamldependencycondition), its persistence template, and the MinIOExternalNamealias exposed to the computing-unit pool namespace. Default install renders identically.lakefsConfigto a region-only S3 blockstore with credentials injected viaextraEnvVarsfor external S3.STORAGE_S3_*through the sharedtexera.s3.*helpers (added in feat(k8s): make app-tier object storage configurable via storage.s3 #5932); addlakekeeperInit.createBucket(skipmc mbfor pre-existing external buckets whose IAM principal lacksCreateBucket); addwarehouse.flavor(s3-compatvs nativeaws, the latter omitting endpoint + path-style access) and an optionalwarehouse.keyPrefixto isolate the warehouse within a shared bucket.values-aws.yamlexample overlay: a complete, copy-pasteable all-external-S3 configuration (MinIO off,storage.s3, LakeFS S3 blockstore, Lakekeeperflavor: aws+createBucket: false).Non-breaking: with no overrides the chart renders the same set of resources as today.
Affected Area
Deployment / Infrastructure, Storage / Metadata