Traefik/step1 config and daemonset - #149
Conversation
Adds SplunkConfigurationSpec.CACertRef (CABundleRef{Name,Namespace,Key})
so SAIA/SLIM pods can mount a caller-supplied CA bundle and set
REQUESTS_CA_BUNDLE/SSL_CERT_FILE for TLS-validated HEC/Splunk API calls
in external telemetry mode, with a checksum-based restart annotation on
CA rotation. Wires the same opt-in Splunk telemetry config block into
the k0s installer and its cluster-config, and adds a Traefik HTTPS
ingress design doc for follow-up implementation.
Covers caCertRef setup for both in-cluster Splunk (automated via the k0s installer's provision_splunk_cert) and external Splunk (manual CA secret + caCertSecretName), plus verification steps and the OpenShift/EKS installer gap where caCertRef must be set manually on the CR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The SAIA Service already conditionally exposes an https:8443 ServicePort when MTLS.Enabled && Termination=="operator" (reconcileSAIAService), but nginx only ever listened on 8080 with no TLS directives — so that Service port had no live listener behind it. Extract the shared routing/CORS location blocks into locationsConf and reuse them in a new conditional TLS server block on 8443, backed by the cert-manager Secret already mounted for mTLS. Also mount that Secret and add the matching container port in reconcileNginxDeployment. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…opology Broaden the target from k0s-on-EC2 specifically to any k0s infrastructure (cloud VM, bare metal, on-prem), calling out cloud-specific notes only where the underlying mechanism differs. Add Requirements (1a) and TLS/mTLS Topology and External Exposure (5a) sections.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f467c6d5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - name: ${splunkweb_svc} | ||
| port: 8000 |
There was a problem hiding this comment.
When the internal Standalone is installed, this commit enables Splunk Web SSL, so service port 8000 expects HTTPS. Traefik CRD services default scheme to http unless the port is 443 or the port name contains https (https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/service/#configuration-options), so this route forwards plaintext HTTP to Splunk Web's TLS listener and the advertised https://<worker>:8000 endpoint will fail. Add scheme: https plus appropriate backend trust/ServersTransport, or use TCP passthrough for this route.
Useful? React with 👍 / 👎.
| tls: | ||
| secretName: internal-domain-tls |
There was a problem hiding this comment.
This IngressRoute is applied in ${AI_NS}, but internal-domain-tls is created only in the ingress namespace. Traefik documents tls.secretName as a Secret in the same namespace as the IngressRoute (https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/ingressroute/#configuration-options), so SAIA and Splunk Web will not use the cert whose CA the installer tells users to trust and will fall back to a default/missing certificate. Create or copy the TLS secret into ${AI_NS}, or configure a Traefik default TLS store instead.
Useful? React with 👍 / 👎.
| effect: NoSchedule | ||
| containers: | ||
| - name: traefik | ||
| image: ${TRAEFIK_IMAGE} |
There was a problem hiding this comment.
For air-gapped or private-registry installs, TRAEFIK_IMAGE can point at a private image and the function does create pull secrets in the ingress namespace, but this DaemonSet never sets imagePullSecrets and the traefik ServiceAccount is not patched to use them. In that configuration the Traefik pods will hit ImagePullBackOff; either add the created secrets to the pod spec or patch the traefik ServiceAccount before deploying.
Useful? React with 👍 / 👎.
| tls: | ||
| passthrough: true |
There was a problem hiding this comment.
When management ingress is enabled, this passthrough route preserves Splunk's own certificate, but provision_splunk_cert only adds Splunk service DNS names and localhost to that cert. Clients following the printed https://<worker-ip-or-hostname>:8089 URL therefore validate it against the worker IP/hostname and hit a SAN mismatch. Add the ingress hostname/worker IP SANs to the Splunk server Certificate, or terminate/re-encrypt with a cert valid for the external host.
Useful? React with 👍 / 👎.
| key = "ca.crt" | ||
| } | ||
| secret := &corev1.Secret{} | ||
| if err := c.Get(ctx, types.NamespacedName{Name: ref.Name, Namespace: ai.Namespace}, secret); err != nil { |
There was a problem hiding this comment.
The CRD exposes caCertRef.namespace as if an explicit namespace can be selected, but the implementation always reads and mounts the Secret from the AIService namespace. If a user points at a CA Secret in another namespace, the checksum is omitted or taken from the wrong same-named Secret while the pod still mounts the wrong namespace's Secret. Validate that ref.Namespace is empty or equal to ai.Namespace, or copy the Secret into the workload namespace.
Useful? React with 👍 / 👎.
…CertRef into OTel TLS splunkConfiguration.endpoint (mgmt/JWKS, :8089) was being reused as the OTel sidecar's HEC ingestion URL, which silently misdelivers telemetry whenever HEC and management run on different ports. Add hecEndpoint (falls back to endpoint for backward compatibility) and update the k0s installer to set it. Also mount CACertRef into the OTel sidecar so it can verify Splunk's HEC cert instead of always skipping TLS verification. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ch, combined trust bundle - Reject caCertRef.namespace values that differ from the CR's own namespace in both AIPlatform and AIService webhooks; the Secret is always mounted same-namespace at runtime, so a mismatched value was silently ignored. - Watch CACertRef Secrets in both controllers so rotating the CA bundle's content in place (without renaming the Secret) triggers a reconcile, instead of relying on an unrelated trigger to pick up the checksum change. - Add a splunk-ca-merge initContainer to SAIA (v1, v2, v2-worker) and SLIM Deployments that concatenates the system trust store with the private CA onto a shared emptyDir; SSL_CERT_FILE/REQUESTS_CA_BUNDLE now point at the combined bundle instead of the private CA alone, which previously dropped all public CA trust for the process. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Certificate SANs and Splunk endpoint URLs hardcoded .svc.cluster.local, breaking clusters configured with a non-default clusterDomain. Add CLUSTER_DOMAIN (default cluster.local, overridable via kubernetes.clusterDomain in the config YAML) and use it everywhere the literal was previously inlined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
install_cert_manager() applied the stock v1.13.0 manifest with no --feature-gates flag, but the Splunk leaf Certificate's additionalOutputFormats: CombinedPEM (needed to produce tls-combined.pem for sslConfig.serverCert) is gated behind AdditionalCertificateOutputFormats, an alpha gate that defaults to off. Without it cert-manager silently never writes tls-combined.pem and splunkd fails to start. Patch the controller Deployment's args post-install since the raw manifest offers no values mechanism. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
helm-chart/splunk-ai-operator/crds/ and tools/cluster_setup/artifacts.yaml were missing caCertRef, hecEndpoint, and trustedIssuers — added to config/crd/bases/ai.splunk.com_aiplatforms.yaml by earlier AIP-4614 work but never propagated to these two separately-maintained copies. Helm chart synced via `make helm-sync`; artifacts.yaml (which no Makefile target touches) updated manually to match config/crd/bases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
buildSAIACABundleEnv/buildSlimCABundleEnv mounted the entire caCertRef
Secret via SecretVolumeSource{SecretName} with no key projection. When
caCertRef points at a leaf-cert Secret (e.g. the installer's
ai-splunk-server-tls, which also holds tls.key), Splunk's private key
was exposed in the SAIA/SLIM container filesystem alongside the CA cert
it actually needs. Project only the resolved CA key via Items so the
private key is never mounted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CACertRef.Key (a CRD-settable field) flowed unescaped into a fmt.Sprintf-built /bin/sh -c command string in buildSAIACABundleEnv and buildSlimCABundleEnv. A key value containing shell metacharacters could break out of the intended cat/cp invocation. Pass the resolved paths via env vars instead and use a static script string, so the key's contents are never re-parsed as shell syntax. Added regression tests covering a key containing quote/semicolon/comment metacharacters. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
WithEventFilter(...) populates controller-runtime's globalPredicates, which get ANDed onto every Watches()/Owns() source, not just the primary resource. That silently defeated the CACertRef Secret watch added for AIP-4614 Tier 1 item 4: Secrets never bump metadata.generation, and a pure .data rotation touches neither annotations nor labels, so the global Or(GenerationChanged, AnnotationChanged, LabelChanged) filter evaluated false and killed the event before SecretChangedPredicate ever ran. Rotating a CA bundle's content in place (without renaming the Secret) never triggered a reconcile, despite the code's own comments claiming otherwise. Move the primary-resource-scoped predicate into For()'s own WithPredicates instead, which only applies to that one watch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
renderOtelConf defaulted the splunk_hec exporter's TLS config to insecure_skip_verify: true whenever CACertRef was unset, silently accepting any certificate. Default to false instead (verify against the collector image's system trust store), matching how SAIA/SLIM already behave without a CACertRef — correct for a publicly trusted HEC cert, and fails loudly instead of silently accepting anything when a private CA is actually needed. Also fix the OTel collector's splunk-ca volume mount, which had the same whole-Secret-mount issue fixed for SAIA/SLIM in a prior commit: CACertRef may point at a leaf-cert Secret containing tls.key, so project only the resolved CA key instead of mounting the entire Secret. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tion renderOtelConf silently fell back from hecEndpoint to the management/JWKS endpoint when hecEndpoint was unset, pointing the OTel exporter at the wrong port whenever the two listeners differ. Now emits a warning Event on fallback, and errors out if neither field is set instead of shipping telemetry to an empty URL. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MTLSConfig.Enabled only makes nginx present a server certificate; there is no ssl_verify_client directive anywhere in the codebase, so no client certificate is ever requested or validated. Update the field's doc comments (and regenerate the CRDs/helm chart/artifacts.yaml copies) so the CRD schema itself no longer implies real mTLS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Secure CA projection and rotation handling, fail closed for OTel TLS, separate management and HEC endpoints, remove the CombinedPEM dependency, and synchronize generated CRDs, installers, documentation, and regression tests.
Companion guide to TRAEFIK_HTTPS_DESIGN.md, aimed at end users enabling ingress.enabled — covers config, firewall prerequisites (infra-agnostic), cert trust import, multi-worker IP guidance, FIPS, airgap, and rollback.
Adds Traefik as a hostPort DaemonSet fronting SAIA (:8443), Splunk Web (:8000), and Splunk mgmt (:8089, TCP passthrough), backed by a cert-manager self-signed cert chain with SANs covering every worker IP. Gated behind ingress.enabled (default false) so existing NodePort/tunnel access is unaffected. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
7f467c6 to
5567a19
Compare
1 participant
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits