feat(AIP-4614): add Splunk TLS CA bundle support for k0s installer - #148
feat(AIP-4614): add Splunk TLS CA bundle support for k0s installer#148kupratyu-splunk wants to merge 16 commits into
Conversation
Adds SplunkConfigurationSpec.CACertRef (CABundleRef{Name,Namespace,Key})
so SAIA/SLIM pods can mount a caller-supplied CA bundle and set
REQUESTS_CA_BUNDLE/SSL_CERT_FILE for TLS-validated HEC/Splunk API calls
in external telemetry mode, with a checksum-based restart annotation on
CA rotation. Wires the same opt-in Splunk telemetry config block into
the k0s installer and its cluster-config, and adds a Traefik HTTPS
ingress design doc for follow-up implementation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cfb01e1045
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // system trust store — always true for the installer's self-signed internal | ||
| // chain, and common for external Splunk behind a private/internal CA. | ||
| // +kubebuilder:validation:Optional | ||
| CACertRef *CABundleRef `json:"caCertRef,omitempty"` |
There was a problem hiding this comment.
Adding the Go type and base CRDs is not enough for the install paths that apply packaged manifests: the default k0s script still applies tools/cluster_setup/artifacts.yaml (SPLUNK_AI_FILE defaults there), and rg caCertRef tools/cluster_setup/artifacts.yaml helm-chart/splunk-ai-operator/crds finds no schema entry for this field. With those CRDs installed, the caCertRef emitted by the installer or Helm values is pruned/rejected before the reconciler sees it, so internal/private-CA Splunk installs still deploy SAIA/SLIM without the CA bundle.
Useful? React with 👍 / 👎.
| // Namespace is the Secret's namespace. Defaults to the AIPlatform/AIService's | ||
| // own namespace when omitted. | ||
| // +kubebuilder:validation:Optional | ||
| Namespace string `json:"namespace,omitempty"` |
There was a problem hiding this comment.
This field is exposed as the Secret namespace and documented as defaulting only when omitted, but both CA volume creation and checksum lookup ignore it and always use the AIService namespace. If a customer follows the API and points at a CA Secret in a shared/security namespace, the generated pods try to mount a same-named Secret from the service namespace and the checksum is computed from the wrong location; either validate that the namespace is empty/local or implement a supported copy/reference flow.
Useful? React with 👍 / 👎.
| name: ${SPLUNK_EXTERNAL_SECRET_NAME} | ||
| namespace: ${AI_NS} | ||
| ${trusted_issuers_yaml} | ||
| ${external_ca_cert_yaml}${trusted_issuers_yaml} |
There was a problem hiding this comment.
When external mode sets both splunk.external.caCertSecretName and splunk.trustedIssuers, external_ca_cert_yaml comes from command substitution, which strips its trailing newline, and this concatenation puts trustedIssuers: on the same line as key: ca.crt. That renders invalid YAML for private-CA external Splunk installs that also need explicit issuers, so kubectl apply fails before the AIPlatform is created.
Useful? React with 👍 / 👎.
| duration: 2160h # 90d leaf lifetime — pinned explicitly (Part G.1) | ||
| renewBefore: 720h # reissue 30d before expiry |
There was a problem hiding this comment.
The internal Splunk leaf certificate now expires after 90 days and is renewed 30 days before expiry, but the installer never changes the Standalone pod template or otherwise restarts Splunk when ai-splunk-server-tls is updated; the comment above also notes splunkd will not pick up the new leaf without a pod restart. Internal k0s clusters left running through the first rotation can keep serving the old certificate/JWKS key until manual intervention, breaking SAIA/SLIM TLS or token validation after expiry.
Useful? React with 👍 / 👎.
Covers caCertRef setup for both in-cluster Splunk (automated via the k0s installer's provision_splunk_cert) and external Splunk (manual CA secret + caCertSecretName), plus verification steps and the OpenShift/EKS installer gap where caCertRef must be set manually on the CR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The SAIA Service already conditionally exposes an https:8443 ServicePort when MTLS.Enabled && Termination=="operator" (reconcileSAIAService), but nginx only ever listened on 8080 with no TLS directives — so that Service port had no live listener behind it. Extract the shared routing/CORS location blocks into locationsConf and reuse them in a new conditional TLS server block on 8443, backed by the cert-manager Secret already mounted for mTLS. Also mount that Secret and add the matching container port in reconcileNginxDeployment. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…opology Broaden the target from k0s-on-EC2 specifically to any k0s infrastructure (cloud VM, bare metal, on-prem), calling out cloud-specific notes only where the underlying mechanism differs. Add Requirements (1a) and TLS/mTLS Topology and External Exposure (5a) sections.
…CertRef into OTel TLS splunkConfiguration.endpoint (mgmt/JWKS, :8089) was being reused as the OTel sidecar's HEC ingestion URL, which silently misdelivers telemetry whenever HEC and management run on different ports. Add hecEndpoint (falls back to endpoint for backward compatibility) and update the k0s installer to set it. Also mount CACertRef into the OTel sidecar so it can verify Splunk's HEC cert instead of always skipping TLS verification. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ch, combined trust bundle - Reject caCertRef.namespace values that differ from the CR's own namespace in both AIPlatform and AIService webhooks; the Secret is always mounted same-namespace at runtime, so a mismatched value was silently ignored. - Watch CACertRef Secrets in both controllers so rotating the CA bundle's content in place (without renaming the Secret) triggers a reconcile, instead of relying on an unrelated trigger to pick up the checksum change. - Add a splunk-ca-merge initContainer to SAIA (v1, v2, v2-worker) and SLIM Deployments that concatenates the system trust store with the private CA onto a shared emptyDir; SSL_CERT_FILE/REQUESTS_CA_BUNDLE now point at the combined bundle instead of the private CA alone, which previously dropped all public CA trust for the process. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Certificate SANs and Splunk endpoint URLs hardcoded .svc.cluster.local, breaking clusters configured with a non-default clusterDomain. Add CLUSTER_DOMAIN (default cluster.local, overridable via kubernetes.clusterDomain in the config YAML) and use it everywhere the literal was previously inlined. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
install_cert_manager() applied the stock v1.13.0 manifest with no --feature-gates flag, but the Splunk leaf Certificate's additionalOutputFormats: CombinedPEM (needed to produce tls-combined.pem for sslConfig.serverCert) is gated behind AdditionalCertificateOutputFormats, an alpha gate that defaults to off. Without it cert-manager silently never writes tls-combined.pem and splunkd fails to start. Patch the controller Deployment's args post-install since the raw manifest offers no values mechanism. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
helm-chart/splunk-ai-operator/crds/ and tools/cluster_setup/artifacts.yaml were missing caCertRef, hecEndpoint, and trustedIssuers — added to config/crd/bases/ai.splunk.com_aiplatforms.yaml by earlier AIP-4614 work but never propagated to these two separately-maintained copies. Helm chart synced via `make helm-sync`; artifacts.yaml (which no Makefile target touches) updated manually to match config/crd/bases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
buildSAIACABundleEnv/buildSlimCABundleEnv mounted the entire caCertRef
Secret via SecretVolumeSource{SecretName} with no key projection. When
caCertRef points at a leaf-cert Secret (e.g. the installer's
ai-splunk-server-tls, which also holds tls.key), Splunk's private key
was exposed in the SAIA/SLIM container filesystem alongside the CA cert
it actually needs. Project only the resolved CA key via Items so the
private key is never mounted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CACertRef.Key (a CRD-settable field) flowed unescaped into a fmt.Sprintf-built /bin/sh -c command string in buildSAIACABundleEnv and buildSlimCABundleEnv. A key value containing shell metacharacters could break out of the intended cat/cp invocation. Pass the resolved paths via env vars instead and use a static script string, so the key's contents are never re-parsed as shell syntax. Added regression tests covering a key containing quote/semicolon/comment metacharacters. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
WithEventFilter(...) populates controller-runtime's globalPredicates, which get ANDed onto every Watches()/Owns() source, not just the primary resource. That silently defeated the CACertRef Secret watch added for AIP-4614 Tier 1 item 4: Secrets never bump metadata.generation, and a pure .data rotation touches neither annotations nor labels, so the global Or(GenerationChanged, AnnotationChanged, LabelChanged) filter evaluated false and killed the event before SecretChangedPredicate ever ran. Rotating a CA bundle's content in place (without renaming the Secret) never triggered a reconcile, despite the code's own comments claiming otherwise. Move the primary-resource-scoped predicate into For()'s own WithPredicates instead, which only applies to that one watch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
renderOtelConf defaulted the splunk_hec exporter's TLS config to insecure_skip_verify: true whenever CACertRef was unset, silently accepting any certificate. Default to false instead (verify against the collector image's system trust store), matching how SAIA/SLIM already behave without a CACertRef — correct for a publicly trusted HEC cert, and fails loudly instead of silently accepting anything when a private CA is actually needed. Also fix the OTel collector's splunk-ca volume mount, which had the same whole-Secret-mount issue fixed for SAIA/SLIM in a prior commit: CACertRef may point at a leaf-cert Secret containing tls.key, so project only the resolved CA key instead of mounting the entire Secret. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tion renderOtelConf silently fell back from hecEndpoint to the management/JWKS endpoint when hecEndpoint was unset, pointing the OTel exporter at the wrong port whenever the two listeners differ. Now emits a warning Event on fallback, and errors out if neither field is set instead of shipping telemetry to an empty URL. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MTLSConfig.Enabled only makes nginx present a server certificate; there is no ssl_verify_client directive anywhere in the codebase, so no client certificate is ever requested or validated. Update the field's doc comments (and regenerate the CRDs/helm chart/artifacts.yaml copies) so the CRD schema itself no longer implies real mTLS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Secure CA projection and rotation handling, fail closed for OTel TLS, separate management and HEC endpoints, remove the CombinedPEM dependency, and synchronize generated CRDs, installers, documentation, and regression tests.
1 participant
https://splunk.atlassian.net/browse/VULN-78636: Lack of In-Cluster TLS for Service-to-Service Communication
Adds SplunkConfigurationSpec.CACertRef (CABundleRef{Name,Namespace,Key}) so SAIA/SLIM pods can mount a caller-supplied CA bundle and set REQUESTS_CA_BUNDLE/SSL_CERT_FILE for TLS-validated HEC/Splunk API calls in external telemetry mode, with a checksum-based restart annotation on CA rotation. Wires the same opt-in Splunk telemetry config block into the k0s installer and its cluster-config, and adds a Traefik HTTPS ingress design doc for follow-up implementation.
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits