Migrating to Single Model Deployment Config and allowing replica count to be configurable through cluster config - #130
Draft
vavarshn wants to merge 91 commits into
Draft
Migrating to Single Model Deployment Config and allowing replica count to be configurable through cluster config#130vavarshn wants to merge 91 commits into
vavarshn wants to merge 91 commits into
Conversation
feat(AIP-3938): AI tier openshift support
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
feat(AIP-4163): Ai pod script enhancements
feat(AIP-4161): Openshift air gap install script
Bring ai-tier-ga in sync with main
… script issues (#126) * feat(openshift): route GPU/CPU workers via ai-tier-node label * resolve codex comments
chore: Merge ai-pod-feature-branch to ai-tier-ga
Creates EC2 infrastructure consumed by k0s_cluster_with_stack.sh: - VPC + public subnet + IGW in a single AZ - Security group with self-referencing rule (private-IP k0s comms) - RHEL 9 instances: controller(s), CPU workers, GPU workers, installer - Installer gets an EIP; k0s nodes stay on private IPs only - Separate EBS volumes for /var/lib/k0s (GPU) and /data/minio (installer) - Auto-creates or reuses AWS key pair, downloads .pem locally - Optional MinIO install via install_minio_ec2.sh after provisioning - Generates my-k0s-config.yaml on the installer with private IPs pre-filled - Commands: provision / output / status / destroy / validate / dry-run - Test config (t3.medium, ~$1-2) for Level 2 validation - Design + test plan in K0S_AWS_PROVISION.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…mount fix
Rewrites CloudFormation-based provisioner to direct AWS CLI calls,
bypassing SCP p-m68tib3s which blocks ec2:RunInstances via CloudFormation.
Key changes:
- IMDSv2 required (HttpTokens=required) on all instances — required by SCP
- EBS encryption on all volumes — required by SCP
- Existing VPC only (vpc-09b191e89c83d588e) — no new VPC creation
- Separate subnets: k0s nodes in private subnet, installer in public subnet with EIP
- SSH ProxyCommand instead of -J flag (fixes key forwarding on macOS)
- mount_disk_via_ssh: mount EBS data disks via SSH after attach, not UserData
(UserData runs before EBS volumes are attached by the provisioner)
- destroy --yes / -y flag for non-interactive teardown
- Fix _INSTANCE grep pattern to include INSTALLER_INSTANCE (was missing)
- Fix ${var,,} bash 3.2 incompatibility with tr [:upper:] [:lower:]
Validated end-to-end: provision → status → output → destroy all pass.
All 4 instances, EBS volumes, EIP, SG, key pair created and destroyed cleanly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on steps, failure history Updates the provisioner doc to reflect the rewrite from CloudFormation to direct AWS CLI calls. Adds: - SCP compliance table (IMDSv2, EBS encryption, existing VPC requirement) - Corrected architecture diagram (public vs private subnet split) - Full k0s integration guide (Option A from installer, Option B from laptop + ProxyJump) - Development history: 9 documented failures with root causes and fixes (SCP block, VPC, set -e false negative, private subnet EIP, SSH -J key, EBS mount timing, INSTALLER_INSTANCE grep, destroy TTY, bash 3.2 compat) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s-west-2 VPC When network.vpcId is empty (or omitted in non-us-west-2 regions), the provisioner now creates the full network stack automatically: VPC → IGW → public subnet + route (0.0.0.0/0 → IGW) → NAT GW EIP → NAT GW → private subnet + route (0.0.0.0/0 → NAT GW) All six resources are tagged and tracked in the state file so destroy tears them down cleanly in reverse order (private RT → private subnet → NAT GW → NAT EIP → public RT → public subnet → IGW → VPC). For us-west-2, the default vpcId remains vpc-09b191e89c83d588e (SCP in splunkcloud-ai-dev blocks new-VPC creation; leaving the default avoids breaking existing deployments). Any other region defaults to auto-create. Config additions: network.vpcCidr (default 10.0.0.0/16) network.publicSubnetCidr (default 10.0.1.0/24) network.privateSubnetCidr (default 10.0.2.0/24) Docs updated with Network Modes table, mandatory-fields list, and full config reference with inline comments for each network field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… to not create AWS resources - Add WARNING comments to ensure_network and destroy_network (untested code path) - Emit runtime warn() when auto-create mode activates - dry-run: skip pick_subnet in auto-create mode (no VPC exists yet) - dry-run: remove ensure_key_pair call (was creating real key pairs in AWS) Dry-run now creates zero AWS resources in both existing-VPC and auto-create modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…writing
Old behaviour: cat > my-k0s-config.yaml (full overwrite with hardcoded defaults)
New behaviour:
1. If my-k0s-config.yaml exists on installer → back it up as
my-k0s-config.bak-<timestamp>.yaml, then yq-patch in-place.
2. If it doesn't exist → copy k0s-cluster-config.yaml (already
uploaded by setup_installer) as the base, then yq-patch.
Only infrastructure fields are written:
cluster.{name,region,sshKeyPath,sshUser}
nodes.existingIPs.{controllers,workers}
storage.objectStore.{type,bucket,endpoint,auth.*} (when minio.enabled)
All other fields (images, operators, aiPlatform, metallb, ecr, etc.)
are preserved from the user's own file.
IPs are serialised via jq to a proper JSON array before yq injection,
avoiding quoting issues with multi-node configs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add TrustedIssuers []string to SplunkConfigurationSpec so operators can supply a list of JWT issuer URLs for Splunk authentication. INTERNAL mode auto-prepends the in-cluster issuer and appends the list; EXTERNAL and DISABLED modes use the list as-is. Updates SAIA reconciler (impl.go) to derive SPLUNK_ISSUERS from the field and adds unit tests covering all three modes and the override vs fill-if-missing behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(): force delete stuck terminating namespaces on node failure
…ation - P1: Always recompute SPLUNK_ISSUERS from spec (not fill-if-missing) so that removing CRRef/endpoint/trustedIssuers clears stale issuer trust. - P1: Use spec.splunkConfiguration.endpoint as the JWT issuer for direct- endpoint installs (k0s/EKS installer sets endpoint, not CRRef), preventing empty SPLUNK_ISSUERS in the default in-cluster deployment. - P2: Derive the in-cluster issuer FQDN from SplunkCustomResourceRef.Name and Kind (same logic as endpoint_resolver.go) instead of a hardcoded name. - P2: Add sha256 checksum annotation on SAIA deployment pod template so that issuer changes trigger a rollout without a manual pod restart. - P2: Sync trustedIssuers field into Helm chart CRDs for AIService and AIPlatform so clusters installed from tracked release manifests accept the new field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…alidateSplunkConfiguration signature Three pre-existing test failures due to the feat/splunk-operator-optional changes: - raybuilder: OTel sidecar annotations now require Splunk to be configured; update both head and worker test cases to include a Splunk endpoint and add a negative case asserting no injection when Splunk is absent. - webhook: validateSplunkConfiguration gained an otelEnabled bool param in PR #124; update all call sites in aiplatform_webhook_test.go and vault_path_validation_test.go to pass false (no OTel in these unit tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SAIA's saia_authorizer.py splits SPLUNK_ISSUERS on comma, but the operator was joining with a space, causing every issuer after the first to be unrecognised and all JWT validation to fail with 401. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
feat(splunk): add TrustedIssuers field and SAIA issuer config
… warn on mutable image tags configure_images() snapshotted artifacts.yaml/splunk-operator-cluster.yaml to a .original file on first run and restored from it on every subsequent run before re-injecting image values. Since .original was never refreshed, any legitimate change to those manifests (new operator release, new env var, new sidecar) was silently reverted on every re-install after the first. Drop the snapshot/restore step; the sed substitutions already only touch their own named fields, so rewriting in place is idempotent. Also warn in validate_image_config() when a configured image has no tag or uses a mutable tag (:latest, :preview, :stable*, :dev, :nightly), since imagePullPolicy: IfNotPresent means re-running install without a new distinct tag will not actually upgrade the running image.
The image: substitutions for the AI operator and Splunk operator matched by content (*splunk*ai*operator*, *splunk*operator*) against the image string itself. This only matched the manifest's pristine default image; once a custom image (e.g. a private registry with no such substring) was written in on a prior run, a later run with a new tag would silently fail to match, leaving the stale image in place. This was previously masked by a since-removed snapshot/restore step that reset the manifest to pristine before each run. Anchor both substitutions on the unique env entry that always immediately precedes the target image: line (RAY_VERSION for artifacts.yaml, POD_NAME for splunk-operator-cluster.yaml) instead of matching image content. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three follow-up issues from PR review on the prior idempotency fix: - Restore the tools/cluster_setup/*.original .gitignore rule (narrowed comment): eks_cluster_with_stack.sh still snapshots/restores .original backups, so deleting the rule left those EKS byproducts untracked in every checkout even though k0s_cluster_with_stack.sh no longer needs it. - Parse the tag from the last path segment only in the mutable-tag check, not from the whole image string. A registry port (e.g. localhost:5000/team/saia-api) was previously read as the tag, so untagged images behind a ported registry silently skipped the no-tag warning. - Extend the mutable-tag check to cover images.fluentBit.image and images.nginx.image (always patched via RELATED_IMAGE_FLUENT_BIT/ RELATED_IMAGE_NGINX) and images.splunk.image (patched via RELATED_IMAGE_SPLUNK_ENTERPRISE, guarded by SPLUNK_MODE=internal since it's only deployed in that mode) — these were patched into the manifest but never checked for the same same-tag-rerun hazard. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
configure_images() patches RELATED_IMAGE_OTEL_COLLECTOR into the manifest, but images.otelCollector.image was missing from the mutable-tag validation list, so an untagged or mutable-tagged OTel collector image silently skipped the same-tag-rerun warning. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
configure_images() patches images.splunk.operatorImage into splunk-operator-cluster.yaml (via the POD_NAME-anchored image: sed) in internal mode, but the mutable-tag validation loop never checked SPLUNK_OPERATOR_IMAGE — so a tagless or mutable-tagged operator image was validated without warning. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
vavarshn
force-pushed
the
vavarshn/deployment_design_changes
branch
from
July 21, 2026 11:17
71d14c1 to
6dc7632
Compare
* docs(AIP-4503): Openshift customer onboarding doc * resolve codex comments
Add two unit-test suites to test_k0s_cluster_with_stack.sh: - configure_images upgrade idempotency: reproduces the exact regression fixed in b07745c — a custom private-registry image containing none of "splunk"/"ai"/"operator" must be correctly overwritten on a second ("upgrade") run, not left stale. - validate_image_config mutable-tag warnings: covers the registry-port tag-parsing fix and every RELATED_IMAGE_* field configure_images patches (operator, ray, weaviate, saia x3, fluentBit, nginx, otelCollector, splunk + splunk-operator under internal mode), so any future field added to one list without the other gets caught. Track test_installer_dry_run.sh (previously untracked) and add a new cluster-setup-tests.yml workflow that runs both bash test suites, wired into main.yml alongside unit-tests. Neither suite needs a cluster, kubectl, or network access. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ternal mode Splunk's oauth2_settings.issuer_uri (the JWT "iss" claim on every interactive token) was hardcoded to a literal string decoupled from splunkConfiguration.endpoint, which feeds SPLUNK_ISSUERS on saia/slim via buildSplunkIssuersVal. The two also differed in scheme (http vs https) and service name (missing the Splunk Operator's own "splunk-" prefix), so CMP auth rejected every real token with "Issuer not allowed"/"Malformed JWT" and AI Tier's refresh_models silently returned no models. Both values now derive from AI_STANDALONE_NAME/AI_NS and match byte-for-byte in the https://splunk-<name>-standalone-service.<ns>.svc.<domain>:8089 form already used by the operator's SplunkCustomResourceRef path. Verified live: minted a real interactive JWT and confirmed slim-api's chat/models endpoint now returns models end-to-end. Co-Authored-By: Claude <noreply@anthropic.com>
- Split the metrics port into a dedicated internal-only ClusterIP Service
(reconcileSlimMetricsService) so NodePort/LoadBalancer public exposure
never publishes /metrics or allocates an extra NodePort for it. The new
Service carries the "component" label so the existing ServiceMonitor
selector (app+component) actually matches it.
- Fix reconcileSlimService to set Type/Ports/Selector inside the
CreateOrUpdate mutate callback instead of before it, since CreateOrUpdate
reloads the live Service first and was discarding the templated NodePort
on both create and update.
- Honor AIPlatformScheme when synthesizing AIPlatformUrl instead of
hardcoding http://, so HTTPS-only Ray endpoints work.
- Populate slim.yaml's instanceScale for all GPU tiers so slim-only
AIPlatforms (saia disabled) actually get GPU worker replicas to schedule
the FmTimeseries model on.
- Guard the SAIA application blocks in applications.yaml with
{{- if .Replicas.X }} so slim-only platforms don't render SAIA templates
referencing replica counts that were never set (avoiding <no value>
Go-template output in the rendered Ray Serve config).
Co-Authored-By: Claude <noreply@anthropic.com>
artifacts.yaml's CRD schema still only allowed "saia" or "seca" for features[].name, so applying an AIPlatform/AIService CR with the slim feature enabled was rejected by API server validation. Add "slim" to both enums. Co-Authored-By: Claude <noreply@anthropic.com>
fix(cluster_setup): stop reverting artifacts.yaml on repeat installs,…
Point MODEL_SCALE_FILE / WORKER_SCALE_FILE at the repo config files in the controller test setup, matching INSTANCE_FILE / APPLICATION_FILE, so the builder can load model-scale.yaml / worker-scale.yaml under envtest. Co-Authored-By: Claude <noreply@anthropic.com>
vavarshn
force-pushed
the
vavarshn/deployment_design_changes
branch
from
July 27, 2026 04:35
75e10b5 to
28cfc39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits