docs: split k0s deployment guide by connectivity - #156
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b794ced5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| scp tools/cluster_setup/k0s_cluster_with_stack.sh \ | ||
| tools/cluster_setup/install_from_airgap_bundle.sh \ | ||
| admin@install-machine:/opt/splunk-ai/ | ||
|
|
||
| scp tools/cluster_setup/my-cluster-config.yaml \ |
There was a problem hiding this comment.
Following this transfer command leaves /opt/splunk-ai without artifacts.yaml, although configure_images unconditionally copies the path from files.aiPlatform (which defaults to ./artifacts.yaml) before installation starts, so the documented offline command exits immediately. Transfer artifacts.yaml as well, and also splunk-operator-cluster.yaml when enabling the in-cluster Splunk mode described later in this guide, or include both manifests in the bundle.
Useful? React with 👍 / 👎.
| ./upload_to_minio.sh | ||
| # Or use upload_to_s3.sh or upload_to_seaweedfs.sh. |
There was a problem hiding this comment.
For any target other than the scripts' localhost defaults, these commands do not upload to the object store configured for the cluster: upload_to_minio.sh defaults to http://127.0.0.1:9000, a default bucket, and minioadmin credentials, while the S3 and SeaweedFS scripts likewise require target-specific environment variables. Since the scripts do not read my-cluster-config.yaml, the guide must show how to export the configured endpoint, bucket, credentials, and region before declaring the models staged and disabling installer staging.
Useful? React with 👍 / 👎.
| | Section | What to set | | ||
| |---|---| | ||
| | `cluster` | Cluster name, SSH private key path, and SSH user | | ||
| | `cluster.airgap` | Set to `false` | | ||
| | `nodes.existingIPs` | Controller and worker IP addresses | | ||
| | `storage.objectStore` | Storage type, endpoint, bucket, and credentials | | ||
| | `images.registry` | Registry hostname | | ||
| | `images.registryInsecure` | `true` only for a plain HTTP registry | | ||
| | `images` | Image names and tags supplied with your build | | ||
| | `aiPlatform.defaultAcceleratorType` | `L40S` or `H100` | | ||
| | `metallb.pool.addresses` | Unused LAN address range for LoadBalancer services | |
There was a problem hiding this comment.
When a user follows this configuration table, the copied template retains splunk.enabled: false; load_config therefore selects disabled mode and the installer skips both the Splunk Operator and Standalone. The later Splunk Web, administrator-secret, and app-installation steps cannot work despite being part of the guide's completion criteria. Both connectivity guides need to instruct users who will connect Splunk AI Assistant to set splunk.enabled: true (and distinguish the external-Splunk mode where applicable).
Useful? React with 👍 / 👎.
| The default service uses NodePort. Find the assigned port: | ||
|
|
||
| ```bash | ||
| kubectl get svc -n ai-platform \ | ||
| -l app.kubernetes.io/name=splunk | ||
| ``` |
There was a problem hiding this comment.
Even after splunk.enabled is corrected, the Standalone created by install_splunk_standalone has no serviceTemplate, so the Splunk Operator creates its normal ClusterIP service rather than a NodePort; aiPlatform.serviceTemplate only exposes SAIA. Consequently no assigned node port exists and http://<worker-node-ip>:<nodePort> is unreachable. Both guides should either configure the Standalone service as NodePort or use the port-forward flow that the installer itself prints.
Useful? React with 👍 / 👎.
| ```bash | ||
| kubectl get secret splunk-standalone-secret -n ai-platform \ | ||
| -o jsonpath='{.data.password}' | base64 --decode && echo |
There was a problem hiding this comment.
With the template's default standaloneName: splunk-standalone, the installer wires the operator-managed secret as splunk-splunk-standalone-standalone-secret-v1; it never creates splunk-standalone-secret. Thus this command returns NotFound and users cannot obtain the password needed by the next step. Update this command and its identical air-gapped counterpart to use the generated versioned name or discover the matching secret safely.
Useful? React with 👍 / 👎.
| kubectl get standalone splunk-standalone -n ai-platform -o json \ | ||
| | jq '.status.appContext.appSrcDeployStatus' | ||
| ``` | ||
|
|
||
| `deployStatus: 3` means the app is installed. |
There was a problem hiding this comment.
The preceding steps upload the archive through Splunk Web, but .status.appContext.appSrcDeployStatus is maintained by the Splunk Operator App Framework for packages discovered through the configured object-store appRepo; it does not report apps installed manually through the UI. In this documented flow the map can therefore be empty even after a successful installation, or contain statuses for unrelated object-store apps. Verify the app through Splunk's app list/REST API instead, and apply the same correction to the air-gapped guide.
Useful? React with 👍 / 👎.
| On the connected preparation machine, install: | ||
|
|
||
| - `curl` | ||
| - `crane` or another container-image copy tool | ||
| - `git` | ||
| - `helm` | ||
| - `tar` | ||
| - `sha256sum` or `shasum` |
There was a problem hiding this comment.
On macOS, ARM64, or any non-Linux-AMD64 preparation machine, the documented bundle build fails because prepare_airgap_bundle.sh always downloads the Linux AMD64 k0s binary and immediately executes it to run k0s airgap list-images and bundle-artifacts; it likewise bundles the Linux AMD64 yq binary. The prerequisites currently imply a generic machine and even accept macOS's shasum. State the required OS/architecture or make the bundle script select compatible host and target binaries.
Useful? React with 👍 / 👎.
| | Node type | Minimum CPU | Minimum RAM | Minimum disk | Count | | ||
| |---|---:|---:|---:|---:| | ||
| | Controller | 4 cores | 8 GB | 100 GB | 1, or 3 for high availability | | ||
| | CPU worker | 8 cores | 32 GB | 200 GB | 1 or more | | ||
| | GPU worker | 48 vCPUs | 384 GiB | 500 GB | 2 | |
There was a problem hiding this comment.
When three controller IPs are configured as this table permits, install_k0s_cluster installs k0s only on CONTROLLER_IPS[0]; the other controller IPs are merely prepared and are never joined as controllers. The installer nevertheless includes all three in its expected node count, so installation waits for nodes that cannot appear and no HA control plane is created. Document one controller until the installer actually provisions the additional controllers, or implement the advertised HA join flow.
Useful? React with 👍 / 👎.
| The minimum production topology is one controller, one CPU worker, and two GPU | ||
| workers. The controller and CPU worker can share one machine for lab testing, | ||
| but this configuration is not supported for production. One GPU worker is not | ||
| enough because AI inference is distributed across both workers. |
There was a problem hiding this comment.
With the two required GPU workers present, omitting a separate CPU-worker IP does not make the controller a CPU worker: label_nodes adds the CPU workload label to the controller only when WORKER_IPS is completely empty, so all CPU-selected platform workloads remain unschedulable. Listing the controller IP again as a worker is worse because the worker install loop removes /var/lib/k0s and the controller service on that host. Do not advertise this lab topology unless the installer can explicitly co-label the controller while other workers exist.
Useful? React with 👍 / 👎.
| ### 3. Stage the model weights | ||
|
|
||
| Run on a connected machine that can reach both Hugging Face and the target | ||
| object store. |
There was a problem hiding this comment.
In a fully separated air gap where the connected and disconnected environments have no network path—the scenario this guide acknowledges in the transfer section—the connected preparation machine cannot also reach the internal target object store as required here. The bundle explicitly excludes model weights, and the later transfer commands never copy model_artifacts, so such users have no way to satisfy modelStaging.enabled: false before installation. Document how to package and physically transfer the downloaded artifacts and then upload them from a machine inside the disconnected environment.
Useful? React with 👍 / 👎.
1 participant
Description
Related Issues
Type of Change
Changes Made
Testing Performed
make test)make lint)Test Environment
Test Steps
Documentation
Checklist
Breaking Changes
Impact:
Migration Path:
Screenshots/Recordings
Additional Notes
Reviewer Notes
Please pay special attention to:
Commit Message Convention: This PR follows Conventional Commits