Skip to content

Field notes from a real deploy: stale :latest redeploys, full-URL imports, runner caveats #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
schlunsen wants to merge 1 commit into from
Closed

Field notes from a real deploy: stale :latest redeploys, full-URL imports, runner caveats #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 40 additions & 3 deletions SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,10 @@ jobs:
docker push ${{ env.IMAGE }}:latest
docker push ${{ env.IMAGE }}:${{ github.sha }}

# Only include this step if the runner has the k3s binary (host-mode runners).
# Containerized runners fail here with exit 127 "k3s: command not found" —
# in that case drop this step and pin the manifest image to the commit-SHA
# tag instead (see "Redeploy" below).
- name: Import into k3s
run: |
docker save ${{ env.IMAGE }}:latest | k3s ctr images import --all-platforms -
Expand All @@ -544,7 +548,7 @@ jobs:
- Both workflows use `docker login ${{ vars.REGISTRY }}` with org-level secrets `REGISTRY_USER` and `REGISTRY_PASSWORD` to authenticate with this workspace's Gitea container registry. The `REGISTRY` **variable** plus these two **secrets** are auto-provisioned at the org level for every workspace, so any new repo inherits them automatically — never hardcode the registry hostname
- Workflow B uses `actions/checkout@v4` for cloning — do NOT use manual `git clone` with hardcoded runner-internal URLs
- Two tags are pushed: `latest` (for the manifest) and either the commit SHA (for rollback) or the upstream tag (for version tracking)
- **Workflow B must include `docker save | k3s ctr images import`** — this imports the image directly into k3s containerd, ensuring it's available for pod scheduling even before skopeo mirroring runs
- **Workflow B should include `docker save | k3s ctr images import` when the runner supports it** — this imports the image directly into k3s containerd, ensuring it's available for pod scheduling even before skopeo mirroring runs. **Not all workspaces have host-mode runners**: on containerized runners the step fails with exit 127 (`k3s: command not found`). Check the runner first, or just rely on commit-SHA image pinning in the manifest (see the Redeploy section) which works everywhere
- **Always detect the repo's default branch** — do not hardcode `main`. Common alternatives: `master`, `canary`, `develop`
- The available runner labels are: `self-hosted` (host mode, has Docker), `ubuntu-latest` (containerized via `node:20-bookworm`), `ubuntu-22.04` (containerized). Use `self-hosted` for any workflow that needs Docker

Expand Down Expand Up @@ -1361,7 +1365,7 @@ Before finalizing, verify:
- [ ] **Upstream images use the full registry path** (e.g., `ghcr.io/org/app:stable`) — NOT just the image name
- [ ] `.gitignore` excludes `node_modules/`, `.env`, `dist/`, etc.
- [ ] `.gitea/workflows/build-and-push.yml` exists (build from source OR mirror upstream)
- [ ] **Workflow includes `docker save | k3s ctr images import` step** (ensures image is available in k3s containerd)
- [ ] **Workflow includes `docker save | k3s ctr images import` step if the runner has k3s** (containerized runners don't — omit the step and pin the manifest image to the commit-SHA tag instead)
- [ ] **Workflow uses a hardcoded lowercase `IMAGE` env var** — NEVER use `${{ github.repository }}` in Docker tags (it preserves uppercase and Docker rejects it)
- [ ] **Workflow branch trigger matches the repo's actual default branch** (not hardcoded `main`)
- [ ] Database services have `volumes` for data persistence
Expand Down Expand Up @@ -1470,6 +1474,13 @@ This fetches `n0-app.json` from the repo, validates it, and creates an `AppDefin
- Short form: `org/repo` (uses workspace's own Gitea)
- Full URL: `https://gitea-clovrlabs.apps.privateprompt.tech/org/repo`

**Prefer the full URL.** On some deployments (observed on prod `app.nzero.pro`) the
short form fails with a misleading `No n0-app.json found in org/repo` error even when
the manifest is present on the default branch; the full-URL form works immediately.

Re-run this same call after changing `n0-app.json` (e.g. bumping a pinned image tag) —
it updates the existing AppDefinition in place (`"created": false` in the response).
Comment on lines +1477 to +1482

### Step 2: Deploy App Instance

```bash
Expand Down Expand Up @@ -1498,6 +1509,23 @@ curl -s -X POST "https://app.privateprompt.tech/api/v1/workspaces/${WS_ID}/apps/
-H "Authorization: Bearer $TOKEN"
```

**WARNING — redeploy does NOT pick up a new `:latest` build by itself.** k3s containerd
caches the image by tag: if the pod's image reference is unchanged (`...:latest`), the
node reuses the cached image and the app keeps serving the old build. A `{"tag": "..."}`
body on the redeploy call is accepted but ignored. Reliable update flow when the CI
workflow cannot import into k3s directly:

1. Push code → CI builds and pushes both `:latest` and `:{commit-sha}` tags
2. Pin the image in `n0-app.json` to the full commit-SHA tag
(`localhost:5000/org/repo:<sha>`) and push
3. Re-import the definition (`POST .../apps/definitions/` with the full repo URL)
Comment on lines +1518 to +1521
4. Redeploy — the changed image reference forces a fresh pull

Verify what's actually deployed by temporarily setting the app public
(`PUT .../apps/{APP_ID}/access` with `{"access_level": "public"}`), curling a content
marker from the site, then reverting to `workspace`. Note: access-level changes only
take effect at the Caddy layer after a redeploy.

### Other Operations

```bash
Expand Down Expand Up @@ -1537,6 +1565,9 @@ Both the loopback address (`127.0.0.1:{port}`) and external hostname (`gitea-{sl
| `Unknown app type` | The `app_type` doesn't match any built-in or AppDefinition slug | Check the slug in n0-app.json |
| Image pull failure | Image not in registry or wrong tag | Check Gitea Actions build succeeded; verify `REGISTRY_USER`/`REGISTRY_PASSWORD` org secrets |
| Build push failure | Stale or missing registry credentials | Re-provision org-level Actions secrets (see below) |
| `No n0-app.json found` on import despite manifest in repo root | Short-form `org/repo` repo_url not resolved on this deployment | Use the full Gitea URL as `repo_url` |
| App serves old version after redeploy | k3s containerd cached the unchanged `:latest` tag | Pin the manifest image to the commit-SHA tag, re-import definition, redeploy |
| Actions run stuck "in_progress" then marked "failure", but image was pushed | Runner's final status report to Gitea timed out — bookkeeping only | Trust the job log (`/actions/runs/{run}/jobs` → `/actions/jobs/{id}/logs`): if it ends with "Job succeeded" and shows push digests, the build is fine |

### Troubleshooting Registry Credentials

Expand Down Expand Up @@ -1669,7 +1700,13 @@ curl -s "$API/workspaces/" \
```

**Gitea personal access token** (for git push):
- Visit `https://gitea-{workspace-slug}.apps.{platform-domain}/-/user/settings/applications`
- **Easiest — mint one via the N0 API** using your platform token (PAT or JWT):
```bash
curl -s -X POST "$API/workspaces/$WS_ID/gitea/token/" \
-H "Authorization: Bearer $N0_TOKEN" -H "Content-Type: application/json" -d '{}'
# → {"token": "...", "username": "...", "gitea_url": "https://gitea-{slug}.apps.{domain}"}
```
- Or visit `https://gitea-{workspace-slug}.apps.{platform-domain}/-/user/settings/applications`
- Click "Generate New Token" → select `write:repository, write:package` scopes → copy token
- Or via Gitea API (if you already have a token with `write:user` scope):
```bash
Expand Down