[Demo] Testdaten für Deployment automatisiert bereitstellen

[adoLime]

User Story

As a Developer, I want representative test data on the SprintStart deployment, so that demos can run against a non-empty system.

Context & Motivation

Our Client demoed SprintStart on Monday (Arena2036, 14:00) and asked for non-empty demo data. After uni-cluster/reset-dev.sh the DB is empty; no re-seed mechanism exists today.

In Scope

• Bring representative test data onto the dev deployment (see Acceptance Criteria for the concrete shape).
• Preferred path: automated, idempotent seeder
• Fallback: if automation does not work out in time, bring the data up manually (create users/projects, assign roles, connect GitHub org, trigger onboarding seeding).

Acceptance Criteria

• At least 2 PM users exist in the system.
• At least 1 of the PMs is assigned to at least 2 projects.
• In each of that PM's projects: at least 2 non-PM users with an onboarding path.
• In each of those projects: at least 1 user with a completed onboarding path (all steps done).
• At least 3 users exist without any project assignment.
• At least one GitHub organization is connected and "Discover GitHub repositories" returns its repos in the UI.
• Demo artifacts are ingested (e.g. sprintstart-ai/tests/rag/demo-corpus/*.md) and visible in the knowledge base.
• A chat use case returns an answer with at least one cited source.
• Approach is documented (automated seeder and/or manual steps), so the demo can be reproduced after a reset.

[LinseCed]

Automated seeder is ready for review: SprintStartProject/uni-cluster#1

Approach: uni-cluster/seed-demo.py, an idempotent script that drives the public HTTP APIs rather than inserting SQL. That choice matters — the Chroma vector store sits on its own PVC, separate from PostgreSQL, so seeding via SQL would leave the knowledge base looking populated while chat returned answers with no citations. Going through the real endpoints means ingestion, embedding and AI path generation happen exactly as they do for a human user.

Verified against a full reset-dev.sh → seed cycle on the dev deployment. All eight machine-checkable criteria pass (./seed-demo.py --verify-only):

[x] at least 2 PM users  (2 found)
[x] a PM assigned to >= 2 projects  (1 such PM(s))
[x] 2+ non-PM users with a path per project
      SprintStart Platform: 2 with path, 1 completed
      Gamify-IT: 2 with path, 1 completed
[x] 1+ completed path per project
[x] at least 3 users without a project  (4 found)
[x] GitHub org connected, discovery works  (7 repos)
[x] demo artifacts in the knowledge base  (5 uploads)
[x] a chat answer cites at least one source  (523 citation(s))

The ninth criterion (documented approach) is covered by a new README section in the same PR.

One thing the issue didn't anticipate: reset-dev.sh wipes Keycloak as well. Keycloak stores its users in a keycloak database on the same PostgreSQL StatefulSet whose PVC the reset deletes, and the realm re-imports with an empty user list — so after a reset there is no account to log in with at all. The seeder recreates the Keycloak accounts too, and the README now documents this.

Two problems found while doing this:

• [Bug] GitHub org discovery 500s when two users connected the same repository sprintstart-backend#165 — "Discover GitHub repositories" returns 500 once two different users have connected the same repository. Invisible on a fresh environment, permanent once it happens. Directly affects the acceptance criterion about repo discovery.
• Raise AI service memory limit to 1Gi on dev sprintstart-k8s#8 (merged) — the AI service was OOMKilled at its 512Mi limit while embedding a freshly connected org, which also killed concurrent onboarding path generation. Dev now runs at 1Gi. This is not seeding-specific: a PM connecting a large org through the UI during a demo could trigger the same thing.

Also worth flagging: the GITHUB_PAT stored in the sprintstart-backend-env secret is expired (returns 401). The seeder takes a token from the environment instead, but that secret value should be rotated or removed.

To reproduce the demo after any future reset:

./reset-dev.sh
# wait for all five pods to report 1/1
export SEED_GITHUB_PAT=$(cat ~/gh)
./seed-demo.py

[LinseCed]

Ran a second, fully clean reset-dev.sh → seed cycle from an empty environment to confirm reproducibility. All eight machine-checkable criteria pass in a single run (exit code 0):

[x] at least 2 PM users  (2 found)
[x] a PM assigned to >= 2 projects  (1 such PM(s))
[x] 2+ non-PM users with a path per project
      SprintStart Platform: 2 with path, 1 completed
      Gamify-IT: 2 with path, 1 completed
[x] 1+ completed path per project
[x] at least 3 users without a project  (4 found)
[x] GitHub org connected, discovery works  (7 repos)
[x] demo artifacts in the knowledge base  (5 uploads)
[x] a chat answer cites at least one source  (889 citation(s))

Caveat worth recording: the run is only clean because the seeder retries. The AI service was OOMKilled twice during the cycle — once during ingestion and once on the path generation right after it — and each time an onboarding path generation died with it. The retry logic absorbed both. Filed as SprintStartProject/sprintstart-ai#148 with the measurements; raising the dev limit to 1Gi (SprintStartProject/sprintstart-k8s#8) reduced but did not eliminate it.

So the demo is reproducible today, but a re-seed shortly before a demo should be given time to retry rather than being run at the last minute.

[LinseCed]

Final state — correcting my earlier comment, which reported a clean cycle on evidence that turned out to be incomplete.

Current state

All criteria pass, and the environment has been verified against the failure modes that were previously hiding behind them:

[x] at least 2 PM users  (2 found)
[x] a PM assigned to >= 2 projects  (1 such PM(s))
[x] 2+ non-PM users with a path per project
      SprintStart Platform: 2 with path, 1 completed
      Gamify-IT: 2 with path, 1 completed
[x] 1+ completed path per project
[x] at least 3 users without a project  (3 found)
[x] GitHub org connected, discovery works  (7 repos)
[x] demo artifacts in the knowledge base  (5 uploads)
[x] every project has artifacts  (all populated)
[x] a chat answer cites at least one source  (3202 citation(s))
[x] all ingestion runs reached the AI index  (all synced)

Project	Artifacts
SprintStart Platform	3660
Gamify-IT	102
Arena2036 Pilot	5

All five pods healthy, no restarts.

What my earlier "all criteria green" report missed

Two criteria were passing while the system was substantially broken, because they counted rows rather than checking the thing that mattered:

1. The knowledge base was 88% unindexed. Ingestion reports COMPLETED once artifacts are stored locally, which says nothing about whether they reached the AI index. Three of four repositories had failed or stuck AI syncs, leaving 3212 of 3659 artifacts invisible to chat. Chat still answered and still cited sources — from the remaining 447 — so the criterion passed.

2. Nothing was visible in the UI. All sources were attached to one project, but the project switcher auto-selects sortProjects()[0]. demo.admin landed on an empty Arena2036 Pilot and pm.nowak on an empty Gamify-IT, while the data sat in a third project. This was found by a human opening the admin account, not by the verification.

Both are now criteria in their own right (all ingestion runs reached the AI index, every project has artifacts), so they fail the run instead of being discovered in the UI.

Infrastructure changes this required

• Raise AI service memory limit to 1Gi on dev sprintstart-k8s#8, Individual Reports should state issue IDs (in the GH project) + dev, test or docu work #9 — AI service 512Mi → 1Gi → 4Gi. Path generation scales with indexed corpus size and needed ~1.2GB; it only appeared to fit earlier because most of the corpus was missing from the index.
• Raise backend memory limit to 1Gi on dev sprintstart-k8s#10 — backend 512Mi → 1Gi. It was OOMKilled mid-chat, surfacing as a 502. It now sits at ~614Mi, i.e. above the old limit.

Open issues

• [Bug] GitHub org discovery 500s when two users connected the same repository sprintstart-backend#165 — more severe than first reported. connect/all is not idempotent, so a single user connecting the same repository twice permanently 500s org discovery for everyone. No disconnect endpoint exists, so recovery means deleting rows across four tables by hand. Worth fixing before anyone touches the connect screen.
• AI service OOMKilled under bulk ingestion, and on the path generation that follows it sprintstart-ai#148 — AI service memory behaviour under ingestion and path generation. Mitigated by headroom, not fixed.
• The GITHUB_PAT in sprintstart-backend-env is expired (401) and should be rotated or removed.

Caveat on reproducibility

The seeder is idempotent and re-runnable, but the full reset-then-seed path has not been re-run end to end since the last few fixes landed. The current environment was reached by seeding, then repairing. A re-seed before a demo should be given time to retry, not run at the last minute.

[LinseCed]

Reproducibility confirmed. Ran a full reset-dev.sh → seed cycle from an empty environment with all fixes in place.

This one was clean in the strict sense — not just "criteria pass", but no failures along the way:

• 0 pod restarts across all five pods over the 35-minute run
• 0 retries, 0 errors in the seeder log (earlier cycles absorbed 2–3 AI OOM kills at this point)
• 8 of 8 ingestion runs SUCCEEDED
• 0 duplicate repository connections
• All four onboarding paths generated on the first attempt — the previous cycle failed 3 of 3
• Exit code 0

[x] at least 2 PM users  (2 found)
[x] a PM assigned to >= 2 projects  (1 such PM(s))
[x] 2+ non-PM users with a path per project
      SprintStart Platform: 2 with path, 1 completed
      Gamify-IT: 2 with path, 1 completed
[x] 1+ completed path per project
[x] at least 3 users without a project  (3 found)
[x] GitHub org connected, discovery works  (7 repos)
[x] demo artifacts in the knowledge base  (5 uploads)
[x] every project has artifacts  (all populated)
[x] a chat answer cites at least one source  (2256 citation(s))
[x] all ingestion runs reached the AI index  (all synced)

Project	Artifacts	Sources
SprintStart Platform	3660	4 SprintStartProject repos + demo corpus
Gamify-IT	102	2 Gamify-IT repos
Arena2036 Pilot	5	demo corpus

Every account now lands on a populated project: demo.admin → Arena2036 Pilot, pm.nowak → Gamify-IT, learners → SprintStart Platform.

To reproduce:

./reset-dev.sh
# wait for all five pods to report 1/1
export SEED_GITHUB_PAT=$(cat ~/gh)
./seed-demo.py        # ~35 min

Remaining open items are unchanged: SprintStartProject/sprintstart-backend#165 (connect is not idempotent — worth fixing before anyone uses the connect screen), SprintStartProject/sprintstart-ai#148 (AI memory, mitigated by headroom only), the expired GITHUB_PAT in sprintstart-backend-env, and the fact that prod still runs the AI service and backend at the base 512Mi limits that proved insufficient on dev.