From d04df365049635d3a8e8f25a3598af5983aab447 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 11:46:34 +0500 Subject: [PATCH 01/36] docs: record app engineering handbook scope --- docs/WORK_LOG.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 74175b7..1771d88 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,25 @@ claims as completed work. ## Current status +## App engineering handbook — 2026-09-13 + +- **Outcome:** create a complete printable PDF explaining the app from beginner + to advanced level, with layer diagrams, daily selection examples, catalog + exhaustion/refill, notifications, email, credentials by purpose, release flow, + tradeoffs, and a seven-day study digest. +- **Scope:** documentation and a reproducible PDF source only; no application, + production configuration, content generation, or messaging changes. +- **Branch/base:** `codex/app-engineering-handbook` from freshly fetched + `origin/develop` at `3cc5af3`. GitHub read-only checks confirm #191 and #193 + merged; `main` is `2e537f6`. Older release status below is historical. +- **Planned commits:** scope; source-based handbook; PDF builder and navigation; + validation and PR handoff. Generated PDFs/previews remain untracked outputs. +- **Evidence:** inspect current implementation before prose, verify relevant + provider documentation, distinguish code/defaults from live service settings, + and omit credential values. Render and inspect every final PDF page. + +## Previous release handoff (historical) + - [Release PR #191](https://github.com/Coding-Moves/one-concept/pull/191) is open from **develop → main** for **1.8.0**, with the six-benefit one-time card and runtime **1.3.0**. Feature/fix PRs #183–#186, #188, and #189 are included; From 15fd0a341bbce4cffc6c9d7dcbb03c8662cbedb1 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 11:54:55 +0500 Subject: [PATCH 02/36] docs: explain app architecture and complete learning lifecycle --- docs/handbook/ONE_CONCEPT_HANDBOOK.md | 974 ++++++++++++++++++++++++++ 1 file changed, 974 insertions(+) create mode 100644 docs/handbook/ONE_CONCEPT_HANDBOOK.md diff --git a/docs/handbook/ONE_CONCEPT_HANDBOOK.md b/docs/handbook/ONE_CONCEPT_HANDBOOK.md new file mode 100644 index 0000000..0a17bec --- /dev/null +++ b/docs/handbook/ONE_CONCEPT_HANDBOOK.md @@ -0,0 +1,974 @@ +# 01 | Your app in twelve answers + +One Concept is a daily microlearning app: a signed-in learner receives one short technical concept, reads its explanation and example, and can mark it learned, like it, save it, or share it. Topic follows shape future assignments. The backend is the authority for assignments and learning progress. + +| Your question | What the inspected code actually does | +| --- | --- | +| Does everyone get the same lesson today? | No shared lesson of the day. Each account receives its own assignment; two accounts can coincidentally receive the same concept. | +| Is selection random? | Partly. It prefers a least recently assigned followed topic, then randomly chooses an eligible concept among the tied candidates. | +| Can my two phones get different daily lessons? | The database permits only one assignment for your account and local date. Both read the stored winner. | +| Is one topic a 25-lesson course? | No. 25 is the default shared published-catalog target for the scheduled generator. It is not a course length or personal allowance. | +| Does reading use up a lesson for everyone? | No. Published content stays in the shared catalog. Each user's assignment history is separate. | +| What happens after my followed topics run out? | The API tries the wider published catalog, excluding anything previously assigned to you. | +| What if I exhaust the entire catalog? | There is no repeat assignment. The app shows an exhausted state until eligible new content exists. | +| Does Gemini answer every app open? | No. It writes stored lessons through background generation paths. Daily requests select database content. | +| What sends learning reminders? | A backend reminder worker calls Expo Push; delivery continues through the platform push service. | +| What sends account emails? | Supabase Auth, through the sender configured in its dashboard. These are separate from learning reminders. | +| Is there a weekly learning digest email? | No implementation was found. This handbook includes a weekly study plan for you. | +| Is this already released? | GitHub confirms v1.8.0 and a successful release workflow. Live device adoption and Railway runtime settings were not checked. | + +**Read carefully:** an assigned lesson is treated as consumed by the no-repeat selector even when the learner never marks it learned. Opening the card is also not proof of comprehension. Several current UI phrases say "read" or "learned" more broadly than the backend rule. + +Sources: `backend/app/services/selection.py`; `backend/app/config.py`; `mobile/src/screens/TodayScreen.tsx`; `backend/app/services/reminders.py`. + +# 02 | The complete system, layer by layer + +Think of the system as a shared library with personal reading records. The phone is the reading desk, FastAPI is the librarian, PostgreSQL stores the shelves and records, and Gemini helps write new library content. Supabase Auth supplies identity. GitHub and Expo deliver software updates rather than daily lesson text. + + + +| Layer | Responsibility | Where it runs | +| --- | --- | --- | +| Presentation | Screens, cards, navigation, theme, accessibility | The user's phone; web build for development/validation | +| Client state | Cached content, optimistic actions, offline queue | React contexts and device storage | +| Identity | Signup, password sign-in, sessions, account emails | Supabase Auth plus its mobile client | +| Application API | Verify identity; select lessons; write progress | FastAPI/Uvicorn, packaged in Docker for Railway | +| Data | Catalog, follows, assignments, interactions, job records | Supabase PostgreSQL | +| Background work | Generate catalog content and send due reminders | Python workers; API event-loop prefetch | +| Delivery and operations | Git history, reviews, audits, OTA and APK builds | GitHub Actions, EAS, Railway configuration | + +**The key boundary:** the mobile app talks directly to Supabase for authentication, and to FastAPI for application data. Gemini and database passwords belong behind the backend boundary. No Firestore, custom payment service, Redis queue, or in-app AI chat appears in the inspected application. + +Sources: `mobile/App.tsx`; `mobile/src/lib/supabase.ts`; `mobile/src/api/client.ts`; `backend/app/main.py`; `backend/Dockerfile`. + +# 03 | Product vocabulary and learning behavior + +The word "card" can mean three different things in this app. Separating them makes the rest of the system much easier to understand. + +| Term | Meaning | Does it create another daily lesson? | +| --- | --- | --- | +| Topic/category | A subject such as Computer Science or Mathematics | Following it influences a future unassigned day. | +| Concept/lesson | One catalog row: title, summary, example, topic | It can become an eligible daily assignment. | +| Today's card | The presentation of your assigned concept | Reopening or completing it does not advance to a second daily concept. | +| Assignment | Link between a user, concept and local calendar date | Created on first successful daily/state request for that date. | +| Learned | Assignment with a completion timestamp | Counts toward history and streaks. | +| Like | Personal appreciation, also contributing to a public count | Does not change selection ranking. | +| Save/bookmark | A personal reading collection | Supports revisiting; it is independent of learned status. | +| What's New card | Release highlights bundled with the app version | It announces software improvements, not course content. | + +The seeded topics are Artificial Intelligence, Software Engineering, Computer Science, Mathematics, and Linux & Systems. The initial database migration seeds 20 written concepts; a later migration provides 150 curated backlog titles. A backlog title is a writing task, not a published lesson. These are repository seed counts, not a live inventory of production content. + +**The learning loop:** choose interests, open the app, read a short explanation and concrete example, mark the day learned, and revisit useful ideas in Saved. This builds a small daily habit. The code does not currently assess understanding with quizzes, enforce prerequisites, adapt difficulty to performance, or schedule spaced-repetition reviews. "Learned" is self-reported completion. + +**No chapter progression:** topic rotation is about variety. The selector does not promise lesson 1 before lesson 2, even though a stored difficulty field exists. Someone following only AI receives AI while eligible AI content remains; someone following five subjects receives a mixture. + +Sources: `backend/migrations/0003_seed_topics.sql`; `backend/migrations/0004_seed_concepts.sql`; `backend/migrations/0006_seed_backlog.sql`; `backend/app/services/selection.py`; `mobile/src/components/ConceptActions.tsx`. + +# 04 | Frontend stack and why each piece exists + +The frontend is a TypeScript React Native app built with Expo. It is not a website wrapped inside a phone shell. React describes the interface; React Native supplies native UI components; Expo supplies the development, native-module, build, and update ecosystem. + +| Technology | Version declared in this repository | Job in this app | +| --- | --- | --- | +| Expo | ~57.0.20, SDK 57 | Development runtime and managed native integrations | +| React Native | 0.86.3 | Mobile views, input, layout and interaction | +| React / React DOM | 19.2.8 / 19.2.8 | Component state; matching web renderer | +| TypeScript | ~7.0.2 | Compile-time checking of data and component contracts | +| React Navigation | Major 7 packages | Bottom tabs, nested profile stack, detail modal | +| Supabase JS | ^2.116.0 | Auth session lifecycle and password flows | +| AsyncStorage | 2.2.0 | Cached lessons, progress, outbox and preferences | +| Expo SecureStore | ~57.0.3 | Native session storage through a chunking adapter | +| Expo Notifications | ~57.0.17 | Permission, push token, Android channel, foreground display | +| Expo Updates | ~57.0.21 | Compatible downloaded JavaScript updates | +| Native screens / safe area | ~4.26.0 / ~5.7.0 | Navigation integration and screen inset handling | +| Fonts / icons | Space Grotesk; Expo vector icons | Typography and interface icons | + +These are manifest declarations. A tilde or caret is a permitted version range; `package-lock.json` resolves the install used by `npm ci`. `mobile/app.config.js` declares the app release as **1.8.0** and its native runtime as **1.3.0**. The package manifest's `1.0.0` is not the release authority. + +Styling uses React Native styles and a shared theme module for colors, typography, spacing, radii and shadows. There is no Tailwind, Redux, Zustand, React Query, or Expo Router dependency in the manifest. React contexts and custom repositories handle state and persistence. + +**Engineering assessment:** this keeps a small product approachable and avoids a second UI codebase. The tradeoff is that custom caching, reconciliation, navigation, and update compatibility still need deliberate testing; Expo does not make those product rules automatic. + +Sources: `mobile/package.json`; `mobile/package-lock.json`; `mobile/app.config.js`; `mobile/src/theme/index.ts`; `mobile/App.tsx`. + +# 05 | Screen map: what the learner sees + + + +The splash screen stays visible until the custom font is ready or has failed and the root has laid out. Authentication loading follows. A missing session shows Auth; a restored session shows the four main tabs. + +| Screen | What it shows | Important detail | +| --- | --- | --- | +| Today | Daily card, explanation/example, learned action, streak, like/save/share | A cached assignment can remain visible offline. A real account never receives an invented demo replacement. | +| History | Recent completed concepts with dates | The visible feed is capped at the last 10, although the backend retains more and exposes pagination. | +| Stats | Current/longest streak, total learned, topic progress | Topic denominators come from the server's published catalog; they can increase as content grows. | +| Profile | Account information, saved count, reminder toggle, theme, sign-out | Reminder times are displayed. The current screen has an on/off switch, not a time editor. | +| Personalization | Dynamic server topics and follow controls | The visible count is shared published concepts, not your personal remaining count. | +| Saved | Saved collection, search and topic filters | Older metadata is paged in; full text must finish downloading before offline reading. | +| Concept detail | Full lesson, example and actions | It can open above any tab; it prefers cached text and refreshes online. | +| About | App and organization information | Separate from account and learning state. | + +The authentication screen supports signup, sign-in, password recovery and show/hide password. Password visibility resets on submission or mode change. The root also displays an offline banner and eligible release highlights. + +**Beginner distinction:** browsing a saved lesson is not the same as receiving another daily assignment. The daily action records the current eligible day; there is no bulk "complete all 25" course flow. You may revisit content without earning multiple daily completions. + +Sources: `mobile/App.tsx`; `mobile/src/screens/TodayScreen.tsx`; `mobile/src/screens/HistoryScreen.tsx`; `mobile/src/screens/ProfileScreen.tsx`; `mobile/src/screens/SavedScreen.tsx`. + +# 06 | Startup, state and the first API request + + + +`App.tsx` nests SafeArea, Theme, Connectivity, Auth, and Progress providers. Each has a distinct job. Theme persists a device preference. Auth restores identity and supplies tokens. Progress is the common state owner so Today, History, Stats and Saved can agree after an action. + +**On startup:** the app reads cached identity and progress to paint quickly. It then revalidates through `GET /v1/me/state?compact=true`. That response includes profile data, follows, likes, saves, progress totals, recent collection metadata, and today's full lesson. If today's assignment does not exist, this GET creates it through the normal selector. + +One startup HTTP request does not mean the entire screen uses one database query or one network request. The backend performs an aggregate state query and separate daily-selection queries. The app also loads topics, registers push, synchronizes timezone, and may download missing saved bodies. It avoids blocking the initial screen on all those downloads. + +**Compact mode:** up to 50 enriched learned rows and 50 saved rows accompany exact totals, continuation cursors and older-topic aggregates. Like/bookmark membership remains complete. Stats adds recent/optimistic entries to older aggregates. Older clients can still use the full legacy response. Saved loads further metadata in pages of 50 when needed. + +**UI responsiveness:** tapping Like updates the interface immediately. Network mutations run on a serialized promise chain, so delayed responses cannot casually overwrite later taps. Pending user intentions are merged back into fresh server snapshots. This is optimistic UI with later confirmation, not proof that a write has already reached PostgreSQL. + +**Boundary to remember:** `localProgressRepository` and a deterministic bundled selector still exist for local/demo support. Signed-in Today uses server data; the visible signed-out route is Auth. Do not infer the production algorithm from the local prototype helper. + +Sources: `mobile/src/context/ProgressContext.tsx`; `mobile/src/services/remoteProgressRepository.ts`; `backend/app/api/v1/me.py`; `backend/app/services/state.py`. + +# 07 | Authentication and trustworthy identity + + + +The app sends the email and password to Supabase Auth using `signInWithPassword` or `signUp`. FastAPI does not store or check those passwords. A successful session provides an access token and refresh machinery. On native platforms the app uses Expo SecureStore with 1,800-character chunks; web uses AsyncStorage. + +For an application request, the HTTP client obtains the current access token and attaches `Authorization: Bearer ...`. FastAPI validates the token before deriving the user ID. A user ID supplied in a request body is not an identity claim the server trusts. + +| Check | Why it matters | +| --- | --- | +| Algorithm is ES256 | Rejects unsigned tokens and unexpected signing algorithms. | +| Key ID resolves in Supabase JWKS | Uses the project's public verification keys. | +| Issuer matches this project | A token from another issuer is not sufficient. | +| Audience is authenticated | Rejects a token intended for a different audience. | +| Expiry and subject are valid | Expired sessions and missing identity fail verification. | +| Queries use the verified user | Protects one user's progress from another user's request. | + +JWKS is a public key set, not the signing secret. The backend caches it for an hour and throttles refreshes for unknown key IDs to 30 seconds. The legacy HS256 secret setting is declared but not used by the current verifier. + +**Offline session recovery:** a cached identity can unlock already-cached browsing during a retryable refresh failure. It cannot make the API accept an invalid token. There is an eight-second startup failsafe; network/auth fetches use a separate 15-second timeout. + +**Sign-out:** the app tries to deregister this phone's push token before revoking the session. If global sign-out fails offline, it falls back to local sign-out. Account caches and queued intentions are cleared, with guards against late callbacks. Already-issued access-token validity is still governed by its verification/expiry rules; the backend does not perform a revocation lookup on every request. + +Sources: `mobile/src/context/AuthContext.tsx`; `mobile/src/lib/secureStorage.ts`; `mobile/src/services/authSession.ts`; `backend/app/core/security.py`; `backend/app/deps.py`. + +# 08 | Account emails: trigger, template and delivery + +There are three separate decisions in an account email: **why it is triggered**, **what it looks like**, and **which service delivers it**. Supabase Auth owns the trigger. Repository HTML supplies the design after manual installation. The configured SMTP sender handles delivery. + + + +| Email | Trigger | Current integration | +| --- | --- | --- | +| Confirm signup | Signup when email confirmation is enabled | `confirm-signup.html`; verified link then confirmation landing page | +| Reset password | User selects Forgot password and submits an address | `reset-password.html`; verified recovery link to a browser form | +| Password changed | A completed password change, with Supabase security notification enabled | `password-changed.html`; informational security email | +| Weekly learning digest | No current worker, template or preference found | A possible future feature, not part of these Auth emails | + +The work log records that the owner installed all three templates and enabled the password-change notification. That is reported operational evidence; this task did not inspect the live dashboard or send test emails. Actual inbox delivery remains unverified. + +**Which provider are we using?** The app integration is Supabase Auth. The current SMTP provider cannot be proven from repository code. The work log mentions an existing Gmail setup that needs configuration/testing; Resend setup remains a separate open draft, PR #187. There is no active Resend SDK/API call in the inspected backend. It would be inaccurate to say the app definitely sends through Resend today. + +As checked in the provider documentation, Supabase's default sender is intended for testing, restricts recipients to project-team addresses, and currently permits two messages per hour. Custom templates do not remove those limits. A configured custom SMTP service changes the transport; it does not require rewriting the mobile signup flow. [Supabase SMTP](https://supabase.com/docs/guides/auth/auth-smtp). + +Merging HTML into GitHub or publishing an app release does not install the templates. Supabase dashboard changes and a controlled inbox test are separate operational steps. + +Sources: `docs/EMAIL_TEMPLATES.md`; `backend/email-templates/`; `mobile/src/context/AuthContext.tsx`; `docs/WORK_LOG.md`. + +# 09 | Signup and password reset, end to end + + + +**Signup:** Supabase creates the Auth user. The database's `on_auth_user_created` trigger initializes the profile, notification preferences and default follows. With confirmation enabled, signup may return a user but no authenticated session. The UI asks the person to check email. The verified link redirects to the configured Site URL, intended to be the backend's `/confirmed` page, which tells them to return to the app and sign in. + +**Recovery:** the app calls `resetPasswordForEmail` with the API's absolute `/reset-password` URL when configured. The response stays neutral about whether an account exists. Supabase verifies the emailed link and redirects to the browser page with recovery information in the URL fragment. + +The page reads the fragment, checks `type=recovery`, and submits the new password directly to Supabase's `/auth/v1/user` endpoint. It uses the public anon key plus the recovery access token. The fragment is not sent as part of the normal HTTP request to FastAPI. After success, the page removes it from browser history and directs the person back to sign-in. The page also handles invalid or expired links and mismatched passwords. + +| Operational item | Why it must be correct | +| --- | --- | +| Supabase Site URL | Signup must land on a meaningful page after verification. | +| Allowed redirect URLs | Recovery must be permitted to reach this backend's `/reset-password`. | +| Backend SUPABASE_ANON_KEY | The browser form needs public Auth client configuration; otherwise it shows reset unavailable. | +| Template ConfirmationURL placeholder | The button must visit Supabase verification, not jump straight to a success page. | +| Security notification toggle | Password-change email is independent of the reset email. | + +There is no configured native URL scheme in the inspected app config. These flows use HTTPS landing pages and explicit return-to-app instructions; they are not a fully automatic deep-link sign-in flow. + +Sources: `backend/app/api/v1/pages.py`; `mobile/src/context/AuthContext.tsx`; `backend/migrations/0001_schema.sql`; `docs/EMAIL_TEMPLATES.md`. + +# 10 | Credentials: what is used, where and why + +This inventory names settings and credentials without displaying their values. A URL, app identifier, public project key, user session token and server secret have different security roles. + +| Name or credential | Location and consumer | Classification / purpose | +| --- | --- | --- | +| EXPO_PUBLIC_API_BASE_URL | EAS/mobile build environment; HTTP client | Public backend address; not a password | +| EXPO_PUBLIC_SUPABASE_URL | Mobile Auth client | Public project address | +| EXPO_PUBLIC_SUPABASE_ANON_KEY | Mobile Auth client | Public project key; not a logged-in identity | +| SUPABASE_URL / SUPABASE_JWKS_URL | Backend config / JWT verifier | Public project and verification-key endpoints | +| SUPABASE_ANON_KEY | Backend reset page | Public Auth key intentionally emitted into that page | +| DATABASE_URL | Railway API/workers | Secret PostgreSQL connection; application transaction pooler | +| DIRECT_URL | Migration/operator environment | Secret database connection for DDL; configured session connection | +| GEMINI_API_KEY | Backend generation paths | Secret provider credential; sent in x-goog-api-key header | +| EXPO_TOKEN | GitHub Actions secret | Allows EAS CLI to build/publish; not a learner's push token | +| GITHUB_TOKEN | Ephemeral Actions token | Workflow permissions for releases, dispatch and audit issues | +| User access / refresh tokens | Auth session storage | Sensitive account credentials; never publish them | +| Expo push token | Phone and device_tokens table | A sensitive delivery address, not a login credential | + +`SUPABASE_SERVICE_ROLE_KEY` and `SUPABASE_JWT_SECRET` are declared optional settings, but searches found no current consumers beyond their declarations. FastAPI connects using `DATABASE_URL`, not a service-role REST client. The active JWT verifier uses ES256/JWKS. + +Android configuration references `google-services.json`. Firebase service-account credentials for FCM/EAS are private and distinct from client Firebase configuration. Their presence or validity in EAS was not inspected. iOS push would require Apple credentials; the checked-in automated build flow targets Android. + +SMTP host, user, password/API credential and sender identity live in the email provider/Supabase setup, not the app bundle. No `RESEND_API_KEY` is consumed by current app code. All `EXPO_PUBLIC_*` values are readable from the bundle, so they must never hold backend secrets. Public keys identify an application; JWTs identify a user. [Supabase API keys](https://supabase.com/docs/guides/getting-started/api-keys). + +Sources: `backend/app/config.py`; `backend/.env.example`; `mobile/.env.example`; `mobile/app.config.js`; `.github/workflows/release.yml`. + +# 11 | Backend stack and request lifecycle + +FastAPI is the application's trusted rule layer. It accepts HTTP/JSON, validates input, verifies the caller, runs service logic, and reads/writes PostgreSQL. Uvicorn is the web server that runs the application. Docker packages that Python process for Railway. + + + +| Component | Pinned version | Purpose | +| --- | --- | --- | +| Python | 3.12 Docker base | Backend language/runtime | +| FastAPI | 0.141.1 | Routing, dependency injection, response contracts | +| Uvicorn | 0.52.4 | ASGI HTTP server | +| SQLAlchemy async | 2.0.52 | Sessions, pooling and async SQL execution | +| asyncpg | 0.31.0 | PostgreSQL network driver | +| Pydantic Settings | 2.15.0 | Environment configuration; schemas use Pydantic models | +| PyJWT with crypto | 2.13.0 | ES256 signature and token-claim validation | +| httpx | 0.28.1 | HTTP calls to Gemini, Expo and Supabase JWKS | +| pytest / pytest-asyncio | 9.1.1 / 1.4.0 | Unit, API and asynchronous integration tests | + +The structure separates routes (`api/v1`), request/response schemas, service logic, database session setup, and workers. Many services use explicit SQL through SQLAlchemy's `text()` rather than hiding the selection and concurrency rules behind ORM queries. ORM models mirror the schema; SQL migrations are the authority. + +Configuration comes from environment variables, with local `.env` support. Missing required database or Auth settings fail startup. CORS controls allowed browser origins; it is not a replacement for authentication. Production disables the interactive docs and raw OpenAPI route. `/health` includes a database query, so it checks more than whether a Python process exists. + +The default container listens on Railway's `PORT`, falling back to 8000. Railway configuration declares Docker builds, a `/health` check with a 30-second timeout, and failure restarts up to three retries. These files describe deployment behavior; they do not prove current service health or deployment settings. + +Sources: `backend/requirements.txt`; `backend/requirements-dev.txt`; `backend/app/main.py`; `backend/app/db/session.py`; `backend/Dockerfile`; `backend/railway.json`. + +# 12 | API map: screen actions become requests + +The `/v1` feature API is authenticated. The backend derives user identity from the verified token, so the phone does not choose the account whose data gets changed. + +| Method and path | Use / result | +| --- | --- | +| GET /health | Public service and database health | +| GET /confirmed; /reset-password | Public human-facing Auth landing pages | +| GET /v1/topics | Active topics, global published counts and your follow state | +| GET /v1/me/state?compact=true | Startup snapshot plus today's full lesson; may create today's assignment | +| GET /v1/daily | Get/create today's assignment; 409 catalog_exhausted if no candidate | +| POST /v1/daily/complete | Mark the most recent eligible assignment learned; return its day and streaks | +| GET /v1/me/stats | Current/longest streak and total learned | +| GET /v1/me/history | Cursor-paged completed history | +| GET /v1/me/saved | Cursor-paged saved metadata | +| PUT /v1/me/topics?compact=true | Replace the whole followed-topic set | +| PATCH /v1/me?compact=true | Update display name and/or validated timezone | +| GET / PUT /v1/me/notifications | Read or replace reminder preferences | +| POST / DELETE /v1/me/push-token | Register/reassign or remove a device's push token | +| GET /v1/concepts/{slug} | Read a published lesson's full content | +| PUT / DELETE /v1/concepts/{slug}/like | Set or clear the caller's like | +| PUT / DELETE /v1/concepts/{slug}/save | Set or clear the caller's bookmark | + +Collection endpoints default to 50 rows and permit 1-100. History uses a date cursor; Saved uses a timestamp/UUID ordering cursor so ties can be traversed. These are live views, not frozen snapshots; a refreshed first page discovers new records above an older cursor. + +**Error meanings:** 401 means the request is not authorized by a valid session; 400 can mean invalid topics, timezone or cursor; 404 can mean a missing lesson/assignment; 422 is input-schema validation; 409 is daily catalog exhaustion; 5xx is a server failure. The client also represents transport failure as status 0, which is not an HTTP status returned by the server. + +`/me/state` keeps the rest of the account usable by returning `daily: null` on exhaustion instead of failing the entire response. Like/save requests confirm the requested state. Repeating a save remains saved, although the backend may refresh its timestamp; state idempotency does not mean all metadata is byte-for-byte unchanged. + +Sources: `backend/app/api/v1/`; `backend/app/schemas/`; `backend/app/services/collections.py`; `mobile/src/api/client.ts`. + +# 13 | Database relationships: shared content, personal state + + + +PostgreSQL stores relationships, not a separate copy of every lesson for every person. A concept row belongs to one topic. A daily assignment points from a user to that concept and a date. Two users can point to the same concept without sharing completion status. + +| Table group | Tables | Scope | +| --- | --- | --- | +| Identity bridge | profiles, linked to Supabase auth.users | One app profile per Auth identity | +| Shared learning content | topics, concepts | Shared catalog and content provenance | +| Personal learning choices | user_topics | Many users follow many topics | +| Personal daily progress | daily_assignments | One user's concept/date/completion | +| Personal interactions | concept_interactions | Independent liked_at and saved_at on one user/concept row | +| Reminder configuration | notification_preferences, device_tokens | Per-user settings and one or more handsets | +| Generation operations | concept_backlog, generation_daily_usage | Shared writing queue and daily call budget | +| Reminder operations | reminder_log | Claimed user/date/time slots | + +There are eleven application tables in the inspected migrations, plus Supabase's managed Auth tables. UUIDs are database identifiers; stable slugs such as `hash-tables` identify lessons in API paths and mobile state. + +**Example:** Concept C is stored once. Ali's assignment references C with a completion timestamp; Sara's assignment references C with no completion. Ali's like and Sara's bookmark are separate interaction records. Neither action edits the lesson body. Public like totals aggregate likes across accounts, while the API excludes the viewer's own like and the UI adds it locally for immediate feedback. + +This model makes shared generation economical and per-user progress clear. It also means changing a published concept changes the shared content readers may see on later refresh. Cached copies can stay older until refreshed. + +Sources: `backend/migrations/0001_schema.sql`; `backend/migrations/0005_concept_backlog.sql`; `backend/migrations/0007_reminder_log.sql`; `backend/migrations/0010_generation_daily_usage.sql`; `backend/app/db/models.py`. + +# 14 | Database guarantees, triggers and migrations + +| Rule | Enforcement | Why it matters | +| --- | --- | --- | +| At most one assignment per user/day | UNIQUE(user_id, assigned_for) | Two devices cannot give the account two daily slots. | +| Never reassign a concept to one user | UNIQUE(user_id, concept_id) | No-repeat survives races and application mistakes. | +| Stable concept identity | Unique concept slug | Prevents duplicate catalog slugs. | +| One follow per user/topic | Composite primary key | Repeated follows do not duplicate rows. | +| Independent like/save state | One interaction per user/concept | Liking does not automatically save or complete. | +| One reminder claim per slot | Primary key(user_id, local_date, slot) | Overlapping workers do not separately claim the same reminder. | +| Nonnegative generation usage | Daily primary key and check constraint | Atomic shared counter is durable across process restarts. | + +**Actual SQL triggers:** inserting `auth.users` runs `handle_new_user`, which creates a profile, default reminders, and follows for all active topics at that time. Several tables run `touch_updated_at` before updates. A backend bootstrap helper is a fallback when state loading finds no profile. + +**What is not a SQL trigger:** lesson reading does not call Gemini through PostgreSQL. Notifications are not sent by a database email trigger. Generation and reminder decisions are Python logic invoked by a request/background task or an external cron schedule. + +Row Level Security is enabled. Client policies limit access by owner or published/active state. Operational tables have no client policies. The privileged backend connection can bypass those client protections, so every backend query still needs correct user scoping. RLS does not repair a backend query that uses a privileged connection and forgets its ownership filter. + +Migrations are ordered SQL files: core schema and RLS; topic/lesson seeds; backlog schema/seeds; reminder claims; stale-generation timestamps; like index; shared generation budget. Applied files are immutable. A new schema change needs a new migration, not an edit to old history. + +The production ledger lists all ten SQL filenames. Migration 0010's actual application and independent verification are recorded in the work log. Deployment does not apply migrations automatically. The GitHub check compares filenames with the ledger; it does not connect to production and prove the schema. Verification must happen before recording a filename as applied. + +Sources: `backend/migrations/`; `backend/app/services/users.py`; `.github/workflows/migrations.yml`; `RELEASING.md`; `docs/WORK_LOG.md`. + +# 15 | The exact daily-selection algorithm + + + +1. PostgreSQL derives today's calendar date using the user's stored profile timezone. Production callers cannot supply an arbitrary assignment date. +2. If today's assignment exists, return it unchanged. Completion, refreshes and follow changes do not replace it. +3. Find published concepts in followed topics, excluding **every concept ever assigned** to this account, including unfinished assignments. +4. Prefer topics with the oldest last-assigned date; never-seen topics come first. Use SQL `random()` to break ties among eligible concepts. +5. If the followed pool is empty, request background refill for an eligible stale followed topic, then immediately search the global published catalog with the same no-repeat rule. +6. If no global candidate exists, return exhausted. Otherwise insert the assignment. A uniqueness conflict lets the concurrent winner's row be returned. +7. After a new followed-topic assignment, count that user's remaining unassigned content in its topic. At five or fewer, request background prefetch. + +**Random does not mean reshuffle on refresh.** Randomness is used when creating the assignment. Persistence makes the result stable for the date. Topic rotation reduces the chance that a huge topic dominates a small one; it is not a fixed weekday schedule or a strict 20% quota. + +The first fallback response sets `outside_followed_topics=true`. That flag is not stored with the assignment; the existing-assignment branch returns false on later fetches, so the explanatory banner can disappear while the assigned lesson stays the same. + +Selection does not rank by likes, reading speed, quiz performance or difficulty. It does not call a language model to decide what a particular person needs next. It is a deterministic rule order plus a random tie-break, followed by a durable database assignment. + +Sources: `backend/app/services/selection.py`; `backend/app/api/v1/daily.py`; `backend/app/api/v1/me.py`; `backend/migrations/0001_schema.sql`. + +# 16 | Two users, one week: same or different? + +This is an illustrative valid outcome, not a prediction or a dump of real user records. Ali follows AI and Mathematics. Sara follows Software Engineering only. Both are new, open the app each listed day, and have sufficient eligible content. + +| Day | Ali's possible assignment | Sara's possible assignment | +| --- | --- | --- | +| Monday | AI: Overfitting | Software: Idempotency | +| Tuesday | Mathematics: Bayes' theorem | Software: ACID transactions | +| Wednesday | AI: Tokenization | Software: Connection pooling | +| Thursday | Mathematics: Eigenvectors | Software: Retries and backoff | +| Friday | AI: Transfer learning | Software: Dependency injection | +| Saturday | Mathematics: Gradient descent | Software: Feature flags | +| Sunday | AI: Regularization | Software: Database migrations | + + + +Ali's first topic could instead be Mathematics. Rotation uses his assignment history, so Sara's reading does not change which topic is least recent for him. If two people follow the same subjects and have similar history, they can get the same or different concepts. A concept is not reserved globally when one person receives it. + +**Different start dates:** a new user can receive a lesson that an older user finished months earlier. The new user does not inherit the older user's progress, and the app does not force everyone to read the newest generated concept. + +**Changing follows:** if Ali unfollows AI after Monday's card is created, Monday remains assigned. The next unassigned day uses the updated followed set. If every followed topic is exhausted, the global fallback can give a lesson outside that set. Following no topics also leads to the global fallback; it is not a way to disable the daily assignment. + +**Adding content:** a newly published concept becomes a candidate for everyone who has never been assigned it. Existing assignments stay pinned. A user without today's assignment can receive new content on a later request, including later that same day after an earlier exhausted response. + +Sources: `backend/app/services/selection.py`; `backend/app/services/interactions.py`; `backend/app/schemas/me.py`. + +# 17 | Completion, missed days and streaks + + + +Marking learned writes `completed_at` with the server clock. Repeating the completion call preserves the existing timestamp. A completion normally applies to today; the API can complete yesterday only if there is no newer assignment and yesterday is still the most recent eligible day. Older dates cannot be backfilled through this endpoint. + +| Situation | Result | +| --- | --- | +| Read today's card and tap learned twice | One completed assignment; no extra lesson and no double streak credit. | +| Open a card but never mark it learned | It stays uncompleted, yet no-repeat excludes it from future assignments. | +| Never open the app that day | No assignment is created merely by midnight or by the reminder worker. | +| Read at 23:58, complete at 00:01 with no new assignment | The API can credit yesterday's assignment. | +| A new day's assignment already exists | Completion targets that newer assignment, not yesterday's card. | +| Queue an offline completion then reconnect next day | The mobile replay code drops the stale-day action; it does not backfill the streak. | +| Save or like an old lesson | It changes the interaction, not daily learning credit. | + +Streaks are derived from completed `assigned_for` dates, not a mutable counter supplied by the phone. Consecutive dates form runs. The current streak can end today **or yesterday**, so an unfinished morning does not immediately break a streak. A full missed day creates a gap. The longest streak is the longest historical run. + +Example: completed Monday, Tuesday and Wednesday; Thursday is unfinished. During Thursday, current remains 3. If Friday arrives with Thursday still missed, current is 0 until a new run begins; longest remains 3. Completing Friday yields current 1 and total learned 4. + +Changing timezone changes future day boundaries; existing `assigned_for` dates are not rewritten. The device sends its IANA timezone on sign-in/session activation as best effort. PostgreSQL validates the zone. A failed sync can leave UTC or an older zone until a successful later sync; changing the phone clock is not the server's source of truth. + +Sources: `backend/app/services/interactions.py`; `backend/app/services/streaks.py`; `mobile/src/services/remoteProgressRepository.ts`; `mobile/src/services/notifications.ts`. + +# 18 | What really happens after all 25 lessons + +**This is the main catalog-growth limitation found in the source.** The system measures a user's remaining lessons to request prefetch, but the generator's stopping rules measure the total shared published catalog. Reading a concept never removes it from that total. + + + +| Quantity | Current rule | Example after Ali finishes a 25-concept topic | +| --- | --- | --- | +| Shared published count | Number of published concept rows in the topic | Still 25 | +| Ali's unassigned count | Published concepts never assigned to Ali | 0 | +| Sara's unassigned count | Same calculation for Sara | Could still be 20 if she has received 5 | +| Scheduled top-up target | MIN_POOL_PER_TOPIC, default 25 | Deficit = 25 - 25 = 0; no generation | +| Prefetch trigger | Personal unassigned count is at most 5 | Can request a job | +| Prefetch stop target | Shared published count reaches 10 | 25 already exceeds 10; job stops without a new lesson | + +So "I finished 25; the system automatically adds the next 25" is **not** the implemented behavior. Even on Ali's twentieth assignment, when five remain, requesting prefetch can do no work because the global shelf already holds 25. A pending backlog and a working Gemini key do not override this stop condition. + +On the next unassigned day, Ali gets an eligible concept from another followed topic if possible. When all followed topics are dry, the selector widens to other published topics. If the whole catalog is exhausted for Ali, `/v1/daily` returns 409 and `/me/state` returns `daily: null`. Sara continues receiving her own eligible lessons normally. + +The exhausted UI says new concepts are on the way, but current code does not guarantee when new content will arrive. This conclusion is a direct source-based deduction, not a production experiment or a change made in this task. + +**Possible future fix:** base replenishment on a defined reserve of unassigned lessons for active readers, or an explicit publishing cadence, while retaining the shared budget and bounded work. Raising the shared target can add content temporarily, but it still does not model each user's remaining runway. + +Sources: `backend/app/services/prefetch.py`; `backend/app/services/pool.py`; `backend/app/services/selection.py`; `backend/app/config.py`; `mobile/src/screens/TodayScreen.tsx`. + +# 19 | Content generation: from title to published lesson + + + +Gemini writes explanations for curated subjects; it does not choose the syllabus. A backlog row supplies a stable slug, title, topic, optional angle and difficulty. The generator asks for JSON containing a summary and example. The model setting defaults to `gemini-3.1-flash-lite`; a runtime environment can override it. + +| Stage | Actual behavior | +| --- | --- | +| Entry point | Scheduled pool_topup, request-triggered prefetch, or deliberate rewrite_catalog maintenance | +| Claim | Oldest eligible pending title, fewer than 3 attempts; PostgreSQL row locking skips work claimed by another worker | +| Budget | Reserve a shared daily call slot in the same transaction as the claim | +| Commit | Commit before the provider request so a DB transaction is not held while the model writes | +| Provider request | httpx POST to Gemini generateContent, with a 45-second timeout | +| Prompt | Version 2026-08-v2; short everyday explanation and one concrete example | +| Validation | JSON shape, required text, length limits, no specified filler/meta openings or code fences | +| Publication | Insert published concept with model/prompt provenance; mark backlog done only if insertion succeeded | + +The request uses temperature 0.7, up to 800 output tokens and a JSON response schema. Validation requires a 100-420 character summary and a 40-300 character example. It rejects identical summary/example text and selected undesirable phrasing. These checks improve formatting; they do not establish factual correctness or expert educational review. Valid output is published automatically. + +Scheduled refill compares active topics with `MIN_POOL_PER_TOPIC` and fills deficits subject to backlog, switches and budget. The backend README recommends a daily Railway job but does not commit a precise live cron schedule. Request-triggered prefetch is an `asyncio` task inside the API process, deduplicated by topic in that process, with at most five generated lessons and a global published target of ten. + +A new published lesson needs no mobile release. It appears through API data on a later request. Generation success does not itself send a "new content" push, change today's pinned assignment, or create a new topic. + +Sources: `backend/app/services/generation.py`; `backend/app/services/pool.py`; `backend/app/services/prefetch.py`; `backend/app/workers/pool_topup.py`. + +# 20 | Generation budget, retries and failure states + + + +The default **200-call cap** is an application limit shared across API prefetch, scheduled refill and catalog rewrites. It is not a guaranteed Gemini allowance, a user limit, or a token/spending cap. The database reserves one slot before each attempted call. Every process must use the same configured cap. + +The budget day is PostgreSQL's current date in `America/Los_Angeles`, matching Gemini's documented midnight Pacific RPD reset. During Pacific daylight time that is 12:00 in Karachi; during Pacific standard time it is 13:00. The learner's midnight is a different clock. Provider limits also depend on project, model and tier; multiple keys for the same project do not create independent provider quotas. [Gemini rate limits](https://ai.google.dev/gemini-api/docs/rate-limits). + +| Outcome | Backlog consequence | Shared budget consequence | +| --- | --- | --- | +| No eligible backlog | No title claimed | No reservation | +| Budget denied or claim commit fails | Claim and attempt roll back | No committed call slot | +| Valid lesson | Mark done after actual insert | Reserved slot remains spent | +| Invalid response/provider failure | Return pending or retire after 3 attempts | Committed slot remains spent | +| Provider 429 | Return pending and refund backlog attempt | Daily slot is still spent | +| Crash after commit | Row may remain generating | Slot remains spent; later top-up can reclaim stale work | +| Slug collision on publication | Mark failed; no new concept | Call was already spent | + +Scheduled top-up reclaims generating rows older than 30 minutes or with a missing claim time. Its rate-limit retry starts at 15 seconds, doubles up to a 120-second backoff component, respects a longer provider delay, and stops after five consecutive rate limits. Normal scheduled pacing defaults to six seconds between calls. Prefetch stops on a rate limit and has no equivalent pacing loop. + +**Safety controls:** generation defaults off; it needs the master enable switch and key. Prefetch also needs `GENERATION_ON_DEMAND`. A cap of zero blocks new reservations. These settings stop new work; they cannot recall an already-sent request. No distributed per-minute rate limiter is implemented, so parallel processes may still hit provider RPM limits below the daily cap. + +Sources: `backend/app/services/generation_budget.py`; `backend/app/services/pool.py`; `backend/app/services/prefetch.py`; `backend/app/workers/rewrite_catalog.py`; `backend/app/config.py`. + +# 21 | Adding more lessons and new topic cards + +There are three different expansion jobs: writing another lesson in an existing topic, adding a new topic, and announcing an app release. They use different data and do not automatically trigger one another. + + + +**More lessons in an existing topic:** add reviewed, unique backlog subjects through the repository's migration/operational process. A generator can turn them into published concepts only when the switches, target/deficit rule and budget permit it. Adding pending titles alone does not make them visible. The current shared-count limitation still applies even with a large backlog. + +**A new topic:** add an active topic row with a stable slug, display name, description and sort order; provide actual published content or eligible backlog and generation configuration. `GET /v1/topics` supplies the topic to Personalization and Stats. The mobile UI reads this dynamic list, so a basic new topic does not inherently require an APK. New styles or features may require client work. + +The new-user trigger follows every active topic **when that account is created**. It does not automatically add a later topic to all existing users. Existing users can choose to follow it. Global fallback may still select from it after their followed catalog is exhausted. Topic appearance and default follow policy are separate choices. + +**What counts change?** A published addition raises the topic's shared count. A learner at 25/25 can later show 25/30 without losing a completion. This is a larger denominator, not a reset. A draft, failed backlog row or archived concept is not a new published candidate. + +| Change | Needs new content data? | Needs app release? | Automatically sends push? | +| --- | --- | --- | --- | +| Another lesson, existing shape | Yes | Usually no | No | +| Another basic topic | Yes | Usually no | No | +| New daily-card UI | Not necessarily | Yes, usually OTA if compatible | No | +| Native permission/module | Not necessarily | New native build/runtime | No | +| Release highlights | Bundled highlights entry | Yes | No; displayed in app | + +There is no built-in admin CMS or mobile "add topic" screen in this repository. Extending the syllabus is an operator/developer activity. A future editorial tool should preserve unique slugs, review status and the generation budget. + +Sources: `backend/migrations/0001_schema.sql`; `backend/migrations/0006_seed_backlog.sql`; `backend/app/api/v1/topics.py`; `mobile/src/hooks/useTopics.ts`; `mobile/src/screens/StatsScreen.tsx`. + +# 22 | Push registration and the delivery chain + + + +On account activation, the mobile Auth effect starts reminder registration as best effort. The code requires `Device.isDevice`, requests notification permission if needed, creates Android's `reminders` channel at high importance, gets an Expo push token using the EAS project ID, and sends it to FastAPI. + +The backend stores that token in `device_tokens`, scoped to the authenticated user. A unique token can move to a new user when the same handset signs into another account. A user can have several tokens, so a single reminder slot can fan out to several devices. + +The backend sends to Expo's HTTPS push endpoint. Expo handles the downstream platform services: FCM for Android and APNs for iOS. Firebase here is a notification delivery dependency, not the application's main database or authentication provider. The inspected workflow automates Android distribution; source-level iOS support does not prove a working shipped iOS build. [Expo push overview](https://docs.expo.dev/push-notifications/overview/). + +| Layer | Responsible for | +| --- | --- | +| User and phone OS | Permission, app/channel settings, focus modes and final display | +| Mobile code | Register token and configure foreground presentation/channel | +| FastAPI and PostgreSQL | Store token and account reminder preferences | +| Reminder worker | Decide which user/time slots are due | +| Expo Push | Accept messages and route to platform push infrastructure | +| Platform service | Deliver to the registered application/device | + +The payload contains a generic title/body, default sound, high priority and the Android channel ID. It contains no lesson slug or custom navigation data. No custom notification-response navigation handler was found; the code does not guarantee a tap routes to a particular lesson screen. + +If a notification arrives while the app is open, the handler shows the banner/list quietly: no sound and no badge update. If it arrives while closed, the app need not be executing a JavaScript timer for the OS to show a remote push. Device and delivery settings can still suppress or delay it. + +Sources: `mobile/src/services/notifications.ts`; `mobile/src/context/AuthContext.tsx`; `backend/app/api/v1/me.py`; `backend/app/services/reminders.py`; `mobile/app.config.js`. + +# 23 | The reminder worker's exact decision + + + +`python -m app.workers.reminders` runs one pass and exits. Its documented cadence is every 15 minutes on Railway, with a matching 15-minute look-back window. The actual schedule is configured outside the checked-in `railway.json`; it was not inspected live in this task. + +For every enabled user's stored reminder times, the SQL considers occurrences on both today and yesterday in that user's timezone. A slot is due if it is at or before local now and strictly later than now minus 15 minutes. This handles slots near midnight without wrapping bare time values incorrectly. + +The worker requires at least one registered device and skips a completed scheduled date. It also suppresses yesterday's late slot when the current day has already been completed. It does **not** require an existing daily assignment or available content. A user who has not opened the app, or has exhausted the catalog, can therefore still qualify for the generic reminder. + +The due slots are atomically inserted into `reminder_log` with conflict handling. Only the slots won by this pass are sent. Claims commit before network delivery. Messages fan out to registered devices in batches of at most 100 with a 30-second HTTP timeout. + +| Result | Current handling | +| --- | --- | +| Expo ticket status ok | Increment the accepted/sent counter | +| DeviceNotRegistered ticket | Remove that stale device token | +| Other ticket rejection | Log it | +| HTTP batch failure | Log and continue; the slot remains claimed | +| Overlapping worker or rerun | Existing primary key prevents a second claim | + +**Guarantee:** this favors one server claim/attempt per slot over retrying until delivered. It is not end-to-end exactly-once phone delivery. A crash or failed batch after the claim can miss a reminder. The code processes initial tickets but does not poll Expo delivery receipts, and the `sent_at` column is stamped when claimed rather than proving handset display. + +Completion can race with an already-claimed batch. It suppresses later decisions; it cannot retract a push already sent. The generic copy should be understood in that light. + +Sources: `backend/app/services/reminders.py`; `backend/app/workers/reminders.py`; `backend/migrations/0007_reminder_log.sql`. + +# 24 | Notification timing: a day in Karachi + +Defaults are **08:00, 14:00 and 20:00 in the profile timezone**, with reminders enabled. The API accepts one to three unique valid HH:MM values. The current Profile UI displays those times and an enabled switch; there is no custom-time editor on that screen. + + + +| Example | Expected server behavior | +| --- | --- | +| 08:00, day unfinished | A due worker pass can claim and send the morning reminder. | +| 08:10, learner completes online | Completion becomes authoritative in PostgreSQL. | +| 14:00 and 20:00 afterward | Those slots are suppressed because the date is complete. | +| Learner completes at 15:00 instead | Morning and afternoon can have sent; evening is suppressed. | +| Learner never completes | Up to three configured slot claims per day, each sent to all registered devices. | +| Completion only exists offline at 13:55 | Server may still send at 14:00 until the queued completion reaches it. | +| User disables reminders online | Future due queries exclude that user; an already-sent message may remain. | +| Slot at 23:58; worker runs at 00:05 | Yesterday's occurrence lies inside the window and may be sent, unless completion suppresses it. | + +For `Asia/Karachi`, 08:00 corresponds to 03:00 UTC, 14:00 to 09:00 UTC and 20:00 to 15:00 UTC. Railway's scheduler uses UTC, but the SQL translates the clock per user; you do not create a separate cron job per timezone. A 14:07 custom slot would normally be caught by the next suitable pass, not necessarily at exactly 14:07. + +Railway documentation says cron timing can vary and a new run is skipped if the previous run is still active. If a pass is delayed beyond the 15-minute look-back, a slot can be missed; this implementation has no unlimited catch-up. A worker must close resources and exit. [Railway cron jobs](https://docs.railway.com/cron-jobs). + +**Different clocks:** progress and reminders use the profile's local calendar day; the generation budget uses Pacific day; GitHub audit schedules use UTC. These clocks should never be substituted for one another. + +Sources: `backend/migrations/0001_schema.sql`; `backend/app/schemas/notifications.py`; `backend/app/services/reminders.py`; `mobile/src/screens/ProfileScreen.tsx`. + +# 25 | Every major trigger, compared + +| Event or condition | What runs | Observable effect / limit | +| --- | --- | --- | +| New Supabase Auth user row | Database bootstrap trigger | Profile, default reminders and active-topic follows | +| Signup needing confirmation | Supabase Auth email system | Confirmation email through configured sender | +| Forgot password | Supabase Auth recovery flow | Reset email, then web password form | +| Successful password change | Enabled Supabase security email | Informational account notification | +| App session activates | Mobile Auth effect | Push registration and timezone sync, best effort | +| First daily/state request on a new local date | Daily selection service | Persist a new eligible assignment | +| Five or fewer personal unassigned concepts in the selected followed topic | API background prefetch request | May generate if shared count, switches, backlog and budget allow | +| Followed catalog exhausted | Prefetch request plus global fallback | Existing global content serves today; generation is not awaited | +| Scheduled pool worker | Shared catalog deficit calculation | Publish enough eligible backlog toward the configured target | +| Due reminder time, incomplete day | Reminder worker | Claim slot then send generic Expo pushes | +| Online learned action | Completion service | Store completion; later reminder decisions become silent | +| Reconnect or reopen with queued work | Mobile sync loop | Replay and reconcile while the app is active | +| Native app goes to background | Mobile lifecycle listener | Pause JS sync timers; remote push delivery is separate | +| New content row published | Later topic/daily/detail reads | Catalog grows; no automatic content announcement | +| Mobile changes pushed to develop | Preview EAS Update workflow | Preview-channel OTA, subject to workflow paths | +| Release merge into main | Release workflow; separate Railway deploy | OTA and release tag; APK gate; backend deployment | +| New app version with highlights | In-app release-card hook | Show until dismissed for that device/version | +| Monday 00:00 UTC or main push | GitHub audit | Scan dependencies/code; may create deduplicated findings issues | + +There is no automatic chain from "a user finishes a topic" to "a new topic is created" to "everyone gets an email." Each would need explicit product rules and implementation. Likewise, midnight makes a new date eligible, but does not by itself assign lessons to all accounts. + +Sources: `backend/migrations/0001_schema.sql`; `backend/app/services/selection.py`; `backend/app/services/reminders.py`; `mobile/src/context/ProgressContext.tsx`; `.github/workflows/`. + +# 26 | Offline storage: what survives no internet + + + +The app retains more than a card title. It stores full downloaded lesson bodies, including examples, and tries to download any missing saved lessons after successful state loads. It uses three concurrent download workers and individual cache entries rather than one giant lesson blob. + +| Data | Storage / behavior | Offline limit | +| --- | --- | --- | +| Auth session | Native SecureStore adapter; web AsyncStorage | Cached identity helps browsing; server requests still need valid authentication. | +| Daily/progress snapshot | AsyncStorage plus in-memory repository | Last known state may be stale. No fresh server assignment can be created offline. | +| Full concept bodies | Per-slug OfflineCache | Only completed downloads can be read offline. | +| Saved metadata | Account-keyed paged cache | Older titles can also be recovered from downloaded bodies. | +| Topics/follows | Shared cached topic store | Queued local choices override stale responses until replay. | +| Pending mutations | Durable serialized outbox | Survives restart while the account remains; not a cloud backup. | +| Reminder preferences | Last known settings cache | Toggle update needs the API; failure reverts the switch. | +| Theme / release dismissal | Device-scoped AsyncStorage | Persists independently of account progress. | + +**A bookmark is not a completed download.** Saving a concept online can start a background body fetch; closing the app or losing connectivity before it finishes can leave an unavailable detail page offline. Opening a previously cached lesson is more reliable than assuming all saved bodies have downloaded. + +Signed-in screens preserve useful caches and offer retry/empty states when data is missing. They do not substitute invented totals or demo lessons for real account data. Native connectivity is inferred from requests; there is no native network-status listener in this APK. The web build also listens for browser online/offline events. + +Sign-out clears server state, legacy daily cache, downloaded lesson cache, saved metadata, topics/follows, notification preferences and the outbox. Epoch guards prevent late responses from repopulating cleared data. Theme and What's New dismissal remain device preferences. Clearing app storage or reinstalling removes local availability; server-confirmed progress can be fetched again after sign-in. + +Sources: `mobile/src/services/conceptApi.ts`; `mobile/src/services/offlineCache.ts`; `mobile/src/services/accountCaches.ts`; `mobile/src/services/remoteProgressRepository.ts`; `mobile/src/services/savedApi.ts`. + +# 27 | Offline actions and safe replay + + + +The outbox stores the user's latest intent for each like, save, topic set or completion date. A like followed by an unlike should replay the final desired state, rather than blindly replaying every tap. Disk writes are serialized, and dequeue checks avoid deleting a newer action with the same key. + +The visible UI updates optimistically. Network writes and queue replay share a mutation chain. Fresh server responses are reconciled with pending intentions, so a still-queued save is not lost simply because the server snapshot has not seen it yet. + +| Replay situation | What the code does | +| --- | --- | +| Like/save | PUT for desired on; DELETE for desired off | +| Topic changes | Replace the complete followed set | +| Completion from today | Attempt the server completion endpoint | +| Completion from an older device day | Drop it; do not repair historical streaks | +| Network failure | Retain outstanding work and retry later | +| Server 5xx | Keep that intention for a later attempt | +| Rejected 4xx | Drop that entry so an unreplayable action does not block forever | +| Sign-out/account change | Invalidate old work and clear its caches/queue | + +When work remains or requests cannot reach the API, retries back off through 5, 10, 20 and 30 seconds, then stay bounded at 30 seconds. Partial success does not reset a failed cycle into a tight request loop. When reachable and the queue is empty, routine retry polling stops. + +Foregrounding the app wakes the loop; browser reconnect can wake it immediately. Going into the background pauses its timers. There is no closed-app OS background sync worker. Restoring Wi-Fi while the native app is closed does not itself guarantee queued completion is uploaded before the next reminder. + +**Tradeoff:** these rules preserve fast interaction and make common retries safe, but offline progress is provisional. The server's day and completion rules remain authoritative. A phone left offline across midnight can lose yesterday's queued learning credit even though downloaded lessons remain readable. Conflicting choices from two devices are not merged into a collaborative document; later accepted writes determine the stored state. + +Sources: `mobile/src/services/mutationOutbox.ts`; `mobile/src/services/remoteProgressRepository.ts`; `mobile/src/services/syncLoop.ts`; `mobile/src/context/ProgressContext.tsx`; `mobile/src/services/pendingProgress.ts`. + +# 28 | GitHub to production: the release path + + + +The repository is `Coding-Moves/one-concept`. Team feature/fix branches use a descriptive `codex/` prefix by default and target `develop`. The owner's workflow preserves focused commits and reviews one coherent chunk per PR. A production release is a separate `develop` to `main` PR. + +Before release, `mobile/app.config.js` supplies a new marketing version and a matching nonempty What's New entry. Pending database migrations are applied and verified separately, then recorded in `applied.txt`. The migration check must pass. Merging into `main` triggers production effects. + +| Release component | What happens | +| --- | --- | +| Production OTA | EAS publishes the mobile update with the production environment. | +| Preview OTA | The same release updates preview devices too. | +| Version tag / GitHub Release | Created only after the OTA job succeeds; an already-used tag causes failure. | +| Release APK workflow | Explicitly dispatched because token-created release events alone are insufficient for this workflow chain. | +| APK native gate | Rebuilds only when the shipped native runtime differs from the one recorded for apk-latest. | +| Backend | Railway auto-deployment is documented as independently following main; it is not gated behind EAS success. | + +`EXPO_TOKEN` authorizes EAS. GitHub's workflow token creates tags/releases and dispatches the APK job. The release workflow serializes releases so quick merges do not race the version tag. EAS public environment values are bundled during OTA publication; missing configuration can break an update even if the code is valid. + +**Important deployment distinction:** GitHub merge, successful OTA publish, Railway deployment health and actual phone adoption are four different observations. A successful release job proves the workflow completed its steps; it does not prove every phone has launched and downloaded the update. + +Current evidence: release PR #191 and migration PR #193 are merged. GitHub release v1.8.0 was published on 12 September 2026 at 19:29 UTC, which is 13 September at 00:29 in Karachi. The release and migration workflows succeeded. Live Railway rollout and generation-cap settings remain unverified in this handbook. + +Sources: `RELEASING.md`; `.github/workflows/release.yml`; `.github/workflows/release-apk.yml`; `.github/workflows/eas-update.yml`; `mobile/eas.json`. + +# 29 | OTA, native builds and the What's New card + + + +The app has a native layer installed in its APK and a compatible JavaScript/assets layer that EAS Update can replace. Its marketing version is **1.8.0** while `runtimeVersion` stays **1.3.0**. They answer different questions: "which release is this?" and "which native binary can run this update?" + +| Change | Delivery decision | +| --- | --- | +| Lesson data in PostgreSQL | API data refresh; no software update needed | +| Compatible screen/state JavaScript | OTA on the intended channel | +| New native module/permission/config | New native build and runtime procedure | +| Backend endpoint implementation | Railway backend deployment; compatible API contract still matters | +| Email template | Supabase template installation; separate from APK/OTA | + +The checked-in config checks for updates on load with a zero fallback wait. That favors quick startup using the cached/bundled code. It is not a promise that a freshly published update is applied instantly in the currently open session; download readiness, later launch and Expo's update behavior matter. Runtime matching is required for a compatible update. [Expo runtime versions](https://docs.expo.dev/eas-update/runtime-versions/). + +The stable GitHub `apk-latest` asset is the sideload installation path. A JavaScript-only release can ship without producing a new APK. EAS profiles distinguish development, preview, production and production-apk; the channel identifies which update stream a build follows. + +**What's New:** a bundled entry matches the current app version. The hook checks the last dismissed version stored on the device. An authenticated user sees the card when a matching entry exists and that version differs from the last dismissed value. Got it persists dismissal; long highlight lists scroll within the card. + +This is device-wide, not per account: signing out does not reset the dismissal. Reinstalling, clearing storage, or a failed storage write can make it appear again. The implementation stores the last dismissed version, not a permanent set of every version ever seen, so rolling between versions can also show cards again. No email, cron or push is needed to display it. + +Sources: `mobile/app.config.js`; `mobile/eas.json`; `mobile/src/data/whatsNew.ts`; `mobile/src/hooks/useWhatsNew.ts`; `mobile/src/services/whatsNewStore.ts`. + +# 30 | CI, dependency bots and validation + +| Mechanism | Trigger | What it establishes | +| --- | --- | --- | +| Preview OTA | Qualifying mobile pushes to develop; manual dispatch | Publishes to selected EAS channel; not a full regression suite | +| Manual Android build | Explicit workflow dispatch | Starts an EAS build with selected profile | +| Release | Main push | Production/preview OTA, tag and release, APK workflow dispatch | +| Migration ledger check | Main push; PR into main; manual | Every SQL filename is listed as applied; no live DB inspection | +| Security & Code Audit | Monday 00:00 UTC; main push; manual | pip-audit, npm high/critical findings, Ruff F/E9 and TypeScript | +| Dependabot | Weekly for Python, npm and Actions | Opens dependency-update PRs; Expo-managed major/minor updates are restricted | +| Stale cleanup | Daily 01:00 UTC | Marks automated issues stale after 30 days, closes after 7 more; excludes PRs | + +The audit creates deduplicated findings issues; it does not change code or automatically merge fixes. Tool failure is distinct from discovered vulnerabilities. The checked-in workflows do not include a general pull-request pytest job, so a green publish is not equivalent to all backend tests passing. + +**Local validation tools:** mobile TypeScript uses `npm run typecheck`; Node 24 runs the built-in regression tests. Browser acceptance scripts exercise password UI, saved offline reading, replay races and release-card layout. Android/web exports check packaging but are not a physical-device run. The backend uses `.venv/bin/python -m pytest`, with disposable PostgreSQL 16 via Podman for integration tests and mocked provider calls. + +The 1.8.0 preparation log records 145 backend tests and 33 Node tests passing without skips, plus TypeScript, exports and relevant browser scenarios. Those are historical results from release preparation, not tests rerun for this documentation task. Physical-phone behavior and actual inbox delivery were not established there. + +GitHub currently shows the 1.8.0 release and migration check succeeded, but the Security & Code Audit run failed. That run must be investigated before calling the operational checks all green; this handbook does not equate the failure with a confirmed vulnerability. Documentation validation here consists of source/link checks and PDF text/render inspection. + +Sources: `.github/workflows/`; `.github/dependabot.yml`; `mobile/tests/README.md`; `backend/tests/conftest.py`; `docs/WORK_LOG.md`. + +# 31 | Performance, scale and cost drivers + +The strongest cost decision is storing generated lessons once for everyone. If one lesson is read by 1,000 accounts, it needs one successful publication, plus any failed generation attempts, rather than 1,000 separate personalized model calls. Reading still costs API/database traffic, storage and delivery. + + + +| Area | Existing optimization | Practical limit | +| --- | --- | --- | +| Startup | Cached paint; compact state includes today's body | Membership arrays and some aggregate work still grow with account history. | +| Collections | Bounded enriched rows and cursor pages | Saved intentionally loads older metadata for full search; total local collection can grow. | +| Offline reading | Per-entry bodies; 3 download workers | Downloading a large collection still uses network and device storage. | +| Database | Pool of 5 with up to 5 overflow per process; pre-ping; LIFO reuse | Multiple API processes/workers multiply possible connections. | +| Idle connection latency | Default 30-second warm-up; 5-second probe budgets | Keeps a hot connection, not every possible burst connection; adds probe traffic. | +| Model calls | Shared catalog, bounded jobs and daily reservations | No factual-review gate or shared RPM limiter; failed attempts still spend reservations. | +| Reminders | Atomic claims and batches of 100 messages | More users/devices mean more fan-out; claim deduplication does not guarantee delivery. | + +Connections recycle after 1,800 seconds. Transaction-pooler mode disables prepared-statement caching and avoids pinning a transaction across the Gemini request. The API lifecycle manages warm-up and cleanup; cron workers do not start the API warm-up task. + +The work log's controlled 365-record fixture shrank startup state from 151,007 to 55,676 bytes, about 63%, using compact mode. This is one local fixture, not a production percentile or proof of a constant response size. Local connection warm-up experiments also showed benefits under induced idle expiry, not measured Railway/Supabase production latency. + +**Cost model:** total operating cost is hosting + database/storage/traffic + Auth/email + build/update delivery + model calls. Model spending depends on attempted calls, input/output tokens, selected model and tier. The 200-call setting is neither a dollar budget nor a guarantee of 200 published lessons. No invoices, current paid tiers or real monthly usage were read for this report. + +As traffic grows, measure API latency and failures, database connections, assignment exhaustion, remaining unassigned lessons per active reader/topic, queue delays, generation outcomes and push receipts before adding more infrastructure. + +Sources: `backend/app/db/session.py`; `backend/app/db/keepalive.py`; `backend/app/services/state.py`; `mobile/src/services/conceptApi.ts`; `docs/WORK_LOG.md`. + +# 32 | Architecture choices: advantages and tradeoffs + +These comparisons are engineering judgments about this app's requirements, not claims that one vendor or language is universally best. The existing design is a reasonable fit for a small daily-learning product with shared content and account-specific progress. + +| Current choice | Alternative design | Why the current approach fits / what it costs | +| --- | --- | --- | +| React Native + Expo | Separate native Android and iOS apps | Shares UI/product logic and release tooling; still requires native compatibility and device testing. | +| TypeScript client + Python API | One language on both ends | Strong frontend tooling plus readable backend services; contracts must stay aligned across languages. | +| Supabase Auth + custom FastAPI | Direct mobile application writes to hosted database APIs | Centralizes trusted-day, completion and budget rules; adds a hosted API to operate. | +| PostgreSQL relationships | Document-oriented personal lesson copies | Natural uniqueness, joins and transactional claims; complex SQL must be reviewed and indexed. | +| Shared generated catalog | Generate separately per learner/request | Amortizes model work and gives stable reading latency; less individual tailoring and finite catalog runway. | +| Curated titles + generated prose | Fully automatic topic invention | Controls syllabus and slug identity; curation and factual quality still need maintenance. | +| Topic rotation + random tie-break | Ordered prerequisite curriculum | Variety and simple personalization; no guaranteed beginner-to-advanced teaching sequence. | +| Server reminder schedule | Device-only local reminder schedule | Uses server-confirmed completion across devices; depends on worker health, network and push delivery. | +| Custom local outbox | Always-online interaction | Supports unreliable connections; introduces replay, date-boundary and account-cleanup complexity. | +| PostgreSQL job claims + process prefetch | Dedicated durable task queue | Fewer moving parts; API prefetch can stop when its process dies and lacks queue-level delivery guarantees. | +| EAS OTA with pinned runtime | New binary for every screen edit | Faster compatible updates; incorrect runtime discipline can break installed apps. | + +**What I would keep:** a shared catalog, verified JWT identity, database uniqueness rules, server-owned dates, durable offline intentions, and a separate production release branch. They directly support the product's core promises. + +**What deserves the next engineering investment:** reliable content runway, visible delivery outcomes, automated regression gates and clear operational configuration. Adding microservices or a new frontend framework would not by itself fix catalog exhaustion or missed reminders. + +Sources: `mobile/package.json`; `backend/app/services/selection.py`; `backend/app/services/pool.py`; `backend/app/services/reminders.py`; `RELEASING.md`. + +# 33 | Current limits and sensible next priorities + +This is a source-based assessment, not a claim that these improvements have already been built. No implementation changes were made while producing the handbook. + +| Priority | Finding | Next useful step | +| --- | --- | --- | +| 1 | Shared count targets can stop growth for a reader who has used all 25 concepts | Define required personal runway and change replenishment policy with concurrent-user/exhaustion tests. | +| 2 | Catalog-exhausted accounts can still receive the generic lesson reminder | Decide the intended exhausted-state reminder policy; make eligibility/copy reflect available learning. | +| 3 | Reminder claims survive failed delivery; no receipt polling | Add delivery visibility and an explicit retry/idempotency design if reliable reminders are a product promise. | +| 4 | Email transport/inbox success remains unverified | Verify configured SMTP, non-team delivery and complete signup/recovery/password-change flows. | +| 5 | No general PR regression suite in workflow YAML | Gate important branches with the existing backend and mobile tests, including real disposable PostgreSQL. | +| 6 | Daily budget is shared, per-minute pacing is not | Coordinate provider throughput if multiple processes start generating together. | +| 7 | Offline completion can expire across midnight | Choose and document a deliberate product policy before altering trusted-date semantics. | +| 8 | Generated prose passes format checks, not factual review | Add editorial sampling or a reviewed publication state if teaching accuracy requires it. | + +Other boundaries worth understanding: visible History contains the last ten records; full history endpoints exist but the screen does not expose full paging. New topics are not automatically followed by existing users. Custom reminder times are supported by the API but not editable in the current Profile UI. Background synchronization works while active/reopened, not as a closed-app OS task. + +The project currently does not implement a learning digest email, adaptive quizzes, spaced repetition, a prerequisite graph, a content CMS, payments, or a dedicated analytics/crash-reporting SDK. These would be new features. A future weekly digest would need learning aggregation, user preference/unsubscribe behavior, a scheduler, a delivery provider/template and deduplication records; Auth security emails are a different subsystem. + +Supabase's current documentation recommends publishable/secret keys over legacy anon/service-role names. This repository still uses the legacy naming contract. Treat key migration as planned configuration/compatibility work, not a reason to put secrets in mobile code. [Supabase API keys](https://supabase.com/docs/guides/getting-started/api-keys). + +Sources: `backend/app/services/prefetch.py`; `backend/app/services/reminders.py`; `mobile/src/screens/HistoryScreen.tsx`; `mobile/src/screens/ProfileScreen.tsx`; `.github/workflows/`. + +# 34 | Practical operating and troubleshooting guide + +Start with the layer that owns the symptom. A frontend retry cannot repair a missing database table; an app release cannot install an email template; a new Gemini key cannot bypass the generator's shared-count stop rule. + +| Symptom | Inspect first | What separates the likely causes | +| --- | --- | --- | +| Today unavailable on one phone | Connectivity, session and state response | Cached body missing versus API failure versus real daily:null exhaustion | +| Same lesson keeps showing today | Assignment date and ID | Normal daily pinning; completion does not unlock a second lesson | +| All followed lessons exhausted | Per-user assigned set and global candidates | Other users may still have plenty of eligible content | +| No new lessons generated | Enabled/key/model, shared target, backlog, daily cap, worker logs | A full global shelf stops refill even if a reader has zero remaining | +| Backlog stuck generating | claimed_at and scheduled-worker execution | Top-up reclaims rows older than 30 minutes; verify the worker actually runs | +| Reminder missing | Permission/channel, stored token, enabled prefs, timezone, completion, claim log | Claimed is not delivered; a failed send can leave a used slot | +| Reminder after learning | Is completion on server or only offline? | Also check a claim/send race and the current assigned_for date | +| Email never arrives | Supabase Auth logs and actual SMTP configuration | Default sender restrictions, provider failure, spam handling or invalid recipient | +| Reset page unavailable | Backend public anon key and redirect URL | App and email can work while the recovery page is misconfigured | +| Update absent on phone | Channel, runtime, EAS publish and relaunch | Marketing version alone does not establish native compatibility | +| Backend query fails after release | Migration application and deployed revision | Ledger consistency alone is not proof of schema or healthy rollout | +| Saved title opens to no offline body | Whether body download completed | Saved metadata and downloaded content are separate | + +**Useful local commands:** from `mobile/`, `npm ci`, `npm run typecheck`, and `npm test` with Node 24. From `backend/`, use `.venv/bin/python -m pytest` with test configuration. Development uses `python -m uvicorn app.main:app --reload --port 8000`. A physical phone needs a reachable LAN backend address, not its own `localhost`. + +**Operational worker commands:** `python -m app.workers.reminders` sends real messages under real configuration; `python -m app.workers.pool_topup` can spend Gemini quota and publish content; `rewrite_catalog` can change existing prose. They are actions, not harmless health checks. Use controlled test configuration for exploration. This report did not execute them against production. + +Read logs and aggregate counts without printing connection strings, bearer tokens, recovery URLs or provider credentials. Observe delivery and content availability separately from process liveness. + +Sources: `backend/README.md`; `mobile/tests/README.md`; `backend/app/workers/`; `backend/app/config.py`; `mobile/.env.example`. + +# 35 | Your seven-day learning digest + +Use this as a one-week self-study plan. It is part of the PDF, not an automation or an email subscription. Spend roughly 30-45 minutes a day: read, redraw one flow from memory, and explain it aloud as if onboarding another engineer. + +| Day | Read these chapters | One concrete outcome | +| --- | --- | --- | +| 1 - Product and stack | 01-06 | Draw the phone/Auth/API/database map. Explain topic, concept, assignment and learned without mixing them. | +| 2 - Identity and email | 07-10 | Trace signup and password reset. Identify which values are public configuration and which are sensitive credentials. | +| 3 - API and database | 11-17 | Trace a daily assignment and a completion through routes, SQL and uniqueness constraints. Explain why two devices agree. | +| 4 - Catalog and AI | 18-21 | Reproduce the 25-concept example on paper. Explain why unread=0 does not mean published=0. | +| 5 - Notifications | 22-25 | Draw registration and delivery separately. Walk through the three reminder times and an offline completion. | +| 6 - Offline and release | 26-30 | Trace an offline save through restart/replay, then a change from develop to compatible production OTA. | +| 7 - Engineering judgment | 31-39 | Explain costs, current limits and evidence boundaries. Present a prioritized improvement proposal. | + +**Weekly recap to keep:** the mobile app is the presentation and cache; Supabase Auth establishes identity; FastAPI owns application rules; PostgreSQL owns durable records and concurrency constraints; Gemini writes shared content; Python workers decide refills/reminders; Expo and platform push services deliver notifications; GitHub/EAS/Railway deliver software. + +The three questions you should be able to answer confidently are: **Who owns this fact? What event changes it? What happens if the operation runs twice or fails halfway?** Those questions explain assignments, bookmarks, notifications, migrations and budget reservations with the same engineering discipline. + +For hands-on study, use a disposable database and dummy provider responses. Simulate two users with different follows, a same-day second device, an exhausted catalog, a late completion and a failed push. The goal is to understand observable behavior, not spend quota or send test messages to actual learners. + +Sources: `docs/CODEBASE_MAP.md`; `backend/tests/test_selection.py`; `backend/tests/test_reminders.py`; `mobile/tests/README.md`. + +# 36 | Check your understanding: scenarios and answers + +| Scenario | Answer you should be able to explain | +| --- | --- | +| Ali and Sara both follow AI. Must Monday match? | No. Each has a separate candidate set and assignment. The same concept is allowed across users. | +| Ali opens Monday's lesson but skips learned. Can it return Tuesday? | No. Selection excludes prior assignments, not only completions. | +| Ali changes follows after loading today's card. What changes now? | Follow state changes; today's assignment remains. The next unassigned date uses the updated set. | +| A topic holds 25 published concepts and Ali has received all of them. Is the shared count zero? | No, still 25. His remaining eligible count is zero, which exposes the current refill mismatch. | +| Gemini is disabled but the shared catalog has unseen lessons. Does Today work? | Yes. Daily reading selects stored published rows. | +| Cap is 200 and two workers have each used 100 calls. Does a restart grant another 200? | No. The shared daily database ledger persists across workers/restarts. | +| A rate-limited attempt is refunded in backlog. Is its daily quota refunded too? | No. The committed reservation remains; backlog retry accounting is a different counter. | +| Learned is queued offline before 14:00. Is the afternoon reminder suppressed? | Only if server completion has arrived before the worker's relevant decision; local UI alone is insufficient. | +| Reminder log row exists but the Expo HTTP request failed. Will the next pass resend it? | Not in the current implementation; the slot is already claimed. | +| A saved title is visible offline. Must its example be available? | No. The full body must have finished downloading. | +| Version rises to 1.8.1 with only compatible JS changes. Must runtime rise? | No. Runtime changes only with the native release procedure. | +| A new topic row is inserted. Do existing accounts follow it automatically? | No. Default follows are established at account bootstrap, not retroactively. | +| A GitHub release exists. Did Supabase email HTML update too? | No. Templates are installed separately. | +| Does a green migration filename check prove the production table exists? | No. Actual application and verification must precede the ledger entry. | + +**Advanced design exercise:** propose a refill policy that keeps at least seven unassigned concepts for engaged readers in each followed topic, without generating seven new copies per person. Define the active-user window, bounded job size, shared budget, concurrent-worker behavior, sparse/new-topic policy and the exhausted-user experience. This is a proposed design exercise; seven is not a current app setting. + +**Acceptance mindset:** write down an observable outcome before coding. For example, under the proposed policy, a reader near exhaustion should gain future eligible content even when the topic already has 25 shared concepts, while a second reader continues seeing the same shared catalog. + +Sources: `backend/app/services/selection.py`; `backend/app/services/prefetch.py`; `backend/app/services/generation_budget.py`; `backend/app/services/reminders.py`. + +# 37 | Glossary: beginner words to advanced concepts + +| Term | Plain explanation in this app | +| --- | --- | +| Frontend / backend | What runs on the user's device / the trusted service behind it. | +| API / endpoint | A network contract / a specific path and method implementing it. | +| JSON | Structured text used for request and response data. | +| JWT / bearer token | Signed identity claims / a credential sent with an API request. | +| JWKS / ES256 | Public verification-key set / the selected signature algorithm. | +| Authentication / authorization | Prove who the caller is / decide which records or actions they may access. | +| PostgreSQL / SQL | The relational database / the language used to query and change it. | +| UUID / slug | A unique database identifier / a readable stable identifier used in routes. | +| Foreign key | A database relationship that points to an existing row. | +| Unique constraint | A rule the database enforces even when requests race. | +| RLS | Row Level Security: database policies limiting client-visible records. | +| Transaction / commit | A group of related changes / making those changes durable together. | +| Idempotency | Repeating an action keeps the intended state, such as saved remaining saved. | +| Race condition | Concurrent operations whose timing can change the result without protection. | +| SKIP LOCKED | Let one worker skip another worker's claimed row instead of duplicating that task. | +| Cache / stale | A local copy / a copy that may no longer match the server. | +| Optimistic UI | Show the intended change before the server confirms it. | +| Outbox / replay | Persisted pending actions / retrying them later. | +| Epoch guard | A generation marker rejecting a callback from an old account/cache lifecycle. | +| Backoff | Wait longer between repeated failures instead of hammering the service. | +| Cron / worker | A timed launch rule / a process that performs background work. | +| Push token / push receipt | Device delivery address / downstream delivery feedback. | +| SMTP | Email transport used by the configured account-email sender. | +| Migration | An ordered database change with an immutable history after application. | +| CI / CD | Automated checks / automated delivery or deployment steps. | +| OTA / APK | Downloaded compatible app update / Android installation binary. | +| Runtime version / channel | Native compatibility identifier / chosen update stream. | +| Watermark / runway | Threshold prompting work / how much eligible content remains for a reader. | + +**Three different "tokens":** a Gemini output token is a unit of model text processing; an access token is an account credential; a push token is a delivery address. They are unrelated and should never be substituted for one another. + +Sources: `docs/CODEBASE_MAP.md`; `backend/app/core/security.py`; `backend/app/services/pool.py`; `mobile/src/services/mutationOutbox.ts`. + +# 38 | Code atlas: where to look when you forget + +All repository source links in this handbook are pinned to the inspected base revision `3cc5af3382ba8e735be573d0ccb7f79ef6af9181`. This avoids a future branch edit silently changing the evidence behind an explanation. Paths are relative to the repository root. + +| If you want to understand... | Start here | +| --- | --- | +| Screen composition and providers | `mobile/App.tsx` | +| UI appearance | `mobile/src/theme/index.ts`, `mobile/src/components/` | +| Auth, sign-in, signup, sign-out | `mobile/src/context/AuthContext.tsx` | +| Session recovery and secure storage | `mobile/src/services/authSession.ts`, `mobile/src/lib/secureStorage.ts` | +| API headers, timeout and connectivity | `mobile/src/api/client.ts`, `mobile/src/api/fetchWithTimeout.ts` | +| Shared screen state | `mobile/src/context/ProgressContext.tsx` | +| Server cache and action replay | `mobile/src/services/remoteProgressRepository.ts` | +| Full offline reading | `mobile/src/services/conceptApi.ts`, `mobile/src/services/offlineCache.ts` | +| Saved paging and topic choices | `mobile/src/hooks/useSavedConcepts.ts`, `mobile/src/services/topicStore.ts` | +| Navigation to backend routes | `backend/app/api/v1/router.py` and adjacent route files | +| Daily lesson algorithm | `backend/app/services/selection.py` | +| Learned state and streaks | `backend/app/services/interactions.py`, `backend/app/services/streaks.py` | +| Catalog writing and concurrency | `backend/app/services/generation.py`, `backend/app/services/pool.py` | +| Refill thresholds and shared budget | `backend/app/services/prefetch.py`, `backend/app/services/generation_budget.py` | +| Reminder selection and delivery | `backend/app/services/reminders.py`, `mobile/src/services/notifications.ts` | +| Schema, constraints and SQL triggers | `backend/migrations/` | +| Public account pages and email HTML | `backend/app/api/v1/pages.py`, `backend/email-templates/` | +| Production release behavior | `RELEASING.md`, `.github/workflows/release.yml` | +| Native runtime and build profiles | `mobile/app.config.js`, `mobile/eas.json` | +| Recorded tests and operating decisions | `docs/WORK_LOG.md`, `mobile/tests/README.md`, `backend/tests/` | + +Older comments and documents sometimes describe a local-only client, synchronous on-demand generation or one query for the whole startup response. Current execution paths take precedence: authenticated mobile state is remote, generation is off the daily response path, and folded daily selection uses extra database queries. + +Sources: `docs/CODEBASE_MAP.md`; `docs/WORK_LOG.md`. + +# 39 | Evidence, references and what was verified + +**Snapshot date: 13 September 2026, Asia/Karachi.** The handbook describes the source tree at develop `3cc5af3`, whose tree matched the released application inspected here. Production main was `2e537f6`. Configuration defaults are labeled as defaults; secrets and real user records were not read into the report. + +| Evidence level | Established for this report | +| --- | --- | +| Source inspected | Mobile stack/screens/state, Auth, API, SQL, selection, generation, reminders, caches and release workflows | +| GitHub checked live | #191 and #193 merged; v1.8.0 exists; release/migration/APK workflow success; audit workflow failure; #187 open draft | +| Historical record | 1.8.0 tests, migration 0010 production verification, owner-reported template installation | +| Not verified live | Railway deployment/cron settings, current catalog counts, model/key overrides, provider limits for this account, SMTP delivery, push credentials/receipts, phone adoption | +| Not executed | Production SQL, real Gemini generation, real reminders/emails, mobile/backend regression suites for this documentation-only task | + +**GitHub evidence:** [release PR #191](https://github.com/Coding-Moves/one-concept/pull/191), [migration PR #193](https://github.com/Coding-Moves/one-concept/pull/193), [v1.8.0 release](https://github.com/Coding-Moves/one-concept/releases/tag/v1.8.0), [release workflow](https://github.com/Coding-Moves/one-concept/actions/runs/34714212293), [migration check](https://github.com/Coding-Moves/one-concept/actions/runs/34714212291), [APK workflow](https://github.com/Coding-Moves/one-concept/actions/runs/34714318696), [failed audit](https://github.com/Coding-Moves/one-concept/actions/runs/34714212289), [deferred email setup #187](https://github.com/Coding-Moves/one-concept/pull/187). A successful APK workflow can mean its runtime gate skipped rebuilding. + +Provider documentation checked on the snapshot date supports the platform behavior, while repository source determines how this app uses it: + +- [Expo push overview](https://docs.expo.dev/push-notifications/overview/): Expo routes notifications through FCM/APNs. +- [Expo runtime versions](https://docs.expo.dev/eas-update/runtime-versions/): native/update compatibility and runtime discipline. +- [Supabase SMTP](https://supabase.com/docs/guides/auth/auth-smtp): default-sender restrictions and custom transport. +- [Supabase API keys](https://supabase.com/docs/guides/getting-started/api-keys): public component keys versus elevated server keys and user identity. +- [Gemini rate limits](https://ai.google.dev/gemini-api/docs/rate-limits): project/model limits and midnight Pacific daily reset. +- [Railway cron jobs](https://docs.railway.com/cron-jobs): externally configured UTC schedules, exit requirements and timing limits. + +Diagrams are explanatory models drawn from these flows, not screenshots or measurements of live infrastructure. User names, week schedules and numeric runway exercises are examples. Recommendations are separated from implemented behavior. The most important finding is the distinction between shared published stock and each learner's unassigned stock. + +Sources: `docs/WORK_LOG.md`; `docs/CODEBASE_MAP.md`; `mobile/app.config.js`; `backend/app/config.py`. From 159b564930186d2d4388ceab29bf841d87b96c0f Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:14:09 +0500 Subject: [PATCH 03/36] docs: compare issue 195 sustainable learning design and current behavior --- docs/WORK_LOG.md | 5 + docs/handbook/ONE_CONCEPT_HANDBOOK.md | 210 +++++++++++++++++++++++++- 2 files changed, 214 insertions(+), 1 deletion(-) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 1771d88..b86ba34 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -23,6 +23,11 @@ claims as completed work. - **Evidence:** inspect current implementation before prose, verify relevant provider documentation, distinguish code/defaults from live service settings, and omit credential values. Render and inspect every final PDF page. +- **Scope addition:** the owner requested discussion of future issue #195 and + concluding before/after Q&A. Read the open issue and its dated production + inventory; explain its five proposed work areas, benefits, tradeoffs and + undecided parameters. This is design documentation, not authorization to + implement #195, generate content, change production or close that issue. ## Previous release handoff (historical) diff --git a/docs/handbook/ONE_CONCEPT_HANDBOOK.md b/docs/handbook/ONE_CONCEPT_HANDBOOK.md index 0a17bec..e5f0682 100644 --- a/docs/handbook/ONE_CONCEPT_HANDBOOK.md +++ b/docs/handbook/ONE_CONCEPT_HANDBOOK.md @@ -843,7 +843,7 @@ Use this as a one-week self-study plan. It is part of the PDF, not an automation | 4 - Catalog and AI | 18-21 | Reproduce the 25-concept example on paper. Explain why unread=0 does not mean published=0. | | 5 - Notifications | 22-25 | Draw registration and delivery separately. Walk through the three reminder times and an offline completion. | | 6 - Offline and release | 26-30 | Trace an offline save through restart/replay, then a change from develop to compatible production OTA. | -| 7 - Engineering judgment | 31-39 | Explain costs, current limits and evidence boundaries. Present a prioritized improvement proposal. | +| 7 - Engineering judgment | 31-47 | Explain current limits, then trace issue #195's future design and answer the before/after scenarios. Allow extra study time for this final day. | **Weekly recap to keep:** the mobile app is the presentation and cache; Supabase Auth establishes identity; FastAPI owns application rules; PostgreSQL owns durable records and concurrency constraints; Gemini writes shared content; Python workers decide refills/reminders; Expo and platform push services deliver notifications; GitHub/EAS/Railway deliver software. @@ -955,6 +955,7 @@ Sources: `docs/CODEBASE_MAP.md`; `docs/WORK_LOG.md`. | Source inspected | Mobile stack/screens/state, Auth, API, SQL, selection, generation, reminders, caches and release workflows | | GitHub checked live | #191 and #193 merged; v1.8.0 exists; release/migration/APK workflow success; audit workflow failure; #187 open draft | | Historical record | 1.8.0 tests, migration 0010 production verification, owner-reported template installation | +| Issue #195 checked live | Open future plan; its dated 125-lesson inventory is attributed to the issue in chapter 40, not independently re-queried here | | Not verified live | Railway deployment/cron settings, current catalog counts, model/key overrides, provider limits for this account, SMTP delivery, push credentials/receipts, phone adoption | | Not executed | Production SQL, real Gemini generation, real reminders/emails, mobile/backend regression suites for this documentation-only task | @@ -972,3 +973,210 @@ Provider documentation checked on the snapshot date supports the platform behavi Diagrams are explanatory models drawn from these flows, not screenshots or measurements of live infrastructure. User names, week schedules and numeric runway exercises are examples. Recommendations are separated from implemented behavior. The most important finding is the distinction between shared published stock and each learner's unassigned stock. Sources: `docs/WORK_LOG.md`; `docs/CODEBASE_MAP.md`; `mobile/app.config.js`; `backend/app/config.py`. + +# 40 | Issue #195: sustainable learning beyond exhaustion + +**Status: proposed future design, open issue, not implemented by this handbook.** Issue #195 is titled "[P1] Sustain daily learning with continuous replenishment, curriculum growth, and review." It addresses the finite-catalog failure as a content lifecycle problem, rather than increasing 25 to another fixed ceiling. + +The issue reports a read-only production inventory dated 13 September 2026. This table reproduces that issue's reported snapshot; this PDF task did not independently query production. It does not prove the current worker schedule or generation switches are enabled. + +| Subject | Published | Pending titles | Failed items | +| --- | --- | --- | --- | +| Artificial Intelligence | 25 | 5 | 4 | +| Computer Science | 25 | 3 | 6 | +| Linux & Systems | 25 | 3 | 6 | +| Mathematics | 25 | 5 | 4 | +| Software Engineering | 25 | 3 | 6 | +| Total | 125 | 19 | 26 | + +The 150-title original queue consists of 105 generated entries, 19 pending and 26 failed entries. Together with the 20 initial seeded lessons, that explains 125 published lessons. Pending and failed rows are not ready-to-read stock. The supply is finite even if every remaining title could eventually be published. + + + +The five work areas are: ongoing replenishment based on reader availability; an expandable reviewed curriculum; useful review/exploration when fresh content is unavailable; preservation of the shared library and bounded costs; and protected operational reporting with actionable alerts. + +**Benefit for learners:** a temporary writing or provider outage need not turn Today into a dead end. **Benefit for the owner:** content demand, queue health and publishing become visible and manageable. **Benefit for engineering:** the proposal preserves JWT identity, backend writes, no repeated new assignments and shared budgets, while introducing explicit records for the new behaviors. + +The issue requests eventual implementation in one cohesive PR into develop with focused commits and regression coverage. This handbook only explains that design; its documentation PR is not the implementation PR for #195. + +Future-design source: [Issue #195](https://github.com/Coding-Moves/one-concept/issues/195), read 13 September 2026; no comments were present. + +# 41 | Future replenishment: reserve, demand and cadence + +**The proposed change:** keep the initial published-catalog target separate from ongoing reader supply. Use the daily selector's eligibility rule when measuring unseen content: previously assigned concepts do not count as new, even when they were skipped. + + + +The issue suggests considering a **60-90-day reserve per subject for experienced active readers**. This is a starting proposal, not a configured guarantee. The implementation must choose an affordable target, an active-reader definition, consumption assumptions, and the warning threshold before enabling generation. + +| Measure | Illustrative meaning, not a chosen production setting | +| --- | --- | +| Remaining eligible stock | 12 approved concepts never assigned to an experienced reader in a subject | +| Consumption assumption | 1 new concept from that subject per day for a subject-only daily reader | +| Estimated reserve | 12 / 1 = 12 days; a reader with several follows consumes each subject differently | +| Reserve target | A chosen goal such as 60 days; the issue also suggests considering 90 | +| Refill gap | At 1/day, growing 12 to 60 requires 48 additional eligible approved concepts | +| Sustainable cadence | Ongoing approved publication must match or exceed relevant consumption, or the reserve shrinks | + +If consumption is one/day and approved publication is one/day, the reserve stays roughly level; it does not build a depleted reserve back to 60. If publication is two/day during recovery, net growth is one/day before failed drafts and other constraints. These are simple planning examples, not projected real throughput. With no approved output, a 60-day reserve buys about 60 such consumption days, not unlimited new content. + +**Durable coalescing:** store one bounded subject refill goal/work request, or use an equivalent durable scheduled mechanism. Ten users and three API processes reporting low supply should update the same bounded objective, rather than add thirteen new batches or repeatedly raise the target. Workers claim work safely and consume the existing shared budget. + +**Avoid misleading averages:** new accounts may have 125 unseen concepts while long-time readers have zero. An average across all users hides that failure. The final design needs a definition of experienced active readers and an explicit aggregation rule, such as a selected low-availability cohort or percentile. Those are design options, not decisions already made in #195. + +Daily API reads continue returning stored content quickly. Supply aggregation belongs in bounded background work, not a scan of every user's history on every phone open. + +Future-design source: [Issue #195, work areas 1 and 4](https://github.com/Coding-Moves/one-concept/issues/195). + +# 42 | Future curriculum: plan, draft, review, publish + +The proposal replaces a finite list of titles with a repeatable curriculum-maintenance process. Each subject can contain foundations, intermediate concepts and advanced applications, with learning objectives, stable identities, difficulty and prerequisites where useful. + + + +| Stage | What an operator does | What becomes learner-visible | +| --- | --- | --- | +| Plan/import titles | Extend subject coverage in reviewed batches; check exact and likely semantic duplicates | Nothing yet; a title is not a lesson | +| Draft | Use AI assistance or editorial writing with references and version information | Draft stays separate from published material | +| Review | Check explanation/example accuracy, usefulness, objective and overlap | Only approved content becomes eligible | +| Publish | Release an approved version into the shared catalog | Eligible users can receive it on later selections | +| Correct | Review a revision while existing content remains available | Stable references survive; approved correction can replace the current reading version | + +Different titles can describe the same idea. Unique slugs catch exact identifiers but do not establish semantic novelty. The proposed workflow should flag likely overlaps for review instead of counting a reworded duplicate as another day of meaningful new learning. + +**Handling the 26 failed items:** group failures by cause, inspect the relevant records and fix the cause before retrying. A stale claim, duplicate slug, provider failure and invalid text need different responses. Preserve attempts and budget rules; blindly resetting every failed row would risk repeated spending without improving quality. + +**Stable identities and versions:** keep existing concept IDs/slugs, saved references and assignment history. A corrected explanation should not create a fake "new concept" just to gain another catalog count. Explicit content versions make it possible to reason about cached older text, editorial history and what a learner reviewed. The exact version schema and cache invalidation policy remain implementation decisions. + +The issue asks for a responsible operator and recurring content maintenance. Sustainable generation also requires sustainable title planning and review capacity. Producing hundreds of unreviewed drafts does not satisfy a reserve of approved lessons. + +Prerequisites in curriculum metadata do not automatically create an ordered or adaptive daily selector. The implementation must define how, if at all, prerequisites affect eligibility and existing users. The issue establishes structured curriculum goals; it does not specify a complete adaptive teaching algorithm. + +Future-design source: [Issue #195, work area 2](https://github.com/Coding-Moves/one-concept/issues/195). + +# 43 | Future daily practice and honest progress metrics + +When a fresh lesson is unavailable, Today should offer **Review a previous lesson** and **Explore another subject** when those options exist. Reviews should draw from previously completed lessons, favor ideas not reviewed recently, and clearly identify the activity as review. + + + +| Metric or record | Today | Proposed #195 behavior | +| --- | --- | --- | +| New assignment | One new concept per user/date; never repeat a concept for that user | Preserve these constraints | +| Review activity | No distinct durable review model | Add a separate persistent activity/review record | +| Unique concepts learned | Completed new assignments | Review does not increase this count | +| Review total | Not tracked separately | Count completed review activities separately | +| Daily learning streak | Consecutive completed assignment dates | A completed new lesson or qualifying review can satisfy the day's activity | +| Mere card open | Not learned | Still not a completed activity | + +**Example:** Ali has 125 unique concepts learned and a 10-day streak. On day 11 there is no fresh eligible lesson, so he completes a labeled review. The intended result is 125 unique concepts, one additional completed review, and an 11-day activity streak. Sara receives a fresh lesson and completes it; her unique count rises by one. Their activities differ, but both practiced that day. + +A useful conceptual model is a separate activity ledger referencing the user, concept, activity kind, assigned local date and completion state. That is an explanatory model, not a final table or API contract. Exact SQL/API design is not fixed in the issue. The important guarantees are durable identity, stable daily activity, idempotent completion and no duplicate new-concept assignment. + +Changing streak semantics requires explicit documentation and compatibility handling. Preserve past assignments and completed days. For the new metric, completed new/review activities can contribute a distinct set of local dates; two completions on the same date must not earn two streak days. Show the difference between unique knowledge coverage and repeated practice. + +If a user has no completed history, there may be nothing eligible to review. The app must show an honest unavailable state and retry/exploration options rather than invent content or promise a new lesson at an unverified time. The no-repeat promise still applies to **new assignments**, while repeats are intentional and labeled in review. + +Future-design source: [Issue #195, work area 3](https://github.com/Coding-Moves/one-concept/issues/195). + +# 44 | Future review flow: two devices, offline and rollout + + + +The intended review experience should retain the app's current responsiveness without weakening server authority. A completed review needs one logical identity that can survive retries, two-device access and offline replay. A client timestamp must not become permission to backdate arbitrary learning. + +| Scenario | Proposed required behavior | +| --- | --- | +| Two devices open the same day's activity | Both resolve a stable activity; no competing daily choices that overwrite one another. | +| A completion is tapped twice | One logical completion and one day's streak credit. | +| A cached review is completed offline | Persist account-scoped intent and reconcile safely when allowed by the defined day/grace rules. | +| New content appears during review | Keep the activity in progress stable; do not swap cards or count the day twice. | +| Sign-out occurs during replay | Clear account data and reject late old-account callbacks. | +| Reviewed lesson is corrected | Preserve concept references and apply an explicit content-version/cache policy. | +| Neither fresh content nor a cached review body exists | Honest unavailable/retry state; no invented lesson body. | + +**Decisions still needed:** whether offline review eligibility/activity identity is prepared during an earlier online session; exactly how review day/grace rules map to the current completion behavior; whether one active activity is allowed per user/day or more activities share one daily credit; how delayed completions are reconciled. Issue #195 demands consistent, idempotent behavior but does not settle every protocol detail. + +A practical rollout sequence would be: add new immutable migrations and backward-compatible backend support; preserve legacy assignments and metrics; release the new client/cache/outbox behavior; verify controlled scenarios; enable the new replenishment/review policy gradually with measured limits. This sequence is an engineering recommendation, not a deployment performed here. + +Old APKs and offline queues must retain valid semantics during the transition. Existing `POST /daily/complete` behavior cannot silently start completing unrelated reviews for legacy clients. New activity contracts should make the target clear. Native changes, if implementation introduces them, require the native release procedure; a review feature does not automatically mean a new APK is required. + +The issue requires at least a year of simulated multi-user activity with outages, replenished planning queues, different follows and bounded generation. That long simulation checks whether the design stays useful beyond the original catalog, not just whether day 126 happens to pass. + +Future-design source: [Issue #195, work area 3 and implementation plan](https://github.com/Coding-Moves/one-concept/issues/195). + +# 45 | Future content operations: visible and bounded + +Sustainability needs both product behavior and an operating process. Issue #195 asks for a minimal protected operator report/view, a responsible maintainer and a runbook. It does not require a separate large analytics platform; broader crash/metrics work remains linked to #161. + + + +| Signal | What it helps the owner decide | +| --- | --- | +| Published/approved supply and reader availability | Whether content exists but engaged readers have exhausted it | +| Estimated reserve days and low watermark | Whether planned publication can keep pace with consumption | +| Planned/pending/failed/stale work | Whether the bottleneck is ideas, writing, validation or a stopped worker | +| Last successful generation and publication | Whether drafts are being produced but no approved content is reaching users | +| Daily reserved calls, retry/limit state | Whether provider spending or quota is stopping progress | +| Generation enabled and schedule health | Whether the system is deliberately paused or unexpectedly inactive | + +Alerts should identify actionable conditions, deduplicate repeated failures and report recovery. A routine healthy pass should not repeatedly notify the operator. This is proposed **operator alerting**, separate from current learner reminders and Supabase account emails. The issue does not select an alert transport/provider or authorize sending new messages today. + +The runbook should cover title import, review/publication, correction, failure inspection, schedule/config verification, pausing generation and restoration from a tested backup. Reports must be restricted to authorized maintainers and avoid exposed credentials, sensitive raw errors and user-identifying details. + +**Cost tradeoff:** reviews let the product stay useful without forcing a paid model call whenever a learner opens Today. New approved lessons still serve every eligible reader. A reserve and extra editorial stages increase storage, operator effort and planned generation. The shared Pacific-day ledger remains in place; a 60-90-day goal does not override an affordable daily cap. + +**How I would judge success:** experienced readers have a monitored supply buffer; supply failures produce clear operational causes; outages allow eligible review practice; unique learning counts remain honest; repeated opens/workers cannot grow jobs without bound. A permanently empty title queue plus an ever-growing counter is not sustainability. Neither is a large pile of unreviewed drafts. + +Future-design source: [Issue #195, work areas 4 and 5](https://github.com/Coding-Moves/one-concept/issues/195). + +# 46 | Before and after: the design comparison + +The right-hand column describes the intended outcome of issue #195 after a complete, validated implementation. It is not the current app and should not be presented to learners as an already-shipped promise. + +| Concern | Before: current source | After: intended #195 design | +| --- | --- | --- | +| Meaning of 25 | Shared target can stop refilling | Initial catalog size separated from continuing reserve/cadence | +| Content supply signal | Global stock plus a personal trigger with a global stop gate | Experienced active readers' eligible supply, measured in background | +| Many users run low | Process-local prefetch deduplication | Durable coalesced per-subject work with a bounded persistent goal | +| Planned curriculum | Finite seeded titles | Repeatable reviewed import/extension workflow | +| AI quality | Format/style checks then publication | Draft separated from approved publication, with factual/usefulness review | +| Same idea, different title | Unique slugs alone do not detect semantic overlap | Likely duplicates surfaced for review | +| Fresh catalog exhausted | Global fallback, then no daily activity | Honest review/explore choices when available | +| Repetition | Cannot create a repeat new assignment | Preserve that rule; intentional labeled reviews use separate records | +| Streak during content gap | No assignment completion can break the run | A completed eligible review can provide daily activity credit | +| Unique learned total | New completions | Still unique new completions; review total shown separately | +| Content corrections | Shared body can change on refresh | Stable concept identity plus explicit review/version history | +| Offline review | No distinct review workflow | Cached bodies and account-scoped idempotent review replay | +| Model outage | Existing unseen content works until exhausted | Stored fresh content or eligible review; no synchronous model wait | +| Owner visibility | Logs and existing counters | Protected supply/queue/budget/worker report and actionable alerts | +| Spending control | Existing shared cap, switches and claims | Preserve them; reserve goals cannot create unbounded calls | + +**Decisions to document before enabling the future system:** the active/experienced-reader definition; reserve target and publication assumptions; semantic-duplicate review process; approval ownership; activity identity and timezone/grace contract; metric names; compatibility and cache-version policy; alert transport and thresholds; rollout/backup/disable procedure. + +**Benefit with a limit:** the proposal makes learning more resilient and content maintenance more deliberate. It cannot guarantee an endless stream of accurate fresh lessons with no editorial labor, no provider budget and no planned curriculum. Reviews are a useful fallback and learning activity, not evidence that new supply is healthy. + +Future-design source: [Issue #195](https://github.com/Coding-Moves/one-concept/issues/195); interpretations and rollout suggestions in chapters 41-45 are explicitly labeled. + +# 47 | Final Q&A: before and after issue #195 + +| Question | Before: what happens now? | After: what #195 intends | +| --- | --- | --- | +| I finish all 25 AI lessons. What is tomorrow? | Other eligible followed/global content, or exhaustion. Refill can stop because shared stock is already 25. | Reader availability can drive bounded refill; if fresh content is not ready, offer eligible review/exploration. | +| Do Ali and Sara still get different lessons? | They can; their follows and histories produce separate assignments. | Yes, progress stays personal and content stays shared. Review and new activities may also differ. | +| Do we generate a new lesson separately for each person? | No, content is shared. | Still no. Grow the curriculum once and reuse each approved concept across eligible readers. | +| What if everyone runs low together? | Several processes can trigger work; database claims protect backlog rows and budget. | Coalesce demand into durable bounded subject goals as well as preserving claim/budget protection. | +| Can the app repeat something I learned? | Not as another new daily assignment. Saved reading is manual revisiting. | Yes as an explicit review activity, using a separate record. New-assignment no-repeat stays intact. | +| Does a review turn 125 learned into 126? | There is no separate review completion. | No. Unique learned stays 125; the review total changes and the day can count toward activity streak. | +| Will my streak survive a Gemini outage? | Only while a new eligible assignment can be completed; exhaustion has no practice completion. | A qualifying completed review can preserve activity continuity. No review history/body may still mean unavailable. | +| Can I just open a review and get credit? | Opening is not a completed learned action. | Still no. Explicit completion is required. | +| What if a new lesson appears during review? | No separate review session exists. | Keep the active review stable. Use fresh content on a later eligible selection without double credit. | +| Can I review offline on two devices? | Existing offline queue handles current likes/saves/follows and same-day learned actions. | The new review protocol must support durable identity, safe replay and one logical completion under defined date rules. | +| Does 90 days of reserve mean infinite lessons? | No reserve-day policy exists. | No. It is a proposed buffer that shrinks if approved publication falls behind consumption. | +| Will this add a weekly learning email? | No learning-email digest exists. | #195 does not request that feature. Its operational alerts and review practice are different. | +| Will finishing a review stop reminders? | Current reminders check completed assignments only. | Reminder eligibility must be explicitly integrated with qualifying activity completion if it should stop after review; the exact contract must be decided and tested. | +| Is this future design built already? | No; issue #195 is open. | Only a complete implementation, migration/compatibility rollout and acceptance evidence can make these outcomes real. | + +**What to remember:** today the app protects a daily new-concept assignment. Issue #195 would preserve that guarantee while adding sustainable supply management and a separate daily practice path. The final reminder answer identifies an integration decision needed to keep the new streak/activity meaning consistent across the product; it is not a feature already specified in detail or implemented. + +Future-design source: [Issue #195](https://github.com/Coding-Moves/one-concept/issues/195), including its acceptance criteria. From eff0673f3e50ac14a0c0d9b589030d5b25fdbe7b Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:16:52 +0500 Subject: [PATCH 04/36] docs: add printable handbook renderer and vector flowcharts --- .gitignore | 4 + docs/CODEBASE_MAP.md | 1 + docs/handbook/README.md | 52 ++++ docs/handbook/build_pdf.py | 609 +++++++++++++++++++++++++++++++++++++ 4 files changed, 666 insertions(+) create mode 100644 docs/handbook/README.md create mode 100644 docs/handbook/build_pdf.py diff --git a/.gitignore b/.gitignore index 06baa9f..c7ab573 100644 --- a/.gitignore +++ b/.gitignore @@ -18,3 +18,7 @@ venv/ # Firebase service-account keys are real secrets — never commit them *firebase-adminsdk*.json + +# Generated handbook PDFs and render/QA intermediates +/output/pdf/ +/tmp/pdfs/ diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index ce4544c..03418f2 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -19,6 +19,7 @@ learned history, streaks, likes, saved concepts, and push reminders. | Documentation | Root `README.md`, `RELEASING.md`, `CONTRIBUTING.md`, `docs/ARCHITECTURE.md`, `docs/ROADMAP.md`, and the backend/mobile guides. | | Agent guidance | Root `AGENTS.md`; `mobile/AGENTS.md` adds Expo documentation requirements and `mobile/CLAUDE.md` references it. | | Authentication email | `backend/email-templates/` contains branded signup, recovery, and password-changed HTML; `docs/EMAIL_TEMPLATES.md` covers manual Supabase installation and activation checks. Templates use the configured sender and are not installed by app deployment. | +| Engineering handbook | `docs/handbook/ONE_CONCEPT_HANDBOOK.md` explains the full stack and learning lifecycle; `docs/handbook/build_pdf.py` renders the printable guide with vector diagrams. Build and verification instructions are in `docs/handbook/README.md`. | ## Mobile navigation and presentation diff --git a/docs/handbook/README.md b/docs/handbook/README.md new file mode 100644 index 0000000..78cee34 --- /dev/null +++ b/docs/handbook/README.md @@ -0,0 +1,52 @@ +# One Concept engineering handbook + +The [handbook source](ONE_CONCEPT_HANDBOOK.md) covers the mobile client, identity, +API, database, daily selection, generation, reminders, account emails, offline +state, releases and operational tradeoffs. It includes a seven-day study digest +and scenario answers. Chapters 40-47 discuss the proposed sustainable lifecycle +in issue #195, its benefits and open decisions, ending with before/after Q&A. +That design is described as future work, not an implemented feature. +The source snapshot and live-evidence limits are recorded +in chapter 39. Chapter 18 explains the current shared-count refill limitation; +it does not claim that the limitation has been fixed. + +## Build the PDF + +Requires Python 3, `reportlab`, and Liberation Sans/Mono TrueType fonts. From the +repository root: + +```bash +python docs/handbook/build_pdf.py +``` + +Use `--font-dir /path/to/fonts` if fonts are not in a detected system directory. +The directory must contain `LiberationSans-Regular.ttf`, +`LiberationSans-Bold.ttf`, `LiberationSans-Italic.ttf`, and +`LiberationMono-Regular.ttf` (or `DejaVuSansMono.ttf`). The builder also recognizes +the bundled document runtime's font directory when available. + +Default outputs are `output/pdf/one-concept-engineering-handbook.pdf` and +`tmp/pdfs/handbook-layout.json`. `--output` and `--qa-report` override these paths. +Generated PDF and QA files are not committed. The PDF includes selectable text, +clickable contents, bookmarks, pinned repository source links and vector diagrams. + +## Update and verify + +Each numbered top-level heading starts one chapter/page. The small Markdown +subset is paragraphs, numbered or hyphen list items, pipe tables, inline bold, +code and HTTPS links. `` inserts a named vector diagram +from the builder. Keep the renderer's chapter/figure assertions and cover counts +aligned when changing structure. The source revision used for links is explicit +in the renderer and handbook; update it only after verifying the new source. + +The builder measures text, tables and diagram labels, rejects overflow, and +records per-page font scale and remaining space. It does not replace visual QA. +Render every final page, inspect the images and verify extracted text and links: + +```bash +pdftoppm -scale-to 1500 -png output/pdf/one-concept-engineering-handbook.pdf tmp/pdfs/page +git diff --check +``` + +Application tests are not needed for prose/layout edits. Changes to the app +itself follow the repository's normal validation and release requirements. diff --git a/docs/handbook/build_pdf.py b/docs/handbook/build_pdf.py new file mode 100644 index 0000000..2987d3f --- /dev/null +++ b/docs/handbook/build_pdf.py @@ -0,0 +1,609 @@ +"""Render the engineering handbook with selectable text and vector diagrams. + +Usage: python docs/handbook/build_pdf.py +Requires reportlab. See README.md in this directory for reproducible setup. +""" + +from __future__ import annotations + +import argparse +import html +import json +import math +from pathlib import Path +import re + +from reportlab.lib import colors +from reportlab.lib.enums import TA_CENTER +from reportlab.lib.pagesizes import A4 +from reportlab.lib.styles import ParagraphStyle +from reportlab.pdfbase import pdfmetrics +from reportlab.pdfbase.ttfonts import TTFont +from reportlab.pdfgen import canvas +from reportlab.platypus import Flowable, Paragraph, Spacer, Table, TableStyle + +ROOT = Path(__file__).resolve().parents[2] +HERE = Path(__file__).resolve().parent +REV = "3cc5af3382ba8e735be573d0ccb7f79ef6af9181" +REPO = "https://github.com/Coding-Moves/one-concept" +CHAPTERS = 47 +DIAGRAMS = 29 +PW, PH = A4 +MARGIN = 43 +WIDTH = PW - 2 * MARGIN +NAVY = colors.HexColor("#162D45") +TEAL = colors.HexColor("#087F82") +INK = colors.HexColor("#25394B") +MUTED = colors.HexColor("#5C6F7F") +PALE = colors.HexColor("#F2F6F8") +LINE = colors.HexColor("#DCE5EA") +AMBER = colors.HexColor("#A36810") + + +def fonts(font_dir: Path | None): + runtime = Path.home() / ".cache/codex-runtimes/codex-primary-runtime/dependencies" + options = [font_dir] if font_dir else [] + options += [Path("/usr/share/fonts/truetype/liberation2"), + Path("/usr/share/fonts/truetype/liberation"), + runtime / "native/libreoffice-headless/libreoffice/share/fonts/truetype"] + for directory in options: + if directory and (directory / "LiberationSans-Regular.ttf").is_file(): + for name, filename in [("Body", "LiberationSans-Regular.ttf"), + ("Bold", "LiberationSans-Bold.ttf"), + ("Italic", "LiberationSans-Italic.ttf"), + ("Mono", "LiberationMono-Regular.ttf")]: + path = directory / filename + if name == "Mono" and not path.exists(): + path = directory / "DejaVuSansMono.ttf" + pdfmetrics.registerFont(TTFont(name, str(path))) + pdfmetrics.registerFontFamily("Body", normal="Body", bold="Bold", italic="Italic") + return + raise RuntimeError("Install Liberation fonts or pass --font-dir containing Liberation Sans and Mono.") + + +def inline(text: str) -> str: + """Only render the small Markdown subset used by the source, escaping first.""" + text = html.escape(text) + text = re.sub(r"\[([^\]]+)\]\((https://[^ )]+)\)", + r'\1', text) + text = re.sub(r"\*\*(.+?)\*\*", r"\1", text) + + def code(match): + raw = html.unescape(match[1]) + p = ROOT / raw + label = f'{match[1]}' + if p.exists() and ("/" in raw or raw.endswith(".md")): + kind = "tree" if p.is_dir() else "blob" + return f'{label}' + return label + + return re.sub(r"`([^`]+)`", code, text) + + +def styles(scale=1): + return { + "body": ParagraphStyle("body", fontName="Body", fontSize=10.15 * scale, + leading=14.25 * scale, textColor=INK, spaceAfter=7 * scale), + "cell": ParagraphStyle("cell", fontName="Body", fontSize=9.0 * scale, + leading=11.85 * scale, textColor=INK), + "head": ParagraphStyle("head", fontName="Bold", fontSize=9.0 * scale, + leading=11.85 * scale, textColor=colors.white), + "source": ParagraphStyle("source", fontName="Body", fontSize=7.8 * scale, + leading=10.5 * scale, textColor=MUTED), + } + + +class Diagram(Flowable): + """Vector boxes/arrows; node labels are measured to prohibit clipping.""" + + def __init__(self, key, number, scale=1): + super().__init__() + self.key, self.number = key, number + self.factor = (WIDTH / 520) * scale + self.width = WIDTH + self.base_h = 188 if key not in {"selection", "database", "release"} else 215 + if key == "editorial": + self.base_h = 176 + self.height = self.base_h * self.factor + 8 + self.nodes = {} + + def box(self, key, x, y, w, h, title, body="", tone="teal"): + self.nodes[key] = (x, y, w, h, title, body, tone) + + def edge(self, a, b, label="", via=None): + ca = self.nodes[a][:4] + cb = self.nodes[b][:4] + ax, ay = ca[0] + ca[2] / 2, ca[1] + ca[3] / 2 + bx, by = cb[0] + cb[2] / 2, cb[1] + cb[3] / 2 + dx, dy = bx - ax, by - ay + if via: + points = via + else: + k1 = min(ca[2] / 2 / abs(dx) if dx else float("inf"), + ca[3] / 2 / abs(dy) if dy else float("inf")) + k2 = min(cb[2] / 2 / abs(dx) if dx else float("inf"), + cb[3] / 2 / abs(dy) if dy else float("inf")) + points = [(ax + k1 * dx, ay + k1 * dy), (bx - k2 * dx, by - k2 * dy)] + self.arrow(points, label) + + def arrow(self, points, label=""): + c = self.canv + pts = [(x, self.base_h - y) for x, y in points] + c.setStrokeColor(MUTED) + c.setFillColor(MUTED) + c.setLineWidth(0.9) + p = c.beginPath() + p.moveTo(*pts[0]) + for point in pts[1:]: + p.lineTo(*point) + c.drawPath(p) + (x1, y1), (x2, y2) = pts[-2:] + angle = math.atan2(y2 - y1, x2 - x1) + p = c.beginPath() + p.moveTo(x2, y2) + p.lineTo(x2 - 5 * math.cos(angle - .45), y2 - 5 * math.sin(angle - .45)) + p.lineTo(x2 - 5 * math.cos(angle + .45), y2 - 5 * math.sin(angle + .45)) + p.close() + c.drawPath(p, fill=1, stroke=0) + if label: + lx, ly = (pts[0][0] + pts[-1][0]) / 2, (pts[0][1] + pts[-1][1]) / 2 + 4 + c.setFont("Body", 7.5) + tw = pdfmetrics.stringWidth(label, "Body", 7.5) + c.setFillColor(PALE) + c.rect(lx - tw / 2 - 3, ly - 1, tw + 6, 10, fill=1, stroke=0) + c.setFillColor(MUTED) + c.drawCentredString(lx, ly, label) + + def note(self, text, y=171): + c = self.canv + c.setFont("Italic", 8) + c.setFillColor(MUTED) + c.drawCentredString(260, self.base_h - y, text) + + def row(self, items, y=43, height=49, gap=25): + w = (488 - gap * (len(items) - 1)) / len(items) + keys = [] + for i, item in enumerate(items): + key = str(len(self.nodes)) + self.box(key, 16 + i * (w + gap), y, w, height, *item) + keys.append(key) + for a, b in zip(keys, keys[1:]): + self.edge(a, b) + return keys + + def sequence(self, actors, steps): + c = self.canv + count = len(actors) + xs = [62 + i * 396 / (count - 1) for i in range(count)] + for i, name in enumerate(actors): + self.box(str(i), xs[i] - 49, 31, 98, 25, name) + c.setDash(2, 3) + c.setStrokeColor(LINE) + c.line(xs[i], self.base_h - 58, xs[i], 24) + c.setDash() + for i, (a, b, text) in enumerate(steps): + y = 75 + i * 22 + self.arrow([(xs[a], y), (xs[b], y)], text) + + def draw(self): + c = self.canv + c.saveState() + c.scale(self.factor, self.factor) + c.setFillColor(PALE) + c.roundRect(0, 0, 520, self.base_h, 9, fill=1, stroke=0) + c.setFillColor(TEAL) + c.setFont("Bold", 8) + title = self.key.replace("_", " ").upper() + c.drawString(14, self.base_h - 17, f"FIGURE {self.number:02d} / {title}") + b, e = self.box, self.edge + key = self.key + if key == "architecture": + b("m", 15, 35, 137, 46, "Mobile app", "UI, cache, outbox") + b("a", 191, 35, 137, 46, "FastAPI", "Trusted app rules") + b("d", 367, 35, 137, 46, "PostgreSQL", "Shared + personal data") + b("u", 15, 112, 137, 42, "Supabase Auth", "Sessions + account email") + b("w", 191, 112, 137, 42, "Python workers", "Refill + reminder decisions") + b("p", 367, 112, 137, 42, "Gemini / Expo Push", "Write content / route pushes") + e("m", "a", "JWT"); e("a", "d", "SQL") + e("u", "m", "identity"); e("w", "a", "shared services") + e("w", "p", "HTTPS"); e("d", "w", "job state") + elif key == "navigation": + self.row([("Launch", "Splash + session"), ("Auth or tabs", "Session gates access"), + ("Detail modal", "Open a lesson")], 40, 44) + for i, (title, body) in enumerate([("Today", "Daily lesson"), ("History", "Last 10 learned"), + ("Stats", "Totals + streaks"), ("Profile", "Saved, topics, settings")]): + b("tab" + str(i), 16 + i * 124, 116, 116, 40, title, body) + self.note("Profile contains Personalization, Saved and About.") + elif key == "startup": + self.sequence(["Device cache", "React state", "FastAPI", "PostgreSQL"], + [(0, 1, "paint last known state"), (1, 2, "GET state + token"), + (2, 3, "aggregate + get/create daily"), (2, 1, "reconcile with pending actions")]) + elif key == "auth": + self.sequence(["Phone", "Supabase Auth", "FastAPI", "Database"], + [(0, 1, "email + password"), (1, 0, "session / access token"), + (0, 2, "request with bearer JWT"), (2, 3, "verified user-scoped SQL")]) + elif key == "email": + self.row([("Account action", "Signup / recovery / change"), ("Supabase Auth", "Select template + link"), + ("Configured SMTP", "Deliver to inbox")], 47, 58) + self.note("HTML source in GitHub must be installed in Supabase separately.", 142) + self.note("Learning reminder pushes use a different system.", 160) + elif key == "recovery": + self.sequence(["App", "Supabase Auth", "Inbox", "Reset web page"], + [(0, 1, "request recovery"), (1, 2, "email verification link"), + (2, 3, "verified redirect + fragment"), (3, 1, "new password + recovery JWT")]) + elif key == "backend": + self.row([("HTTP request", "Bearer token + JSON"), ("FastAPI route", "Auth + schema"), + ("Service", "Selection / interaction"), ("Database", "Commit + response")], 48, 62, 17) + self.note("Generation requests run outside the daily response path.", 145) + elif key == "database": + for args in [("u", 16, 34, 140, 39, "auth.users", "Supabase-managed identity"), + ("p", 16, 105, 140, 39, "profiles", "One per user"), + ("a", 189, 105, 140, 39, "daily_assignments", "User + date + concept"), + ("c", 363, 105, 140, 39, "concepts", "Shared written lessons"), + ("t", 363, 34, 140, 39, "topics", "Shared subject catalog"), + ("f", 189, 34, 140, 39, "user_topics", "Follow relationships"), + ("n", 16, 172, 140, 30, "Preferences + tokens", ""), + ("i", 189, 172, 140, 30, "Interactions", "Like / save"), + ("o", 363, 172, 140, 30, "Operational tables", "Backlog, budget, claims")]: + b(*args) + for a, z, label in [("u", "p", "1:1"), ("p", "a", "1:many"), + ("t", "c", "1:many"), ("c", "a", "1:many"), + ("p", "f", "follows"), ("t", "f", "topic"), + ("p", "n", "owns"), ("p", "i", "owns")]: e(a, z, label) + elif key == "selection": + b("s", 17, 34, 152, 38, "Existing assignment?", "For this user's local day") + b("r", 348, 34, 155, 38, "Return stored lesson", "No re-selection") + b("f", 17, 95, 152, 38, "Followed candidates?", "Exclude all prior assignments") + b("p", 193, 95, 135, 38, "Pick candidate", "Topic recency + random") + b("i", 348, 95, 155, 38, "Persist assignment", "Unique user/day + user/concept") + b("g", 17, 159, 152, 38, "Global candidates?", "Same no-repeat rule") + b("x", 193, 159, 135, 38, "Exhausted", "No eligible global concept", "amber") + e("s", "r", "yes"); e("s", "f", "no"); e("f", "p", "yes") + e("p", "i"); e("f", "g", "no"); e("g", "x", "no") + e("g", "p", "yes", [(169, 178), (183, 178), (183, 114), (193, 114)]) + elif key == "multiuser": + b("c", 188, 36, 144, 42, "One shared catalog", "Lessons stay after reading") + b("a", 23, 112, 213, 42, "Ali: AI + Mathematics", "Own assignment and completion history") + b("s", 285, 112, 213, 42, "Sara: Software Engineering", "Own assignment and completion history") + e("c", "a", "eligible for Ali"); e("c", "s", "eligible for Sara") + elif key == "completion": + self.row([("Tap learned", "Immediate UI response"), ("POST complete", "Server picks eligible day"), + ("Persist timestamp", "Idempotent completion")], 43, 50) + b("s", 55, 121, 175, 36, "Derived streaks", "Completed assigned_for dates") + b("n", 291, 121, 175, 36, "Later reminder checks", "Completed day is excluded") + e("2", "n"); e("2", "s") + elif key == "exhaustion": + b("p", 17, 40, 148, 51, "Shared published = 25", "Reading does not delete rows") + b("a", 188, 40, 148, 51, "Ali remaining = 0", "All 25 previously assigned", "amber") + b("s", 359, 40, 148, 51, "Sara remaining = 20", "Only 5 previously assigned") + b("w", 17, 117, 148, 39, "Worker target = 25", "No shared deficit") + b("f", 188, 117, 148, 39, "Prefetch target = 10", "Already exceeded: no refill", "amber") + b("r", 359, 117, 148, 39, "Sara continues", "Her candidates remain") + e("p", "w"); e("a", "f"); e("s", "r") + elif key == "generation": + self.row([("Curated title", "Pending backlog"), ("Claim + budget", "Atomic DB transaction"), + ("Gemini request", "Commit precedes call")], 40, 48) + b("v", 190, 114, 143, 42, "Validate text", "Shape, length, phrasing") + b("p", 17, 114, 143, 42, "Publish globally", "Concept + provenance") + e("2", "v"); e("v", "p") + self.note("One successful publication can serve many independent users.") + elif key == "budget": + b("a", 16, 39, 143, 43, "API prefetch", "") + b("b", 189, 39, 143, 43, "Scheduled refill", "") + b("c", 362, 39, 143, 43, "Manual rewrite", "") + b("l", 123, 115, 275, 42, "Shared Pacific-day ledger", "Reserve atomically; commit before provider call") + for x in ("a", "b", "c"): e(x, "l") + self.note("Restarting a process does not reset the day's reserved calls.") + elif key == "growth": + self.row([("New topic or title", "Curated data change"), ("Published concept", "Seed or allowed generation"), + ("Future API selection", "User has never received it")], 45, 58) + self.note("New catalog data does not replace today's pinned assignment.", 140) + self.note("Publishing content does not send a new-content notification.", 158) + elif key == "push": + self.row([("Permission", "Phone + OS"), ("Expo token", "Register via API"), + ("Due worker", "Choose user/time slot"), ("Expo Push", "FCM / APNs to device")], 47, 63, 16) + self.note("Registration identifies where to send; the worker decides when.", 146) + elif key == "reminders": + b("t", 16, 38, 148, 47, "15-minute pass", "Translate clock per profile") + b("d", 189, 38, 148, 47, "Due and enabled?", "Token exists; day unfinished") + b("c", 362, 38, 148, 47, "Claim slot", "Unique user/date/time") + b("s", 362, 119, 148, 40, "Send Expo batch", "Claims already committed") + b("n", 16, 119, 321, 40, "Skip or already claimed", "No candidate, completed date or conflicting claim") + e("t", "d"); e("d", "c", "yes"); e("c", "s", "won") + e("d", "n", "no"); e("c", "n", "lost") + elif key == "reminder_day": + self.row([("08:00", "Morning reminder"), ("08:10", "Online completion"), + ("14:00", "Suppressed"), ("20:00", "Suppressed")], 48, 58, 19) + self.note("Example in the user's profile timezone, assuming normal worker execution.", 141) + self.note("Only server-confirmed completion affects the worker's next decision.", 161) + elif key == "offline_storage": + b("s", 171, 35, 178, 41, "Successful online state load", "Render now; download saved bodies") + for i, (t, body) in enumerate([("Progress snapshot", "Dates, totals, membership"), + ("Full concept bodies", "Explanation + example"), + ("Topics + saved metadata", "Choices, titles, filters")]): + b(str(i), 16 + i * 174, 112, 144, 43, t, body); e("s", str(i)) + self.note("Only completed downloads are available on an offline restart.") + elif key == "sync": + self.row([("Tap while offline", "Optimistic UI"), ("Durable outbox", "Latest desired state"), + ("Foreground retry", "5 / 10 / 20 / 30 seconds")], 42, 50) + b("r", 16, 118, 230, 40, "Success: reconcile", "Merge server state with remaining intentions") + b("f", 281, 118, 222, 40, "Failure: retain or reject", "Network/5xx retry; stale day/4xx dropped") + e("2", "f"); e("2", "r") + elif key == "release": + self.row([("Feature branch", "Focused commits"), ("PR to develop", "Review + preview"), + ("Release PR", "Version + migration check")], 36, 45) + b("m", 362, 112, 143, 40, "Merge main", "Production trigger") + b("e", 189, 112, 143, 40, "EAS production OTA", "Then preview + GitHub release") + b("a", 16, 112, 143, 40, "APK workflow", "Rebuild only for new runtime") + b("r", 362, 173, 143, 29, "Railway deployment", "Independent backend path") + e("2", "m"); e("m", "e"); e("e", "a"); e("m", "r") + elif key == "ota": + b("b", 21, 42, 213, 61, "Installed APK", "Native runtime 1.3.0") + b("u", 286, 42, 213, 61, "Compatible OTA", "App version 1.8.0; runtime 1.3.0") + e("u", "b", "match") + self.note("Marketing version can change while the native runtime remains fixed.", 138) + self.note("A matching highlight entry is shown until this device dismisses it.", 157) + elif key == "cost": + b("g", 17, 49, 146, 55, "One generated lesson", "Shared writing cost") + b("c", 190, 49, 146, 55, "One catalog row", "Reusable published content") + b("u", 363, 49, 146, 55, "Many learners", "Separate progress records") + e("g", "c"); e("c", "u") + self.note("Illustration: 1 lesson can serve 1,000 readers; failed attempts still cost calls.", 143) + elif key == "future_lifecycle": + self.row([("Plan curriculum", "Reviewed title pipeline"), ("Draft + approve", "Quality-controlled content"), + ("Shared publication", "Bounded continuing supply")], 38, 49) + b("l", 16, 118, 230, 39, "Daily fresh learning", "Personal eligibility; stored approved content") + b("r", 282, 118, 221, 39, "Review when fresh is unavailable", "Separate practice and honest metrics") + e("2", "l"); e("2", "r") + self.note("PROPOSED #195: operations monitor both supply and learning continuity.") + elif key == "runway": + self.row([("Reader availability", "Experienced active cohort"), ("Durable subject goal", "Coalesce repeated signals"), + ("Bounded worker", "Claims + shared call budget")], 40, 52) + b("p", 118, 119, 282, 36, "Approved new supply increases the reserve", "Consumption reduces it; future checks remeasure it") + e("2", "p") + self.note("PROPOSED #195: a reserve is a buffer, not unlimited content.") + elif key == "editorial": + self.row([("Plan / import", "Objectives + overlap check"), ("Draft", "AI-assisted or editorial"), + ("Review", "Correctness + usefulness"), ("Publish", "Approved version")], 43, 58, 16) + self.note("PROPOSED #195: an unreviewed draft does not count as available supply.", 139) + self.note("Corrections retain the concept identity and its saved/history references.", 159) + elif key == "review_schema": + b("a", 18, 38, 217, 44, "Existing new-concept assignments", "Preserve unique user/day and user/concept") + b("r", 284, 38, 217, 44, "Proposed review/activity records", "Separate identity, date and completion") + b("m", 122, 122, 280, 35, "Daily activity: distinct completed local dates", "One day of streak credit; separate new/review totals") + e("a", "m"); e("r", "m") + self.note("Conceptual model for #195; final table and API contracts are not chosen.") + elif key == "review_flow": + self.row([("Stable review target", "Server-owned identity/day"), ("Read cached body", "Offline if available"), + ("Complete / queue", "Account-scoped intention"), ("Reconcile once", "Server day + deduplication")], 43, 63, 16) + self.note("PROPOSED #195: two devices and repeated taps share one logical completion.", 139) + self.note("A fresh lesson arriving now must not replace a review already in progress.", 159) + elif key == "content_ops": + self.row([("Supply + job metrics", "Reserve, queue, calls, health"), ("Protected report", "Actionable alerts + recovery"), + ("Authorized maintainer", "Review, repair, pause, restore")], 44, 57) + self.note("PROPOSED #195: report causes, deduplicate alerts and preserve spending limits.", 141) + self.note("Operator alerts are separate from learner reminders and account emails.", 161) + else: + raise ValueError(f"Unknown diagram: {key}") + + # Draw boxes after connectors so lines cannot overprint the labels. + for _, (x, y, w, h, title, body, tone) in self.nodes.items(): + c.setFillColor(colors.white) + c.setStrokeColor(LINE) + c.roundRect(x, self.base_h - y - h, w, h, 5, fill=1, stroke=1) + accent = AMBER if tone == "amber" else TEAL + c.setFillColor(accent) + c.roundRect(x, self.base_h - y - h, 3, h, 1, fill=1, stroke=0) + st = ParagraphStyle("node", fontName="Body", fontSize=8.2, leading=10.4, + textColor=INK, alignment=TA_CENTER) + tx = "" + html.escape(title) + "" + if body: tx += '
' + html.escape(body) + "" + p = Paragraph(tx, st) + _, ph = p.wrap(w - 13, h) + if ph > h - 5: + raise ValueError(f"Diagram label overflow in {key}: {title} ({ph} > {h})") + p.drawOn(c, x + 7, self.base_h - y - (h + ph) / 2) + c.restoreState() + + +def blocks(body, scale, fig_start, chapter): + st = styles(scale) + lines = body.strip().splitlines() + result, i, fig = [], 0, fig_start + while i < len(lines): + line = lines[i].strip() + if not line: + i += 1 + continue + if line.startswith(" Curriculum[Curated curriculum import] + Curriculum --> Queue[Durable topic supply targets] + Queue --> Drafts[Budgeted background generation] + Drafts --> Review[Operator review and versioned publication] + Review --> Library[Shared published library] + Library --> Daily[Daily new lesson] + Library --> Practice[Daily review] + Daily --> Progress[Account progress and learning streak] + Practice --> Progress + Queue --> Operations[Protected operational report] + Review --> Operations +``` + +- Topic UUIDs and slugs remain stable. Retiring a topic hides it from discovery, + new selection and generation without deleting concepts, follows or history. +- New assignments preserve one concept per user/day and no repeated new concept. + Review has its own daily record; both completion types count learning days, + while unique learned totals remain based on completed new assignments only. +- A profile row lock serializes daily selection and review selection. A review + already chosen for today wins over content that arrives later that day. +- Legacy clients retain the existing daily payload. Review-aware clients opt in + to a separate review payload; old clients never mistake a review for new learning. +- Publication is explicit operator work. Generation prepares drafts, not truth. + Corrections use an optimistic base version and preserve the concept identity. +- New subjects need registry/curriculum data, not new routes or mobile screens. + Topic retirement is reversible; physical deletion is not a routine operation. + +## Supply and curriculum + +The daily path only signals low availability. Durable topic targets are computed +from published content plus the reader's deficit, and merged with GREATEST. With +unchanged assignments, publication increases published and unseen counts equally, +so repeated requests do not increase the target. Scheduled planning accounts for +active readers from the last 90 days and bootstraps new subjects. Default target +reserve is 60 lessons; the low watermark is 5. These are configurable operating +choices, not a guarantee that approved material or provider capacity exists. + +Generation claims are serialized per subject and count drafts toward work in +progress so an approval backlog cannot cause unlimited drafts. Every provider +call still reserves from the shared Pacific-day ledger. Daily reading never waits +for a model; stale work, empty curricula and exhausted budgets have bounded exits. + +Curriculum imports are validated and idempotent. Stable slugs, objectives, +difficulty, prerequisite slugs and references keep expansion deliberate. Similar +titles are surfaced for operator review. Import, correction, review, publication +and retirement are maintainer CLI actions; they are not public mobile API powers. + +## Review and offline behaviour + +An exhausted review-aware reader receives a previously completed lesson chosen +by least recent review. Explicit completion may keep the learning streak alive, +but never increases the unique learned count. Server dates and the existing +one-day grace determine accepted completion. A completion identifies its review +record, so a stale offline request cannot complete a different day's activity. + +The review payload travels in the existing account cache. Pending completion uses +the durable outbox and existing reconnect scheduler. New account-scoped state is +cleared on sign-out and guarded against late responses. The UI distinguishes +Review from New, and offers subject discovery or a retry when no review exists. + +## Delivery plan + +1. Reproduce the refill stop; introduce durable supply planning and concurrency tests. +2. Add generic subject lifecycle and structured curriculum import. +3. Add versioned drafts, explicit review/publication and safe corrections. +4. Add review selection/completion and activity-based streaks with compatibility tests. +5. Add account-safe offline review and Today UI. +6. Add protected operational reports, condition transitions and maintenance runbook. +7. Validate long-running supply, retirement/addition, outages and full regressions; + document the complete architecture and open one PR into develop. + +New migrations must be applied before backend deployment. Their filenames stay +out of applied.txt until production application is verified. This PR does not +change production settings, generate live content or deploy a release. diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 74175b7..d5bbe4c 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,19 @@ claims as completed work. ## Current status +## Sustainable learning (#195) — 2026-09-13 + +- Implement all five content-lifecycle work areas in one PR, with a portable + subject registry supporting the existing five subjects and future additions/ + retirement. Work in isolated `/tmp/one-concept-195` on + `codex/195-sustainable-learning` from `develop` (`3cc5af3`). +- Preserve the unrelated handbook checkout and its local edits. Read Expo SDK + 57 documentation before mobile changes. No production migration, generation, + app version bump or deployment is part of this implementation PR. +- Architecture and planned focused commits are in `CONTENT_ARCHITECTURE.md`. + Validate durable refill, curriculum/publication, review/streak/offline behaviour, + subject lifecycle and operational reporting before publishing the PR. + - [Release PR #191](https://github.com/Coding-Moves/one-concept/pull/191) is open from **develop → main** for **1.8.0**, with the six-benefit one-time card and runtime **1.3.0**. Feature/fix PRs #183–#186, #188, and #189 are included; From 6df5ffcf387cb013a758b6a784e8a8e8ae8394c4 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:19:18 +0500 Subject: [PATCH 06/36] docs: record handbook validation and handoff --- docs/WORK_LOG.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index b86ba34..9f199ca 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -28,6 +28,29 @@ claims as completed work. inventory; explain its five proposed work areas, benefits, tradeoffs and undecided parameters. This is design documentation, not authorization to implement #195, generate content, change production or close that issue. +- **Delivered:** a 49-page handbook with 47 chapters and 29 vector diagrams, + clickable contents/bookmarks, pinned source links, credential names/purposes + without values, a seven-day study guide, and final before/after Q&A. Current + refill behavior is distinguished from the proposed sustainable design in #195. + Reproducible source and build instructions are in `docs/handbook/`; the local + output is `output/pdf/one-concept-engineering-handbook.pdf` (ignored by Git). +- **Commits:** `d04df36` records scope; `15fd0a3` explains the current app; + `159b564` adds the #195 comparison and Q&A; `eff0673` adds the PDF renderer, + diagrams, build guide and navigation. This validation entry is committed as + `docs: record handbook validation and handoff`. +- **Validation (passed):** rebuilt and rendered all 49 pages with Poppler; + visually reviewed every page and individually rechecked revised diagrams. + Final automated checks confirm 29 figures, 291 link annotations, 176 valid + local source-path references, correct chapter order, text within page bounds, + valid build-script syntax and no secret-looking token patterns. README local + links and `git diff --check` pass. Minimum body/table fonts are 9.13/8.1 pt. +- **Limits / not run:** application tests were not rerun for documentation/layout + work. Live database contents, actual cron/SMTP settings, inbox delivery and + physical-device push delivery were not tested. The handbook labels historical + test evidence, dated issue inventory and future-design decisions explicitly. +- **Handoff:** prepare one documentation PR into `develop`, preserve its focused + commits and leave #195 open for its separate implementation. Another task's + appended issue-creation note in this shared log remains unstaged and preserved. ## Previous release handoff (historical) From 283475f0c2e6f69deaa5de21ace507e0b21b068d Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:20:53 +0500 Subject: [PATCH 07/36] docs: record handbook PR handoff --- docs/WORK_LOG.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 9f199ca..04685f1 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -48,9 +48,16 @@ claims as completed work. work. Live database contents, actual cron/SMTP settings, inbox delivery and physical-device push delivery were not tested. The handbook labels historical test evidence, dated issue inventory and future-design decisions explicitly. -- **Handoff:** prepare one documentation PR into `develop`, preserve its focused - commits and leave #195 open for its separate implementation. Another task's - appended issue-creation note in this shared log remains unstaged and preserved. +- **Handoff:** [PR #196](https://github.com/Coding-Moves/one-concept/pull/196) + targets `develop` with the focused commits above plus validation `6df5ffc`. + Preserve the commits; #195 remains open for its separate implementation. + The PDF is delivered locally and the PR provides its reproducible source. + Another task's appended issue-creation note remains unstaged and preserved. +- **Publication:** the initial automatic-review destination concern was resolved + by verifying the existing public origin, owner ADMIN access and public-source + scope without credential values. The approved push used the existing GitHub + credential helper after plain HTTPS authentication was unavailable. No merge, + deployment or production mutation was performed. ## Previous release handoff (historical) From 11adafc8a571a61ef7df41384f7237988c88fa44 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:23:16 +0500 Subject: [PATCH 08/36] fix: replenish shared lessons beyond the initial topic inventory --- backend/app/config.py | 6 +- backend/app/services/pool.py | 18 ++- backend/app/services/prefetch.py | 14 +- backend/app/services/selection.py | 9 +- backend/app/services/supply.py | 87 +++++++++++ backend/app/workers/pool_topup.py | 2 + backend/migrations/0011_content_supply.sql | 13 ++ backend/tests/test_content_supply.py | 166 +++++++++++++++++++++ backend/tests/test_generation.py | 11 +- backend/tests/test_generation_limits.py | 5 +- 10 files changed, 313 insertions(+), 18 deletions(-) create mode 100644 backend/app/services/supply.py create mode 100644 backend/migrations/0011_content_supply.sql create mode 100644 backend/tests/test_content_supply.py diff --git a/backend/app/config.py b/backend/app/config.py index 92fd5c0..3807795 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -36,7 +36,11 @@ class Settings(BaseSettings): gemini_api_key: str = "" gemini_model: str = "gemini-3.1-flash-lite" generation_enabled: bool = False - min_pool_per_topic: int = 25 + min_pool_per_topic: int = Field(default=25, ge=0) + content_reserve_per_topic: int = Field(default=60, ge=1, le=365) + content_low_watermark: int = Field(default=5, ge=0, le=30) + content_active_days: int = Field(default=90, ge=1, le=365) + content_generation_batch: int = Field(default=5, ge=1, le=25) # Shared by all generation paths; zero prevents new reservations. generation_daily_call_cap: int = Field(default=200, ge=0) # Seconds between worker calls; the free tier allows ~10 requests a minute. diff --git a/backend/app/services/pool.py b/backend/app/services/pool.py index 9deea69..3ff7912 100644 --- a/backend/app/services/pool.py +++ b/backend/app/services/pool.py @@ -16,6 +16,7 @@ from app.config import get_settings from app.services.generation_budget import GenerationBudgetExhausted, reserve_generation_call from app.services.generation import GenerationError, RateLimitedError, generate_concept +from app.services.supply import target_for log = logging.getLogger(__name__) @@ -75,6 +76,7 @@ where b.id = ( select b2.id from public.concept_backlog b2 where b2.status = 'pending' + and exists(select 1 from public.topics active where active.id=b2.topic_id and active.is_active) and (cast(:topic_id as uuid) is null or b2.topic_id = cast(:topic_id as uuid)) and b2.attempts < 3 order by b2.created_at @@ -133,11 +135,20 @@ class TopUpResult: async def generate_one( session: AsyncSession, api_key: str, model: str, topic_id: uuid.UUID | None = None, - *, call_cap: int | None = None, + *, call_cap: int | None = None, supply_target: int | None = None, ) -> uuid.UUID | None: """Claim a title and daily budget together, then generate outside the transaction.""" cap = get_settings().generation_daily_call_cap if call_cap is None else call_cap try: + if supply_target is not None and topic_id is not None: + # Serialize capacity checks and claims, then release before model I/O. + active = await session.scalar(text('select is_active from public.topics where id=:t for update'), {'t':topic_id}) + inventory = await session.scalar(text("""select + (select count(*) from public.concepts where topic_id=:t and status in ('published','draft')) + + (select count(*) from public.concept_backlog where topic_id=:t and status='generating')"""), {'t':topic_id}) + if not active or inventory >= supply_target: + await session.commit() + return None claimed = (await session.execute(_CLAIM, {"topic_id": topic_id})).first() if claimed is not None: await reserve_generation_call(session, cap) @@ -225,13 +236,14 @@ async def top_up( backoff = BACKOFF_START_SECONDS rate_limit_streak = 0 for topic in (await session.execute(_POOL_COUNTS)).all(): - deficit = minimum_per_topic - topic.published + target = await target_for(session, topic.id, minimum_per_topic) + deficit = target - topic.published if deficit <= 0: continue remaining = min(deficit, topic.pending) while remaining > 0: try: - concept_id = await generate_one(session, api_key, model, topic.id, call_cap=call_cap) + concept_id = await generate_one(session, api_key, model, topic.id, call_cap=call_cap, supply_target=target) except GenerationBudgetExhausted: log.info("stopping: shared daily call cap of %s reached", call_cap) return TopUpResult(generated, failed, "daily call cap reached") diff --git a/backend/app/services/prefetch.py b/backend/app/services/prefetch.py index 7615440..9c2ea3d 100644 --- a/backend/app/services/prefetch.py +++ b/backend/app/services/prefetch.py @@ -20,6 +20,7 @@ from app.services.generation import RateLimitedError from app.services.generation_budget import GenerationBudgetExhausted from app.services.pool import generate_one +from app.services.supply import target_for log = logging.getLogger(__name__) @@ -29,9 +30,9 @@ where topic_id = :topic_id and status = 'published' """) -# Start topping a topic up once a user's unread published concepts in it fall to -# this many; a run generates until the topic reaches TARGET_PUBLISHED, capped by -# PREFETCH_BATCH lessons so one trigger can't run away. +# Legacy callers without a durable reader signal retain this small bootstrap +# floor. Reader-aware calls use the shared, advancing supply target instead. +# Each run is bounded by the configured batch and the shared daily call budget. LOW_WATERMARK = 5 PREFETCH_BATCH = 5 TARGET_PUBLISHED = LOW_WATERMARK + PREFETCH_BATCH @@ -74,7 +75,7 @@ async def _run(topic_id: uuid.UUID) -> None: # Its own session: the request's session is closed the moment the # response returns, long before this finishes. async with SessionLocal() as session: - for _ in range(PREFETCH_BATCH): + for _ in range(settings.content_generation_batch): # Re-check against a shared target each iteration so a prefetch # in another process (its lessons land in the same catalog) can # satisfy the topic and let this one stop early — bounding the @@ -82,12 +83,13 @@ async def _run(topic_id: uuid.UUID) -> None: published = await session.scalar( _PUBLISHED_IN_TOPIC, {"topic_id": topic_id} ) - if published is not None and published >= TARGET_PUBLISHED: + target = await target_for(session, topic_id, TARGET_PUBLISHED) + if published is not None and published >= target: break try: concept_id = await generate_one( session, settings.gemini_api_key, settings.gemini_model, topic_id, - call_cap=settings.generation_daily_call_cap, + call_cap=settings.generation_daily_call_cap, supply_target=target, ) except GenerationBudgetExhausted: log.info("prefetch for topic %s stopped: daily call cap reached", topic_id) diff --git a/backend/app/services/selection.py b/backend/app/services/selection.py index 6d90cae..70214db 100644 --- a/backend/app/services/selection.py +++ b/backend/app/services/selection.py @@ -23,7 +23,9 @@ from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession -from app.services.prefetch import LOW_WATERMARK, request_prefetch +from app.services.prefetch import request_prefetch +from app.config import get_settings +from app.services.supply import signal_reader @dataclass @@ -75,6 +77,7 @@ class DailyResult: with pool as ( select c.id, c.topic_id from public.concepts c + join public.topics t on t.id=c.topic_id and t.is_active where c.status = 'published' and (:ignore_follows or c.topic_id in ( select topic_id from public.user_topics where user_id = :uid)) @@ -188,6 +191,7 @@ async def get_or_create_daily( await session.execute(_FOLLOWED_TOPIC_BY_STALENESS, {"uid": user_id}) ).scalar_one_or_none() if stale_topic is not None: + await signal_reader(session, user_id, stale_topic) request_prefetch(stale_topic) outside = True @@ -225,7 +229,8 @@ async def get_or_create_daily( watermark = ( await session.execute(_TOPIC_UNREAD, {"uid": user_id, "cid": concept_id}) ).first() - if watermark and watermark.topic_id is not None and watermark.unread <= LOW_WATERMARK: + if watermark and watermark.topic_id is not None and watermark.unread <= get_settings().content_low_watermark: + await signal_reader(session, user_id, watermark.topic_id) request_prefetch(watermark.topic_id) row = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).one() diff --git a/backend/app/services/supply.py b/backend/app/services/supply.py new file mode 100644 index 0000000..7e61762 --- /dev/null +++ b/backend/app/services/supply.py @@ -0,0 +1,87 @@ +"""Durable, coalesced demand. Publication cannot inflate a reader's target. + +The target is assigned published concepts + a reserve. Adding a new published +concept increases both the catalog and unread supply, leaving this target fixed. +Only consumption, not repeated GETs or a second handset, advances it. +""" + +import uuid + +from sqlalchemy import text +from sqlalchemy.ext.asyncio import AsyncSession + +from app.config import get_settings + +_SIGNAL = text(""" + with inventory as ( + select count(*)::int as published, + count(*) filter (where exists (select 1 from public.daily_assignments a + where a.user_id=:uid and a.concept_id=c.id))::int as assigned + from public.concepts c join public.topics t on t.id=c.topic_id + where c.topic_id=:tid and c.status='published' and t.is_active + ) + insert into public.content_supply_targets(topic_id,target_count,expires_at) + select :tid, assigned+:reserve, now()+make_interval(days=>:active_days) + from inventory where published-assigned<=:low + and exists(select 1 from public.topics where id=:tid and is_active) + on conflict(topic_id) do update set + target_count=greatest(content_supply_targets.target_count,excluded.target_count), + requested_at=now(), expires_at=excluded.expires_at +""") + + +async def signal_reader( + session: AsyncSession, user_id: uuid.UUID, topic_id: uuid.UUID +) -> None: + settings = get_settings() + await session.execute( + _SIGNAL, + { + "uid": user_id, + "tid": topic_id, + "reserve": settings.content_reserve_per_topic, + "low": settings.content_low_watermark, + "active_days": settings.content_active_days, + }, + ) + await session.commit() + + +async def plan_active_readers(session: AsyncSession) -> None: + """One aggregate per scheduled run, never a scan of every user on a GET.""" + settings = get_settings() + await session.execute( + text(""" + with active as ( + select distinct user_id from public.daily_assignments + where assigned_at>=now()-make_interval(days=>:days) + ), demand as ( + select ut.topic_id, coalesce(max(seen.n),0)::int+:reserve as target + from active a join public.user_topics ut on ut.user_id=a.user_id + join public.topics t on t.id=ut.topic_id and t.is_active + left join lateral ( + select count(*) as n from public.daily_assignments da + join public.concepts c on c.id=da.concept_id + where da.user_id=a.user_id and c.topic_id=ut.topic_id and c.status='published' + ) seen on true group by ut.topic_id + ) + insert into public.content_supply_targets(topic_id,target_count,expires_at) + select topic_id,target,now()+make_interval(days=>:days) from demand + on conflict(topic_id) do update set target_count=excluded.target_count, + requested_at=now(),expires_at=excluded.expires_at + """), + { + "days": settings.content_active_days, + "reserve": settings.content_reserve_per_topic, + }, + ) + await session.commit() + + +async def target_for(session: AsyncSession, topic_id: uuid.UUID, floor: int = 0) -> int: + return await session.scalar( + text("""select greatest(:floor,coalesce( + (select target_count from public.content_supply_targets + where topic_id=:tid and expires_at>now()),0))"""), + {"tid": topic_id, "floor": floor}, + ) diff --git a/backend/app/workers/pool_topup.py b/backend/app/workers/pool_topup.py index f03833f..441ef5a 100644 --- a/backend/app/workers/pool_topup.py +++ b/backend/app/workers/pool_topup.py @@ -10,6 +10,7 @@ from app.config import get_settings from app.db.session import SessionLocal, engine from app.services.pool import top_up +from app.services.supply import plan_active_readers async def main() -> None: @@ -17,6 +18,7 @@ async def main() -> None: settings = get_settings() async with SessionLocal() as session: + await plan_active_readers(session) result = await top_up( session, api_key=settings.gemini_api_key, diff --git a/backend/migrations/0011_content_supply.sql b/backend/migrations/0011_content_supply.sql new file mode 100644 index 0000000..f22bf1f --- /dev/null +++ b/backend/migrations/0011_content_supply.sql @@ -0,0 +1,13 @@ +-- Persistent demand, shared by every reader and generator of a subject. +begin; +create table public.content_supply_targets ( + topic_id uuid primary key references public.topics(id) on delete restrict, + target_count integer not null check (target_count >= 0), + requested_at timestamptz not null default now(), + expires_at timestamptz not null +); +alter table public.content_supply_targets enable row level security; +-- Backend/operator only; no mobile policies. +create index daily_assignments_recent_reader_idx + on public.daily_assignments (assigned_at, user_id); +commit; diff --git a/backend/tests/test_content_supply.py b/backend/tests/test_content_supply.py new file mode 100644 index 0000000..48a885a --- /dev/null +++ b/backend/tests/test_content_supply.py @@ -0,0 +1,166 @@ +"""Supply must follow reader demand, even after a catalog has reached its floor.""" + +import asyncio +import uuid +from datetime import UTC, datetime, timedelta + +import pytest +from app.config import get_settings +from app.services import pool, prefetch, selection +from app.services.generation import GeneratedConcept +from sqlalchemy import text + + +@pytest.fixture +async def full_topic(session, user): + slug = "supply-" + uuid.uuid4().hex + tid = await session.scalar( + text( + "insert into public.topics(slug,name) values (:s,'Supply fixture') returning id" + ), + {"s": slug}, + ) + for i in range(25): + cid = await session.scalar( + text("""insert into public.concepts(topic_id,slug,title,summary) + values (:t,:s,'Fixture','Stored explanation') returning id"""), + {"t": tid, "s": f"{slug}-{i}"}, + ) + await session.execute( + text("""insert into public.daily_assignments(user_id,concept_id,assigned_for,completed_at) + values (:u,:c,:d,now())"""), + {"u": user, "c": cid, "d": datetime.now(UTC).date() - timedelta(days=i + 1)}, + ) + await session.execute( + text("delete from public.user_topics where user_id=:u"), {"u": user} + ) + await session.execute( + text("insert into public.user_topics(user_id,topic_id) values (:u,:t)"), + {"u": user, "t": tid}, + ) + for i in range(6): + await session.execute( + text("""insert into public.concept_backlog(topic_id,slug,title) + values (:t,:s,'A new lesson')"""), + {"t": tid, "s": f"{slug}-new-{i}"}, + ) + await session.commit() + yield tid + await session.rollback() + await session.execute( + text("update public.topics set is_active=false where id=:t"), {"t": tid} + ) + await session.commit() + + +async def test_experienced_reader_refills_a_full_topic( + empty_generation_budget, + session, + user, + full_topic, + sessionmaker_for_test, + monkeypatch, +): + settings = get_settings() + monkeypatch.setattr(settings, "generation_enabled", True) + monkeypatch.setattr(settings, "gemini_api_key", "fixture-key") + monkeypatch.setattr(prefetch, "SessionLocal", sessionmaker_for_test) + + async def generate(**kwargs): + return GeneratedConcept( + summary="A reviewed-size explanation. " * 10, + example="A concrete example of this concept.", + model="fixture", + ) + + monkeypatch.setattr(pool, "generate_concept", generate) + await selection.get_or_create_daily(session, user) + if prefetch._tasks: + await asyncio.gather(*list(prefetch._tasks)) + count = await session.scalar( + text("select count(*) from public.concepts where topic_id=:t"), + {"t": full_topic}, + ) + assert count > 25, ( + "A reader who exhausted 25 lessons must generate new drafts beyond the stocking floor" + ) + + +async def test_repeated_readers_and_publication_do_not_inflate_target( + session, user, full_topic, sessionmaker_for_test +): + from app.services.supply import signal_reader, target_for + + async def signal(): + async with sessionmaker_for_test() as other: + await signal_reader(other, user, full_topic) + + await asyncio.gather(*(signal() for _ in range(12))) + target = await target_for(session, full_topic) + assert target == 25 + get_settings().content_reserve_per_topic + await session.execute( + text("""insert into public.concepts(topic_id,slug,title,summary) + values (:t,:s,'New shared lesson','Published after demand')"""), + {"t": full_topic, "s": uuid.uuid4().hex}, + ) + await session.commit() + await signal_reader(session, user, full_topic) + assert await target_for(session, full_topic) == target + + +async def test_capacity_claims_include_other_workers_in_flight( + empty_generation_budget, session, full_topic, sessionmaker_for_test, monkeypatch +): + called = 0 + entered = asyncio.Event() + release = asyncio.Event() + + async def generate(**kwargs): + nonlocal called + called += 1 + entered.set() + await release.wait() + return GeneratedConcept( + summary="An explanation. " * 20, + example="A concrete example.", + model="fixture", + ) + + monkeypatch.setattr(pool, "generate_concept", generate) + + async def run(): + async with sessionmaker_for_test() as other: + return await pool.generate_one( + other, "k", "m", full_topic, call_cap=100, supply_target=26 + ) + + first = asyncio.create_task(run()) + await asyncio.wait_for(entered.wait(), 5) + try: + assert await asyncio.wait_for(run(), 5) is None + finally: + release.set() + assert await first is not None + assert called == 1 + + +async def test_retirement_disables_new_selection_and_generation( + empty_generation_budget, session, user, full_topic, monkeypatch +): + await session.execute( + text("update public.topics set is_active=false where id=:t"), {"t": full_topic} + ) + await session.commit() + + async def unexpected(**kwargs): + raise AssertionError("retired subject reached provider") + + monkeypatch.setattr(pool, "generate_concept", unexpected) + assert ( + await pool.generate_one( + session, "k", "m", full_topic, call_cap=100, supply_target=80 + ) + is None + ) + result = await selection.get_or_create_daily(session, user) + assert result.concept is None or not result.concept.topic_slug.startswith("supply-") diff --git a/backend/tests/test_generation.py b/backend/tests/test_generation.py index fe39ac7..9dc696e 100644 --- a/backend/tests/test_generation.py +++ b/backend/tests/test_generation.py @@ -180,11 +180,10 @@ async def test_failed_generation_leaves_the_item_retryable(empty_generation_budg async def test_repeated_failures_retire_the_item(empty_generation_budget, session, patch_httpx): patch_httpx(_stub_transport({}, status=500)) - # An inactive topic of its own, so this cannot disturb the shared backlog - # that the other tests draw from. + # Use an active isolated topic; retire it after exercising provider failure. topic_id = (await session.execute(text(""" insert into public.topics (slug, name, is_active, sort_order) - values ('test-cursed', 'Cursed Topic', false, 99) + values ('test-cursed', 'Cursed Topic', true, 99) returning id """))).scalar_one() await session.execute(text(""" @@ -200,6 +199,8 @@ async def test_repeated_failures_retire_the_item(empty_generation_budget, sessio "select status, attempts from public.concept_backlog where slug = 'cursed-title'"))).one() assert row.status == "failed", "a title that never works must stop blocking the queue" assert row.attempts == 3 + await session.execute(text('update public.topics set is_active=false where id=:id'), {'id':topic_id}) + await session.commit() async def test_rate_limit_releases_the_claim_and_refunds_the_attempt(empty_generation_budget, session, patch_httpx): @@ -266,7 +267,7 @@ async def test_slug_collision_does_not_mark_the_backlog_done(empty_generation_bu patch_httpx(_stub_transport(_gemini_response(GOOD_SUMMARY, GOOD_EXAMPLE))) topic_id = (await session.execute(text(""" insert into public.topics (slug, name, is_active, sort_order) - values ('test-collision', 'Collision Topic', false, 97) + values ('test-collision', 'Collision Topic', true, 97) returning id """))).scalar_one() # A published concept already owns the slug the backlog row will generate. @@ -290,6 +291,8 @@ async def test_slug_collision_does_not_mark_the_backlog_done(empty_generation_bu count = await session.scalar( text("select count(*) from public.concepts where slug = 'dup-slug'")) assert count == 1 + await session.execute(text('update public.topics set is_active=false where id=:id'), {'id':topic_id}) + await session.commit() response = httpx.Response(429, headers={"retry-after": "30"}, text="{}") assert generation._retry_after_seconds(response) == 30.0 assert generation._retry_after_seconds(httpx.Response(429, text="{}")) is None diff --git a/backend/tests/test_generation_limits.py b/backend/tests/test_generation_limits.py index 2f5e65e..08d151b 100644 --- a/backend/tests/test_generation_limits.py +++ b/backend/tests/test_generation_limits.py @@ -23,7 +23,7 @@ async def topic(session, monkeypatch): tid = uuid.uuid4() await session.execute(text(""" insert into public.topics (id, slug, name, is_active) - values (:id, :slug, 'Budget fixture', false) + values (:id, :slug, 'Budget fixture', true) """), {"id": tid, "slug": f"budget-{tid}"}) await session.execute(text(""" insert into public.concept_backlog (topic_id, slug, title) @@ -162,7 +162,8 @@ def test_negative_daily_cap_is_rejected(): @pytest.fixture def prefetch_config(monkeypatch, sessionmaker_for_test): config = SimpleNamespace(generation_enabled=True, generation_on_demand=True, - gemini_api_key="test", gemini_model="test", generation_daily_call_cap=2) + gemini_api_key="test", gemini_model="test", generation_daily_call_cap=2, + content_generation_batch=5) monkeypatch.setattr(prefetch, "get_settings", lambda: config) monkeypatch.setattr(prefetch, "SessionLocal", sessionmaker_for_test) return config From 1a106c0a352d655d322b97fb9f1632e8330c11cd Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:26:19 +0500 Subject: [PATCH 09/36] feat: add portable subjects and validated curriculum imports --- backend/app/services/curriculum.py | 192 ++++++++++++++++++ backend/content/subjects.json | 7 + .../0012_curriculum_publication.sql | 32 +++ backend/tests/test_curriculum.py | 79 +++++++ 4 files changed, 310 insertions(+) create mode 100644 backend/app/services/curriculum.py create mode 100644 backend/content/subjects.json create mode 100644 backend/migrations/0012_curriculum_publication.sql create mode 100644 backend/tests/test_curriculum.py diff --git a/backend/app/services/curriculum.py b/backend/app/services/curriculum.py new file mode 100644 index 0000000..f2b66df --- /dev/null +++ b/backend/app/services/curriculum.py @@ -0,0 +1,192 @@ +"""Maintainer-only catalog operations. Callers commit the whole import or roll back. + +Stable slugs identify records; importing a registry never deletes omitted topics. +The advisory transaction lock serializes imports and publication across operators. +""" + +import re +from difflib import SequenceMatcher +from typing import Annotated + +from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator +from sqlalchemy import text +from sqlalchemy.ext.asyncio import AsyncSession + +Slug = Annotated[str, Field(pattern=r"^[a-z0-9]+(?:-[a-z0-9]+)*$", max_length=120)] + + +class StrictModel(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + +class Subject(StrictModel): + slug: Slug + name: str = Field(min_length=2, max_length=80) + description: str = Field(default="", max_length=500) + sort_order: int = Field(default=0, ge=0, le=32767) + is_active: bool = True + + +class Reference(StrictModel): + title: str = Field(min_length=3, max_length=200) + url: HttpUrl + + +class Curriculum(StrictModel): + objective: str = Field(min_length=15, max_length=500) + # 1 = foundations, 2 = intermediate, 3 = advanced applications. + difficulty: int = Field(ge=1, le=3) + prerequisites: list[Slug] = Field(default_factory=list, max_length=20) + references: list[Reference] = Field(min_length=1, max_length=10) + + @field_validator("prerequisites") + @classmethod + def unique_prerequisites(cls, values): + if len(values) != len(set(values)): + raise ValueError("Duplicate prerequisites") + return values + + +class PlannedLesson(StrictModel): + slug: Slug + topic_slug: Slug + title: str = Field(min_length=3, max_length=160) + angle: str = Field(default="", max_length=1000) + curriculum: Curriculum + + +def normalized(value: str) -> str: + return " ".join(re.findall(r"\w+", value.casefold())) + + +async def catalog_lock(session: AsyncSession) -> None: + await session.execute(text("select pg_advisory_xact_lock(195, 1)")) + + +async def import_subjects(session: AsyncSession, items: list[Subject]) -> int: + if len({item.slug for item in items}) != len(items): + raise ValueError("Duplicate subject slug in import") + await catalog_lock(session) + for item in items: + await session.execute( + text("""insert into public.topics + (slug,name,description,sort_order,is_active) + values (:slug,:name,:description,:sort_order,:is_active) + on conflict(slug) do update set name=excluded.name, + description=excluded.description,sort_order=excluded.sort_order, + is_active=excluded.is_active"""), + item.model_dump(), + ) + return len(items) + + +async def validate_graph(session: AsyncSession, additions: dict[str, dict]) -> None: + rows = ( + await session.execute( + text("""select slug,curriculum from public.concepts + union all select slug,curriculum from public.concept_backlog""") + ) + ).all() + graph = {r.slug: r.curriculum.get("prerequisites", []) for r in rows} + graph.update( + {slug: data.get("prerequisites", []) for slug, data in additions.items()} + ) + visited, visiting = set(), set() + + def visit(slug): + if slug in visiting: + raise ValueError(f"Prerequisite cycle includes {slug}") + if slug in visited: + return + if slug not in graph: + raise ValueError(f"Unknown prerequisite {slug}") + visiting.add(slug) + for prereq in graph[slug]: + visit(prereq) + visiting.remove(slug) + visited.add(slug) + + for slug in additions: + visit(slug) + + +async def import_lessons( + session: AsyncSession, items: list[PlannedLesson] +) -> list[str]: + """Idempotent exact re-import; changed existing plans need explicit editorial work.""" + import json + + if len({item.slug for item in items}) != len(items): + raise ValueError("Duplicate lesson slug in import") + await catalog_lock(session) + await validate_graph( + session, {i.slug: i.curriculum.model_dump(mode="json") for i in items} + ) + existing = ( + await session.execute( + text("""select slug,title,curriculum from public.concepts + union all select slug,title,curriculum from public.concept_backlog""") + ) + ).all() + titles = {normalized(r.title): r.slug for r in existing} + objectives = { + normalized(r.curriculum["objective"]): r.slug + for r in existing + if r.curriculum.get("objective") + } + warnings = [] + for item in items: + title_key, objective_key = ( + normalized(item.title), + normalized(item.curriculum.objective), + ) + for key, index in ((title_key, titles), (objective_key, objectives)): + if key in index and index[key] != item.slug: + raise ValueError(f"Exact duplicate of {index[key]}: {item.slug}") + for key, slug in titles.items(): + if ( + slug != item.slug + and SequenceMatcher(None, title_key, key).ratio() >= 0.78 + ): + warnings.append(f"Check overlap: {item.slug} and {slug}") + titles[title_key], objectives[objective_key] = item.slug, item.slug + topic = await session.scalar( + text("select id from public.topics where slug=:s and is_active"), + {"s": item.topic_slug}, + ) + if topic is None: + raise ValueError(f"Unknown or retired subject {item.topic_slug}") + previous = ( + await session.execute( + text("select * from public.concept_backlog where slug=:s"), + {"s": item.slug}, + ) + ).first() + data = item.curriculum.model_dump(mode="json") + if previous: + if ( + previous.topic_id, + previous.title, + previous.angle or "", + previous.curriculum, + ) != (topic, item.title, item.angle, data): + raise ValueError( + f"{item.slug} already exists with a different plan; review it explicitly" + ) + continue + if any(row.slug == item.slug for row in existing): + raise ValueError(f"{item.slug} already exists in the catalog") + await session.execute( + text("""insert into public.concept_backlog + (topic_id,slug,title,angle,difficulty,curriculum) values + (:t,:s,:title,:angle,:difficulty,cast(:data as jsonb))"""), + { + "t": topic, + "s": item.slug, + "title": item.title, + "angle": item.angle, + "difficulty": item.curriculum.difficulty, + "data": json.dumps(data), + }, + ) + return sorted(set(warnings)) diff --git a/backend/content/subjects.json b/backend/content/subjects.json new file mode 100644 index 0000000..6743866 --- /dev/null +++ b/backend/content/subjects.json @@ -0,0 +1,7 @@ +[ + {"slug":"artificial-intelligence","name":"Artificial Intelligence","description":"Machine learning, deep learning, LLMs, computer vision, and NLP.","sort_order":1}, + {"slug":"software-engineering","name":"Software Engineering","description":"Architecture, APIs, databases, testing, design patterns, and delivery.","sort_order":2}, + {"slug":"computer-science","name":"Computer Science","description":"Algorithms, data structures, operating systems, networks, and compilers.","sort_order":3}, + {"slug":"mathematics","name":"Mathematics","description":"Algebra, calculus, probability, statistics, and the mathematics behind AI.","sort_order":4}, + {"slug":"linux-systems","name":"Linux & Systems","description":"The kernel, processes, filesystems, networking, and system administration.","sort_order":5} +] diff --git a/backend/migrations/0012_curriculum_publication.sql b/backend/migrations/0012_curriculum_publication.sql new file mode 100644 index 0000000..e5cb375 --- /dev/null +++ b/backend/migrations/0012_curriculum_publication.sql @@ -0,0 +1,32 @@ +-- Curriculum and editorial history are shared catalog data, never user data. +begin; +alter table public.concept_backlog add column curriculum jsonb not null default '{}'; +alter table public.concepts + add column curriculum jsonb not null default '{}', + add column content_version integer not null default 1 check (content_version >= 0), + add column published_at timestamptz; +update public.concepts set published_at=created_at where status='published'; +create table public.concept_revisions ( + id uuid primary key default gen_random_uuid(), + concept_id uuid not null references public.concepts(id) on delete restrict, + base_version integer not null check (base_version >= 0), + body jsonb not null, + status text not null default 'draft' check(status in ('draft','published','rejected')), + created_at timestamptz not null default now(), + reviewed_at timestamptz, + reviewed_by text, + review_note text +); +create index concept_revisions_pending_idx on public.concept_revisions(concept_id) + where status='draft'; +alter table public.concept_revisions enable row level security; +-- No client policies: drafts, review notes and operator identities stay private. +create table public.content_retry_log ( + id bigint generated always as identity primary key, + backlog_id uuid not null references public.concept_backlog(id) on delete restrict, + reason text not null, + operator text not null, + created_at timestamptz not null default now() +); +alter table public.content_retry_log enable row level security; +commit; diff --git a/backend/tests/test_curriculum.py b/backend/tests/test_curriculum.py new file mode 100644 index 0000000..173c807 --- /dev/null +++ b/backend/tests/test_curriculum.py @@ -0,0 +1,79 @@ +import uuid + +import pytest +from app.services.curriculum import ( + PlannedLesson, + Subject, + import_lessons, + import_subjects, +) +from sqlalchemy import text + + +def plan(slug, topic, **changes): + data = { + "slug": slug, + "topic_slug": topic, + "title": f"Learning {slug}", + "curriculum": { + "objective": f"Explain and demonstrate {slug}", + "difficulty": 1, + "references": [ + {"title": "Reference manual", "url": "https://docs.python.org/3/"} + ], + }, + } + data.update(changes) + return PlannedLesson.model_validate(data) + + +async def test_generic_subject_import_and_retirement_preserve_identity(session): + slug = "subject-" + uuid.uuid4().hex + subject = Subject(slug=slug, name="Future subject") + await import_subjects(session, [subject]) + topic_id = await session.scalar( + text("select id from public.topics where slug=:s"), {"s": slug} + ) + lesson = plan("lesson-" + uuid.uuid4().hex, slug) + assert await import_lessons(session, [lesson]) == [] + assert await import_lessons(session, [lesson]) == [] + await import_subjects(session, [subject.model_copy(update={"is_active": False})]) + assert ( + await session.scalar( + text("select id from public.topics where slug=:s"), {"s": slug} + ) + == topic_id + ) + assert ( + await session.scalar( + text("select count(*) from public.concept_backlog where topic_id=:t"), + {"t": topic_id}, + ) + == 1 + ) + with pytest.raises(ValueError, match="retired"): + await import_lessons(session, [plan("other-" + uuid.uuid4().hex, slug)]) + await session.rollback() + + +async def test_curriculum_rejects_cycles_missing_prerequisites_and_duplicate_objectives( + session, +): + slug = "subject-" + uuid.uuid4().hex + await import_subjects(session, [Subject(slug=slug, name="Fixture subject")]) + first, second = ( + plan("one-" + uuid.uuid4().hex, slug), + plan("two-" + uuid.uuid4().hex, slug), + ) + first.curriculum.prerequisites = [second.slug] + with pytest.raises(ValueError, match="Unknown prerequisite"): + await import_lessons(session, [first]) + second.curriculum.prerequisites = [first.slug] + with pytest.raises(ValueError, match="cycle"): + await import_lessons(session, [first, second]) + first.curriculum.prerequisites = [] + second.curriculum.prerequisites = [] + second.curriculum.objective = first.curriculum.objective + with pytest.raises(ValueError, match="Exact duplicate"): + await import_lessons(session, [first, second]) + await session.rollback() From 72e3a9b9e1025650b38e8d4bcaa3452324582b97 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:34:08 +0500 Subject: [PATCH 10/36] feat: require editorial review before publishing generated lessons --- backend/app/services/pool.py | 86 ++++++++--- backend/app/services/publication.py | 161 ++++++++++++++++++++ backend/app/workers/rewrite_catalog.py | 76 +++++++--- backend/tests/test_generation.py | 11 +- backend/tests/test_generation_limits.py | 3 +- backend/tests/test_publication.py | 187 ++++++++++++++++++++++++ 6 files changed, 479 insertions(+), 45 deletions(-) create mode 100644 backend/app/services/publication.py create mode 100644 backend/tests/test_publication.py diff --git a/backend/app/services/pool.py b/backend/app/services/pool.py index 3ff7912..f4b8607 100644 --- a/backend/app/services/pool.py +++ b/backend/app/services/pool.py @@ -6,6 +6,7 @@ """ import asyncio +import json import logging import uuid from dataclasses import dataclass @@ -14,8 +15,11 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.config import get_settings -from app.services.generation_budget import GenerationBudgetExhausted, reserve_generation_call from app.services.generation import GenerationError, RateLimitedError, generate_concept +from app.services.generation_budget import ( + GenerationBudgetExhausted, + reserve_generation_call, +) from app.services.supply import target_for log = logging.getLogger(__name__) @@ -31,7 +35,7 @@ _POOL_COUNTS = text(""" select t.id, t.slug, t.name, (select count(*) from public.concepts c - where c.topic_id = t.id and c.status = 'published')::int as published, + where c.topic_id = t.id and c.status in ('published','draft'))::int as published, (select count(*) from public.concept_backlog b where b.topic_id = t.id and b.status = 'pending')::int as pending from public.topics t @@ -78,13 +82,13 @@ where b2.status = 'pending' and exists(select 1 from public.topics active where active.id=b2.topic_id and active.is_active) and (cast(:topic_id as uuid) is null or b2.topic_id = cast(:topic_id as uuid)) - and b2.attempts < 3 + and b2.attempts < 3 + (select count(*) from public.content_retry_log r where r.backlog_id=b2.id) order by b2.created_at for update skip locked limit 1 ) and t.id = b.topic_id - returning b.id, b.slug, b.title, b.angle, b.difficulty, b.topic_id, + returning b.id, b.slug, b.title, b.angle, b.difficulty, b.topic_id, b.curriculum, t.name as topic_name """) @@ -92,11 +96,16 @@ with inserted as ( insert into public.concepts (topic_id, slug, title, summary, example, difficulty, - status, source, model, prompt_version) + status, source, model, prompt_version, curriculum, content_version) values (:topic_id, :slug, :title, :summary, :example, :difficulty, - 'published', 'gemini', :model, :prompt_version) + 'draft', 'gemini', :model, :prompt_version, cast(:curriculum as jsonb), 0) on conflict (slug) do nothing - returning id + returning * + ), revision as ( + insert into public.concept_revisions(concept_id,base_version,body) + select id,0,jsonb_build_object('title',title,'summary',summary,'example',example, + 'curriculum',curriculum,'model',model,'prompt_version',prompt_version) + from inserted returning id ) update public.concept_backlog -- Mark done ONLY when a concept was actually inserted. A slug collision @@ -112,7 +121,7 @@ _FAIL = text(""" update public.concept_backlog - set status = case when attempts >= 3 then 'failed' else 'pending' end, + set status = case when attempts >= 3 + (select count(*) from public.content_retry_log r where r.backlog_id=concept_backlog.id) then 'failed' else 'pending' end, last_error = :error, claimed_at = null where id = :backlog_id """) @@ -134,18 +143,29 @@ class TopUpResult: async def generate_one( - session: AsyncSession, api_key: str, model: str, topic_id: uuid.UUID | None = None, - *, call_cap: int | None = None, supply_target: int | None = None, + session: AsyncSession, + api_key: str, + model: str, + topic_id: uuid.UUID | None = None, + *, + call_cap: int | None = None, + supply_target: int | None = None, ) -> uuid.UUID | None: """Claim a title and daily budget together, then generate outside the transaction.""" cap = get_settings().generation_daily_call_cap if call_cap is None else call_cap try: if supply_target is not None and topic_id is not None: # Serialize capacity checks and claims, then release before model I/O. - active = await session.scalar(text('select is_active from public.topics where id=:t for update'), {'t':topic_id}) - inventory = await session.scalar(text("""select + active = await session.scalar( + text("select is_active from public.topics where id=:t for update"), + {"t": topic_id}, + ) + inventory = await session.scalar( + text("""select (select count(*) from public.concepts where topic_id=:t and status in ('published','draft')) + - (select count(*) from public.concept_backlog where topic_id=:t and status='generating')"""), {'t':topic_id}) + (select count(*) from public.concept_backlog where topic_id=:t and status='generating')"""), + {"t": topic_id}, + ) if not active or inventory >= supply_target: await session.commit() return None @@ -167,7 +187,19 @@ async def generate_one( result = await generate_concept( title=claimed.title, topic_name=claimed.topic_name, - angle=claimed.angle, + angle="\n".join( + filter( + None, + [ + claimed.angle, + f"Learning objective: {claimed.curriculum['objective']}" + if claimed.curriculum.get("objective") + else None, + f"Difficulty: {claimed.difficulty or 1}. Prerequisites: {claimed.curriculum.get('prerequisites', [])}", + f"Editorial references: {claimed.curriculum.get('references', [])}", + ], + ) + ), api_key=api_key, model=model, ) @@ -179,7 +211,9 @@ async def generate_one( # Leave it pending for another attempt; give up after three so one bad # title cannot block the queue forever. log.warning("generation failed for %s: %s", claimed.slug, exc) - await session.execute(_FAIL, {"backlog_id": claimed.id, "error": str(exc)[:500]}) + await session.execute( + _FAIL, {"backlog_id": claimed.id, "error": str(exc)[:500]} + ) await session.commit() return None @@ -196,12 +230,13 @@ async def generate_one( "model": result.model, "prompt_version": result.prompt_version, "backlog_id": claimed.id, + "curriculum": json.dumps(claimed.curriculum), }, ) ).scalar_one_or_none() await session.commit() if concept_id is not None: - log.info("published %s", claimed.slug) + log.info("drafted %s", claimed.slug) else: # The insert was a no-op (slug already exists); _PUBLISH marked the row # failed rather than done, so the title is flagged, not silently retired. @@ -227,7 +262,9 @@ async def top_up( # Start every run by reclaiming rows a previous worker abandoned mid-flight, # so a crash cannot permanently lose a title from the pool (issue #37). - reaped = (await session.execute(_REAP_STALE, {"max_minutes": STALE_CLAIM_MINUTES})).all() + reaped = ( + await session.execute(_REAP_STALE, {"max_minutes": STALE_CLAIM_MINUTES}) + ).all() await session.commit() if reaped: log.warning("reclaimed %s stale 'generating' backlog rows", len(reaped)) @@ -240,17 +277,26 @@ async def top_up( deficit = target - topic.published if deficit <= 0: continue - remaining = min(deficit, topic.pending) + remaining = min(deficit, topic.pending, get_settings().content_generation_batch) while remaining > 0: try: - concept_id = await generate_one(session, api_key, model, topic.id, call_cap=call_cap, supply_target=target) + concept_id = await generate_one( + session, + api_key, + model, + topic.id, + call_cap=call_cap, + supply_target=target, + ) except GenerationBudgetExhausted: log.info("stopping: shared daily call cap of %s reached", call_cap) return TopUpResult(generated, failed, "daily call cap reached") except RateLimitedError as exc: rate_limit_streak += 1 if rate_limit_streak >= MAX_CONSECUTIVE_RATE_LIMITS: - log.warning("stopping: %s consecutive rate limits", rate_limit_streak) + log.warning( + "stopping: %s consecutive rate limits", rate_limit_streak + ) return TopUpResult(generated, failed, "rate limited") delay = max(exc.retry_after or 0.0, backoff) log.warning("rate limited; retrying %s in %.0fs", topic.slug, delay) diff --git a/backend/app/services/publication.py b/backend/app/services/publication.py new file mode 100644 index 0000000..2a25f62 --- /dev/null +++ b/backend/app/services/publication.py @@ -0,0 +1,161 @@ +"""Explicit review gates and version-safe corrections for the shared library.""" + +import uuid + +from pydantic import Field +from sqlalchemy import text +from sqlalchemy.ext.asyncio import AsyncSession + +from app.services.curriculum import ( + Curriculum, + StrictModel, + catalog_lock, + normalized, + validate_graph, +) + + +class LessonBody(StrictModel): + title: str = Field(min_length=3, max_length=160) + summary: str = Field(min_length=100, max_length=600) + example: str = Field(min_length=40, max_length=500) + curriculum: Curriculum + model: str | None = None + prompt_version: str | None = None + + +async def stage_revision( + session: AsyncSession, slug: str, body: LessonBody +) -> uuid.UUID: + """Leave the published text intact until this exact draft passes review.""" + await catalog_lock(session) + concept = ( + await session.execute( + text( + "select id,content_version from public.concepts where slug=:s for update" + ), + {"s": slug}, + ) + ).first() + if concept is None: + raise ValueError("Unknown concept") + await validate_graph(session, {slug: body.curriculum.model_dump(mode="json")}) + return await session.scalar( + text("""insert into public.concept_revisions + (concept_id,base_version,body) values (:id,:v,cast(:body as jsonb)) returning id"""), + { + "id": concept.id, + "v": concept.content_version, + "body": body.model_dump_json(), + }, + ) + + +async def publish_revision( + session: AsyncSession, revision_id: uuid.UUID, reviewer: str, note: str +) -> int: + if not reviewer.strip() or len(note.strip()) < 10: + raise ValueError( + "Record the reviewer and a substantive correctness/source review note" + ) + await catalog_lock(session) + row = ( + await session.execute( + text("""select r.*,c.slug,c.content_version,t.is_active + from public.concept_revisions r join public.concepts c on c.id=r.concept_id + join public.topics t on t.id=c.topic_id where r.id=:id for update of r,c,t"""), + {"id": revision_id}, + ) + ).first() + if row is None: + raise ValueError("Unknown revision") + if row.status == "published": + return row.base_version + 1 # idempotent retries never publish twice + if row.status != "draft" or row.base_version != row.content_version: + raise ValueError( + "Revision is rejected or stale; prepare a new draft from the current version" + ) + if not row.is_active: + raise ValueError("Cannot publish into a retired subject") + body = LessonBody.model_validate(row.body) + await validate_graph(session, {row.slug: body.curriculum.model_dump(mode="json")}) + # A prerequisite must be available before a dependent lesson can be published. + for slug in body.curriculum.prerequisites: + if not await session.scalar( + text( + "select exists(select 1 from public.concepts where slug=:s and status='published')" + ), + {"s": slug}, + ): + raise ValueError(f"Publish prerequisite {slug} first") + others = ( + await session.execute( + text( + "select slug,title,curriculum from public.concepts where id<>:id and status='published'" + ), + {"id": row.concept_id}, + ) + ).all() + for other in others: + if normalized(other.title) == normalized(body.title) or normalized( + other.curriculum.get("objective", "") + ) == normalized(body.curriculum.objective): + raise ValueError( + f"Exact duplicate of {other.slug}; resolve overlap before publication" + ) + await session.execute( + text("""update public.concepts set title=:title,summary=:summary, + example=:example,curriculum=cast(:curriculum as jsonb),difficulty=:difficulty, + model=:model,prompt_version=:prompt_version,content_version=content_version+1, + status='published',published_at=now() where id=:id"""), + { + "id": row.concept_id, + "title": body.title, + "summary": body.summary, + "example": body.example, + "curriculum": body.curriculum.model_dump_json(), + "difficulty": body.curriculum.difficulty, + "model": body.model, + "prompt_version": body.prompt_version, + }, + ) + await session.execute( + text("""update public.concept_revisions set status='published', + reviewed_by=:reviewer,review_note=:note,reviewed_at=now() where id=:id"""), + {"id": revision_id, "reviewer": reviewer.strip(), "note": note.strip()}, + ) + return row.content_version + 1 + + +async def retry_failed( + session: AsyncSession, slug: str, operator: str, reason: str +) -> None: + """Grant one audited retry after correcting a cause; never reset lifetime attempts.""" + if not operator.strip() or len(reason.strip()) < 10: + raise ValueError("Record an operator and the corrected cause") + row = ( + await session.execute( + text("""select b.id,b.attempts from public.concept_backlog b + join public.topics t on t.id=b.topic_id where b.slug=:s and b.status='failed' + and t.is_active for update of b"""), + {"s": slug}, + ) + ).first() + if row is None: + raise ValueError("Choose a failed item in an active subject") + if await session.scalar( + text("select exists(select 1 from public.concepts where slug=:s)"), {"s": slug} + ): + raise ValueError("Slug already exists; correct the existing draft instead") + # Each grant permits one further attempt, while preserving attempts as history. + await session.execute( + text("""insert into public.content_retry_log(backlog_id,operator,reason) + values (:id,:operator,:reason)"""), + {"id": row.id, "operator": operator, "reason": reason}, + ) + await session.execute( + text( + "update public.concept_backlog set status='pending',claimed_at=null where id=:id" + ), + {"id": row.id}, + ) diff --git a/backend/app/workers/rewrite_catalog.py b/backend/app/workers/rewrite_catalog.py index 0309afd..c551954 100644 --- a/backend/app/workers/rewrite_catalog.py +++ b/backend/app/workers/rewrite_catalog.py @@ -1,9 +1,9 @@ """Run with: python -m app.workers.rewrite_catalog One-off (but re-runnable) pass that rewrites every published concept with the -current prompt. Skips lessons already written by the current PROMPT_VERSION, +current prompt as review drafts. Skips lessons with pending drafts or the current PROMPT_VERSION, so an interrupted run resumes where it stopped. Titles, slugs, ids, and every -user's history stay untouched — only the words change. +user's history and published words stay untouched until explicit approval. """ import asyncio @@ -13,13 +13,16 @@ from app.config import get_settings from app.db.session import SessionLocal, engine -from app.services.generation_budget import GenerationBudgetExhausted, reserve_generation_call from app.services.generation import ( PROMPT_VERSION, GenerationError, RateLimitedError, generate_concept, ) +from app.services.generation_budget import ( + GenerationBudgetExhausted, + reserve_generation_call, +) log = logging.getLogger(__name__) @@ -27,23 +30,29 @@ BACKOFF_START, BACKOFF_MAX, MAX_RATE_LIMIT_STREAK = 15.0, 120.0, 5 _TODO = text(""" - select c.id, c.title, t.name as topic_name + select c.id, c.title, c.content_version, c.curriculum, t.name as topic_name from public.concepts c join public.topics t on t.id = c.topic_id where c.status = 'published' + and t.is_active and coalesce(c.prompt_version, '') <> :pv + and not exists (select 1 from public.concept_revisions r where r.concept_id=c.id + and r.status='draft') order by c.created_at """) _UPDATE = text(""" - update public.concepts - set summary = :summary, example = :example, - model = :model, prompt_version = :pv, source = 'gemini' - where id = :id + insert into public.concept_revisions(concept_id,base_version,body) + select id,content_version,jsonb_build_object('title',title,'summary',cast(:summary as text), + 'example',cast(:example as text),'curriculum',curriculum,'model',cast(:model as text),'prompt_version',cast(:pv as text)) + from public.concepts where id=:id and content_version=:version + and not exists(select 1 from public.concept_revisions r where r.concept_id=:id and r.status='draft') """) async def main() -> None: - logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + logging.basicConfig( + level=logging.INFO, format="%(levelname)s %(name)s: %(message)s" + ) settings = get_settings() try: @@ -67,24 +76,39 @@ async def main() -> None: for row in todo: while True: try: - await reserve_generation_call(session, settings.generation_daily_call_cap) + await reserve_generation_call( + session, settings.generation_daily_call_cap + ) await session.commit() result = await generate_concept( - title=row.title, topic_name=row.topic_name, angle=None, - api_key=settings.gemini_api_key, model=settings.gemini_model, + title=row.title, + topic_name=row.topic_name, + angle=None, + api_key=settings.gemini_api_key, + model=settings.gemini_model, ) except GenerationBudgetExhausted: await session.rollback() - log.info("daily call cap reached: rewritten %s, failed %s (resume on a later day)", rewritten, failed) + log.info( + "daily call cap reached: rewritten %s, failed %s (resume on a later day)", + rewritten, + failed, + ) return except RateLimitedError as exc: streak += 1 if streak >= MAX_RATE_LIMIT_STREAK: log.warning("giving up: %s consecutive rate limits", streak) - log.info("rewritten %s, failed %s (resume by re-running)", rewritten, failed) + log.info( + "rewritten %s, failed %s (resume by re-running)", + rewritten, + failed, + ) return delay = max(exc.retry_after or 0.0, backoff) - log.warning("rate limited; retrying %s in %.0fs", row.title, delay) + log.warning( + "rate limited; retrying %s in %.0fs", row.title, delay + ) await asyncio.sleep(delay) backoff = min(backoff * 2, BACKOFF_MAX) continue @@ -94,13 +118,25 @@ async def main() -> None: failed += 1 break streak, backoff = 0, BACKOFF_START - await session.execute(_UPDATE, { - "id": row.id, "summary": result.summary, "example": result.example, - "model": result.model, "pv": result.prompt_version, - }) + await session.execute( + _UPDATE, + { + "id": row.id, + "version": row.content_version, + "summary": result.summary, + "example": result.example, + "model": result.model, + "pv": result.prompt_version, + }, + ) await session.commit() rewritten += 1 - log.info("rewrote %s (%s/%s)", row.title, rewritten, len(todo)) + log.info( + "staged revision for %s (%s/%s)", + row.title, + rewritten, + len(todo), + ) break await asyncio.sleep(PACE_SECONDS) diff --git a/backend/tests/test_generation.py b/backend/tests/test_generation.py index 9dc696e..cf438a6 100644 --- a/backend/tests/test_generation.py +++ b/backend/tests/test_generation.py @@ -152,7 +152,7 @@ async def test_generate_one_publishes_and_marks_the_backlog(empty_generation_bud from public.concepts c join public.concept_backlog b on b.slug = c.slug where c.id = :id """), {"id": concept_id})).one() - assert row.status == "published" + assert row.status == "draft" assert row.source == "gemini" assert row.model and row.prompt_version, "provenance must be recorded" assert row.backlog_status == "done" @@ -355,6 +355,9 @@ async def test_top_up_stops_at_the_call_cap(empty_generation_budget, session, pa async def test_top_up_fills_only_topics_below_the_threshold(empty_generation_budget, session, patch_httpx): + # Isolate the bootstrap floor from durable demand created by other tests. + await session.execute(text("delete from public.content_supply_targets")) + await session.commit() patch_httpx(_stub_transport(_gemini_response(GOOD_SUMMARY, GOOD_EXAMPLE))) threshold = 6 @@ -362,12 +365,12 @@ async def test_top_up_fills_only_topics_below_the_threshold(empty_generation_bud # earlier tests in this session may already have published concepts. deficits = (await session.execute(text(""" select greatest(:t - (select count(*) from public.concepts c - where c.topic_id = tp.id and c.status = 'published'), 0) as deficit, + where c.topic_id = tp.id and c.status in ('published','draft')), 0) as deficit, (select count(*) from public.concept_backlog b where b.topic_id = tp.id and b.status = 'pending') as pending from public.topics tp where tp.is_active """), {"t": threshold})).all() - expected = sum(min(d.deficit, d.pending) for d in deficits) + expected = sum(min(d.deficit, d.pending, 5) for d in deficits) result = await top_up(session, api_key="k", model="gemini-2.0-flash", enabled=True, minimum_per_topic=threshold, call_cap=100) @@ -378,7 +381,7 @@ async def test_top_up_fills_only_topics_below_the_threshold(empty_generation_bud select tp.slug from public.topics tp where tp.is_active and (select count(*) from public.concepts c - where c.topic_id = tp.id and c.status = 'published') < :t + where c.topic_id = tp.id and c.status in ('published','draft')) < :t """), {"t": threshold})).scalars().all() assert short == [], f"topics left below the threshold: {short}" diff --git a/backend/tests/test_generation_limits.py b/backend/tests/test_generation_limits.py index 08d151b..54188e4 100644 --- a/backend/tests/test_generation_limits.py +++ b/backend/tests/test_generation_limits.py @@ -36,6 +36,7 @@ async def topic(session, monkeypatch): )).bindparams(tid=tid)) yield tid await session.rollback() + await session.execute(text("delete from public.concept_revisions where concept_id in (select id from public.concepts where topic_id=:id)"), {"id": tid}) await session.execute(text("delete from public.concepts where topic_id = :id"), {"id": tid}) await session.execute(text("delete from public.topics where id = :id"), {"id": tid}) await session.commit() @@ -243,7 +244,7 @@ async def rewrite_config(topic, session, generator, sessionmaker_for_test, monke async def rewritten_count(session, topic): return await session.scalar(text(""" - select count(*) from public.concepts where topic_id = :tid and prompt_version = :pv + select count(*) from public.concept_revisions r join public.concepts c on c.id=r.concept_id where c.topic_id = :tid and r.body->>'prompt_version' = :pv """), {"tid": topic, "pv": rewrite.PROMPT_VERSION}) diff --git a/backend/tests/test_publication.py b/backend/tests/test_publication.py new file mode 100644 index 0000000..2dfa31b --- /dev/null +++ b/backend/tests/test_publication.py @@ -0,0 +1,187 @@ +import json +import uuid +from unittest.mock import AsyncMock + +import pytest +from app.services import pool +from app.services.curriculum import ( + PlannedLesson, + Subject, + import_lessons, + import_subjects, +) +from app.services.generation import GeneratedConcept +from app.services.publication import LessonBody, publish_revision, stage_revision +from sqlalchemy import text + + +@pytest.fixture +async def draft(session, monkeypatch, empty_generation_budget): + slug = "editorial-" + uuid.uuid4().hex + await import_subjects(session, [Subject(slug=slug, name="Editorial fixture")]) + data = { + "objective": f"Explain and demonstrate {slug}", + "difficulty": 1, + "references": [ + {"title": "Python reference", "url": "https://docs.python.org/3/"} + ], + } + await import_lessons( + session, + [ + PlannedLesson( + slug=slug, topic_slug=slug, title=f"Lesson {slug}", curriculum=data + ) + ], + ) + tid = await session.scalar( + text("select id from public.topics where slug=:s"), {"s": slug} + ) + await session.commit() + monkeypatch.setattr( + pool, + "generate_concept", + AsyncMock( + return_value=GeneratedConcept( + summary="A useful explanation with a precise learning objective. " * 3, + example="A concrete worked example that demonstrates the idea clearly.", + model="fixture", + ) + ), + ) + cid = await pool.generate_one(session, "fixture", "fixture", tid, call_cap=10) + yield slug, cid + await session.rollback() + await session.execute( + text("update public.topics set is_active=false where id=:t"), {"t": tid} + ) + await session.commit() + + +async def test_generated_drafts_require_review_and_publish_once(session, draft): + slug, cid = draft + row = ( + await session.execute( + text("select status,content_version from public.concepts where id=:id"), + {"id": cid}, + ) + ).one() + assert tuple(row) == ("draft", 0) + revision = await session.scalar( + text("select id from public.concept_revisions where concept_id=:id"), + {"id": cid}, + ) + with pytest.raises(ValueError, match="reviewer"): + await publish_revision(session, revision, "", "looks good") + assert ( + await publish_revision( + session, + revision, + "Maintainer", + "Checked explanation and example against the listed source.", + ) + == 1 + ) + assert ( + await publish_revision( + session, + revision, + "Maintainer", + "Duplicate submission of the reviewed draft.", + ) + == 1 + ) + await session.commit() + assert ( + await session.scalar( + text( + "select content_version from public.concepts where slug=:s and status='published'" + ), + {"s": slug}, + ) + == 1 + ) + + +async def test_corrections_preserve_text_until_review_and_reject_stale_versions( + session, draft +): + slug, cid = draft + original = ( + await session.execute( + text("select id,body from public.concept_revisions where concept_id=:id"), + {"id": cid}, + ) + ).one() + await publish_revision( + session, original.id, "Maintainer", "Verified original text against sources." + ) + body = LessonBody.model_validate(original.body) + body.summary = ( + "A corrected explanation that remains available under the same identity. " * 3 + ) + first = await stage_revision(session, slug, body) + second = await stage_revision(session, slug, body) + assert ( + await session.scalar( + text("select summary from public.concepts where id=:id"), {"id": cid} + ) + == original.body["summary"].strip() + ) + assert ( + await publish_revision( + session, + first, + "Maintainer", + "Verified the correction and its worked example.", + ) + == 2 + ) + with pytest.raises(ValueError, match="stale"): + await publish_revision( + session, + second, + "Maintainer", + "This older draft must not overwrite a newer correction.", + ) + assert ( + await session.scalar( + text("select id from public.concepts where slug=:s"), {"s": slug} + ) + == cid + ) + await session.commit() + + +async def test_retired_subject_cannot_publish_and_incomplete_metadata_cannot_bypass_review( + session, draft +): + slug, cid = draft + revision = await session.scalar( + text("select id from public.concept_revisions where concept_id=:id"), + {"id": cid}, + ) + await session.execute( + text("update public.topics set is_active=false where slug=:s"), {"s": slug} + ) + with pytest.raises(ValueError, match="retired"): + await publish_revision( + session, revision, "Maintainer", "Verified source and example." + ) + await session.execute( + text("update public.topics set is_active=true where slug=:s"), {"s": slug} + ) + await session.execute( + text( + "update public.concept_revisions set body=jsonb_set(body,'{curriculum}',cast(:data as jsonb)) where id=:id" + ), + {"id": revision, "data": json.dumps({})}, + ) + with pytest.raises(ValueError): + await publish_revision( + session, + revision, + "Maintainer", + "Review must include a learning objective and references.", + ) + await session.rollback() From c5d305e33f8bda5207d59ba3bbdf086fa7b94e3a Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:38:25 +0500 Subject: [PATCH 11/36] feat: add stable daily reviews with activity-based learning streaks --- backend/app/api/v1/daily.py | 1 + backend/app/api/v1/me.py | 24 +++- backend/app/api/v1/reviews.py | 26 ++++ backend/app/api/v1/router.py | 4 +- backend/app/db/models.py | 5 +- backend/app/schemas/daily.py | 5 + backend/app/schemas/me.py | 4 +- backend/app/services/concepts.py | 1 + backend/app/services/interactions.py | 3 + backend/app/services/reminders.py | 4 + backend/app/services/reviews.py | 89 ++++++++++++ backend/app/services/selection.py | 61 ++++++-- backend/app/services/state.py | 7 +- backend/app/services/streaks.py | 8 +- backend/app/services/supply.py | 14 +- backend/migrations/0013_daily_reviews.sql | 17 +++ backend/tests/test_api.py | 18 +++ backend/tests/test_reviews.py | 162 ++++++++++++++++++++++ backend/tests/test_selection.py | 2 +- backend/tests/test_state_pagination.py | 2 +- 20 files changed, 428 insertions(+), 29 deletions(-) create mode 100644 backend/app/api/v1/reviews.py create mode 100644 backend/app/services/reviews.py create mode 100644 backend/migrations/0013_daily_reviews.sql create mode 100644 backend/tests/test_reviews.py diff --git a/backend/app/api/v1/daily.py b/backend/app/api/v1/daily.py index 2c5c30d..a555016 100644 --- a/backend/app/api/v1/daily.py +++ b/backend/app/api/v1/daily.py @@ -60,6 +60,7 @@ async def get_daily( topic_slug=concept.topic_slug, topic_name=concept.topic_name, like_count=concept.like_count, + content_version=concept.content_version, ), ) diff --git a/backend/app/api/v1/me.py b/backend/app/api/v1/me.py index b6c7e42..cc0b3c0 100644 --- a/backend/app/api/v1/me.py +++ b/backend/app/api/v1/me.py @@ -6,10 +6,16 @@ from app.db.session import get_db from app.deps import CurrentUser, get_current_user -from app.schemas.daily import ConceptOut, DailyOut +from app.schemas.daily import ConceptOut, DailyOut, ReviewOut from app.schemas.me import ( - HistoryPageOut, LearnedOut, ProfileIn, SavedConceptOut, SavedPageOut, StateOut, - StreakOut, TopicsIn, + HistoryPageOut, + LearnedOut, + ProfileIn, + SavedConceptOut, + SavedPageOut, + StateOut, + StreakOut, + TopicsIn, ) from app.schemas.notifications import NotificationPrefs, PushTokenIn from app.services.collections import history_page, saved_page @@ -40,6 +46,7 @@ def _daily_out_or_none(result: DailyResult) -> DailyOut | None: concept=ConceptOut( id=c.id, slug=c.slug, title=c.title, summary=c.summary, example=c.example, topic_slug=c.topic_slug, topic_name=c.topic_name, like_count=c.like_count, + content_version=c.content_version, ), ) @@ -73,6 +80,7 @@ def _to_state_out(state) -> StateOut: @router.get("/state", response_model=StateOut) async def get_state( compact: bool = False, + reviews: bool = False, user: CurrentUser = Depends(get_current_user), db: AsyncSession = Depends(get_db), ) -> StateOut: @@ -89,7 +97,15 @@ async def get_state( out = _to_state_out(state) # Fold today's concept in so the app needs one startup round trip (#102). # Same create-on-first-call behaviour as GET /v1/daily. - out.daily = _daily_out_or_none(await get_or_create_daily(db, user.id)) + result = await get_or_create_daily(db, user.id, allow_review=reviews) + out.daily = _daily_out_or_none(result) + if result.status == "review": + out.review = ReviewOut( + review_id=result.review_id,assigned_for=result.assigned_for, + assigned_at=result.assigned_at,completed_at=result.completed_at, + learned=result.completed_at is not None, + concept=ConceptOut(**vars(result.concept)), + ) return out diff --git a/backend/app/api/v1/reviews.py b/backend/app/api/v1/reviews.py new file mode 100644 index 0000000..1bfa22f --- /dev/null +++ b/backend/app/api/v1/reviews.py @@ -0,0 +1,26 @@ +import uuid + +from fastapi import APIRouter, Depends +from sqlalchemy.ext.asyncio import AsyncSession + +from app.db.session import get_db +from app.deps import CurrentUser, get_current_user +from app.schemas.me import CompletedOut, StreakOut +from app.services.reviews import complete_review +from app.services.streaks import compute_streaks + +router = APIRouter(prefix="/reviews", tags=["reviews"]) + + +@router.post("/{review_id}/complete", response_model=CompletedOut) +async def complete( + review_id: uuid.UUID, + user: CurrentUser = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + day = await complete_review(db, user.id, review_id) + return CompletedOut( + completed=True, + assigned_for=day, + stats=StreakOut(**vars(await compute_streaks(db, user.id))), + ) diff --git a/backend/app/api/v1/router.py b/backend/app/api/v1/router.py index 413382a..d09a1c5 100644 --- a/backend/app/api/v1/router.py +++ b/backend/app/api/v1/router.py @@ -1,9 +1,11 @@ from fastapi import APIRouter -from app.api.v1 import concepts, daily, me, topics +from app.api.v1 import concepts, daily, me, reviews, topics api_router = APIRouter(prefix="/v1") api_router.include_router(topics.router) api_router.include_router(daily.router) api_router.include_router(concepts.router) api_router.include_router(me.router) + +api_router.include_router(reviews.router) diff --git a/backend/app/db/models.py b/backend/app/db/models.py index c83680e..3b5d0bc 100644 --- a/backend/app/db/models.py +++ b/backend/app/db/models.py @@ -18,7 +18,7 @@ Time, UniqueConstraint, ) -from sqlalchemy.dialects.postgresql import ARRAY +from sqlalchemy.dialects.postgresql import ARRAY, JSONB from sqlalchemy.dialects.postgresql import UUID as PgUUID from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column @@ -63,6 +63,9 @@ class Concept(Base): difficulty: Mapped[int | None] = mapped_column(SmallInteger) status: Mapped[str] = mapped_column(Text, nullable=False, default="published") source: Mapped[str] = mapped_column(Text, nullable=False, default="seed") + content_version: Mapped[int] = mapped_column(Integer, nullable=False, default=1) + curriculum: Mapped[dict] = mapped_column(JSONB, nullable=False, default=dict) + published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) # Provenance of a generated lesson (null for seeded rows). model: Mapped[str | None] = mapped_column(Text) prompt_version: Mapped[str | None] = mapped_column(Text) diff --git a/backend/app/schemas/daily.py b/backend/app/schemas/daily.py index 9d03b0b..c93aed9 100644 --- a/backend/app/schemas/daily.py +++ b/backend/app/schemas/daily.py @@ -14,6 +14,7 @@ class ConceptOut(BaseModel): topic_name: str # Likes from other users; the client adds the viewer's own like. like_count: int = 0 + content_version: int = 1 class DailyOut(BaseModel): @@ -31,3 +32,7 @@ class DailyExhaustedOut(BaseModel): assigned_for: date reason: str = "catalog_exhausted" detail: str = "You have already been assigned every available concept." + + +class ReviewOut(DailyOut): + review_id: uuid.UUID diff --git a/backend/app/schemas/me.py b/backend/app/schemas/me.py index ab1553e..55bec25 100644 --- a/backend/app/schemas/me.py +++ b/backend/app/schemas/me.py @@ -2,13 +2,14 @@ from pydantic import BaseModel, Field -from app.schemas.daily import DailyOut +from app.schemas.daily import DailyOut, ReviewOut class StreakOut(BaseModel): current: int longest: int total_learned: int + total_reviews: int = 0 class SavedConceptOut(BaseModel): @@ -58,6 +59,7 @@ class StateOut(BaseModel): # (issue #102). Null when the catalog is exhausted for this user. This GET # creates the day's assignment on first call, exactly like GET /v1/daily. daily: DailyOut | None = None + review: ReviewOut | None = None class TopicsIn(BaseModel): diff --git a/backend/app/services/concepts.py b/backend/app/services/concepts.py index 9003059..d94d1c7 100644 --- a/backend/app/services/concepts.py +++ b/backend/app/services/concepts.py @@ -46,4 +46,5 @@ async def get_concept_out(db: AsyncSession, user_id, slug: str) -> ConceptOut | topic_slug=topic_slug, topic_name=topic_name, like_count=likes, + content_version=concept.content_version, ) diff --git a/backend/app/services/interactions.py b/backend/app/services/interactions.py index db9b15c..c264d1d 100644 --- a/backend/app/services/interactions.py +++ b/backend/app/services/interactions.py @@ -51,6 +51,8 @@ async def set_interaction( select id from public.daily_assignments where user_id = :uid and assigned_for in (cast(:today as date), cast(:today as date) - 1) + and not exists (select 1 from public.daily_reviews r where r.user_id=:uid + and r.assigned_for >= daily_assignments.assigned_for) order by assigned_for desc limit 1 ) @@ -71,6 +73,7 @@ async def complete_today(session: AsyncSession, user_id: uuid.UUID, today) -> Co The completion timestamp is the server's, not the client's; the returned `assigned_for` is the day it counts towards. """ + await session.execute(text("select id from public.profiles where id=:uid for update"), {"uid":user_id}) row = (await session.execute(_COMPLETE, {"uid": user_id, "today": today})).first() if row is None: raise HTTPException( diff --git a/backend/app/services/reminders.py b/backend/app/services/reminders.py index bc1c151..603b322 100644 --- a/backend/app/services/reminders.py +++ b/backend/app/services/reminders.py @@ -74,6 +74,8 @@ where da.user_id = o.user_id and da.assigned_for = o.local_date and da.completed_at is not null) + and not exists (select 1 from public.daily_reviews r + where r.user_id=o.user_id and r.assigned_for=o.local_date and r.completed_at is not null) and not exists ( -- A finished CURRENT day also silences yesterday's late slot: the -- push says "today's concept is waiting", and past midnight the @@ -82,6 +84,8 @@ where da.user_id = o.user_id and da.assigned_for = o.local_now::date and da.completed_at is not null) + and not exists (select 1 from public.daily_reviews r + where r.user_id=o.user_id and r.assigned_for=o.local_now::date and r.completed_at is not null) ) insert into public.reminder_log (user_id, local_date, slot) select user_id, local_date, slot from due diff --git a/backend/app/services/reviews.py b/backend/app/services/reviews.py new file mode 100644 index 0000000..f45e033 --- /dev/null +++ b/backend/app/services/reviews.py @@ -0,0 +1,89 @@ +"""Daily review selection and explicit, idempotent completion. + +Selection runs under the same profile row lock as new daily assignments. Review +never inserts an assignment and can only use a previously completed lesson. +""" + +import uuid +from datetime import date + +from fastapi import HTTPException +from sqlalchemy import text +from sqlalchemy.ext.asyncio import AsyncSession + +_EXISTING = text("""select r.id as review_id,r.assigned_for,r.assigned_at,r.completed_at, + c.id,c.slug,c.title,c.summary,c.example,c.content_version, + t.slug as topic_slug,t.name as topic_name, + (select count(*) from public.concept_interactions i where i.concept_id=c.id + and i.liked_at is not null and i.user_id<>:uid)::int as like_count + from public.daily_reviews r join public.concepts c on c.id=r.concept_id + join public.topics t on t.id=c.topic_id where r.user_id=:uid and r.assigned_for=:today""") + + +async def existing_review(session: AsyncSession, user_id: uuid.UUID, today: date): + from app.services.selection import _row_to_result + + row = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).first() + if row is None: + return None + result = _row_to_result(row, outside=False) + result.status = "review" + result.review_id = row.review_id + return result + + +async def choose_review(session: AsyncSession, user_id: uuid.UUID, today: date): + cid = await session.scalar( + text("""select a.concept_id from public.daily_assignments a + join public.concepts c on c.id=a.concept_id + left join lateral (select max(r.assigned_for) as last_review from public.daily_reviews r + where r.user_id=:uid and r.concept_id=a.concept_id) seen on true + where a.user_id=:uid and a.completed_at is not null and c.status='published' + order by seen.last_review asc nulls first,a.assigned_for,a.concept_id limit 1"""), + {"uid": user_id}, + ) + if cid is None: + return None + await session.execute( + text("""insert into public.daily_reviews(user_id,concept_id,assigned_for) + values (:uid,:cid,:today) on conflict(user_id,assigned_for) do nothing"""), + {"uid": user_id, "cid": cid, "today": today}, + ) + return await existing_review(session, user_id, today) + + +async def complete_review( + session: AsyncSession, + user_id: uuid.UUID, + review_id: uuid.UUID, + *, + today: date | None = None, +) -> date: + # Same lock/order as selection. A just-past-midnight tap may complete yesterday + # only while no later activity has been assigned. No client date is accepted. + server_today = await session.scalar( + text( + "select (now() at time zone timezone)::date from public.profiles where id=:uid for update" + ), + {"uid": user_id}, + ) + today = today or server_today + row = ( + await session.execute( + text("""update public.daily_reviews r + set completed_at=coalesce(completed_at,now()) where r.id=:id and r.user_id=:uid + and (r.completed_at is not null or ( + r.assigned_for in (cast(:today as date),cast(:today as date)-1) + and not exists(select 1 from public.daily_assignments a where a.user_id=:uid and a.assigned_for>r.assigned_for) + and not exists(select 1 from public.daily_reviews newer where newer.user_id=:uid and newer.assigned_for>r.assigned_for))) + returning assigned_for"""), + {"uid": user_id, "id": review_id, "today": today}, + ) + ).first() + if row is None: + await session.rollback() + raise HTTPException( + 409, detail="Review is unavailable or its completion window has ended" + ) + await session.commit() + return row.assigned_for diff --git a/backend/app/services/selection.py b/backend/app/services/selection.py index 70214db..017fd34 100644 --- a/backend/app/services/selection.py +++ b/backend/app/services/selection.py @@ -23,8 +23,8 @@ from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession -from app.services.prefetch import request_prefetch from app.config import get_settings +from app.services.prefetch import request_prefetch from app.services.supply import signal_reader @@ -40,11 +40,13 @@ class ConceptPayload: # Likes from OTHER users; the client adds the viewer's own like on top, so a # like/unlike is an instant +/-1 with no server round trip to see it. like_count: int = 0 + content_version: int = 1 @dataclass class DailyResult: - status: str # "ok" | "exhausted" + status: str # "ok" | "review" | "exhausted" + review_id: uuid.UUID | None = None assigned_for: date | None = None assigned_at: datetime | None = None completed_at: datetime | None = None @@ -55,12 +57,12 @@ class DailyResult: _TODAY = text(""" select timezone, (now() at time zone timezone)::date as today - from public.profiles where id = :uid + from public.profiles where id = :uid for update """) _EXISTING = text(""" select a.assigned_for, a.assigned_at, a.completed_at, - c.id, c.slug, c.title, c.summary, c.example, + c.id, c.slug, c.title, c.summary, c.example, c.content_version, t.slug as topic_slug, t.name as topic_name, (select count(*) from public.concept_interactions ci where ci.concept_id = c.id and ci.liked_at is not null @@ -119,8 +121,6 @@ class DailyResult: group by c.topic_id ) ls on ls.topic_id = ut.topic_id where ut.user_id = :uid - and exists (select 1 from public.concept_backlog b - where b.topic_id = ut.topic_id and b.status = 'pending') order by ls.seen_on asc nulls first limit 1 """) @@ -157,12 +157,13 @@ def _row_to_result(row, outside: bool) -> DailyResult: topic_slug=row.topic_slug, topic_name=row.topic_name, like_count=row.like_count, + content_version=row.content_version, ), outside_followed_topics=outside, ) -async def get_or_create_daily( +async def _select_new( session: AsyncSession, user_id: uuid.UUID, *, today: date | None = None ) -> DailyResult: """`today` is derived from the user's timezone in production. @@ -173,7 +174,9 @@ async def get_or_create_daily( if today is None: today = (await session.execute(_TODAY, {"uid": user_id})).one().today - existing = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).first() + existing = ( + await session.execute(_EXISTING, {"uid": user_id, "today": today}) + ).first() if existing: return _row_to_result(existing, outside=False) @@ -191,7 +194,7 @@ async def get_or_create_daily( await session.execute(_FOLLOWED_TOPIC_BY_STALENESS, {"uid": user_id}) ).scalar_one_or_none() if stale_topic is not None: - await signal_reader(session, user_id, stale_topic) + await signal_reader(session, user_id, stale_topic, commit=False) request_prefetch(stale_topic) outside = True @@ -211,14 +214,15 @@ async def get_or_create_daily( _INSERT, {"uid": user_id, "cid": concept_id, "today": today} ) ).scalar_one_or_none() - await session.commit() except IntegrityError: # Another device won the race, or the concept was assigned concurrently. await session.rollback() inserted = None if inserted is None: - row = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).first() + row = ( + await session.execute(_EXISTING, {"uid": user_id, "today": today}) + ).first() if row: return _row_to_result(row, outside=False) return DailyResult(status="exhausted", assigned_for=today) @@ -229,9 +233,40 @@ async def get_or_create_daily( watermark = ( await session.execute(_TOPIC_UNREAD, {"uid": user_id, "cid": concept_id}) ).first() - if watermark and watermark.topic_id is not None and watermark.unread <= get_settings().content_low_watermark: - await signal_reader(session, user_id, watermark.topic_id) + if ( + watermark + and watermark.topic_id is not None + and watermark.unread <= get_settings().content_low_watermark + ): + await signal_reader(session, user_id, watermark.topic_id, commit=False) request_prefetch(watermark.topic_id) row = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).one() return _row_to_result(row, outside=outside) + + +async def get_or_create_daily( + session: AsyncSession, + user_id: uuid.UUID, + *, + today: date | None = None, + allow_review: bool = False, +) -> DailyResult: + """Serialize the day's choice across devices and old/new client versions.""" + from app.services.reviews import choose_review, existing_review + + clock = (await session.execute(_TODAY, {"uid": user_id})).one() + today = today or clock.today + review = await existing_review(session, user_id, today) + if review: + await session.commit() + return ( + review + if allow_review + else DailyResult(status="exhausted", assigned_for=today) + ) + result = await _select_new(session, user_id, today=today) + if result.status == "exhausted" and allow_review: + result = await choose_review(session, user_id, today) or result + await session.commit() + return result diff --git a/backend/app/services/state.py b/backend/app/services/state.py index 3a6945b..e539344 100644 --- a/backend/app/services/state.py +++ b/backend/app/services/state.py @@ -121,7 +121,8 @@ class UserState: where a.user_id = :uid and a.assigned_for = (select today from prof) ), -- Gaps and islands: consecutive dates share (date - row_number()). - days as (select distinct assigned_for as d from learned_rows), + days as (select assigned_for as d from learned_rows + union select assigned_for from public.daily_reviews where user_id=:uid and completed_at is not null), grouped as (select d, d - (row_number() over (order by d))::int as grp from days), runs as (select grp, count(*)::int as len, max(d) as ends_on from grouped group by grp) select @@ -140,7 +141,8 @@ class UserState: where ends_on in (prof.today, prof.today - 1) order by ends_on desc limit 1), 0) as current_streak, coalesce((select max(len) from runs), 0) as longest_streak, - (select count(*)::int from days) as total_learned + (select count(*)::int from learned_rows) as total_learned, + (select count(*)::int from public.daily_reviews where user_id=:uid and completed_at is not null) as total_reviews from prof, followed, learned, interactions, saved """) @@ -184,6 +186,7 @@ async def load_state(session: AsyncSession, user_id: uuid.UUID, *, compact: bool current=row.current_streak, longest=row.longest_streak, total_learned=row.total_learned, + total_reviews=row.total_reviews, ), learned_before_window=dict(row.learned_before_window) if compact else None, history_next_cursor=(row.learned[-1]["on"] diff --git a/backend/app/services/streaks.py b/backend/app/services/streaks.py index f2902df..98938f0 100644 --- a/backend/app/services/streaks.py +++ b/backend/app/services/streaks.py @@ -19,6 +19,7 @@ class StreakStats: current: int longest: int total_learned: int + total_reviews: int = 0 # Gaps and islands: consecutive dates share (date - row_number()), so each @@ -28,6 +29,8 @@ class StreakStats: select distinct assigned_for as d from public.daily_assignments where user_id = :uid and completed_at is not null + union select assigned_for from public.daily_reviews + where user_id=:uid and completed_at is not null ), grouped as ( select d, d - (row_number() over (order by d))::int as grp from days ), runs as ( @@ -39,7 +42,8 @@ class StreakStats: where ends_on in (:today, :yesterday) order by ends_on desc limit 1), 0) as current, coalesce((select max(len) from runs), 0) as longest, - (select count(*)::int from days) as total_learned + (select count(*)::int from public.daily_assignments where user_id=:uid and completed_at is not null) as total_learned, + (select count(*)::int from public.daily_reviews where user_id=:uid and completed_at is not null) as total_reviews """) _TODAY = text(""" @@ -65,4 +69,4 @@ async def compute_streaks( ).one() # A run ending yesterday still counts as current, so an unfinished today # never shows the user a broken streak before the day is over. - return StreakStats(current=row.current, longest=row.longest, total_learned=row.total_learned) + return StreakStats(current=row.current, longest=row.longest, total_learned=row.total_learned, total_reviews=row.total_reviews) diff --git a/backend/app/services/supply.py b/backend/app/services/supply.py index 7e61762..6c2ddde 100644 --- a/backend/app/services/supply.py +++ b/backend/app/services/supply.py @@ -31,7 +31,11 @@ async def signal_reader( - session: AsyncSession, user_id: uuid.UUID, topic_id: uuid.UUID + session: AsyncSession, + user_id: uuid.UUID, + topic_id: uuid.UUID, + *, + commit: bool = True, ) -> None: settings = get_settings() await session.execute( @@ -44,7 +48,8 @@ async def signal_reader( "active_days": settings.content_active_days, }, ) - await session.commit() + if commit: + await session.commit() async def plan_active_readers(session: AsyncSession) -> None: @@ -55,6 +60,8 @@ async def plan_active_readers(session: AsyncSession) -> None: with active as ( select distinct user_id from public.daily_assignments where assigned_at>=now()-make_interval(days=>:days) + union select user_id from public.daily_reviews + where assigned_at>=now()-make_interval(days=>:days) ), demand as ( select ut.topic_id, coalesce(max(seen.n),0)::int+:reserve as target from active a join public.user_topics ut on ut.user_id=a.user_id @@ -67,7 +74,8 @@ async def plan_active_readers(session: AsyncSession) -> None: ) insert into public.content_supply_targets(topic_id,target_count,expires_at) select topic_id,target,now()+make_interval(days=>:days) from demand - on conflict(topic_id) do update set target_count=excluded.target_count, + on conflict(topic_id) do update set target_count=greatest(excluded.target_count, + case when content_supply_targets.expires_at>now() then content_supply_targets.target_count else 0 end), requested_at=now(),expires_at=excluded.expires_at """), { diff --git a/backend/migrations/0013_daily_reviews.sql b/backend/migrations/0013_daily_reviews.sql new file mode 100644 index 0000000..e4db910 --- /dev/null +++ b/backend/migrations/0013_daily_reviews.sql @@ -0,0 +1,17 @@ +-- Review is a separate daily activity; unique new-concept assignments stay intact. +begin; +create table public.daily_reviews ( + id uuid primary key default gen_random_uuid(), + user_id uuid not null references public.profiles(id) on delete cascade, + concept_id uuid not null references public.concepts(id) on delete restrict, + assigned_for date not null, + assigned_at timestamptz not null default now(), + completed_at timestamptz, + unique(user_id,assigned_for) +); +create index daily_reviews_rotation_idx on public.daily_reviews(user_id,concept_id,assigned_for desc); +create index daily_reviews_streak_idx on public.daily_reviews(user_id,assigned_for) + where completed_at is not null; +alter table public.daily_reviews enable row level security; +-- Backend-only writes. No mobile direct access. +commit; diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py index ed57c85..a345e17 100644 --- a/backend/tests/test_api.py +++ b/backend/tests/test_api.py @@ -372,3 +372,21 @@ async def test_state_folds_in_todays_concept(client, sessionmaker_for_test, user # The assignment was created by this call — the daily endpoint now agrees. daily = (await client.get("/v1/daily")).json() assert daily["concept"]["slug"] == state["daily"]["concept"]["slug"] + + +async def test_review_opt_in_and_completion_contract(client, session, user): + await session.execute(text("""insert into public.daily_assignments(user_id,concept_id,assigned_for,completed_at) + select :u,id,date '2000-01-01'+(row_number() over(order by id))::int,now() + from public.concepts where status='published'"""), {'u':user}) + await session.commit() + legacy = (await client.get('/v1/me/state?compact=true')).json() + assert legacy['daily'] is None and legacy['review'] is None + body = (await client.get('/v1/me/state?compact=true&reviews=true')).json() + assert body['daily'] is None and body['review']['learned'] is False + review_id = body['review']['review_id'] + response = await client.post(f'/v1/reviews/{review_id}/complete',json={'user_id':str(uuid.uuid4()),'assigned_for':'2000-01-01'}) + assert response.status_code == 200, response.text + assert response.json()['assigned_for']==body['today'] + assert response.json()['stats']['total_learned']==body['stats']['total_learned'] + assert response.json()['stats']['total_reviews']==1 + assert (await client.get('/v1/daily')).status_code==409 diff --git a/backend/tests/test_reviews.py b/backend/tests/test_reviews.py new file mode 100644 index 0000000..d7fa4c2 --- /dev/null +++ b/backend/tests/test_reviews.py @@ -0,0 +1,162 @@ +import asyncio +import uuid +from datetime import date, timedelta + +import pytest +from app.services.interactions import complete_today +from app.services.reviews import complete_review +from app.services.selection import get_or_create_daily +from app.services.state import load_state +from app.services.streaks import compute_streaks, local_today +from fastapi import HTTPException +from sqlalchemy import text + +DAY = date(2030, 1, 10) + + +@pytest.fixture +async def exhausted(session, user): + await session.execute( + text("""insert into public.daily_assignments(user_id,concept_id,assigned_for,completed_at) + select :u,id,cast(:day as date)-(row_number() over(order by id))::int,now() + from public.concepts where status='published' """), + {"u": user, "day": DAY}, + ) + await session.commit() + return user + + +async def test_review_is_explicit_rotates_and_does_not_inflate_learning( + session, exhausted +): + uid = exhausted + before = await compute_streaks(session, uid, DAY) + result = await get_or_create_daily(session, uid, today=DAY, allow_review=True) + assert result.status == "review" and result.completed_at is None + assert (await compute_streaks(session, uid, DAY)) == before + for _ in range(2): + assert await complete_review(session, uid, result.review_id, today=DAY) == DAY + after = await compute_streaks(session, uid, DAY) + assert after.total_learned == before.total_learned + assert after.total_reviews == 1 and after.current == before.current + 1 + assert ( + await get_or_create_daily(session, uid, today=DAY, allow_review=True) + ).completed_at + next_day = await get_or_create_daily( + session, uid, today=DAY + timedelta(days=1), allow_review=True + ) + assert next_day.concept.id != result.concept.id + + +async def test_two_devices_choose_and_complete_one_review( + session, exhausted, sessionmaker_for_test +): + async def select(): + async with sessionmaker_for_test() as other: + return await get_or_create_daily( + other, exhausted, today=DAY, allow_review=True + ) + + results = await asyncio.gather(*(select() for _ in range(6))) + assert len({r.review_id for r in results}) == 1 + + async def complete(): + async with sessionmaker_for_test() as other: + await complete_review(other, exhausted, results[0].review_id, today=DAY) + + await asyncio.gather(*(complete() for _ in range(6))) + assert (await compute_streaks(session, exhausted, DAY)).total_reviews == 1 + + +async def test_arriving_content_and_legacy_client_cannot_replace_review( + session, exhausted +): + result = await get_or_create_daily(session, exhausted, today=DAY, allow_review=True) + cid = await session.scalar( + text("""insert into public.concepts(topic_id,slug,title,summary) + select id,:slug,'Fresh lesson','Fresh body' from public.topics where is_active limit 1 returning id"""), + {"slug": "arrival-" + uuid.uuid4().hex}, + ) + await session.commit() + assert ( + await get_or_create_daily(session, exhausted, today=DAY) + ).status == "exhausted" + assert ( + await get_or_create_daily(session, exhausted, today=DAY, allow_review=True) + ).review_id == result.review_id + assert ( + await get_or_create_daily( + session, exhausted, today=DAY + timedelta(days=1), allow_review=True + ) + ).concept.id == cid + await session.execute(text('delete from public.daily_assignments where user_id=:u and concept_id=:c'), {'u':exhausted,'c':cid}) + await session.execute(text('delete from public.concepts where id=:c'), {'c':cid}) + await session.commit() + + +async def test_grace_rejects_backdating_wrong_account_and_newer_activity( + session, exhausted +): + review = await get_or_create_daily(session, exhausted, today=DAY, allow_review=True) + with pytest.raises(HTTPException): + await complete_review(session, uuid.uuid4(), review.review_id, today=DAY) + with pytest.raises(HTTPException): + await complete_review( + session, exhausted, review.review_id, today=DAY + timedelta(days=2) + ) + assert ( + await complete_review( + session, exhausted, review.review_id, today=DAY + timedelta(days=1) + ) + == DAY + ) + second = await get_or_create_daily( + session, exhausted, today=DAY + timedelta(days=1), allow_review=True + ) + await get_or_create_daily( + session, exhausted, today=DAY + timedelta(days=2), allow_review=True + ) + with pytest.raises(HTTPException): + await complete_review( + session, exhausted, second.review_id, today=DAY + timedelta(days=2) + ) + with pytest.raises(HTTPException): + await complete_today(session, exhausted, DAY) + await session.rollback() + + +async def test_no_completed_history_has_honest_exhaustion(session, exhausted): + await session.execute( + text("update public.daily_assignments set completed_at=null where user_id=:u"), + {"u": exhausted}, + ) + await session.commit() + assert ( + await get_or_create_daily(session, exhausted, today=DAY, allow_review=True) + ).status == "exhausted" + + +async def test_review_uses_profile_timezone_and_state_matches_streaks( + session, exhausted +): + await session.execute( + text("update public.profiles set timezone='Pacific/Kiritimati' where id=:u"), + {"u": exhausted}, + ) + await session.commit() + day = await local_today(session, exhausted) + review = await get_or_create_daily(session, exhausted, allow_review=True) + assert review.assigned_for == day + # Fixture history is in 2030; completion here remains constrained by the + # latest assignment. Move old history into the past for the timezone check. + await session.execute( + text( + "update public.daily_assignments set assigned_for=assigned_for-10000 where user_id=:u" + ), + {"u": exhausted}, + ) + await session.commit() + await complete_review(session, exhausted, review.review_id) + assert (await load_state(session, exhausted)).stats == await compute_streaks( + session, exhausted + ) diff --git a/backend/tests/test_selection.py b/backend/tests/test_selection.py index 24914cd..c8c353e 100644 --- a/backend/tests/test_selection.py +++ b/backend/tests/test_selection.py @@ -44,7 +44,7 @@ async def test_never_repeats_and_reports_exhaustion(session, user): generation can legitimately add concepts. """ catalog = await session.scalar( - text("select count(*) from public.concepts where status = 'published'") + text("select count(*) from public.concepts c join public.topics t on t.id=c.topic_id where c.status = 'published' and t.is_active") ) seen = [] for offset in range(catalog): diff --git a/backend/tests/test_state_pagination.py b/backend/tests/test_state_pagination.py index eb63eed..bd7968b 100644 --- a/backend/tests/test_state_pagination.py +++ b/backend/tests/test_state_pagination.py @@ -55,7 +55,7 @@ async def test_compact_state_bounds_details_without_losing_totals(collection_cli assert len(before["learned"]) == len(before["saved"]) == 365 assert len(after["learned"]) == len(after["saved"]) == 50 assert len(after["likes"]) == len(after["bookmarks"]) == 365 - assert after["stats"] == before["stats"] == {"current": 365, "longest": 365, "total_learned": 365} + assert after["stats"] == before["stats"] == {"current": 365, "longest": 365, "total_learned": 365, "total_reviews": 0} assert after["learned_before_window"] == {"Computer Science": 315} assert after["history_next_cursor"] and after["saved_next_cursor"] assert len(compact.content) < len(legacy.content) / 2 From ebf4c286a3fe6810978b7b2acf112265ec4daad2 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 12:38:29 +0500 Subject: [PATCH 12/36] feat: persist review completion through offline replay and account changes --- mobile/src/context/ProgressContext.tsx | 16 +++++ mobile/src/services/conceptApi.ts | 2 + mobile/src/services/dailyApi.ts | 1 + mobile/src/services/mutationOutbox.ts | 5 +- mobile/src/services/pendingProgress.ts | 18 +++++ mobile/src/services/progressRepository.ts | 3 + .../src/services/remoteProgressRepository.ts | 71 +++++++++++++++---- mobile/src/services/topics.ts | 4 +- mobile/src/types/index.ts | 14 ++-- mobile/tests/reviewProgress.test.mjs | 39 ++++++++++ 10 files changed, 150 insertions(+), 23 deletions(-) create mode 100644 mobile/tests/reviewProgress.test.mjs diff --git a/mobile/src/context/ProgressContext.tsx b/mobile/src/context/ProgressContext.tsx index 783f2ce..90366e6 100644 --- a/mobile/src/context/ProgressContext.tsx +++ b/mobile/src/context/ProgressContext.tsx @@ -22,6 +22,7 @@ import { createSyncLoop } from '../services/syncLoop'; import { fetchTopics } from '../services/topicsApi'; import { useAuth } from './AuthContext'; import { EMPTY_PROGRESS } from '../services/storage'; +import { withCompletedReview } from '../services/pendingProgress'; import { computeStreaks, StreakStats } from '../services/streak'; export interface ProgressContextValue { @@ -41,6 +42,7 @@ export interface ProgressContextValue { * signed in) so the recorded concept, title, and topic match it; falls back * to the locally-selected concept when omitted. */ markLearned: (target?: Concept) => void; + completeReview: (reviewId: string) => void; toggleTopic: (category: Category) => void; toggleLike: (conceptId: string) => void; toggleBookmark: (conceptId: string, title?: string, topicName?: string) => void; @@ -265,6 +267,7 @@ export function ProgressProvider({ children, repository: override }: Props) { ...prev, learned, stats: { + ...prev.stats, current, longest: Math.max(prev.stats?.longest ?? 0, current), totalLearned: (prev.stats?.totalLearned ?? prev.learned.length) + 1, @@ -285,6 +288,7 @@ export function ProgressProvider({ children, repository: override }: Props) { learned: prev.learned.filter((r) => r.date !== today), stats: prev.stats ? { + ...prev.stats, current: Math.max(0, prev.stats.current - 1), longest: prev.stats.longest, totalLearned: Math.max(0, prev.stats.totalLearned - 1), @@ -294,6 +298,16 @@ export function ProgressProvider({ children, repository: override }: Props) { ); }, [apply, concept, repository, today]); + const completeReview = useCallback((reviewId: string) => { + if (!repository.completeReview) return; + const before = progress; + void apply( + state => withCompletedReview(state, reviewId), + () => repository.completeReview!(reviewId), + state => ({ ...state, serverDaily: before.serverDaily, stats: before.stats }), + ); + }, [apply, repository, progress]); + const toggleTopic = useCallback( (category: Category) => { const toggle = (prev: ProgressState) => ({ @@ -351,6 +365,7 @@ export function ProgressProvider({ children, repository: override }: Props) { hasLearned, streaks, markLearned, + completeReview, toggleTopic, toggleLike, toggleBookmark, @@ -365,6 +380,7 @@ export function ProgressProvider({ children, repository: override }: Props) { hasLearned, streaks, markLearned, + completeReview, toggleTopic, toggleLike, toggleBookmark, diff --git a/mobile/src/services/conceptApi.ts b/mobile/src/services/conceptApi.ts index 76186c3..061abbb 100644 --- a/mobile/src/services/conceptApi.ts +++ b/mobile/src/services/conceptApi.ts @@ -15,6 +15,7 @@ interface ConceptResponse { topic_slug: string; topic_name: string; like_count?: number; + content_version?: number; } /** @@ -31,6 +32,7 @@ async function downloadConcept(slug: string): Promise { summary: c.summary, example: c.example ?? undefined, likeCount: c.like_count ?? 0, + contentVersion: c.content_version ?? 1, }; } diff --git a/mobile/src/services/dailyApi.ts b/mobile/src/services/dailyApi.ts index eeeda74..af86d0b 100644 --- a/mobile/src/services/dailyApi.ts +++ b/mobile/src/services/dailyApi.ts @@ -19,6 +19,7 @@ export function toConcept(payload: DailyPayload): Concept { summary: payload.concept.summary, example: payload.concept.example ?? undefined, likeCount: payload.concept.like_count ?? 0, + contentVersion: payload.concept.content_version ?? 1, }; } diff --git a/mobile/src/services/mutationOutbox.ts b/mobile/src/services/mutationOutbox.ts index 4e952b9..0ea18fc 100644 --- a/mobile/src/services/mutationOutbox.ts +++ b/mobile/src/services/mutationOutbox.ts @@ -10,7 +10,8 @@ export type QueuedMutation = | { kind: 'topics'; slugs: string[] } // The date it was completed: /v1/daily/complete only completes "today", so a // 'learn' queued on a previous day must be dropped, not replayed (#133). - | { kind: 'learn'; date: string }; + | { kind: 'learn'; date: string } + | { kind: 'review'; reviewId: string; date: string }; /** Stable coalescing key — one pending intent per (kind, target). */ export function keyOf(m: QueuedMutation): string { @@ -23,6 +24,8 @@ export function keyOf(m: QueuedMutation): string { return 'topics'; case 'learn': return 'learn'; + case 'review': + return `review:${m.reviewId}`; } } diff --git a/mobile/src/services/pendingProgress.ts b/mobile/src/services/pendingProgress.ts index 63617d4..1d688dc 100644 --- a/mobile/src/services/pendingProgress.ts +++ b/mobile/src/services/pendingProgress.ts @@ -21,6 +21,8 @@ export function withPendingProgress( if (mutation.desired && row) saved.push(row); state = { ...state, savedConcepts: saved }; } + } else if (mutation.kind === 'review') { + state = withCompletedReview(state, mutation.reviewId); } else if (mutation.kind === 'learn' && mutation.date === state.assignment?.date) { const record = previous.learned.find(row => row.date === mutation.date); if (record && !state.learned.some(row => row.date === mutation.date)) { @@ -30,3 +32,19 @@ export function withPendingProgress( } return state; } + + +/** Apply once to the matching server-assigned activity, never a new lesson. + * The payload date identifies the learning day; no device timestamp is sent. */ +export function withCompletedReview(state: ProgressState, reviewId: string): ProgressState { + const daily = state.serverDaily; + if (daily?.status !== 'review' || daily.payload.review_id !== reviewId || daily.payload.learned) return state; + const alreadyLearnedDay = state.learned.some(row => row.date === daily.payload.assigned_for); + const stats = state.stats ? { + ...state.stats, + current: state.stats.current + (alreadyLearnedDay ? 0 : 1), + longest: Math.max(state.stats.longest, state.stats.current + (alreadyLearnedDay ? 0 : 1)), + totalReviews: (state.stats.totalReviews ?? 0) + 1, + } : undefined; + return { ...state, stats, serverDaily: { ...daily, payload: { ...daily.payload, learned: true } } }; +} diff --git a/mobile/src/services/progressRepository.ts b/mobile/src/services/progressRepository.ts index e691b1e..4d1877c 100644 --- a/mobile/src/services/progressRepository.ts +++ b/mobile/src/services/progressRepository.ts @@ -30,6 +30,9 @@ export interface ProgressRepository { topicName?: string ): Promise; + /** Complete the identified review; never changes unique learned records. */ + completeReview?(reviewId: string): Promise; + /** Follow / unfollow a topic. Server-side: PUT /v1/me/topics. */ toggleTopic(category: Category): Promise; diff --git a/mobile/src/services/remoteProgressRepository.ts b/mobile/src/services/remoteProgressRepository.ts index 1566d58..51727e1 100644 --- a/mobile/src/services/remoteProgressRepository.ts +++ b/mobile/src/services/remoteProgressRepository.ts @@ -1,10 +1,10 @@ import AsyncStorage from '@react-native-async-storage/async-storage'; import { ApiError, apiRequest } from '../api/client'; -import { Category, DailyPayload, ProgressState } from '../types'; +import { Category, DailyPayload, ReviewPayload, ProgressState } from '../types'; import { todayKey } from './dates'; import { cacheSavedConcepts, conceptCache } from './conceptApi'; import { toConcept } from './dailyApi'; -import { withPendingProgress } from './pendingProgress'; +import { withPendingProgress, withCompletedReview } from './pendingProgress'; import { clearQueue, dequeue, enqueue, keyOf, pending, QueuedMutation } from './mutationQueue'; import { ProgressRepository } from './progressRepository'; import { EMPTY_PROGRESS } from './storage'; @@ -34,9 +34,10 @@ interface StatePayload { saved?: { concept_slug: string; title?: string; topic_name?: string; like_count?: number }[]; learned_before_window?: Record | null; saved_next_cursor?: string | null; - stats: { current: number; longest: number; total_learned: number }; + stats: { current: number; longest: number; total_learned: number; total_reviews?: number }; assignment_slug: string | null; daily?: DailyPayload | null; + review?: ReviewPayload | null; } function toProgressState(payload: StatePayload): ProgressState { @@ -70,12 +71,15 @@ function toProgressState(payload: StatePayload): ProgressState { current: payload.stats.current, longest: payload.stats.longest, totalLearned: payload.stats.total_learned, + totalReviews: payload.stats.total_reviews ?? 0, }, // Today's concept, folded in (#102). A fresh fetch is never "stale"; the // offline flag is set only when load() falls back to cache after a failure. serverDaily: payload.daily ? { status: 'ok', payload: payload.daily, stale: false } - : { status: 'exhausted' }, + : payload.review + ? { status: 'review', payload: payload.review, stale: false } + : { status: 'exhausted' }, }; } @@ -107,8 +111,8 @@ export class RemoteProgressRepository implements ProgressRepository { // Keep each day's full text for later History/Saved reading, and download // saved bodies without holding up the initial screen. const contentEpoch = conceptCache.epoch; - if (payload.daily) { - const concept = toConcept(payload.daily); + if (payload.daily || payload.review) { + const concept = toConcept((payload.daily ?? payload.review)!); await conceptCache.set(concept.id, concept, contentEpoch).catch(() => {}); } if (epoch !== this.epoch) return EMPTY_PROGRESS; @@ -132,7 +136,7 @@ export class RemoteProgressRepository implements ProgressRepository { try { this.cache = JSON.parse(raw) as ProgressState; // Also upgrade an existing installation's cached Today while offline. - if (this.cache.serverDaily?.status === 'ok') { + if (this.cache.serverDaily?.status === 'ok' || this.cache.serverDaily?.status === 'review') { const concept = toConcept(this.cache.serverDaily.payload); await conceptCache.set(concept.id, concept, contentEpoch).catch(() => {}); } @@ -146,7 +150,7 @@ export class RemoteProgressRepository implements ProgressRepository { async load(): Promise { const epoch = this.epoch; try { - return await this.fromState(await apiRequest('/v1/me/state?compact=true'), epoch); + return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { const raw = await AsyncStorage.getItem(CACHE_KEY).catch(() => null); if (raw && epoch === this.epoch) { @@ -156,7 +160,7 @@ export class RemoteProgressRepository implements ProgressRepository { // cache-first preview (loadCached) leaves it not-stale, so the banner // still doesn't flash during a normal load (#92). const offline: ProgressState = - cached.serverDaily?.status === 'ok' + (cached.serverDaily?.status === 'ok' || cached.serverDaily?.status === 'review') ? { ...cached, serverDaily: { ...cached.serverDaily, stale: true } } : cached; this.cache = offline; @@ -181,7 +185,7 @@ export class RemoteProgressRepository implements ProgressRepository { let done: { completed: boolean; assigned_for: string; - stats: { current: number; longest: number; total_learned: number }; + stats: { current: number; longest: number; total_learned: number; total_reviews?: number }; }; try { done = await apiRequest('/v1/daily/complete', { method: 'POST' }); @@ -200,6 +204,7 @@ export class RemoteProgressRepository implements ProgressRepository { already || !this.cache.stats ? this.cache.stats : { + ...this.cache.stats, current: this.cache.stats.current + 1, longest: Math.max(this.cache.stats.longest, this.cache.stats.current + 1), totalLearned: this.cache.stats.totalLearned + 1, @@ -216,7 +221,7 @@ export class RemoteProgressRepository implements ProgressRepository { // guess (the caller's concept id, no title). Without this the History tab // only caught up on a full reload, i.e. an app restart (issue #91). try { - return await this.fromState(await apiRequest('/v1/me/state?compact=true'), epoch); + return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { // The completion already persisted; a failed reload must not roll it back. // Patch in place using the caller's concept id (the cached assignment can @@ -232,11 +237,46 @@ export class RemoteProgressRepository implements ProgressRepository { current: done.stats.current, longest: done.stats.longest, totalLearned: done.stats.total_learned, + totalReviews: done.stats.total_reviews ?? this.cache.stats?.totalReviews ?? 0, }, }, epoch); } } + async completeReview(reviewId: string): Promise { + const epoch = this.epoch; + const daily = this.cache.serverDaily; + if (daily?.status !== 'review' || daily.payload.review_id !== reviewId) return this.cache; + // Persist intent before I/O. A successful server write followed by a lost + // response or process restart is safe to replay using this exact review ID. + await enqueue({ kind: 'review', reviewId, date: daily.payload.assigned_for }); + if (epoch !== this.epoch) return EMPTY_PROGRESS; + const before = this.cache; + const optimistic = withCompletedReview(before, reviewId); + await this.remember(optimistic, epoch); + try { + const done = await apiRequest<{ + stats: {current: number; longest: number; total_learned: number; total_reviews: number}; + }>(`/v1/reviews/${encodeURIComponent(reviewId)}/complete`, { method: 'POST' }); + if (epoch !== this.epoch) return EMPTY_PROGRESS; + await dequeue(`review:${reviewId}`); + return this.remember({ ...this.cache, stats: { + current: done.stats.current, longest: done.stats.longest, + totalLearned: done.stats.total_learned, totalReviews: done.stats.total_reviews, + } }, epoch); + } catch (error) { + if (epoch !== this.epoch) return EMPTY_PROGRESS; + if (isOffline(error) || (error instanceof ApiError && (error.status >= 500 || error.status === 429))) { + return optimistic; // durable retry; do not turn an uncertain write into a lost tap + } + await dequeue(`review:${reviewId}`); + // Restore the uncompleted snapshot even if the reconciliation request + // also fails; an expired review must not keep an invented completed day. + await this.remember({ ...this.cache, serverDaily: before.serverDaily, stats: before.stats }, epoch); + return this.load(); + } + } + async toggleTopic(category: Category): Promise { const epoch = this.epoch; const following = this.cache.followedTopics.includes(category); @@ -350,7 +390,7 @@ export class RemoteProgressRepository implements ProgressRepository { // by the UI. Fall back to patching in place; the saved list catches up on // the next successful load. try { - return await this.fromState(await apiRequest('/v1/me/state?compact=true'), epoch); + return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { return this.remember(patched(), epoch); } @@ -387,14 +427,14 @@ export class RemoteProgressRepository implements ProgressRepository { } catch (err) { if (epoch !== this.epoch) return null; if (isOffline(err)) return null; // still offline — keep the rest queued - if (err instanceof ApiError && err.status >= 500) continue; // transient — retry next time + if (err instanceof ApiError && (err.status >= 500 || err.status === 429)) continue; // transient — retry next time await dequeue(keyOf(m), m); // 4xx: unfixable, drop so it can't block forever } } if (epoch !== this.epoch) return null; try { - return await this.fromState(await apiRequest('/v1/me/state?compact=true'), epoch); + return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { return null; } @@ -415,6 +455,9 @@ export class RemoteProgressRepository implements ProgressRepository { case 'topics': await apiRequest('/v1/me/topics?compact=true', { method: 'PUT', body: { topics: m.slugs } }); return; + case 'review': + await apiRequest(`/v1/reviews/${encodeURIComponent(m.reviewId)}/complete`, { method: 'POST' }); + return; case 'learn': await apiRequest('/v1/daily/complete', { method: 'POST' }); return; diff --git a/mobile/src/services/topics.ts b/mobile/src/services/topics.ts index 8100425..fcb6d70 100644 --- a/mobile/src/services/topics.ts +++ b/mobile/src/services/topics.ts @@ -17,9 +17,9 @@ const CATEGORY_BY_SLUG: Record = Object.fromEntries( ) as Record; export function toCategory(slug: string): Category | null { - return CATEGORY_BY_SLUG[slug] ?? null; + return CATEGORY_BY_SLUG[slug] ?? slug; } export function toSlug(category: Category): string { - return SLUG_BY_CATEGORY[category]; + return SLUG_BY_CATEGORY[category] ?? category; } diff --git a/mobile/src/types/index.ts b/mobile/src/types/index.ts index f1ddeef..834609a 100644 --- a/mobile/src/types/index.ts +++ b/mobile/src/types/index.ts @@ -9,14 +9,11 @@ export interface Concept { example?: string; /** Likes from other users; the viewer's own like is added on top for display. */ likeCount?: number; + contentVersion?: number; } -export type Category = - | 'Artificial Intelligence' - | 'Software Engineering' - | 'Computer Science' - | 'Mathematics' - | 'Linux & Systems'; +/** Display labels come from the subject registry. The bundled five are demo data. */ +export type Category = string; export const CATEGORIES: Category[] = [ 'Artificial Intelligence', @@ -65,12 +62,16 @@ export interface DailyPayload { topic_slug: string; topic_name: string; like_count?: number; + content_version?: number; }; } /** The server's daily result: today's concept, exhausted, or unavailable. */ +export interface ReviewPayload extends DailyPayload { review_id: string } + export type DailyOutcome = | { status: 'ok'; payload: DailyPayload; stale: boolean } + | { status: 'review'; payload: ReviewPayload; stale: boolean } | { status: 'exhausted' } | { status: 'unavailable' }; @@ -85,6 +86,7 @@ export interface StreakStats { current: number; longest: number; totalLearned: number; + totalReviews?: number; } /** Everything the app persists locally. */ diff --git a/mobile/tests/reviewProgress.test.mjs b/mobile/tests/reviewProgress.test.mjs new file mode 100644 index 0000000..c7dc15c --- /dev/null +++ b/mobile/tests/reviewProgress.test.mjs @@ -0,0 +1,39 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { withCompletedReview, withPendingProgress } from '../src/services/pendingProgress.ts'; +import { MutationOutbox } from '../src/services/mutationOutbox.ts'; + +const state = { + learned: [{conceptId:'known',date:'2026-09-12'}], assignment:null, + followedTopics:[],likes:[],bookmarks:[], + stats:{current:8,longest:10,totalLearned:25,totalReviews:2}, + serverDaily:{status:'review',stale:false,payload:{review_id:'review-1',assigned_for:'2026-09-13',learned:false,concept:{slug:'known'}}}, +}; +test('review completion advances activity once and leaves unique learning untouched', () => { + const once=withCompletedReview(state,'review-1'); + assert.deepEqual(once.learned,state.learned); + assert.deepEqual(once.stats,{current:9,longest:10,totalLearned:25,totalReviews:3}); + assert.equal(withCompletedReview(once,'review-1'),once); + assert.equal(withCompletedReview(state,'different-review'),state); +}); +test('pending completion survives stale refresh but cannot complete another day or review', () => { + const queued=[{kind:'review',reviewId:'review-1',date:'2026-09-13'}]; + const once=withCompletedReview(state,'review-1'); + const merged=withPendingProgress(state,once,queued); + assert.equal(merged.serverDaily.payload.learned,true); + assert.equal(merged.stats.totalReviews,3); + assert.equal(withPendingProgress(merged,once,queued).stats.totalReviews,3); + const next={...state,serverDaily:{...state.serverDaily,payload:{...state.serverDaily.payload,review_id:'review-2'}}}; + assert.equal(withPendingProgress(next,once,queued).serverDaily.payload.learned,false); +}); +test('review outbox survives restart, coalesces two taps, and clears on account change', async () => { + const rows=new Map(); + const disk={getItem:async k=>rows.get(k)??null,setItem:async(k,v)=>rows.set(k,v),removeItem:async k=>rows.delete(k)}; + const first=new MutationOutbox(disk,'queue'); + const action={kind:'review',reviewId:'review-1',date:'2026-09-13'}; + await Promise.all([first.enqueue(action),first.enqueue(action)]); + const restarted=new MutationOutbox(disk,'queue'); + assert.deepEqual(await restarted.pending(),[action]); + await restarted.clear(); + assert.deepEqual(await restarted.pending(),[]); +}); From 9bc36a858bfb1c446c39c7d585faf74b0cd8e476 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 13:06:53 +0500 Subject: [PATCH 13/36] feat: expose protected curriculum maintenance and content health operations --- backend/.env.example | 8 + backend/app/config.py | 2 + backend/app/services/content_health.py | 201 ++++++++++++++++++ backend/app/services/curriculum.py | 28 ++- backend/app/workers/content.py | 172 +++++++++++++++ backend/app/workers/pool_topup.py | 59 +++-- backend/content/curriculum.example.json | 95 +++++++++ .../migrations/0014_content_operations.sql | 18 ++ backend/tests/test_content_health.py | 99 +++++++++ backend/tests/test_curriculum.py | 60 ++++++ 10 files changed, 722 insertions(+), 20 deletions(-) create mode 100644 backend/app/services/content_health.py create mode 100644 backend/app/workers/content.py create mode 100644 backend/content/curriculum.example.json create mode 100644 backend/migrations/0014_content_operations.sql create mode 100644 backend/tests/test_content_health.py diff --git a/backend/.env.example b/backend/.env.example index a27c6ed..153276c 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -50,3 +50,11 @@ GENERATION_ON_DEMAND=true # --- HTTP ----------------------------------------------------------------- # Comma-separated. Expo web dev server during development. ALLOWED_ORIGINS=http://localhost:8081 + +# Sustainable shared content supply and operations (see docs/CONTENT_OPERATIONS.md) +CONTENT_RESERVE_PER_TOPIC=60 +CONTENT_LOW_WATERMARK=5 +CONTENT_ACTIVE_DAYS=90 +CONTENT_PLANNED_RESERVE=90 +CONTENT_GENERATION_BATCH=5 +GENERATION_MAX_CONCURRENT=3 diff --git a/backend/app/config.py b/backend/app/config.py index 3807795..c391d78 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -40,8 +40,10 @@ class Settings(BaseSettings): content_reserve_per_topic: int = Field(default=60, ge=1, le=365) content_low_watermark: int = Field(default=5, ge=0, le=30) content_active_days: int = Field(default=90, ge=1, le=365) + content_planned_reserve: int = Field(default=90, ge=1, le=1000) content_generation_batch: int = Field(default=5, ge=1, le=25) # Shared by all generation paths; zero prevents new reservations. + generation_max_concurrent: int = Field(default=3, ge=1, le=20) generation_daily_call_cap: int = Field(default=200, ge=0) # Seconds between worker calls; the free tier allows ~10 requests a minute. generation_pace_seconds: float = 6.0 diff --git a/backend/app/services/content_health.py b/backend/app/services/content_health.py new file mode 100644 index 0000000..9fa7976 --- /dev/null +++ b/backend/app/services/content_health.py @@ -0,0 +1,201 @@ +"""Backend-only operational snapshot. No user IDs or raw provider errors escape.""" + +from sqlalchemy import text +from sqlalchemy.ext.asyncio import AsyncSession + +from app.config import get_settings + +_FAILURE_CATEGORY = """case + when last_error like '%slug%' then 'slug_collision' + when last_error ~* 'summary|example|json|length|style|schema' then 'content_validation' + when last_error ~* 'timeout|connect|network' then 'provider_transport' + when last_error ~* '429|rate.limit' then 'provider_throttled' + when last_error ~* '(http|gemini).*[45][0-9][0-9]' then 'provider_http' + else 'other_or_unclassified' end""" + + +async def failed_items(session: AsyncSession, after: str = "") -> list[dict]: + rows = await session.execute( + text(f"""select b.slug,t.slug as topic,b.title, + b.attempts,b.updated_at,{_FAILURE_CATEGORY} as category, + (select count(*) from public.content_retry_log r where r.backlog_id=b.id) as retry_grants + from public.concept_backlog b join public.topics t on t.id=b.topic_id + where b.status='failed' and b.slug>:after order by b.slug limit 100"""), + {"after": after}, + ) + return [dict(r) for r in rows.mappings()] + + +_REPORT = text("""with active as ( + select user_id from public.daily_assignments where assigned_at>=now()-make_interval(days=>:days) + union select user_id from public.daily_reviews where assigned_at>=now()-make_interval(days=>:days) +), consumed as ( + select ut.topic_id,max(seen.n)::int as max_assigned from active a + join public.user_topics ut on ut.user_id=a.user_id + left join lateral (select count(*) as n from public.daily_assignments da + join public.concepts c on c.id=da.concept_id where da.user_id=a.user_id + and c.topic_id=ut.topic_id and c.status='published') seen on true group by ut.topic_id +) +select t.slug,t.is_active, + (select count(*) from public.concepts c where c.topic_id=t.id and status='published')::int as published, + (select count(*) from public.concepts c where c.topic_id=t.id and status='draft')::int as drafts, + (select count(*) from public.concept_revisions r join public.concepts c on c.id=r.concept_id + where c.topic_id=t.id and r.status='draft')::int as revisions_awaiting_review, + (select count(*) from public.concept_backlog b where b.topic_id=t.id and status='pending' + and attempts<3+(select count(*) from public.content_retry_log l where l.backlog_id=b.id))::int as pending, + (select count(*) from public.concept_backlog b where b.topic_id=t.id and (status='failed' + or (status='pending' and attempts>=3+(select count(*) from public.content_retry_log l where l.backlog_id=b.id))))::int as failed, + (select count(*) from public.concept_backlog b where b.topic_id=t.id and status='generating')::int as generating, + (select count(*) from public.concept_backlog b where b.topic_id=t.id and status='generating' + and (claimed_at is null or claimed_atnow()),0)) as target +from public.topics t left join consumed on consumed.topic_id=t.id order by t.sort_order,t.slug""") + + +async def health_report(session: AsyncSession) -> dict: + settings = get_settings() + rows = ( + ( + await session.execute( + _REPORT, + { + "days": settings.content_active_days, + "floor": settings.min_pool_per_topic, + }, + ) + ) + .mappings() + .all() + ) + topics = [] + conditions = set() + for row in rows: + topic = dict(row) + topic["unseen_for_experienced_reader"] = max( + 0, row["published"] - row["experienced_assigned"] + ) + # Conservative one lesson/subject/day. This is a stock estimate, not a + # promise of publication dates or measured per-reader consumption speed. + topic["estimated_reserve_days"] = topic["unseen_for_experienced_reader"] + topics.append(topic) + if not row["is_active"]: + continue + for name, active in { + "low_reserve": topic["estimated_reserve_days"] + <= settings.content_low_watermark, + "below_reserve_target": topic["estimated_reserve_days"] + < settings.content_reserve_per_topic, + "empty_queue": row["pending"] == 0, + "low_planned_reserve": row["pending"] < settings.content_planned_reserve, + "failed_generation": row["failed"] > 0, + "stale_claims": row["stale"] > 0, + "awaiting_review": row["revisions_awaiting_review"] > 0, + }.items(): + if active: + conditions.add(f"{row['slug']}:{name}") + used = await session.scalar( + text( + "select coalesce((select calls_used from public.generation_daily_usage where budget_day=(statement_timestamp() at time zone 'America/Los_Angeles')::date),0)" + ) + ) + worker = ( + ( + await session.execute( + text( + "select *,finished_at= settings.generation_daily_call_cap: + conditions.add("daily_budget_exhausted") + if ( + not worker + or worker["finished_at"] is None + or worker["overdue"] + or worker["outcome"] == "failed" + ): + conditions.add("scheduled_worker_needs_attention") + corrections = ( + ( + await session.execute( + text("""select count(*)::int as generating, + count(*) filter(where created_at list[dict]: + """Return transitions once, including recovery; caller commits before emitting.""" + await session.execute(text("select pg_advisory_xact_lock(195,2)")) + previous = dict( + ( + await session.execute( + text("select key,active from public.content_conditions") + ) + ).all() + ) + active = set(conditions) + transitions = [] + for key in sorted(set(previous) | active): + value = key in active + if previous.get(key, False) != value: + transitions.append( + {"condition": key, "state": "active" if value else "recovered"} + ) + await session.execute( + text("""insert into public.content_conditions(key,active) values (:k,:v) + on conflict(key) do update set active=excluded.active,observed_at=now(), + changed_at=case when content_conditions.active<>excluded.active then now() else content_conditions.changed_at end"""), + {"k": key, "v": value}, + ) + return transitions diff --git a/backend/app/services/curriculum.py b/backend/app/services/curriculum.py index f2b66df..880a673 100644 --- a/backend/app/services/curriculum.py +++ b/backend/app/services/curriculum.py @@ -83,8 +83,8 @@ async def import_subjects(session: AsyncSession, items: list[Subject]) -> int: async def validate_graph(session: AsyncSession, additions: dict[str, dict]) -> None: rows = ( await session.execute( - text("""select slug,curriculum from public.concepts - union all select slug,curriculum from public.concept_backlog""") + text("""select slug,curriculum from public.concept_backlog + union all select slug,curriculum from public.concepts""") ) ).all() graph = {r.slug: r.curriculum.get("prerequisites", []) for r in rows} @@ -111,7 +111,7 @@ def visit(slug): async def import_lessons( - session: AsyncSession, items: list[PlannedLesson] + session: AsyncSession, items: list[PlannedLesson], *, revise: bool = False ) -> list[str]: """Idempotent exact re-import; changed existing plans need explicit editorial work.""" import json @@ -158,22 +158,38 @@ async def import_lessons( raise ValueError(f"Unknown or retired subject {item.topic_slug}") previous = ( await session.execute( - text("select * from public.concept_backlog where slug=:s"), + text("select * from public.concept_backlog where slug=:s for update"), {"s": item.slug}, ) ).first() data = item.curriculum.model_dump(mode="json") if previous: + if previous.topic_id != topic: + raise ValueError("A plan cannot move between subject identities") if ( previous.topic_id, previous.title, previous.angle or "", previous.curriculum, ) != (topic, item.title, item.angle, data): - raise ValueError( - f"{item.slug} already exists with a different plan; review it explicitly" + if not revise or previous.status not in ("pending", "failed"): + raise ValueError( + f"{item.slug} already exists with a different plan; only pending/failed plans can be revised explicitly" + ) + await session.execute( + text("""update public.concept_backlog set title=:title,angle=:angle, + difficulty=:difficulty,curriculum=cast(:data as jsonb) where id=:id"""), + { + "id": previous.id, + "title": item.title, + "angle": item.angle, + "difficulty": item.curriculum.difficulty, + "data": json.dumps(data), + }, ) continue + if revise: + raise ValueError(f"No existing plan for {item.slug}") if any(row.slug == item.slug for row in existing): raise ValueError(f"{item.slug} already exists in the catalog") await session.execute( diff --git a/backend/app/workers/content.py b/backend/app/workers/content.py new file mode 100644 index 0000000..f2f7c0d --- /dev/null +++ b/backend/app/workers/content.py @@ -0,0 +1,172 @@ +"""Protected maintainer CLI: python -m app.workers.content --help. + +Uses backend credentials; intentionally has no public HTTP route. Import files +are validated before mutation and every operation is a single transaction. +""" + +import argparse +import asyncio +import json +import uuid +from pathlib import Path + +from pydantic import TypeAdapter +from sqlalchemy import text + +from app.db.session import SessionLocal, engine +from app.services.curriculum import ( + PlannedLesson, + Subject, + import_lessons, + import_subjects, +) +from app.services.publication import ( + LessonBody, + publish_revision, + retry_failed, + stage_revision, +) + + +def parser(): + root = argparse.ArgumentParser(description=__doc__) + sub = root.add_subparsers(dest="command", required=True) + for name in ("import-subjects", "import-curriculum", "revise-plan"): + p = sub.add_parser(name) + p.add_argument("file", type=Path) + p = sub.add_parser( + "stage", help="Stage a corrected lesson body without replacing published text" + ) + p.add_argument("slug") + p.add_argument("file", type=Path) + sub.add_parser("drafts") + p = sub.add_parser( + "report", help="Content health; --observe emits only condition transitions" + ) + p.add_argument("--observe", action="store_true") + p = sub.add_parser("failures", help="Redacted failed titles, up to 100 per page") + p.add_argument("--after", default="", help="Continue after the last slug") + p = sub.add_parser("show") + p.add_argument("revision", type=uuid.UUID) + for name in ("publish", "reject"): + p = sub.add_parser(name) + p.add_argument("revision", type=uuid.UUID) + p.add_argument("--reviewed-by", required=True) + p.add_argument("--note", required=True) + p = sub.add_parser("retry") + p.add_argument("slug") + p.add_argument("--operator", required=True) + p.add_argument("--reason", required=True) + return root + + +async def run(args): + try: + async with SessionLocal() as session: + async with session.begin(): + if args.command == "failures": + from app.services.content_health import failed_items + + result = await failed_items(session, args.after) + elif args.command == "report": + from app.services.content_health import ( + health_report, + observe_conditions, + ) + + result = await health_report(session) + if args.observe: + result = await observe_conditions(session, result["conditions"]) + elif args.command == "import-subjects": + rows = TypeAdapter(list[Subject]).validate_json( + args.file.read_text() + ) + result = {"subjects": await import_subjects(session, rows)} + elif args.command in ("import-curriculum", "revise-plan"): + rows = TypeAdapter(list[PlannedLesson]).validate_json( + args.file.read_text() + ) + result = { + "planned": len(rows), + "overlap_warnings": await import_lessons( + session, rows, revise=args.command == "revise-plan" + ), + } + elif args.command == "stage": + result = { + "revision": str( + await stage_revision( + session, + args.slug, + LessonBody.model_validate_json(args.file.read_text()), + ) + ) + } + elif args.command == "publish": + result = { + "version": await publish_revision( + session, args.revision, args.reviewed_by, args.note + ) + } + elif args.command == "reject": + if not args.reviewed_by.strip() or len(args.note.strip()) < 10: + raise ValueError("Record reviewer and rejection reason") + count = await session.scalar( + text("""with changed as ( + update public.concept_revisions set status='rejected',reviewed_at=now(), + reviewed_by=:who,review_note=:note where id=:id and status='draft' returning id + ) select count(*) from changed"""), + { + "id": args.revision, + "who": args.reviewed_by, + "note": args.note, + }, + ) + result = {"rejected": count} + elif args.command == "retry": + await retry_failed(session, args.slug, args.operator, args.reason) + result = {"retry_granted": args.slug} + elif args.command == "show": + row = ( + ( + await session.execute( + text( + "select * from public.concept_revisions where id=:id" + ), + {"id": args.revision}, + ) + ) + .mappings() + .first() + ) + if row is None: + raise ValueError("Unknown revision") + result = dict(row) + else: + result = [ + dict(r) + for r in ( + await session.execute( + text("""select r.id,c.slug, + r.base_version,r.created_at from public.concept_revisions r + join public.concepts c on c.id=r.concept_id where r.status='draft' + order by r.created_at limit 100""") + ) + ).mappings() + ] + if args.command != "report" or not args.observe or result: + print(json.dumps(result, default=str, indent=2)) + finally: + await engine.dispose() + + +def main(): + args = parser().parse_args() + try: + asyncio.run(run(args)) + except ValueError as exc: + raise SystemExit(str(exc)) from exc + + +if __name__ == "__main__": + main() diff --git a/backend/app/workers/pool_topup.py b/backend/app/workers/pool_topup.py index 441ef5a..c548687 100644 --- a/backend/app/workers/pool_topup.py +++ b/backend/app/workers/pool_topup.py @@ -7,6 +7,8 @@ import asyncio import logging +from sqlalchemy import text + from app.config import get_settings from app.db.session import SessionLocal, engine from app.services.pool import top_up @@ -14,28 +16,57 @@ async def main() -> None: - logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + logging.basicConfig( + level=logging.INFO, format="%(levelname)s %(name)s: %(message)s" + ) settings = get_settings() - async with SessionLocal() as session: - await plan_active_readers(session) - result = await top_up( - session, - api_key=settings.gemini_api_key, - model=settings.gemini_model, - enabled=settings.generation_enabled, - minimum_per_topic=settings.min_pool_per_topic, - call_cap=settings.generation_daily_call_cap, - pace_seconds=settings.generation_pace_seconds, - ) + try: + async with SessionLocal() as session: + await session.execute( + text("""insert into public.content_worker_runs(worker,outcome) + values ('pool_topup','running') on conflict(worker) do update + set started_at=now(),finished_at=null,outcome='running',generated=0,failed=0""") + ) + await session.commit() + try: + await plan_active_readers(session) + result = await top_up( + session, + api_key=settings.gemini_api_key, + model=settings.gemini_model, + enabled=settings.generation_enabled, + minimum_per_topic=settings.min_pool_per_topic, + call_cap=settings.generation_daily_call_cap, + pace_seconds=settings.generation_pace_seconds, + ) + except BaseException: + await session.rollback() + await session.execute( + text( + "update public.content_worker_runs set outcome='failed',finished_at=now() where worker='pool_topup'" + ) + ) + await session.commit() + raise + await session.execute( + text("""update public.content_worker_runs set outcome=:outcome, + finished_at=now(),generated=:generated,failed=:failed where worker='pool_topup'"""), + { + "outcome": result.skipped_reason or "completed", + "generated": result.generated, + "failed": result.failed, + }, + ) + await session.commit() + finally: + await engine.dispose() if result.skipped_reason: logging.info("nothing to do: %s", result.skipped_reason) else: logging.info("generated %s, failed %s", result.generated, result.failed) - await engine.dispose() - if __name__ == "__main__": asyncio.run(main()) diff --git a/backend/content/curriculum.example.json b/backend/content/curriculum.example.json new file mode 100644 index 0000000..5fb9cd6 --- /dev/null +++ b/backend/content/curriculum.example.json @@ -0,0 +1,95 @@ +[ + { + "slug": "evaluation-data-leakage", + "topic_slug": "artificial-intelligence", + "title": "Evaluation Data Leakage", + "angle": "Use one concrete example and verify the distinction against the reference.", + "curriculum": { + "objective": "Identify information that must stay outside model fitting and explain why leaking it makes evaluation misleading.", + "difficulty": 1, + "prerequisites": [ + "cross-validation" + ], + "references": [ + { + "title": "scikit-learn: Common pitfalls", + "url": "https://scikit-learn.org/stable/common_pitfalls.html" + } + ] + } + }, + { + "slug": "read-committed-snapshots", + "topic_slug": "software-engineering", + "title": "Read Committed Snapshots", + "angle": "Use one concrete example and verify the distinction against the reference.", + "curriculum": { + "objective": "Explain why two queries in one Read Committed transaction can observe different committed data.", + "difficulty": 2, + "prerequisites": [ + "acid-transactions" + ], + "references": [ + { + "title": "PostgreSQL 16: Transaction isolation", + "url": "https://www.postgresql.org/docs/16/transaction-iso.html" + } + ] + } + }, + { + "slug": "double-ended-queues", + "topic_slug": "computer-science", + "title": "Double-ended Queues", + "angle": "Use one concrete example and verify the distinction against the reference.", + "curriculum": { + "objective": "Choose a deque when a workload frequently inserts and removes elements at both ends.", + "difficulty": 2, + "prerequisites": [ + "stacks-and-queues" + ], + "references": [ + { + "title": "Python: collections.deque", + "url": "https://docs.python.org/3/library/collections.html#collections.deque" + } + ] + } + }, + { + "slug": "probability-sample-spaces", + "topic_slug": "mathematics", + "title": "Probability Sample Spaces", + "angle": "Use one concrete example and verify the distinction against the reference.", + "curriculum": { + "objective": "List a sample space and distinguish an individual outcome from an event containing several outcomes.", + "difficulty": 1, + "prerequisites": [], + "references": [ + { + "title": "OpenStax: Probability terminology", + "url": "https://openstax.org/books/introductory-statistics-2e/pages/3-1-terminology" + } + ] + } + }, + { + "slug": "redirection-evaluation-order", + "topic_slug": "linux-systems", + "title": "Redirection Evaluation Order", + "angle": "Use one concrete example and verify the distinction against the reference.", + "curriculum": { + "objective": "Trace how left-to-right redirection changes which destination receives a command’s standard error.", + "difficulty": 3, + "prerequisites": [ + "pipes-and-redirection" + ], + "references": [ + { + "title": "GNU Bash: Redirections", + "url": "https://www.gnu.org/software/bash/manual/html_node/Redirections.html" + } + ] + } + } +] diff --git a/backend/migrations/0014_content_operations.sql b/backend/migrations/0014_content_operations.sql new file mode 100644 index 0000000..6c668bb --- /dev/null +++ b/backend/migrations/0014_content_operations.sql @@ -0,0 +1,18 @@ +begin; +create table public.content_worker_runs ( + worker text primary key, + started_at timestamptz not null default now(), + finished_at timestamptz, + outcome text not null, + generated integer not null default 0, + failed integer not null default 0 +); +alter table public.content_worker_runs enable row level security; +create table public.content_conditions ( + key text primary key, + active boolean not null, + changed_at timestamptz not null default now(), + observed_at timestamptz not null default now() +); +alter table public.content_conditions enable row level security; +commit; diff --git a/backend/tests/test_content_health.py b/backend/tests/test_content_health.py new file mode 100644 index 0000000..b368836 --- /dev/null +++ b/backend/tests/test_content_health.py @@ -0,0 +1,99 @@ +import asyncio +import uuid +from unittest.mock import AsyncMock + +from app.config import get_settings +from app.services import pool +from app.services.content_health import health_report, observe_conditions +from app.services.generation import GenerationError +from app.services.publication import retry_failed +from sqlalchemy import text + + +async def test_report_redacts_raw_errors_and_counts_exhausted_attempts( + session, monkeypatch +): + settings = get_settings() + monkeypatch.setattr(settings, "generation_enabled", False) + slug = "health-" + uuid.uuid4().hex + tid = await session.scalar( + text( + "insert into public.topics(slug,name) values (:s,'Health fixture') returning id" + ), + {"s": slug}, + ) + await session.execute( + text("""insert into public.concept_backlog(topic_id,slug,title,status,attempts,last_error) + values (:t,:s,'Fixture','failed',3,'invalid JSON secret-token=do-not-display')"""), + {"t": tid, "s": slug}, + ) + report = await health_report(session) + assert "secret-token" not in str(report) + assert str(tid) not in str(report) + assert "generation_disabled" in report["conditions"] + assert f"{slug}:failed_generation" in report["conditions"] + topic = next(t for t in report["topics"] if t["slug"] == slug) + assert topic["pending"] == 0 and topic["failed"] == 1 + await session.rollback() + + +async def test_condition_transitions_are_deduplicated_and_report_recovery( + session, sessionmaker_for_test +): + await session.execute(text("delete from public.content_conditions")) + await session.commit() + + async def observe(): + async with sessionmaker_for_test() as other: + changes = await observe_conditions(other, ["fixture:low_reserve"]) + await other.commit() + return changes + + batches = await asyncio.gather(*(observe() for _ in range(5))) + assert sum(len(b) for b in batches) == 1 + assert await observe_conditions(session, []) == [ + {"condition": "fixture:low_reserve", "state": "recovered"} + ] + await session.commit() + assert await observe_conditions(session, []) == [] + await session.rollback() + + +async def test_failed_retry_preserves_attempts_and_grants_only_one_more_call( + session, empty_generation_budget, monkeypatch +): + slug = "retry-" + uuid.uuid4().hex + tid = await session.scalar( + text( + "insert into public.topics(slug,name) values (:s,'Retry fixture') returning id" + ), + {"s": slug}, + ) + await session.execute( + text("""insert into public.concept_backlog(topic_id,slug,title,status,attempts) + values (:t,:s,'Fixture','failed',3)"""), + {"t": tid, "s": slug}, + ) + await retry_failed( + session, + slug, + "Maintainer", + "Corrected the prompt and reviewed the provider failure.", + ) + await session.commit() + generate = AsyncMock(side_effect=GenerationError("bad response")) + monkeypatch.setattr(pool, "generate_concept", generate) + assert await pool.generate_one(session, "k", "m", tid, call_cap=10) is None + assert await pool.generate_one(session, "k", "m", tid, call_cap=10) is None + assert generate.await_count == 1 + row = ( + await session.execute( + text("select status,attempts from public.concept_backlog where slug=:s"), + {"s": slug}, + ) + ).one() + assert tuple(row) == ("failed", 4) + await session.execute( + text("update public.topics set is_active=false where id=:t"), {"t": tid} + ) + await session.commit() diff --git a/backend/tests/test_curriculum.py b/backend/tests/test_curriculum.py index 173c807..c3fa4a3 100644 --- a/backend/tests/test_curriculum.py +++ b/backend/tests/test_curriculum.py @@ -77,3 +77,63 @@ async def test_curriculum_rejects_cycles_missing_prerequisites_and_duplicate_obj with pytest.raises(ValueError, match="Exact duplicate"): await import_lessons(session, [first, second]) await session.rollback() + + +async def test_revising_a_failed_plan_preserves_attempts_and_requires_explicit_retry( + session, +): + slug = "revision-" + uuid.uuid4().hex + await import_subjects(session, [Subject(slug=slug, name="Revision fixture")]) + item = plan(slug, slug) + await import_lessons(session, [item]) + await session.execute( + text( + "update public.concept_backlog set attempts=3,status='failed' where slug=:s" + ), + {"s": slug}, + ) + item.angle = "Clarify the known validation failure without changing identity." + with pytest.raises(ValueError, match="explicitly"): + await import_lessons(session, [item]) + await import_lessons(session, [item], revise=True) + row = ( + await session.execute( + text( + "select attempts,status,angle from public.concept_backlog where slug=:s" + ), + {"s": slug}, + ) + ).one() + assert tuple(row) == (3, "failed", item.angle) + await session.rollback() + + +async def test_checked_in_registry_and_extension_examples_import_without_changing_subject_ids( + session, +): + from pathlib import Path + from pydantic import TypeAdapter + + folder = Path(__file__).resolve().parents[1] / "content" + before = ( + await session.execute( + text("select slug,id from public.topics where is_active order by slug") + ) + ).all() + subjects = TypeAdapter(list[Subject]).validate_json( + (folder / "subjects.json").read_text() + ) + lessons = TypeAdapter(list[PlannedLesson]).validate_json( + (folder / "curriculum.example.json").read_text() + ) + await import_subjects(session, subjects) + assert len(subjects) == len(lessons) == 5 + await import_lessons(session, lessons) + await import_lessons(session, lessons) + after = ( + await session.execute( + text("select slug,id from public.topics where is_active order by slug") + ) + ).all() + assert before == after + await session.rollback() From 49ad80f4e6bcb395830b3ae6f5c5c707ffdfd38e Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 13:58:57 +0500 Subject: [PATCH 14/36] fix: bound concurrent generation and recover abandoned editorial claims --- backend/app/services/generation_budget.py | 25 +++++++ backend/app/services/pool.py | 9 ++- backend/app/services/prefetch.py | 21 ++++-- backend/app/workers/rewrite_catalog.py | 70 ++++++++++++++++--- .../0015_revision_generation_claims.sql | 8 +++ backend/tests/conftest.py | 15 ++++ backend/tests/test_generation.py | 10 ++- backend/tests/test_generation_limits.py | 37 ++++++++++ 8 files changed, 177 insertions(+), 18 deletions(-) create mode 100644 backend/migrations/0015_revision_generation_claims.sql diff --git a/backend/app/services/generation_budget.py b/backend/app/services/generation_budget.py index cdebbad..66c89dc 100644 --- a/backend/app/services/generation_budget.py +++ b/backend/app/services/generation_budget.py @@ -36,3 +36,28 @@ async def reserve_generation_call(session: AsyncSession, call_cap: int) -> None: used = (await session.execute(_RESERVE, {"cap": call_cap})).scalar_one_or_none() if used is None: raise GenerationBudgetExhausted("daily generation call cap reached") + + +class GenerationBusy(RuntimeError): + """All shared provider slots are in use; retry on a later worker run.""" + + +async def check_generation_capacity(session: AsyncSession) -> None: + """Call AFTER reserving quota and BEFORE committing a durable work claim. + + The day's quota row lock serializes this count with all other claimers. + Uncommitted claims do not reach a provider. Denial rolls back both the claim + and its reservation; finished work immediately releases its counted slot. + """ + from app.config import get_settings + + await session.execute(text("select pg_advisory_xact_lock(195,3)")) + count = await session.scalar( + text("""select + (select count(*) from public.concept_backlog where status='generating' + and claimed_at>=now()-interval '30 minutes') + + (select count(*) from public.concept_revisions where status='generating' + and created_at>=now()-interval '30 minutes')""") + ) + if count > get_settings().generation_max_concurrent: + raise GenerationBusy("shared generation concurrency limit reached") diff --git a/backend/app/services/pool.py b/backend/app/services/pool.py index f4b8607..0b016db 100644 --- a/backend/app/services/pool.py +++ b/backend/app/services/pool.py @@ -18,6 +18,8 @@ from app.services.generation import GenerationError, RateLimitedError, generate_concept from app.services.generation_budget import ( GenerationBudgetExhausted, + GenerationBusy, + check_generation_capacity, reserve_generation_call, ) from app.services.supply import target_for @@ -66,7 +68,7 @@ # too rather than leaving them stuck forever. _REAP_STALE = text(""" update public.concept_backlog - set status = 'pending', claimed_at = null + set status = case when attempts >= 3 + (select count(*) from public.content_retry_log r where r.backlog_id=concept_backlog.id) then 'failed' else 'pending' end, claimed_at = null where status = 'generating' and (claimed_at is null or claimed_at < now() - make_interval(mins => :max_minutes)) @@ -172,6 +174,7 @@ async def generate_one( claimed = (await session.execute(_CLAIM, {"topic_id": topic_id})).first() if claimed is not None: await reserve_generation_call(session, cap) + await check_generation_capacity(session) await session.commit() except BaseException: # Quota denial/DB failure/cancellation must undo the claim and its attempt. @@ -210,7 +213,7 @@ async def generate_one( except GenerationError as exc: # Leave it pending for another attempt; give up after three so one bad # title cannot block the queue forever. - log.warning("generation failed for %s: %s", claimed.slug, exc) + log.warning("generation failed for %s (%s); inspect the protected backlog", claimed.slug, type(exc).__name__) await session.execute( _FAIL, {"backlog_id": claimed.id, "error": str(exc)[:500]} ) @@ -288,6 +291,8 @@ async def top_up( call_cap=call_cap, supply_target=target, ) + except GenerationBusy: + return TopUpResult(generated, failed, "generation capacity busy") except GenerationBudgetExhausted: log.info("stopping: shared daily call cap of %s reached", call_cap) return TopUpResult(generated, failed, "daily call cap reached") diff --git a/backend/app/services/prefetch.py b/backend/app/services/prefetch.py index 9c2ea3d..1323a1e 100644 --- a/backend/app/services/prefetch.py +++ b/backend/app/services/prefetch.py @@ -18,7 +18,7 @@ from app.config import get_settings from app.db.session import SessionLocal from app.services.generation import RateLimitedError -from app.services.generation_budget import GenerationBudgetExhausted +from app.services.generation_budget import GenerationBudgetExhausted, GenerationBusy from app.services.pool import generate_one from app.services.supply import target_for @@ -88,11 +88,24 @@ async def _run(topic_id: uuid.UUID) -> None: break try: concept_id = await generate_one( - session, settings.gemini_api_key, settings.gemini_model, topic_id, - call_cap=settings.generation_daily_call_cap, supply_target=target, + session, + settings.gemini_api_key, + settings.gemini_model, + topic_id, + call_cap=settings.generation_daily_call_cap, + supply_target=target, ) + except GenerationBusy: + log.info( + "prefetch for topic %s stopped: generation capacity busy", + topic_id, + ) + break except GenerationBudgetExhausted: - log.info("prefetch for topic %s stopped: daily call cap reached", topic_id) + log.info( + "prefetch for topic %s stopped: daily call cap reached", + topic_id, + ) break except RateLimitedError: log.info("prefetch for topic %s stopped: rate limited", topic_id) diff --git a/backend/app/workers/rewrite_catalog.py b/backend/app/workers/rewrite_catalog.py index c551954..4c28f9e 100644 --- a/backend/app/workers/rewrite_catalog.py +++ b/backend/app/workers/rewrite_catalog.py @@ -21,6 +21,8 @@ ) from app.services.generation_budget import ( GenerationBudgetExhausted, + GenerationBusy, + check_generation_capacity, reserve_generation_call, ) @@ -36,19 +38,53 @@ and t.is_active and coalesce(c.prompt_version, '') <> :pv and not exists (select 1 from public.concept_revisions r where r.concept_id=c.id - and r.status='draft') + and r.status in ('draft','generating')) order by c.created_at """) +_CLAIM = text(""" + insert into public.concept_revisions(concept_id,base_version,body,status) + select id,content_version,'{}'::jsonb,'generating' from public.concepts c + where id=:id and content_version=:version and status='published' + and exists(select 1 from public.topics t where t.id=c.topic_id and t.is_active) + and not exists(select 1 from public.concept_revisions r where r.concept_id=c.id + and r.status in ('draft','generating')) returning id +""") + _UPDATE = text(""" - insert into public.concept_revisions(concept_id,base_version,body) - select id,content_version,jsonb_build_object('title',title,'summary',cast(:summary as text), - 'example',cast(:example as text),'curriculum',curriculum,'model',cast(:model as text),'prompt_version',cast(:pv as text)) - from public.concepts where id=:id and content_version=:version - and not exists(select 1 from public.concept_revisions r where r.concept_id=:id and r.status='draft') + update public.concept_revisions r set body=jsonb_build_object('title',c.title, + 'summary',cast(:summary as text),'example',cast(:example as text), + 'curriculum',c.curriculum,'model',cast(:model as text),'prompt_version',cast(:pv as text)), + status='draft' + from public.concepts c where r.id=:revision and r.concept_id=c.id + and r.status='generating' and c.content_version=r.base_version """) +async def _claim(session, row, cap): + await session.execute( + text("select id from public.concepts where id=:id for update"), {"id": row.id} + ) + revision = await session.scalar( + _CLAIM, {"id": row.id, "version": row.content_version} + ) + if revision: + await reserve_generation_call(session, cap) + await check_generation_capacity(session) + await session.commit() + return revision + + +async def _release(session, revision): + await session.execute( + text( + "delete from public.concept_revisions where id=:id and status='generating'" + ), + {"id": revision}, + ) + await session.commit() + + async def main() -> None: logging.basicConfig( level=logging.INFO, format="%(levelname)s %(name)s: %(message)s" @@ -63,6 +99,11 @@ async def main() -> None: log.info("no API key configured; catalog unchanged") return async with SessionLocal() as session: + await session.execute( + text( + "delete from public.concept_revisions where status='generating' and created_at None: for row in todo: while True: try: - await reserve_generation_call( - session, settings.generation_daily_call_cap + revision = await _claim( + session, row, settings.generation_daily_call_cap ) - await session.commit() + if not revision: + break result = await generate_concept( title=row.title, topic_name=row.topic_name, @@ -87,6 +129,10 @@ async def main() -> None: api_key=settings.gemini_api_key, model=settings.gemini_model, ) + except GenerationBusy: + await session.rollback() + log.info("shared generation capacity busy; resume on a later run") + return except GenerationBudgetExhausted: await session.rollback() log.info( @@ -96,6 +142,7 @@ async def main() -> None: ) return except RateLimitedError as exc: + await _release(session, revision) streak += 1 if streak >= MAX_RATE_LIMIT_STREAK: log.warning("giving up: %s consecutive rate limits", streak) @@ -113,8 +160,9 @@ async def main() -> None: backoff = min(backoff * 2, BACKOFF_MAX) continue except GenerationError as exc: + await _release(session, revision) # Leave it on the old prompt version; a later run retries it. - log.warning("skipping %s: %s", row.title, exc) + log.warning("skipping %s (%s)", row.title, type(exc).__name__) failed += 1 break streak, backoff = 0, BACKOFF_START @@ -122,7 +170,7 @@ async def main() -> None: _UPDATE, { "id": row.id, - "version": row.content_version, + "revision": revision, "summary": result.summary, "example": result.example, "model": result.model, diff --git a/backend/migrations/0015_revision_generation_claims.sql b/backend/migrations/0015_revision_generation_claims.sql new file mode 100644 index 0000000..319e17a --- /dev/null +++ b/backend/migrations/0015_revision_generation_claims.sql @@ -0,0 +1,8 @@ +-- Durable claims for bulk correction drafting; no database transaction spans model I/O. +begin; +alter table public.concept_revisions drop constraint concept_revisions_status_check; +alter table public.concept_revisions add constraint concept_revisions_status_check + check(status in ('generating','draft','published','rejected')); +create index concept_revisions_generating_idx on public.concept_revisions(created_at) + where status='generating'; +commit; diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index eab4a5b..8b918c0 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -128,3 +128,18 @@ async def empty_generation_budget(session): await session.rollback() await session.execute(text("delete from public.generation_daily_usage")) await session.commit() + + +@pytest.fixture(autouse=True) +def no_live_http(monkeypatch): + """All HTTP integration uses ASGI/MockTransport; never contact a real provider.""" + import httpx + + async def reject_async(self, request): + raise AssertionError('Live HTTP transport is disabled in tests; use a mock transport') + + def reject_sync(self, request): + raise AssertionError('Live HTTP transport is disabled in tests; use a mock transport') + + monkeypatch.setattr(httpx.AsyncHTTPTransport, 'handle_async_request', reject_async) + monkeypatch.setattr(httpx.HTTPTransport, 'handle_request', reject_sync) diff --git a/backend/tests/test_generation.py b/backend/tests/test_generation.py index cf438a6..189fd46 100644 --- a/backend/tests/test_generation.py +++ b/backend/tests/test_generation.py @@ -247,7 +247,10 @@ async def test_stale_generating_rows_are_reclaimed(empty_generation_budget, sess """), {"tid": topic_id}) await session.commit() - # minimum_per_topic=0 → no generation happens; only the reaper runs. + # Clear reader demand too: the bootstrap floor alone no longer controls + # ongoing refill. This test exercises only the stale-claim reaper. + await session.execute(text('delete from public.content_supply_targets')) + await session.commit() await top_up(session, api_key="k", model="m", enabled=True, minimum_per_topic=0, call_cap=100) @@ -258,6 +261,11 @@ async def test_stale_generating_rows_are_reclaimed(empty_generation_budget, sess assert rows["stranded"] == "pending", "the abandoned claim must be reclaimed" assert rows["pre-migration"] == "pending", "a NULL-claimed leftover is stale too" assert rows["in-flight"] == "generating", "a fresh claim must be left alone" + # This artificial live claim must not occupy a global provider slot for + # later tests in the shared database. + await session.execute(text('delete from public.concept_backlog where topic_id=:tid'), {'tid':topic_id}) + await session.execute(text('delete from public.topics where id=:tid'), {'tid':topic_id}) + await session.commit() async def test_slug_collision_does_not_mark_the_backlog_done(empty_generation_budget, session, patch_httpx): diff --git a/backend/tests/test_generation_limits.py b/backend/tests/test_generation_limits.py index 54188e4..132552a 100644 --- a/backend/tests/test_generation_limits.py +++ b/backend/tests/test_generation_limits.py @@ -302,3 +302,40 @@ async def test_rewrite_cancelled_provider_keeps_budget_and_cleans_up(topic, gene assert await calls_used(session) == 1 assert await rewritten_count(session, topic) == 0 rewrite.engine.dispose.assert_awaited_once() + + +async def test_two_rewrite_workers_claim_one_revision_before_reserving(topic,generator,session,rewrite_config,sessionmaker_for_test): + row=(await session.execute(rewrite._TODO,{'pv':rewrite.PROMPT_VERSION})).first() + await session.commit() + async def claim(): + async with sessionmaker_for_test() as other: + return await rewrite._claim(other,row,10) + claims=await asyncio.gather(*(claim() for _ in range(8))) + assert sum(c is not None for c in claims)==1 + assert await calls_used(session)==1 + generator.assert_not_awaited() + + +async def test_global_concurrency_denial_refunds_unstarted_call(topic,generator,session,monkeypatch,sessionmaker_for_test): + from app.config import get_settings + from app.services.generation_budget import GenerationBusy + monkeypatch.setattr(get_settings(),'generation_max_concurrent',1) + entered,release=asyncio.Event(),asyncio.Event() + async def blocked(**kwargs): + entered.set() + await release.wait() + return GeneratedConcept(summary='Fixture',example='Fixture',model='fixture') + generator.side_effect=blocked + async def run(): + async with sessionmaker_for_test() as other: + return await pool.generate_one(other,'k','m',topic,call_cap=10) + first=asyncio.create_task(run()) + await asyncio.wait_for(entered.wait(),5) + try: + with pytest.raises(GenerationBusy): + await run() + assert await calls_used(session)==1 + finally: + release.set() + await first + assert generator.await_count==1 From 64285a4bf282a3f10dc00f07c4df43f5af24e99c Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 13:59:32 +0500 Subject: [PATCH 15/36] fix: sequence curriculum lessons and start refill after target commit --- backend/app/api/v1/me.py | 8 +++++++- backend/app/services/selection.py | 28 ++++++++++++++++++++++------ backend/tests/test_selection.py | 17 +++++++++++++++++ 3 files changed, 46 insertions(+), 7 deletions(-) diff --git a/backend/app/api/v1/me.py b/backend/app/api/v1/me.py index cc0b3c0..55603be 100644 --- a/backend/app/api/v1/me.py +++ b/backend/app/api/v1/me.py @@ -87,8 +87,14 @@ async def get_state( """Everything the app needs to render, in one request. Bootstrapping only runs when the state query finds no profile, so the - common path costs a single round trip. + profile lock keeps the state totals consistent with concurrent completions. """ + # Keep completion totals and the folded activity from the same point in + # time when another device completes a review during startup. + await db.execute( + text("select id from public.profiles where id=:uid for update"), + {"uid": user.id}, + ) state = await load_state(db, user.id, compact=compact) if state is None: await ensure_bootstrapped(db, user.id, user.email) diff --git a/backend/app/services/selection.py b/backend/app/services/selection.py index 017fd34..7fb95ea 100644 --- a/backend/app/services/selection.py +++ b/backend/app/services/selection.py @@ -77,7 +77,12 @@ class DailyResult: # within that topic. _CANDIDATE = text(""" with pool as ( - select c.id, c.topic_id + select c.id, c.topic_id,coalesce(c.difficulty,1) as level, + (select count(*) from jsonb_array_elements_text(coalesce(c.curriculum->'prerequisites','[]'::jsonb)) required(slug) + where not exists(select 1 from public.daily_assignments learned + join public.concepts prior on prior.id=learned.concept_id + where learned.user_id=:uid and learned.completed_at is not null + and prior.slug=required.slug)) as unmet from public.concepts c join public.topics t on t.id=c.topic_id and t.is_active where c.status = 'published' @@ -97,7 +102,7 @@ class DailyResult: select p.id from pool p left join last_seen ls on ls.topic_id = p.topic_id - order by ls.seen_on asc nulls first, random() + order by ls.seen_on asc nulls first, p.unmet asc, p.level asc, random() limit 1 """) @@ -164,7 +169,11 @@ def _row_to_result(row, outside: bool) -> DailyResult: async def _select_new( - session: AsyncSession, user_id: uuid.UUID, *, today: date | None = None + session: AsyncSession, + user_id: uuid.UUID, + *, + today: date | None = None, + prefetch_topics: set[uuid.UUID], ) -> DailyResult: """`today` is derived from the user's timezone in production. @@ -195,7 +204,7 @@ async def _select_new( ).scalar_one_or_none() if stale_topic is not None: await signal_reader(session, user_id, stale_topic, commit=False) - request_prefetch(stale_topic) + prefetch_topics.add(stale_topic) outside = True concept_id = ( @@ -239,7 +248,7 @@ async def _select_new( and watermark.unread <= get_settings().content_low_watermark ): await signal_reader(session, user_id, watermark.topic_id, commit=False) - request_prefetch(watermark.topic_id) + prefetch_topics.add(watermark.topic_id) row = (await session.execute(_EXISTING, {"uid": user_id, "today": today})).one() return _row_to_result(row, outside=outside) @@ -265,8 +274,15 @@ async def get_or_create_daily( if allow_review else DailyResult(status="exhausted", assigned_for=today) ) - result = await _select_new(session, user_id, today=today) + prefetch_topics: set[uuid.UUID] = set() + result = await _select_new( + session, user_id, today=today, prefetch_topics=prefetch_topics + ) if result.status == "exhausted" and allow_review: result = await choose_review(session, user_id, today) or result await session.commit() + # Publish the durable target before waking another session. Starting the + # task earlier can see the old bootstrap floor and immediately stop. + for topic_id in prefetch_topics: + request_prefetch(topic_id) return result diff --git a/backend/tests/test_selection.py b/backend/tests/test_selection.py index c8c353e..30d4ae6 100644 --- a/backend/tests/test_selection.py +++ b/backend/tests/test_selection.py @@ -148,3 +148,20 @@ async def test_completion_is_reflected(session, user): again = await get_or_create_daily(session, user, today=DAY) assert again.concept.id == result.concept.id assert again.completed_at is not None + + +async def test_curriculum_prefers_foundations_before_advanced_applications(session,user): + import uuid + slug='ordered-'+uuid.uuid4().hex + tid=await session.scalar(text("insert into public.topics(slug,name) values (:s,'Ordered fixture') returning id"),{'s':slug}) + await session.execute(text("delete from public.user_topics where user_id=:u"),{'u':user}) + await session.execute(text("insert into public.user_topics(user_id,topic_id) values (:u,:t)"),{'u':user,'t':tid}) + for level in [3,2,1]: + await session.execute(text("insert into public.concepts(topic_id,slug,title,summary,difficulty) values (:t,:s,'Fixture','Body',:level)"),{'t':tid,'s':f'{slug}-{level}','level':level}) + await session.commit() + try: + first=await get_or_create_daily(session,user,today=DAY) + assert first.concept.slug==f'{slug}-1' + finally: + await session.execute(text('update public.topics set is_active=false where id=:t'),{'t':tid}) + await session.commit() From bd166edaff40d2953fb351568bc4c794bbb29f6e Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 13:59:44 +0500 Subject: [PATCH 16/36] fix: fence progress cache writes during account cleanup --- .../src/services/remoteProgressRepository.ts | 36 ++++++++++--------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/mobile/src/services/remoteProgressRepository.ts b/mobile/src/services/remoteProgressRepository.ts index 51727e1..8ecd30c 100644 --- a/mobile/src/services/remoteProgressRepository.ts +++ b/mobile/src/services/remoteProgressRepository.ts @@ -7,10 +7,11 @@ import { toConcept } from './dailyApi'; import { withPendingProgress, withCompletedReview } from './pendingProgress'; import { clearQueue, dequeue, enqueue, keyOf, pending, QueuedMutation } from './mutationQueue'; import { ProgressRepository } from './progressRepository'; +import { OfflineCache } from './offlineCache'; import { EMPTY_PROGRESS } from './storage'; import { toCategory, toSlug } from './topics'; -const CACHE_KEY = 'one-concept/server-state/v1'; +const CACHE_PREFIX = 'one-concept/server-state/'; /** True for a network failure (no response) — the signal to queue offline. */ function isOffline(err: unknown): boolean { @@ -96,12 +97,15 @@ export class RemoteProgressRepository implements ProgressRepository { // wipe happened while its request was in flight, its late result must not // be re-persisted — that would resurrect the signed-out account's data. private epoch = 0; + private disk = new OfflineCache(AsyncStorage, CACHE_PREFIX); private async remember(state: ProgressState, epoch: number): Promise { - if (epoch !== this.epoch) return state; + if (epoch !== this.epoch) return EMPTY_PROGRESS; this.cache = state; - AsyncStorage.setItem(CACHE_KEY, JSON.stringify(state)).catch(() => {}); - return state; + // Preserve the v1 disk format, using the same tested write fence as lesson + // bodies. Sign-out waits for an in-flight write before removing account data. + await this.disk.set('v1', state, epoch).catch(() => {}); + return epoch === this.epoch ? state : EMPTY_PROGRESS; } private async fromState(payload: StatePayload, epoch: number): Promise { @@ -122,19 +126,21 @@ export class RemoteProgressRepository implements ProgressRepository { /** Drop the in-memory state; the module singleton outlives a sign-out. The * offline queue is account data too, so it goes with it. */ - forget(): Promise { + async forget(): Promise { this.epoch += 1; this.cache = EMPTY_PROGRESS; - return clearQueue(); + await Promise.all([clearQueue(), this.disk.clear()]); } async loadCached(): Promise { const epoch = this.epoch; const contentEpoch = conceptCache.epoch; - const raw = await AsyncStorage.getItem(CACHE_KEY).catch(() => null); - if (!raw || epoch !== this.epoch) return null; + const parsed = await this.disk.get('v1', epoch); + if (!parsed || epoch !== this.epoch) return null; try { - this.cache = JSON.parse(raw) as ProgressState; + const restored = withPendingProgress(parsed, parsed, await pending()); + if (epoch !== this.epoch) return null; + this.cache = restored; // Also upgrade an existing installation's cached Today while offline. if (this.cache.serverDaily?.status === 'ok' || this.cache.serverDaily?.status === 'review') { const concept = toConcept(this.cache.serverDaily.payload); @@ -152,9 +158,10 @@ export class RemoteProgressRepository implements ProgressRepository { try { return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { - const raw = await AsyncStorage.getItem(CACHE_KEY).catch(() => null); - if (raw && epoch === this.epoch) { - const cached = JSON.parse(raw) as ProgressState; + const parsed = await this.disk.get('v1', epoch); + if (parsed && epoch === this.epoch) { + const cached = withPendingProgress(parsed, parsed, await pending()); + if (epoch !== this.epoch) return EMPTY_PROGRESS; // Genuinely offline: the fetch failed and we're serving the saved copy, // so flag the daily stale — that's what drives the "Offline" banner. The // cache-first preview (loadCached) leaves it not-stale, so the banner @@ -470,8 +477,5 @@ export const remoteProgressRepository = new RemoteProgressRepository(); /** Forget everything: the disk cache AND the singleton's in-memory copy. * Called on sign-out so the next account can never see this one's data. */ export async function clearServerStateCache(): Promise { - await Promise.all([ - remoteProgressRepository.forget(), - AsyncStorage.removeItem(CACHE_KEY), - ]); + await remoteProgressRepository.forget(); } From 5a569b0af97591daaf07a16c3e0c70359bfd3222 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 13:59:59 +0500 Subject: [PATCH 17/36] feat: offer daily review practice and separate review statistics --- mobile/src/screens/StatsScreen.tsx | 4 ++ mobile/src/screens/TodayScreen.tsx | 39 ++++++++++--- mobile/tests/README.md | 15 +++++ mobile/tests/review.browser.cjs | 91 ++++++++++++++++++++++++++++++ 4 files changed, 140 insertions(+), 9 deletions(-) create mode 100644 mobile/tests/review.browser.cjs diff --git a/mobile/src/screens/StatsScreen.tsx b/mobile/src/screens/StatsScreen.tsx index 6154ebf..8ec07a2 100644 --- a/mobile/src/screens/StatsScreen.tsx +++ b/mobile/src/screens/StatsScreen.tsx @@ -106,6 +106,10 @@ export function StatsScreen() { ) : ( <> + + Reviews completed: {progress.stats?.totalReviews ?? 0} + New lessons and completed reviews count toward your learning streak. Reviews do not increase concepts learned. + {error && topics.length === 0 ? ( createStyles(colors), [colors]); + const navigation = useNavigation}>>(); + const explore = () => navigation.navigate('Profile', {screen: 'Personalization'}); const outcome = serverDaily; + const review = outcome?.status === 'review'; // An authenticated user can read a cached assignment, never an invented demo lesson. const serverConcept = - outcome && outcome.status === 'ok' ? toConcept(outcome.payload) : null; + outcome && (outcome.status === 'ok' || outcome.status === 'review') ? toConcept(outcome.payload) : null; const concept = serverConcept ?? (session ? null : localConcept); // The shown concept is done if today's date is marked (the instant @@ -43,10 +49,10 @@ export function TodayScreen() { // server concept qualifies: the local fallback (selectDailyConcept) can // recycle an already-learned concept once the bundled pool is exhausted, // and that must still show the button. - const done = learnedToday || (!!serverConcept && hasLearned(serverConcept.id)); + const done = review ? outcome.payload.learned : learnedToday || (!!serverConcept && hasLearned(serverConcept.id)); const loading = localLoading; const exhausted = outcome?.status === 'exhausted'; - const offline = outcome?.status === 'ok' && outcome.stale; + const offline = (outcome?.status === 'ok' || outcome?.status === 'review') && outcome.stale; const outsideTopics = outcome?.status === 'ok' && outcome.payload.outside_followed_topics; @@ -74,7 +80,7 @@ export function TodayScreen() { {loading ? ( <> - Today’s concept + {review ? "Today’s review" : "Today’s concept"} @@ -82,7 +88,7 @@ export function TodayScreen() { <> - Today’s concept + {review ? "Today’s review" : "Today’s concept"} {offline ? ( @@ -104,7 +110,16 @@ export function TodayScreen() { - You’ve learned every concept available. New ones are on the way. + No new lesson is available for you right now. Complete a lesson to build your review library, or explore the subjects. + + + ) : null} + + {review ? ( + + + + Review a previous lesson. Recall the idea before rereading, then explain the example in your own words. Completing this review counts toward your streak. ) : null} @@ -125,15 +140,21 @@ export function TodayScreen() { {concept && (done ? ( - Learned today — see you tomorrow! + {review ? "Review complete — your learning day counts." : "Learned today — see you tomorrow!"} ) : ( markLearned(concept ?? undefined)} + label={review ? "Complete review" : "Mark as learned"} + onPress={() => review ? completeReview(outcome.payload.review_id) : markLearned(concept ?? undefined)} disabled={!concept} /> ))} + {exhausted || review ? ( + <> + + {exhausted ? : null} + + ) : null} )} diff --git a/mobile/tests/README.md b/mobile/tests/README.md index 57b44fd..8ef0f9e 100644 --- a/mobile/tests/README.md +++ b/mobile/tests/README.md @@ -105,3 +105,18 @@ downloads, while explicit sign-out clears account caches. The mocked browser checks cover web storage and app behavior; native keyboard/autofill and device storage still require the preview checks above. A production release is needed to deliver the changes to installations still running the older `main` build. + + +## Daily review and continued learning (#195) + +`reviewProgress.test.mjs` checks review outbox persistence and pending-state +reconciliation without increasing unique learned totals. The actual exported +app can be exercised with `node tests/review.browser.cjs /path/to/web-export` +using the same Playwright environment variables as the other browser scripts. +Use dummy API/Auth configuration pointing to `http://127.0.0.1:4781` (API path +`/api`); no live user or provider is needed. The scenario covers light/dark, +review labelling, future-subject discovery, offline completion/restart, +reconnect, separate Stats totals, and enlarged text at a narrow viewport. + +Physical-device font scaling, screen readers and native storage still require +manual acceptance. Syncing remains foreground/reopen JS work on the current APK. diff --git a/mobile/tests/review.browser.cjs b/mobile/tests/review.browser.cjs new file mode 100644 index 0000000..aaa040e --- /dev/null +++ b/mobile/tests/review.browser.cjs @@ -0,0 +1,91 @@ +const {chromium,expect}=require(process.env.PLAYWRIGHT_TEST_MODULE || 'playwright/test'); +const http=require('node:http'),fs=require('node:fs'),path=require('node:path'),assert=require('node:assert/strict'); +const root=process.argv[2]; +if(!root || !fs.existsSync(path.join(root,'index.html'))) throw Error('Pass the exported web directory'); +const version=require('../app.config.js').expo.version; +const today=new Date().toISOString().slice(0,10); +const concept={id:'33333333-3333-4333-8333-333333333333',slug:'known-lesson',title:'Reviewing invariants',summary:'An invariant is a rule that stays true while a system changes. Use it to check whether each operation keeps your data consistent.',example:'A library book can have one active borrower. Returning and lending it should preserve that rule.',topic_slug:'computer-science',topic_name:'Computer Science',content_version:2,like_count:0}; +const session={access_token:'fixture',refresh_token:'fixture-refresh',token_type:'bearer',expires_in:864000,expires_at:Math.floor(Date.now()/1000)+864000,user:{id:'11111111-1111-1111-1111-111111111111',email:'fixture@example.invalid',aud:'authenticated',role:'authenticated',app_metadata:{},user_metadata:{},created_at:'2026-01-01T00:00:00Z'}}; +const server=http.createServer((req,res)=>{ + const relative=decodeURIComponent(new URL(req.url,'http://localhost').pathname); + const file=path.join(root,relative==='/'?'index.html':relative); + try {res.setHeader('Content-Type',({'.html':'text/html','.js':'application/javascript','.ttf':'font/ttf','.png':'image/png'})[path.extname(file)]||'application/octet-stream');res.end(fs.readFileSync(file));} + catch{res.statusCode=404;res.end();} +}); +(async()=>{ + await new Promise(r=>server.listen(4781,'127.0.0.1',r)); + const browser=await chromium.launch({executablePath:process.env.PLAYWRIGHT_CHROMIUM_PATH,headless:true,args:['--no-sandbox']}); + try{ + for(const theme of ['light','dark']){ + let online=true,completions=0,stateReads=0; + const errors=[]; + const state={display_name:'Reader',timezone:'UTC',today,followed_topics:['computer-science'],learned:[{concept_slug:concept.slug,learned_on:'2026-01-01',title:concept.title,topic_name:concept.topic_name}],likes:[],bookmarks:[],saved:[],stats:{current:4,longest:8,total_learned:25,total_reviews:2},assignment_slug:null,daily:null,review:{review_id:'22222222-2222-4222-8222-222222222222',assigned_for:today,assigned_at:today+'T08:00:00Z',completed_at:null,learned:false,outside_followed_topics:false,concept}}; + const context=await browser.newContext({viewport:{width:390,height:844},timezoneId:'UTC'}); + await context.addInitScript(({session,version,theme})=>{ + if(!localStorage.getItem('fixture-seeded')){ + localStorage.setItem('sb-127-auth-token',JSON.stringify(session)); + localStorage.setItem('one-concept/last-seen-version/v1',version); + localStorage.setItem('one-concept/theme/v1',theme); + localStorage.setItem('fixture-seeded','yes'); + } + },{session,version,theme}); + await context.route('**/api/**',async route=>{ + if(!online) return route.abort('internetdisconnected'); + const req=route.request(),url=new URL(req.url()),endpoint=url.pathname.replace('/api',''); + let body={}; + if(endpoint==='/v1/me/state'){ + assert.equal(url.searchParams.get('reviews'),'true');stateReads++;body=state; + } else if(endpoint.startsWith('/v1/reviews/') && endpoint.endsWith('/complete')){ + assert.equal(endpoint,`/v1/reviews/${state.review.review_id}/complete`); + completions++; + if(!state.review.learned){state.review.learned=true;state.review.completed_at=today+'T09:00:00Z';state.stats.current=5;state.stats.total_reviews=3;} + body={completed:true,assigned_for:today,stats:state.stats}; + } else if(endpoint==='/v1/topics') body=[{slug:'computer-science',name:'Computer Science',concept_count:25,following:true},{slug:'future-subject',name:'Future subject',concept_count:4,following:false}]; + else if(endpoint==='/v1/me/notifications') body={enabled:false,reminder_times:['08:00']}; + else if(endpoint.startsWith('/v1/concepts/')) body=concept; + await route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(body)}); + }); + await context.route('**/auth/v1/**',route=>route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(session)})); + const page=await context.newPage();page.on('pageerror',e=>errors.push(e.message)); + await page.goto('http://127.0.0.1:4781'); + await expect(page.getByText('Today’s review',{exact:true})).toBeVisible(); + // Use the actual theme switch; persisted key is verified by the app itself. + const switcher=page.getByRole('button',{name:theme==='dark'?'Switch to dark mode':'Switch to light mode'}); + if(await switcher.count()) await switcher.click(); + await expect(page.getByRole('button',{name:'Complete review',exact:true})).toBeVisible(); + assert.equal(completions,0); + await page.getByRole('button',{name:'Explore another subject',exact:true}).click(); + await expect(page.getByText('Future subject',{exact:true})).toBeVisible(); + await page.getByRole('tab',{name:'Today'}).click(); + online=false;await page.evaluate(()=>window.dispatchEvent(new Event('offline'))); + await page.getByRole('button',{name:'Complete review',exact:true}).click(); + await expect(page.getByText('Review complete — your learning day counts.',{exact:true})).toBeVisible(); + await expect.poll(()=>page.evaluate(()=>JSON.parse(localStorage.getItem('one-concept/server-state/v1')||'{}').stats?.totalReviews)).toBe(3); + await page.reload(); + await expect(page.getByText('Review complete — your learning day counts.',{exact:true})).toBeVisible(); + assert.equal(completions,0); + // A narrow viewport with enlarged text must retain reachable actions. + await page.setViewportSize({width:320,height:568}); + await page.evaluate(()=>document.querySelectorAll('div,span').forEach(el=>{ + if(el.childNodes.length===1 && el.firstChild?.nodeType===Node.TEXT_NODE){const style=getComputedStyle(el);const size=parseFloat(style.fontSize);const line=parseFloat(style.lineHeight);el.style.fontSize=(size*1.5)+'px';if(Number.isFinite(line))el.style.lineHeight=(line*1.5)+'px';} + })); + await page.getByRole('button',{name:'Explore another subject',exact:true}).scrollIntoViewIfNeeded(); + await expect(page.getByRole('button',{name:'Explore another subject',exact:true})).toBeInViewport(); + await page.screenshot({path:`/tmp/one-concept-195-review-${theme}.png`,fullPage:true}); + await expect(page.getByRole('button',{name:'Explore another subject',exact:true})).toBeVisible(); + online=true;await page.evaluate(()=>window.dispatchEvent(new Event('online'))); + await expect.poll(()=>completions,{timeout:20000}).toBe(1); + await expect.poll(()=>page.evaluate(()=>localStorage.getItem('one-concept/mutation-queue/v1')||'{}')).toBe('{}'); + assert.equal(state.stats.total_learned,25); + assert.equal(state.stats.total_reviews,3); + await page.reload(); + await expect(page.getByText('Review complete — your learning day counts.',{exact:true})).toBeVisible(); + assert.equal(completions,1); + await page.getByRole('tab',{name:'Stats'}).click(); + await expect(page.getByText('Reviews completed: 3',{exact:true})).toBeVisible(); + assert.deepEqual(errors,[]); + await context.close(); + console.log(`${theme}: cached review, offline completion/restart, reconnect, subject exploration, separate stats passed (${stateReads} state reads)`); + } + }finally{await browser.close();server.close();} +})().catch(error=>{console.error(error);server.close();process.exitCode=1;}); From 465a47a0245a22c1509a19759c12291b30a93ef6 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 14:00:21 +0500 Subject: [PATCH 18/36] test: exercise a year of learning and restore content backups --- backend/tests/test_content_backup.py | 61 ++++++++ backend/tests/test_sustainable_year.py | 188 +++++++++++++++++++++++++ 2 files changed, 249 insertions(+) create mode 100644 backend/tests/test_content_backup.py create mode 100644 backend/tests/test_sustainable_year.py diff --git a/backend/tests/test_content_backup.py b/backend/tests/test_content_backup.py new file mode 100644 index 0000000..53685cc --- /dev/null +++ b/backend/tests/test_content_backup.py @@ -0,0 +1,61 @@ +"""Restore the disposable database into another database before relying on backups.""" + +import json +import subprocess + +from conftest import CONTAINER + + +_CHECK = """select json_build_object( + 'concepts',(select count(*) from public.concepts), + 'revisions',(select count(*) from public.concept_revisions), + 'assignments',(select count(*) from public.daily_assignments), + 'reviews',(select count(*) from public.daily_reviews), + 'constraints',(select count(*) from pg_constraint where contype in ('p','u','f')), + 'rls',(select count(*) from pg_class where relrowsecurity))""" + + +def test_backup_restores_catalog_progress_and_schema(database, tmp_path): + def command(*args, input=None): + result = subprocess.run( + ["podman", "exec", "-i", CONTAINER, *args], input=input, capture_output=True + ) + assert result.returncode == 0, result.stderr.decode()[:500] + return result.stdout + + backup = command( + "pg_dump", "-U", "postgres", "-Fc", "--no-owner", "--no-acl", "postgres" + ) + file = tmp_path / "fixture.dump" + file.write_bytes(backup) + command("createdb", "-U", "postgres", "content_restore_fixture") + try: + command( + "pg_restore", + "-U", + "postgres", + "--no-owner", + "--no-acl", + "--exit-on-error", + "-d", + "content_restore_fixture", + input=file.read_bytes(), + ) + original = json.loads( + command("psql", "-U", "postgres", "-d", "postgres", "-Atc", _CHECK) + ) + restored = json.loads( + command( + "psql", + "-U", + "postgres", + "-d", + "content_restore_fixture", + "-Atc", + _CHECK, + ) + ) + assert original == restored + assert restored["concepts"] >= 20 and restored["rls"] >= 15 + finally: + command("dropdb", "-U", "postgres", "content_restore_fixture") diff --git a/backend/tests/test_sustainable_year.py b/backend/tests/test_sustainable_year.py new file mode 100644 index 0000000..cff736b --- /dev/null +++ b/backend/tests/test_sustainable_year.py @@ -0,0 +1,188 @@ +"""A real PostgreSQL simulation of continued learning beyond the original catalog.""" + +import uuid +from datetime import date, timedelta +from unittest.mock import AsyncMock + +from sqlalchemy import text + +from app.services import pool, selection +from app.services.curriculum import PlannedLesson, import_lessons +from app.services.generation import GeneratedConcept +from app.services.interactions import complete_today, set_followed_topics +from app.services.publication import publish_revision +from app.services.reviews import complete_review +from app.services.selection import get_or_create_daily +from app.services.streaks import compute_streaks +from app.services.supply import signal_reader, target_for + + +async def test_three_readers_learn_for_a_year_through_refills_and_outages( + session, empty_generation_budget, monkeypatch +): + prefix = "year-" + uuid.uuid4().hex + # Other regression modules populate the shared test database with thousands + # of collection rows. Scope only the simulation's catalog, retaining the + # real eligibility/rotation SQL and the existing five subject records. + monkeypatch.setattr( + selection, + "_CANDIDATE", + text( + str(selection._CANDIDATE).replace( + "where c.status = 'published'", + "where c.status = 'published' and starts_with(c.slug,:fixture_prefix)", + ) + ).bindparams(fixture_prefix=prefix), + ) + topics = ( + await session.execute( + text( + "select id,slug from public.topics where is_active order by sort_order limit 5" + ) + ) + ).all() + assert len(topics) == 5 + users = [uuid.uuid4() for _ in range(3)] + for uid in users: + await session.execute( + text("insert into auth.users(id,email) values (:u,:e)"), + {"u": uid, "e": f"{uid}@example.invalid"}, + ) + # Five topics, 25 new fixture lessons each. Existing seed lessons stay intact. + for topic in topics: + await session.execute( + text("""insert into public.concepts(topic_id,slug,title,summary) + select :t,:prefix||'-'||n,'Fixture '||n,'Fixture content' + from generate_series(1,25) n"""), + {"t": topic.id, "prefix": prefix + "-" + topic.slug}, + ) + await session.commit() + await set_followed_topics(session, users[0], [t.slug for t in topics]) + await set_followed_topics(session, users[1], [topics[0].slug]) + await set_followed_topics(session, users[2], [topics[1].slug, topics[2].slug]) + generate = AsyncMock( + return_value=GeneratedConcept( + summary="A complete, useful explanation of this simulated learning objective. " + * 3, + example="A concrete example demonstrates the simulated lesson to its reader.", + model="fixture", + ) + ) + monkeypatch.setattr(pool, "generate_concept", generate) + start = date(2024, 1, 1) + seen = {uid: set() for uid in users} + review_days = 0 + try: + for offset in range(365): + day = start + timedelta(days=offset) + # A long simulated outage, plus delayed weekly publication. New + # titles are imported in batches after the original queue was used. + if offset % 7 == 0 and not 140 <= offset <= 210: + topic = topics[(offset // 7) % 5] + slug = f"{prefix}-new-{offset}" + lesson = PlannedLesson( + slug=slug, + topic_slug=topic.slug, + title=f"Simulation objective {offset}", + curriculum={ + "objective": f"Explain the distinct simulated behaviour numbered {offset}", + "difficulty": 1, + "references": [ + { + "title": "Fixture source", + "url": "https://docs.python.org/3/", + } + ], + }, + ) + await import_lessons(session, [lesson]) + # Exercise a bounded supply claim, restricting only the fixture's + # queue identity so real seeded pending titles remain untouched. + await session.commit() + with monkeypatch.context() as patch: + patch.setattr( + pool, + "_CLAIM", + text( + str(pool._CLAIM).replace( + "where b2.status = 'pending'", + "where b2.status = 'pending' and b2.slug=:only_slug", + ) + ).bindparams(only_slug=slug), + ) + inventory = await session.scalar( + text( + "select count(*) from public.concepts where topic_id=:t and status in ('published','draft')" + ), + {"t": topic.id}, + ) + cid = await pool.generate_one( + session, + "fixture", + "fixture", + topic.id, + call_cap=60, + supply_target=inventory + 1, + ) + rid = await session.scalar( + text("select id from public.concept_revisions where concept_id=:c"), + {"c": cid}, + ) + await publish_revision( + session, + rid, + "Fixture reviewer", + "Verified simulated lesson and source for the yearly regression.", + ) + await session.commit() + for uid in users: + activity = await get_or_create_daily( + session, uid, today=day, allow_review=True + ) + assert activity.status in ("ok", "review"), ( + f"No useful activity on day {offset}" + ) + if activity.status == "ok": + assert activity.concept.id not in seen[uid] + seen[uid].add(activity.concept.id) + await complete_today(session, uid, day) + else: + review_days += 1 + await complete_review(session, uid, activity.review_id, today=day) + if offset % 30 == 0: + await signal_reader(session, uid, topics[0].id) + before = await target_for(session, topics[0].id) + await signal_reader(session, uid, topics[0].id) + assert await target_for(session, topics[0].id) == before + for uid in users: + stats = await compute_streaks(session, uid, start + timedelta(days=364)) + assert stats.current == stats.longest == 365 + assert stats.total_learned == len(seen[uid]) > 125 + assert stats.total_learned + stats.total_reviews == 365 + assert review_days > 0 + assert generate.await_count <= 60 + used = await session.scalar( + text("select sum(calls_used) from public.generation_daily_usage") + ) + assert used == generate.await_count + finally: + await session.rollback() + # The shared suite keeps its original seed inventory and users. + await session.execute( + text("delete from auth.users where id=any(:ids)"), {"ids": users} + ) + await session.execute( + text( + "delete from public.concept_revisions where concept_id in (select id from public.concepts where slug like :p)" + ), + {"p": prefix + "%"}, + ) + await session.execute( + text("delete from public.concept_backlog where slug like :p"), + {"p": prefix + "%"}, + ) + await session.execute( + text("delete from public.concepts where slug like :p"), {"p": prefix + "%"} + ) + await session.execute(text("delete from public.content_supply_targets")) + await session.commit() From 23799f954e7862b35580c4173a95527674bd9deb Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 14:01:02 +0500 Subject: [PATCH 19/36] fix: retain legacy lesson snapshots before publishing corrections --- backend/app/services/publication.py | 14 ++++++++++++++ backend/tests/test_publication.py | 20 ++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/backend/app/services/publication.py b/backend/app/services/publication.py index 2a25f62..8f91a98 100644 --- a/backend/app/services/publication.py +++ b/backend/app/services/publication.py @@ -103,6 +103,20 @@ async def publish_revision( raise ValueError( f"Exact duplicate of {other.slug}; resolve overlap before publication" ) + # Preserve the pre-correction version, including migrated seed lessons that + # predate editorial records. Do not invent an original reviewer or date. + await session.execute( + text("""insert into public.concept_revisions + (concept_id,base_version,body,status,review_note) + select c.id,c.content_version-1,jsonb_build_object('title',c.title, + 'summary',c.summary,'example',c.example,'curriculum',c.curriculum, + 'model',c.model,'prompt_version',c.prompt_version),'published', + 'Legacy version captured before correction; original review was not recorded.' + from public.concepts c where c.id=:id and c.content_version>0 + and not exists(select 1 from public.concept_revisions r where r.concept_id=c.id + and r.status='published' and r.base_version=c.content_version-1)"""), + {"id": row.concept_id}, + ) await session.execute( text("""update public.concepts set title=:title,summary=:summary, example=:example,curriculum=cast(:curriculum as jsonb),difficulty=:difficulty, diff --git a/backend/tests/test_publication.py b/backend/tests/test_publication.py index 2dfa31b..b93a9aa 100644 --- a/backend/tests/test_publication.py +++ b/backend/tests/test_publication.py @@ -185,3 +185,23 @@ async def test_retired_subject_cannot_publish_and_incomplete_metadata_cannot_byp "Review must include a learning objective and references.", ) await session.rollback() + + +async def test_legacy_correction_keeps_original_text_without_inventing_review(session, draft): + slug, cid = draft + original = (await session.execute( + text("select id,body from public.concept_revisions where concept_id=:id"), + {"id": cid}, + )).one() + await publish_revision(session, original.id, "Maintainer", "Checked original against references.") + # Simulate a migrated published lesson which predates revision records. + await session.execute(text("delete from public.concept_revisions where concept_id=:id"), {"id": cid}) + body = LessonBody.model_validate(original.body) + body.summary = "A corrected explanation with a stable concept identity. " * 3 + revision = await stage_revision(session, slug, body) + assert await publish_revision(session, revision, "Maintainer", "Checked corrected example and source.") == 2 + legacy = (await session.execute(text("select body,reviewed_by,reviewed_at,review_note from public.concept_revisions where concept_id=:id and base_version=0"), {"id": cid})).one() + assert legacy.body["summary"] == original.body["summary"].strip() + assert legacy.reviewed_by is None and legacy.reviewed_at is None + assert "original review was not recorded" in legacy.review_note + await session.commit() From b6b863ff9b5593df6e9a0a9d451872c9734aabb2 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 14:02:10 +0500 Subject: [PATCH 20/36] docs: document content architecture operations and verified handoff --- backend/README.md | 11 ++ docs/CODEBASE_MAP.md | 48 +++++- docs/CONTENT_ARCHITECTURE.md | 276 +++++++++++++++++++++++++---------- docs/CONTENT_OPERATIONS.md | 234 +++++++++++++++++++++++++++++ docs/WORK_LOG.md | 53 +++++-- 5 files changed, 530 insertions(+), 92 deletions(-) create mode 100644 docs/CONTENT_OPERATIONS.md diff --git a/backend/README.md b/backend/README.md index fd723d0..cebd74d 100644 --- a/backend/README.md +++ b/backend/README.md @@ -13,6 +13,17 @@ Status: **Phase 7 — reminders.** Reads, writes, Gemini generation from a curated backlog, and timezone-aware push reminders that stop once the day is learned. +## Sustainable content lifecycle + +See [the architecture](../docs/CONTENT_ARCHITECTURE.md) and +[the operator runbook](../docs/CONTENT_OPERATIONS.md). Content now uses durable +reader-aware supply targets, structured curriculum imports, explicit reviewed +publication, and separate daily reviews when new lessons are exhausted. +Run `python -m app.workers.content --help` for maintainer operations. +Migrations 0011–0015 must precede this backend; keep old generators disabled +during rollout. Daily review clients opt into `reviews=true` on `/v1/me/state` +and complete an identified activity through `/v1/reviews/{id}/complete`. + ## Layout ``` diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index ce4544c..288a5b1 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -1,6 +1,6 @@ # Codebase map -Source inspection: 2026-09-11. This is a navigation guide to the implementation, +Source inspection: 2026-09-13. This is a navigation guide to the implementation, not a claim that deployed services or every runtime behavior have been verified. Refresh the relevant sections when the code changes. @@ -14,6 +14,7 @@ learned history, streaks, likes, saved concepts, and push reminders. | Mobile | `mobile/index.ts` registers `mobile/App.tsx`; Expo SDK 57, React Native 0.86, React 19, TypeScript. | | Backend | `backend/app/main.py`; FastAPI, async SQLAlchemy/asyncpg, Pydantic settings, ES256 JWT verification. Docker uses Python 3.12. | | Database | `backend/migrations/`; Supabase PostgreSQL schema, RLS, seeds, and incremental migrations. | +| Content lifecycle | `docs/CONTENT_ARCHITECTURE.md`, `docs/CONTENT_OPERATIONS.md`; portable subject/curriculum imports, durable refill, reviewed publication, daily review, protected health report. | | Content engine | `backend/app/services/generation.py`, `pool.py`, `prefetch.py`; Gemini lessons from a curated backlog. | | Operations | `.github/workflows/`, `backend/railway.json`, `backend/Dockerfile`, `mobile/eas.json`, `mobile/app.config.js`. | | Documentation | Root `README.md`, `RELEASING.md`, `CONTRIBUTING.md`, `docs/ARCHITECTURE.md`, `docs/ROADMAP.md`, and the backend/mobile guides. | @@ -161,7 +162,7 @@ models live in `schemas/daily.py`, `me.py`, `notifications.py`, and `topics.py`. viewer's own like; the mobile UI adds that one locally. - `services/generation.py` builds versioned prompts, calls Gemini through httpx, validates output, and exposes rate-limit errors. `pool.py` claims backlog work - with `FOR UPDATE SKIP LOCKED`, publishes validated rows, refunds throttled + with `FOR UPDATE SKIP LOCKED`, stages generated concepts and editorial revisions, refunds throttled attempts and reclaims stale work. Claims and daily call reservations commit together before contacting the provider; quota denial rolls back the claim. - `services/generation_budget.py` atomically reserves from the shared @@ -169,7 +170,7 @@ models live in `schemas/daily.py`, `me.py`, `notifications.py`, and `topics.py`. API prefetch, scheduled refill, and manual rewrite calls share this budget. Failed/uncertain calls retain their reservation; restarts do not reset it. - `services/prefetch.py` schedules bounded background top-ups, with a low unread - watermark, a published-count target, and per-process in-flight topic tracking. + watermark, durable per-topic targets, and per-process in-flight topic tracking. Exhausted shared budget is a normal stop condition. - `services/reminders.py` claims due user/day/time slots before sending Expo push batches, handles timezone and midnight windows, suppresses completed days, and @@ -179,12 +180,41 @@ models live in `schemas/daily.py`, `me.py`, `notifications.py`, and `topics.py`. lessons through Gemini with the shared budget and generation kill switch; do not run it merely to inspect the project. +## Sustainable learning additions (#195) + +- `services/curriculum.py` validates subject/plan imports, duplicate candidates + and prerequisite graphs. `backend/content/subjects.json` retains the five + subjects; `curriculum.example.json` shows future data-only expansion. +- `services/supply.py` persists assigned-count-plus-reserve demand and plans for + active readers. `pool.py` counts drafts/in-flight claims in capacity and calls + the shared quota/concurrency checks before committing any provider request. + `prefetch.py` starts only after the demand transaction commits. +- `services/publication.py` stages corrections, validates explicit approval and + increments versions while preserving identities and prior bodies. + `workers/rewrite_catalog.py` now drafts revisions under durable claims. +- `services/reviews.py` and `api/v1/reviews.py` select/complete separate review + records. `selection.py` locks profiles across daily choice; `streaks.py`, + `state.py` and reminders count completed review days without increasing unique + learned totals. `me/state?reviews=true` opts into a separate review payload. +- `workers/content.py` exposes maintainer-only imports, revision inspection, + approval, failed-plan correction/retry and health reports. `content_health.py` + computes supply/queue/quota/worker conditions and deduplicates transitions. + This uses backend credentials, with no public administration API. +- Mobile review payloads and versioned concept bodies use the existing caches; + review IDs key durable outbox intents. `pendingProgress.ts` merges pending + completion without new learned rows. Today labels review/exploration and + Stats separates review totals. Dynamic subject labels use existing generic UI. +- Added regressions cover imports, publication races, two-device review, grace, + refill/call bounds, a 365-day three-reader simulation, protected health changes, + backup restoration, and browser offline/reconnect in both themes. + ## Schema and migrations `db/models.py` mirrors the SQL schema; migrations are the schema authority. -The eleven tables cover profiles, topics, concepts, user topics, daily assignments, +The seventeen tables cover profiles, topics, concepts, user topics, daily assignments, concept interactions, notification preferences, device tokens, the concept -backlog, reminder logs, and shared daily generation usage. Unique constraints +backlog, reminder logs, shared daily generation usage, supply targets, editorial +revisions/retry logs, daily reviews, worker runs and health conditions. Unique constraints enforce one daily assignment and no concept repeats per user. RLS adds isolation behind backend identity checks. | Migration | Purpose | @@ -197,8 +227,14 @@ enforce one daily assignment and no concept repeats per user. RLS adds isolation | `0008_backlog_claimed_at.sql` | Timestamp for reclaiming abandoned generation. | | `0009_like_count_index.sql` | Index for public like counts. | | `0010_generation_daily_usage.sql` | Backend-only daily Gemini call reservations shared by all generation paths. | +| `0011_content_supply.sql` | Durable, coalesced demand beyond bootstrap inventory. | +| `0012_curriculum_publication.sql` | Structured curriculum, content versions, review drafts and audited retry grants. | +| `0013_daily_reviews.sql` | Separate review activities; preserves new-assignment uniqueness. | +| `0014_content_operations.sql` | Worker heartbeat and deduplicated condition state. | +| `0015_revision_generation_claims.sql` | Durable claims for correction drafting. | -All ten filenames are recorded in `migrations/applied.txt`; migration 0010 was +Only migrations 0001–0010 are recorded in `migrations/applied.txt`; new migrations +0011–0015 are unapplied in production at this PR handoff. Migration 0010 was applied and independently verified in production during the 1.8.0 release follow-up. The ledger is repository evidence, not a live check of production. Application connections use the transaction pooler; migration DDL uses `DIRECT_URL` diff --git a/docs/CONTENT_ARCHITECTURE.md b/docs/CONTENT_ARCHITECTURE.md index 471269a..8c70c8b 100644 --- a/docs/CONTENT_ARCHITECTURE.md +++ b/docs/CONTENT_ARCHITECTURE.md @@ -1,84 +1,208 @@ # Sustainable learning architecture -Issue #195 is delivered as one PR with focused commits. The current five subjects -remain the starting catalog. Subjects are database records, not application modes; -future additions and retirement use the same registry and content workflow. +One shared library supports a continuing daily learning habit. The existing five +subjects remain the initial registry. Future subjects use the same data contracts +and operations, without adding subject-specific routes or screens. #195 is one +implementation PR targeting `develop`; it does not deploy a production release. -## Boundaries and invariants +## Components and ownership ```mermaid flowchart LR - Registry[Subject registry] --> Curriculum[Curated curriculum import] - Curriculum --> Queue[Durable topic supply targets] - Queue --> Drafts[Budgeted background generation] - Drafts --> Review[Operator review and versioned publication] - Review --> Library[Shared published library] - Library --> Daily[Daily new lesson] - Library --> Practice[Daily review] - Daily --> Progress[Account progress and learning streak] - Practice --> Progress - Queue --> Operations[Protected operational report] - Review --> Operations + Operator[Authorized content maintainer] --> Registry[Subject registry] + Operator --> Curriculum[Validated curriculum imports] + Registry --> Curriculum + Curriculum --> Backlog[Shared planned-title backlog] + Reader[Daily request] --> Selection[Stored-content selection] + Selection --> Demand[Durable subject demand] + Planner[Scheduled reader planning] --> Demand + Demand --> Worker[Bounded background drafting] + Backlog --> Worker + Budget[Shared quota and concurrency] --> Worker + Worker --> Draft[Versioned editorial drafts] + Operator --> Approval[Explicit source and correctness review] + Draft --> Approval + Approval --> Library[Shared published concepts] + Library --> Selection + Selection --> New[Daily new assignment] + Selection --> Review[Daily review of a completed lesson] + New --> Activity[Learning days and streaks] + Review --> Activity + New --> Unique[Unique concepts learned] + Review --> ReviewTotal[Separate review total] + Demand --> Health[Protected operational report] + Backlog --> Health + Draft --> Health + Budget --> Health ``` -- Topic UUIDs and slugs remain stable. Retiring a topic hides it from discovery, - new selection and generation without deleting concepts, follows or history. -- New assignments preserve one concept per user/day and no repeated new concept. - Review has its own daily record; both completion types count learning days, - while unique learned totals remain based on completed new assignments only. -- A profile row lock serializes daily selection and review selection. A review - already chosen for today wins over content that arrives later that day. -- Legacy clients retain the existing daily payload. Review-aware clients opt in - to a separate review payload; old clients never mistake a review for new learning. -- Publication is explicit operator work. Generation prepares drafts, not truth. - Corrections use an optimistic base version and preserve the concept identity. -- New subjects need registry/curriculum data, not new routes or mobile screens. - Topic retirement is reversible; physical deletion is not a routine operation. - -## Supply and curriculum - -The daily path only signals low availability. Durable topic targets are computed -from published content plus the reader's deficit, and merged with GREATEST. With -unchanged assignments, publication increases published and unseen counts equally, -so repeated requests do not increase the target. Scheduled planning accounts for -active readers from the last 90 days and bootstraps new subjects. Default target -reserve is 60 lessons; the low watermark is 5. These are configurable operating -choices, not a guarantee that approved material or provider capacity exists. - -Generation claims are serialized per subject and count drafts toward work in -progress so an approval backlog cannot cause unlimited drafts. Every provider -call still reserves from the shared Pacific-day ledger. Daily reading never waits -for a model; stale work, empty curricula and exhausted budgets have bounded exits. - -Curriculum imports are validated and idempotent. Stable slugs, objectives, -difficulty, prerequisite slugs and references keep expansion deliberate. Similar -titles are surfaced for operator review. Import, correction, review, publication -and retirement are maintainer CLI actions; they are not public mobile API powers. - -## Review and offline behaviour - -An exhausted review-aware reader receives a previously completed lesson chosen -by least recent review. Explicit completion may keep the learning streak alive, -but never increases the unique learned count. Server dates and the existing -one-day grace determine accepted completion. A completion identifies its review -record, so a stale offline request cannot complete a different day's activity. - -The review payload travels in the existing account cache. Pending completion uses -the durable outbox and existing reconnect scheduler. New account-scoped state is -cleared on sign-out and guarded against late responses. The UI distinguishes -Review from New, and offers subject discovery or a retry when no review exists. - -## Delivery plan - -1. Reproduce the refill stop; introduce durable supply planning and concurrency tests. -2. Add generic subject lifecycle and structured curriculum import. -3. Add versioned drafts, explicit review/publication and safe corrections. -4. Add review selection/completion and activity-based streaks with compatibility tests. -5. Add account-safe offline review and Today UI. -6. Add protected operational reports, condition transitions and maintenance runbook. -7. Validate long-running supply, retirement/addition, outages and full regressions; - document the complete architecture and open one PR into develop. - -New migrations must be applied before backend deployment. Their filenames stay -out of applied.txt until production application is verified. This PR does not -change production settings, generate live content or deploy a release. +The FastAPI backend owns application writes and verifies JWT identity. The mobile +app reads the API and stores account-scoped caches/intents. Model credentials and +operator access remain on the backend. Curriculum import, approval and operational +reports use a maintainer CLI rather than exposing administrative routes to users. + +| Layer | Main implementation | Durable identity | +| --- | --- | --- | +| Registry and curriculum | `curriculum.py`, `content/subjects.json` | Topic UUID + immutable slug; planned concept slug | +| Demand and generation | `supply.py`, `pool.py`, `prefetch.py`, `generation_budget.py` | One demand target per topic; backlog UUID | +| Editorial changes | `publication.py`, `concept_revisions` | Concept UUID/slug + monotonically increasing content version | +| Daily activity | `selection.py`, `reviews.py`, `interactions.py` | New assignment or review UUID + server local date | +| Mobile persistence | `remoteProgressRepository.ts`, `mutationOutbox.ts` | Account epoch; review UUID for replay | +| Operations | `content_health.py`, `workers/content.py`, `workers/pool_topup.py` | Condition key and worker name | + +Paths in the table refer to `backend/app/services/` unless qualified; the mobile +files live in `mobile/src/services/`. See [the codebase map](CODEBASE_MAP.md) for +routes and [the operations runbook](CONTENT_OPERATIONS.md) for commands. + +## Continuous supply without per-user generation + +For subject `t` and reader `u`: + +- `P(t)` = published concepts in the active subject. +- `A(u,t)` = those published concepts already assigned to that reader, including skipped assignments. +- `U(u,t) = P(t) − A(u,t)` = unseen supply, matching daily eligibility. +- When `U` reaches the low watermark, requested shared inventory is `A + reserve`. + +Requests coalesce into one row using the maximum target. With no additional +assignments, publishing one lesson increases `P` and `U` together; the target +stays fixed. Repeated opens, multiple devices and multiple users do not create +independent catalogs or ever-increasing targets. The target advances with actual +consumption. Demand commits **before** a background task is woken; otherwise a +second session can see the old bootstrap floor and wrongly stop at 25 lessons. + +Scheduled planning aggregates the most experienced active reader in each followed +subject. Active means an assignment or review was created within the configured +90-day window. Demand expires without continued activity; the 25-lesson bootstrap +floor remains available for subjects without active demand. Existing greater +unexpired targets are retained so a concurrent planner cannot overwrite a reader +signal. Inactive old targets are not grounds for deleting approved content. + +Default reserve is 60, urgent watermark 5, and a normal pass attempts at most five +items per subject. These settings are operating choices. The planner cannot +manufacture an approved curriculum or a human review schedule. Empty curricula, +low reserves and approval backlogs must remain visible instead of being hidden +behind a larger fixed global pool. + +## Generation and publication transactions + +A normal generator locks the topic for a short capacity check, counts published +concepts, drafts and in-flight work, and claims one eligible backlog item with +`FOR UPDATE SKIP LOCKED`. A model call is reserved from the existing Pacific-day +usage ledger in that transaction. A shared transaction advisory lock serializes +the provider-slot check across the midnight budget-row change as well. The +transaction commits before provider I/O; no database connection remains pinned +across model latency or backoff. + +The global provider-slot default is three. Capacity denial rolls back an +**unstarted** claim and reservation. Failed, throttled or uncertain calls that +actually started retain their quota reservation. Throttling refunds the title's +attempt, not quota. Normal failures stop after three attempts; an operator can +grant one audited additional attempt after correcting the cause. Generation +switches apply to every worker entry point. Stale claims have a 30-minute recovery +window, comfortably beyond the provider request timeout. + +Successful generation writes a **draft concept and draft revision atomically**. +It does not publish to readers. Drafts count toward capacity, so an approval +backlog cannot cause unlimited generation. The shared catalog stores a lesson +once, independently of the number of readers. + +The maintainer validates an objective, difficulty, prerequisites and references, +then records a substantive review note. Publication checks the current base +version and updates the existing concept atomically. Repeating the same approved +revision returns its version without another publication. Competing corrections +cannot silently overwrite each other. Existing text remains visible during +review, and prior versions are retained. Legacy snapshots explicitly record +that their original review information was unavailable. + +Bulk correction drafting also uses durable revision claims, daily quota and +provider slots. A second worker skips an in-progress/pending correction. A crash +leaves a recoverable claim, not an unreviewed replacement of live text. + +## Portable subjects and deliberate curriculum growth + +Topic UUIDs and slugs are stable identities; display names and ordering are data. +Imports upsert only the supplied subjects. Retirement sets `is_active=false`: +new discovery, assignment and generation stop, while saved/history links and +already chosen activities survive. Retirement is reversible; routine operations +never physically delete a subject or reassign its identity. + +`curriculum.example.json` demonstrates adding plans for the existing five +subjects through data. Difficulty defines foundations (1), intermediate ideas +(2) and advanced applications (3). Objectives and prerequisite references are +validated; missing references and cycles are rejected. Exact title/objective +duplicates are rejected and title-similarity warnings surface likely overlaps. +This heuristic supplements editorial review; it cannot establish semantic novelty. + +Prerequisites must be published before dependent content is approved. Within topic +rotation, selection prefers satisfied prerequisites and lower difficulty. They +are preferences rather than eligibility locks: skipped lessons still leave the +new-assignment pool under the existing no-repeat rule. Unseen supply therefore +uses the same eligibility rule as selection. + +Only pending/failed plans can be explicitly revised; attempts and status are +preserved. Done or generating plans are not silently rewritten. Published +corrections use the versioned editorial workflow. No applied seed migration is +modified when the operator imports the next batch. + +## Daily review and compatibility + +The existing unique constraints still enforce one new concept per user/day and +no repeated new concept per user. Review has a separate durable table and +completion record; it never fabricates another new assignment. + +Every selection locks the user's profile while deciding between activity types: + +1. Return a review already chosen today, including its completion state. +2. Otherwise return an existing new assignment or select an unseen published lesson. +3. If the catalog is exhausted and the client opted in, choose a previously + completed published lesson, preferring the least recently reviewed. +4. If no completed lesson exists, return honest exhaustion with exploration/retry UI. + +Fresh publication during a review cannot replace the selected activity. A legacy +client requesting the same day sees its compatible exhausted response, not a +review disguised as a new lesson. Updated clients request +`GET /v1/me/state?compact=true&reviews=true`, which adds a separate `review` +payload while preserving `daily`. `POST /v1/reviews/{id}/complete` accepts no +client-selected user/date. Ownership comes from the verified JWT. + +Completion normally counts for the review's server-local assigned day. +Yesterday is accepted only while no newer activity has been assigned. Older +uncompleted activity cannot repair a streak. A completed review can be safely +acknowledged again without changing its timestamp or day. Profile locking +serializes selection and both completion types across devices. + +Streaks use the union of completed new-assignment and review dates. A day counts +once; opening a lesson counts nothing. `total_learned` continues to count unique +completed new assignments, and `total_reviews` counts completed review records. +Both state and standalone stats use these semantics. Reminders are suppressed +when either kind of activity completed the relevant day. + +## Offline and correction behaviour + +A review payload contains its full stored lesson and stable version. It uses the +existing account cache and concept cache. The outbox stores review completion +by review UUID, separately from new completion intents. It persists before +network I/O, coalesces repeated taps, retries offline/server failures and accepts +idempotent acknowledgements. A stale review intent cannot complete a different +activity. Optimistic review completion updates activity metrics without adding +a learned-history row; the server reconciles timezone/grace decisions. + +The existing account epoch and serialized storage cleanup fence late callbacks +and clear review data during sign-out. There is no new native worker: syncing +runs while the existing APK is open or reopened. Offline content stays on its +cached version until fetched online; detail reads refresh corrected text using +the same slug, preserving saved/history references. + +## Operational limits and rollout + +The protected report exposes supply, drafts, queue failures, stale claims, +reservation usage and worker health without user identifiers or raw provider +errors. Shared condition state deduplicates transitions and records recovery; +the CLI's full snapshot remains the source for investigation. Hosting job logs +are the initial delivery surface, not a new email/notification integration. + +Apply migrations 0011–0015 before backend deployment, then ship the JS update. +Their filenames remain out of the production ledger until actual application is +verified. Keep old generators stopped during rollout. Restore/pause procedures, +editorial responsibilities, publication cadence and remaining live-environment +checks are in [CONTENT_OPERATIONS.md](CONTENT_OPERATIONS.md). diff --git a/docs/CONTENT_OPERATIONS.md b/docs/CONTENT_OPERATIONS.md new file mode 100644 index 0000000..8e1ec1d --- /dev/null +++ b/docs/CONTENT_OPERATIONS.md @@ -0,0 +1,234 @@ +# Content operations runbook + +This runbook implements [the sustainable learning architecture](CONTENT_ARCHITECTURE.md) +for #195. The repository owner is the initial content operator and reviewer. +Broader crash reporting and application observability remain in #161. + +## Access and rollout + +All commands below run from `backend/` with its virtual environment and a +maintainer's backend database configuration. The content CLI has no public HTTP +route. New operational tables have RLS enabled and no mobile policies. Never +put backend credentials in an import file, screenshot, PR or mobile configuration. + +Before deploying this backend, back up the database and apply migrations +**0011–0015 in order** through the existing direct/session migration connection. +Do not add their names to `migrations/applied.txt` until production application +has actually been verified. These migrations add supply targets, editorial +history, reviews, health state and correction claims; they preserve existing +lesson IDs, assignment constraints and completed dates. + +Pause API demand generation and scheduled generation during the transition. +Deploy the backend before the JS update. Old APKs retain the existing daily +contract; updated clients opt into `reviews=true` on the compact state endpoint. +No native module, app version or runtime change is included in this feature PR. +Deploy this backend consistently to the API and workers: an old worker would +bypass the new draft gate. Restart paused jobs only after all old generators are +stopped, all five migrations are verified, and the shared daily quota rollout +in [the backend guide](../backend/README.md#shared-generation-budget) is satisfied. + +## Operating policy and schedule + +| Setting | Default | Meaning | +| --- | ---: | --- | +| `MIN_POOL_PER_TOPIC` | 25 | Bootstrap inventory floor; never the continuing-learning ceiling. | +| `CONTENT_RESERVE_PER_TOPIC` | 60 | Desired unseen lessons for the most experienced active reader. | +| `CONTENT_LOW_WATERMARK` | 5 | Urgent supply warning and demand-trigger threshold. | +| `CONTENT_ACTIVE_DAYS` | 90 | Reader had a new assignment or review assigned within this many days. | +| `CONTENT_PLANNED_RESERVE` | 90 | Warning threshold for available pending titles per subject. | +| `CONTENT_GENERATION_BATCH` | 5 | Maximum attempts per subject in a normal scheduled/prefetch pass, excluding bounded throttling retries. | +| `GENERATION_MAX_CONCURRENT` | 3 | Shared simultaneous provider calls, including correction drafting. | +| `GENERATION_DAILY_CALL_CAP` | 200 | Existing default Pacific-day reservation cap; set a lower affordable cap before enabling. | +| `GENERATION_ENABLED` | false | Kill switch for scheduled, demand and correction workers. | +| `GENERATION_ON_DEMAND` | true | Permit bounded background wakes from daily demand when the kill switch allows generation. | + +The reserve estimate assumes **one new lesson per subject per day**, a +conservative bound for a reader concentrating on one subject. Following five +subjects does not create five daily assignments. Start with a reviewed publication +plan the operator can sustain: for example, review enough material weekly to +publish around one new lesson per active subject per day. This is an operating +target, not an automatic guarantee. A five-call daily cap can limit initial +spend; validation failures consume calls and may reduce approved output. The +configured cap is not proof of the provider's quota or pricing. + +Configure the existing hosting scheduler to run: + +- `python -m app.workers.pool_topup` at least daily, before the editorial review session. +- `python -m app.workers.content report --observe` hourly, retaining job output. +- A human curriculum/publication review weekly, with a brief daily check of urgent conditions. + +Verify scheduler history and the report's worker finish time after configuration. +The PR does not configure a live scheduler. A run overdue by 36 hours is reported. +`report --observe` writes shared condition state and emits only transitions, +including recovery; unchanged runs print nothing. It does not send Slack/email +messages. A job-output collection failure can miss a transition, so use the full +`report` snapshot during incident investigation. Do not run overlapping manual +bulk rewrites deliberately; durable claims also protect accidental overlap. + +## Inspect supply and failures + +```bash +.venv/bin/python -m app.workers.content report +.venv/bin/python -m app.workers.content report --observe +.venv/bin/python -m app.workers.content failures +.venv/bin/python -m app.workers.content drafts +.venv/bin/python -m app.workers.content show REVISION_UUID +``` + +The report separates published, draft, pending, failed, generating and stale +work; it includes experienced-reader unseen availability, estimated reserve, +last generation/publication, daily reservations, and worker finish state. +Drafts occupy inventory capacity until reviewed; an approval backlog must not +cause unlimited new drafts. Low planned reserve means the curriculum needs work, +not that the model needs more automatic retries. + +A read-only production classification on 2026-09-13 found the existing **26 failed +entries**: **18 throttling-related, four content-validation failures and four +unclassified**. All had three attempts. No failures were retried or reset in this +PR. Categories are approximate diagnostics; inspect the specific title/cause +privately before granting a retry. Historical throttling entries may predate the +current attempt-refund behaviour. + +`failures` lists up to 100 title slugs with category and attempt/grant counts. +Continue with `failures --after LAST_SLUG`. It never prints raw provider errors. +For unclassified failures, inspect the stored diagnostic privately in the database +console and redact it before sharing. + +For each affected title, correct the source, scope, wording or provider setting +first. `revise-plan` can update only pending/failed plans, preserving the slug, +subject identity, status and lifetime attempts. Use a JSON array in the same +format as the import example. Then grant one audited extra attempt: + +```bash +.venv/bin/python -m app.workers.content revise-plan corrected-plans.json +.venv/bin/python -m app.workers.content retry TITLE_SLUG \ + --operator 'Muawiya Amir' --reason 'Describe the diagnosed cause and correction' +``` + +A retry uses the normal shared budget and concurrency limits. A slug collision +requires correcting the existing concept/draft; it cannot be solved by resetting +attempts. Never blanket-reset all failed rows. Stale backlog and correction claims +are recovered after 30 minutes by their workers. If the provider is still slow, +first pause generation and inspect scheduler health instead of deleting live claims. + +## Add or retire a subject + +`content/subjects.json` describes the current five subjects. Copy the relevant +record into a small import file. To add another subject later, supply a new +stable lowercase slug, display name, description and sort order. New topics do +not require a mobile release or a new API route. + +```bash +.venv/bin/python -m app.workers.content import-subjects subject-changes.json +``` + +Imports upsert the listed slugs only. **Omitting a subject does not delete it.** +Set `is_active: false` explicitly to retire it; set it back to true to restore it. +Do not rename a slug or delete the database record as a retirement mechanism. +Retired subjects leave discovery/new assignment/generation. Existing daily cards, +completed lessons, likes, saved links and reviews remain accessible. A subject +rename changes its display label; the underlying identity remains unchanged. + +New users initially follow all active subjects. Existing users choose newly added +subjects through Personalization; adding a subject does not silently rewrite +an existing user's preferences. After a change, check `/v1/topics`, follow it on +a test account, and verify next-day selection. Today's activity stays fixed. + +## Extend, draft and approve a curriculum + +1. Prepare a JSON array using + [the five-subject example](../backend/content/curriculum.example.json). + It demonstrates extension, not a production seed or a complete reserve. +2. Give every concept a stable unique slug, one learning objective, difficulty + **1 foundations / 2 intermediate / 3 advanced applications**, prerequisite + slugs where useful, and relevant source references. Separate distinct ideas; + a renamed duplicate is not library growth. +3. Import with `python -m app.workers.content import-curriculum FILE.json`. + Exact re-import is safe. Unknown subjects/prerequisites, cycles, duplicate + slugs and exact title/objective matches are rejected. Similar title warnings + require editorial inspection; this inexpensive heuristic is not semantic + proof. Check objectives and source material for conceptual duplication too. +4. Allow the background worker to draft within the shared quota. Daily HTTP + requests never wait for this work. Use `drafts` and `show` to inspect results. +5. Verify factual correctness, scope, example usefulness, reading length, + prerequisite availability and references. Import warnings are not approval. + A reference URL is not evidence that the generated text actually follows it. +6. If a draft needs changes, save its lesson body as JSON and run + `python -m app.workers.content stage SLUG BODY.json`. The body contains + `title`, `summary`, `example`, `curriculum`, optional `model` and + `prompt_version`. The CLI prints a revision UUID. +7. Publish the exact reviewed revision: + + ```bash + .venv/bin/python -m app.workers.content publish REVISION_UUID \ + --reviewed-by 'Muawiya Amir' \ + --note 'Explain which source, factual claims and example were verified' + ``` + +Publication requires complete metadata, valid prerequisites already published, +an active subject, and a current base version. It rejects exact duplicates and +stale competing revisions. Repeating an approved revision is a no-op. Approve +prerequisites before dependent lessons. Selection favours known prerequisites +and lower difficulty within topic rotation; prerequisites are a sequencing +preference rather than a hard lock that strands readers who skipped a lesson. + +Legacy published content remains readable. Legacy pending titles can still +produce drafts, but their initially empty curriculum metadata cannot pass the +new publication gate: complete it through `stage` before approval. The example +extension and existing plans must continue to grow through maintainer work; +there is no automatic source of infinite high-quality titles. + +Reject an unsuitable revision with `reject REVISION_UUID --reviewed-by NAME +--note REASON`. Rejection preserves the audit trail. A new draft concept remains +in inventory so it can be corrected with `stage`; do not repeatedly generate +new copies to evade review. Resolve rejected inventory during the weekly review. + +## Correct a published lesson + +Use `stage` with the same slug, then review and publish its revision. Existing +published text stays visible while the correction is pending. Publication +increments `content_version`, preserves the concept UUID/slug and records the +reviewer/note. The previous published body is retained, including an explicitly +labelled legacy snapshot when the original version predates review history. + +Two simultaneous corrections cannot overwrite each other silently. After a +stale-version rejection, read the latest body and prepare a fresh revision. +Saved/history references retain their meaning. Offline readers retain their +cached version until the lesson is fetched online; opening details refreshes +it. There is no promise of instantaneous correction on an offline device. + +`python -m app.workers.rewrite_catalog` now creates correction drafts instead of +replacing published text. Its durable claims prevent overlapping workers from +drafting the same current revision, and every attempted call spends from the +same budget as new lessons. It obeys the kill switch and requires review afterward. + +## Backup, recovery and safe pause + +The automated restore rehearsal dumps the disposable PostgreSQL 16 database, +restores into a second database, and compares catalog/progress counts, +constraints and RLS. Production backup/restore must still be rehearsed against +an isolated environment with the actual Supabase Auth setup and access controls. +Never restore over production as an exploratory check. + +For a real backup, use the approved direct connection through `PGSERVICE` or a +permission-restricted `.pgpass`, never a password pasted into command history: + +```bash +pg_dump --format=custom --no-owner --no-acl --file=one-concept.dump +# Point the approved connection configuration at an empty isolated restore DB: +pg_restore --no-owner --no-acl --exit-on-error --dbname=RESTORE_DATABASE one-concept.dump +``` + +Provision required Auth schemas/roles in the rehearsal environment according to +the database provider's procedure. Verify counts for concepts, revisions, +assignments, reviews, interactions and budget usage; verify constraints/RLS and +read/complete flows with test identities. Encrypt/restrict backup storage and +record the backup time and restore result privately. + +To pause costs, set `GENERATION_ENABLED=false` consistently on API and workers, +and stop manual drafting. Keep the API and stored library online: readers can +continue new lessons and reviews. For a bad publication, stage a correction +from the retained version rather than rewriting history or dropping tables. +An application rollback must retain the new tables and data. Rolling back to +an old generator would bypass editorial gates; keep generation disabled until +compatible workers are restored. Never delete review records to roll back a UI. diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index d5bbe4c..c7635d9 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -9,16 +9,49 @@ claims as completed work. ## Sustainable learning (#195) — 2026-09-13 -- Implement all five content-lifecycle work areas in one PR, with a portable - subject registry supporting the existing five subjects and future additions/ - retirement. Work in isolated `/tmp/one-concept-195` on - `codex/195-sustainable-learning` from `develop` (`3cc5af3`). -- Preserve the unrelated handbook checkout and its local edits. Read Expo SDK - 57 documentation before mobile changes. No production migration, generation, - app version bump or deployment is part of this implementation PR. -- Architecture and planned focused commits are in `CONTENT_ARCHITECTURE.md`. - Validate durable refill, curriculum/publication, review/streak/offline behaviour, - subject lifecycle and operational reporting before publishing the PR. +- Implemented all five lifecycle work areas in one feature PR targeting `develop`: + durable reader-based refill; portable subject/curriculum imports; reviewed, + versioned shared content; daily review with offline replay; protected operations. + The existing five subjects remain; future addition/retirement uses data imports. +- Work is isolated in `/tmp/one-concept-195`, branch + `codex/195-sustainable-learning`, based on `develop` (`3cc5af3`). The original + handbook checkout and its unrelated edits remain untouched. Expo SDK 57 docs + were read before mobile work. No app version/runtime change was made. +- Reproduced the 25-lesson refill ceiling with actual selection/prefetch against + disposable PostgreSQL and mocked drafting. The regression now passes. Review + also caught and fixed a worker wake before its durable target committed. +- Commits: `e0eb494` architecture scope; `11adafc` refill; `1a106c0` curriculum; + `72e3a9b` editorial gate; `c5d305e` review API/streaks; `ebf4c28` review outbox; + `9bc36a8` operations; `49ad80f` generation concurrency/recovery; + `64285a4` selection consistency; `bd166ed` cache cleanup; `5a569b0` review UI; + `465a47a` year simulation/restore. Further preservation/handoff commits are + identified by their subjects; preserve every meaningful commit when merging. +- Validation: full backend suite **171 passed, no skips**, using disposable + PostgreSQL 16 with live HTTP blocked. Added legacy-snapshot regression afterward: + publication suite **4 passed** (172 backend tests now collected). The yearly + simulation covers three readers, five subjects, queue extension and a prolonged + drafting outage; each reader reaches 365 learning days without inflating unique + learned totals. Backup dump/restore passed in a second disposable database. +- Mobile: Node 24 typecheck and **36 tests passed, no skips**; Android/iOS/web + exports passed. Mocked browser checks passed in both themes, narrow/enlarged + text, review offline restart/reconnect, future-subject exploration, timer replay, + transient failure, in-flight sign-out and all 365 saved bodies. Inspected review + screenshots. Physical-device font scaling, screen readers, native storage and + real two-device acceptance remain manual; backend concurrency tests cover races. +- Read-only production classification found 26 old failures: 18 throttling, + four validation, four unclassified. None was retried/reset. No production writes, + model generation, scheduler configuration or deployment occurred. Migrations + 0011–0015 remain intentionally absent from the applied ledger; apply/verify them + before backend rollout, then deliver the JS update. Pause old generation workers + during migration/deployment so they cannot bypass the editorial gate. +- Architecture and operating procedures: [CONTENT_ARCHITECTURE.md](CONTENT_ARCHITECTURE.md) + and [CONTENT_OPERATIONS.md](CONTENT_OPERATIONS.md). Human curriculum expansion + and source review are required; title-similarity checks do not prove originality. + Operational transitions appear in protected job output, with no external alerts + configured. Production backup/Auth restore remains a separate live rehearsal. +- PR publication and final local documentation checks are pending below. + +## Previous release status - [Release PR #191](https://github.com/Coding-Moves/one-concept/pull/191) is open from **develop → main** for **1.8.0**, with the six-benefit one-time card and From e5c680b415d5246d47426d774711d606929cc2d1 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 14:03:59 +0500 Subject: [PATCH 21/36] docs: record sustainable learning PR handoff --- docs/WORK_LOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index c7635d9..50cc877 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -49,7 +49,11 @@ claims as completed work. and source review are required; title-similarity checks do not prove originality. Operational transitions appear in protected job output, with no external alerts configured. Production backup/Auth restore remains a separate live rehearsal. -- PR publication and final local documentation checks are pending below. +- [PR #197](https://github.com/Coding-Moves/one-concept/pull/197) is open for review + from `codex/195-sustainable-learning` into `develop`, with the full architecture, + validation and rollout detail. Local documentation links and whitespace checks + passed. No merge or production release was performed. The final bookkeeping + commit is `docs: record sustainable learning PR handoff`. ## Previous release status From 9e99f0484669fbbf1c6539afdbeb1c436c74b811 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 15:49:03 +0500 Subject: [PATCH 22/36] fix: roll back rejected offline reviews before removing queued intent --- mobile/src/services/pendingProgress.ts | 17 ++++++++++++- .../src/services/remoteProgressRepository.ts | 20 +++++++++++---- mobile/src/types/index.ts | 2 ++ mobile/tests/review.browser.cjs | 25 ++++++++++++++++++- mobile/tests/reviewProgress.test.mjs | 14 ++++++++++- 5 files changed, 70 insertions(+), 8 deletions(-) diff --git a/mobile/src/services/pendingProgress.ts b/mobile/src/services/pendingProgress.ts index 1d688dc..7bff0a8 100644 --- a/mobile/src/services/pendingProgress.ts +++ b/mobile/src/services/pendingProgress.ts @@ -46,5 +46,20 @@ export function withCompletedReview(state: ProgressState, reviewId: string): Pro longest: Math.max(state.stats.longest, state.stats.current + (alreadyLearnedDay ? 0 : 1)), totalReviews: (state.stats.totalReviews ?? 0) + 1, } : undefined; - return { ...state, stats, serverDaily: { ...daily, payload: { ...daily.payload, learned: true } } }; + return { ...state, stats, pendingReviewStats: { reviewId, stats: state.stats }, serverDaily: { ...daily, payload: { ...daily.payload, learned: true } } }; +} + +/** Undo only this pending review; a rejection must not replace a newer activity. */ +export function withRejectedReview(state: ProgressState, reviewId: string): ProgressState { + const daily = state.serverDaily; + if (daily?.status !== 'review' || daily.payload.review_id !== reviewId || !daily.payload.learned) return state; + const snapshot = state.pendingReviewStats; + return { + ...state, + // Older pre-release caches may lack a snapshot. Invalidate their unverified + // aggregate instead of retaining an invented review/streak count. + stats: snapshot?.reviewId === reviewId ? snapshot.stats : undefined, + pendingReviewStats: undefined, + serverDaily: { ...daily, payload: { ...daily.payload, learned: false, completed_at: null } }, + }; } diff --git a/mobile/src/services/remoteProgressRepository.ts b/mobile/src/services/remoteProgressRepository.ts index 8ecd30c..e4c154c 100644 --- a/mobile/src/services/remoteProgressRepository.ts +++ b/mobile/src/services/remoteProgressRepository.ts @@ -4,7 +4,7 @@ import { Category, DailyPayload, ReviewPayload, ProgressState } from '../types'; import { todayKey } from './dates'; import { cacheSavedConcepts, conceptCache } from './conceptApi'; import { toConcept } from './dailyApi'; -import { withPendingProgress, withCompletedReview } from './pendingProgress'; +import { withPendingProgress, withCompletedReview, withRejectedReview } from './pendingProgress'; import { clearQueue, dequeue, enqueue, keyOf, pending, QueuedMutation } from './mutationQueue'; import { ProgressRepository } from './progressRepository'; import { OfflineCache } from './offlineCache'; @@ -267,7 +267,7 @@ export class RemoteProgressRepository implements ProgressRepository { }>(`/v1/reviews/${encodeURIComponent(reviewId)}/complete`, { method: 'POST' }); if (epoch !== this.epoch) return EMPTY_PROGRESS; await dequeue(`review:${reviewId}`); - return this.remember({ ...this.cache, stats: { + return this.remember({ ...this.cache, pendingReviewStats: undefined, stats: { current: done.stats.current, longest: done.stats.longest, totalLearned: done.stats.total_learned, totalReviews: done.stats.total_reviews, } }, epoch); @@ -279,7 +279,7 @@ export class RemoteProgressRepository implements ProgressRepository { await dequeue(`review:${reviewId}`); // Restore the uncompleted snapshot even if the reconciliation request // also fails; an expired review must not keep an invented completed day. - await this.remember({ ...this.cache, serverDaily: before.serverDaily, stats: before.stats }, epoch); + await this.remember({ ...this.cache, serverDaily: before.serverDaily, stats: before.stats, pendingReviewStats: undefined }, epoch); return this.load(); } } @@ -416,6 +416,7 @@ export class RemoteProgressRepository implements ProgressRepository { const entries = await pending(); if (entries.length === 0) return null; + let rejectedReview = false; const today = todayKey(); for (const m of entries) { if (epoch !== this.epoch) return null; // signed out mid-flush @@ -433,8 +434,17 @@ export class RemoteProgressRepository implements ProgressRepository { await dequeue(keyOf(m), m); // guarded: don't clobber a newer same-key intent } catch (err) { if (epoch !== this.epoch) return null; - if (isOffline(err)) return null; // still offline — keep the rest queued + if (isOffline(err)) return rejectedReview ? this.cache : null; // keep remaining intents queued if (err instanceof ApiError && (err.status >= 500 || err.status === 429)) continue; // transient — retry next time + if (m.kind === 'review') { + const restored = withRejectedReview(this.cache, m.reviewId); + // Persist the rollback BEFORE deleting the intent. A storage failure + // must leave the action retryable, and sign-out still fences the write. + await this.disk.set('v1', restored, epoch); + if (epoch !== this.epoch) return null; + this.cache = restored; + rejectedReview = true; + } await dequeue(keyOf(m), m); // 4xx: unfixable, drop so it can't block forever } } @@ -443,7 +453,7 @@ export class RemoteProgressRepository implements ProgressRepository { try { return await this.fromState(await apiRequest('/v1/me/state?compact=true&reviews=true'), epoch); } catch { - return null; + return epoch === this.epoch && rejectedReview ? this.cache : null; } } diff --git a/mobile/src/types/index.ts b/mobile/src/types/index.ts index 834609a..250ea29 100644 --- a/mobile/src/types/index.ts +++ b/mobile/src/types/index.ts @@ -114,6 +114,8 @@ export interface ProgressState { * Absent for purely local state, where the client derives them instead. */ stats?: StreakStats; + /** Pre-completion totals, retained until a queued review is acknowledged. */ + pendingReviewStats?: { reviewId: string; stats?: StreakStats }; /** * Today's concept, folded into the server state so startup needs one request * (#102). Present only for server-backed state; the signed-out demo picks the diff --git a/mobile/tests/review.browser.cjs b/mobile/tests/review.browser.cjs index aaa040e..253c40f 100644 --- a/mobile/tests/review.browser.cjs +++ b/mobile/tests/review.browser.cjs @@ -1,6 +1,7 @@ const {chromium,expect}=require(process.env.PLAYWRIGHT_TEST_MODULE || 'playwright/test'); const http=require('node:http'),fs=require('node:fs'),path=require('node:path'),assert=require('node:assert/strict'); const root=process.argv[2]; +const rejectReview=process.argv.includes('--reject-review'); if(!root || !fs.existsSync(path.join(root,'index.html'))) throw Error('Pass the exported web directory'); const version=require('../app.config.js').expo.version; const today=new Date().toISOString().slice(0,10); @@ -17,7 +18,7 @@ const server=http.createServer((req,res)=>{ const browser=await chromium.launch({executablePath:process.env.PLAYWRIGHT_CHROMIUM_PATH,headless:true,args:['--no-sandbox']}); try{ for(const theme of ['light','dark']){ - let online=true,completions=0,stateReads=0; + let online=true,completions=0,stateReads=0,rejectReplay=false; const errors=[]; const state={display_name:'Reader',timezone:'UTC',today,followed_topics:['computer-science'],learned:[{concept_slug:concept.slug,learned_on:'2026-01-01',title:concept.title,topic_name:concept.topic_name}],likes:[],bookmarks:[],saved:[],stats:{current:4,longest:8,total_learned:25,total_reviews:2},assignment_slug:null,daily:null,review:{review_id:'22222222-2222-4222-8222-222222222222',assigned_for:today,assigned_at:today+'T08:00:00Z',completed_at:null,learned:false,outside_followed_topics:false,concept}}; const context=await browser.newContext({viewport:{width:390,height:844},timezoneId:'UTC'}); @@ -33,11 +34,13 @@ const server=http.createServer((req,res)=>{ if(!online) return route.abort('internetdisconnected'); const req=route.request(),url=new URL(req.url()),endpoint=url.pathname.replace('/api',''); let body={}; + if(endpoint==='/v1/me/state' && rejectReplay) return route.fulfill({status:503,contentType:'application/json',body:'{}'}); if(endpoint==='/v1/me/state'){ assert.equal(url.searchParams.get('reviews'),'true');stateReads++;body=state; } else if(endpoint.startsWith('/v1/reviews/') && endpoint.endsWith('/complete')){ assert.equal(endpoint,`/v1/reviews/${state.review.review_id}/complete`); completions++; + if(rejectReplay) return route.fulfill({status:409,contentType:'application/json',body:JSON.stringify({detail:'Review is unavailable or its completion window has ended'})}); if(!state.review.learned){state.review.learned=true;state.review.completed_at=today+'T09:00:00Z';state.stats.current=5;state.stats.total_reviews=3;} body={completed:true,assigned_for:today,stats:state.stats}; } else if(endpoint==='/v1/topics') body=[{slug:'computer-science',name:'Computer Science',concept_count:25,following:true},{slug:'future-subject',name:'Future subject',concept_count:4,following:false}]; @@ -73,9 +76,29 @@ const server=http.createServer((req,res)=>{ await expect(page.getByRole('button',{name:'Explore another subject',exact:true})).toBeInViewport(); await page.screenshot({path:`/tmp/one-concept-195-review-${theme}.png`,fullPage:true}); await expect(page.getByRole('button',{name:'Explore another subject',exact:true})).toBeVisible(); + rejectReplay=rejectReview; online=true;await page.evaluate(()=>window.dispatchEvent(new Event('online'))); await expect.poll(()=>completions,{timeout:20000}).toBe(1); await expect.poll(()=>page.evaluate(()=>localStorage.getItem('one-concept/mutation-queue/v1')||'{}')).toBe('{}'); + if(rejectReview){ + // Server rejected the expired completion; reconciliation also fails. The + // corrected cache must be visible now and remain corrected after restart. + await expect(page.getByRole('button',{name:'Complete review',exact:true})).toBeVisible(); + await page.reload(); + await expect(page.getByRole('button',{name:'Complete review',exact:true})).toBeVisible(); + const disk=await page.evaluate(()=>JSON.parse(localStorage.getItem('one-concept/server-state/v1'))); + assert.equal(disk.serverDaily.payload.learned,false); + assert.equal(disk.stats.totalReviews,2); + assert.equal(disk.stats.current,4); + assert.equal(disk.stats.longest,8); + assert.equal(disk.stats.totalLearned,25); + assert.equal(disk.pendingReviewStats,undefined); + assert.equal(state.review.learned,false); + assert.deepEqual(errors,[]); + await context.close(); + console.log(`${theme}: rejected replay rolls back completion/totals despite failed refresh and restart`); + continue; + } assert.equal(state.stats.total_learned,25); assert.equal(state.stats.total_reviews,3); await page.reload(); diff --git a/mobile/tests/reviewProgress.test.mjs b/mobile/tests/reviewProgress.test.mjs index c7dc15c..4708e3b 100644 --- a/mobile/tests/reviewProgress.test.mjs +++ b/mobile/tests/reviewProgress.test.mjs @@ -1,6 +1,6 @@ import assert from 'node:assert/strict'; import { test } from 'node:test'; -import { withCompletedReview, withPendingProgress } from '../src/services/pendingProgress.ts'; +import { withCompletedReview, withPendingProgress, withRejectedReview } from '../src/services/pendingProgress.ts'; import { MutationOutbox } from '../src/services/mutationOutbox.ts'; const state = { @@ -37,3 +37,15 @@ test('review outbox survives restart, coalesces two taps, and clears on account await restarted.clear(); assert.deepEqual(await restarted.pending(),[]); }); + +test('rejected review restores exact totals after disk restart without touching saved work', () => { + const optimistic=withCompletedReview({...state,stats:{...state.stats,current:10,longest:10}},'review-1'); + const restarted=JSON.parse(JSON.stringify(optimistic)); + const rejected=withRejectedReview({...restarted,bookmarks:['another-lesson']},'review-1'); + assert.equal(rejected.serverDaily.payload.learned,false); + assert.deepEqual(rejected.stats,{...state.stats,current:10,longest:10}); + assert.deepEqual(rejected.bookmarks,['another-lesson']); + assert.equal(rejected.pendingReviewStats,undefined); + assert.equal(withRejectedReview(rejected,'review-1'),rejected); + assert.equal(withRejectedReview(state,'old-review'),state); +}); From b3e08de8c1c233bc4323ff620ee44beddf7a0cff Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 15:49:20 +0500 Subject: [PATCH 23/36] docs: record rejected review regression and validation --- docs/WORK_LOG.md | 17 +++++++++++++++++ mobile/tests/README.md | 4 ++++ 2 files changed, 21 insertions(+) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 8687411..dae0df0 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,23 @@ claims as completed work. ## Current status +### PR #197 review correction + +- Fixed rejected offline review replay in the same PR. Pre-completion statistics + now survive restart; a terminal replay rejection restores the matching activity + and its exact totals on disk before removing the queued intent. A failed refresh + returns the corrected cache immediately. Newer activities and unrelated saves + are preserved; sign-out keeps its existing write fence. +- The new `--reject-review` browser regression fails on the original export and + passes on the fixed export in both themes, including failed refresh and restart. + The normal successful replay browser scenario also passed in both themes. +- Validation: Node 24 typecheck, **37 tests passed with no skips**, web export, + browser scenarios and `git diff --check`. Backend/native code is unchanged; + backend tests and physical-device checks were not rerun for this JS-only fix. +- Focused implementation/test commit: `fix: roll back rejected offline reviews + before removing queued intent`. Test instructions and this handoff are a + separate documentation commit. No merge, release or production change. + ## Sustainable learning (#195) — 2026-09-13 - Implemented all five lifecycle work areas in one feature PR targeting `develop`: diff --git a/mobile/tests/README.md b/mobile/tests/README.md index 8ef0f9e..79a0d4a 100644 --- a/mobile/tests/README.md +++ b/mobile/tests/README.md @@ -118,5 +118,9 @@ Use dummy API/Auth configuration pointing to `http://127.0.0.1:4781` (API path review labelling, future-subject discovery, offline completion/restart, reconnect, separate Stats totals, and enlarged text at a narrow viewport. +Add `--reject-review` to exercise an expired offline completion (409) followed +by a failed state refresh (503). The UI and disk must restore the uncompleted +review and exact pre-tap totals before and after restart, in both themes. + Physical-device font scaling, screen readers and native storage still require manual acceptance. Syncing remains foreground/reopen JS work on the current APK. From e2eed9bf3a549105ebae029455452b3bb2319750 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:29:56 +0500 Subject: [PATCH 24/36] feat: browse and cache complete learning history in bounded pages --- mobile/src/hooks/useHistory.ts | 54 +++++++++++++ mobile/src/screens/HistoryScreen.tsx | 34 +++++--- mobile/src/services/accountCaches.ts | 2 + mobile/src/services/historyApi.ts | 30 +++++++ .../src/services/remoteProgressRepository.ts | 2 + mobile/src/types/index.ts | 1 + mobile/tests/history.browser.cjs | 78 +++++++++++++++++++ 7 files changed, 192 insertions(+), 9 deletions(-) create mode 100644 mobile/src/hooks/useHistory.ts create mode 100644 mobile/src/services/historyApi.ts create mode 100644 mobile/tests/history.browser.cjs diff --git a/mobile/src/hooks/useHistory.ts b/mobile/src/hooks/useHistory.ts new file mode 100644 index 0000000..8ef5a1a --- /dev/null +++ b/mobile/src/hooks/useHistory.ts @@ -0,0 +1,54 @@ +import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; +import { useAuth } from '../context/AuthContext'; +import { fetchHistoryPage, historyPageCache, HistoryPage } from '../services/historyApi'; +import { ProgressState } from '../types'; + +/** Load one bounded page per action; a refresh/account change invalidates late responses. */ +export function useHistory(progress: ProgressState) { + const { session } = useAuth(); + const owner = session?.user.id ?? null; + const key = JSON.stringify([owner, progress.learned, progress.historyNextCursor]); + const [loaded, setLoaded] = useState<{key: string; pages: HistoryPage[]}>({key, pages: []}); + const [loading, setLoading] = useState(false); + const [failed, setFailed] = useState(false); + const generation = useRef(0); + const busy = useRef(false); + useEffect(() => { + generation.current++; + busy.current = false; + setLoading(false); + setFailed(false); + setLoaded({key, pages: []}); + return () => { generation.current++; }; + }, [key]); + const pages = loaded.key === key ? loaded.pages : []; + const records = useMemo(() => { + const rows = new Map(pages.flatMap(page => page.items).map(row => [row.conceptId, row])); + progress.learned.forEach(row => rows.set(row.conceptId, row)); + return [...rows.values()].sort((a,b) => b.date.localeCompare(a.date)); + }, [pages, progress.learned]); + // Legacy cached states predate the cursor field but carry complete totals. + const initialCursor = progress.historyNextCursor === undefined + ? ((progress.stats?.totalLearned ?? 0) > progress.learned.length + ? [...progress.learned].sort((a,b) => a.date.localeCompare(b.date))[0]?.date : null) + : progress.historyNextCursor; + const cursor = pages.length ? pages[pages.length - 1].nextCursor : initialCursor; + const loadMore = useCallback(async () => { + if (!owner || !cursor || busy.current) return; + busy.current = true; + const request = generation.current; + const epoch = historyPageCache.epoch; + setLoading(true); + setFailed(false); + try { + const page = await fetchHistoryPage(owner, cursor, epoch); + if (request !== generation.current || epoch !== historyPageCache.epoch) return; + setLoaded(previous => ({key, pages: [...(previous.key === key ? previous.pages : []), page]})); + } catch { + if (request === generation.current) setFailed(true); + } finally { + if (request === generation.current) { busy.current = false; setLoading(false); } + } + }, [owner, cursor, key]); + return {records, loading, failed, hasMore: !!owner && !!cursor, loadMore}; +} diff --git a/mobile/src/screens/HistoryScreen.tsx b/mobile/src/screens/HistoryScreen.tsx index b0d3fee..422e92f 100644 --- a/mobile/src/screens/HistoryScreen.tsx +++ b/mobile/src/screens/HistoryScreen.tsx @@ -2,8 +2,10 @@ import { Ionicons } from '@expo/vector-icons'; import { CompositeNavigationProp, ParamListBase, useNavigation } from '@react-navigation/native'; import { BottomTabNavigationProp } from '@react-navigation/bottom-tabs'; import { NativeStackNavigationProp } from '@react-navigation/native-stack'; -import { useMemo } from 'react'; -import { FlatList, Pressable, StyleSheet, Text, View } from 'react-native'; +import { useMemo, useState } from 'react'; +import { FlatList, Pressable, TextInput, StyleSheet, Text, View } from 'react-native'; +import { useHistory } from '../hooks/useHistory'; +import { PrimaryButton } from '../components/PrimaryButton'; import { CategoryChip } from '../components/CategoryChip'; import { LikeCount } from '../components/LikeCount'; import { SkeletonRow } from '../components/Skeleton'; @@ -17,8 +19,6 @@ import { formatDateKey } from '../services/dates'; import { scaleIcon, scaleFont, radius, shadows, spacing, ThemeColors, typography } from '../theme'; import { Category, LearnedRecord } from '../types'; -// Keep the feed focused on recent activity (issue #124). -const HISTORY_LIMIT = 10; function prettify(slug: string): string { return slug.replace(/-/g, ' ').replace(/\b\w/g, (c) => c.toUpperCase()); @@ -75,10 +75,11 @@ export function HistoryScreen() { > >(); - // Newest first, capped at the last HISTORY_LIMIT to keep the feed focused. - const records = [...progress.learned] - .sort((a, b) => (a.date < b.date ? 1 : -1)) - .slice(0, HISTORY_LIMIT); + const history = useHistory(progress); + const [query, setQuery] = useState(''); + const records = history.records.filter(record => + `${record.title ?? prettify(record.conceptId)} ${record.topicName ?? ''}` + .toLowerCase().includes(query.trim().toLowerCase())); const likedIds = useMemo(() => new Set(progress.likes), [progress.likes]); const open = (conceptId: string, title: string) => @@ -101,7 +102,10 @@ export function HistoryScreen() { ListHeaderComponent={ History - Your last {HISTORY_LIMIT} concepts. + {history.records.length} of {progress.stats?.totalLearned ?? history.records.length} learned concepts loaded. + } ListEmptyComponent={ @@ -111,6 +115,8 @@ export function HistoryScreen() { + ) : query.trim() ? ( + No matches in loaded history. Load older lessons to keep looking. ) : !online && !progress.stats ? ( ) : ( @@ -123,6 +129,14 @@ export function HistoryScreen() { ) } + ListFooterComponent={history.hasMore ? ( + + {history.failed ? Older lessons couldn’t be loaded. Downloaded pages remain available offline. : null} + + + ) : null} + keyboardShouldPersistTaps="handled" ItemSeparatorComponent={() => } /> @@ -133,6 +147,8 @@ type Styles = ReturnType; const createStyles = (colors: ThemeColors) => StyleSheet.create({ + search: { color: colors.text, borderColor: colors.border, borderWidth: 1, borderRadius: radius.sm, padding: spacing.md, fontSize: scaleFont(16) }, + footer: { paddingVertical: spacing.lg, gap: spacing.sm }, screen: { flex: 1, backgroundColor: colors.background, diff --git a/mobile/src/services/accountCaches.ts b/mobile/src/services/accountCaches.ts index 48ba5c1..e887ed8 100644 --- a/mobile/src/services/accountCaches.ts +++ b/mobile/src/services/accountCaches.ts @@ -12,6 +12,7 @@ import { clearDailyCache } from './dailyApi'; import { conceptCache } from './conceptApi'; import { clearNotificationPrefsCache } from './notifications'; import { clearServerStateCache } from './remoteProgressRepository'; +import { historyPageCache } from './historyApi'; import { savedCollectionCache } from './savedApi'; import { clearTopicsCache } from './topicsApi'; @@ -22,6 +23,7 @@ export async function clearAccountCaches(): Promise { clearDailyCache(), conceptCache.clear(), savedCollectionCache.clear(), + historyPageCache.clear(), clearTopicsCache(), clearNotificationPrefsCache(), ]); diff --git a/mobile/src/services/historyApi.ts b/mobile/src/services/historyApi.ts new file mode 100644 index 0000000..d6b96a2 --- /dev/null +++ b/mobile/src/services/historyApi.ts @@ -0,0 +1,30 @@ +import AsyncStorage from '@react-native-async-storage/async-storage'; +import { apiRequest } from '../api/client'; +import { LearnedRecord } from '../types'; +import { OfflineCache } from './offlineCache'; + +export interface HistoryPage { items: LearnedRecord[]; nextCursor: string | null } +export const historyPageCache = new OfflineCache(AsyncStorage, 'one-concept/history-pages/v1/'); + +/** Pages stay separate from compact startup state and participate in sign-out. */ +export async function fetchHistoryPage(userId: string, cursor: string, epoch: number): Promise { + const key = `${userId}/${cursor}`; + try { + const page = await apiRequest<{ + items: { concept_slug: string; learned_on: string; title: string; topic_name: string; like_count: number }[]; + next_cursor: string | null; + }>(`/v1/me/history?limit=50&before=${encodeURIComponent(cursor)}`); + if (page.next_cursor && page.next_cursor >= cursor) throw new Error('History cursor did not advance'); + const result = { + items: page.items.map(row => ({ conceptId: row.concept_slug, date: row.learned_on, + title: row.title, topicName: row.topic_name, likeCount: row.like_count })), + nextCursor: page.next_cursor, + }; + await historyPageCache.set(key, result, epoch).catch(() => {}); + return result; + } catch (error) { + const cached = await historyPageCache.get(key, epoch); + if (cached) return cached; + throw error; + } +} diff --git a/mobile/src/services/remoteProgressRepository.ts b/mobile/src/services/remoteProgressRepository.ts index e4c154c..80c0cd8 100644 --- a/mobile/src/services/remoteProgressRepository.ts +++ b/mobile/src/services/remoteProgressRepository.ts @@ -35,6 +35,7 @@ interface StatePayload { saved?: { concept_slug: string; title?: string; topic_name?: string; like_count?: number }[]; learned_before_window?: Record | null; saved_next_cursor?: string | null; + history_next_cursor?: string | null; stats: { current: number; longest: number; total_learned: number; total_reviews?: number }; assignment_slug: string | null; daily?: DailyPayload | null; @@ -66,6 +67,7 @@ function toProgressState(payload: StatePayload): ProgressState { })), learnedBeforeWindow: payload.learned_before_window ?? undefined, savedNextCursor: payload.saved_next_cursor, + historyNextCursor: payload.history_next_cursor, // Server-computed, so the day boundary comes from the user's stored // timezone rather than whatever the device clock happens to say. stats: { diff --git a/mobile/src/types/index.ts b/mobile/src/types/index.ts index 250ea29..f0203fd 100644 --- a/mobile/src/types/index.ts +++ b/mobile/src/types/index.ts @@ -109,6 +109,7 @@ export interface ProgressState { learnedBeforeWindow?: Record; /** Continuation after the embedded saved window; absent on legacy responses. */ savedNextCursor?: string | null; + historyNextCursor?: string | null; /** * Streaks as computed by the server, when the state came from the server. * Absent for purely local state, where the client derives them instead. diff --git a/mobile/tests/history.browser.cjs b/mobile/tests/history.browser.cjs new file mode 100644 index 0000000..346c39d --- /dev/null +++ b/mobile/tests/history.browser.cjs @@ -0,0 +1,78 @@ +const {chromium,expect}=require(process.env.PLAYWRIGHT_TEST_MODULE || 'playwright/test'); +const http=require('node:http'),fs=require('node:fs'),path=require('node:path'),assert=require('node:assert/strict'); +const root=process.argv[2]; +const rejectReview=process.argv.includes('--reject-review'); +if(!root || !fs.existsSync(path.join(root,'index.html'))) throw Error('Pass the exported web directory'); +const version=require('../app.config.js').expo.version; +const today=new Date().toISOString().slice(0,10); +const concept={id:'33333333-3333-4333-8333-333333333333',slug:'known-lesson',title:'Reviewing invariants',summary:'An invariant is a rule that stays true while a system changes. Use it to check whether each operation keeps your data consistent.',example:'A library book can have one active borrower. Returning and lending it should preserve that rule.',topic_slug:'computer-science',topic_name:'Computer Science',content_version:2,like_count:0}; +const session={access_token:'fixture',refresh_token:'fixture-refresh',token_type:'bearer',expires_in:864000,expires_at:Math.floor(Date.now()/1000)+864000,user:{id:'11111111-1111-1111-1111-111111111111',email:'fixture@example.invalid',aud:'authenticated',role:'authenticated',app_metadata:{},user_metadata:{},created_at:'2026-01-01T00:00:00Z'}}; +const server=http.createServer((req,res)=>{ + const relative=decodeURIComponent(new URL(req.url,'http://localhost').pathname); + const file=path.join(root,relative==='/'?'index.html':relative); + try {res.setHeader('Content-Type',({'.html':'text/html','.js':'application/javascript','.ttf':'font/ttf','.png':'image/png'})[path.extname(file)]||'application/octet-stream');res.end(fs.readFileSync(file));} + catch{res.statusCode=404;res.end();} +}); +(async()=>{ + await new Promise(r=>server.listen(4781,'127.0.0.1',r)); + const browser=await chromium.launch({executablePath:process.env.PLAYWRIGHT_CHROMIUM_PATH,headless:true,args:['--no-sandbox']}); + try { + let online=true, failPage=false, holdPage=false, releasePage, pageRequests=0; + const records=Array.from({length:120},(_,i)=>({concept_slug:`lesson-${i}`,title:`Historical lesson ${i}`,topic_name:'Computer Science',like_count:0,learned_on:new Date(Date.now()-(i+1)*86400000).toISOString().slice(0,10)})); + const state={display_name:'Reader',timezone:'UTC',today,followed_topics:['computer-science'],learned:records.slice(0,50),history_next_cursor:records[49].learned_on,likes:[],bookmarks:[],saved:[],stats:{current:120,longest:120,total_learned:120,total_reviews:0},assignment_slug:concept.slug,daily:{assigned_for:today,assigned_at:today+'T08:00:00Z',learned:false,completed_at:null,outside_followed_topics:false,concept}}; + const context=await browser.newContext({viewport:{width:390,height:844}}); + await context.addInitScript(({session,version})=>{ + if(!localStorage.getItem('fixture-seeded')){ + localStorage.setItem('sb-127-auth-token',JSON.stringify(session)); + localStorage.setItem('one-concept/last-seen-version/v1',version); + localStorage.setItem('fixture-seeded','yes'); + } + },{session,version}); + await context.route('**/api/**',async route=>{ + if(!online) return route.abort('internetdisconnected'); + const url=new URL(route.request().url()), endpoint=url.pathname.replace('/api','');let body={}; + if(endpoint==='/v1/me/state') {assert.equal(url.searchParams.get('compact'),'true');body=state;} + else if(endpoint==='/v1/me/history') { + pageRequests++;assert.equal(url.searchParams.get('limit'),'50'); + if(holdPage) await new Promise(r=>{releasePage=r;}); + if(failPage) return route.fulfill({status:503,contentType:'application/json',body:'{}'}); + const remaining=records.filter(r=>r.learned_on50?items.at(-1).learned_on:null}; + } else if(endpoint==='/v1/topics') body=[{slug:'computer-science',name:'Computer Science',concept_count:125,following:true}]; + else if(endpoint==='/v1/me/notifications') body={enabled:false,reminder_times:['08:00']}; + else if(endpoint.startsWith('/v1/concepts/')) body={...concept,slug:decodeURIComponent(endpoint.split('/').pop()),title:'Historical lesson 119',summary:'Downloaded historical explanation.'}; + await route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(body)}); + }); + await context.route('**/auth/v1/**',route=>route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(session)})); + const page=await context.newPage(), errors=[];page.on('pageerror',e=>errors.push(e.message)); + await page.goto('http://127.0.0.1:4781'); + await expect(page.getByText('Today’s concept',{exact:true})).toBeVisible();assert.equal(pageRequests,0); + await page.getByRole('tab',{name:'History'}).click(); + await expect(page.getByText('50 of 120 learned concepts loaded.',{exact:true})).toBeVisible();assert.equal(pageRequests,0); + failPage=true;await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await expect(page.getByRole('button',{name:'Retry older lessons',exact:true})).toBeVisible();failPage=false; + await page.getByRole('button',{name:'Retry older lessons',exact:true}).click(); + await expect(page.getByText('100 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); + await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await expect(page.getByText('120 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); + await page.getByPlaceholder('Search loaded history').fill('Historical lesson 119'); + await page.getByRole('button',{name:'Open Historical lesson 119',exact:true}).click(); + await expect(page.getByText('Downloaded historical explanation.',{exact:true})).toBeVisible(); + online=false;await page.reload();await page.getByRole('tab',{name:'History'}).click(); + await expect(page.getByText('50 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); + await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await expect(page.getByText('100 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); + await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await expect(page.getByText('120 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); + await page.getByPlaceholder('Search loaded history').fill('Historical lesson 119'); + await page.getByRole('button',{name:'Open Historical lesson 119',exact:true}).click(); + await expect(page.getByText('Downloaded historical explanation.',{exact:true})).toBeVisible(); + console.log('PASS: bounded History pages, retry, search and downloaded history/detail after offline restart'); + online=true;holdPage=true;await page.reload();await page.getByRole('tab',{name:'History'}).click(); + await page.getByRole('button',{name:'Load older lessons',exact:true}).click();await expect.poll(()=>!!releasePage).toBe(true); + await page.getByRole('tab',{name:'Profile'}).click();await page.getByText('Sign out',{exact:true}).click(); + await expect(page.getByText('Welcome back — sign in to pick up your streak.',{exact:true})).toBeVisible();releasePage(); + await expect.poll(()=>page.evaluate(()=>Object.keys(localStorage).filter(k=>k.startsWith('one-concept/history-pages/')).length)).toBe(0); + assert.deepEqual(errors,[]);console.log('PASS: sign-out clears History pages and fences a late page response');await context.close(); + } finally {await browser.close();server.close();} +})().catch(e=>{console.error(e);server.close();process.exitCode=1;}); From afa8e963b4ee144b4aa18f4802fdf4bd38c4be39 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:34:33 +0500 Subject: [PATCH 25/36] fix: improve offline contrast and large-text account layouts --- mobile/src/components/OfflineBanner.tsx | 6 +++--- mobile/src/screens/ConceptDetailScreen.tsx | 8 ++++---- mobile/src/screens/ProfileScreen.tsx | 4 ++-- mobile/src/theme/index.ts | 6 ++++++ mobile/tests/accessibility.test.mjs | 14 ++++++++++++++ 5 files changed, 29 insertions(+), 9 deletions(-) create mode 100644 mobile/tests/accessibility.test.mjs diff --git a/mobile/src/components/OfflineBanner.tsx b/mobile/src/components/OfflineBanner.tsx index 7a60064..a5bcbe7 100644 --- a/mobile/src/components/OfflineBanner.tsx +++ b/mobile/src/components/OfflineBanner.tsx @@ -20,7 +20,7 @@ export function OfflineBanner() { accessibilityLabel="You are offline. Changes will sync when you reconnect." > - + Offline — changes will sync when you reconnect @@ -29,7 +29,7 @@ export function OfflineBanner() { const createStyles = (colors: ThemeColors) => StyleSheet.create({ - container: { backgroundColor: colors.textMuted }, + container: { backgroundColor: colors.offlineBackground }, row: { flexDirection: 'row', alignItems: 'center', @@ -38,5 +38,5 @@ const createStyles = (colors: ThemeColors) => paddingVertical: spacing.xs + 2, paddingHorizontal: spacing.md, }, - text: { fontSize: scaleFont(12), fontWeight: '600', color: colors.onPrimary }, + text: { flexShrink: 1, fontSize: scaleFont(14), fontWeight: '600', color: colors.offlineText }, }); diff --git a/mobile/src/screens/ConceptDetailScreen.tsx b/mobile/src/screens/ConceptDetailScreen.tsx index 352e461..2f85d94 100644 --- a/mobile/src/screens/ConceptDetailScreen.tsx +++ b/mobile/src/screens/ConceptDetailScreen.tsx @@ -19,7 +19,7 @@ type Status = 'loading' | 'ready' | 'error'; export function ConceptDetailScreen() { const navigation = useNavigation(); const { params } = useRoute>(); - const { conceptId, title } = params; + const { conceptId } = params; const { colors } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); @@ -63,8 +63,8 @@ export function ConceptDetailScreen() { return ( - - {title ?? 'Concept'} + + Concept navigation.goBack()} @@ -109,7 +109,7 @@ const createStyles = (colors: ThemeColors) => padding: spacing.lg, }, heading: { ...typography.title, fontSize: scaleFont(22), color: colors.text, flexShrink: 1 }, - closeButton: { padding: spacing.xs }, + closeButton: { minWidth: 44, minHeight: 44, alignItems: 'center', justifyContent: 'center', flexShrink: 0 }, content: { paddingHorizontal: spacing.lg, paddingBottom: spacing.xl, gap: spacing.md }, center: { flex: 1, alignItems: 'center', justifyContent: 'center', gap: spacing.sm, padding: spacing.xl }, }); diff --git a/mobile/src/screens/ProfileScreen.tsx b/mobile/src/screens/ProfileScreen.tsx index a399bd7..02c8d3b 100644 --- a/mobile/src/screens/ProfileScreen.tsx +++ b/mobile/src/screens/ProfileScreen.tsx @@ -75,10 +75,10 @@ export function ProfileScreen() { - + {email ? email.split('@')[0] : 'Learner'} - + {email ?? 'Signed out'} diff --git a/mobile/src/theme/index.ts b/mobile/src/theme/index.ts index 0d73605..0445e69 100644 --- a/mobile/src/theme/index.ts +++ b/mobile/src/theme/index.ts @@ -17,6 +17,8 @@ export interface ThemeColors { primary: string; primaryPressed: string; onPrimary: string; + offlineBackground: string; + offlineText: string; success: string; successSurface: string; streak: string; @@ -37,6 +39,8 @@ export const lightColors: ThemeColors = { primary: '#6366F1', primaryPressed: '#4F46E5', onPrimary: '#FFFFFF', + offlineBackground: '#3730A3', + offlineText: '#FFFFFF', success: '#16A34A', successSurface: '#E9F8EF', streak: '#F97316', @@ -56,6 +60,8 @@ export const darkColors: ThemeColors = { primary: '#818CF8', primaryPressed: '#6366F1', onPrimary: '#FFFFFF', + offlineBackground: '#C7D2FE', + offlineText: '#1E1B4B', success: '#34D399', successSurface: '#10291F', streak: '#FB923C', diff --git a/mobile/tests/accessibility.test.mjs b/mobile/tests/accessibility.test.mjs new file mode 100644 index 0000000..eb33564 --- /dev/null +++ b/mobile/tests/accessibility.test.mjs @@ -0,0 +1,14 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { lightColors, darkColors } from '../src/theme/index.ts'; +const luminance = hex => { + const rgb=hex.slice(1).match(/../g).map(v=>parseInt(v,16)/255) + .map(v=>v<=0.04045?v/12.92:((v+0.055)/1.055)**2.4); + return 0.2126*rgb[0]+0.7152*rgb[1]+0.0722*rgb[2]; +}; +for (const [name, colors] of Object.entries({light:lightColors,dark:darkColors})) { + test(`${name} offline status meets normal-text contrast`, () => { + const a=luminance(colors.offlineBackground), b=luminance(colors.offlineText); + assert.ok((Math.max(a,b)+0.05)/(Math.min(a,b)+0.05)>=4.5); + }); +} From 920c8d61c86a2ec4363c4a2506bb056be12334c8 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:40:42 +0500 Subject: [PATCH 26/36] feat: add deliberate refresh controls across learning screens --- mobile/src/hooks/useRefreshControl.tsx | 35 ++++++++++ mobile/src/screens/ConceptDetailScreen.tsx | 20 ++++-- mobile/src/screens/HistoryScreen.tsx | 4 ++ mobile/src/screens/PersonalizationScreen.tsx | 6 +- mobile/src/screens/ProfileScreen.tsx | 7 +- mobile/src/screens/SavedScreen.tsx | 4 ++ mobile/src/screens/StatsScreen.tsx | 8 ++- mobile/src/screens/TodayScreen.tsx | 5 +- mobile/src/services/conceptApi.ts | 3 +- mobile/tests/history.browser.cjs | 23 ++++--- mobile/tests/learning-ui.browser.cjs | 72 ++++++++++++++++++++ 11 files changed, 166 insertions(+), 21 deletions(-) create mode 100644 mobile/src/hooks/useRefreshControl.tsx create mode 100644 mobile/tests/learning-ui.browser.cjs diff --git a/mobile/src/hooks/useRefreshControl.tsx b/mobile/src/hooks/useRefreshControl.tsx new file mode 100644 index 0000000..76065da --- /dev/null +++ b/mobile/src/hooks/useRefreshControl.tsx @@ -0,0 +1,35 @@ +import { useCallback, useEffect, useRef, useState } from 'react'; +import { Pressable, RefreshControl, Text, View } from 'react-native'; +import { useTheme } from '../context/ThemeContext'; +import { scaleFont, spacing } from '../theme'; + +/** One action for native pull gestures and an accessible button on every platform. */ +export function useRefreshControl(label: string, task: () => Promise) { + const { colors } = useTheme(); + const [refreshing, setRefreshing] = useState(false); + const [failed, setFailed] = useState(false); + const busy = useRef(false); + const mounted = useRef(true); + useEffect(() => { mounted.current = true; return () => { mounted.current = false; }; }, []); + const onRefresh = useCallback(async () => { + if (busy.current) return; + busy.current = true; + setRefreshing(true); + setFailed(false); + try { await task(); } catch { if (mounted.current) setFailed(true); } + finally { busy.current = false; if (mounted.current) setRefreshing(false); } + }, [task]); + const control = ; + const action = + + + {refreshing ? 'Refreshing…' : `Refresh ${label}`} + + + {failed ? Couldn’t refresh. Try again. : null} + ; + return { control, action, refreshing, onRefresh }; +} diff --git a/mobile/src/screens/ConceptDetailScreen.tsx b/mobile/src/screens/ConceptDetailScreen.tsx index 2f85d94..fdd0259 100644 --- a/mobile/src/screens/ConceptDetailScreen.tsx +++ b/mobile/src/screens/ConceptDetailScreen.tsx @@ -1,6 +1,7 @@ +import { useRefreshControl } from '../hooks/useRefreshControl'; import { Ionicons } from '@expo/vector-icons'; import { RouteProp, useNavigation, useRoute } from '@react-navigation/native'; -import { useEffect, useMemo, useState } from 'react'; +import { useEffect, useMemo, useRef, useState } from 'react'; import { ActivityIndicator, Pressable, ScrollView, StyleSheet, Text, View } from 'react-native'; import { ConceptActions } from '../components/ConceptActions'; import { ConceptCard } from '../components/ConceptCard'; @@ -27,18 +28,25 @@ export function ConceptDetailScreen() { const [status, setStatus] = useState('loading'); const [attempt, setAttempt] = useState(0); const online = useOnline(); + const request = useRef(0); + const refreshUI = useRefreshControl('lesson', async () => { + const current = ++request.current; + const next = await fetchConcept(conceptId, undefined, true); + if (request.current === current) { setConcept(next); setStatus('ready'); } + }); useEffect(() => { let active = true; + const current = ++request.current; setStatus('loading'); fetchConcept(conceptId, (cached) => { - if (active) { + if (active && current === request.current) { setConcept(cached); setStatus('ready'); } }) .then((c) => { - if (active) { + if (active && current === request.current) { setConcept(c); setStatus('ready'); } @@ -47,7 +55,7 @@ export function ConceptDetailScreen() { // Offline or not found: the bundled catalog covers the signed-out demo // set; anything else we can't show, so say so rather than hang. const local = CONCEPTS_BY_ID.get(conceptId); - if (!active) return; + if (!active || current !== request.current) return; if (local) { setConcept(local); setStatus('ready'); @@ -57,6 +65,7 @@ export function ConceptDetailScreen() { }); return () => { active = false; + request.current++; }; }, [conceptId, attempt]); @@ -89,7 +98,8 @@ export function ConceptDetailScreen() { /> ) : ( - + + {refreshUI.action} diff --git a/mobile/src/screens/HistoryScreen.tsx b/mobile/src/screens/HistoryScreen.tsx index 422e92f..2051a7f 100644 --- a/mobile/src/screens/HistoryScreen.tsx +++ b/mobile/src/screens/HistoryScreen.tsx @@ -1,3 +1,4 @@ +import { useRefreshControl } from '../hooks/useRefreshControl'; import { Ionicons } from '@expo/vector-icons'; import { CompositeNavigationProp, ParamListBase, useNavigation } from '@react-navigation/native'; import { BottomTabNavigationProp } from '@react-navigation/bottom-tabs'; @@ -65,6 +66,7 @@ export function HistoryScreen() { const online = useOnline(); const { colors } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); + const refreshUI = useRefreshControl('history', refresh); // Composite: History is a tab screen that reaches up to the root stack's // concept-detail modal (#124). const navigation = @@ -88,6 +90,7 @@ export function HistoryScreen() { return ( `${r.date}-${r.conceptId}`} renderItem={({ item }) => ( @@ -102,6 +105,7 @@ export function HistoryScreen() { ListHeaderComponent={ History + {refreshUI.action} {history.records.length} of {progress.stats?.totalLearned ?? history.records.length} learned concepts loaded. createStyles(colors), [colors]); return ( - + + {refreshUI.action} Personalization >(); - const { progress, streaks } = useProgress(); + const { progress, streaks, refresh } = useProgress(); const { email, signOut } = useAuth(); const { colors, mode, toggle } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); + const refreshUI = useRefreshControl('profile', refresh); // The list itself now lives on a dedicated Saved screen (issue #131); the // Profile only needs the count. Use bookmarks.length — the same source as the @@ -69,7 +71,8 @@ export function ProfileScreen() { }, [prefs]); return ( - + + {refreshUI.action} diff --git a/mobile/src/screens/SavedScreen.tsx b/mobile/src/screens/SavedScreen.tsx index 086feb0..f98bd6a 100644 --- a/mobile/src/screens/SavedScreen.tsx +++ b/mobile/src/screens/SavedScreen.tsx @@ -1,3 +1,4 @@ +import { useRefreshControl } from '../hooks/useRefreshControl'; import { Ionicons } from '@expo/vector-icons'; import { CompositeNavigationProp, useNavigation } from '@react-navigation/native'; import { NativeStackNavigationProp } from '@react-navigation/native-stack'; @@ -38,6 +39,7 @@ export function SavedScreen() { const { savedConcepts, loading, failed, retry } = useSavedConcepts(progress); const { colors } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); + const refreshUI = useRefreshControl('saved lessons', async () => { await refresh(); retry(); }); const [query, setQuery] = useState(''); const [category, setCategory] = useState(ALL); @@ -88,6 +90,7 @@ export function SavedScreen() { return ( + {refreshUI.action} navigation.goBack()} @@ -174,6 +177,7 @@ export function SavedScreen() { ) : null} c.id} contentContainerStyle={styles.listContent} diff --git a/mobile/src/screens/StatsScreen.tsx b/mobile/src/screens/StatsScreen.tsx index 8ec07a2..cfd33ee 100644 --- a/mobile/src/screens/StatsScreen.tsx +++ b/mobile/src/screens/StatsScreen.tsx @@ -1,3 +1,5 @@ +import { fetchTopics } from '../services/topicsApi'; +import { useRefreshControl } from '../hooks/useRefreshControl'; import { useMemo } from 'react'; import { ScrollView, StyleSheet, Text, View } from 'react-native'; import { SkeletonBlock } from '../components/Skeleton'; @@ -65,12 +67,13 @@ function ProgressBar({ fraction, styles }: { fraction: number; styles: Styles }) } export function StatsScreen() { - const { loading, progress, streaks } = useProgress(); + const { loading, progress, streaks, refresh } = useProgress(); const { loading: topicsLoading, topics, error, retry } = useTopics(); const { session } = useAuth(); const online = useOnline(); const { colors } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); + const refreshUI = useRefreshControl('stats', async () => { await refresh(); await fetchTopics(); }); // A failed catalog request must not substitute demo totals for a real account. const serverMode = !!session; @@ -92,7 +95,8 @@ export function StatsScreen() { }; return ( - + + {refreshUI.action} Stats Your learning progress over time. diff --git a/mobile/src/screens/TodayScreen.tsx b/mobile/src/screens/TodayScreen.tsx index b9f4471..230e8f2 100644 --- a/mobile/src/screens/TodayScreen.tsx +++ b/mobile/src/screens/TodayScreen.tsx @@ -1,3 +1,4 @@ +import { useRefreshControl } from '../hooks/useRefreshControl'; import { useNavigation, NavigationProp, NavigatorScreenParams } from '@react-navigation/native'; import type { ProfileStackParamList } from './ProfileScreen'; import { Ionicons } from '@expo/vector-icons'; @@ -32,6 +33,7 @@ export function TodayScreen() { const online = useOnline(); const { colors, mode, toggle } = useTheme(); const styles = useMemo(() => createStyles(colors), [colors]); + const refreshUI = useRefreshControl('today', refresh); const navigation = useNavigation}>>(); const explore = () => navigation.navigate('Profile', {screen: 'Personalization'}); @@ -57,7 +59,8 @@ export function TodayScreen() { outcome?.status === 'ok' && outcome.payload.outside_followed_topics; return ( - + + {refreshUI.action} One Concept diff --git a/mobile/src/services/conceptApi.ts b/mobile/src/services/conceptApi.ts index 061abbb..3d1aa91 100644 --- a/mobile/src/services/conceptApi.ts +++ b/mobile/src/services/conceptApi.ts @@ -40,12 +40,13 @@ async function downloadConcept(slug: string): Promise { export async function fetchConcept( slug: string, onCached?: (concept: Concept) => void, + forceRefresh = false, ): Promise { const epoch = conceptCache.epoch; const cached = await conceptCache.get(slug, epoch); if (cached) { onCached?.(cached); - if (!getConnectivity()) return cached; + if (!getConnectivity() && !forceRefresh) return cached; } try { const concept = await downloadConcept(slug); diff --git a/mobile/tests/history.browser.cjs b/mobile/tests/history.browser.cjs index 346c39d..ec950d2 100644 --- a/mobile/tests/history.browser.cjs +++ b/mobile/tests/history.browser.cjs @@ -1,7 +1,6 @@ const {chromium,expect}=require(process.env.PLAYWRIGHT_TEST_MODULE || 'playwright/test'); const http=require('node:http'),fs=require('node:fs'),path=require('node:path'),assert=require('node:assert/strict'); const root=process.argv[2]; -const rejectReview=process.argv.includes('--reject-review'); if(!root || !fs.existsSync(path.join(root,'index.html'))) throw Error('Pass the exported web directory'); const version=require('../app.config.js').expo.version; const today=new Date().toISOString().slice(0,10); @@ -17,7 +16,7 @@ const server=http.createServer((req,res)=>{ await new Promise(r=>server.listen(4781,'127.0.0.1',r)); const browser=await chromium.launch({executablePath:process.env.PLAYWRIGHT_CHROMIUM_PATH,headless:true,args:['--no-sandbox']}); try { - let online=true, failPage=false, holdPage=false, releasePage, pageRequests=0; + let online=true, failPage=false, holdPage=false, releasePage, pageRequests=0, detailRequests=0; const records=Array.from({length:120},(_,i)=>({concept_slug:`lesson-${i}`,title:`Historical lesson ${i}`,topic_name:'Computer Science',like_count:0,learned_on:new Date(Date.now()-(i+1)*86400000).toISOString().slice(0,10)})); const state={display_name:'Reader',timezone:'UTC',today,followed_topics:['computer-science'],learned:records.slice(0,50),history_next_cursor:records[49].learned_on,likes:[],bookmarks:[],saved:[],stats:{current:120,longest:120,total_learned:120,total_reviews:0},assignment_slug:concept.slug,daily:{assigned_for:today,assigned_at:today+'T08:00:00Z',learned:false,completed_at:null,outside_followed_topics:false,concept}}; const context=await browser.newContext({viewport:{width:390,height:844}}); @@ -40,36 +39,42 @@ const server=http.createServer((req,res)=>{ const items=remaining.slice(0,50);body={items,next_cursor:remaining.length>50?items.at(-1).learned_on:null}; } else if(endpoint==='/v1/topics') body=[{slug:'computer-science',name:'Computer Science',concept_count:125,following:true}]; else if(endpoint==='/v1/me/notifications') body={enabled:false,reminder_times:['08:00']}; - else if(endpoint.startsWith('/v1/concepts/')) body={...concept,slug:decodeURIComponent(endpoint.split('/').pop()),title:'Historical lesson 119',summary:'Downloaded historical explanation.'}; + else if(endpoint.startsWith('/v1/concepts/')) { detailRequests++;body={...concept,slug:decodeURIComponent(endpoint.split('/').pop()),title:'Historical lesson 119',summary:'Downloaded historical explanation.'}; } await route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(body)}); }); await context.route('**/auth/v1/**',route=>route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(session)})); const page=await context.newPage(), errors=[];page.on('pageerror',e=>errors.push(e.message)); + // Keyboard activation avoids racing the virtualized footer's scroll settling. + const older=async label=>{const button=page.getByRole('button',{name:label,exact:true});await button.focus();await button.press('Enter');}; await page.goto('http://127.0.0.1:4781'); await expect(page.getByText('Today’s concept',{exact:true})).toBeVisible();assert.equal(pageRequests,0); await page.getByRole('tab',{name:'History'}).click(); await expect(page.getByText('50 of 120 learned concepts loaded.',{exact:true})).toBeVisible();assert.equal(pageRequests,0); - failPage=true;await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + failPage=true;await older('Load older lessons'); await expect(page.getByRole('button',{name:'Retry older lessons',exact:true})).toBeVisible();failPage=false; - await page.getByRole('button',{name:'Retry older lessons',exact:true}).click(); + await older('Retry older lessons'); await expect(page.getByText('100 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); - await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await older('Load older lessons'); await expect(page.getByText('120 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); await page.getByPlaceholder('Search loaded history').fill('Historical lesson 119'); await page.getByRole('button',{name:'Open Historical lesson 119',exact:true}).click(); await expect(page.getByText('Downloaded historical explanation.',{exact:true})).toBeVisible(); online=false;await page.reload();await page.getByRole('tab',{name:'History'}).click(); await expect(page.getByText('50 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); - await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await older('Load older lessons'); await expect(page.getByText('100 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); - await page.getByRole('button',{name:'Load older lessons',exact:true}).click(); + await older('Load older lessons'); await expect(page.getByText('120 of 120 learned concepts loaded.',{exact:true})).toBeVisible(); await page.getByPlaceholder('Search loaded history').fill('Historical lesson 119'); await page.getByRole('button',{name:'Open Historical lesson 119',exact:true}).click(); await expect(page.getByText('Downloaded historical explanation.',{exact:true})).toBeVisible(); + await page.getByRole('button',{name:'Refresh lesson',exact:true}).click(); + const previousDetails=detailRequests;online=true; + await page.getByRole('button',{name:'Refresh lesson',exact:true}).click(); + await expect.poll(()=>detailRequests).toBe(previousDetails+1); console.log('PASS: bounded History pages, retry, search and downloaded history/detail after offline restart'); online=true;holdPage=true;await page.reload();await page.getByRole('tab',{name:'History'}).click(); - await page.getByRole('button',{name:'Load older lessons',exact:true}).click();await expect.poll(()=>!!releasePage).toBe(true); + await older('Load older lessons');await expect.poll(()=>!!releasePage).toBe(true); await page.getByRole('tab',{name:'Profile'}).click();await page.getByText('Sign out',{exact:true}).click(); await expect(page.getByText('Welcome back — sign in to pick up your streak.',{exact:true})).toBeVisible();releasePage(); await expect.poll(()=>page.evaluate(()=>Object.keys(localStorage).filter(k=>k.startsWith('one-concept/history-pages/')).length)).toBe(0); diff --git a/mobile/tests/learning-ui.browser.cjs b/mobile/tests/learning-ui.browser.cjs new file mode 100644 index 0000000..fc7bfc9 --- /dev/null +++ b/mobile/tests/learning-ui.browser.cjs @@ -0,0 +1,72 @@ +const {chromium,expect}=require(process.env.PLAYWRIGHT_TEST_MODULE || 'playwright/test'); +const http=require('node:http'),fs=require('node:fs'),path=require('node:path'),assert=require('node:assert/strict'); +const root=process.argv[2]; +if(!root || !fs.existsSync(path.join(root,'index.html'))) throw Error('Pass the exported web directory'); +const version=require('../app.config.js').expo.version; +const today=new Date().toISOString().slice(0,10); +const concept={id:'33333333-3333-4333-8333-333333333333',slug:'known-lesson',title:'Reviewing invariants',summary:'An invariant is a rule that stays true while a system changes. Use it to check whether each operation keeps your data consistent.',example:'A library book can have one active borrower. Returning and lending it should preserve that rule.',topic_slug:'computer-science',topic_name:'Computer Science',content_version:2,like_count:0}; +const session={access_token:'fixture',refresh_token:'fixture-refresh',token_type:'bearer',expires_in:864000,expires_at:Math.floor(Date.now()/1000)+864000,user:{id:'11111111-1111-1111-1111-111111111111',email:'fixture@example.invalid',aud:'authenticated',role:'authenticated',app_metadata:{},user_metadata:{},created_at:'2026-01-01T00:00:00Z'}}; +const server=http.createServer((req,res)=>{ + const relative=decodeURIComponent(new URL(req.url,'http://localhost').pathname); + const file=path.join(root,relative==='/'?'index.html':relative); + try {res.setHeader('Content-Type',({'.html':'text/html','.js':'application/javascript','.ttf':'font/ttf','.png':'image/png'})[path.extname(file)]||'application/octet-stream');res.end(fs.readFileSync(file));} + catch{res.statusCode=404;res.end();} +}); +(async()=>{ + await new Promise(r=>server.listen(4781,'127.0.0.1',r)); + const browser=await chromium.launch({executablePath:process.env.PLAYWRIGHT_CHROMIUM_PATH,headless:true,args:['--no-sandbox']}); + try { + for(const theme of ['light','dark']) { + let online=true,reads=0,detailReads=0,hold=false,release; + const errors=[]; + const state={display_name:'Reader',timezone:'UTC',today,followed_topics:['computer-science'],learned:[],likes:[],bookmarks:[],saved:[],stats:{current:0,longest:0,total_learned:0,total_reviews:0},assignment_slug:concept.slug,daily:{assigned_for:today,assigned_at:today+'T08:00:00Z',learned:false,completed_at:null,outside_followed_topics:false,concept}}; + const context=await browser.newContext({viewport:{width:320,height:700}}); + await context.addInitScript(({session,version,theme})=>{ + localStorage.setItem('sb-127-auth-token',JSON.stringify(session)); + localStorage.setItem('one-concept/last-seen-version/v1',version); + localStorage.setItem('one-concept/theme/v1',theme); + },{session:{...session,user:{...session.user,email:'averylongreadernamefortextsizing@example.invalid'}},version,theme}); + await context.route('**/api/**',async route=>{ + if(!online)return route.abort('internetdisconnected'); + const endpoint=new URL(route.request().url()).pathname.replace('/api','');let body={}; + if(endpoint==='/v1/me/state'){reads++;if(hold)await new Promise(r=>{release=r;});body=state;} + else if(endpoint==='/v1/topics')body=[{slug:'computer-science',name:'Computer Science',concept_count:125,following:true}]; + else if(endpoint==='/v1/me/notifications')body={enabled:false,reminder_times:['08:00']}; + else if(endpoint.startsWith('/v1/concepts/')){detailReads++;body=concept;} + await route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(body)}); + }); + await context.route('**/auth/v1/**',route=>route.fulfill({status:200,contentType:'application/json',body:JSON.stringify(session)})); + const page=await context.newPage();page.on('pageerror',e=>errors.push(e.message)); + await page.goto('http://127.0.0.1:4781'); + await expect(page.getByText(concept.summary,{exact:true})).toBeVisible(); + const before=reads;hold=true; + await page.getByRole('button',{name:'Refresh today',exact:true}).click(); + await expect.poll(()=>!!release).toBe(true); + await expect(page.getByRole('button',{name:'Refresh today',exact:true})).toBeDisabled(); + hold=false;release(); + await expect(page.getByRole('button',{name:'Refresh today',exact:true})).toBeEnabled();assert.equal(reads,before+1); + online=false;await page.evaluate(()=>window.dispatchEvent(new Event('offline'))); + await page.getByRole('button',{name:'Refresh today',exact:true}).click(); + await expect(page.getByText(concept.summary,{exact:true})).toBeVisible(); + const banner=page.getByRole('alert',{name:'You are offline. Changes will sync when you reconnect.'}); + await expect(banner).toBeVisible(); + const ratio=await banner.evaluate(el=>{ + const rgb=s=>s.match(/[0-9.]+/g).slice(0,3).map(Number).map(v=>{v/=255;return v<=.04045?v/12.92:((v+.055)/1.055)**2.4;}); + const lum=s=>{const c=rgb(s);return .2126*c[0]+.7152*c[1]+.0722*c[2];}; + const bg=lum(getComputedStyle(el).backgroundColor); + const text=[...el.querySelectorAll('div')].find(x=>x.textContent==='Offline — changes will sync when you reconnect'); + const fg=lum(getComputedStyle(text).color);return(Math.max(bg,fg)+.05)/(Math.min(bg,fg)+.05); + });assert.ok(ratio>=4.5,`contrast ${ratio}`); + await page.getByRole('tab',{name:'Profile'}).click(); + await page.evaluate(()=>document.querySelectorAll('div,span').forEach(el=>{ + if(el.childNodes.length===1&&el.firstChild?.nodeType===Node.TEXT_NODE){const s=getComputedStyle(el);el.style.fontSize=(parseFloat(s.fontSize)*1.8)+'px';const line=parseFloat(s.lineHeight);if(Number.isFinite(line))el.style.lineHeight=(line*1.8)+'px';} + })); + const email=page.getByText('averylongreadernamefortextsizing@example.invalid',{exact:true}); + await email.scrollIntoViewIfNeeded();await expect(email).toBeVisible(); + const bounds=await email.boundingBox();assert.ok(bounds.x>=0&&bounds.x+bounds.width<=321); + await page.screenshot({path:`/tmp/next-profile-${theme}.png`,fullPage:true}); + assert.deepEqual(errors,[]);console.log(`${theme}: refresh completion/disabled state, offline content, contrast ${ratio.toFixed(2)}:1 and enlarged profile passed`); + await context.close(); + } + }finally{await browser.close();server.close();} +})().catch(e=>{console.error(e);server.close();process.exitCode=1;}); From 3e1ed7b24b17a59f9d22eea6871e644bde269eb2 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:41:31 +0500 Subject: [PATCH 27/36] docs: record learning experience validation and release handoff --- docs/CODEBASE_MAP.md | 7 ++++++- docs/WORK_LOG.md | 30 ++++++++++++++++++++++++++++++ mobile/tests/README.md | 22 ++++++++++++++++++++++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index f098c95..5d2a658 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -32,7 +32,7 @@ inside a root stack, with a concept-detail modal above them. | Screen | Responsibility | | --- | --- | | `TodayScreen.tsx` | Daily lesson, learned action, streak, loading/exhausted/offline states. | -| `HistoryScreen.tsx` | Learned records and navigation to concept details. | +| `HistoryScreen.tsx` | Paginated learning history, search within loaded records, offline pages and navigation to concept details. | | `StatsScreen.tsx` | Streak and topic statistics. | | `ProfileScreen.tsx` | Account, reminder preferences, theme, sign-out, and links to profile subpages. | | `PersonalizationScreen.tsx` | Server topic catalog and follow controls through `useTopics`. | @@ -49,6 +49,11 @@ like counts, streak/flame visuals, buttons, skeletons, the offline banner, typography, shadows, and scaling; `ThemeContext` persists light/dark preference. `src/navigation.ts` types the root stack. +History uses `hooks/useHistory.ts` and `services/historyApi.ts` to load one +50-item page per request from the existing history endpoint. Page caches join +account cleanup; the startup progress aggregate remains compact. Data screens +share `hooks/useRefreshControl.tsx` for native pull gestures and refresh buttons. + ## Mobile state, persistence, and API boundaries - `src/lib/supabase.ts` creates the Auth client. `secureStorage.ts` chunks native diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index dae0df0..75d46ab 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,36 @@ claims as completed work. ## Current status +## Learning experience batch (#158, #159, #160) + +- Implemented one PR from merged `develop` (`3937927`), branch + `codex/learning-experience-polish`, isolated in `/tmp/one-concept-next`. +- Confirmed History still stopped at ten, refresh gestures were absent, and + offline banner contrast was insufficient. Existing explicit retry screens + were already present and remain available. +- `e2eed9b`: full History through the existing 50-item cursor API, explicit older + page loading, search within loaded records, account-scoped page caching and + sign-out cleanup. Startup remains compact; previously opened lesson bodies + remain readable offline. `afa8e96`: high-contrast offline colors, wrapping + profile identity and a compact detail header with a 44px close target. +- `feat: add deliberate refresh controls across learning screens`: shared native + pull controls and accessible buttons, disabled during active refresh, preserving + offline content and allowing explicit detail refresh to probe reconnection. +- Validation: Node 24 typecheck and **39 tests passed**, no skips; Android/iOS/web + exports passed. Mocked browser checks passed for 120-item History, 503 retry, + offline restart/pages/detail, explicit detail reconnection and in-flight sign-out; + rejected-review replay in both themes; refresh busy state, offline content, + banner contrast and 320px enlarged profile in both themes. Inspected screenshots. + Banner contrast measured **9.93:1 light / 10.72:1 dark**. Local links and whitespace + checked. Backend code and schema are unchanged; backend tests were not rerun. +- Browser footer activation was made deterministic with keyboard interaction + after a scroll-timing flake; request tracing confirmed page boundaries and + stale-response fencing. Temporary instrumentation was removed from the source. +- Native pull gestures, Dynamic Type, TalkBack/VoiceOver still need device checks. + Search covers loaded History pages; older bodies require prior download. + PR closes the three issues when merged. No manual issue closure or release yet. + Release preparation and its one-time What's New card follow the feature merge. + ### PR #197 review correction - Fixed rejected offline review replay in the same PR. Pre-completion statistics diff --git a/mobile/tests/README.md b/mobile/tests/README.md index 79a0d4a..6ef3530 100644 --- a/mobile/tests/README.md +++ b/mobile/tests/README.md @@ -124,3 +124,25 @@ review and exact pre-tap totals before and after restart, in both themes. Physical-device font scaling, screen readers and native storage still require manual acceptance. Syncing remains foreground/reopen JS work on the current APK. + + +## Learning experience (#158–#160) + +Using the same dummy-config export and Playwright environment above, run these +scripts sequentially (each uses port 4781): + +```sh +node tests/history.browser.cjs /path/to/export +node tests/learning-ui.browser.cjs /path/to/export +``` + +History checks 120 lessons with 50-item pages, retry after 503, search within the +loaded pages, offline page/detail reading after restart, and sign-out during a +page request. Older metadata is requested only when Load older lessons is used; +search explicitly covers loaded history. Lesson bodies must have been opened or +saved online to be available offline. + +UI checks manual refresh, disabled controls during a request, retained offline +content, actual banner contrast in both themes, and an enlarged profile at +320px. Native pull gestures, OS Dynamic Type, TalkBack and VoiceOver still need +physical-device acceptance. Refresh buttons provide an accessible alternative. From 80e14e1b955f921012c42539f955e178688fa970 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:42:24 +0500 Subject: [PATCH 28/36] docs: link combined learning experience PR --- docs/WORK_LOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 75d46ab..fc88626 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -34,7 +34,8 @@ claims as completed work. stale-response fencing. Temporary instrumentation was removed from the source. - Native pull gestures, Dynamic Type, TalkBack/VoiceOver still need device checks. Search covers loaded History pages; older bodies require prior download. - PR closes the three issues when merged. No manual issue closure or release yet. + [PR #198](https://github.com/Coding-Moves/one-concept/pull/198) targets `develop` + and closes the three issues when merged. No manual issue closure or release yet. Release preparation and its one-time What's New card follow the feature merge. ### PR #197 review correction From b38fd15918076e54625a4d644f0f8474565c6b46 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:53:13 +0500 Subject: [PATCH 29/36] fix: use the public history cursor parameter --- mobile/src/services/historyApi.ts | 2 +- mobile/tests/history.browser.cjs | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/mobile/src/services/historyApi.ts b/mobile/src/services/historyApi.ts index d6b96a2..5db215a 100644 --- a/mobile/src/services/historyApi.ts +++ b/mobile/src/services/historyApi.ts @@ -13,7 +13,7 @@ export async function fetchHistoryPage(userId: string, cursor: string, epoch: nu const page = await apiRequest<{ items: { concept_slug: string; learned_on: string; title: string; topic_name: string; like_count: number }[]; next_cursor: string | null; - }>(`/v1/me/history?limit=50&before=${encodeURIComponent(cursor)}`); + }>(`/v1/me/history?limit=50&cursor=${encodeURIComponent(cursor)}`); if (page.next_cursor && page.next_cursor >= cursor) throw new Error('History cursor did not advance'); const result = { items: page.items.map(row => ({ conceptId: row.concept_slug, date: row.learned_on, diff --git a/mobile/tests/history.browser.cjs b/mobile/tests/history.browser.cjs index ec950d2..aeedb76 100644 --- a/mobile/tests/history.browser.cjs +++ b/mobile/tests/history.browser.cjs @@ -35,7 +35,9 @@ const server=http.createServer((req,res)=>{ pageRequests++;assert.equal(url.searchParams.get('limit'),'50'); if(holdPage) await new Promise(r=>{releasePage=r;}); if(failPage) return route.fulfill({status:503,contentType:'application/json',body:'{}'}); - const remaining=records.filter(r=>r.learned_on!cursor || r.learned_on50?items.at(-1).learned_on:null}; } else if(endpoint==='/v1/topics') body=[{slug:'computer-science',name:'Computer Science',concept_count:125,following:true}]; else if(endpoint==='/v1/me/notifications') body={enabled:false,reminder_times:['08:00']}; From cc9b6240408c25e32a3185ff2e76f76c67c47523 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:53:38 +0500 Subject: [PATCH 30/36] docs: record history cursor regression verification --- docs/WORK_LOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index fc88626..67b44b2 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,17 @@ claims as completed work. ## Current status +### PR #198 cursor review follow-up + +- `b38fd15` fixes the confirmed History query mismatch (`before` vs backend + `cursor`) in the same PR, with the browser fixture matching the real contract. +- Before the fix, the corrected fixture reproduced 50/120 records and a retry + error instead of loading the next page. After the fix, all 120 records load. +- Validation: Node 24 typecheck, all 39 unit tests, web export and the corrected + History browser test passed (paging, retry, offline restart/detail, sign-out + and late responses). Whitespace checks passed. No backend changes; native + gesture/device checks remain unverified as recorded below. + ## Learning experience batch (#158, #159, #160) - Implemented one PR from merged `develop` (`3937927`), branch From 6166d80581afd230f7d7cde68506c2cfb491f684 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:56:59 +0500 Subject: [PATCH 31/36] release: prepare 1.9.0 with one-time learning highlights --- mobile/app.config.js | 2 +- mobile/src/data/whatsNew.ts | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/mobile/app.config.js b/mobile/app.config.js index ba62a98..770aa01 100644 --- a/mobile/app.config.js +++ b/mobile/app.config.js @@ -18,7 +18,7 @@ module.exports = { name: 'One Concept', slug: 'one-concept', owner: 'coding-moves', - version: '1.8.0', + version: '1.9.0', orientation: 'portrait', icon: './assets/icon.png', userInterfaceStyle: 'automatic', diff --git a/mobile/src/data/whatsNew.ts b/mobile/src/data/whatsNew.ts index 4be8f70..335d8b0 100644 --- a/mobile/src/data/whatsNew.ts +++ b/mobile/src/data/whatsNew.ts @@ -21,6 +21,15 @@ export interface WhatsNewEntry { } export const WHATS_NEW: WhatsNewEntry[] = [ + { + version: '1.9.0', + highlights: [ + 'Keep learning with daily reviews when you have explored the available new lessons.', + 'Revisit your complete learning history and search the lessons you have loaded.', + 'Refresh your learning screens with a pull or a tap, while keeping downloaded content available offline.', + 'Read clearer offline notices and see your full profile details at larger text sizes.', + ], + }, { version: '1.8.0', highlights: [ From ac4452d4841ab285c9d13902974e1f1ae39c0c85 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 21:57:49 +0500 Subject: [PATCH 32/36] docs: record 1.9.0 validation and rollout prerequisites --- docs/WORK_LOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 67b44b2..180f0c7 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,20 @@ claims as completed work. ## Current status +### Version 1.9.0 release preparation + +- Preparing the merged #197/#198 work for a `develop` to `main` release. +- Version 1.9.0 includes a matching one-time What's New card; native runtime + stays at 1.3.0. Preserve the existing per-device dismissal behavior. +- Release must remain draft until production backup, generator pause and verified + application of migrations 0011–0015 are complete. No production operations or + ledger claims are included in this preparation. +- `6166d80` adds version/card together. Node 24 typecheck and all 39 tests + passed; Android/iOS/web exports and whitespace checks passed. Physical-device + checks remain outstanding. The release PR is intentionally draft pending the + production rollout; its migration check will fail until verified application. + + ### PR #198 cursor review follow-up - `b38fd15` fixes the confirmed History query mismatch (`before` vs backend From 297eb6ce49e981ef8219cec265b83df595079d59 Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 22:45:08 +0500 Subject: [PATCH 33/36] release: record verified production migrations 0011 through 0015 --- backend/migrations/applied.txt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/backend/migrations/applied.txt b/backend/migrations/applied.txt index 0ff1ffc..cafc655 100644 --- a/backend/migrations/applied.txt +++ b/backend/migrations/applied.txt @@ -18,3 +18,8 @@ 0008_backlog_claimed_at.sql 0009_like_count_index.sql 0010_generation_daily_usage.sql +0011_content_supply.sql +0012_curriculum_publication.sql +0013_daily_reviews.sql +0014_content_operations.sql +0015_revision_generation_claims.sql From 87ee0fab4ec2ceacab081018e11827c9613f066f Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 22:45:08 +0500 Subject: [PATCH 34/36] docs: record production verification and remaining rollout steps --- docs/CODEBASE_MAP.md | 8 ++++---- docs/WORK_LOG.md | 17 +++++++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index 5d2a658..4860a3a 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -239,10 +239,10 @@ enforce one daily assignment and no concept repeats per user. RLS adds isolation | `0014_content_operations.sql` | Worker heartbeat and deduplicated condition state. | | `0015_revision_generation_claims.sql` | Durable claims for correction drafting. | -Only migrations 0001–0010 are recorded in `migrations/applied.txt`; new migrations -0011–0015 are unapplied in production at this PR handoff. Migration 0010 was -applied and independently verified in production during the 1.8.0 release -follow-up. The ledger is repository evidence, not a live check of production. +Migrations 0001–0015 are recorded in `migrations/applied.txt`. The owner applied +0011–0015 during 1.9.0 release preparation, and a separate read-only production +connection verified their tables, RLS, columns, indexes, constraints and backfill. +The backend/worker rollout remains pending; the ledger does not prove deployment. Application connections use the transaction pooler; migration DDL uses `DIRECT_URL` and the session pooler. Applied migrations must not be rewritten. diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 180f0c7..1298c1f 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,23 @@ claims as completed work. ## Current status +### Version 1.9.0 production migration verification + +- Owner reported generation paused and manually applied migrations 0011–0015 + in order through SQL Editor, each reporting success. +- Independently verified the original production project through a read-only + session-pooler transaction: six new tables, RLS enabled with no client policies, + four added columns, five indexes, foreign keys/checks, daily-review uniqueness, + final revision-status constraint and published-lesson timestamp backfill. +- Record all five migrations only after that verification. No production writes + were performed by this verification; migration files remain immutable. +- Local pre-release dump exists and its archive index is readable. Full restore + remains untested; owner explicitly deferred the separate cloud Backup project. +- Backend/worker rollout and ordering before OTA remain outstanding. Keep + generation paused and release #200 draft; do not merge main yet. +- Validation: migration file/ledger comparison and whitespace checks passed. + No application code changed; application tests were not rerun for this entry. + ### Version 1.9.0 release preparation - Preparing the merged #197/#198 work for a `develop` to `main` release. From fe5315b91bd174dd82fdc3b7ebc21c6d5cd725bd Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 22:50:16 +0500 Subject: [PATCH 35/36] fix: publish production OTA only after deployed revision confirmation --- .github/scripts/check-release-revision.sh | 13 +++++++++++ .../scripts/check-release-revision.test.cjs | 20 ++++++++++++++++ .github/workflows/eas-update.yml | 19 +-------------- .github/workflows/release-validation.yml | 17 ++++++++++++++ .github/workflows/release.yml | 23 +++++++++++++------ 5 files changed, 67 insertions(+), 25 deletions(-) create mode 100644 .github/scripts/check-release-revision.sh create mode 100644 .github/scripts/check-release-revision.test.cjs create mode 100644 .github/workflows/release-validation.yml diff --git a/.github/scripts/check-release-revision.sh b/.github/scripts/check-release-revision.sh new file mode 100644 index 0000000..570bf2e --- /dev/null +++ b/.github/scripts/check-release-revision.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail +# This validates an operator's deployment attestation, not Railway health itself. +if [[ "${GITHUB_REF:-}" != refs/heads/main ]]; then + echo '::error::Publish releases from main only.' + exit 1 +fi +if [[ ! "${VERIFIED_BACKEND_SHA:-}" =~ ^[0-9a-f]{40}$ ]] || + [[ "$VERIFIED_BACKEND_SHA" != "${GITHUB_SHA:-}" ]] || + [[ "$VERIFIED_BACKEND_SHA" != "${REMOTE_MAIN_SHA:-}" ]]; then + echo '::error::Verify the current main commit on the production API and workers, then provide its full SHA.' + exit 1 +fi diff --git a/.github/scripts/check-release-revision.test.cjs b/.github/scripts/check-release-revision.test.cjs new file mode 100644 index 0000000..f0ea9d3 --- /dev/null +++ b/.github/scripts/check-release-revision.test.cjs @@ -0,0 +1,20 @@ +const {test}=require('node:test'); +const assert=require('node:assert/strict'); +const {spawnSync}=require('node:child_process'); +const path=require('node:path'); +const sha='a'.repeat(40), other='b'.repeat(40); +const baseline={...process.env,GITHUB_REF:'refs/heads/main',GITHUB_SHA:sha,VERIFIED_BACKEND_SHA:sha,REMOTE_MAIN_SHA:sha}; +for(const [name,override,success] of [ + ['matching main deployment',{},true], + ['preview branch',{GITHUB_REF:'refs/heads/develop'},false], + ['tag dispatch',{GITHUB_REF:'refs/tags/v1.9.0'},false], + ['missing attestation',{VERIFIED_BACKEND_SHA:''},false], + ['short SHA',{VERIFIED_BACKEND_SHA:'aaaaaaa'},false], + ['older backend',{VERIFIED_BACKEND_SHA:other},false], + ['main advanced after dispatch',{REMOTE_MAIN_SHA:other},false], + ['remote lookup failed',{REMOTE_MAIN_SHA:''},false], + ['shell characters',{VERIFIED_BACKEND_SHA:'$(exit 0)'},false], +]) test(name,()=>{ + const result=spawnSync('bash',[path.join(__dirname,'check-release-revision.sh')],{env:{...baseline,...override},encoding:'utf8'}); + assert.equal(result.status,success?0:1,result.stderr+result.stdout); +}); diff --git a/.github/workflows/eas-update.yml b/.github/workflows/eas-update.yml index 99aec6d..1ecff76 100644 --- a/.github/workflows/eas-update.yml +++ b/.github/workflows/eas-update.yml @@ -1,5 +1,5 @@ # Preview-channel OTA for the develop branch (the developer's own devices), -# plus a manual dispatch that publishes the chosen channel. Production OTA +# plus a manual preview dispatch. Production OTA # for a release lives in release.yml, so the release tag is only cut after # that publish succeeds. Native changes need a new build — see # mobile/DEPLOYMENT.md. @@ -15,12 +15,6 @@ on: - "mobile/**" - "!mobile/**.md" workflow_dispatch: - inputs: - channel: - description: "Update channel (manual runs only)" - type: choice - default: preview - options: [preview, production] jobs: update: @@ -46,18 +40,7 @@ jobs: # --environment pulls the EXPO_PUBLIC_* variables from EAS into the # bundle; without it the update ships with empty config and crashes. - - name: Publish OTA update (production) - if: github.event_name == 'workflow_dispatch' && github.event.inputs.channel == 'production' - env: - MSG: ${{ github.event.head_commit.message || 'manual dispatch' }} - run: eas update --channel production --environment production --message "$MSG" --non-interactive - - # A push to develop refreshes preview; a manual run publishes only the - # chosen channel. - name: Publish OTA update (preview) - if: >- - (github.event_name == 'push') || - (github.event_name == 'workflow_dispatch' && github.event.inputs.channel == 'preview') env: MSG: ${{ github.event.head_commit.message || 'manual dispatch' }} run: eas update --channel preview --environment preview --message "$MSG" --non-interactive diff --git a/.github/workflows/release-validation.yml b/.github/workflows/release-validation.yml new file mode 100644 index 0000000..97f6bd3 --- /dev/null +++ b/.github/workflows/release-validation.yml @@ -0,0 +1,17 @@ +name: Release guard validation +on: + pull_request: + paths: + - '.github/scripts/check-release-revision*' + - '.github/workflows/release*.yml' + - '.github/workflows/eas-update.yml' +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-node@v7 + with: + node-version: 24 + - run: bash -n .github/scripts/check-release-revision.sh + - run: node --test --test-isolation=none .github/scripts/check-release-revision.test.cjs diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 4152006..a8901e2 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,13 +1,14 @@ -# Merging a release PR into main cuts a versioned release. The production OTA -# publishes FIRST; the tag and GitHub Release are created only if that publish -# succeeds, so a Release can never claim a version that never reached phones. -# (Railway deploys the backend independently on the same push — see -# mobile/DEPLOYMENT.md; it is not gated here.) +# Railway deploys main independently. Publish only after the operator verifies +# the same commit on the API and workers and checks production health. name: Release on: - push: - branches: [main] + workflow_dispatch: + inputs: + backend_revision: + description: 'Full main commit SHA verified on healthy production API and workers' + required: true + type: string # One release at a time: two quick merges must not race the tag check. concurrency: @@ -27,6 +28,14 @@ jobs: steps: - uses: actions/checkout@v7 + - name: Verify deployed revision before publishing + env: + VERIFIED_BACKEND_SHA: ${{ inputs.backend_revision }} + run: | + REMOTE_MAIN_SHA=$(git ls-remote origin refs/heads/main | cut -f1) + export REMOTE_MAIN_SHA + bash ../.github/scripts/check-release-revision.sh + - uses: actions/setup-node@v7 with: node-version: 22 From fd74b15bccfcfe1ea80077842d79a5cb8a7f55ad Mon Sep 17 00:00:00 2001 From: Muawiya Amir Date: Sun, 13 Sep 2026 22:50:40 +0500 Subject: [PATCH 36/36] docs: explain backend-first release and local backup status --- AGENTS.md | 3 ++- RELEASING.md | 19 +++++++++++++++---- docs/CODEBASE_MAP.md | 6 ++++-- docs/WORK_LOG.md | 21 +++++++++++++++++++++ mobile/DEPLOYMENT.md | 25 ++++++++++++------------- 5 files changed, 54 insertions(+), 20 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ad8690e..3900d82 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,8 @@ requires the exact Expo SDK 57 documentation before writing mobile code. Use a descriptive `codex/` branch for a new chunk unless the owner specifies a branch. Check the available base revision; do not assume local refs are current. - `main` is production. A production release uses a `develop` to `main` PR and - the release runbook. Merging there triggers deployment and release automation. + the release runbook. Merging deploys the backend; publish the mobile release + separately after verifying the deployed API and worker revision. - Every release PR must include a one-time What's New card. During release preparation, automatically add a nonempty entry in `mobile/src/data/whatsNew.ts` matching `mobile/app.config.js`'s version; do not wait for the owner to remind you. diff --git a/RELEASING.md b/RELEASING.md index 0223f1e..65c25e5 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -7,7 +7,8 @@ Production release flow. Keep it boring and repeatable. (This is our internal flow — the project doesn't take outside PRs; see [CONTRIBUTING.md](CONTRIBUTING.md).) - `main` is production. A release is a single PR **develop → main** (no `release/*` branch). -- Merging to `main` triggers `.github/workflows/release.yml`. +- Merging to `main` triggers Railway backend deployment. Mobile publication is + a separate manual run of `.github/workflows/release.yml` after verification. ## Cutting a release 1. **Bump the version on `develop` first.** Edit `mobile/app.config.js` → `expo.version` @@ -26,9 +27,19 @@ Production release flow. Keep it boring and repeatable. release PR.** 4. Open the release PR **develop → main**. It must pass the required **"Migrations applied check"** and get its approval, then merge. -5. On merge, `release.yml` publishes the production + preview OTA, cuts the `vX.Y.Z` - tag + GitHub Release, and dispatches the APK build (which **skips** unless - runtimeVersion changed). Railway auto-deploys the `api` service from `main`. +5. On merge, Railway auto-deploys the API from `main`. Keep generation paused + during backend/worker transitions. Verify the new main commit is deployed to + the API and workers, `/health` succeeds, and the release-specific smoke checks + pass. Do not resume old generators against the new editorial schema. +6. Open **GitHub Actions → Release → Run workflow**, select **main**, and enter + the full 40-character main commit SHA you verified on production API/workers + in `backend_revision`. This is an operator attestation; the workflow does not + inspect Railway deployments itself. Do not submit it until checks are complete. +7. The workflow rejects another branch, an older deployment or a main revision + that advanced before validation. It then publishes production + preview OTA, + cuts the version tag/GitHub Release, and dispatches the native-gated APK build. + Do not merge another release while publication is running. The standalone + EAS Update workflow publishes preview only; production uses this release path. ## Database migrations — MANUAL, every release The deploy does **not** auto-migrate. Files in `backend/migrations/*.sql` must be run diff --git a/docs/CODEBASE_MAP.md b/docs/CODEBASE_MAP.md index 4860a3a..1f7763c 100644 --- a/docs/CODEBASE_MAP.md +++ b/docs/CODEBASE_MAP.md @@ -265,9 +265,11 @@ and the session pooler. Applied migrations must not be rewritten. on port 55433, applies every migration, and disables live generation. HTTP calls to Gemini/Expo are mocked. Database-dependent tests skip if Podman cannot start. - `.github/workflows/eas-update.yml` publishes preview OTA on qualifying mobile - pushes to `develop`; manual dispatch can select a channel. `eas-build.yml` is + pushes to `develop`; manual dispatch also publishes preview only. `eas-build.yml` is a manual Android build workflow. -- `release.yml` runs on `main`, publishes production then preview OTA, creates a +- `release.yml` is manually dispatched on `main` after operator confirmation of + the deployed backend/worker SHA; its guard rejects missing/mismatched revisions + and non-main dispatches. It publishes production then preview OTA, creates a version tag/GitHub release, and dispatches `release-apk.yml`. APK publication is gated on native `runtimeVersion` changes. Railway deploys the backend independently; follow `RELEASING.md` for migration and release ordering. diff --git a/docs/WORK_LOG.md b/docs/WORK_LOG.md index 1298c1f..6bd6190 100644 --- a/docs/WORK_LOG.md +++ b/docs/WORK_LOG.md @@ -7,6 +7,27 @@ claims as completed work. ## Current status +### Release #200 readiness and deployment ordering + +- Owner requested a merge-ready release, with production merge left for approval. + Railway screenshot confirms main auto-deploy with `/backend` root directory. +- Separating the backend merge/deploy from mobile publication. Release becomes + manual on main with a required full deployed-commit attestation; a guard rejects + wrong branches, missing/mismatched SHAs and a main revision changed since dispatch. + Standalone EAS Update remains preview-only so it cannot bypass the release gate. +- This attestation is not automatic Railway verification. After merging main, + inspect API/worker deployments and health before dispatching Release. Keep + generation paused until compatible workers are confirmed. No production merge, + mobile publication or backend deployment is performed by this preparation. +- Local backup was created by the owner on their computer. Its index was checked; + nothing was uploaded/restored. Production SQL migrations were applied directly + by the owner and independently verified; they did not reload the backup. +- Validation: nine release-guard cases passed, including rejected stale/missing + revisions and non-main branches; shell syntax, all workflow YAML parsing and + production/preview wiring checks passed. No application code changed; retain + prior app test/export evidence. GitHub checks are verified at handoff. + + ### Version 1.9.0 production migration verification - Owner reported generation paused and manually applied migrations 0011–0015 diff --git a/mobile/DEPLOYMENT.md b/mobile/DEPLOYMENT.md index 6d3bbaf..2faa1c5 100644 --- a/mobile/DEPLOYMENT.md +++ b/mobile/DEPLOYMENT.md @@ -61,11 +61,11 @@ The `production` profile builds an AAB for Play Store submission; use ## OTA update (JS, UI, styling, assets — no reinstall) -```bash -cd mobile -npm run update:production # eas update --channel production -npm run update:preview # eas update --channel preview -``` +For production, follow [the release runbook](../RELEASING.md): merge the release, +verify the new backend and workers, then manually run **Release** on `main` with +the verified full commit SHA. Merging alone does not publish a mobile update. +Use **EAS Update (OTA)** for preview updates. Direct local production publishing +bypasses the deployment check and is not the normal release procedure. Installed apps fetch the update on next launch (`checkAutomatically: ON_LOAD`). @@ -80,12 +80,11 @@ OTA cannot ship native code. Instead: ## GitHub automation -Two workflows in [.github/workflows](../.github/workflows), both need the -`EXPO_TOKEN` repository secret (expo.dev → Account settings → Access tokens; -add at GitHub → Settings → Secrets and variables → Actions): +Workflows in [.github/workflows](../.github/workflows) use the `EXPO_TOKEN` +repository secret for EAS: -- **eas-update.yml** — every push to `main` touching `mobile/**` publishes an - OTA update to the **production** channel automatically. Manual dispatch lets - you pick `preview` instead. -- **eas-build.yml** — manual dispatch only (builds cost quota); choose the - profile. +- **release.yml** — manual production release after backend/worker verification, + followed by preview OTA, release tagging and native-gated APK dispatch. +- **eas-update.yml** — qualifying mobile pushes to `develop` and manual runs + publish preview only. +- **eas-build.yml** — manual build; choose the profile.