diff --git a/docs/admin/configuration/codemie/api-configuration.md b/docs/admin/configuration/codemie/api-configuration.md index 69d02736..2d30587e 100644 --- a/docs/admin/configuration/codemie/api-configuration.md +++ b/docs/admin/configuration/codemie/api-configuration.md @@ -608,14 +608,14 @@ Caches user-to-budget resolution results to reduce database load on high-traffic Manages automatic reset of per-member budget windows aligned with LiteLLM's reset cycle. -| Parameter | Type | Default | Description | -| ---------------------------------------------------- | ------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `LITELLM_BUDGET_RESET_TRACKER_ENABLED` | boolean | `false` | Enables the background job that monitors soon-to-reset project budget windows. | -| `LITELLM_BUDGET_RESET_TRACKER_SCHEDULE` | string | `"*/10 * * * *"` | Cron schedule (UTC) for the reset-window tracker job. Defaults to every 10 minutes. | -| `LITELLM_BUDGET_RESET_WINDOW_MINUTES` | integer | `15` | Look-ahead window in minutes for detecting project budgets that will reset soon. | -| `LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED` | boolean | `false` | Enables the daily reconciliation job that re-syncs reset state at midnight UTC. | -| `LITELLM_BUDGET_RESET_RECONCILIATION_SCHEDULE` | string | `"10 0 * * *"` | Cron schedule (UTC) for the reset reconciliation job. Must run within `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` of midnight. | -| `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` | integer | `10` | Allowed execution window in minutes after midnight UTC for the reconciliation job. | +| Parameter | Type | Default | Description | +| ---------------------------------------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `LITELLM_BUDGET_RESET_TRACKER_ENABLED` | boolean | `false` | Enables the budget reset-window tracker job. Periodically checks which members have a budget reset approaching within the configured look-ahead window. | +| `LITELLM_BUDGET_RESET_TRACKER_SCHEDULE` | string | `"*/10 * * * *"` | Cron schedule (UTC) for the reset-window tracker job. Defaults to every 10 minutes. | +| `LITELLM_BUDGET_RESET_WINDOW_MINUTES` | integer | `15` | Look-ahead window in minutes. Members whose budget resets within this window are captured by the tracker job. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED` | boolean | `false` | Enables the daily reset reconciliation job. Runs shortly after midnight UTC and reconciles actual budget resets, correcting any mismatches between LiteLLM and CodeMie state. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_SCHEDULE` | string | `"10 0 * * *"` | Cron schedule (UTC) for the reset reconciliation job. Must run within `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` of midnight. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` | integer | `10` | Allowed execution window in minutes after 00:00 UTC. The reconciliation schedule must run within this window — if not, the application will fail to start. | ### LiteLLM Spend Tracking diff --git a/docs/admin/configuration/codemie/project-budget-management.md b/docs/admin/configuration/codemie/project-budget-management.md index 65d6702b..caad6b6e 100644 --- a/docs/admin/configuration/codemie/project-budget-management.md +++ b/docs/admin/configuration/codemie/project-budget-management.md @@ -93,7 +93,7 @@ When a project budget is created, CodeMie automatically distributes the budget e ### Equal allocation (default) -- `max_budget` and `soft_budget` are divided equally among all members using precise decimal arithmetic (9 decimal places). +- `max_budget` and `soft_budget` are divided equally among all members using precise decimal arithmetic. - Values are rounded to the nearest cent for display and enforcement. - If a rounding remainder exists (e.g., `$0.01` that cannot be split evenly), it is assigned to the last member determined by ascending `user_id` sort order. - The total of all member allocations never exceeds the project budget. @@ -174,12 +174,13 @@ Spend data updates on the collect schedule (nightly by default). There is typica ## Background Jobs and Environment Variables -Two background jobs maintain budget state. All jobs use PostgreSQL advisory locks so only one pod runs each job in a multi-replica deployment. +Three background jobs maintain budget state. All jobs use PostgreSQL advisory locks so only one pod runs each job in a multi-replica deployment. | Job | Purpose | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Spend collector** (`litellm_spend_collector`) | Polls LiteLLM for spend data and writes deltas to `project_spend_tracking` | | **Budget reset tracker** (`litellm_budget_reset_tracker`) | Detects budget period rollovers and adjusts spend records accordingly | +| **Budget reset reconciliation** | Runs daily after midnight UTC: verifies all budget resets were applied correctly in both LiteLLM and CodeMie | | **Startup reconciliation** | One-time on pod start: aligns DB state with LiteLLM (predefined budgets, user assignments, project budget assignments) | ### Environment variables @@ -190,6 +191,7 @@ Two background jobs maintain budget state. All jobs use PostgreSQL advisory lock | `LITELLM_SPEND_COLLECTOR_SCHEDULE` | `0 23 * * *` | Cron expression (UTC) for the spend collector. See [API Configuration](../api-configuration) for details. | | `LITELLM_BUDGET_RESET_TRACKER_ENABLED` | `false` | Enable the budget reset tracker background job | | `LITELLM_BUDGET_RESET_TRACKER_SCHEDULE` | `*/10 * * * *` | Cron expression (UTC) for the reset tracker | +| `LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED` | `false` | Enable the daily budget reset reconciliation background job | | `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED` | `false` | Enable one-time startup reconciliation on pod start | | `LLM_PROXY_BUDGET_RECONCILIATION_TIMEOUT_SECONDS` | `600` | Timeout in seconds for the startup reconciliation job | @@ -204,7 +206,7 @@ extraEnv: - name: LITELLM_BUDGET_RESET_TRACKER_ENABLED value: 'true' - name: LITELLM_BUDGET_RESET_TRACKER_SCHEDULE - value: '0 0 * * *' + value: '*/10 * * * *' - name: LLM_PROXY_BUDGET_RECONCILIATION_ENABLED value: 'true' ``` diff --git a/docs/admin/configuration/extensions/litellm-proxy/budget-configuration.md b/docs/admin/configuration/extensions/litellm-proxy/budget-configuration.md index 347d7942..6a8b700e 100644 --- a/docs/admin/configuration/extensions/litellm-proxy/budget-configuration.md +++ b/docs/admin/configuration/extensions/litellm-proxy/budget-configuration.md @@ -93,29 +93,102 @@ The `budget_category` field controls which type of usage the budget applies to: Budget enforcement is disabled by default. To activate it, set the following environment variables in the CodeMie API deployment: -| Variable | Type | Default | Description | -| ----------------------------------- | ------- | ------- | -------------------------------------------------------------------------------- | -| `LLM_PROXY_BUDGET_CHECK_ENABLED` | boolean | `false` | Enables budget limit checking for LLM proxy requests | -| `LLM_PROXY_BUDGET_SYNC_ENABLED` | boolean | `false` | Syncs predefined budgets from `budgets-config.yaml` into the database on startup | -| `LLM_PROXY_BUDGET_BACKFILL_ENABLED` | boolean | `false` | Backfills user budget assignments from LiteLLM on startup for existing users | +| Variable | Type | Default | Description | +| ----------------------------------------- | ------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `LLM_PROXY_BUDGET_CHECK_ENABLED` | boolean | `false` | Enables LLM budget enforcement. When `true`, CodeMie actively enforces spending limits - LLM requests from users or projects that have exceeded their budget are blocked. Also enables budget API routes and background budget maintenance. | +| `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED` | boolean | `false` | Triggers a **one-time** reconciliation job on startup that syncs predefined budgets from `budgets-config.yaml`, backfills user and project budget assignments, and aligns spending state with LiteLLM. Must be disabled after reconciliation completes — it is not intended to stay enabled. | -**In Helm Values** (`values.yaml`): +### How budget enforcement works + +When `LLM_PROXY_BUDGET_CHECK_ENABLED` is enabled, CodeMie registers the budget enforcement provider and enables budget API routes. LLM requests are checked against active budgets - requests from users or projects that have exceeded their `max_budget` are rejected; requests approaching `soft_budget` generate warnings. + +**Predefined budgets from `budgets-config.yaml` are not loaded by `LLM_PROXY_BUDGET_CHECK_ENABLED` alone.** To sync the YAML config into the database and LiteLLM, run the one-time reconciliation by temporarily setting `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED=true`. The reconciliation job runs once after startup and: + +1. Syncs predefined budgets from `budgets-config.yaml` into PostgreSQL and LiteLLM - config is the **source of truth**: existing budgets are created or updated, spending counters are preserved unless `budget_duration` changes +2. Syncs current spending state from LiteLLM back into CodeMie +3. Backfills budget assignments for existing users and projects + +Without running reconciliation, changes to `budgets-config.yaml` never reach the database even after a pod restart. + +### Running reconciliation + +Reconciliation is a one-time operation. Run it whenever you update `budgets-config.yaml` or on initial setup. + +When multiple replicas are running, a PostgreSQL advisory lock ensures only one pod executes the reconciliation — other pods acquire the lock, skip, and log `"Budget reconciliation lock held by another instance, skipping"`. You do not need to scale down to a single replica. + +1. Set `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED: "true"` in `extraEnv`. +2. Apply the Helm values and wait for pods to restart. +3. Confirm reconciliation completed successfully in pod logs. +4. Remove the variable (or set it to `"false"`) and re-apply. + +**In AI/Run CodeMie Backend values** (`values.yaml`) — enable budget enforcement permanently, run reconciliation once: ```yaml -api: - env: - - name: LLM_PROXY_BUDGET_CHECK_ENABLED - value: "true" - - name: LLM_PROXY_BUDGET_SYNC_ENABLED - value: "true" - - name: LLM_PROXY_BUDGET_BACKFILL_ENABLED - value: "true" +extraEnv: + - name: LLM_PROXY_BUDGET_CHECK_ENABLED + value: "true" + # Set to "true" temporarily during reconciliation, then remove + - name: LLM_PROXY_BUDGET_RECONCILIATION_ENABLED + value: "true" ``` -:::warning -`LLM_PROXY_BUDGET_SYNC_ENABLED` must be `true` for predefined budgets from `budgets-config.yaml` to be loaded into the database. Without it, budget definitions in the config file have no effect. +## Spend Tracking Background Jobs + +CodeMie includes background APScheduler jobs that collect spend data from LiteLLM, track upcoming budget resets, and reconcile daily resets. All three jobs are disabled by default and require `LLM_PROXY_ENABLED=true` to function. + +:::warning Prerequisite +All spend tracking jobs require the LiteLLM proxy to be enabled (`LLM_PROXY_ENABLED=true`). If any of these variables is set to `true` while `LLM_PROXY_ENABLED=false`, the scheduler will not start and a warning will be logged. ::: +### Variables + +| Variable | Type | Default | Description | +| ---------------------------------------------------- | ------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `LITELLM_SPEND_COLLECTOR_ENABLED` | boolean | `false` | Enables the nightly spend collector job. Pulls spend data from LiteLLM and inserts rows into the spend tracking table. | +| `LITELLM_SPEND_COLLECTOR_SCHEDULE` | string | `0 23 * * *` | Cron schedule (UTC) for the spend collector. Default: every day at 23:00 UTC. | +| `LITELLM_BUDGET_RESET_TRACKER_ENABLED` | boolean | `false` | Enables the budget reset-window tracker job. Periodically checks which members have a budget reset approaching within the configured look-ahead window. | +| `LITELLM_BUDGET_RESET_TRACKER_SCHEDULE` | string | `*/10 * * * *` | Cron schedule (UTC) for the reset-window tracker. Default: every 10 minutes. | +| `LITELLM_BUDGET_RESET_WINDOW_MINUTES` | integer | `15` | Look-ahead window in minutes. Members whose budget resets within this window are captured by the tracker job. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED` | boolean | `false` | Enables the daily reset reconciliation job. Runs shortly after midnight UTC and reconciles actual budget resets, correcting any mismatches between LiteLLM and CodeMie state. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_SCHEDULE` | string | `10 0 * * *` | Cron schedule (UTC) for the reconciliation job. Must be a daily schedule that falls within the midnight UTC window defined by `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES`. | +| `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` | integer | `10` | Allowed execution window in minutes after 00:00 UTC. The reconciliation schedule must run within this window — if not, the application will fail to start. | + +### How each job works + +**Spend Collector** (`LITELLM_SPEND_COLLECTOR_ENABLED`) + +Runs nightly at 23:00 UTC. Queries LiteLLM for accumulated spend data across applications and inserts the results into the CodeMie spend tracking table. In a multi-replica deployment, an advisory lock ensures only one pod runs the job. + +**Budget Reset Tracker** (`LITELLM_BUDGET_RESET_TRACKER_ENABLED`) + +Runs every 10 minutes. Scans member budget reset windows and records which members are approaching a reset within the next `LITELLM_BUDGET_RESET_WINDOW_MINUTES` minutes. Keeps the budget reset state up to date between nightly reconciliations. + +**Budget Reset Reconciliation** (`LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED`) + +Runs daily at 00:10 UTC (default). After budget periods roll over at midnight, this job verifies that all resets were applied correctly in both LiteLLM and CodeMie. It logs `updated`, `failed`, and `mismatch_warnings` counts on completion. + +:::info Schedule validation +When `LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED=true`, CodeMie validates `LITELLM_BUDGET_RESET_RECONCILIATION_SCHEDULE` at startup. The schedule must be a daily cron expression and must fire within `LITELLM_BUDGET_RESET_RECONCILIATION_WINDOW_MINUTES` minutes of midnight UTC. If validation fails, the application will not start. +::: + +### Enabling all three jobs + +For a complete spend tracking setup, enable all three jobs together: + +**In AI/Run CodeMie Backend values** (`values.yaml`): + +```yaml +extraEnv: + - name: LITELLM_SPEND_COLLECTOR_ENABLED + value: "true" + - name: LITELLM_BUDGET_RESET_TRACKER_ENABLED + value: "true" + - name: LITELLM_BUDGET_RESET_RECONCILIATION_ENABLED + value: "true" +``` + +The three jobs are independent and can be enabled individually, but together they form a complete pipeline: the collector captures historical spend, the tracker monitors upcoming resets, and the reconciliation job corrects any state mismatches after midnight. + ## Customizing Budgets via Helm To override the default budget or add category-specific budgets, mount a custom `budgets-config.yaml` using the Helm chart's `extraVolumeMounts`, `extraVolumes`, and `extraObjects` values. @@ -278,6 +351,18 @@ When `LITELLM_PREMIUM_MODELS_ALIASES` is set to an empty array (`'[]'`), premium The `/spending` endpoint returns an additional `premium_current_spending` field when this feature is enabled, so you can monitor premium model costs separately. +## UI Configuration + +### AI/Run CodeMie UI + +In the `codemie-ui` Helm chart `values.yaml`, set: + +```yaml +viteEnableBudgetManagement: true +``` + +`viteEnableBudgetManagement` enables budget columns and the budget management section on project detail pages. Set it to `false` if your deployment does not use budget tracking. + ## See Also - [LiteLLM Proxy Installation Guide](../../../deployment/extensions/litellm-proxy/index.md) diff --git a/docs/admin/update/release-notes.md b/docs/admin/update/release-notes.md index 2d8b8ab1..0963832e 100644 --- a/docs/admin/update/release-notes.md +++ b/docs/admin/update/release-notes.md @@ -188,9 +188,7 @@ No third-party component updates in this release.

Configuration Changes

-1. **Update LiteLLM budget env vars** — remove `LITELLM_SPEND_COLLECTOR_SCHEDULE` and set `LLM_PROXY_BUDGET_BACKFILL_ENABLED: "true"`. See [Budget Configuration](../configuration/extensions/litellm-proxy/budget-configuration.md). - -2. **One-time reconciliation via `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED`** +1. **One-time reconciliation via `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED`** :::warning One-time operation Enable only on a **single API replica**, wait for reconciliation to complete (check pod logs), then disable and scale replicas back. @@ -220,7 +218,15 @@ No third-party component updates in this release.

Configuration Changes

-No breaking configuration changes were introduced in this release. +1. **`LLM_PROXY_BUDGET_SYNC_ENABLED` and `LLM_PROXY_BUDGET_BACKFILL_ENABLED` removed** — both variables are replaced by `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED`. + + :::tip Configuration housekeeping + The following variables are no longer supported and should be removed from your `extraEnv`: + - `LLM_PROXY_BUDGET_SYNC_ENABLED` + - `LLM_PROXY_BUDGET_BACKFILL_ENABLED` + + Use `LLM_PROXY_BUDGET_RECONCILIATION_ENABLED` instead. + :::

Known Issues