Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions src/codex/quota.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,17 @@ export function isCompleteCodexQuotaRecoverySnapshot(
// Recovery still fails closed on MISSING EVIDENCE — a credits-only or windowless payload
// carries no usage reading at all and must never clear a cooldown. What it does not do is
// fail closed on an unfamiliar plan NAME, which only ever meant "cooled forever".
const required = codexQuotaWindowForPlan(plan) === "monthly"
? quota.monthlyPercent
: quota.weeklyPercent;
return typeof required === "number" && Number.isFinite(required);
//
// The parser classifies windows by DURATION, not by plan name: a Team response whose
// primary window is explicitly monthly parses to monthlyPercent only (no secondary
// window exists), so requiring weeklyPercent because the plan is not go/free would
// strand exactly those accounts until their predicted expiry. Accept whichever window(s)
// the parser actually wrote; Go/Free never carry a weekly value, so monthly-only is
// required there.
if (codexQuotaWindowForPlan(plan) === "monthly") {
return typeof quota.monthlyPercent === "number" && Number.isFinite(quota.monthlyPercent);
}
return hasKnownQuotaValue(quota);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Distinguish monthly primaries from tertiary-only snapshots

For a weekly plan such as Team, parseUsageQuota() can produce monthlyPercent from a tertiary window even when the weekly primary/secondary window is entirely absent; tests/codex-routing.test.ts:1199-1205 explicitly preserves that tertiary-only representation. This fallback now treats such a response as complete recovery evidence, so a low optional 30-day reading can clear a reset-derived shared cooldown without any fresh reading for the weekly quota that caused it, immediately routing traffic back to an account that may still be exhausted. Preserve whether the monthly value came from an explicitly monthly primary and accept monthly-only recovery for weekly-named plans only in that case, rather than accepting every parsed monthly value.

Useful? React with 👍 / 👎.

}

export function normalizeUsagePercent(value: unknown): number | undefined {
Expand Down
22 changes: 20 additions & 2 deletions src/codex/routing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ export type CodexQuotaRecoveryProbeClaim = {
leaseId: string;
cooldownGeneration: number;
credentialGeneration: number;
/** Claim-time `replacedAt`; unchanged after a probe-owned refresh, stamped on external replacement. */
credentialReplacedAt?: number;
};

export type CodexQuotaRecoveryProbeProof = {
Expand Down Expand Up @@ -439,6 +441,7 @@ export function claimDueCodexQuotaRecoveryProbes(
scope?: CodexQuotaScope;
health: CodexUpstreamHealth;
credentialGeneration: number;
credentialReplacedAt?: number;
order: number;
}> = [];
for (const [order, account] of (config.codexAccounts ?? []).entries()) {
Expand Down Expand Up @@ -467,6 +470,7 @@ export function claimDueCodexQuotaRecoveryProbes(
...(candidate.scope ? { scope: candidate.scope } : {}),
health: candidate.health,
credentialGeneration: record.generation,
...(record.replacedAt !== undefined ? { credentialReplacedAt: record.replacedAt } : {}),
order,
});
}
Expand All @@ -491,6 +495,9 @@ export function claimDueCodexQuotaRecoveryProbes(
leaseId,
cooldownGeneration: candidate.health.cooldownGeneration ?? 0,
credentialGeneration: candidate.credentialGeneration,
...(candidate.credentialReplacedAt !== undefined
? { credentialReplacedAt: candidate.credentialReplacedAt }
: {}),
};
});
}
Expand All @@ -506,10 +513,21 @@ export function settleCodexQuotaRecoveryProbe(
? scopedHealthFor(claim.accountId, claim.scope)
: upstreamHealth.get(claim.accountId);
if (!health || health.probeLeaseId !== claim.leaseId) return false;
const currentRecord = readCodexAccountRecord(claim.accountId);
const proofGeneration = proof.credentialGeneration;
// A probe-owned token refresh (getValidCodexToken) advances the credential generation by
// exactly one while preserving `replacedAt`; an external credential replacement bumps the
// generation too but stamps a fresh `replacedAt`. Accept the +1 transition only when the
// claim-time lineage is intact AND the generation the fresh quota was proven under is live.
const generationFenced = proofGeneration !== undefined
&& (proofGeneration === claim.credentialGeneration
? isCodexAccountGenerationLive(claim.accountId, proofGeneration)
: proofGeneration === claim.credentialGeneration + 1
&& currentRecord?.replacedAt === claim.credentialReplacedAt
&& isCodexAccountGenerationLive(claim.accountId, proofGeneration));
Comment on lines +525 to +527

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept all later generations from the same refresh lineage

When another caller such as the token guardian is already refreshing this cooled account, the probe can join that getValidCodexToken() flight; the owner persists generation N+1, then the same-account join path at src/codex/account-store.ts:406-422 persists the returned credential again and reports N+2. The fresh WHAM result is therefore proven under a live, unchanged-replacedAt generation but this exact +1 check rejects it, leaving the cooldown in place for at least another probe interval. Since replacedAt already distinguishes external replacement, accept any later live generation with the same claim-time lineage (or avoid the redundant same-account generation bump).

Useful? React with 👍 / 👎.

const fenced = (health.cooldownGeneration ?? 0) === claim.cooldownGeneration
&& (health.probeLeaseGeneration ?? 0) === claim.cooldownGeneration
&& claim.credentialGeneration === proof.credentialGeneration
&& isCodexAccountGenerationLive(claim.accountId, claim.credentialGeneration);
&& generationFenced;
if (!recovered || !fenced) {
const released = withProbeLeaseReleased(health, now);
if (claim.scope) setScopedHealth(claim.accountId, claim.scope, released);
Expand Down
64 changes: 59 additions & 5 deletions tests/codex-cooldown-recovery.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,55 @@ describe("Codex cooldown recovery worker", () => {
expect(routed).toEqual(["b", "b"]);
});

test("recovers a Team account from a duration-classified monthly snapshot", async () => {
// WHAM can legitimately return only an explicitly monthly primary window for a Team
// plan (30.4-day window, no secondary). parseUsageQuota then writes monthlyPercent only,
// so recovery must accept the window the parser actually classified instead of demanding
// a weekly reading because the plan name is not "go"/"free".
const config = makeConfig(["a"]);
saveCredential("a");
cool(config, "a");
globalThis.fetch = async () => usageResponse(0, {
plan_type: "team",
rate_limit: {
primary_window: { used_percent: 6, reset_at: 1_900_000_000, limit_window_seconds: 2_628_000 },
secondary_window: null,
tertiary_window: null,
},
rate_limit_reset_credits: { available_count: 0 },
});
await runCodexCooldownRecoveryProbes(config, due());
expect(getCodexQuotaHealthSnapshot("a", "shared", due() + 1)).toBeNull();
});

test("recovers when the probe's own token refresh advances the credential generation", async () => {
// A near-expiry access token makes getValidCodexToken() refresh it inside the probe
// fetch, bumping the credential generation from 1 to 2 before WHAM completes. The fresh
// quota is proven under the new live generation, so settling against the claim-time
// generation must accept this probe's own refresh, not treat it as a replacement.
const config = makeConfig(["a"]);
saveCodexAccountCredential("a", {
accessToken: "access-a",
refreshToken: "refresh-a",
expiresAt: Date.now() + 30_000,
chatgptAccountId: "acct-a",
});
cool(config, "a");
globalThis.fetch = async input => {
const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;
if (url.includes("/oauth/token")) {
return new Response(JSON.stringify({
access_token: "access-a-2",
refresh_token: "refresh-a-2",
expires_in: 3600,
}), { status: 200, headers: { "Content-Type": "application/json" } });
}
return usageResponse(12);
};
await runCodexCooldownRecoveryProbes(config, due());
expect(getCodexQuotaHealthSnapshot("a", "shared", due() + 1)).toBeNull();
});

test.each([
["still exhausted", () => usageResponse(100)],
["credits only", () => usageResponse(0, { plan_type: "team", rate_limit_reset_credits: { available_count: 1 } })],
Expand Down Expand Up @@ -290,13 +339,18 @@ describe("Codex cooldown recovery worker", () => {

for (const plan of snapshotPlans) {
const monthly = codexQuotaWindowForPlan(plan) === "monthly";
const filled = monthly ? { monthlyPercent: 12 } : { weeklyPercent: 12 };
const empty = monthly ? { weeklyPercent: 12 } : { monthlyPercent: 12 };
expect(isCompleteCodexQuotaRecoverySnapshot(filled, plan)).toBe(true);
// The other window is not evidence for this plan, in either direction.
expect(isCompleteCodexQuotaRecoverySnapshot(empty, plan)).toBe(false);
// The parser classifies windows by duration, so a weekly-billed plan can carry a
// monthly-only reading (30-day primary, no secondary). Any window the parser actually
// wrote is evidence; only Go/Free never carry a weekly value.
expect(isCompleteCodexQuotaRecoverySnapshot({ weeklyPercent: 12 }, plan)).toBe(!monthly);
expect(isCompleteCodexQuotaRecoverySnapshot({ monthlyPercent: 12 }, plan)).toBe(true);
}

// Monthly-billed Go/Free parse to monthlyPercent only; a weekly-only reading is not
// evidence for them.
expect(isCompleteCodexQuotaRecoverySnapshot({ weeklyPercent: 12 }, "go")).toBe(false);
expect(isCompleteCodexQuotaRecoverySnapshot({ weeklyPercent: 12 }, "free")).toBe(false);

// Absent plan follows the parser's weekly default.
expect(isCompleteCodexQuotaRecoverySnapshot({ weeklyPercent: 12 }, undefined)).toBe(true);
// Missing EVIDENCE still fails closed — that is the guard that matters.
Expand Down
Loading