Skip to content

[Outage] files.upload → 503 Service Unavailable #2489

@prashant-kikani

Description

@prashant-kikani

Originally posted in developer forum: https://discuss.ai.google.dev/t/outage-files-upload-503-service-unavailabl/146858

TL;DR

The Gemini Files API (POST /upload/v1beta/files) has been returning HTTP 503 with an empty error body for at least 2 hours, while models.generateContent continues to work normally. The failure is reproducible across two independent API keys (different GCP projects), multiple file types (MP3, PNG), and trivially small file sizes (~129 B PNG, ~4.5 KB MP3). Our production traffic from asia-northeast1 is affected.
Error shape (identical for every request)

{"error":{"message":"","code":503,"status":"Service Unavailable"}}

Note the empty message field — this makes it hard to tell whether the cause is a backend incident,
an undocumented quota, or something else. A non-empty message would help a lot.

Timeline (UTC)

┌──────────────────┬────────────────────────────────────────────────────────────────────────────┐
│ Time │ Event │
├──────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ 2026-05-21 05:15 │ First 503 in our logs (intermittent — some uploads still succeed) │
├──────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ 2026-05-21 06:43 │ Last successful files.upload in our logs. Goes to a hard outage from here. │
├──────────────────┼────────────────────────────────────────────────────────────────────────────┤
│ 2026-05-21 07:15 │ Still 503 on every probe at the time of writing │
└──────────────────┴────────────────────────────────────────────────────────────────────────────┘

In the 05:15–07:15 UTC window we recorded 702 distinct upload failures spanning 84 unique workloads
and 11 Cloud Run instances from one project, plus an independent project (different API key)
producing the same 503.

Scope — what works, what doesn't

Endpoint: models.generateContent (gemini-2.5-flash, text prompt)
SDK call: genAI.models.generateContent(...)
Result: OK (~2.9s)
────────────────────────────────────────
Endpoint: files.upload (PNG, 129 B, image/png)
SDK call: genAI.files.upload(...)
Result: 503 in ~1.1s
────────────────────────────────────────
Endpoint: files.upload (MP3, 4.5 KB, audio/mpeg)
SDK call: genAI.files.upload(...)
Result: 503 after ~81s (SDK appears to be doing internal retries against the resumable-upload
protocol before surfacing the error)
────────────────────────────────────────
Endpoint: Same probe with a second API key from a different GCP project
SDK call: same call
Result: 503, identical body

Everything points to a backend-side issue with the Files API specifically.

Environment

  • SDK: @google/genai 1.43.0 (Node.js 24)
  • Region the requests originate from: asia-northeast1 (also reproduced from a developer machine in
    Korea)
  • Two independent GCP projects, two API keys created in different consoles — both fail identically.

Minimal reproducer (Node + @google/genai)

import { GoogleGenAI } from "@google/genai";

const genAI = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! });
const t0 = Date.now();
try {
const r = await genAI.files.upload({
file: "probe.png", // any small file
config: { mimeType: "image/png" },
});
console.log("OK", Date.now() - t0, r.name);
} catch (err: any) {
console.log("FAIL", Date.now() - t0, JSON.stringify(err));
}

Output right now (truncated):
FAIL 1082 {"error":{"message":"","code":503,"status":"Service Unavailable"}}

curl reproducer

KEY=YOUR_KEY
curl -X POST
"https://generativelanguage.googleapis.com/upload/v1beta/files?uploadType=media&key=$KEY"
-H "Content-Type: image/png"
--data-binary @probe.png -i

Returns HTTP/2 503 with the JSON body shown above.

Ask

  1. Can someone confirm an incident on the Files API and post the status page link? I don't see a
    corresponding entry on the public Google Cloud / AI Studio status pages yet.
  2. Any ETA for recovery would help us decide whether to drain queues vs keep retrying.
  3. Please consider returning a non-empty message field on these 503s — even a generic "backend
    unavailable, retry with exponential backoff" would be vastly better than empty.

Happy to provide more logs, request IDs, or run additional probes on request. Thanks!

Metadata

Metadata

Assignees

Labels

priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions