Skip to content

hotfix: stop per-request OOM on location export endpoints - #761

Closed
jirhiker wants to merge 121 commits into
productionfrom
claude/mystifying-panini-4ded82
Closed

hotfix: stop per-request OOM on location export endpoints#761
jirhiker wants to merge 121 commits into
productionfrom
claude/mystifying-panini-4ded82

Conversation

@jirhiker

@jirhiker jirhiker commented Jul 6, 2026

Copy link
Copy Markdown
Member

Problem

App Engine terminated the process mid-request: "the process that handled this request was found to be using too much memory and was terminated." Already on F4 (1GB), 1 worker — so it's a per-request memory spike, not an undersized instance.

Two endpoints loaded the entire SampleLocation table into RAM on a single request:

  • GET /location/feature_collection.all() the whole table, then built a second list of dicts (double memory).
  • GET /location/shapefile.all() the whole table and wrote locations.shp/.zip to CWD. On App Engine the app dir is read-only; only /tmp is writable and it's RAM-backed (counts against instance memory).

Both were async def doing blocking sync DB/file work, stalling the event loop.

Fix

api/base.py

  • feature_collectionStreamingResponse, emits GeoJSON row-by-row via yield_per(1000). Whole table never buffered. Output shape unchanged.
  • shapefile → writes to tempfile.mkdtemp() (not read-only CWD), streams rows with yield_per, cleans up the temp dir via BackgroundTask after send.
  • Both async defdef (blocking work runs in threadpool).

db/__init__.py

  • Production async engine echo=TrueFalse (was logging every statement).
  • Bounded pools on both engines: pool_size=5, max_overflow=2, pool_recycle=1800, pool_pre_ping=True.

Not in this PR (ops / follow-up)

  • app.yaml entrypoint updated locally with --max-requests 200 --max-requests-jitter 40 --timeout 120 (worker recycling as a residual-leak safety net). Not committed — app.yaml is untracked and contains secrets.
  • ⚠️ app.yaml holds a plaintext DB password + GCP service-account private key. Move to Secret Manager and rotate those credentials.

Testing

  • Syntax verified. Not load-tested here — verify a feature_collection / shapefile call against the full table post-deploy.

🤖 Generated with Claude Code

jirhiker and others added 30 commits June 18, 2025 11:11
WIP: This is a POC replacement for bureau geospatial databases
Numbers are limited to 10 digits.
This reduces the risk of catastrophic backtracking.
Bumps [alembic](https://github.com/sqlalchemy/alembic) from 1.16.1 to 1.16.2.
- [Release notes](https://github.com/sqlalchemy/alembic/releases)
- [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES)
- [Commits](https://github.com/sqlalchemy/alembic/commits)

---
updated-dependencies:
- dependency-name: alembic
  dependency-version: 1.16.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 3.2.2 to 3.2.3.
- [Changelog](https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst)
- [Commits](python-greenlet/greenlet@3.2.2...3.2.3)

---
updated-dependencies:
- dependency-name: greenlet
  dependency-version: 3.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
jirhiker and others added 27 commits July 1, 2025 23:56
Bumps [pillow](https://github.com/python-pillow/Pillow) from 11.2.1 to 11.3.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@11.2.1...11.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 11.3.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
…-11.3.0

Bump pillow from 11.2.1 to 11.3.0
…-1.16.2

Bump alembic from 1.16.1 to 1.16.2
…-0.115.14

Bump fastapi from 0.115.12 to 0.115.14
…-0.35.0

Bump uvicorn from 0.34.3 to 0.35.0
…-pagination-0.13.3

Bump fastapi-pagination from 0.13.2 to 0.13.3
…t-3.2.3

Bump greenlet from 3.2.2 to 3.2.3
…n permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
The /location/feature_collection and /location/shapefile endpoints loaded
the entire SampleLocation table into memory on a single request, spiking
memory enough for App Engine to terminate the process.

- feature_collection: stream GeoJSON row-by-row via StreamingResponse +
  yield_per instead of building two full in-memory lists. Output shape
  unchanged.
- shapefile: write to a tempfile.mkdtemp() dir (App Engine app dir is
  read-only; /tmp is RAM-backed) with yield_per streaming and clean up
  the temp dir via BackgroundTask after send.
- Both endpoints changed from async def to def so blocking sync DB/file
  work runs in the threadpool instead of stalling the event loop.
- db: disable echo on the production async engine and add bounded pools
  (pool_size, max_overflow, pool_recycle, pool_pre_ping) on both engines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jirhiker
jirhiker changed the base branch from geoserver-iac to production July 6, 2026 20:18
@jirhiker

jirhiker commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Superseded by #762 — clean hotfix cherry-picked onto production (production has a refactored codebase; the original branch targeted the geoserver-iac line and produced a 100-commit diff against production).

@jirhiker jirhiker closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant