hotfix: stop per-request OOM on location export endpoints - #761
Closed
jirhiker wants to merge 121 commits into
Closed
hotfix: stop per-request OOM on location export endpoints#761jirhiker wants to merge 121 commits into
jirhiker wants to merge 121 commits into
Conversation
WIP: This is a POC replacement for bureau geospatial databases
Numbers are limited to 10 digits. This reduces the risk of catastrophic backtracking.
Bumps [alembic](https://github.com/sqlalchemy/alembic) from 1.16.1 to 1.16.2. - [Release notes](https://github.com/sqlalchemy/alembic/releases) - [Changelog](https://github.com/sqlalchemy/alembic/blob/main/CHANGES) - [Commits](https://github.com/sqlalchemy/alembic/commits) --- updated-dependencies: - dependency-name: alembic dependency-version: 1.16.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [greenlet](https://github.com/python-greenlet/greenlet) from 3.2.2 to 3.2.3. - [Changelog](https://github.com/python-greenlet/greenlet/blob/master/CHANGES.rst) - [Commits](python-greenlet/greenlet@3.2.2...3.2.3) --- updated-dependencies: - dependency-name: greenlet dependency-version: 3.2.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pillow](https://github.com/python-pillow/Pillow) from 11.2.1 to 11.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@11.2.1...11.3.0) --- updated-dependencies: - dependency-name: pillow dependency-version: 11.3.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…-11.3.0 Bump pillow from 11.2.1 to 11.3.0
…-1.16.2 Bump alembic from 1.16.1 to 1.16.2
…-0.115.14 Bump fastapi from 0.115.12 to 0.115.14
…-0.35.0 Bump uvicorn from 0.34.3 to 0.35.0
…-pagination-0.13.3 Bump fastapi-pagination from 0.13.2 to 0.13.3
…t-3.2.3 Bump greenlet from 3.2.2 to 3.2.3
…n permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
The /location/feature_collection and /location/shapefile endpoints loaded the entire SampleLocation table into memory on a single request, spiking memory enough for App Engine to terminate the process. - feature_collection: stream GeoJSON row-by-row via StreamingResponse + yield_per instead of building two full in-memory lists. Output shape unchanged. - shapefile: write to a tempfile.mkdtemp() dir (App Engine app dir is read-only; /tmp is RAM-backed) with yield_per streaming and clean up the temp dir via BackgroundTask after send. - Both endpoints changed from async def to def so blocking sync DB/file work runs in the threadpool instead of stalling the event loop. - db: disable echo on the production async engine and add bounded pools (pool_size, max_overflow, pool_recycle, pool_pre_ping) on both engines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
|
Superseded by #762 — clean hotfix cherry-picked onto production (production has a refactored codebase; the original branch targeted the geoserver-iac line and produced a 100-commit diff against production). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
App Engine terminated the process mid-request: "the process that handled this request was found to be using too much memory and was terminated." Already on F4 (1GB), 1 worker — so it's a per-request memory spike, not an undersized instance.
Two endpoints loaded the entire
SampleLocationtable into RAM on a single request:GET /location/feature_collection—.all()the whole table, then built a second list of dicts (double memory).GET /location/shapefile—.all()the whole table and wrotelocations.shp/.zipto CWD. On App Engine the app dir is read-only; only/tmpis writable and it's RAM-backed (counts against instance memory).Both were
async defdoing blocking sync DB/file work, stalling the event loop.Fix
api/base.pyfeature_collection→StreamingResponse, emits GeoJSON row-by-row viayield_per(1000). Whole table never buffered. Output shape unchanged.shapefile→ writes totempfile.mkdtemp()(not read-only CWD), streams rows withyield_per, cleans up the temp dir viaBackgroundTaskafter send.async def→def(blocking work runs in threadpool).db/__init__.pyecho=True→False(was logging every statement).pool_size=5, max_overflow=2, pool_recycle=1800, pool_pre_ping=True.Not in this PR (ops / follow-up)
app.yamlentrypoint updated locally with--max-requests 200 --max-requests-jitter 40 --timeout 120(worker recycling as a residual-leak safety net). Not committed —app.yamlis untracked and contains secrets.app.yamlholds a plaintext DB password + GCP service-account private key. Move to Secret Manager and rotate those credentials.Testing
feature_collection/shapefilecall against the full table post-deploy.🤖 Generated with Claude Code