Skip to content

Commit 4f37faa

Browse files
committed
Add support for GCS-backed plot uploads and update documentation
- Configure GCS as the default `IMAGE_UPLOAD_BACKEND` in `Dockerfile`, with fallback to S3 if overridden. - Enhance `pyproject.toml` to include `gcs` extra for `lf_toolkit`. - Update `CLAUDE.md` to reflect GCS integration and backend switching details. - Clarify environment variable requirements for both GCS and S3 upload options.
1 parent 7f93bd3 commit 4f37faa

3 files changed

Lines changed: 18 additions & 4 deletions

File tree

CLAUDE.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ All source lives in `evaluation_function/`:
99
| File | Role |
1010
|------|------|
1111
| `main.py` | IPC server entry point; registers `evaluation_function` and `preview_function` with lf_toolkit |
12-
| `evaluation.py` | Core evaluation pipeline: security check → subprocess execution → output comparison → S3 plot upload → structured feedback |
12+
| `evaluation.py` | Core evaluation pipeline: security check → subprocess execution → output comparison → plot upload (GCS/S3 via lf_toolkit) → structured feedback |
1313
| `preview.py` | AST-based pre-execution security validator (`_SecurityVisitor`) |
1414
| `dev.py` | CLI wrapper for local manual testing |
1515

@@ -20,7 +20,7 @@ All source lives in `evaluation_function/`:
2020
- **`demo`**: execute code with no stdin; return stdout/plots as `output` feedback (no pass/fail)
2121
- **`io_test`**: for each test in `params["tests"]`, execute with `test["input"]` as stdin and compare stdout against `test["expected_output"]`; upload matplotlib plots on pass or fail
2222
- **`unit_test`**: append `params["test_code"]` + unit-runner harness to student code; execute once; parse JSON results; supports plain `test_*` functions, `unittest.TestCase` subclasses, and Hypothesis-based tests
23-
3. Upload any captured matplotlib figures to S3 (`_UPLOAD_FOLDER = "evaluatePython"`)
23+
3. Upload any captured matplotlib figures via `lf_toolkit` `upload_image` (`_UPLOAD_FOLDER = "evaluatePython"`); backend is GCS or S3 per `IMAGE_UPLOAD_BACKEND`
2424
4. Return a `Result` with feedback tags: `pass`, `fail`, `hidden_fail`, `error`, `output`, `summary`
2525

2626
### Request shape
@@ -151,7 +151,9 @@ CI runs on Python 3.12 and uploads JUnit XML results (`.github/workflows/test-li
151151
| `FUNCTION_ARGS` | `-m,evaluation_function.main` | lf_toolkit runner |
152152
| `FUNCTION_RPC_TRANSPORT` | `stdio` | shimmy↔worker transport (stdio so it survives the sandbox mount namespace) |
153153
| `LOG_LEVEL` | `debug` | Logging verbosity |
154-
| `AWS_*` / boto3 credentials | Runtime env | Required for S3 plot uploads |
154+
| `IMAGE_UPLOAD_BACKEND` | `gcs` | Plot upload backend in lf_toolkit (`gcs` set in Dockerfile; override to `s3` on the service to use AWS) |
155+
| `GCS_BUCKET` | Runtime env | Target bucket for matplotlib plot uploads; set per-environment on the Cloud Run service. Auth is via the runtime service account (ADC) — no keys |
156+
| `AWS_*` / `S3_BUCKET_URI` | Runtime env | Only for the legacy S3 plot-upload backend (`IMAGE_UPLOAD_BACKEND=s3`) |
155157
| `SANDBOX_ENABLED` | `true` | Wrap the worker in shimmy's nsjail sandbox (needs `--privileged` at run time) |
156158
| `SANDBOX_SECCOMP` | `true` | nsjail seccomp syscall filter |
157159
| `SANDBOX_RO_BINDS` | `/usr:/lib:/lib64:/bin:/sbin:/etc:/app` | Read-only bind mounts visible inside the jail |

Dockerfile

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ ENV FUNCTION_RPC_TRANSPORT="stdio"
4949
# needs CAP_SYS_ADMIN for unshare(CLONE_NEWNS). Running as uid 0, nsjail's "auto"
5050
# userns handling then drops CLONE_NEWUSER (a nested userns' unprivileged gid_map
5151
# write fails); --user still drops the worker to nobody. Network stays up so
52-
# matplotlib plots can be uploaded to S3 via boto3.
52+
# matplotlib plots can be uploaded to object storage (see below).
5353
#
5454
# SANDBOX_RO_BINDS / _RW_BINDS: shimmy splits these env vars on COMMA. Bind "/"
5555
# read-only (whole rootfs -- arch-independent, where an explicit list would need
@@ -61,6 +61,13 @@ ENV FUNCTION_RPC_TRANSPORT="stdio"
6161
# creation on some hosts ("pthread_create ... Invalid argument"); the mount and
6262
# user namespaces still isolate the filesystem and privileges.
6363
#
64+
# Network stays up so matplotlib plots (see evaluation.py::_upload_plots) can be
65+
# pushed to object storage. On GCP we use lf_toolkit's GCS backend
66+
# (IMAGE_UPLOAD_BACKEND=gcs): the worker authenticates with Application Default
67+
# Credentials via the Cloud Run runtime service account -- no static keys -- and
68+
# needs GCS_BUCKET set on the service (staging/prod differ). To fall back to S3,
69+
# override IMAGE_UPLOAD_BACKEND=s3 on the service and set S3_BUCKET_URI / AWS_*.
70+
#
6471
# No seccomp (nsjail has no built-in default policy; the fixed shimmy takes a
6572
# kafel policy via SANDBOX_SECCOMP_STRING / _POLICY_FILE if wanted) and no
6673
# rlimits (the RPC worker is long-lived and shared; per-run limits are the
@@ -70,4 +77,8 @@ ENV SANDBOX_ENABLED="true" \
7077
SANDBOX_RW_BINDS="/tmp" \
7178
SANDBOX_DISABLE_CLONE_NEWPID="true"
7279

80+
# Plot upload backend (lf_toolkit). GCS_BUCKET is supplied per-environment on the
81+
# Cloud Run service.
82+
ENV IMAGE_UPLOAD_BACKEND="gcs"
83+
7384
ENV LOG_LEVEL="debug"

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ python = ">=3.11,<4.0"
1515
typing_extensions = "^4.12.2"
1616
lf_toolkit = { git = "https://github.com/lambda-feedback/toolkit-python.git", branch = "main", extras = [
1717
"ipc",
18+
"gcs",
1819
] }
1920
numpy = ">=1.24"
2021
pandas = ">=2.0"

0 commit comments

Comments
 (0)