Fix large-file ingest (500 → works) + SDK max_file_size guard#113
Conversation
isValidBase64 ran a regex whose '(?:...){4})*' quantifier overflowed V8's call stack on base64 strings >~1MB, so ingesting files >~750KB returned 500 during validation, before any DB work. Skip the regex for strings over 1MB and rely on the native canonical round-trip check (Buffer.from(s,'base64').toString('base64') === s), which never overflows.
Also bump just-bash ^2.14.5 -> ^3.0.1 (3.0.0 stdin byte/utf8 overhaul makes cat/pipes/append/concat byte-correct on binary) and document real ingest size limits in the api skill.
Client(max_file_size=...) (default 64 MiB) rejects oversized files on every write path (ingest_files, fs.write, fs.write_files) BEFORE any base64 encoding or network I/O, raising ValidationError(code='EFILE_TOO_LARGE') naming each offending path/size. Threaded to every Sandbox the client creates or attaches; set 0 to disable. Bumps the Python SDK to 0.3.0 and updates the py-sdk skill docs (client/errors/sandbox/SKILL).
|
Warning Review limit reached
More reviews will be available in 36 minutes and 23 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (15)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@codex review |
Root README: add MAX_REQUEST_BODY_BYTES, MAX_INGEST_BYTES, MAX_INGEST_FILES, MAX_INGEST_PATHS_CONCURRENCY to the env-var table (the ceilings governing ingest). SDK README: document Client(max_file_size=...) (default 64 MiB, client-side, 0 disables) and the client-side EFILE_TOO_LARGE ValidationError.
|
Codex Review: Didn't find any major issues. Breezy! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Summary
Fixes large-file ingest (which returned 500 for files larger than ~750 KB) and adds a client-side per-file size guard to the Python SDK. Verified end-to-end by ingesting the full
just-bashrepo (1603 files / 20.6 MB, including two ~4 MB binaries) byte-exact.What changed
Server —
sql-fs-api(fix, patch)isValidBase64ran a structural regex whose(?:[A-Za-z0-9+/]{4})*quantifier overflowed V8's call stack (RangeError: Maximum call stack size exceeded) on base64 strings beyond ~1 MB — so ingesting any file over ~750 KB failed with500 INTERNAL_ERRORduring request validation, before any DB work. The regex is now skipped for strings over 1 MB, relying on the native canonical round-trip check (Buffer.from(s, "base64").toString("base64") === s), which never overflows.just-bash^2.14.5→^3.0.1. The 3.0.0 stdin byte/utf8 overhaul makescat, pipes (|), append (>>), and concatenation byte-correct on binary content (2.14.5 re-encoded binary as UTF-8). FS-level ops (cp/mv/rm) and content-addressable storage were always byte-correct.Python SDK —
sql-fs-sdk0.2.5 → 0.3.0 (feat)Client(max_file_size=...)(default 64 MiB) enforces a per-file ceiling on every write path (ingest_files,fs.write,fs.write_files) before any content is base64-encoded or sent. Oversized files raiseValidationError(code="EFILE_TOO_LARGE")(withstatus=None) naming each offending path/size. Threaded to everySandbox; setmax_file_size=0to disable.Docs
apiandpy-sdkskills: real ingest size limits, the large-file fix, and the newmax_file_sizeknob /EFILE_TOO_LARGEerror.Testing
just-bashfolder (1603 files, 20.6 MB) → all stored,md5summatches for sampled files including the two ~4 MB binaries.max_file_size: rejects 65 MB, accepts 64 MB, honors custom limits,0disables.pnpm typecheck+ pre-commit lint/typecheck hooks pass.Notes
just-bashbump is intentionally without a changeset — it rides into the next release without its own changelog entry.head -c N/tail -c Nreading from a file argument still decode binary as UTF-8 and mis-count bytes (piping in,cat bin | head -c N, is byte-clean). Tracked separately.