Skip to content

feat(run): ck run on the file backend (LAB-5118) - #2

Merged
27Bslash6 merged 16 commits into
mainfrom
lab-5118-ck-run-file-backend
Oct 3, 2026
Merged

27Bslash6 merged 16 commits into
mainfrom
lab-5118-ck-run-file-backend

Conversation

@27Bslash6

@27Bslash6 27Bslash6 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Adds ck run on the file backend. It wraps a command and caches its stdout when the command exits 0.

ck requires cachekit-rs 0.9. In 0.8, FileBackend::build read every entry in the cache directory on each start, looking for orphaned temp files, and against 10,000 entries that put ck over its 5 ms hit-path budget. 0.9.0 moves that sweep to a backend's first set, so a hit no longer pays it: on the same Linux host, the added time drops from 3.8 ms to 0.5 ms. 0.9.0 also requires a raw master key of exactly 32 bytes, so ck now checks for exactly 32 itself.

What ck run does

  • Freshness is the reader's call. The encrypted envelope stores {stdout, stored_at}. Each call judges age on its own clock against its own --ttl and --stale, so lowering --stale takes effect on the next call. The backend TTL is the writer's ttl + stale and only collects garbage.
  • Keys. ns:ck-run:v1:<64 hex> is a keyed BLAKE2b-256 over the length-prefixed --scope, then each argv element. The hash key comes from cachekit-core's derive_domain_key(master, "cache_keys", tenant), which is the protocol's cache_key_salt. The failure marker sits at the value key plus :neg. Both use the same EncryptionLayer, so the AAD binds each one to its own key.
  • Failure marker. It stores {exit, failures, retry_at}. Backoff is min(30 s × 2^(n−1), 15 min), and the marker's backend TTL is (retry_at − now) + 15 min. A marker is active only while now < retry_at ≤ now + 15 min. A fresh value always wins, and the marker is read only when no fresh value exists. Only ck's own spawn failure (exit 127 or 126) leaves the marker untouched.
  • Fill lock. An exclusive flock on ~/.cache/ck/locks/<hash>, opened O_CLOEXEC and never unlinked. A contended call serves stale output when it has some, and otherwise waits. Entries live in ~/.cache/ck/data.
  • File key. Without CACHEKIT_MASTER_KEY, ck creates ~/.config/ck/file.key on first use. It writes the key under a unique temp name with O_EXCL, mode 0600, and fsyncs it. Then it link()s the temp to file.key, fsyncs the directory, and removes the temp name. A key file that is not 64 hex characters, not owned by the user, or not mode 0600 exits 125 and is left alone.
  • stdin. Only /dev/null (or a closed fd 0) is cached. A terminal, pipe, file or socket runs the command uncached with one warning that names < /dev/null.
  • Child environment. Every CACHEKIT_* variable is removed before the command starts.
  • Signals. SIGTERM and SIGHUP are forwarded to the child; SIGINT and SIGQUIT are not. A run counts as interrupted only when ck itself received one of these four. The signal is recorded inside its handler, and ck's helper threads block all four, so the handler always runs on the main thread. If the command dies of a handled signal before ck's handler has run, ck waits up to 50 ms for it, because some kernels signal a process group one member at a time. A command that dies of a signal on its own is an ordinary failure, and stale output is still served. Every later SIGTERM and SIGHUP is still forwarded, so a command that traps the first one can still be stopped. An interrupted run discards the output ck was holding back, stores nothing, and exits 128 + the number of a signal it received (which of several close together is unspecified): once the command itself has exited (a background job still holding its stdout does not keep ck alive), or at once when no command is running. Output already written, by an uncached call or past the cap, stays written; an interrupted run streamed past the cap can lose the tail ck was still relaying. A SIGINT or SIGQUIT landing while ck starts the command may not reach it. A signal ck inherited as ignored, as under nohup, stays ignored and is never forwarded. An inherited ignored SIGCHLD is the exception: ck handles it so the child's exit status survives, and sets it back to ignored in the child before exec. A run whose exit status cannot be collected, or whose output could not be written, sets no marker.
  • Output cap. Past 20 MiB, ck writes out what it holds, streams the rest, and does not cache the run.
  • Errors. Flags and local faults exit 125 before anything runs: a bad duration, ttl + stale > 30d, a CACHEKIT_MASTER_KEY that is not exactly 32 bytes of hex, an unusable file key, or a cache directory FileBackend::build refuses (that message names chmod 700). A backend read error runs the command uncached with one warning. A file-backend Timeout and a decrypt failure are both misses. Any write error after the run is a warning, and the exit code stays the command's.

ck drives cachekit-rs at the Backend trait with default-features = false and encryption, file. It parses arguments by hand, because one subcommand does not justify a parser framework on a path with a 5 ms budget.

Repo changes

  • The binary is now ck ([[bin]] name = "ck"), and the CI binary grep scans target/release/ck.
  • rust-version = "1.85" with resolver = "3", so the lockfile resolves to 1.85-compatible versions. A new msrv job runs cargo +1.85 check --locked --all-targets.
  • deny.toml bans a direct HTTP client: deny = [{ crate = "reqwest", wrappers = ["cachekit-rs"] }]. Adding reqwest directly fails cargo deny check bans. With cachekit-rs's cachekitio feature on, it passes, so the wrapper list is exact.
  • The licence allow list grows from MIT to exactly what the graph carries: Apache-2.0 (ring), BSD-3-Clause (subtle), BSL-1.0 (xxhash-rust), ISC (ring, untrusted), and Unicode-3.0 (the ICU crates under url, plus unicode-ident).
  • One crate-scoped exception: cbindgen (MPL-2.0). It is an unconditional build dependency of cachekit-core, but its build script only runs it under the ffi feature, which ck does not enable. None of its code is linked into ck. No copyleft licence goes on the allow list. Making cbindgen optional in cachekit-core would remove the exception.
  • New direct dependencies: cachekit-rs, plus cachekit-core, blake2, hex, getrandom, libc, rustix and tokio (rt only), which cachekit-rs already pulls in. signal-hook and signal-hook-registry (MIT OR Apache-2.0) are the only crates new to the graph. Dev-only: tempfile.
  • unsafe_code moves from forbid to deny, with three audited exceptions in src/child.rs. inherited_ignored is a sigaction call with a null new action, which only reads a disposition. ignore_sigchld_after_fork is a pre_exec hook that calls only async-signal-safe signal. block_handled_signals is a pthread_sigmask call that blocks the four handled signals on the calling thread only. No dependency offers any of these safely.

Tests

cargo test --locked runs 12 unit tests, 40 separate-process tests and a README test. The separate-process tests drive the real binary with its own HOME and a PATH shim that counts its calls. They cover:

  • an outage with a stale value: 10 calls all serve stale with exit 0, and the command runs at most twice;
  • an outage with no value: at most 2 runs in 10 calls;
  • 10 concurrent cold calls: exactly 1 run;
  • a contended lock with stale output: stale is served without waiting;
  • a late marker: inert while the value is fresh;
  • a spawn failure sets no marker, and a child exiting 127 does;
  • a marker 16 minutes ahead is inactive, and one 14 minutes ahead suppresses;
  • lowering --stale stops an old entry being served;
  • a terminal, pipe, socket and file on stdin each run uncached with one warning, and /dev/null caches;
  • the child sees no CACHEKIT_* variable;
  • 10 concurrent first runs agree on one file key, with no temp name left behind;
  • invalid file keys (not hex, short, a trailing newline, mode 0644) exit 125 and are left untouched;
  • a group- or other-writable cache directory exits 125;
  • bad flags exit 125 before anything runs;
  • --refresh never serves stale;
  • output past 20 MiB is streamed and not cached;
  • SIGTERM is forwarded and stores nothing;
  • SIGINT waits for the child, prints nothing and stores nothing;
  • a TERM after a trapped group INT, and a second TERM after a trapped first, both reach the command, with the first signal's exit code;
  • a signal after the command exited, with only a background job holding stdout, stops ck;
  • on Linux, 50 repeated SIGTERMs add no threads to ck;
  • a signal while ck serves stale output into a full pipe still stops it;
  • a waiter blocked on the lock exits on a signal;
  • a process-group SIGINT, as from Ctrl-C, exits 130 over 40 rounds with a stale value present and stores no marker;
  • a signal while ck is writing output stops it, and nothing is cached;
  • a closed reader ends a streamed command and sets no marker;
  • a SIGHUP ignored by the caller (nohup) stays ignored;
  • a command that traps SIGINT and exits 1 is still read as interrupted, over 20 rounds;
  • a command that sends itself TERM or INT is a failure that serves stale output;
  • SIGTERM stops ck as soon as the command exits, even with a background job still holding its stdout;
  • on Linux, every helper thread blocks the four handled signals;
  • a FIFO at file.key exits 125 without hanging;
  • an inherited ignored SIGCHLD: a success is cached rather than read as a failure, and the command still inherits SIGCHLD ignored;
  • a success too large to store still clears the marker;
  • a corrupted entry is a miss and is never served;
  • scope and argv are both in the key.

tests/readme.rs runs every sh block in the README with a stand-in op on PATH. It also fails if an example lacks < /dev/null.

tests/bench.rs is the hit-path gate. It times ck run -- true against true over 300 interleaved runs, after filling the cache directory with 10,000 entries, and fails if the median adds more than 5 ms. CI runs it in release mode on Linux and macOS.

Locally on Linux, fmt, clippy -D warnings, cargo test --locked, cargo deny check and the 1.85 check all pass. The suite also passes pinned to one CPU (taskset -c 0), which is how the Ctrl-C race reproduces on the earlier code. With cachekit-rs 0.9.0 the benchmark adds 0.5 ms locally on Linux (ext4), against 3.8 ms on 0.8.0 on the same host.

ck run wraps a command and caches its stdout when it exits 0, on the
cachekit-rs file backend with client-side encryption.

- Reader-side freshness from a stored_at timestamp, so each call applies its
  own --ttl and --stale.
- A keyed BLAKE2b-256 cache key over the length-prefixed scope and argv.
- A failure marker with escalating backoff and a 15-minute active window.
- A per-key flock fill lock, so concurrent cold calls run the command once.
- An auto-created file key, published with link() so racing first runs agree.
- Only /dev/null stdin is cached; CACHEKIT_* is stripped from the child.
- SIGTERM and SIGHUP are forwarded; output past 20 MiB streams uncached.

The binary is named ck. CI gains an MSRV 1.85 build and the hit-path
budget benchmark, cargo-deny bans a direct HTTP client, and the licence
allow list carries exactly the permissive licences in the graph.
- Record a received signal inside its handler, so a child killed by a
  process-group SIGINT is never read as an ordinary failure. A child that
  dies of SIGINT, SIGQUIT, SIGTERM or SIGHUP also counts as interrupted.
- After the child exits, a signal ends ck at once instead of being dropped.
- Leave signals inherited as ignored alone, so `nohup ck run` keeps its
  SIGHUP immunity and the command inherits it. Reading the disposition is
  the crate's one unsafe block; unsafe_code is deny instead of forbid.
- Delete the marker on every exit-0 run, even one too large to store.
- Stop streaming and drop the pipe once stdout fails, so the command gets
  SIGPIPE.
- Treat stdin as uncacheable when /dev/null cannot be inspected.
- Read file.key through one O_NOFOLLOW descriptor that is also fstat'ed.
- Reject a non-ASCII duration instead of panicking; never panic on a
  closed stderr.
With SIGCHLD ignored, the kernel reaps the child itself and its exit status
is lost, so a successful run read as exit 126 and set a failure marker.

- When SIGCHLD is inherited as ignored, ck handles it so the status can be
  collected, and sets it back to ignored in the child before exec.
- An exit status that cannot be collected is no evidence the origin failed:
  no marker, stale output if servable, otherwise exit 126 with a warning.
- A waitid error other than EINTR is no longer swallowed.
- A run is interrupted only when ck itself received the signal. A command
  that dies of a signal on its own (`kill -INT $$`, `pkill op`) is an
  ordinary failure, so a stale value is still served.
- ck's helper threads (the signal thread and tokio's blocking pool) block
  the handled signals, so their handlers always run on the main thread and
  the flag is set before the main thread reads the child's status.
- When ck's reader goes away mid-stream, the command's SIGPIPE death sets
  no failure marker.
- A signal that arrives while only a background job holds the command's
  stdout ends ck at once.
- file.key is opened O_NONBLOCK | O_NOCTTY, so a FIFO there exits 125
  instead of hanging.
- On a handled signal while the command runs, the signal thread forwards
  it (TERM and HUP), waits for the command to exit, and exits 128 + n. A
  background job left holding the command's stdout no longer keeps ck
  alive.
- An interrupted run prints nothing, as well as storing nothing.
- When the command dies of a handled signal before ck's own handler has
  run, wait up to 50 ms for it: kernels that signal a process group one
  member at a time give no order against the child's exit.
- Rename Stdout::ReaderGone to OutputLost, since any stdout write failure
  sets it.
- README: the Signals section now matches the behaviour.
- Tests poll a readiness file instead of sleeping, and a Linux test checks
  that every helper thread blocks the handled signals.
- The signal thread forwards every SIGTERM and SIGHUP while the command
  runs, so a command that traps the first one can still be stopped. On the
  first signal it records it and starts a waiter that ends ck once the
  command itself has exited.
- Every path that ends ck on a signal exits 128 + the first signal, so two
  different signals give one deterministic exit code.
- README and docs: a signal discards the output ck was holding back; output
  already written stays written.
- Tests: INT then TERM and TERM then TERM against commands that trap the
  first signal, the exited-command case again, and readiness polls in place
  of fixed sleeps.
- record_first reports whether it recorded the signal, and only that call
  starts the waiter, so repeated TERM or HUP signals add no threads.
- A signal that arrived while the spawn path held the state lock ends the
  run before the command starts.
- README: signals that arrive together count in signal-number order; a
  streamed run can lose its tail on a signal; a signal inside the spawn
  itself is honoured when the command exits.
- Tests: a Linux thread-count check, readiness polls in place of the last
  fixed pauses, and one wait_exit helper.
…-5118)

- Signals::new now runs before the flag handlers, so a signal landing
  between the two still reaches the signal thread, which exits 128 + n.
- A signal found before spawn releases the lock and lets the signal thread
  decide the first signal, so signals keep signal-number order there too.
- README: only SIGINT and SIGQUIT can miss a command that is starting, and
  only an interrupted streamed run can lose its tail.
- Tests: ck runs as a process-group leader and wait_exit kills the whole
  group on timeout; the thread-count test waits on a baseline.
signal-hook-registry installs a signal's OS handler before it publishes the
action list, so a signal in that window reached neither the iterator nor the
flag. install now blocks the handled signals on the main thread for the
whole of installation and the signal-thread spawn, then restores the
caller's mask with SIG_SETMASK. A signal sent meanwhile is delivered after
every action is published, to both handlers.

The mask code is one sigmask helper, still the crate's single unsafe site
for it. settle_first waits up to 5 s for the signal thread's record, now
that every flagged signal is guaranteed to reach that thread. Redundant
per-test process_group calls are gone.
…LAB-5118)

Every return in cached() that runs no command (stale on a contended lock,
a fresh value read under the lock, stale or suppression under an active
marker) and the spawn-failure path now checks Supervisor::interrupted()
first. A signal ck has already received ends the call with 128 + n and no
output, instead of being ignored or answered alongside served output.

A signal that arrives after the check is still answered by the signal
thread, so ck stays killable while it blocks writing a served value.

README and docs: ck exits 128 plus the number of a signal it received;
which of several close together is unspecified.
- Supervisor::run's spawn-failure arm now releases the state lock and
  returns Interrupted when ck has already received a signal, so the
  uncached path (stdin not /dev/null, lock or backend failure) exits
  128 + n instead of racing to 127 or 126. The separate check in
  record() is gone: one site covers both callers.
- README: a signal while ck prints stored output can cut it off, and ck
  still exits 128 + n, so $(...) callers must check the exit code.
- Comments no longer claim every no-command return checks, or that the
  signal thread sees signals in order.
cachekit-rs 0.9.0 sweeps orphaned temp files on a file backend's first
set instead of in FileBackend::build. A hit no longer reads the whole
cache directory, so the 10,000-entry hit-path bench drops from 3.8 ms
to 0.5 ms added on Linux.

0.9.0 also makes EncryptionLayer::new reject any master key that is not
exactly 32 bytes. ck accepted 32 or more, so a longer CACHEKIT_MASTER_KEY
would now fail inside the library with advice meant for Rust callers.
ck checks for exactly 32 bytes itself and names `openssl rand -hex 32`,
and the README says the key is exactly 64 hex characters.
@27Bslash6

Copy link
Copy Markdown
Contributor Author

@kody review

@kodus-27b

kodus-27b Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Code Review Completed! 🔥

The code review was successfully completed based on your current configurations.

Kody Guide: Usage and Configuration
Interacting with Kody
  • Request a Review: Ask Kody to review your PR manually by adding a comment with the `@kody start-review` command at the root of your PR.

  • Provide Feedback: Help Kody learn and improve by reacting to its comments with a 👍 for helpful suggestions or a 👎 if improvements are needed.

Providing Context (Files & MCPs)

Add these hints in your PR description (or a comment) to unlock deeper checks:

  • Ticket / Acceptance Criteria: `Refs: ABC-123` (Linear/Jira/Asana/ClickUp/Trello) or a direct ticket link.
  • Bugfix Validation: a Sentry/Datadog/Bugsnag event link (or paste the stack trace/error message).
  • Endpoint Risk: mention the route (e.g., `POST /api/payments`) or controller/action name.
  • Attach a repo file as context: use an explicit marker like `@file:docs/guide.mdx#L10-L50` (replace with your real path).
  • API Contract Docs: include `@file:openapi.yaml` or `@file:swagger.json` when changing routes/schemas.
  • Definition of Done / Standards: include `@file:DOD.md` or `@file:CONTRIBUTING.md` if your repo has them.
  • Design System Source of Truth: include `@file:ui/index.ts` (replace with your DS entrypoint path).
  • Feature Flags: include the flag key/name and `@file:flags.ts` / `@file:config.json` (and optionally the PostHog flag name).
  • Edge/CDN Rules: link the Cloudflare rule/zone or describe the intended redirect/header behavior.
  • Attach an MCP tool output: use `@mcp<provider|tool>` (replace with an installed MCP provider + tool, e.g., `@mcp<sentry|events.search>`).
Current Kody Configuration
Review Options

The following review options are enabled or disabled:

Options Enabled
Bug ✅
Performance ✅
Security ✅
Business Logic ✅

Access your configuration settings here.

Kody Code Review — 2 suggested fixes.
Paste the prompt below to your agent and all review fixed at once!

🛠️ Open Agent Prompt
A code review identified the following issues in this pull request.
Each section describes what was found and includes a reference implementation where available.

Files involved:
- README.md:15
- src/store.rs:97

---

### [1/2] README.md:15
Issue identified during code review:
Violates team rule 'No hardcoded API key/token literal in credential-named variable': WHAT: Flag a string literal assigned to a variable, constant, struct field, or default value whose name indicates a credential (api_key, token, secret, password, access_key, client_secret) in non-test source code.
HOW: In ADDED lines, look for `let/const/static <credential-ish name> = "..."` or `credential_field: "..."` where the right-hand side is a literal instead of an env/secret lookup.

---

### [2/2] src/store.rs:97
Issue identified during code review:
Swallowed error in src/store.rs: the `Err(_) => Ok(None)` arm turns every decrypt or AAD failure into a cache miss and discards the error without logging it. When the encryption key is rotated, or a stored entry is corrupted or tampered with, every affected read silently returns a miss, and operators get no signal explaining the jump in the miss rate. Fix: bind the error and log it before returning, without logging key material, e.g. `Err(e) => { tracing::warn!(key = %key, error = %e, "decrypt failed; treating as miss"); Ok(None) }`.

---

Review each issue in context, use the reference implementations as guidance, and apply fixes that are consistent with the surrounding codebase.

Comment thread README.md
every new shell would otherwise call the secret manager again:

```sh
API_TOKEN="$(ck run --ttl 12h --stale 7d -- op read op://Private/api/token < /dev/null)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kody code-review Kody Rules high

Violates team rule 'No hardcoded API key/token literal in credential-named variable': WHAT: Flag a string literal assigned to a variable, constant, struct field, or default value whose name indicates a credential (api_key, token, secret, password, access_key, client_secret) in non-test source code.
HOW: In ADDED lines, look for let/const/static <credential-ish name> = "..." or credential_field: "..." where the right-hand side is a literal instead of an env/secret lookup.

Kody rule violation: Treat suspicious hardcoded tokens/keys as secrets and move them to a secret store

Prompt for LLM

File README.md:

Line 15:

Violates team rule 'No hardcoded API key/token literal in credential-named variable': WHAT: Flag a string literal assigned to a variable, constant, struct field, or default value whose name indicates a credential (api_key, token, secret, password, access_key, client_secret) in non-test source code.
HOW: In ADDED lines, look for `let/const/static <credential-ish name> = "..."` or `credential_field: "..."` where the right-hand side is a literal instead of an env/secret lookup.

Talk to Kody by mentioning @kody

Was this suggestion helpful? React with 👍 or 👎 to help Kody learn from this interaction.

​

​

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a hardcoded credential. API_TOKEN="$(ck run … -- op read op://Private/api/token < /dev/null)" assigns the output of a command substitution: the token is read from 1Password at run time, and no literal secret appears in the README.

Comment thread src/store.rs Outdated
A decrypt or AAD failure stays a miss that is never served, but it now
prints one stderr line per call. A new master key also changes the keyed
file name, so key rotation never reaches this path: a failure here means a
corrupted or tampered entry, which should not pass silently. The message
carries the library's error text and no key material.
@kodus-27b

kodus-27b Bot commented Oct 3, 2026

Copy link
Copy Markdown

Code Review Completed! 🔥

The code review was successfully completed based on your current configurations.

Kody Guide: Usage and Configuration
Interacting with Kody
  • Request a Review: Ask Kody to review your PR manually by adding a comment with the `@kody start-review` command at the root of your PR.

  • Provide Feedback: Help Kody learn and improve by reacting to its comments with a 👍 for helpful suggestions or a 👎 if improvements are needed.

Providing Context (Files & MCPs)

Add these hints in your PR description (or a comment) to unlock deeper checks:

  • Ticket / Acceptance Criteria: `Refs: ABC-123` (Linear/Jira/Asana/ClickUp/Trello) or a direct ticket link.
  • Bugfix Validation: a Sentry/Datadog/Bugsnag event link (or paste the stack trace/error message).
  • Endpoint Risk: mention the route (e.g., `POST /api/payments`) or controller/action name.
  • Attach a repo file as context: use an explicit marker like `@file:docs/guide.mdx#L10-L50` (replace with your real path).
  • API Contract Docs: include `@file:openapi.yaml` or `@file:swagger.json` when changing routes/schemas.
  • Definition of Done / Standards: include `@file:DOD.md` or `@file:CONTRIBUTING.md` if your repo has them.
  • Design System Source of Truth: include `@file:ui/index.ts` (replace with your DS entrypoint path).
  • Feature Flags: include the flag key/name and `@file:flags.ts` / `@file:config.json` (and optionally the PostHog flag name).
  • Edge/CDN Rules: link the Cloudflare rule/zone or describe the intended redirect/header behavior.
  • Attach an MCP tool output: use `@mcp<provider|tool>` (replace with an installed MCP provider + tool, e.g., `@mcp<sentry|events.search>`).
Current Kody Configuration
Review Options

The following review options are enabled or disabled:

Options Enabled
Bug ✅
Performance ✅
Security ✅
Business Logic ✅

Access your configuration settings here.

@27Bslash6
27Bslash6 merged commit 04d6376 into main Oct 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant