fix(server): refuse to start a second server against a live data directory - #8442
Conversation
…ctory
Two servers pointed at one `--base-dir` both open `state.sqlite` and both write
`settings.json`, and they overwrite each other. Observed: a desktop app
auto-updated to a newer server while the old one was still running, the new
process found its port taken, silently bound a random one, and ran blind against
shared state. The visible symptom was a settings toggle that would not stick --
hours away from the cause, and nothing about it is detectable afterwards.
So refuse at startup. The lock is claimed before anything binds a port or opens
the database, and is provided into `HttpServerLive` rather than merged beside it
so the ordering is structural: the lock is a dependency of the thing it protects.
An advisory `flock` would be the better primitive, since the kernel drops it when
the holder dies. Node has no binding for it and a native dependency for one lock
is the worse trade, so this is an atomically created file holding the owner's
identity, with liveness checked by signal 0. The tradeoff is stated in the module:
a killed server whose pid is later reused blocks startup until the file is
removed, which is the safe direction, and the message names the file.
A lock whose owner is gone, or which a crash tore in half mid-write, is reclaimed
rather than treated as permanent. Reclaiming re-races the exclusive create, so two
servers starting together still produce one winner. Release only removes a lock
this process still owns, so a successor is never evicted.
The bound port is stamped onto the lock afterwards purely so a later server's
refusal names an address the user can open rather than just a pid.
Verified end to end against two real servers: the second refuses with the message
below, exits 1, and never binds; shutdown releases; restart is unblocked.
Another T3 Code server is already using this data directory.
data directory: /tmp/t3-smoke-basedir/userdata
held by: pid 285163, listening on port 39977
since: 2026-08-27T16:59:03.329Z
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
github-actions
Bot
added
vouch:unvouched
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a006bdbdb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Effect.gen(function* () { | ||
| const failure = yield* claimLock({ stateDir, lockPath, startedAt }); | ||
| if (failure !== undefined) return yield* failure; |
There was a problem hiding this comment.
When this release starts while any pre-lock T3 version is still running against the same data directory—the desktop auto-update scenario described by this change—there is no server.lock, so claimLock succeeds and both servers continue writing shared state. The previous version already persists its live PID in server-runtime.json via serverRuntimeState.ts, so the first lock-enabled release needs to reject a live legacy runtime state before creating the new lock; otherwise the rollout does not fix the incident until every running server has already upgraded once.
Useful? React with 👍 / 👎.
| // Owner is gone, or the file is unreadable because a crash tore a write in | ||
| // half. Reclaim and re-race the exclusive create, which still yields one | ||
| // winner when two servers start together. | ||
| yield* fs.remove(input.lockPath).pipe(Effect.ignore); |
There was a problem hiding this comment.
When two servers concurrently start after a crash left a stale lock, both can read the stale holder, then the first can remove it and create its own lock before the second reaches this unconditional removal. The second process then unlinks the first process's newly acquired lock and succeeds on its next wx attempt, leaving both servers running against the same state directory. Reclamation must conditionally remove the exact stale object or use a locking primitive/protocol that cannot delete a successor's claim.
Useful? React with 👍 / 👎.
| const holder = yield* readHolder(lockPath); | ||
| if (holder === undefined || holder.pid !== process.pid) return; | ||
| yield* fs.writeFileString(lockPath, encodeHolder({ ...holder, port })).pipe(Effect.ignore); |
There was a problem hiding this comment.
When another server attempts startup while the holder records its bound port, this ordinary writeFileString opens the lock with truncation before rewriting the JSON. A contender can fail its exclusive create during that window, decode the empty or partial file as an undefined holder, remove it as stale, and acquire the directory while the original server remains live. Write the metadata through an atomic replacement so readers never mistake an in-progress update for a crashed owner.
Useful? React with 👍 / 👎.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| // Owner is gone, or the file is unreadable because a crash tore a write in | ||
| // half. Reclaim and re-race the exclusive create, which still yields one | ||
| // winner when two servers start together. | ||
| yield* fs.remove(input.lockPath).pipe(Effect.ignore); |
There was a problem hiding this comment.
🟠 High src/serverSingleton.ts:153
A stale-lock reclaim can delete a new lock owned by another starter, so both servers proceed from claimLock while only one lock file remains. After A reads a dead holder, B can remove and recreate the file, then A's unconditional fs.remove(input.lockPath) removes B's lock before A creates its own. Reclaim only the same file instance that was inspected (or otherwise make stale-lock removal conditional) before retrying.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/serverSingleton.ts around line 153:
A stale-lock reclaim can delete a new lock owned by another starter, so both servers proceed from `claimLock` while only one lock file remains. After A reads a dead holder, B can remove and recreate the file, then A's unconditional `fs.remove(input.lockPath)` removes B's lock before A creates its own. Reclaim only the same file instance that was inspected (or otherwise make stale-lock removal conditional) before retrying.
| const fs = yield* FileSystem.FileSystem; | ||
| const holder = yield* readHolder(lockPath); | ||
| if (holder === undefined || holder.pid !== process.pid) return; | ||
| yield* fs.writeFileString(lockPath, encodeHolder({ ...holder, port })).pipe(Effect.ignore); |
There was a problem hiding this comment.
🟠 High src/serverSingleton.ts:183
recordServerLockPort can cause two live servers to own the same data directory: its in-place rewrite temporarily leaves lockPath empty or partially written, so a concurrent claimLock treats the live lock as stale, removes it, and acquires a replacement lock. Update the holder atomically (for example, write a temporary file and rename it) so readers never observe an incomplete lock.
Also found in 2 other location(s)
apps/server/src/server.ts:533
Calling
recordServerLockPortrewrites the live lock file with a non-atomic truncate/write. A second server whose exclusive create loses during that interval reads an empty/partial file, treats it as stale inclaimLock, removes it, and can acquire a replacement lock while the first server remains running. This re-enables concurrent servers against the same data directory and the corruption the lock is intended to prevent.
apps/server/src/server.ts:713
Providing
ServerSingletonLiveenables a lock protocol with a startup race: the winningwriteFileString(..., { flag: "wx" })creates the lock before its JSON payload has been fully written, while a losing concurrent starter treats an unreadable/empty lock as stale and removes it. The loser can then acquire the replacement lock even though the original server is alive, so two simultaneously started servers can both open and mutate the same data directory.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/serverSingleton.ts around line 183:
`recordServerLockPort` can cause two live servers to own the same data directory: its in-place rewrite temporarily leaves `lockPath` empty or partially written, so a concurrent `claimLock` treats the live lock as stale, removes it, and acquires a replacement lock. Update the holder atomically (for example, write a temporary file and rename it) so readers never observe an incomplete lock.
Also found in 2 other location(s):
- apps/server/src/server.ts:533 -- Calling `recordServerLockPort` rewrites the live lock file with a non-atomic truncate/write. A second server whose exclusive create loses during that interval reads an empty/partial file, treats it as stale in `claimLock`, removes it, and can acquire a replacement lock while the first server remains running. This re-enables concurrent servers against the same data directory and the corruption the lock is intended to prevent.
- apps/server/src/server.ts:713 -- Providing `ServerSingletonLive` enables a lock protocol with a startup race: the winning `writeFileString(..., { flag: "wx" })` creates the lock before its JSON payload has been fully written, while a losing concurrent starter treats an unreadable/empty lock as stale and removes it. The loser can then acquire the replacement lock even though the original server is alive, so two simultaneously started servers can both open and mutate the same data directory.
There was a problem hiding this comment.
Reviewed the new apps/server/src/serverSingleton.ts and its wiring in apps/server/src/server.ts against the Effect service conventions.
Imports (namespace subpath imports), Effect.fn usage, FileSystem/Path acquisition from the environment, scoped acquisition via Effect.acquireRelease inside Layer.effectDiscard, and the new tests all look consistent with the conventions. Two findings in the error-modelling area are commented inline: the underlying PlatformError from the exclusive lock write is discarded instead of being classified/preserved as cause, and ServerLockUnavailableError carries the message as a free-form single-value reason string.
Posted via Macroscope — Effect Service Conventions
| lockPath: Schema.String, | ||
| reason: Schema.String, | ||
| }, | ||
| ) { | ||
| override get message(): string { | ||
| return `Could not claim the server lock at ${this.lockPath}: ${this.reason}`; |
There was a problem hiding this comment.
reason is a single fixed prose string (the only construction site passes "the lock was repeatedly reclaimed by another starting server") and the message is built from it, so the message text is stored as data next to an already-specific error tag, and no cause is available when the failure came from a real filesystem error. Consider modelling the exhaustion structurally and keeping an optional cause, deriving message from those attributes.
{
lockPath: Schema.String,
- reason: Schema.String,
+ attempts: Schema.Int,
+ cause: Schema.optional(Schema.Defect()),
},
) {
override get message(): string {
- return `Could not claim the server lock at ${this.lockPath}: ${this.reason}`;
+ return `Could not claim the server lock at ${this.lockPath} after ${this.attempts} attempts; another starting server kept reclaiming it.`;
}The construction site then becomes new ServerLockUnavailableError({ lockPath: input.lockPath, attempts: 2 }).
Posted via Macroscope — Effect Service Conventions
| const created = yield* fs.writeFileString(input.lockPath, payload, { flag: "wx" }).pipe( | ||
| Effect.as(true), | ||
| Effect.orElseSucceed(() => false), | ||
| ); |
There was a problem hiding this comment.
Effect.orElseSucceed(() => false) erases the underlying PlatformError, so a non-AlreadyExists failure (permission denied, read-only or missing parent, out of disk) is silently reinterpreted as "the lock is taken" and surfaces as ServerAlreadyRunningError/ServerLockUnavailableError with no cause and a message that names the wrong problem. Consider treating only the structural AlreadyExists reason as "not created" and letting other platform errors propagate (typed) as packages/shared/src/relayClient.ts does for its install lock.
| const created = yield* fs.writeFileString(input.lockPath, payload, { flag: "wx" }).pipe( | |
| Effect.as(true), | |
| Effect.orElseSucceed(() => false), | |
| ); | |
| const created = yield* fs.writeFileString(input.lockPath, payload, { flag: "wx" }).pipe( | |
| Effect.as(true), | |
| Effect.catch((error) => | |
| error.reason._tag === "AlreadyExists" ? Effect.succeed(false) : Effect.fail(error), | |
| ), | |
| ); |
Posted via Macroscope — Effect Service Conventions
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7a006bd. Configure here.
| // Owner is gone, or the file is unreadable because a crash tore a write in | ||
| // half. Reclaim and re-race the exclusive create, which still yields one | ||
| // winner when two servers start together. | ||
| yield* fs.remove(input.lockPath).pipe(Effect.ignore); |
There was a problem hiding this comment.
Live lock stolen while file unreadable
Medium Severity
claimLock reclaims whenever the lock file is missing or not valid JSON, without proving the owner is gone. Exclusive create and recordServerLockPort's truncating overwrite both leave the file empty or partial while the holder is still alive, so a second process can delete it and start. releaseServerLock also removes a lock it cannot decode, which can evict a successor in that same window. Two live servers then share state.sqlite and settings.json.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 7a006bd. Configure here.
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR adds a production-wide server lock and changes startup/lifecycle behavior across processes, rather than making a small local correction. Unresolved review findings identify races and legacy-state cases that could still permit concurrent servers to share and corrupt the same data directory. Not approved because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
1 participant
What Changed
A new
apps/server/src/serverSingleton.tsclaims the data directory at server startup. A second server started against a directory another live server already holds now exits 1 with an explanatory message instead of starting anyway.The lock is
Layer.provided intoHttpServerLiverather than merged beside it, so the ordering is structural rather than incidental: the lock is a dependency of the thing it protects, and no server reaches a listening socket while another holds the directory.Three files, no refactors, no behaviour change for the single-server case. Only
t3 serveandt3 startbuild the server layer, so no other subcommand is affected.Why
Two T3 Code servers pointed at the same
--base-dirboth openstate.sqliteand both writesettings.json, and they overwrite each other. Nothing refuses the second start, and nothing reports it afterwards.The port check does not save you. When
:3775is already taken, the second server binds a different port and starts normally — so it looks perfectly healthy while running blind against shared state.Repro:
Before this change both start. Both hold
/tmp/t3-repro/userdata/state.sqliteopen and both write/tmp/t3-repro/userdata/settings.json; whichever writes last wins, and the other UI silently reverts.How I hit it in the wild: a desktop app auto-updated to a newer server while the old one was still running. The server does not self-upgrade, so
npx t3@<newer>started a second server against the same data directory. The visible symptom was a settings toggle that would not stick — about an hour away from the actual cause, and nothing about it is detectable after the fact.After this change, the second server exits 1 without binding a port:
Why a pid file and not
flockAn advisory
flockis the better primitive — the kernel drops it when the holder dies, so a crash leaves nothing stale. Node has no binding for it, and pulling in a native dependency for one lock seemed the worse trade. So this is a file created atomically withwxholding the owner's identity, with liveness checked via signal 0 (treatingEPERMas alive, since a server running as another user must not be trampled).The tradeoff is stated in the module rather than hidden: a server that is killed and whose pid is later reused by an unrelated process will block startup until the lock file is removed. That is the safe direction to fail, and the message names the file so recovery is one
rm. If you would rather take the native dependency and use a realflock, say so and I will redo it that way.Staleness is handled rather than fatal: a lock whose owner is gone, or which a crash tore in half mid-write, is reclaimed. Reclaiming re-races the exclusive create, so two servers starting simultaneously still produce exactly one winner. Release only removes a lock this process still owns, so a successor is never evicted by its predecessor's shutdown.
The bound port is stamped onto the lock after binding, purely so a later refusal can name an address the user can open rather than a bare pid.
One thing worth flagging for review: the lock is structurally guaranteed to precede the HTTP server binding, which is what the refusal path depends on. I did not attempt to order it against every sibling layer, so I would not claim it strictly precedes the first SQLite open in all compositions.
UI Changes
None — server startup behaviour only.
Tests
apps/server/src/serverSingleton.test.ts— 9 tests, following the existingit.layer(NodeServices.layer)convention used elsewhere in this directory: claim/release on scope exit, refusal while held, stale-pid reclaim, half-written-file reclaim, successor not evicted on release, port recorded and surfaced in the refusal, separate directories independent, and pid liveness.I checked the tests actually bite rather than just passing: changing the exclusive create from
{ flag: "wx" }to{ flag: "w" }fails "refuses a second server while the first holds the directory" and "records the bound port so the next server can name it".Verified locally:
vp test run src/serverSingleton.test.ts→ 9 passedapps/serversuite → 246 files passed, 2824 passed / 10 skipped (baseline on this commit's parent: 245 files, 2815 passed)vp run typecheck→ exit 0vp fmt --check→ clean;vp lintreports nothing new for the changed filesAlso verified end to end with two real server processes against one
--base-dir: the second refuses, exits 1, and never binds; SIGKILLing the holder leaves a stale lock that the next start reclaims; ordinary shutdown releases the lock and a restart is unblocked.Checklist
Note
Medium Risk
Changes server startup ordering and adds a global gate on shared state; mis-ordering or lock edge cases (e.g. pid reuse) could block legitimate starts, but single-server behavior is intended unchanged.
Overview
Prevents two T3 Code processes from sharing one
--base-dirand silently corruptingstate.sqliteandsettings.json(including the case where the second process binds a different port after the first took the default).Startup now claims the state directory with a
server.lockpid file (exclusive create + liveness viakill(pid, 0), with stale or half-written locks reclaimed). A live holder yieldsServerAlreadyRunningErrorwith data dir, pid, optional port, and recovery hints instead of starting.ServerSingletonLiveisLayer.provided intoHttpServerLiveso the lock runs before the HTTP server can listen; after bind,recordServerLockPortupdates the lock so refusal messages can say which port the other server is on. Lock releases on normal shutdown scope exit.Adds
serverSingleton.test.tscovering claim/release, double-start refusal, stale lock reclaim, port in error message, and per-directory isolation.Reviewed by Cursor Bugbot for commit 7a006bd. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add exclusive lock to prevent concurrent servers on the same data directory
ServerSingletonmodule in serverSingleton.ts that acquires aserver.lockfile in the configured state directory before the HTTP server binds ports.claimLockatomically creates the lock file with PID and timestamp; if another live process holds it, startup aborts withServerAlreadyRunningError. Stale or half-written locks are reclaimed on retry.recordServerLockPortso subsequent startup failures can report which port the holder is using.ServerSingletonLivelayer is piped as a dependency ofHttpServerLivein server.ts to enforce lock-before-listen ordering.stateDirwill now fail withServerAlreadyRunningErrorinstead of silently sharing the directory.📊 Macroscope summarized 7a006bd. 2 files reviewed, 4 issues evaluated, 2 issues filtered, 2 comments posted
🗂️ Filtered Issues
apps/server/src/server.ts — 0 comments posted, 2 evaluated, 2 filtered
recordServerLockPortrewrites the live lock file with a non-atomic truncate/write. A second server whose exclusive create loses during that interval reads an empty/partial file, treats it as stale inclaimLock, removes it, and can acquire a replacement lock while the first server remains running. This re-enables concurrent servers against the same data directory and the corruption the lock is intended to prevent. [ Cross-file consolidated ]ServerSingletonLiveenables a lock protocol with a startup race: the winningwriteFileString(..., { flag: "wx" })creates the lock before its JSON payload has been fully written, while a losing concurrent starter treats an unreadable/empty lock as stale and removes it. The loser can then acquire the replacement lock even though the original server is alive, so two simultaneously started servers can both open and mutate the same data directory. [ Cross-file consolidated ]