Code Duel — A Real-Time 1v1 Competitive Programming Platform

Built for the Coding Club, BITS Pilani. Adopted by 500+ users across live contests.

The source repositories are private (club-owned). This document is a technical writeup of the system I designed and built: the architecture, the engineering decisions, the trade-offs, and the measurements.

---

Table of Contents

1. What It Is
2. System Architecture
3. The Life of a Submission
4. Deep Dive — The Remote Code Execution Engine (Go)
5. Deep Dive — Matchmaking
6. Deep Dive — Real-Time Duel State
7. Scoring and Leaderboards
8. Contest Lifecycle and Durability
9. Data Model
10. Reliability and Failure Handling
11. Observability
12. Load Testing and Results
13. Technology Choices and Trade-Offs
14. Testing Strategy
15. Known Limitations and Roadmap
16. Repository Layout

---

1. What It Is

Code Duel is a head-to-head competitive programming platform. Two participants are matched against each other, receive the same algorithmic problem calibrated to their combined skill level, and race to solve it. The first correct submission wins the duel; the winner gains rating points weighted by opponent strength, solve speed, and win streak. Contests run for a fixed duration, during which participants duel repeatedly and climb a live leaderboard.

Everything that makes that experience feel instant is the hard part:

Requirement	Engineering consequence
Untrusted code must run safely	Every submission executes in a locked-down, single-use Docker sandbox
Verdicts must feel immediate	Container startup is removed from the critical path via a pre-warmed pool
Bursty traffic (everyone submits at once)	Queue-backed, demand-proportional autoscaling of the sandbox pool
Duels are live and stateful	WebSocket state machine with reconnection, forfeit timers, and draw resolution
Two players can't be double-booked	Atomic Redis claim + idempotency token on match creation
A contest can't lose scores if a process dies	Redis is the hot path; PostgreSQL is checkpointed continuously

Four independently deployable services, four languages judged (C++, Java, Python, PyPy), all coordinated through Redis.

---

2. System Architecture

flowchart TB
    subgraph client["Client"]
        FE["Next.js 15 Frontend<br/>React · Monaco editor · Socket.IO client"]
    end

    subgraph core["Core Services"]
        BE["Backend — Node.js / Express / TypeScript<br/>REST API · Socket.IO · better-auth"]
        MM["Matchmaking Service — Node.js / TypeScript<br/>independent pairing loop"]
        RCE["RCE Engine — Go<br/>sandbox pool · autoscaler · judge"]
        ADMIN["Admin Portal — Django<br/>unmanaged models over the same schema"]
    end

    subgraph data["State"]
        REDIS[("Redis Stack<br/>queues · JSON state · sorted sets")]
        PG[("PostgreSQL<br/>Drizzle ORM · source of truth")]
    end

    subgraph sandbox["Sandboxing"]
        DOCKER["Docker Engine"]
        C1["sandbox: cpp"]
        C2["sandbox: java"]
        C3["sandbox: python"]
        C4["sandbox: pypy"]
    end

    FE <-->|"WebSocket + REST"| BE
    BE --> PG
    BE <--> REDIS
    MM <--> REDIS
    MM -->|"POST /api/v1/match-found"| BE
    BE -->|"LPUSH submissionsQueue:{lang}"| REDIS
    RCE -->|"BRPOP"| REDIS
    RCE --> DOCKER
    DOCKER --> C1 & C2 & C3 & C4
    RCE -->|"POST result callback"| BE
    ADMIN --> PG

Loading

Why four services instead of one

The single most important boundary is the RCE engine. Node.js is single-threaded; orchestrating Docker lifecycles and marshalling compile output on the same event loop that serves every live WebSocket would turn one heavy C++ compile into a latency spike for every connected player. Moving execution into a separate Go process means:

• Blast radius containment — the service that touches the Docker socket and runs untrusted code is not the service that holds user sessions.
• The right concurrency model — one goroutine per job, bounded by a semaphore, is a natural fit for orchestrating hundreds of concurrent container lifecycles.
• Independent scaling — the judge is CPU- and memory-bound; the API is I/O-bound. They scale on different axes.

The matchmaking service is separated for a different reason: it is a periodic workload (a loop that wakes on an interval and pairs everyone waiting), not a request/response workload. Keeping it out of the API server means the pairing loop can be tuned, restarted, or scoped to a single contest without touching live connections.

Services communicate asynchronously through Redis rather than direct HTTP. The backend pushes a job and returns immediately; the engine pulls when it has capacity. That gives natural backpressure: a submission burst grows a queue instead of overwhelming a thread pool, and the queue length is itself the autoscaling signal.

---

3. The Life of a Submission

sequenceDiagram
    autonumber
    actor P as Player
    participant FE as Frontend
    participant BE as Backend (Socket.IO)
    participant R as Redis
    participant RCE as RCE Engine (Go)
    participant SB as Sandbox Container

    P->>FE: Write solution, hit Submit
    FE->>BE: socket.emit("submitCode", {duelId, problemId, code, language})
    BE->>R: Validate duel is in_progress
    BE->>R: Check cooldown:submit:{userId}
    BE->>BE: Reject duplicate of last submission
    BE->>R: SET cooldown (EX 10)
    BE-->>FE: duelUpdate { status: "JUDGING" } (broadcast to both players)
    BE->>R: LPUSH submissionsQueue:{language}
    BE->>BE: INSERT into submissions (PostgreSQL)

    RCE->>R: BRPOP submissionsQueue:{language} (5s timeout)
    RCE->>RCE: Validate size limits, compute time/memory budget
    RCE->>RCE: Acquire semaphore slot, mark job in-flight
    RCE->>SB: Take pre-warmed container from pool
    RCE->>SB: docker exec /app/runner --stdin  ← JSON payload
    SB->>SB: Compile → run every hidden test case → check output
    SB-->>RCE: Verdict JSON on stdout
    RCE->>SB: Destroy container (deferred), replenish pool async
    RCE->>BE: POST /api/v1/submission/result (retry w/ backoff)
    BE->>R: Update duel state, streaks, ZINCRBY leaderboard
    BE->>PG: Persist verdict + duel outcome
    BE-->>FE: submissionStatus + duelStatusChange + leaderboardUpdate
    FE-->>P: Verdict, victory/defeat modal, updated rank

Loading

There are three separate queue families per language, so that a quick "Run against sample cases" never queues behind a full submission judgement:

Queue	Purpose	Result callback
submissionsQueue:{lang}	Real submission, all hidden tests, decides the duel	/api/v1/submission/result
runQueue:{lang}	"Run" against visible sample cases	/api/v1/run/result
runQueueCustom:{lang}	Run against user-supplied custom input	/api/v1/run/result

Four languages × three queues = 12 concurrent BRPOP goroutines, each an independent consumer.

---

4. Deep Dive — The Remote Code Execution Engine (Go)

This is the heart of the project and where most of the engineering went.

4.1 The sandbox

Every submission runs in a single-use container that is destroyed immediately afterwards. Containers are never reused between submissions, so no state — files, processes, memory — can leak from one user's code to another's.

Each sandbox is created with:

hostConfig := &container.HostConfig{
    Resources: container.Resources{
        Memory:    baseMemory,        // hard cgroup memory ceiling
        NanoCPUs:  cpuNanoCPUs,       // 0.5 core — keeps CFS scheduling predictable
        PidsLimit: &pidsLimit,        // 512 — fork-bomb protection
        Ulimits:   []*container.Ulimit{{Name: "fsize", Soft: ..., Hard: ...}},
    },
    ReadonlyRootfs: true,             // immutable root filesystem
    CapDrop:        []string{"ALL"},  // every Linux capability dropped
    SecurityOpt:    []string{"no-new-privileges"},
    NetworkMode:    "none",           // no network namespace at all
    Tmpfs: map[string]string{
        "/workspace": "rw,exec,nosuid,size=256m,uid=1000,gid=1000",
        "/tmp":       "rw,exec,nosuid,size=128m,mode=1777",
    },
}

Layered on top of that:

• The container image creates a non-root rceuser (uid 1000) and runs as it.
• The only writable space is a size-capped in-memory tmpfs that vanishes with the container.
• Runtime memory is enforced twice — by the cgroup at the container level, and inside the container by ulimit-style address-space caps (and -Xmx for the JVM), so an over-allocating program gets a clean MEMORY_LIMIT_EXCEEDED verdict rather than being OOM-killed opaquely.
• Payload guards run before a container is ever acquired: max code size (256 KB), max test-case count, max captured output.

The engine deliberately supports two runner transports, selected by config:

Mode	Mechanism	Why
exec (default)	docker exec /app/runner --stdin, payload on stdin, verdict on stdout, streams demuxed with stdcopy	Sandbox needs no network namespace at all; portable and stable across hosts
http	Runner is an HTTP server on a random mapped host port; engine POSTs the payload	Lower per-job overhead under heavy Linux load

exec is the production default because it lets the sandbox run with NetworkMode: "none" — the strongest isolation posture — at a small latency cost.

4.2 The pre-warmed container pool — the 65% latency win

Creating and starting a container is hundreds of milliseconds. Paying that on every submission is the single largest avoidable cost in a judge. So the engine keeps a pool of already-running, already-idle containers per language.

The pool is, at its core, a buffered channel per language — the channel is the pool, which makes "wait for a free sandbox" a single blocking receive with no custom condition-variable logic:

type ContainerPool struct {
    containers   map[string]chan ContainerInfo // language → ready sandboxes
    currentSizes map[string]int
    targetSizes  map[string]int
    createMus    map[string]*sync.Mutex        // per-language creation lock
    ...
}

func (p *ContainerPool) GetContainer(ctx context.Context, lang string) (ContainerInfo, error) {
    select {
    case info := <-p.containers[lang]:
        return info, nil
    case <-ctx.Done():
        return ContainerInfo{}, ctx.Err()
    }
}

flowchart LR
    subgraph pool["Warm pool — one buffered channel per language"]
        direction LR
        W1["warm"] --- W2["warm"] --- W3["warm"] --- W4["warm"]
    end

    JOB["Job arrives"] -->|"GetContainer — no cold start"| W1
    W1 --> EXEC["exec runner<br/>compile · run tests · check"]
    EXEC --> DESTROY["Destroy container<br/>(deferred, always runs)"]
    DESTROY --> REPL["Replenish asynchronously<br/>off the critical path"]
    REPL --> pool

Loading

Two properties make this work:

1. Destroy-and-replenish is asynchronous. A defer in the executor fires the container teardown and pool refill in a background goroutine, so the user's verdict is not waiting on Docker cleanup.
2. Creation is parallel across languages, serialized within one. A per-language mutex avoids hammering the Docker API with concurrent creates of the same image (a source of daemon-level contention), while all four languages still warm in parallel at startup.

Result: container startup leaves the critical path entirely — measured ~65% reduction in end-to-end evaluation latency versus creating a container per submission.

4.3 Demand-proportional autoscaling — the 5× burst capacity

A fixed pool is wrong in both directions: too small and a submission burst queues behind cold starts; too large and idle containers waste the memory the judge needs to actually run code.

So a scaling manager runs on a 2-second tick and re-sizes each language's pool from live demand:

flowchart TD
    T["Tick every 2s"] --> IF["In-flight jobs per language<br/>(atomic counters)"]
    T --> QL["Queue depth per language<br/>(Redis LLEN × 3 queues)"]
    IF --> D["langDemand = in-flight + queued"]
    QL --> D
    D --> P["proportion = langDemand / totalDemand"]
    P --> TGT["target = baseSize + proportion × scalingCapacity<br/>clamped to [baseSize, maxSize]"]
    TGT --> ADJ{"target vs current"}
    ADJ -->|"higher"| UP["ScaleUp — capped at ±8 per cycle"]
    ADJ -->|"lower"| DOWN["ScaleDown — capped at ±8 per cycle"]
    ADJ -->|"equal"| HOLD["Hold"]
    UP --> T
    DOWN --> T
    HOLD --> T

Loading

Three details that matter:

• Queue length alone is a bad signal. BRPOP drains the queue instantly — jobs move from "queued" to "in-flight" the moment a consumer is free, so a purely queue-based autoscaler sees zero demand while the machine is fully saturated. I added an in-flight tracker (a map of atomic.Int64 per language, incremented when a job takes a semaphore slot and decremented on completion) so demand is queued + in-flight. This was a real bug found under load, not a theoretical one.
• Capacity is allocated proportionally, not equally. If 90% of demand is C++, C++ gets the capacity. Java doesn't sit on 20 idle JVM containers during a C++-heavy problem.
• Adjustment is rate-limited to ±8 containers per cycle, so the pool ramps instead of thrashing.

With container_pool_base_size: 4 and container_pool_max_size: 20 per language, that is 5× burst capacity per language — a steady-state footprint of 16 warm containers that expands to 80 under load, and decays back when the burst passes.

4.4 Memory-aware admission control

The maximum concurrency is not hardcoded. On boot, the engine reads physical host RAM and derives a safe ceiling:

availableBytes := totalRAMBytes - 2*GiB          // reserve for OS, Redis, Go runtime, other services
const workingMemoryPerContainer = 100 * MiB      // ~2× observed peak (≈50 MiB during C++ compile)
safeMaxConcurrent := availableBytes / workingMemoryPerContainer
cfg.App.Server.MaxConcurrentJobs = safeMaxConcurrent

The subtlety worth calling out: the Docker --memory flag (1 GiB) is a security ceiling for runaway processes, not a capacity-planning number. Sizing the pool by that limit would produce a concurrency of ~4 on a 6 GB box. Sizing it by observed working set (~5 MiB idle, ~50 MiB peak during compilation, budgeted at 100 MiB) produces the real safe number. The config file acts as a floor and a ceiling around the computed value, so an operator who knows their workload can still override it.

The computed value feeds a semaphore that bounds concurrent jobs; everything past it waits in Redis rather than in RAM.

4.5 Hardware-independent time limits

A problem tagged "2 second limit" means nothing unless you know how fast the judge is. Running the same contest on a laptop and on a cloud VM would produce different verdicts for identical code.

On startup the engine calibrates itself: for each language it runs a fixed CPU-bound benchmark with a known reference time, and derives a scaling factor:

factor := measuredTime / bench.ReferenceTime
calibratedFactor := factor * 1.1   // 10% safety buffer

Every job's per-test-case limit then becomes problemTimeLimit × languageFactor. A host 1.5× slower than the reference scales every limit by ~1.65×. Calibration failure falls back to a conservative 1.5. Time limits become a property of the problem, not of the machine.

Note that the factor is per language: the Python interpreter and an -O2 C++ binary are affected very differently by the same slower host, so a single global multiplier would be unfair to one of them.

4.6 Compile caching and pre-compiled checkers

Correctness for problems with multiple valid answers (any valid ordering, floating-point tolerance, "output any shortest path") requires a special checker — a testlib.h-style C++ program that validates a submission's output against the input and the reference answer. Compiling that checker on every submission is pure waste, because the checker never changes for a given problem.

Two layers of avoidance:

1. Pre-compiled checker binaries are bind-mounted read-only at /checkers/{problemId}. If the binary exists, the runner copies it in and skips compilation entirely. The mount is optional and auto-detected from the engine's own container inspection, with graceful degradation: if the bind mount can't be created, the pool disables it and keeps serving.
2. Content-addressed compile cache on a shared Docker volume, keyed by SHA-256(checkerCode), so a checker compiled once by any container is reused by all subsequent ones. The cache volume is labelled rce-persist=true and explicitly excluded from the periodic prune, so cleanup never evicts it.

User code is deliberately not cached — it changes on every submission, so a cache lookup would be a guaranteed miss plus overhead.

4.7 Verdict pipeline

The in-container runner produces one of a closed set of verdicts, distinguishing them carefully:

stateDiagram-v2
    [*] --> Compile
    Compile --> COMPILE_ERROR: non-zero exit
    Compile --> RunTests: ok
    RunTests --> TIME_LIMIT_EXCEEDED: context deadline
    RunTests --> MEMORY_LIMIT_EXCEEDED: SIGKILL / SIGXCPU / OutOfMemoryError / bad_alloc
    RunTests --> RUNTIME_ERROR: SIGSEGV or non-zero exit
    RunTests --> Check: clean exit
    Check --> WRONG_ANSWER: checker rejects
    Check --> RunTests: more tests
    Check --> SUCCESS: all tests pass

Loading

Signal-level inspection matters here: a process killed by SIGKILL under an address-space limit is a memory failure, while SIGSEGV is a runtime failure, and the two must not be reported as the same thing to a competitor debugging under time pressure. Java is special-cased (-Xmx injection, OutOfMemoryError string matching) because the JVM manages its own heap and dies differently.

The engine runs in two modes per job: submit short-circuits on the first failure (fast rejection), while run executes every visible case and returns per-test-case detail so the player can see exactly which sample broke.

4.8 Result delivery

Results POST back to the backend over a shared, connection-pooled HTTP client (50 idle connections, 20 per host) — a fresh client per job would burn a TCP handshake on every verdict.

Delivery retries with exponential backoff (3 attempts, 1s → 2s) and, importantly, discriminates error classes: a 5xx is retried, a 4xx is not, because a malformed payload will be malformed on the next attempt too. All result fields are byte-truncated before transmission so a program printing 100 MB of output can't blow up the callback or the database row.

---

5. Deep Dive — Matchmaking

The pairing loop

Users waiting for a duel live in a Redis sorted set per contest, matchmaking:{contestId}, scored by join timestamp — so the set is inherently ordered by wait time and FIFO fairness comes for free. Profile data (rating, score, streak, previous opponent, active duel) lives alongside as Redis JSON documents.

Every cycle, the service reads all contest queues, hydrates and validates each user (with a Zod schema — malformed or stale entries are dropped rather than crashing the loop), and pairs them:

flowchart TD
    START["Cycle begins"] --> CHECK{"Contest still running?"}
    CHECK -->|"no"| STOP["Idle"]
    CHECK -->|"yes"| SCAN["Read all matchmaking:{contestId} sorted sets"]
    SCAN --> HYDRATE["Hydrate + Zod-validate each user<br/>evict anyone already in an in_progress duel"]
    HYDRATE --> ORDER["Order by queue join time (longest wait first)"]
    ORDER --> LOOP["For each unclaimed user"]
    LOOP --> WAIT{"Waited ≥ betterMatchWaitTime?"}
    WAIT -->|"no"| SKIP["Leave in queue — a better opponent may arrive"]
    WAIT -->|"yes"| CRIT{"Early contest?"}
    CRIT -->|"yes"| RATING["Pair on closest rating"]
    CRIT -->|"no"| SCORE["Pair on closest contest score"]
    RATING --> CAND["Rank candidates by criterion distance"]
    SCORE --> CAND
    CAND --> REMATCH{"Best candidate is<br/>the immediately previous opponent?"}
    REMATCH -->|"no"| TOKEN["Build deterministic match token"]
    REMATCH -->|"yes"| HELD{"Waited ≥ rematchWaitTime?"}
    HELD -->|"no"| EXCLUDE["Exclude — keep looking for a fresh opponent"]
    HELD -->|"yes"| TOKEN
    TOKEN --> POST["POST match to backend"]
    POST --> OUT{"Outcome"}
    OUT -->|"created"| CLAIM["Claim both users, ZREM from queue"]
    OUT -->|"duplicate"| CLAIM
    OUT -->|"skipped_busy"| RELEASE["Claim only the genuinely busy users"]

Loading

Fairness-aware pairing

Three mechanisms, each solving a distinct fairness problem:

1. Repeat-opponent avoidance with a graceful fallback. Facing the same person three times in a row makes a contest feel broken. The matcher tracks each user's prevOpponentId and maintains two candidate rankings simultaneously — a strict best match excluding immediate rematches, and a fallback best match allowing them. The strict match always wins if one exists. The fallback only becomes eligible once the pair has waited past rematchWaitTime. Nobody is starved for the sake of variety — variety is preferred, not enforced at any cost.

2. Shifting match criteria. Early in a contest, nobody has a meaningful contest score, so pairing uses global rating. After a configurable window, it switches to live contest score, which by then reflects actual in-contest performance. Ties on the primary criterion break on the other one.

3. A deliberate wait window. A user is not matched the instant they queue. betterMatchWaitTime holds them briefly so the matcher can consider people who arrive moments later. Trading a couple of seconds of wait for a materially closer opponent is the right call in a rating-based system.

Atomic match creation — the concurrency problem

The dangerous scenario: two pairing cycles overlap, or a retry fires after a timeout, and the same two users get placed into two different duels. That corrupts the leaderboard and strands a player in a ghost match.

Three layers of defence:

Deterministic idempotency token. The matcher builds a token from the sorted user IDs plus both queue-join timestamps, so the same logical pairing always produces the same token regardless of which user is "first":

private buildMatchToken(contestId, firstUser, secondUser): string {
  if (firstUser.id.localeCompare(secondUser.id) <= 0) {
    return `${contestId}:${firstUser.id}:${secondUser.id}:${firstUser.queueJoinTime}:${secondUser.queueJoinTime}`;
  }
  return `${contestId}:${secondUser.id}:${firstUser.id}:${secondUser.queueJoinTime}:${firstUser.queueJoinTime}`;
}

Atomic claim in Redis. The backend converts that token into a single-winner lock:

const lockAcquired = await redisClient.set(matchTokenKey, '1', { NX: true, EX: 120 });
if (lockAcquired !== 'OK') {
  return ApiResponse(res, 200, 'Duplicate match ignored', { outcome: 'duplicate' });
}

SET NX EX is a single atomic Redis operation — no read-then-write race window. The TTL means a crashed request can't wedge a pair permanently, and the lock is explicitly released on any downstream failure so a transient error doesn't block a legitimate retry.

In-process guard + busy check. The matcher additionally holds an in-flight token set (so a slow backend call doesn't get re-attempted within the same cycle), and the backend re-verifies both users are not in an in_progress duel before committing. The response is a typed three-way outcome — created / duplicate / skipped_busy — and each is handled differently: created and duplicate both claim the pair, while skipped_busy returns only the genuinely-busy users to a claimed state and frees the other to match with someone else in the same cycle.

Problem selection

Once paired, the duel needs a problem that is fair to both and new to both. The backend:

1. Averages the two players' live contest scores, normalizes to the nearest 100, and clamps to the available problem-rating band (800–1600).
2. Fetches all problems at that rating.
3. Subtracts the union of both players' usedProblems:{contestId}:{userId} Redis sets.
4. Picks randomly from what remains, falling back to the full candidate list if the players have exhausted the band.
5. Records the choice into both players' used-sets in a single pipelined MULTI.

Duel duration is then derived from the problem's rating (15 / 18 / 21 minutes) — harder problems get more clock.

---

6. Deep Dive — Real-Time Duel State

All live duel state is held in Redis (fast, shared across processes) and continuously checkpointed to PostgreSQL (durable). WebSocket connections are authenticated at handshake time through better-auth middleware — an unauthenticated socket never reaches a handler.

Duel state machine

stateDiagram-v2
    [*] --> in_progress: match created, both sockets joined duel room
    in_progress --> ended: a correct submission lands
    in_progress --> forfeited: a player disconnects and the 30s grace expires
    in_progress --> drawn: clock expires, or both players back out
    in_progress --> abandoned: both players disconnect simultaneously
    ended --> [*]
    forfeited --> [*]
    drawn --> [*]
    abandoned --> [*]

Loading

Disconnect handling — the hardest correctness problem

A player closing their laptop mid-duel must not instantly hand their opponent a win (they may be on flaky campus Wi-Fi), but must also not let them stall the opponent indefinitely.

sequenceDiagram
    participant A as Player A
    participant BE as Backend
    participant R as Redis
    participant B as Player B

    A--xBE: socket disconnect
    BE->>R: HSET disconnectMap[A] = disconnectToken (timestamp)
    BE->>R: If A was queued but not duelling, ZREM from matchmaking
    BE->>R: Is B also in disconnectMap?
    alt Both disconnected
        BE->>R: duel.status = "abandoned"
        BE-->>B: duelStatusChange { abandoned }
    else Only A disconnected
        BE-->>B: opponentDisconnected { willForfeitAt: now + 30s }
        Note over BE: 30-second forfeit timer armed
        alt A reconnects in time
            A->>BE: reconnect (new socket)
            BE->>R: HDEL disconnectMap[A]  ← invalidates the stored token
            BE-->>A: reconnectedToDuel { problem, opponent, remaining time }
            BE-->>B: opponentReconnected
            Note over BE: Timer fires, sees token mismatch, exits as a no-op
        else Grace expires
            BE->>R: Re-read duel — still in_progress?
            BE->>BE: endDuel(reason: forfeit) — Redis + PostgreSQL + leaderboard
            BE-->>B: opponentLeft { you win, +score, streak }
        end
    end

Loading

The mechanism I'm most pleased with is the disconnect token. A naive setTimeout forfeit timer is riddled with races: the player reconnects and disconnects again, the opponent solves the problem during the grace window, the duel ends some other way — and a stale timer fires and forfeits someone who is actively playing.

So each disconnect writes a timestamp token into disconnectMap. When the timer fires 30 seconds later it re-reads the token and compares. Reconnection deletes the key; a newer disconnect overwrites it. Either way the stale timer's comparison fails and it exits harmlessly. On top of that, the timer re-reads the duel from Redis before acting, so a duel that ended by any other means during the window is left alone — and if the opponent had also already forfeited, it resolves as a draw rather than awarding a win to someone who left.

Reconnection

Reconnecting is not just "rejoin the room". The connection handler resolves whatever happened while the player was away and replays the correct state: rejoin the duel room and restore the problem, opponent identity and remaining clock if still in_progress; or report a win, loss, draw, abandonment, or forfeit that concluded during the absence — then clear the stale references so the player can queue again cleanly. Handlers are registered synchronously before any async work in the connection path, so events fired immediately on connect are never dropped.

Abuse guards on the submission path

• Per-user cooldown (cooldown:submit:{userId}, 10s TTL) to stop submit-spamming the judge. Notably, the cooldown is set after all validation passes, so a rejected duplicate doesn't burn the user's next legitimate submission window.
• Duplicate detection — resubmitting byte-identical code to the last submission is rejected without ever reaching the judge.
• Duel ownership check — the submitted duelId must match the user's active duel in Redis.

---

7. Scoring and Leaderboards

Duel scoring is Elo-derived with competitive-programming adjustments:

expected     = 1 / (1 + 10^((loserRating − winnerRating) / 400))
baseScore    = MIN_WIN_SCORE (20) + ELO_K (20) × (1 − expected)
streakBonus  = min(streak × 3, 15)
timeBonus    = round((1 − timeTaken / maxTime) × 10)
total        = max(baseScore, baseScore + streakBonus + timeBonus − penalties)

• Upset-weighted — beating a stronger opponent pays more, exactly as Elo intends.
• Speed-rewarded — solving in a third of the clock earns most of the time bonus.
• Streak-rewarded but capped — 3 points per consecutive win, capped at 15, so a hot streak is worth chasing without running away with the contest.
• Floored — the final max(baseScore, total) guarantees bonuses and penalties can never make a win worth less than the base Elo award.

Live leaderboards are Redis sorted sets (leaderboard:{contestId}), so a score update is a single ZINCRBY and a rank lookup is a single ZREVRANK — both O(log N), no scan, no recompute. Rendering the top-N batches user profile lookups into one JSON.MGET round-trip rather than N individual gets, and every player receives their own rank alongside the top-N even when they're ranked #400.

---

8. Contest Lifecycle and Durability

Redis is the hot path; PostgreSQL is the source of truth. Cron-driven schedulers move contests through their lifecycle and keep the two in sync.

stateDiagram-v2
    [*] --> not_started
    not_started --> preloading: T-2min — scheduler claims the contest
    preloading --> preloaded: participants, problems, leaderboard warmed into Redis
    preloading --> not_started: preload failed — released for retry
    preloaded --> in_progress: start time reached
    in_progress --> in_progress: periodic checkpoint of scores + active duels → PostgreSQL
    in_progress --> ended: duration elapsed
    ended --> backed_up: final leaderboard flushed to PostgreSQL
    backed_up --> [*]

Loading

Preloading is the interesting piece. Cold Redis at the moment 500 users hit "join" would mean a stampede of database reads on the most latency-sensitive minute of the event. Instead, a scheduler running every 30 seconds finds contests starting within the next two minutes and warms all participant profiles, problems, and the leaderboard into Redis before the doors open.

The status field doubles as a distributed lock: a scheduler transitions not_started → preloading before doing the work, so a second instance won't duplicate it, and failure resets the status to not_started so the next tick retries rather than leaving the contest stuck.

Checkpointing runs continuously during the contest — leaderboard scores and in-progress duel state are flushed to PostgreSQL — so a Redis failure mid-contest costs one checkpoint interval, not the entire contest.

---

9. Data Model

erDiagram
    USER ||--o{ SUBMISSION : "submits"
    USER ||--o{ DUEL : "plays as A"
    USER ||--o{ DUEL : "plays as B"
    USER ||--o{ SESSION : "authenticates"
    CONTEST ||--o{ DUEL : "contains"
    PROBLEM ||--o{ DUEL : "assigned to"
    PROBLEM ||--o{ SUBMISSION : "judged against"
    DUEL ||--o{ SUBMISSION : "receives"

    USER {
        text id PK
        text name
        text email UK
        int rating
        json scores "per-contest scores"
        int_array contests "registered contest ids"
        enum role "participant | moderator | admin"
    }
    CONTEST {
        serial id PK
        text contest_name UK
        timestamp start_time
        int duration "minutes"
        enum status "not_started → … → backed_up"
    }
    PROBLEM {
        serial id PK
        text problem_statement
        text_array visible_input "samples shown to players"
        text_array complete_input "hidden judge tests"
        text_array complete_output
        int rating "800–1600"
        text checker_code "testlib special judge"
        int time_limit
        int memory_limit
    }
    DUEL {
        serial id PK
        int contest_id FK
        int problem_id FK
        text player_a_id FK
        text player_b_id FK
        enum status
        text winner_id FK
        timestamp duel_start_time
        int duration
    }
    SUBMISSION {
        serial id PK
        int problem_id FK
        text user_id FK
        int duel_id FK
        enum language
        text code
        text verdict
        int test_cases_passed
        int total_test_cases
        timestamp time_of_submission
    }

Loading

Visible and hidden test cases are stored as separate PostgreSQL arrays on the problem, so the "Run" path can never accidentally leak hidden tests — the two sets are fetched by different code paths with different column selections.

Redis keyspace

Key	Type	Purpose
matchmaking:{contestId}	Sorted set	Waiting players, scored by join time
{userId}	JSON	Live profile: rating, score, streak, duelId, prevOpponentId
duel:{duelId}	JSON	Duel status, players, wrong-answer counts, start time, duration
leaderboard:{contestId}	Sorted set	Live rankings
usedProblems:{contestId}:{userId}	Set	Problem-repeat avoidance
userSocketMap / disconnectMap	Hash	Socket routing and disconnect tokens
matchToken:{contestId}:{token}	String (NX, EX 120)	Match-creation idempotency lock
cooldown:submit:{userId}	String (EX 10)	Submission rate limit
submissionsQueue / runQueue / runQueueCustom :{lang}	List	Judge job queues

---

10. Reliability and Failure Handling

Failure	Handling
Redis unavailable at startup	Both Node services retry with exponential backoff and jitter; log output is throttled so a long outage doesn't flood logs
Redis drops mid-run	Reconnect strategy with capped backoff; the matchmaking loop no-ops on a not-ready client instead of crashing; the judge tolerates 5 consecutive errors before backing off
Backend unreachable from the judge	3 attempts, exponential backoff, 4xx not retried (won't succeed), shared pooled HTTP client
Backend unreachable from the matchmaker	4 attempts with jittered backoff; 429 and 5xx retried, other 4xx not
A sandbox hangs	Layered budgets: per-test-case deadline → whole-container timeout (compile + checker + per-test budget + overhead) → executor context deadline
Container leaked by a crash	Periodic prune of stopped containers, dangling volumes, and dangling images every 30 min — with the compile-cache volume explicitly excluded by label
Process killed mid-contest	Graceful shutdown: signal handler cancels polling, in-flight jobs drain via WaitGroup, the entire warm pool is torn down
Checker bind-mount unavailable	Detected at container-create time, disabled at runtime with a warning, judging continues with runtime-compiled checkers
Duplicate match creation	Deterministic token + SET NX EX atomic claim + busy re-check
Stale forfeit timer	Disconnect-token comparison plus duel re-read before acting
Malformed user data in Redis	Zod validation at the boundary; invalid entries are dropped, not crashed on

---

11. Observability

RCE engine exposes /healthz, which is a real health check rather than a static 200: it pings the Docker daemon and Redis, and returns live pool statistics (current vs target size per language), queue depths for all 12 queues, and in-flight job counts. The same endpoint doubles as the operator's window into autoscaler behaviour.

Backend exposes Prometheus metrics at /metrics via prom-client:

• HTTP request rate, error rate, and duration histograms (RED metrics)
• duel_outcomes_total by outcome, duel_duration_ms histograms
• user_rating_delta, user_streak_distribution
• disconnect_events_total by resolution (reconnected / forfeited / abandoned) and disconnect_timeout_ms — this is what told me the 30-second grace window was tuned correctly
• matchmaking_queue_size gauge per contest

Both services emit structured logs (slog JSON in Go) with job, user, duel, and problem IDs on every line, so a single submission can be traced across services.

Admin portal — a Django app mapped onto the same PostgreSQL schema with unmanaged models, giving organizers a UI to author problems, schedule contests, and inspect duels and submissions without touching the database directly. Django's admin was the right call here specifically because it doesn't own the schema: Drizzle owns migrations, and Django gets a mature CRUD UI for free.

---

12. Load Testing and Results

Judging systems fail in ways unit tests don't catch, so the engine ships with a purpose-built load harness (rce/tests/) that:

• Generates 200+ Codeforces-style jobs (100 problems × AC and WA submissions) across all four languages, including deliberate TLE, MLE, runtime-error, and compile-error cases to verify every verdict path under load.
• Pushes them into the real Redis queues so the whole pipeline is exercised end to end, not mocked.
• Samples docker stats every 2 seconds during the run to record peak memory per container — this is what produced the ~50 MiB working-set figure that the memory-aware admission control is built on.
• Reports throughput (jobs/sec) and P50 / P90 / P99 execution latency, split by verdict class.

Headline results:

Metric	Result
Sustained submissions per run	1000+
Throughput	15+ jobs/sec
Evaluation latency vs cold-start-per-job	~65% reduction (pre-warmed pool)
Burst capacity	5× (4 → 20 containers per language, 16 → 80 total)
Verdict correctness	All classes (AC / WA / TLE / MLE / RE / CE) verified under concurrent load

The load harness is also what surfaced the two most valuable bugs in the project: the in-flight tracking gap in the autoscaler (queue-length-only scaling reading zero demand on a saturated machine), and a memory-accounting error where pool sizing was derived from the Docker security limit rather than observed working set.

---

13. Technology Choices and Trade-Offs

Layer	Choice	Why	Trade-off accepted
Judge	Go	Goroutines + channels map directly onto "N concurrent container lifecycles"; predictable latency; first-class Docker SDK	A second language in the stack
API / real-time	Node.js + TypeScript + Socket.IO	Excellent for many idle-but-live WebSocket connections; shared types with the frontend	Single-threaded — which is precisely why judging was moved out
Broker	Redis lists (BRPOP)	Matchmaking already required Redis; blocking pop is a zero-overhead consumer; queue depth is a free autoscaling signal	Weaker durability than Kafka/RabbitMQ — mitigated by AOF, addressed properly in the roadmap
Hot state	Redis JSON + sorted sets	Leaderboards are literally a sorted set; O(log N) rank queries; shared across processes	Requires disciplined checkpointing to PostgreSQL
Persistence	PostgreSQL + Drizzle ORM	Relational data with real foreign keys; typed queries; versioned migrations	—
Sandbox	Docker + cgroups + seccomp defaults	Namespace and cgroup isolation without a hypervisor; universally deployable	Weaker than a microVM — gVisor/Firecracker is on the roadmap
Frontend	Next.js 15 + React 19 + Tailwind	App Router, server components, Monaco editor integration	—
Auth	better-auth	Shared session validation across REST and the WebSocket handshake	—
Admin	Django admin (unmanaged models)	Full CRUD UI for organizers at near-zero cost, without owning the schema	Schema drift risk — mitigated by keeping models unmanaged and Drizzle authoritative

---

14. Testing Strategy

• Unit tests (Vitest) across backend controllers, services, middleware, socket handlers, and utilities — including the tricky paths: forfeit resolution, soft back-out, reconnection, duel-end state transitions, and leaderboard broadcast.
• Contract tests on the matchmaking controller, pinning the three-way created / duplicate / skipped_busy response shape that the matchmaking service depends on — so the two services can't silently drift apart.
• Integration tests for auth, contest, duel, and leaderboard flows against real Redis and PostgreSQL behaviour.
• E2E tests (Playwright) for auth, contest entry, and full duel flows through the browser.
• Load tests (Go) for the judge, as described above.
• Pre-commit hooks running lint and format on push across the TypeScript services.

---

15. Known Limitations and Roadmap

I'd rather state these plainly than pretend they don't exist.

Current limitations

• Single-node judge. The RCE engine scales up (memory-aware concurrency, autoscaling pool) but not out. Multiple engine instances would work today — they'd simply consume from the same Redis queues — but there's no coordinated capacity view across nodes.
• Queue durability. Redis lists have no acknowledgement semantics. A job popped by an engine that then crashes is lost. Redis Streams with consumer groups, or a dedicated broker, would fix this.
• Docker socket exposure. The engine mounts /var/run/docker.sock. That's a meaningful privilege concentration, mitigated by running the engine on a dedicated node with a minimal surface.
• Backend horizontal scaling requires a Socket.IO Redis adapter to fan events across instances.

Roadmap

1. Redis Streams with consumer groups and a dead-letter queue for at-least-once judging.
2. Socket.IO Redis adapter, enabling multiple backend instances.
3. gVisor or Firecracker microVM sandboxes to replace runc, shrinking container-escape risk.
4. Rootless Docker or a socket proxy to eliminate the privileged mount.
5. Distributed tracing (OpenTelemetry) across the submit → queue → judge → callback path.
6. Multi-node judge orchestration with a shared capacity view.

---

16. Repository Layout

code-duel/
├── backend/                  Node.js · Express · Socket.IO · Drizzle · better-auth
│   └── src/
│       ├── controllers/      matchmaking, submission, run, leaderboard, contest, user
│       ├── socket/           handlers (queue, submit, run, disconnect, back-out, timeout)
│       │   └── utils/        duel-end resolution: forfeit / draw / abandon
│       ├── services/         contest lifecycle, cron schedulers
│       ├── metrics/          Prometheus collectors (HTTP, duel, user, disconnect)
│       ├── models/           Drizzle schema
│       └── tests/            unit · integration · e2e
│
├── rce/                      Go execution engine
│   ├── main.go               queue polling, job budgeting, calibration, health server
│   ├── executor/
│   │   ├── pool.go           pre-warmed container pool, sandbox hardening
│   │   ├── scaling_manager.go demand-proportional autoscaler
│   │   ├── inflight_tracker.go atomic per-language in-flight counters
│   │   └── docker_executor.go exec/HTTP dispatch, cleanup, pruning
│   ├── dockerfiles/          per-language images + the in-container runner
│   ├── benchmarks.go         platform calibration workloads
│   └── tests/                load harness, verdict matrix, log parsing
│
├── matchmaking-service/      Node.js pairing loop
│   └── src/                  matchMaker · redisConf · backendClient · schemas
│
├── frontend/                 Next.js 15 · React 19 · Tailwind · Monaco
│   └── src/                  duel arena, lobby, leaderboard, profile, socket store
│
├── admin-portal/             Django admin over the same PostgreSQL schema
└── Makefile                  orchestrates all four services

---

Engineering Summary

Contribution	Detail
Platform	1v1 competitive coding platform for the BITS Pilani Coding Club, adopted by 500+ users
Sandboxed judge	Go execution engine with pre-warmed, single-use Docker sandboxes — read-only rootfs, all capabilities dropped, no network, PID/memory/CPU/file-size capped, non-root, tmpfs-only writes — cutting evaluation latency ~65%
Autoscaling	Demand-proportional pool scaling from in-flight + queued load, giving 5× burst capacity and sustaining 1000+ submissions at 15+ jobs/sec
Matchmaking	Atomic Redis pairing with deterministic idempotency tokens, SET NX EX claims, and fairness-aware selection that avoids repeat opponents without starving anyone
Real-time state	Socket.IO duel lifecycle with token-guarded disconnect/forfeit resolution, full reconnection recovery, and Redis-backed live leaderboards
Fairness engineering	Per-language platform calibration making time limits hardware-independent; rating-banded, repeat-free problem selection
Operations	Prometheus metrics, deep health checks, contest preloading and continuous checkpointing, and a purpose-built load harness that found real bugs