Skip to content

fix: a failing pprof listener must not kill the media server (#64 item 1) - #69

Merged
jason-shen merged 1 commit into
streamcoreai:mainfrom
vsolano9:fix/64-debug-listener-must-not-kill-server
Aug 25, 2026
Merged

fix: a failing pprof listener must not kill the media server (#64 item 1)#69
jason-shen merged 1 commit into
streamcoreai:mainfrom
vsolano9:fix/64-debug-listener-must-not-kill-server

Conversation

@vsolano9

Copy link
Copy Markdown
Contributor

Item 1 of #64. Independent of the rest of that list; items 3–10 are untouched.

The defect

startDebugServer's goroutine ended in log.Fatalf, which is os.Exit(1):

if err := srv.Serve(listener); err != nil && err != http.ErrServerClosed {
    log.Fatalf("debug server error: %v", err)
}

That skips sm.CloseAll() and both graceful shutdowns in main, so an accept error on an optional profiling socket drops every live WebRTC call with no cleanup. As the issue puts it, this mirrors the main HTTP server's pattern in a place where the reasoning does not carry: the process genuinely cannot do its job without the media listener. It can without pprof.

Now log.Printf, and the server keeps running without profiling.

Making it testable

The goroutine literal is lifted into a named serveDebug(srv, listener) so its return is observable. Nothing else moved.

Two tests, because the obvious one is not sufficient on its own:

  • TestServeDebugSurvivesAFailedListener drives serveDebug with a listener whose Accept returns a permanent error, and requires the call to return. The error is deliberately not a net.Error with Temporary() == truenet/http retries those with a backoff, so a temporary error would hang rather than exercise the exit path.
  • TestPublicMuxStillServesAfterTheDebugListenerDies goes through the real startDebugServer, stops the debug listener under its own Serve loop, then requires the public mux to still answer /health. On its own the first test would be satisfied by a serveDebug that nothing calls; this one asserts the thing an operator actually cares about.

Red proof

Reverting only the log.Printf back to log.Fatalf, with both tests in place:

=== RUN   TestServeDebugSurvivesAFailedListener
FAIL	github.com/streamcoreai/streamcore-server	0.355s
FAIL

Worth reading closely: there is no --- FAIL: line. The test does not report a failure — the test binary is killed mid-run, which is the defect reproduced inside the suite. A reader who greps for --- FAIL would conclude nothing ran.

Validation

The exact CI commands from .github/workflows/ci.yml:

  • gofmt -l . — clean
  • go build ./... — clean
  • go vet ./... — clean
  • go test -race ./...ok across all 15 packages

Not claimed

I have not reproduced a real accept failure on a live socket; the listener is a stub, which is what makes the test hermetic. And this changes only the crash behaviour — a failed pprof listener now leaves the server running without profiling, silently apart from the log line. If you would rather it also surface in /health or a metric, that is a design call I did not want to presume, and it belongs with item 7 (warning when profiling settings cannot take effect) rather than here.

Item 1 of streamcoreai#64.

The goroutine in startDebugServer ended in log.Fatalf, which is
os.Exit(1). That path skips sm.CloseAll() and both graceful shutdowns
in main, so an accept error on an optional profiling socket dropped
every live WebRTC call with no cleanup. The main HTTP server keeps
log.Fatalf because the process genuinely cannot serve without that
listener; pprof is opt-in and it can.

Now log.Printf, and the server keeps running without pprof.

The goroutine literal is lifted into a named serveDebug so the return
is observable. Two tests:

- TestServeDebugSurvivesAFailedListener drives serveDebug with a
  listener whose Accept returns a permanent error — the shape that
  actually terminates Serve, since net/http retries temporary ones —
  and requires the call to return.
- TestPublicMuxStillServesAfterTheDebugListenerDies goes through the
  real startDebugServer, stops the debug listener under its own Serve
  loop, and requires the public mux to still answer /health. On its
  own the first test would be satisfied by a serveDebug nobody calls;
  this one asserts the operator-visible claim.

Red proof: with only the log.Printf reverted to log.Fatalf, the first
test does not report a failure — it kills the test binary. `=== RUN`
prints, no result line follows, and the package reports FAIL. That is
the defect itself, reproduced inside the suite.

gofmt clean, go build ./... and go vet ./... clean, and
`go test -race ./...` (the CI command) passes across all 15 packages.

Items 3 through 10 of streamcoreai#64 are deliberately untouched; this is one
independent item as the issue invites.
@jason-shen

Copy link
Copy Markdown
Member

looks good to me, thanks @vsolano9

@jason-shen
jason-shen merged commit ca0cb33 into streamcoreai:main Aug 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants