fix/webserver: recover gRPC handler panics and reject empty query nodes - #1140
Open
vatsalpatel wants to merge 2 commits into
Open
fix/webserver: recover gRPC handler panics and reject empty query nodes#1140vatsalpatel wants to merge 2 commits into
vatsalpatel wants to merge 2 commits into
Conversation
A gRPC request with an empty query message crashes zoekt-webserver. QFromProto read the oneof with a direct field access and panicked in its default branch, so a request with no query field faulted on p.Query and a bare Q{} hit the explicit panic. Both now return an error, which the three handlers already map to codes.InvalidArgument.
grpc-go does not recover handler panics, and defaults.NewServer installed no recovery interceptor, so any panic below a handler took down the process rather than failing the one RPC. Add recovery.UnaryServerInterceptor and recovery.StreamServerInterceptor from go-grpc-middleware, which is already a direct dependency.
Recovery sits last in both chains, closest to the handler, so the resulting Internal error still reaches the metrics interceptor above it. The handler logs the method, panic value and stack, and returns a flat Internal with no detail: panic values here carry shard paths and repository names the caller may have no access to.
Fixes sourcegraph#1139
Signed-off-by: vatsalpatel <vatsalpatel.me@gmail.com>
keegancsmith
requested changes
Aug 17, 2026
keegancsmith
left a comment
Member
There was a problem hiding this comment.
Nice find on the empty Q, indeed that should of been validated before. Minor inline feedback mostly about cleaning up your agents over eagerness for writing lots of test code.
Author
|
@keegancsmith Thank you for the review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A gRPC request with an empty query message crashes zoekt-webserver.
QFromProtopanics on it, and nothing recovers handler panics, so the process goes down instead of the request failing.QFromProto
The oneof was read with a direct field access and the default branch panicked, so
SearchRequest{}faulted onp.Queryand a bareQ{}hit the explicitpanic. Both now return an error. The three handlers incmd/zoekt-webserver/grpc/server/server.goalready turn a returned error intocodes.InvalidArgument, so there are no caller changes. Nested children are covered too, soAnd{children: [{}]}is rejected rather than taking the server down.Recovery interceptor
defaults.NewServernow installsrecovery.UnaryServerInterceptorandrecovery.StreamServerInterceptorfrom go-grpc-middleware, which is already a direct dependency, sogo.modandgo.sumare unchanged.Recovery sits last in both chains, closest to the handler, so the resulting
codes.Internalstill passes back out through the metrics interceptor and shows up ingrpc_server_handled_total. Note that it does not show up in theinternalerrslogging above it: that interceptor only logs errors that look like they came from go-grpc itself, matching on a"grpc: "prefix and two other specific shapes, and a flat"internal error"matches none of them. The logging is therefore done by the recovery handler, which the comment on the ordering now says explicitly.The trade-off of putting recovery innermost is that a panic inside one of the interceptors above it is still fatal. That seemed like the right balance, since the handlers are the exposed surface and the interceptors are not, but I am happy to move it outermost if you would rather cover everything and give up the metrics accuracy.
The handler logs the panic, its stack and the gRPC method through the existing
sglog.Logger, and returns a flatcodes.Internalwith no detail. Nothing about the panic reaches the caller on purpose: panic values on this path routinely carry shard paths, repository names and indexed file names, for example the corrupt shard reports inindex/contentprovider.go, and the caller may have no access to any of it.Tests
TestServerRecoversHandlerPanicsruns a real server built byNewServerwith a panicking service registered, over a real client on a loopback listener. It covers both the unary and the stream interceptor, asserts the client getsInternalwith neither the panic value nor a stack in the message, and asserts a laterListstill succeeds. Removing the two interceptor lines kills the test binary, which is the point: the wiring is what this change is.TestPanicRecoveryHandlercovers the handler directly, for the status code and the absence of a stack in the client-visible message.TestQFromProtoRejectsMissingNodescovers a nil node, an unset oneof, an unset oneof on a child, a nil child, and a wrapper with no child. Every one of those panicked before this change.go test ./...passes, andgo test ./grpc/defaults/ -race -count=30passes.Fixes #1139