test: block outbound network access from the test suite - #256
Merged
Conversation
A gap in a mock currently fails silently in the worst possible way: the request escapes to the real api.spond.com instead of erroring. That makes the suite non-hermetic and non-deterministic, and a test that reaches the network can pass for the wrong reason as easily as it can fail. Add pytest-socket and disable sockets by default, so an unmocked request fails immediately with SocketBlockedError naming the host it tried to resolve, rather than depending on network conditions or credentials the CI runner does not have. `--allow-unix-socket` is required: asyncio builds its event loop self-pipe with socket.socketpair() (AF_UNIX on Linux), so blocking every socket would break the event loop and with it every async test. Verified by attempting a real aiohttp GET to api.spond.com under the guard: blocked at socket.getaddrinfo. The existing 30 tests are unaffected and still pass, confirming the suite is already hermetic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
pytest-socketand disables sockets for the whole suite, so a gap in a mock fails immediately instead of sending a real request toapi.spond.com.Motivation
This is not hypothetical. CI on #236 currently fails like this:
That JSON came from the production Spond API — a GitHub Actions runner made a real outbound call because a change moved
_get_entity()onto an HTTP path that the test's mocks did not cover. The test failed for the right reason there, but only by luck: an unmocked request that happens to succeed passes the test for entirely the wrong reason, and the result depends on network conditions and credentials CI does not have.Under this guard the same situation fails immediately and legibly:
Why
pytest-socketrather than aconftest.pyfixtureA hand-rolled guard typically patches
socket.socket.connect, which has real gaps: aiohttp resolves DNS throughloop.getaddrinfobefore connecting, and asyncio connects non-blocking sockets via the selector. A guard with holes is worse than no guard, because it manufactures false confidence.pytest-socketintercepts at socket creation andgetaddrinfo, which is where the traffic actually starts.Why
--allow-unix-socketLoad-bearing, not decoration. asyncio's
BaseSelectorEventLoopbuilds its self-pipe withsocket.socketpair(), which isAF_UNIXon Linux. Blocking every socket unconditionally breaks the event loop itself, and with it every async test in the suite.Escape hatch
A test that genuinely needs the network can opt in with
@pytest.mark.enable_socket. Nothing in the suite currently does.Testing
poetry run pytest→ 30 passed with the guard active, confirming the existing suite is already hermetic and the AF_UNIX exception is correctaiohttpGET tohttps://api.spond.com/...was blocked atsocket.getaddrinfowithSocketBlockedErrornaming the host. Probe removed before commitpoetry run ruff check→ all checks passedpoetry run ruff format --check→ 13 files already formatted🤖 Generated with Claude Code