fix(fetch): add timeout to robots.txt request in autonomous fetch check - #4776
Open
DawnofGenX wants to merge 1 commit into
Open
fix(fetch): add timeout to robots.txt request in autonomous fetch check#4776DawnofGenX wants to merge 1 commit into
DawnofGenX wants to merge 1 commit into
Conversation
The content-fetch request sets timeout=30, but the robots.txt request issued on every autonomous fetch had no timeout, so a slow or unresponsive robots.txt could hang the check indefinitely. Add the same 30s timeout and a regression test asserting it is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The content-fetch request in
fetch_urlsetstimeout=30, but the robots.txt request issued bycheck_may_autonomously_fetch_url(which runs on every autonomous fetch) had no timeout. A slow or unresponsiverobots.txtcould therefore hang the autonomous-fetch check indefinitely, with no way for the caller to recover.This adds the same 30s timeout to the robots.txt GET so it fails fast (surfacing as the existing connection-error path) instead of hanging, and adds a regression test asserting the timeout is present.
Changes
src/fetch/src/mcp_server_fetch/server.py: addtimeout=30to the robots.txtclient.get(...)call incheck_may_autonomously_fetch_url.src/fetch/tests/test_server.py: newtest_robots_txt_request_uses_timeoutasserting the robots.txt GET carriestimeout=30.Testing
uv run pytest -q→ 21 passed (was 20; +1 new regression test)uv run ruff check .→ All checks passeduv run pyright→ 0 errors, 0 warnings