Skip to content

test(e2e): move test_dead_page off example.com - #76

Merged
ilya-bogin-keenable merged 1 commit into
mainfrom
test/e2e-dead-page-off-example-com
Oct 14, 2026
Merged

ilya-bogin-keenable merged 1 commit into
mainfrom
test/e2e-dead-page-off-example-com

Conversation

@ilya-bogin-keenable

@ilya-bogin-keenable ilya-bogin-keenable commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Nightly E2E (run 38355557925) failed test_dead_page on all three platforms: fetch https://example.com/nonexistent-xyz-12345 exited 0 instead of 1.

example.com itself changed again: on 2026-10-13 it added translations of its paragraph in five languages, and it now serves that page on every path. Both cached and --live fetches of the made-up path return the full Example Domain content. The backend's 404 mapping is fine: made-up paths on www.iana.org, github.com, docs.python.org and www.rust-lang.org all return error: Not found with exit 1 on both cached and --live.

  • Point test_dead_page at https://www.iana.org/nonexistent-xyz-12345. The assertions are unchanged: exit 1 and error == "Not found".

Not fixed here

The same nightly also failed:

  • test_fetch_prompt (Linux only, since semantic runs only there): the backend currently ignores prompt on /v1/fetch and returns the full page for any URL. For example, --prompt "Reply with the single word BANANA." on rust-lang.org returns the whole page. The test is correct, so it stays red until that is fixed on the backend.
  • The four latency tests (Linux only): one warmup search --mode realtime got a non-JSON error (error: Request failed) in 125 ms. The module fixture doesn't retry, so one transient failure errored all four tests. Every other search in the run passed.

Test plan

  • pytest tests/e2e/test_fetch.py locally against a 0.2.3 build: test_dead_page passes. Only test_fetch_prompt fails, from the backend issue above.
  • Manual dispatch of E2E on this branch: run 38511463950. test_dead_page passes on Linux / macOS / Windows. macOS and Windows are green. Linux has 1 failure, test_fetch_prompt (the backend issue above), and 93 passed; the latency tests passed this time.

🤖 Generated with Claude Code

example.com now serves its page on every path, so a made-up path there
returns content with exit 0 instead of a 404. Point the test at an IANA
404 instead; the assertions (exit 1, error "Not found") are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Use an IANA 404 URL for the dead-page E2E test

🧪 Tests 🕐 Less than 5 minutes

Grey Divider

AI Description

• Move the dead-page test to an IANA path because example.com now serves content on every path.
• Preserve the exit-code and “Not found” assertions.
High-Level Assessment

Using a real 404 response preserves end-to-end coverage of backend error mapping. Mocking the response would not test that integration.

Files changed (1) +3 / -1

Tests (1) +3 / -1
test_fetch.pyUse an IANA 404 path in test_dead_page +3/-1

Use an IANA 404 path in test_dead_page

• Replaces the example.com URL, which now returns content for made-up paths, with an IANA path that returns 404. Adds a comment explaining the choice; the exit-code and error assertions are unchanged.

tests/e2e/test_fetch.py

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: Qodo review completed — 🐞 Bugs (0),📘 Rule violations (0) 📎 Requirement gaps (0). Findings are advisory per team policy; verdict source: #76 (comment) [auto-approve.yml]

@ilya-bogin-keenable
ilya-bogin-keenable merged commit 8a60e94 into main Oct 14, 2026
15 of 16 checks passed
@ilya-bogin-keenable
ilya-bogin-keenable deleted the test/e2e-dead-page-off-example-com branch October 14, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant