Skip to content

fix(markdown): resolve relative HTML base URLs - #2352

Open
ktz03 wants to merge 1 commit into
unclecode:developfrom
ktz03:fix/relative-html-base-url-0639
Open

ktz03 wants to merge 1 commit into
unclecode:developfrom
ktz03:fix/relative-html-base-url-0639

Conversation

@ktz03

@ktz03 ktz03 commented Oct 9, 2026

Copy link
Copy Markdown

Summary

Relative <base href> values discard the document origin during Markdown generation, producing incorrect link, image, and citation URLs. Resolve the base element against the document URL once; for example, ../assets/ on https://example.test/pages/index.html now resolves guide.html to https://example.test/assets/guide.html.

List of files changed and why

  • crawl4ai/html2text/__init__.py: resolve relative base URLs in HTML2Text and let CustomHTML2Text delegate without overwriting or resolving the base twice.
  • tests/unit/test_markdown_base_url.py: exercise both converters and the public Markdown generator, with relative bases and absolute, absent, and empty base controls.

How Has This Been Tested?

On Windows with Python 3.12.14:

python -m pytest tests/unit/test_markdown_base_url.py tests/test_table_gfm_compliance.py tests/test_table_rowspan_grid.py tests/test_markdown_generator_validation_1880.py tests/general/test_async_markdown_generator.py tests/general/test_content_source_parameter.py -q
67 passed

The new regression suite has 16 failures and 12 passing controls on the unchanged develop base; all 28 cases pass with the fix. It covers path-relative, parent-relative, root-relative, and protocol-relative base URLs, with citations enabled and disabled. An independent public-API check also passes 96 scenarios across document URLs, base URLs, link forms, and both citation modes. Black checks pass for the new test file and git diff --check passes. A scoped source Black check reports four existing whitespace-only lines on both unchanged develop and the fixed code. Browser, live crawling, and the full repository test suite were not run.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

No documentation change is needed for correcting the existing base URL behavior. The selected local suites pass as detailed above; the full suite was not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant