Skip to content

Fix/scrape empty fields - #23

Merged
HC-ONLINE merged 4 commits into
mainfrom
fix/scrape-empty-fields
Jul 29, 2026
Merged

Fix/scrape empty fields#23
HC-ONLINE merged 4 commits into
mainfrom
fix/scrape-empty-fields

Conversation

@HC-ONLINE

Copy link
Copy Markdown
Owner

Description
Fix scrape results returning empty data for content_type, headers, cookies, links, images, forms, and scripts. The core scrapers (static and dynamic) were not capturing HTTP response metadata or extracting HTML elements, and the service layer hardcoded all these fields to empty defaults ("", {}, []).
Changes:

  • core/scraping/static.py: Extended ScrapeResult dataclass with 8 new fields (content_type, headers, cookies, title, links, images, forms, scripts). Updated scrape() to capture HTTP response headers, cookies, content-type, and extract title/links/images/forms/scripts from parsed HTML.
  • core/scraping/dynamic.py: Extended DynamicScrapeResult with 5 new fields. Updated scrape() to extract title/links/images/forms/scripts from Playwright-rendered HTML.
  • core/scraping/extractor.py: Added extract_forms() and extract_scripts() functions (forms extracts action/method/fields; scripts extracts src/type/attributes).
  • core/scraping/init.py: Exported the 2 new extraction functions.
  • services/scrape_service.py: Updated _scrape_static() and _scrape_dynamic() to populate all fields from core results instead of hardcoding empty defaults. Converts core dicts to export Pydantic models (LinkInfo, ImageInfo, FormInfo, ScriptInfo).
    Related Issue
    Addresses scrape data incompleteness: content_type always empty, headers/cookies always {}, links/images/forms/scripts always [].
    Type of Change
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
    Checklist
  • I have followed the style guidelines of this project (Ruff & Pyright)
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly
    Verification
  • ruff check — 0 errors
  • pyright — 0 errors
  • pytest tests/unit/ — 748/748 passed

@HC-ONLINE
HC-ONLINE merged commit 973f068 into main Jul 29, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant