Skip to content

[CRITICAL] Fix metadata extraction inconsistencies in scraped problems #54

Description

@AVPthegreat

Problem: Problem metadata (titles, descriptions, constraints, I/O samples) is inconsistent across scrapers. Some problems have complete data while others are missing critical fields, leading to incomplete or unusable problem bundles.\n\nImpact: Users receive ZIP files with partial or missing problem data, making the tool unreliable for actual use.\n\nRoot Cause:\n- Scraper selectors may be outdated or fragile\n- HTML structure variations not handled\n- Missing fallback logic for alternative layouts\n- No validation before including problems in output\n\nAcceptance Criteria:\n- ✅ Add metadata completeness validator before including problems in ZIP\n- ✅ Filter out problems missing title, statement, or I/O samples\n- ✅ Log which problems were skipped and why\n- ✅ Update scraper selectors for all platforms to handle common layout variants\n- ✅ Ensure at least 80% of scraped problems have complete metadata\n- ✅ Add unit tests validating metadata completeness for each scraper\n\nPriority: P0 - Blocks core functionality

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical prioritybugSomething isn't workingscraperPlatform scrapers

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions