feat(api): external link rot detection in janitor#81
Open
advancedresearcharray wants to merge 11 commits into
Open
feat(api): external link rot detection in janitor#81advancedresearcharray wants to merge 11 commits into
advancedresearcharray wants to merge 11 commits into
Conversation
Closes kiwifs#423. Opt-in janitor rule scans markdown bodies for external URLs, rate-limits HEAD/GET checks, caches results, and reports broken links in external_links and issue results. Co-authored-by: Cursor <cursoragent@cursor.com>
Add SSRF guards, configurable rate limits, optional domain whitelist, workspace root validation, and expanded regression tests for issue kiwifs#423. Co-authored-by: Cursor <cursoragent@cursor.com>
…test Remove unrelated workspace template scaffolds that broke lint tests and address peer review by adding regression coverage for unreachable URLs. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove unrelated knowledge/runbook/research template files that were accidentally included during cherry-pick; issue kiwifs#423 scope is janitor external link rot detection only. Co-authored-by: Cursor <cursoragent@cursor.com>
Closes acceptance gaps for issue kiwifs#423: verify YAML frontmatter URLs are not probed and cached 404s still surface in external_links on later scans. Co-authored-by: Cursor <cursoragent@cursor.com>
Strengthen external link rot coverage for rate-limit concurrency, User-Agent header, and ScanResult external_links API serialization. Co-authored-by: Cursor <cursoragent@cursor.com>
Log cherry-pick delivery from main with test verification for fleet publish. Co-authored-by: Cursor <cursoragent@cursor.com>
…livery Re-ran janitor and config regression tests after fleet delivery failure; documents passing verification before PR publish. Signed-off-by: Array Fleet <fleet@advancedresearcharray.local> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Array Fleet <fleet@advancedresearcharray.local> Co-authored-by: Cursor <cursoragent@cursor.com>
Block redirect SSRF, sanitize allow/ignore host patterns, refactor probe helpers, and add regression tests for request delay, timeouts, and metadata hosts. Signed-off-by: Array Fleet <fleet@advancedresearcharray.local> Co-authored-by: Cursor <cursoragent@cursor.com>
Hands-on takeover confirms external link rot janitor feature passes all regression tests including SSRF guards, rate limits, and cache behavior. Signed-off-by: Array Fleet <fleet@advancedresearcharray.local> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
external-link-rotthat scans markdown bodies for externalhttp:///https://URLs and HTTP-probes them for rot (4xx/5xx/unreachable).GET /api/kiwi/janitorunderexternal_linksand matchingissuesentries; wired through scheduled janitor,kiwifs check, andkiwifs janitor..kiwi/cache/link-check.jsonwith SSRF guards for private IPs and metadata endpoints.Upstream: kiwifs#423
Test plan
go test ./internal/janitor/... -count=1— external link rot regression testsgo test ./internal/config/... -count=1— external link config parsinggo test ./internal/... -count=1— full internal suite greenexternal_link_check = truein a workspace and confirm broken URLs surface in/api/kiwi/janitorMade with Cursor