Skip to content
#

web-spidering

Star

Here are 5 public repositories matching this topic...

Language: All
Filter by language
All 5 Shell 3 JavaScript 1 Python 1

Pryodon / Webspider

Star 9

Cross-platform Python crawler that finds and verifies downloadable media, documents, and other files, then creates wget-ready URL lists for fast bulk downloading. It also scans sitemap trees and generates validated text or XML sitemaps, with HTTP, HTTPS, FTP, persistent SQLite history, resumable crawls, robots support, and no pip dependencies.

scraper spider web-scraper web-scraping webcrawler spiders scraping-websites web-scraping-software webspider web-scanner web-spider web-scrapping webcrawling webcrawl scraping-python scraping-web web-spidering webcrawlers web-scraping-python web-scraping-python-projects
  • Updated Aug 6, 2026
  • Python

Pryodon / Web-Spider-Linux-shell-script

Star 2

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap.txt or XML file for your website!

bash crawler scraper spider web-crawler web-scraper wget wget-utility scrapers bash-script webcrawler web-scrapers spiders webspider bash-scripting web-spider scraping-tool web-spidering webcrawlers scraping-tools
  • Updated Jul 18, 2026
  • Shell

Add this topic to your repo

To associate your repository with the web-spidering topic, visit your repo's landing page and select "manage topics."

Learn more