安全、可复现的数据采集与解码作品集。
A safe, reproducible portfolio for web extraction and data decoding.
本项目只演示公开或已获授权的数据处理，不提供验证码、访问控制、密码、软件授权、DRM 或加密绕过。
This project handles public or authorized data only. It does not provide CAPTCHA, access-control, password, license, DRM, or encryption bypasses.
| Project | CLI | Web demo | Output |
|---|---|---|---|
| Static List Scraper / 公开列表采集 | Pagination, deduplication | Bundled two-page fixture | .xlsx |
| Dynamic Page Scraper / 动态网页采集 | Playwright rendering | Bundled JavaScript fixture | UTF-8 .csv |
| Data Decoder / 数据解码工具 | Base64, URL, Unicode, JSON, Protobuf | Text processing panel | Text / JSON |
所有自动化测试使用本地合成数据。CLI 中的静态采集命令默认使用专门用于练习的 quotes.toscrape.com；网页演示不依赖外网。
All automated tests use local synthetic data. The static CLI defaults to the purpose-built practice site quotes.toscrape.com; the web demo does not require external network access.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
python -m playwright install chromiumpython3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
python -m playwright install chromiumRun tests / 运行测试：
python -m pytest -qStart the web portfolio / 启动网页作品集：
uvicorn data_toolkit.web.app:create_app --factory --host 127.0.0.1 --port 8000Open http://127.0.0.1:8000.
Decode Base64 / 解码 Base64：
data-toolkit decode base64-decode 5L2g5aW9
# 你好Format nested JSON / 格式化 JSON：
data-toolkit json-format '{"user":{"name":"Ada"}}'Collect a practice list page / 采集练习列表页：
data-toolkit static --pages 2 --output output/static-records.xlsxRun the dynamic demo after serving sample_data/ locally / 启动本地样例服务后采集动态页面：
python -m http.server 9000 --directory sample_data
data-toolkit dynamic --url http://127.0.0.1:9000/dynamic.html --clicks 1 --output output/dynamic-records.csvDecode the bundled Protobuf example / 解析 Protobuf 样例：
data-toolkit protobuf sample_data/person.bin sample_data/person.proto PersonThe included payload hex is 0a034164611025. Convert it to sample_data/person.bin before running the command, or supply your own authorized payload.
docker compose up --buildOpen http://127.0.0.1:8000. The container runs as the non-root pwuser account and stores temporary exports in a named volume.
src/data_toolkit/
├─ static_scraper/ # HTML parser, bounded HTTP service, XLSX export
├─ dynamic_scraper/ # Playwright service, CSV export
├─ decoder/ # Text, JSON, and schema-guided Protobuf tools
├─ web/ # FastAPI routes, Jinja2 templates, CSS
├─ cli.py # Typer commands
├─ config.py # Limits and allowlists
└─ errors.py # User-facing domain errors
Domain modules do not depend on the CLI or web interface. Both adapters call the same tested functions.
- Confirm the target is public or that you have written authorization.
- Review its terms, robots guidance, rate limits, and API options.
- Copy the relevant parser into a new, target-specific module.
- Replace selectors using a small local HTML fixture first.
- Set strict page, request, timeout, and output limits.
- Never commit Cookie, Token, proxy credentials, customer code, or private datasets.
Do not turn the hosted web interface into an arbitrary-URL proxy. Keep public demos restricted to bundled fixtures or an explicit practice-site allowlist.
The decoder intentionally supports a small proto3 subset: one or more top-level messages with scalar string, bytes, bool, int32, int64, uint32, and uint64 fields. It requires a matching schema and does not guess encrypted or unknown formats.
- Uploaded or decoded content is not persisted by the application.
- Web demos do not accept cookies, request headers, browser scripts, selectors, or arbitrary filesystem paths.
- Page counts, upload sizes, navigation timeouts, and output sizes are bounded.
- Debug screenshots remain local and are excluded from Git.
MIT. Before publishing, replace the generic copyright holder only if you want your own name or organization displayed.