Skip to content

Commit ba8f3aa

Browse files
devilscrapesdevilscrapes
committed
GitHub trending scraper in Python
0 parents  commit ba8f3aa

12 files changed

Lines changed: 554 additions & 0 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Your Apify API token: https://console.apify.com/settings/integrations
2+
APIFY_TOKEN=

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
test:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
matrix:
15+
python-version: ["3.11", "3.13"]
16+
steps:
17+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
18+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
19+
with:
20+
python-version: ${{ matrix.python-version }}
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: pytest -q

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
.env
2+
results.json
3+
results.csv
4+
__pycache__/
5+
.venv/
6+
.pytest_cache/
7+
.ruff_cache/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Devil Scrapes
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
<p align="center"><a href="https://apify.com/DevilScrapes/github-trending-scraper"><img src="assets/banner.svg" alt="GitHub Trending Scraper — Python example" width="100%"></a></p>
2+
3+
# GitHub Trending Scraper in Python: trending repositories as a pandas-ready dataset
4+
5+
<p align="center">
6+
<a href="https://apify.com/DevilScrapes/github-trending-scraper"><img alt="Run on Apify" src="https://img.shields.io/badge/Run%20on-Apify-DC2626?logo=apify&logoColor=white"></a>
7+
<img alt="Python" src="https://img.shields.io/badge/Python-3.11%2B-3776AB?logo=python&logoColor=white">
8+
<a href="LICENSE"><img alt="MIT license" src="https://img.shields.io/badge/license-MIT-0A0606"></a>
9+
<a href="https://github.com/devilscrapes/github-trending-scraper-python/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/devilscrapes/github-trending-scraper-python/actions/workflows/ci.yml/badge.svg"></a>
10+
</p>
11+
12+
GitHub's REST and GraphQL APIs know everything about a repository except whether it is trending; that list only exists as a web page. This repository is a compact, tested Python client for the **[GitHub Trending Scraper](https://apify.com/DevilScrapes/github-trending-scraper)** on Apify, which turns that page into rows you can analyse. Choose a period (today, this week or this month), optionally a language, and you receive each listed project with its rank, the stars it collected during that period, its lifetime stars, forks and primary language. The Actor runs in Apify's cloud, so the script on your side is a few dozen lines and needs no GitHub credentials. Output lands as `results.json` and `results.csv`, shaped for a DataFrame, a notebook cell or a nightly cron job.
13+
14+
> **Actor:** [GitHub Trending Scraper](https://apify.com/DevilScrapes/github-trending-scraper) · **Publisher:** [Devil Scrapes](https://apify.com/DevilScrapes) · **Also in JavaScript:** [github-trending-scraper-javascript](https://github.com/devilscrapes/github-trending-scraper-javascript)
15+
16+
## 🔥 What you get
17+
18+
- **One row for every repository on the list**, with its position, the `owner/name` pair split into separate columns, the link, the tagline, the primary language, star and fork counts and the contributor avatars GitHub shows on the card.
19+
- **Star velocity as its own number**: `stars_in_window` counts only the stars earned during the chosen period, kept apart from the lifetime `stars_total`, so you never diff two snapshots to estimate momentum.
20+
- **Three periods, two filters**: switch `since` between `daily`, `weekly` and `monthly`, and narrow by programming language, spoken language or both, all producing the same row shape.
21+
- **GitHub's ordering survives export**: the `rank` column keeps position 1 at position 1 after the rows pass through CSV, a spreadsheet or a database sort.
22+
- **No GitHub token or rate budget of your own**: requests go out from the Apify run, and throttled or failed page loads are retried with backoff before anything is written.
23+
- **Every row is stamped** with the window it came from and a UTC collection time, which makes stacking scheduled snapshots into a history table straightforward.
24+
25+
## 🎯 Use cases
26+
27+
### Measure which ecosystems are gaining attention
28+
29+
Run one job per language you care about, concatenate the CSVs and sum `stars_in_window` by `language`. A weekly cron that appends to a Parquet file gives you a time series of developer attention per ecosystem after a couple of months, without hand-copying anything from the browser.
30+
31+
### Spot breakout projects early
32+
33+
Divide the stars gained in the period by the lifetime total and sort descending. Young repositories with a high ratio are the ones growing fastest relative to their size, a list analysts and investors usually assemble manually from several tabs.
34+
35+
### Produce a weekly digest from a notebook
36+
37+
Pull the `weekly` window for Python or Rust on Friday morning, filter out repositories you have already featured, and export the remainder to Markdown or HTML for a newsletter. The `rank`, `description` and `html_url` columns cover most of what a digest entry needs.
38+
39+
### Keep a historical archive of the list
40+
41+
GitHub only shows the current state of the page. Saving each scheduled run with its `scraped_at` timestamp builds a record you can query later, for example to find every day a given repository appeared and how high it ranked.
42+
43+
## ⚡ Quick start
44+
45+
You need an Apify account and an API token from [Console → Settings → Integrations](https://console.apify.com/settings/integrations). Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.
46+
47+
```bash
48+
git clone https://github.com/devilscrapes/github-trending-scraper-python.git
49+
cd github-trending-scraper-python
50+
python -m venv .venv && source .venv/bin/activate
51+
pip install -r requirements.txt
52+
export APIFY_TOKEN=your_token_here
53+
python main.py # uses input.json
54+
python main.py my-input.json --out data --max-charge 2
55+
```
56+
57+
The script starts the [GitHub Trending Scraper](https://apify.com/DevilScrapes/github-trending-scraper), waits for it to finish, prints a link to the run in Apify Console and saves every result to `results.json` and `results.csv`. It exits with code 1 and the run's status message if the run fails.
58+
59+
## 🧩 The core call
60+
61+
The minimal integration is shown below. `apify-client` starts the Actor, blocks until the run completes and then iterates over the dataset, so a single loop gets you every trending row.
62+
63+
```python
64+
from apify_client import ApifyClient
65+
66+
client = ApifyClient("YOUR_APIFY_TOKEN")
67+
run = client.actor("DevilScrapes/github-trending-scraper").call(run_input={'since': 'weekly', 'language': 'rust', 'maxResults': 10, 'proxyConfiguration': {'useApifyProxy': True}})
68+
for item in client.dataset(run.default_dataset_id).iterate_items():
69+
print(item)
70+
```
71+
72+
## 📥 Input
73+
74+
`main.py` forwards `input.json` to the Actor without changes. The bundled file asks for the weekly Rust list and caps it at 10 repositories, a quick and inexpensive way to see real rows. Edit `since` or `language`, or clear `language` to get the cross-language list.
75+
76+
```json
77+
{
78+
"since": "weekly",
79+
"language": "rust",
80+
"maxResults": 10,
81+
"proxyConfiguration": {
82+
"useApifyProxy": true
83+
}
84+
}
85+
```
86+
87+
| Field | Type | Required | Default | Allowed values |
88+
|---|---|---|---|---|
89+
| `since` | string | no | `"daily"` | `daily`, `weekly`, `monthly` |
90+
| `language` | string | no | `""` | — |
91+
| `spokenLanguage` | string | no | `""` | — |
92+
| `maxResults` | integer | no | `25` | — |
93+
| `proxyConfiguration` | object | no | `{"useApifyProxy": true}` | — |
94+
95+
## 📤 Output
96+
97+
Each dataset row represents a single repository as it appeared on the page at collection time. Counts are integers, the language and tagline can be `null` when GitHub omits them, and `built_by` is a list of avatar URLs that may be empty.
98+
99+
| Field | Type | Always present |
100+
|---|---|---|
101+
| `rank` | integer | yes |
102+
| `full_name` | string | yes |
103+
| `owner` | string | no |
104+
| `name` | string | no |
105+
| `html_url` | string | yes |
106+
| `description` | string or null | no |
107+
| `language` | string or null | no |
108+
| `stars_total` | integer | no |
109+
| `stars_in_window` | integer | no |
110+
| `forks` | integer | no |
111+
| `built_by` | array | no |
112+
| `scraped_at` | string | no |
113+
| `window` | string | no |
114+
115+
## 💰 Pricing
116+
117+
| Event | Price (USD) |
118+
|---|---|
119+
| Actor start | $0.2 |
120+
| Result emitted | $0.002 |
121+
122+
Prices as configured on 2026-08-20. The [Actor page](https://apify.com/DevilScrapes/github-trending-scraper) always shows the current price. Runs are billed to the Apify account that owns the token. Use `--max-charge` to cap what a single run can cost.
123+
124+
## 🛠️ Working with the data in Python
125+
126+
This pandas example (`pip install pandas`) computes a momentum ratio, ranks languages by stars earned in the window and writes the ten fastest-growing repositories to a CSV.
127+
128+
```python
129+
import json
130+
131+
import pandas as pd
132+
133+
rows = pd.DataFrame(json.load(open("results.json", encoding="utf-8")))
134+
rows["language"] = rows["language"].fillna("unknown")
135+
rows["momentum"] = rows["stars_in_window"] / rows["stars_total"].clip(lower=1)
136+
137+
by_language = (
138+
rows.groupby("language")["stars_in_window"]
139+
.agg(["count", "sum"])
140+
.sort_values("sum", ascending=False)
141+
)
142+
print(by_language.to_string())
143+
144+
breakouts = rows.sort_values("momentum", ascending=False).head(10)
145+
breakouts[["rank", "full_name", "stars_in_window", "stars_total", "momentum", "html_url"]].to_csv(
146+
"breakouts.csv", index=False
147+
)
148+
print(breakouts[["full_name", "stars_in_window", "momentum"]].to_string(index=False))
149+
```
150+
151+
## ❓ FAQ
152+
153+
### Why not call the GitHub API for this?
154+
155+
Neither the REST nor the GraphQL API exposes the Trending list. You can search repositories by star count or creation date, but that is a different ranking from the one GitHub publishes on the page. This Actor reads the page itself, so your numbers match what visitors see.
156+
157+
### Do I need a GitHub personal access token?
158+
159+
No. The only credential is your Apify API token. The run does not touch your GitHub account or consume its API rate limit.
160+
161+
### How many repositories does one run return?
162+
163+
GitHub usually lists around 25 projects per period and language combination, and `maxResults` can trim that further. For a broad survey, run the script once per language and combine the CSVs in pandas.
164+
165+
### Can I schedule it from cron or a notebook?
166+
167+
Yes. Call `main.main(["input.json", "--out", "snapshots/2026-10-18"])` from a notebook, or put `python main.py` in a crontab. A nonzero exit code signals a failed run, so a scheduler can alert on it.
168+
169+
### Does it cover the trending developers tab?
170+
171+
No. Only the repositories list is collected. Topic filters are not offered either, because the Trending page itself has none.
172+
173+
## 🧯 Troubleshooting
174+
175+
- **`Set APIFY_TOKEN first`**: export the token in the same shell, or put it in a local `.env` file your shell loads. Never commit it.
176+
- **Run finished with `FAILED` or `ABORTED`**: open the printed Console link; the run log says why. Input validation errors name the field.
177+
- **Fewer rows than expected**: a `--max-charge` cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
178+
- **Something looks wrong in the data**: open an issue on the [Actor page](https://apify.com/DevilScrapes/github-trending-scraper) so the maintainers see it.
179+
180+
## 🔗 More from Devil Scrapes
181+
182+
- [GitHub Trending Scraper on Apify Store](https://apify.com/DevilScrapes/github-trending-scraper)
183+
- [JavaScript version of this example](https://github.com/devilscrapes/github-trending-scraper-javascript)
184+
- [All Devil Scrapes Actors](https://apify.com/DevilScrapes)
185+
- [Apify web scraping cookbook](https://github.com/devilscrapes/apify-web-scraping-cookbook)
186+
187+
Related Actors:
188+
189+
- [GitHub Repo Scraper](https://apify.com/DevilScrapes/github-repo-scraper) — fuller metadata for any repository that catches your eye on Trending.
190+
- [GitHub Organization Scraper](https://apify.com/DevilScrapes/github-org-scraper) — profile data for the organizations behind the projects that keep climbing.
191+
- [Hacker News Scraper](https://apify.com/DevilScrapes/hacker-news-scraper) — see whether a climbing repository is also being discussed on Hacker News.
192+
193+
---
194+
195+
MIT licensed example code. The Actor itself runs on [Apify](https://apify.com/DevilScrapes/github-trending-scraper) and is maintained by [Devil Scrapes](https://apify.com/DevilScrapes). Not affiliated with GitHub, Inc..

‎assets/banner.svg‎

Lines changed: 41 additions & 0 deletions
Loading

‎input.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"since": "weekly",
3+
"language": "rust",
4+
"maxResults": 10,
5+
"proxyConfiguration": {
6+
"useApifyProxy": true
7+
}
8+
}

‎main.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
"""Run the GitHub Trending Scraper on Apify and save its results as JSON and CSV.
2+
3+
Actor page: https://apify.com/DevilScrapes/github-trending-scraper
4+
"""
5+
6+
from __future__ import annotations
7+
8+
import argparse
9+
import json
10+
import os
11+
import sys
12+
from decimal import Decimal
13+
from pathlib import Path
14+
15+
from apify_client import ApifyClient
16+
17+
ACTOR_ID = "DevilScrapes/github-trending-scraper"
18+
TERMINAL_OK = "SUCCEEDED"
19+
20+
21+
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
22+
parser = argparse.ArgumentParser(description="Run the GitHub Trending Scraper on Apify.")
23+
parser.add_argument("input", nargs="?", default="input.json", help="Actor input JSON file (default: input.json)")
24+
parser.add_argument("--out", default=".", help="Folder for results.json and results.csv (default: here)")
25+
parser.add_argument("--max-charge", type=Decimal, help="Stop the run once it has cost this many USD")
26+
parser.add_argument("--memory", type=int, help="Run memory in MB (default: the Actor's own setting)")
27+
return parser.parse_args(argv)
28+
29+
30+
def load_input(path: str) -> dict:
31+
try:
32+
data = json.loads(Path(path).read_text(encoding="utf-8"))
33+
except FileNotFoundError:
34+
sys.exit(f"Input file not found: {path}")
35+
except json.JSONDecodeError as error:
36+
sys.exit(f"Input file {path} is not valid JSON: {error}")
37+
if not isinstance(data, dict):
38+
sys.exit(f"Input file {path} must contain a JSON object")
39+
return data
40+
41+
42+
def main(argv: list[str] | None = None) -> int:
43+
args = parse_args(argv)
44+
token = os.environ.get("APIFY_TOKEN")
45+
if not token:
46+
sys.exit("Set APIFY_TOKEN first. Get a token at https://console.apify.com/settings/integrations")
47+
48+
run_input = load_input(args.input)
49+
client = ApifyClient(token, api_url=os.environ.get("APIFY_API_URL", "https://api.apify.com"))
50+
51+
print(f"Starting {ACTOR_ID} ...")
52+
run = client.actor(ACTOR_ID).call(
53+
run_input=run_input,
54+
max_total_charge_usd=args.max_charge,
55+
memory_mbytes=args.memory,
56+
)
57+
if run is None:
58+
sys.exit("The run could not be started.")
59+
60+
print(f"Run {run.id} finished: {run.status}")
61+
print(f"Details: https://console.apify.com/actors/runs/{run.id}")
62+
if run.status != TERMINAL_OK:
63+
print(run.status_message or "No status message.", file=sys.stderr)
64+
return 1
65+
66+
dataset = client.dataset(run.default_dataset_id)
67+
out = Path(args.out)
68+
out.mkdir(parents=True, exist_ok=True)
69+
items = json.loads(dataset.get_items_as_bytes(item_format="json", clean=True))
70+
(out / "results.json").write_text(json.dumps(items, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
71+
(out / "results.csv").write_bytes(dataset.get_items_as_bytes(item_format="csv", clean=True))
72+
73+
print(f"Saved {len(items)} items to {out / 'results.json'} and {out / 'results.csv'}")
74+
return 0
75+
76+
77+
if __name__ == "__main__":
78+
sys.exit(main())

0 commit comments

Comments
 (0)