Tech Trends — Technology Trend Analysis Platform

---

Project Overview

Challenge	Solution	Impact
Multi-source tech trends with consistent scoring	Canonical ETL pipeline + Trend Score bridge	Stable, explainable rankings
Fragile frontend assumptions	Bridge-first JSON + CSV fallback	Resilient UI under data changes
Weekly refresh without regressions	CI gates + data contracts	Predictable releases

---

Key Metrics (Latest Run)

Metric	Value
Repos classified	931
StackOverflow questions (annual window)	33,356
Reddit mentions (emerging topics)	366
Technologies in ranking	22
Window	2025-03-13 → 2026-03-13 (UTC)

---

Pipeline Architecture

---

Tech Stack

Layer	Technologies
Data	Python, pandas, duckdb, pandera
ETL	Custom pipelines + contracts
Frontend	Flutter Web
Hosting	GitHub Pages

---

Quick Start

# backend
pip install -r backend/requirements.txt
python -m pytest -q

# run ETLs
python backend/github_etl.py
python backend/stackoverflow_etl.py
python backend/reddit_etl.py
python backend/trend_score.py
python backend/sync_assets.py

# frontend
cd frontend
flutter pub get
flutter run -d chrome

---

Automation (GitHub Actions)

1. ETL Weekly Refresh (etl_semanal.yml)
   • Schedule: Monday 08:17 UTC (approximately 03:17 America/Guayaquil).
   • Runs all ETLs, Trend Score, syncs assets, validates contracts, and publishes data.
2. Dependency Security (dependency_security.yml)
   • Schedule: Monday 09:00 UTC
   • Runs pip-audit against backend/requirements.txt.
3. Deploy Frontend (deploy_frontend.yml)
   • Publishes Flutter Web to GitHub Pages.

Delayed weekly run recovery

GitHub Actions schedules can be delayed. First inspect ETL Weekly Data Refresh in the Actions tab and confirm that no run is in progress for the etl-pipeline concurrency group. If the scheduled run is absent or has failed, use Run workflow on main only after that check. Review the run summary and the source freshness guard before considering the published data healthy.

---

Project Structure

Technology-trend-analysis-platform/
├── backend/                       # ETLs + Trend Score + contracts
├── datos/                         # CSV outputs (legacy + latest + history)
├── frontend/                      # Flutter Web UI
├── docs/                          # Technical docs
├── scripts/                       # CI / assets validation helpers
└── .github/workflows/             # CI + ETL automation

---

Author

Samir Caizapasto
Junior Data Engineer & Analyst

---

If you find this project useful, please give the repository a star.