socmon is a self-hosted, reproducible framework for monitoring the Russian internet across social networks. You declare a monitoring job in YAML, and socmon collects, deduplicates, stores, and analyzes posts and comments from VK, Telegram, and several other networks — keeping the raw payloads so you can re-parse later without re-fetching, and recording enough provenance that any run can be repeated and defended.
Read this first: DISCLAIMER.md — legal, privacy, and ethics notice. You are solely responsible for using this tool lawfully and in line with each platform's Terms of Service. socmon is tooling, not legal advice.
Closed monitoring SaaS gives you a dashboard and asks you to trust it. socmon is built around the opposite premise — you own the pipeline end to end:
- Reproducibility & auditability. Every run snapshots its config, model and prompt versions, and content hashes, so a result can be reproduced and justified — for a client report, a compliance trail, or an academic publication.
- Methodological honesty. Coverage limits are made explicit: VK's result ceiling, whether paid Telegram full-text search was used, the best-effort nature of Dzen/Rutube. A bias-and-blind-spots map is part of the output, not hidden.
- Raw data retention. Normalized
Post/Commentrecords plus the original JSON payloads are stored, so you can re-parse, re-analyze, or export to CSV / Parquet at any time. - A provider-agnostic LLM layer. Optional language-model features (topic labeling, config assistance, collection audit, aspect-based sentiment) run against either a hosted API or a local backend, so private data can stay private.
What each network connector can do. Connectors are pluggable; the matrix reflects what each platform realistically exposes.
| Capability | VK | Telegram | OK | Dzen | Rutube | MAX |
|---|---|---|---|---|---|---|
| Global search | yes (capped) | hashtag free / full-text paid | no | no | yes (site search) | no |
| Wall / source search | yes | yes (in channel) | no | no | no | no |
| Source feed | yes | yes | yes | yes (RSS/scrape) | yes | no |
| Comments | yes | yes | optional | best-effort | yes | no |
| Source discovery | yes | yes (TGStat) | no | no | no | no |
| Reactions/metrics | yes | yes | partial | partial | partial | no |
| Robustness | API, stable | user session | scrape, fragile | scrape, fragile | scrape, fragile | unsupported |
Notes: VK global search (newsfeed.search) is capped at ~1000 results per query
window, so it is driven by adaptive time-bisection. Telegram full-text global
search costs money and is gated behind an explicit opt-in. OK, Dzen, and Rutube
are scrape/best-effort and isolated so breakage degrades gracefully. MAX is not
supported for monitoring — its Bot API only sees channels the bot was added to,
and there is no global search or user client; the connector is a stub until a
read/search API appears.
config/job.yaml (declarative)
|
JobConfig (validated)
|
+-------------------+-------------------+
| | |
scheduler ----> collection pipeline ---> storage (PostgreSQL
(APScheduler / | | + pgvector, raw JSONB)
Celery+Redis) | |
| |
connectors pools
(VK/TG/OK/Dzen/ (accounts + proxies:
Rutube; plugin health, rotation,
contract) flood-wait/captcha)
|
analysis
(lemmatize, frequencies, BERTopic topics,
descriptives, spike detection)
|
optional LLM layer (provider-agnostic)
|
dashboards (Streamlit)
Core building blocks (time-bisection, text normalization & content hashing, incremental "stop on known", dedup-key construction, config validation) are pure and fully unit-tested. Network connectors and external integrations are typed, documented stubs that you implement against the cited platform APIs. The package imports with only the light dependency set installed; every heavy dependency is optional and imported lazily.
Requires Python 3.12+, Docker (for PostgreSQL + Redis), and the optional extras for whichever networks and features you need.
# 1. Create an environment and install socmon with the extras you need.
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e '.[telegram,analysis,llm,queue,dashboard]'
# 2. Configure environment (copy and edit placeholder values).
cp .env.example .env
# 3. Start PostgreSQL (with pgvector) and Redis.
docker compose up -d
# 4. Initialize the database schema.
socmon init-db # or: alembic upgrade head
# 5. Run the example monitoring job once.
socmon run config/job.example.yamlThe bundled config/job.example.yaml is a fully
commented brand/topic-monitoring job you can copy and adapt.
socmon is built network by network, value first.
- Phase 0 — Skeleton: data model, PostgreSQL schema, dedup, the VK connector, minimal account/proxy pools, scheduler with hash/cursor increment, baseline analytics, a single-job dashboard. One VK job runs on schedule, fetches only new material, never duplicates.
- Phase 1 — Telegram & collection maturity: Telethon-based Telegram connector, full per-network pools with sticky proxies and health checks, flood-wait/captcha handling, Celery/Redis with separate idempotent queues, metric-snapshot history.
- Phase 2 — Analytics & LLM (targeted): lemmatization (Natasha), TF-IDF / c-TF-IDF word clouds, NER clouds, BERTopic with a Russian embedder plus topics-over-time, a provider-agnostic LLM layer (topic names, config assistant, collection audit, sentiment incl. per-actor) with a local backend option, and a spike detector.
- Phase 3 — Breadth & interpretation: OK and Dzen connectors, near-duplicate and cross-network linking, period summaries, chat-to-data over a SQL allow-list, and a custom front end with filtering and export.
- Phase 4 — Research maturity: corpus export (CSV/Parquet) with codebook and provenance for reproducibility, diffusion/network analysis, geo, extended models, and optional new networks (e.g. MAX) once their APIs allow.
See CONTRIBUTING.md for dev setup, the lint/type/test workflow, the project layout, and the connector-plugin contract.
Apache License 2.0. See LICENSE.