RADAR - Real-time Anomaly Detection and Response

RADAR is a security log monitoring application that collects system logs, stores them in MongoDB, scores each log with an ML autoencoder, and displays anomaly spikes on a Flask dashboard.

Tech Stack

Layer	Technology	Purpose
Frontend	HTML, CSS, JavaScript, Chart.js	Dashboard, charts, KPIs, real-time stream, filters
Web backend	Python, Flask, Flask-CORS	UI routes, authentication, APIs, MongoDB access
ML service	Python, PyTorch, scikit-learn, sentence-transformers	Vectorization and autoencoder-based anomaly scoring
Database	MongoDB Atlas	Stores users and collected event logs
Log collection	Python, PowerShell, Windows Event Log	Collects Windows logs and inserts normalized events into MongoDB
Deployment	Docker, Docker Compose, Render, Kubernetes YAML	Local and cloud deployment options

Project Structure

.
|-- services/
|   |-- web/           Flask dashboard and API service
|   |-- ml/            Autoencoder inference and training code
|   |-- logcollector/  Windows log collector that writes to MongoDB
|   `-- embed/         Reserved embedding service
|-- k8s/               Kubernetes deployment manifests
|-- docker-compose.yml Local Docker orchestration for web + ML
|-- run_locally.ps1    Local Windows runner for ML + collector + web
`-- debug_db.py        MongoDB collection/count debug helper

Main Modules and Use

Module	Important files	What it does
Web dashboard	services/web/app.py, services/web/templates/dashboard.html	Serves the UI, charts, login pages, and dashboard experience
API routes	services/web/routes/*.py	Provides anomaly, latest-log, stream, stats, settings, and auth endpoints
Mongo connector	services/web/database/mongo.py	Connects to MongoDB and reads public/private user log collections
ML client	services/web/ml_client.py	Cleans Mongo records and sends JSON-safe payloads to the ML service for MSE scoring
ML scoring service	services/ml/app.py, services/ml/ml/model_loader.py	Accepts a log document and returns an anomaly score
Preprocessing	services/ml/ml/preprocess.py	Converts message text, source, OS type, and level into model input features
Training	services/ml/ml/train_autoencoder.py	Trains the PyTorch autoencoder and saves model/vectorizer files
Log collector	services/logcollector/app.py	Polls Windows Event Logs and inserts normalized log records into MongoDB

Data Flow

1. The log collector reads Windows Event Logs from channels such as System and Application.
2. Each event is normalized into this MongoDB shape:

{
  "timestamp": "2026-05-15T10:00:00+00:00",
  "hostname": "WIN-APP-01",
  "source": "Service Control Manager",
  "os_type": "Windows",
  "level": "info",
  "message": "Background service entered the running state",
  "collector": "SystemLogCollector"
}

3. The collector inserts the document into RADAR.Event_Logs.
4. The web dashboard fetches logs from MongoDB using /api/anomalies or /api/stream/logs.
5. The web service sends a cleaned version of each log to the ML service.
6. The ML service vectorizes the log and calculates reconstruction error using the autoencoder.
7. The dashboard maps the returned MSE score to severity:

Score range	Severity
< 0.010	Safe
0.010 - 0.019	Low
0.020 - 0.049	Medium
>= 0.050	High / anomaly spike

Run Locally on Windows

Prerequisites:

• Python 3.9+
• PowerShell
• MongoDB Atlas connection string

Start all local services:

.\run_locally.ps1

This starts:

• ML service on http://localhost:5001
• Log collector in the background
• Web dashboard on http://localhost:5000

Open the dashboard at:

http://localhost:5000

Run Individual Services

ML service:

pip install -r services/ml/requirements.txt
python services/ml/app.py

Web service:

pip install -r services/web/requirements.txt
$env:ML_SERVICE_URL="http://localhost:5001"
$env:MONGO_URI="mongodb+srv://..."
python services/web/app.py

Log collector:

pip install -r services/logcollector/requirements.txt
$env:MONGO_URI="mongodb+srv://..."
python services/logcollector/app.py

Log Collector Configuration

Environment variable	Default	Use
MONGO_URI	Project Atlas URI	MongoDB connection string
MONGO_DB	RADAR	Database name
MONGO_COLLECTION	Event_Logs	Log collection
LOG_CHANNELS	System,Application	Windows Event Log channels to collect
LOG_POLL_SECONDS	15	Delay between polling cycles
LOG_LOOKBACK_MINUTES	15	Initial historical window
LOG_MAX_EVENTS_PER_CHANNEL	50	Max events per channel per cycle

To include Security logs, run PowerShell or the collector as Administrator and set:

$env:LOG_CHANNELS="System,Application,Security"

Run with Docker

Docker Compose starts the web and ML services:

docker-compose up --build

The Windows log collector is meant to run on the Windows host because Linux containers cannot directly read Windows Event Logs.

MongoDB Collections

Collection	Purpose
Event_Logs	Public/default log stream used by the dashboard
Users	Login and user profile data
<user email>	Private per-user log collection when private mode is selected

Important Notes

• The dashboard does not trust a database severity field for anomaly spikes.
• Anomaly severity is calculated from the ML service response field score.
• A score of 0.0 means either the event is normal or scoring failed; service logs should be checked if everything appears Safe.
• Mongo _id and Date values must be converted before calling the ML service. services/web/ml_client.py handles this.

Debugging

Check MongoDB collections and document counts:

python debug_db.py

Check whether the web service can reach the ML service:

curl http://localhost:5001/score -Method POST -ContentType "application/json" -Body '{"log":{"timestamp":"2026-05-15T10:00:00Z","hostname":"test","source":"EventLog","os_type":"Windows","level":"info","message":"test log"}}'

Deployment

The repository includes:

• render.yaml for Render service deployment
• k8s/ manifests for Kubernetes
• docker-compose.yml for local container orchestration

For production, store MONGO_URI, SECRET_KEY, and service URLs as environment variables instead of hardcoding them.