Mini Spark + Flink + Firebase — Built for Production Scale
Features • Architecture • Quick Start • Dashboard • API • Documentation
StreamForge is a production-ready, fault-tolerant distributed data processing platform that handles high-volume event streams with:
|
|
stream("events")
.filter(_.type == "click")
.window(sliding(5.min))
.aggregate(sum("value"))
.sink("dashboard") |
|
|
|
┌─────────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ IoT • Logs • Web Events • APIs │
└─────────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌───────────────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ 🔵 KOTLIN API LAYER (Port 8080) │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ REST API │ │ gRPC │ │ WebSocket │ │ Scheduler │ │ │
│ │ │ (Ktor) │ │ Service │ │ Metrics │ │ (Quartz) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ Coroutines • Backpressure • Flow Control │ │
│ └───────────────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ 🔴 SCALA PROCESSING ENGINE │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Custom DSL │───▶│ Stream Graph│───▶│ Operators │───▶│ Windowing │ │ │
│ │ │ Parser │ │ Builder │ │ Map/Filter │ │ Aggregate │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ Akka Streams • Reactive • Backpressure │ │
│ └───────────────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ 🟢 JAVA INFRASTRUCTURE LAYER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Checkpoint │ │ State │ │ Cluster │ │ Recovery │ │ │
│ │ │ Manager │ │ Store │ │ Coordinator │ │ Manager │ │ │
│ │ │ (RocksDB) │ │ (RocksDB) │ │ (ZooKeeper) │ │ (Health) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ Exactly-Once • Fault Tolerance • High Availability │ │
│ └───────────────────────────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ 🟡 FLUTTER DASHBOARD │
│ (Port 3000) │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Cluster │ │ Job │ │ Job │ │
│ │ Health │ │ Metrics │ │ Management │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ Real-Time Charts • WebSocket • Material 3 │
└─────────────────────────────────────────────────────────┘
|
|
|
StreamForge consists of a Kotlin/Scala/Java Backend and a Cross-Platform Flutter Dashboard. You can run the dashboard completely standalone (using its Mock Engine) or alongside the real backend.
- Flutter 3.0+ — Required for the Dashboard App/Website.
- JDK 17+ — Required if running the Kotlin/Scala Backend.
- Docker — Required for running the full cluster easily.
The dashboard has a built-in Stateful Mock Engine. This means you can run it perfectly without ever starting the backend—ideal for testing, design, or showcasing the UI. Because it's built with Flutter, you can run it on any platform from the exact same codebase!
First, navigate to the dashboard directory:
cd dashboard-flutter
flutter pub getTo run the dashboard locally in your browser:
flutter run -d web-server --web-port 3000Then, open http://localhost:3000 in your browser. (Note: On small screens, the UI automatically transforms into a mobile-friendly layout with a bottom navigation bar!)
To run the dashboard as a native mobile application on your connected smartphone or an emulator:
# To see available devices (Emulators, physical devices)
flutter devices
# Run on Android or iOS
flutter runTo compile it into a high-performance native desktop application:
# Enable desktop support if you haven't already
flutter config --enable-windows-desktop
flutter config --enable-macos-desktop
# Run the app natively
flutter run -d windows # or -d macosIf you want to run the real Scala processing engine and Kotlin API:
# Start all microservices (API, Engine, ZooKeeper, Dashboard)
docker-compose up -d
# API is available at: http://localhost:8080
# Dashboard is available at: http://localhost:3000Open two separate terminals in the project root:
Terminal 1 (Backend):
# Build all Java/Scala/Kotlin modules
./gradlew build
# Run the API server
./gradlew :api-kotlin:runTerminal 2 (Dashboard):
cd dashboard-flutter
flutter run -d web-server --web-port 3000curl -X POST http://localhost:8080/api/jobs \
-H "Content-Type: application/json" \
-d '{
"id": "click-analysis",
"name": "Click Stream Analysis",
"type": "STREAMING",
"config": {
"source": "events",
"sink": "dashboard",
"parallelism": 4,
"checkpointInterval": 60000
}
}'curl -X POST http://localhost:8080/api/ingest/event \
-H "Content-Type: application/json" \
-d '{
"id": "evt-001",
"type": "click",
"timestamp": "2025-12-22T20:00:00Z",
"source": "web-app",
"payload": {"page": "/products", "userId": "user-123"}
}'curl http://localhost:8080/api/jobs/click-analysis
Scala Processing Engine DSL • Windowing • Operators |
Java Infrastructure Checkpoints • State • Recovery |
Kotlin API Layer REST • WebSocket • Scheduler |
Flutter Dashboard Charts • Monitoring • Control |
| Metric | Value |
|---|---|
| Throughput | 10,000+ events/second per node |
| Latency | < 20ms average processing |
| Recovery | < 2 seconds from checkpoint |
| Scalability | Horizontal scaling with ZooKeeper |
streamforge/
├── 📄 README.md, LICENSE, CONTRIBUTING.md, CHANGELOG.md
├── 🐳 docker-compose.yml, Dockerfile.api, Dockerfile.engine
│
├── 🔴 engine-scala/ # Scala Processing Engine
│ └── dsl/, window/, operators/, StreamProcessor, BatchProcessor
│
├── 🟢 infrastructure-java/ # Java Infrastructure
│ └── checkpoint/, state/, cluster/, recovery/
│
├── 🔵 api-kotlin/ # Kotlin API Layer
│ └── Application, JobAPI, IngestionAPI, MetricsAPI, Scheduler
│
├── 🟡 dashboard-flutter/ # Flutter Dashboard
│ └── pages/, services/, Dockerfile, nginx.conf
│
├── 📚 docs/ # Documentation
│ └── ARCHITECTURE.md, API.md, DEPLOYMENT.md
│
└── 📝 examples/ # Examples
└── example_pipeline.scala, sample_job.json
| Document | Description |
|---|---|
| Architecture | System design, data flow, components |
| API Reference | REST endpoints, WebSocket, models |
| Deployment | Docker, Kubernetes, production setup |
| ✅ Production Ready | Fault-tolerant with exactly-once semantics |
| ✅ Modern Stack | Multi-language architecture optimized for each layer |
| ✅ Beautiful Dashboard | Real-time visualization with Flutter |
| ✅ Developer Friendly | Declarative DSL, comprehensive APIs |
| ✅ Big-Tech Patterns | Implements designs from Spark, Flink, Samza |
This project is licensed under the MIT License — see LICENSE for details.
