OpenCrawling

OpenCrawling is an enterprise data integration and ingestion platform modeled after Apache ManifoldCF. It leverages modern Java 25 features (such as Structured Concurrency and Virtual Threads), Spring Boot, and vector search infrastructure to orchestrate data flows from various repository connectors to vector search outputs.

---

Architecture Diagram

The diagram below shows the high-level architecture of OpenCrawling, highlighting the newly decoupled, stateless embedding microservice and transformation connectors:

graph TD
    subgraph UI
        UI_App[Admin React UI - oc-admin-ui]
    end

    subgraph PlatformRuntime [OpenCrawling Ingestion Runtime - oc-runtime]
        Runtime_Node([OpenCrawling Ingestion Runtime])
        Core[Core Ingestion Engine - oc-core]
        FS_Conn[Filesystem Repository - oc-filesystem-repository-connector]
        
        Ing_Cons[Ingestion Consumer - IngestionConsumer]
        Tika[Apache Tika Text Extractor]
        Chunker[Token Chunker]
        
        Writer_Cons[Vector Store Writer - VectorStoreWriterConsumer]
        Precompute_Model[PrecomputedEmbeddingModel]
        Vec_Conn[Vector Store Output - oc-vector-output-connector]

        McpServer[Secure MCP Server - McpVectorServer]
        
        Core --> FS_Conn
        Core -->|Publish IngestionMessage| Ingest_Topic[(Kafka Topic: opencrawling-ingestion)]
        
        Ingest_Topic -->|Consume IngestionMessage| Ing_Cons
        Ing_Cons -->|Extract Text| Tika
        Tika --> Chunker
        Chunker -->|Publish Chunks| Chunk_Topic[(Kafka Topic: opencrawling-chunks)]
        
        Embed_Topic[(Kafka Topic: opencrawling-embedded)] -->|Consume EmbeddedMessage| Writer_Cons
        Writer_Cons --> Precompute_Model
        Precompute_Model --> Vec_Conn

        McpServer -->|Queries - Enforces ACLs| Vec_Conn
    end

    subgraph Embedding Service [OpenCrawling Embedding Service - oc-embedding-service]
        Embed_Cons[Embedding Consumer - EmbeddingConsumer]
        Model_Factory[EmbeddingModelFactory]
        
        Chunk_Topic -->|Consume ChunkMessage| Embed_Cons
        Embed_Cons --> Model_Factory
        Embed_Cons -->|Publish Embedded| Embed_Topic
    end

    subgraph Infrastructure [Docker Containers]
        PG[(PostgreSQL + pgvector)]
        Redis[(Redis Cache & Session)]
        Ollama[Ollama AI Embeddings]
        Kafka_Broker[Apache Kafka Broker]
    end

    subgraph External [AI Clients]
        LLM[AI Client / LLM Agent]
        OpenAI[OpenAI Platform]
    end

    UI_App -->|REST API| Runtime_Node
    Vec_Conn -->|Vectors| PG
    Runtime_Node -->|Job Cache| Redis
    
    Model_Factory -->|Local Inference| Ollama
    Model_Factory -->|Cloud Inference| OpenAI
    
    Ingest_Topic --> Kafka_Broker
    Chunk_Topic --> Kafka_Broker
    Embed_Topic --> Kafka_Broker

    LLM -->|Model Context Protocol| McpServer

Loading

---

Administration Dashboard (oc-admin-ui)

The oc-admin-ui provides a modern web-based administration console to monitor and configure your ingestion jobs.

UI Screenshots

📊 Telemetry Dashboard

Real-time graphs monitoring job success rates, Kafka queue load, active crawling threads, and index ingestion speed.

📋 Job Pipeline Scheduler

Schedule, monitor, start, and pause ingestion crawl tasks. Review document indexing status reports.

📁 Connector Configurations

Manage endpoints and credentials for repositories (SharePoint, S3, Filesystem), output vectors, and transformation engines.

⚙️ Ingestion & Embedding Mappings

Configure target models (e.g., Ollama, OpenAI) and tune text chunk sizes/overlap boundaries dynamically.

🪵 Real-Time Ingestion Logs

Inspect live Java logging streams and Kafka consumer offsets to troubleshoot connector execution.

---

Core Technologies

• Java 25 Preview Features: Structured Concurrency, Virtual Threads, and Pattern Matching.
• Spring Boot & Spring AI: High-performance backend orchestrating ingestion jobs.
• Apache Kafka: Decoupled, event-driven document processing using the Claim Check Pattern.
• pgvector: High-dimensional vector similarity search in PostgreSQL.
• Redis Stack: Lightweight caching and session management.
• Ollama & OpenAI: Dynamic embedding generation via local and cloud-based AI engines.
• Vite + React + TailwindCSS: Modern frontend administration dashboard.

---

Getting Started

Prerequisites

Ensure you have the following installed on your machine:

• JDK 25 (Ensure JAVA_HOME points to your JDK 25 directory)
• Maven 3.9+
• Docker & Docker Compose
• Node.js 18+ & npm (for the UI)

---

Step-by-Step Setup

1. Start Infrastructure (Docker)

Spin up the database, cache, message broker, and AI engine. Run from the project root:

docker compose up -d

Services started:

• PostgreSQL (Port 5432): For job metadata, schema migrations, and pgvector storage.
• Redis (Port 6379 / Insight Port 8001): For caching and session management.
• Ollama (Port 11434): For local embeddings.
• Apache Kafka (Port 9092): KRaft-mode broker for decoupled, event-driven document processing.

2. Pull the Embedding Models (Ollama)

OpenCrawling supports configuring different embedding models on a per-job basis and automatically routes them to corresponding PgVector tables. To use the available options, make sure to pull the models you plan to utilize:

• mxbai-embed-large (1024-dim, default):

  docker exec -it ollama ollama pull mxbai-embed-large

• nomic-embed-text (768-dim):

  docker exec -it ollama ollama pull nomic-embed-text

• all-minilm (384-dim):

  docker exec -it ollama ollama pull all-minilm

(Ollama will download the requested models in the background. Once pulled, OpenCrawling will automatically route them to vector_store_1024, vector_store_768, or vector_store_384 respectively).

---

Option A: Run OpenCrawling in Docker Containers (Recommended)

To build and run the OpenCrawling backend runtime, the dynamic embedding microservice, and the administration UI as containerized services, run:

1. Build the images:

   docker compose -f docker-compose-apps.yml build

2. Start the applications:

   docker compose -f docker-compose-apps.yml up -d

• Backend Service: Access the backend runtime and integrated static resources at http://localhost:8080.
• Embedding Service: The dynamic microservice processes embeddings at http://localhost:8082.
• Frontend Service: Access the standalone React Administration Console at http://localhost:3000.

---

Option A.2: Decoupled Multi-Service Deployment (Decoupled Microservices)

To run each microservice component (Repository Crawler, Ingestion Consumer, Embedding Consumer, Vector Store Writer, Secure MCP Server, and Admin UI) as a completely separate containerized process communicating over Kafka:

1. Build the decoupled service images:

   docker compose -f docker-compose-decoupled.yml build

2. Start the complete decoupled pipeline:

   docker compose -f docker-compose-decoupled.yml up -d

This spins up the database/event-stream dependencies alongside five decoupled OpenCrawling service containers. You can view logs, scale individual workers (e.g. docker compose -f docker-compose-decoupled.yml scale oc-embedding-service=3), and monitor the decoupled pipeline.

• React Admin UI Console: Access the administration dashboard at http://localhost:3000.
• Secure MCP Server: Connect your AI Client / IDE directly to http://localhost:8080 over SSE.

Running the Decoupled Integration Test

We provide a fully automated end-to-end integration test script that builds, boots, tests, and cleanses the entire decoupled environment:

./scripts/test-docker-decoupled.sh

This script:

1. Builds all decoupled microservices from source.
2. Boots up the Kafka broker, PostgreSQL + pgvector, Redis, Ollama, and consumer workers.
3. Automatically generates a sample document in the crawler mount, triggers a scan, and waits for consumer ingestion.
4. Queries pgvector directly to verify that the generated embeddings are correctly stored.
5. Verifies the Secure MCP Server SSE endpoint, and tears down the environment upon success.

---

Option B: Run OpenCrawling Locally (Development Mode)

If you wish to run the JVM runtime and React frontend directly on your host machine for development:

1. Build the Project (Maven)

Compile all modules using Java 25. Since we utilize advanced features, preview features must be enabled:

mvn clean install

2. Run the Ingestion Runtime Bootstrap

Start the Spring Boot runtime application:

mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

3. Run the Embedding Service Microservice

Start the Embedding Service application in a separate terminal:

mvn spring-boot:run -pl oc-embedding-service

Running a Sample Ingestion Job on Startup (Optional)

By default, the automatic startup crawl is disabled to prevent unnecessary scans. To trigger a demo crawl job on startup, pass the configuration properties:

mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev \
  -Dspring-boot.run.arguments="--spring.opencrawling.crawl-on-startup=true --spring.opencrawling.scan-path=/your/local/directory/to/scan"

4. Run the Admin UI

To launch the administration dashboard:

cd oc-admin-ui
npm install
npm run dev

Open http://localhost:5173 in your browser.

---

Scaling Out & Performance

OpenCrawling is designed for high-throughput, horizontal scalability. Since the ingestion pipeline is decoupled using Apache Kafka and the Claim Check Pattern, you can scale components independently.

1. Scaling the Ingestion / Processing (Output Connector)

Vector indexing and embedding generation is typically the primary performance bottleneck because of deep learning model inference (Ollama/OpenAI) and database indexing (pgvector).

• Kafka Consumer Group Partitioning: The three main topics (opencrawling-ingestion, opencrawling-chunks, and opencrawling-embedded) are consumed by IngestionConsumer, EmbeddingConsumer (in oc-embedding-service), and VectorStoreWriterConsumer respectively within the OpenCrawling services. By configuring these topics with multiple partitions, Kafka distributes load dynamically among active consumer nodes.
• Horizontal Scaling of Service Instances: You can run multiple instances of the oc-embedding-service application sharing the same consumer group. Kafka automatically distributes partitions and load-balances the messages.
• Ollama Load Balancing: Scale out embedding generation by pointing baseUrl to a load balancer (e.g., NGINX, HAProxy) backed by a cluster of Ollama instances running on GPU-enabled nodes.

2. Scaling the Repository Connectors (Ingestion Source)

The scanning/crawling phase can be distributed by splitting large target sources:

• Partitioned Scans: Run separate bootstrap crawl jobs targeting different sub-directories or repository prefixes.
• Distributed File Shares / Shared Storage: In a multi-node setup, ensure the IngestionConsumer instances have access to the same shared filesystem (e.g., NFS, S3/MinIO bucket, SMB) as the repository crawlers, so the Claim Check reference (path/URI) can be successfully resolved by the consumer node.

3. Claim Check Pattern

To ensure the messaging system remains fast and responsive:

1. The Repository Connector crawls data, but instead of publishing the entire document content (which could be megabytes of binary data) to Kafka, it saves/references the file on a shared storage medium.
2. It publishes a lightweight IngestionMessage (Claim Check record) to the Kafka topic containing the metadata (URI, file path, version).
3. The Consumer Workers process the ingestion:
   • IngestionConsumer pulls the reference, reads the file directly from storage, extracts text with Apache Tika, splits it into semantic chunks, and publishes them to the chunks topic.
   • EmbeddingConsumer (running in the oc-embedding-service microservice) pulls the chunks, reads the dynamically configured Transformation Connector engine configurations, requests embedding vectors from the target model engine (Ollama, OpenAI, Hugging Face, etc.), and publishes the embedded chunks to the embedded topic.
   • VectorStoreWriterConsumer consumes embedded chunks and uses a stateless PrecomputedEmbeddingModel to save them directly to pgvector.

---

Verification & Monitoring

• Database: Access PostgreSQL at localhost:5432 (User: opencrawling, DB: opencrawling).
• Redis Dashboard: Open http://localhost:8001 in your browser to view the Redis Stack Insight dashboard.
• Logs: Monitor console output for the Virtual Thread Executor and Structured Concurrency task logs.

---

Troubleshooting

• Java Version Check: Run java -version to confirm you are using Java 25.
• Preview Features: If your IDE fails to compile structured concurrency code, verify that the --enable-preview JVM argument is configured for compiler and runtime settings. (It is already pre-configured in pom.xml).

---

Trademark

OpenCrawling® is a registered trademark of the OpenCrawling Organization. For guidelines on using the name and logo, please refer to the TRADEMARK.md file.