Intelligent Mind Web Engine

The Intelligent Mind Web Engine is a locally hosted, real-time visual learning and object recognition engine. This application allows users to teach a neural network custom objects using their webcam or file uploads, and instantly recognize them in real time without any external cloud service or API dependencies.

---

Technical Architecture and How It Works

The engine operates on a two-phase pipeline: Feature Extraction (Visual Fingerprinting) and Nearest Neighbors Classification (Vector Matching).

1. Feature Extraction (Visual Fingerprinting)

When an image is submitted (either via a live snapshot or a file upload):

• The input image is decoded and resized to 224x224 pixels.
• The pixel values are normalized to match the preprocessing specifications of the MobileNetV2 architecture.
• The preprocessed image is processed through a MobileNetV2 deep convolutional neural network, pre-trained on the ImageNet dataset.
• The final classification layers of MobileNetV2 are bypassed (configured using include_top=False). A Global Average Pooling layer (pooling='avg') is instead applied to collapse the spatial dimensions (7x7) into a dense 1280-dimensional feature vector. This vector serves as a unique mathematical signature (or visual fingerprint) of the target object.

2. Nearest Neighbors Classification (Vector Matching)

To identify a query object:

• The query image is passed through the same feature extraction pipeline to generate its 1280-dimensional vector.
• The K-Nearest Neighbors (K-NN) algorithm, configured with n_neighbors=1 and a Cosine Similarity metric, calculates the distance between the query vector and all vectors stored in the local database.
• The system returns the label of the closest matching vector, along with the calculated distance.
• Verification Threshold: A threshold of 0.25 is set for the cosine distance. If the distance is less than 0.25 (meaning similarity is greater than 0.75), the system registers a confident match and reports the identification result. If the distance exceeds this threshold, the object is classified as unknown.
• Data Persistence: To preserve taught objects across server reboots, the data (consisting of the feature vectors, corresponding string labels, and raw thumbnail image strings) is automatically serialized to a local storage file (local_memory.pkl) at the root of the workspace. When the application starts up, it checks for this file, automatically deserializes the objects, and re-fits the K-NN index. Deleting an instance or wiping the database updates the array structure and serializes the fresh state back to the disk (or deletes local_memory.pkl entirely when no taught objects remain).

---

Codebase Components

The codebase consists of the following key files:

• backend.py: The FastAPI server that handles static asset delivery, endpoint routing, memory state database operations, and administrative functions.
  • load_base_model: Loads the MobileNetV2 base network.
  • extract_features: Resizes, preprocesses, and converts input images to 1280-D vectors.
  • teach_object: Appends images to the memory database, fits/re-fits the K-NN index, and triggers disk serialization.
  • recognize_object: Compares a query image against the stored database.
  • delete_vector: Deletes a target memory slice, rebuilds the K-NN search index, and updates the local storage file.
  • reset_memory: Wipes all saved instances, clears the active index, and deletes the local storage file.
• static/index.html: The frontend single-page application built using vanilla HTML, CSS, and JavaScript. It renders the webcam stream, captures canvas snapshots, handles file uploads, and acts as the interface for both teaching and identification.
• app.py: An alternative UI constructed with Streamlit, presenting a simple dashboard layout for teaching and real-time verification.
  • load_base_model: Caches and initializes the MobileNetV2 base network.
  • extract_features: Performs the image embedding calculation.
• run.bat: A Windows batch script that automates starting the virtual environment, verifying package installations, launching the FastAPI server, and opening the browser.
• local_memory.pkl: The automatically created local database file containing the pickled structures of training vectors, text labels, and thumbnail graphics.

---

Core Advantages

• Local Persistence: Objects taught to the engine remain stored locally until deleted. Restarting the server does not result in memory loss.
• Zero Cloud Latency and Cost: All classification and neural network inference occur on the local CPU or GPU. There are no API keys, cloud subscriptions, or remote server roundtrips.
• Privacy by Design: All visual representations and images are processed and kept inside the local machine. No client data is transmitted over the internet.
• Instant Learning (Few-Shot): Unlike traditional deep learning classifiers that require hours of training and fine-tuning, the instance database updates instantly. Once a snapshot of an object is saved, the engine can immediately recognize it.
• Lightweight Footprint: By employing MobileNetV2, the application delivers robust performance with minimal resource consumption, making it ideal for standard laptop computers and edge devices.

---

Future Enhancements

• Advanced Database Migration: Transitioning from pickle to a dedicated vector search library such as SQLite with vector search extensions, FAISS, or ChromaDB to support large-scale storage and sub-millisecond indexing.
• Custom Backbone Selection: Provide options to swap the backbone network (e.g., to EfficientNet, ResNet, or Vision Transformers) to suit the host machine's processing capacity.
• Weighted KNN Voting: Generalize the search index query to retrieve K-Nearest Neighbors (where K is greater than 1) and apply similarity-weighted voting to increase detection robustness across complex angles.
• Automated View Capturing: Implement an automated session capture loop that records and stores object angles continuously, guiding the user to rotate the object in front of the camera.
• Interactive Notebook Documentation: Access the technical breakdown and simulation details in the codebase explanation notebook at intelligent_mind_explanation.ipynb.

---

Getting Started

1. Ensure Python 3.12 (or a compatible release) is installed on your machine.
2. Execute run.bat to install requirements and initiate the uvicorn web server.
3. Open http://localhost:8502 to start teaching and recognizing.