feat: add perception UI, multimodal chat, and documentation updates

CHANGELOG.md

@@ -7,7 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Multimodal perception pipeline: Basirah (vision) + Nutq (voice) wired into QCA engine
+- Auditory-first processing (Sam' before Basar) per Quran 16:78/17:36
+- Perception-Qalb integration: emotional context modulates vision and voice analysis
+- `POST /api/perception/analyze` endpoint for multimodal analysis
+- WebSocket `multimodal` message type for real-time perception
+- DNA-inspired quaternary encoding module (`backend/memory/quaternary.py`)
+- Quaternary checksum as 4th integrity layer in Lawh al-Mahfuz
+- VectorStore (ChromaDB) integration in Living Memory novelty gate
+- Hybrid text + semantic similarity in Living Memory `_find_best_match()`
+- `search_entities()` method in KnowledgeGraph (fixes MemoryPyramid layer 4)
+- Perception page in frontend UI with image/audio upload
+- Image and audio attachment support in chat interface
+- `BasirahEngine` and `NutqEngine` exports in perception `__init__.py`
+- NutqEngine: language detection (Arabic, Urdu, English) via Unicode analysis
+- NutqEngine: 8+ intent types (question, command, greeting, farewell, confirmation, negation, request, statement)
+
 ### Fixed
+- BasirahEngine: JSON-structured LLM output parsing (was returning empty extracted_text/key_elements)
+- BasirahEngine: category detection bug (`"error" in raw_lower and "text"` evaluated boolean inside list)
+- NutqEngine: `_adjust_for_tone()` was a no-op, now implements warm/patient/focused text transforms
+- KnowledgeGraph: missing `search_entities()` caused MemoryPyramid layer 4 to silently fail
 - Replace deprecated `datetime.utcnow()` with `datetime.now(timezone.utc)` across all modules
 - Fix Pydantic `class Config` deprecation in `SkillExecuteRequest` (use `model_config` dict)
 - Fix `check_provider_health()` called without `await` in settings endpoint

README.md

@@ -102,6 +102,7 @@ make dev                   # Start backend + frontend
 |---------|---------------|
 | **Chat** | Talk to your AI in the browser or terminal |
 | **Browse the web** | AI can search Google, read websites, extract information |
+| **Analyze images & voice** | Upload images for vision analysis, audio for transcription |
 | **Run code** | AI writes and executes Python, bash scripts |
 | **Manage files** | Read, write, organize files on your computer |
 | **Remember things** | Remembers your conversations and preferences |
@@ -114,8 +115,11 @@ make dev                   # Start backend + frontend
 | Feature | What It Means |
 |---------|---------------|
 | **QALB-7 Cognitive Pipeline** | 7-layer architecture: ethics → deliberation → emotion → conviction → metacognition |
+| **Multimodal Perception** | Sam' (hearing) + Basar (sight) → Fu'ad integration, with Qalb-aware context |
 | **Developmental Stages** | Agents grow from Nutfah (5 tools, 5 turns) to Khalq Akhar (all tools, 25 turns) |
+| **Living Memory** | Novelty gate with hybrid text+vector similarity — never re-stores 1+1=2 |
 | **5-Layer Memory Pyramid** | Unified query across episodic, semantic, neural pathways, vectors, and knowledge graph |
+| **DNA Integrity** | Quaternary (ACGT) checksums with Hamming distance verification in Lawh al-Mahfuz |
 | **Causal Reasoning** | Pearl's 3-rung causal ladder: observation, intervention, counterfactual |
 | **Plugin system** | Add new abilities with a simple Python file |
 | **Event bus + Hooks** | Decoupled communication — modify data at any point in the pipeline |
@@ -282,6 +286,9 @@ Each agent processes every task through these cognitive layers:
 
 | Module | Arabic | Purpose | File |
 |--------|--------|---------|------|
+| **Multimodal Perception** | سمع+بصر | Sam' (hearing) first, then Basar (sight), Qalb-aware context | `perception/basirah.py`, `perception/nutq.py` |
+| **Living Memory** | ذاكرة حية | Novelty gate + hybrid text/vector similarity + Dhikr daemon | `memory/living_memory.py` |
+| **Quaternary Encoding** | تشفير رباعي | DNA-inspired ACGT checksums with Hamming distance verification | `memory/quaternary.py` |
 | **Lawwama Self-Healing** | لوّامة | Immune memory, health metrics, adaptive checkpoint intervals | `core/self_healing.py` |
 | **Parallel Agents** | — | Concurrent task scheduling + skill transfer between agents | `core/parallel_agents.py` |
 | **Imagination** | تصوير | Predictive coding — simulate outcomes before acting | `core/imagination.py` |
@@ -296,14 +303,19 @@ All memory layers are queried through a unified `MemoryPyramid`:
 
 | Layer | Module | Purpose |
 |-------|--------|---------|
+| **Living Memory** | `memory/living_memory.py` | Novelty gate (hybrid text + vector similarity), importance scoring, Dhikr daemon |
 | **Dhikr** | `memory/dhikr.py` | Three-tier persistent memory (episodic, semantic, procedural) |
 | **Masalik** | `memory/masalik.py` | Neural pathway network with spreading activation |
-| **Lawh al-Mahfuz** | `memory/lawh_mahfuz.py` | Immutable core memory with triple-checksum integrity |
-| **VectorStore** | `memory/vector_store.py` | Semantic embedding search (ChromaDB) |
-| **KnowledgeGraph** | `memory/knowledge_graph.py` | Entity-relationship graph (SQLite) |
+| **VectorStore** | `memory/vector_store.py` | Semantic embedding search (ChromaDB) — also used by Living Memory |
+| **KnowledgeGraph** | `memory/knowledge_graph.py` | Entity-relationship graph with full-text search (SQLite) |
+| **Lawh al-Mahfuz** | `memory/lawh_mahfuz.py` | Immutable memory with 4-layer integrity: SHA-256 + CRC-32 + length + quaternary checksum |
 
 Unified query: `memory/memory_pyramid.py` merges, deduplicates, and ranks results by relevance x certainty x recency.
 
+**Living Memory** solves the 1+1=2 problem: seeing the same information again doesn't create a new trace — it activates the existing one. New information enriches existing traces, related info gets linked, and only genuinely novel content is stored.
+
+**Quaternary Encoding** (`memory/quaternary.py`) provides DNA-inspired error detection: binary data is encoded using a 4-symbol alphabet (A, C, G, T), chunked into codons (triplets), and verified using XOR parity and Hamming distance.
+
 ### Developmental Stages (Nafs Levels 1–7)
 
 Agents grow through seven stages, each unlocking new capabilities:
@@ -441,6 +453,45 @@ GET  /api/tasks/history       Get task history
 POST /api/memory/query        Search memories
 POST /api/memory/store        Store a memory
 POST /api/memory/consolidate  Prune old memories
+GET  /api/memory/list         List recent memories
+```
+
+### Perception (Sam' + Basar)
+```
+POST /api/perception/analyze  Multimodal analysis (text + base64 image + base64 audio)
+```
+
+Accepts `MultimodalInput` with fields: `text`, `image_base64`, `audio_base64`, `media_type`, `qalb_state`.
+Processes Sam' (audio) first, then Basar (image), integrates via Fu'ad.
+
+### Cognitive Pipeline
+```
+POST /api/qalb/analyze          Analyze emotional state from text
+GET  /api/qalb/trend/{user_id}  Get emotional trend over time
+POST /api/cognitive/route        Route to best cognitive method
+POST /api/yaqin/tag              Tag knowledge with certainty level
+GET  /api/yaqin/stats            Get Yaqin statistics
+```
+
+### Federation
+```
+GET  /api/federation/status    Federation network status
+POST /api/federation/discover  Discover agents by capability
+POST /api/federation/route     Route task to best agent
+```
+
+### Nafs & Ruh
+```
+GET  /api/nafs/tiers           Get all 7 Nafs tier definitions
+GET  /api/nafs/{agent_id}      Get agent Nafs level and progress
+GET  /api/ruh/{agent_id}       Get agent Ruh energy level
+```
+
+### Knowledge
+```
+POST /api/knowledge/ingest     Ingest from URL or YouTube
+POST /api/knowledge/upload     Upload PDF for knowledge extraction
+GET  /api/knowledge/sources    List ingested knowledge sources
 ```
 
 ### Plugins & Extensibility
@@ -490,6 +541,10 @@ POST /api/shura               Multi-agent consultation
 WS   /ws/{client_id}          WebSocket connection
 ```
 
+**WebSocket message types**: `chat`, `task`, `command`, `multimodal`, `ping`
+
+The `multimodal` type accepts: `{ type: "multimodal", content: "text", image_base64: "...", audio_base64: "...", media_type: "image/png", qalb_state: "neutral" }` and returns a `perception_result` message with full QCA analysis.
+
 ### Channels
 ```
 POST /api/channels/{name}/start   Start a channel adapter

docs/index.html

@@ -1148,6 +1148,9 @@ <h3>Extension Modules</h3>
       <table>
         <thead><tr><th>Module</th><th>Arabic</th><th>Purpose</th><th>File</th></tr></thead>
         <tbody>
+          <tr><td><strong>Multimodal Perception</strong></td><td>سمع+بصر</td><td>Sam' (hearing) first, then Basar (sight), Qalb-aware context modulation</td><td><code>perception/basirah.py</code>, <code>perception/nutq.py</code></td></tr>
+          <tr><td><strong>Living Memory</strong></td><td>ذاكرة حية</td><td>Novelty gate + hybrid text/vector similarity + Dhikr maintenance daemon</td><td><code>memory/living_memory.py</code></td></tr>
+          <tr><td><strong>Quaternary Encoding</strong></td><td>تشفير رباعي</td><td>DNA-inspired ACGT checksums with codon chunking and Hamming verification</td><td><code>memory/quaternary.py</code></td></tr>
           <tr><td><strong>Self-Healing</strong></td><td>لوّامة</td><td>Immune memory, health metrics, adaptive checkpoints</td><td><code>core/self_healing.py</code></td></tr>
           <tr><td><strong>Parallel Agents</strong></td><td>—</td><td>Concurrent task scheduling + skill transfer</td><td><code>core/parallel_agents.py</code></td></tr>
           <tr><td><strong>Imagination</strong></td><td>تصوير</td><td>Predictive coding — simulate before acting</td><td><code>core/imagination.py</code></td></tr>
@@ -1157,19 +1160,21 @@ <h3>Extension Modules</h3>
         </tbody>
       </table>
 
-      <h3>5-Layer Memory Pyramid</h3>
+      <h3>Memory Architecture (6-Layer Pyramid)</h3>
       <p>All memory layers are queried through a unified <code>MemoryPyramid</code>:</p>
       <table>
         <thead><tr><th>Layer</th><th>Module</th><th>Purpose</th></tr></thead>
         <tbody>
+          <tr><td><strong>Living Memory</strong></td><td><code>memory/living_memory.py</code></td><td>Novelty gate with hybrid text + vector similarity, importance scoring, Dhikr daemon</td></tr>
           <tr><td><strong>Dhikr</strong></td><td><code>memory/dhikr.py</code></td><td>Three-tier persistent memory (episodic, semantic, procedural)</td></tr>
           <tr><td><strong>Masalik</strong></td><td><code>memory/masalik.py</code></td><td>Neural pathways with spreading activation</td></tr>
-          <tr><td><strong>Lawh al-Mahfuz</strong></td><td><code>memory/lawh_mahfuz.py</code></td><td>Immutable memory with triple-checksum integrity (SHA-256 + CRC-32 + length)</td></tr>
-          <tr><td><strong>VectorStore</strong></td><td><code>memory/vector_store.py</code></td><td>Semantic embedding search (ChromaDB)</td></tr>
-          <tr><td><strong>KnowledgeGraph</strong></td><td><code>memory/knowledge_graph.py</code></td><td>Entity-relationship graph (SQLite)</td></tr>
+          <tr><td><strong>VectorStore</strong></td><td><code>memory/vector_store.py</code></td><td>Semantic embedding search (ChromaDB) — also used by Living Memory</td></tr>
+          <tr><td><strong>KnowledgeGraph</strong></td><td><code>memory/knowledge_graph.py</code></td><td>Entity-relationship graph with full-text entity search (SQLite)</td></tr>
+          <tr><td><strong>Lawh al-Mahfuz</strong></td><td><code>memory/lawh_mahfuz.py</code></td><td>Immutable memory with 4-layer integrity: SHA-256 + CRC-32 + length + quaternary checksum</td></tr>
         </tbody>
       </table>
       <p>Unified query: <code>memory/memory_pyramid.py</code> merges, deduplicates, and ranks by <code>relevance × certainty × recency</code>.</p>
+      <p><strong>Quaternary Encoding:</strong> DNA-inspired error detection using a 4-symbol alphabet (A, C, G, T). Data is encoded as quaternary strings, chunked into codons (triplets mapping to 64 semantic categories), and verified using XOR parity and Hamming distance checks.</p>
 
       <h3>Yaqin Certainty Engine</h3>
       <p>Every piece of knowledge is tagged with its certainty level:</p>