docs: update project README.md

README.md

@@ -1,63 +1,62 @@
-# Operations Analytics Pipeline: Scalable Integrity Engine
+<div align="center">
 
 [![CI * Code Quality & Tests](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/ci-code.yml/badge.svg)](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/ci-code.yml)
 [![CI * Infra Enforcement](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/ci-infra.yml/badge.svg)](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/ci-infra.yml)
+<br>
 [![CD * Data Pipeline](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/cd-pipeline.yml/badge.svg)](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/cd-pipeline.yml)
 [![CD * Data Extractor](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/cd-extract.yml/badge.svg)](https://github.com/BLMgithub/operations-analytics-pipeline/actions/workflows/cd-extract.yml)
 
-## Overview
-Organizations outgrowing spreadsheet-based workflows often face technical barriers when migrating to relational databases. This transition can lead to inconsistent data structures that limit scaling and reduce reporting reliability.
+</div>
 
-## System Architecture: Event-Driven Integrity
+## System Design: Atomic Medallion Architecture
 
-This project delivers a highly resilient, event-driven data pipeline on Google Cloud Platform designed to defend analytical integrity through a strict Medallion architecture and automated validation gates.
+Event-driven medallion pipeline on Google Cloud Platform. Data moves through gated stages and reaches consumers only as one atomic release, never partial.
 
 ### Isolated Stateless Orchestration
 ![pipeline-orchestration-diagram](assets/diagrams/01-pipeline-orchestration-diagram.png)
 
-To eliminate the risk of cross-run data contamination and memory exhaustion, the system employs isolated execution environments where local compute state is strictly temporary:
-* **Stateless Workspace:** Each run operates in a deterministic `run_id` workspace cleared immediately after completion.
-* **Memory-Optimized Joins:** Maps 36-byte UUID strings to 4-byte UInt32 surrogates, reducing join-key memory overhead by ~16x to maintain serverless resource limits.
-* **Cloud-Native Sync:** After processing the Silver (Contract) layer, the system syncs results to Cloud Storage and purges the local environment.
-* **Linear Integrity Gating:** Stages are strictly gated; failure at any tier (Ingestion, Contract, or Assembly) stops downstream processing to prevent the promotion of partial or malformed data.
-* **Lazy Streaming Engine:** Leverages the Polars Rust engine to process large-scale datasets within the strict memory constraints of serverless Cloud Run instances.
+Each run is isolated; compute state is temporary:
+
+* **Stateless Workspace:** Deterministic `run_id` workspace, cleared immediately after completion.
+* **Cloud-Native Sync:** Silver (Contract) results sync to Cloud Storage, then the local environment is purged.
+* **Linear Integrity Gating:** Failure at any tier (Ingestion, Contract, Assembly) stops downstream processing. Partial or malformed data is never promoted.
+* **Lazy Streaming Engine:** Polars Rust engine processes large-scale datasets within Cloud Run's serverless memory limits.
 
 ### Serverless Infrastructure & Eventarc Triggers
 ![gcp-orchestration-diagram](assets/screenshots/gcp-orchestration-diagram.png)
 
-The underlying infrastructure is entirely serverless, decoupled, and codified via Terraform:
-* **Orchestrated Extraction:** Cloud Scheduler initiates daily extraction via Cloud Run, separating the extraction layer from the main processing logic.
-* **Event-Driven Dispatch:** Eventarc monitors Cloud Storage for `.success` flags, triggering the main processing job via Cloud Workflows only when extraction succeeds.
-* **Zero-Trust Deployment:** GitHub Actions leverage Workload Identity Federation (WIF) for secure, keyless deployments of all infrastructure and containerized jobs.
+Serverless, decoupled, codified via Terraform:
+
+* **Orchestrated Extraction:** Cloud Scheduler initiates daily extraction via Cloud Run, separating extraction from processing logic.
+* **Event-Driven Dispatch:** Eventarc monitors Cloud Storage for `.success` flags, triggering the main job via Cloud Workflows only when extraction succeeds.
+* **Zero-Trust Deployment:** GitHub Actions use Workload Identity Federation (WIF) for keyless deployments of infrastructure and containerized jobs.
 
 ## Data Defense: The Registry Rule Engine
 
-The pipeline actively manages upstream anomalies by enforcing a Medallion architecture governed by a registry-driven validation suite.
+Upstream anomalies are managed by a registry-driven validation suite:
 
 **Bronze (Raw Snapshots)**
-* **Role:** Immutable snapshots of source systems. Data here is assumed to be structurally untrustworthy, containing nulls, duplicates, or orphaned records.
+* Immutable snapshots of source systems. Assumed structurally untrustworthy: nulls, duplicates, orphaned records.
 
 **Silver (The Contract Layer)**
-* **Primitive Integer Pipeline:** Maps 36-byte UUID strings to 4-byte UInt32 surrogates. This reduces  join-key memory overhead by ~16x and ensuring the pipeline stays within serverless memory constraints.
-* **Subtractive-Only Logic:** The pipeline never guesses or "repairs" bad data. Records violating the contract are explicitly dropped and logged in the telemetry report.
-* **Cascade Cleanup:** The system tracks invalidated parent IDs and propagates drops downstream, ensuring child records (like line items) are removed to prevent orphaned data in joins.
-* **Schema Enforcement:** Output files are strictly cast to predefined types and projected to approved columns before storage.
+* **Primitive Integer Pipeline:** Maps 36-byte UUID strings to 4-byte UInt32 surrogates, cutting join-key memory overhead ~16x.
+* **Subtractive-Only Logic:** Never guesses or "repairs" bad data. Non-compliant records are dropped and logged in the telemetry report.
+* **Cascade Cleanup:** Invalidated parent IDs propagate drops to child records (e.g., line items); orphans never reach joins.
+* **Schema Enforcement:** Output strictly cast to predefined types and projected to approved columns before storage.
 
 **Gold (The Semantic Layer)**
-* **Assembly Stage:** Integrates normalized relational tables into a unified analytical dataset, enforcing a 1:1 grain per order.
-* **Semantic Stage:** Transforms events into specialized Fact and Dimension modules tailored for entity-centric analysis (Sellers, Customers, Products).
-* **Strict Grain Enforcement:** Fact tables are deterministically aligned to an ISO-Week grain (`W-MON`) with exactly one row per `(Entity_ID, order_year_week)`.
+* **Assembly Stage:** Integrates normalized relational tables into a unified analytical dataset at a 1:1 grain per order.
+* **Semantic Stage:** Transforms events into entity-centric Fact and Dimension modules (Sellers, Customers, Products).
+* **Strict Grain Enforcement:** Fact tables aligned to an ISO-Week grain (`W-MON`), one row per `(Entity_ID, order_year_week)`.
 
 ### Integrity Gates & Atomic Deployment
 
-* **Dual-Pass Validation Strategy:**
-    * **Initial Raw Gate:** Evaluates raw snapshots. Structural warnings are tolerated but passed to the Silver stage for subtractive cleanup.
-    * **Post-Contract Silver Gate:** Re-validates data after contract rules are applied. Remaining warnings are escalated to fatal errors, triggering a `RuntimeError` to stop downstream corruption.
-* **Atomic BigQuery Publishing:** Final semantic models are delivered via Authorized Views that atomically swap pointers to new data versions. This ensures BI tools always query complete, validated datasets with no downtime during updates.
-
-## Performance & Scalability (Cloud-Native Benchmarks)
+* **Dual-Pass Validation:**
+    * **Initial Raw Gate:** Tolerates structural warnings; passes them to Silver for subtractive cleanup.
+    * **Post-Contract Silver Gate:** Escalates remaining warnings to fatal errors (`RuntimeError`), stopping downstream corruption.
+* **Atomic BigQuery Publishing:** Authorized Views atomically swap pointers to new data versions. BI tools query complete, validated datasets with no downtime.
 
-By leveraging the Polars Rust engine (Lazy API), the system achieves near-optimal resource utilization within the rigid memory constraints of serverless compute.
+## Performance & Scalability
 
 ### GCP Stress-Test Metrics
 
@@ -74,72 +73,55 @@ By leveraging the Polars Rust engine (Lazy API), the system achieves near-optima
 | Efficiency (Processing) | ~307k Rows / Second |
 | Total Runtime (Wall-Clock) | 130 Seconds |
 
-
-*   **Maximized Memory Density:** The **Primitive Integer Pipeline** allows a ~5.34GB analytical model to process within the 8GB RAM limit by shrinking join-key overhead by ~16x.
-*   **Near-Linear Performance Scaling:** The engine saturates available vCPUs, yielding high throughput during streaming execution.
-*   **Zero-Idle Economics:** 100% serverless execution ensures zero billable time during idle periods.
+* **Memory Density:** The Primitive Integer Pipeline shrinks join-key overhead ~16x, fitting a ~5.34GB analytical model inside the 8GB RAM limit.
+* **Zero-Idle Economics:** 100% serverless execution, no billable time when idle.
 
 ### Measurement Methodology
-*   **Performance Profiling:** Captured from production telemetry via the pipeline's native `run_duration` metadata, calculating the precise delta between `started_at` and `completed_at` timestamps.
-*   **Memory Utilization:** Monitored via an integrated [`psutil.virtual_memory().used`](assets/benchmarks/polars/) profiling implementation to verify the actual resource footprint and confirm the physical ceiling for 8GB provision.
-
-### **Scaling Roadmap: From Serverless to Enterprise Lakehouse**
-
-#### **Stage 1: Incremental Delta Propagation**
-*   **Strategy:** Transition to a "Stateless Delta Propagation" model using Polars' streaming engine to process only new `.parquet` deltas, reducing I/O and CPU time by 80-90%.
-
-#### **Stage 2: Event-Driven Real-Time Streaming**
-*   **Strategy:** Integrate GCS Pub/Sub notifications with Cloud Run streaming sinks to trigger sub-minute validation and assembly as files are uploaded.
-
-#### **Stage 3: BigQuery "Engine-as-a-Service"**
-*   **Strategy:** Offload high-volume compute layers entirely to BigQuery using SQL-driven logic. Provides petabyte-scale capacity while the Python pipeline manages integrity gates.
+* **Performance Profiling:** Delta between `started_at` and `completed_at` from the pipeline's native `run_duration` telemetry.
+* **Memory Utilization:** [`psutil.virtual_memory().used`](assets/benchmarks/polars/) profiling verifies the resource footprint against the 8GB ceiling.
 
 ## System Health & Observability
 
 ![ops_dashboard_monitoring](assets/screenshots/ops-analytics-pipeline-db.png)
 
-The pipeline features a comprehensive observability suite managed natively via Google Cloud Monitoring and Cloud Logging, codified entirely in Terraform.
-
-### Monitored Telemetry & Alerting
-The system tracks granular operational metrics to proactively identify resource bottlenecks and execution failures:
+Observability via Cloud Monitoring and Cloud Logging, codified in Terraform:
 
-* **Pipeline Job Metrics:** Tracks execution status (Success/Fail), workflow traffic, and memory allocation bottlenecks against the 8GB threshold.
-* **Extractor Job Metrics:** Monitors Drive API latencies and instance billable time to track API usage costs.
-* **Automated Responders:** Dispatches `CRITICAL` email alerts for ingestion failures, extractor crashes, or pipeline fatal errors (OOMs), ensuring debuggability through resilient lineage tracking.
+* **Pipeline Job Metrics:** Execution status (Success/Fail), workflow traffic, memory allocation against the 8GB threshold.
+* **Extractor Job Metrics:** Drive API latencies and instance billable time for API usage cost tracking.
+* **Automated Responders:** `CRITICAL` email alerts on ingestion failures, extractor crashes, or pipeline fatal errors (OOMs), with lineage tracking for debuggability.
 
 ## Operational Intelligence: BI Decision Support
 
-This architecture serves the Presentation Layer with high reliability, ensuring that dashboards are built upon validated, semantically flattened data models.
+Dashboards are built on validated, semantically flattened models.
 
 > **Dynamic Sensitivity Calibration**
 >
->The reporting suite features interactive "Smoke Detectors." Built with dynamic What-If parameters, these dashboards allow operators to manually adjust alert sensitivity thresholds to match changing business realities.
+>Interactive "Smoke Detectors" with What-If parameters let operators adjust alert sensitivity thresholds to match changing business realities.
 >
 > Explore the  **[Power BI Directory](/power_bi)** to read detailed [operational guides](power_bi/docs) or download the `.pbix` [releases](power_bi/releases/).
 
 ### Customer Experience & Revenue Exposure
-Monitors financial risk by correlating delivery delays with buyer drop-off rates, allowing leadership to quantify the "cost of friction."
+Correlates delivery delays with buyer drop-off rates to quantify financial risk from "cost of friction."
 
 ![Customer Experince Image](assets/gif/customer_experience_exposure_demo.gif)
 
 ### Fulfillment Decision Monitor
-An operational early-warning system focusing on statistical deviations in network speed rather than total failure to identify partners requiring intervention.
+Early-warning system on statistical deviations in network speed, not total failure, to flag partners requiring intervention.
 
 ![Fulfillment Decision Monitor](assets/gif/fulfillment_decision_monitor_demo.gif)
 
 ### Product Friction Monitor
-Identifies structural fulfillment bottlenecks driven by product specifications (e.g., weight outliers) to route items to specialized freight.
+Flags structural fulfillment bottlenecks from product specifications (e.g., weight outliers) to route items to specialized freight.
 
 ![Product Friction Monitor](assets/gif/product_friction_monitor_demo.gif)
 
 ## CI/CD & Security
 
-The project adheres to a strict **Zero-Trust** deployment model. 
+Zero-Trust deployment model:
 
 * **Workload Identity Federation (WIF):** Authenticates GitHub Actions to Google Cloud via short-lived OIDC tokens.
-* **Infrastructure as Code:** IAM bindings and infrastructure are strictly managed via automated Terraform workflows.
-* **Containerized Artifacts:** Codebases are packaged into Docker images and pushed to the GCP Artifact Registry only after passing CI checks.
-
+* **Infrastructure as Code:** IAM bindings and infrastructure managed via automated Terraform workflows.
+* **Containerized Artifacts:** Code is packaged into Docker images and pushed to GCP Artifact Registry only after CI checks pass.
 
 ## Repository Structure
 
@@ -174,4 +156,4 @@ operations-analytics-pipeline/
     ├── .shared/            # Global Dashboards BI assets (e.g. Themes, .json files, etc.)
     ├── dashboards/         # Source Control (PBIP) 
     └── releases            # Deliverables (PBIX)
-```
+```