🎧 SoundSense

Vocal Anomaly Detection: Identifying Emotional Inconsistencies in Speech

SoundSense is a research project focused on detecting emotional mismatches between what is said (text) and how it's said (audio). By combining NLP and voice analysis, we aim to identify vocal anomalies — such as when someone says "I'm fine" but sounds sad.

---

🗂️ Datasets Used

1. CREMA-D – Crowd-sourced Emotional Multimodal Actors Dataset
   🔗 View on Kaggle

2. Emotion Dataset (Text-Based) – Twitter text samples labeled with emotional categories
   🔗 View on Kaggle

---

📌 Project Objectives

• 🎙️ Voice Emotion Classification using audio features like tone, pitch, and pace
• 🧠 Text-Based Emotion Detection using pre-trained BERT models
• ⚖️ Cross-Modal Comparison to flag inconsistencies between vocal tone and spoken content

---

📬 Contact

📩 DM or email me for access to:

• The cleaned datasets
• Trained models

---

🔄 System Flow

The SoundSense anomaly detection server processes user audio through a multi-modal pipeline:

1. Audio Preprocessing: Uploaded audio is normalized to a standard mono 16kHz format and its volume is scaled.
2. Transcription (ASR): The spoken audio is converted to text using a pre-trained Wav2Vec2 Automatic Speech Recognition model.
3. Text Emotion: The transcribed text is analyzed by a RoBERTa model to extract the intended text-based emotion (e.g., joy, sadness, fear).
4. Audio Emotion Ensemble: The vocal tone is analyzed by an ensemble of four distinct models for robustness:
   • Wav2Vec2 (Deep Learning embeddings)
   • CNN (Mel Spectrogram processing)
   • SVM (MFCC, Chroma, ZCR, RMS features via StandardScaler)
   • Random Forest (Traditional ML tree logic via StandardScaler)
5. Anomaly Evaluation: The predicted text emotion is compared against the ensemble's vocal tone predictions. If the tone contradicts the text (e.g., saying "I'm so happy" in a sad tone), the system flags a Vocal Anomaly.

---

🚀 Quick Start

Ensure you have your environment set up and all required models downloaded in their respective folders.

1. Clone the project:

   git clone https://github.com/s4nkar/SoundSense.git
   cd SoundSense

2. Start the Anomaly Detection Server: The backend is powered by FastAPI. Run the server using uvicorn:

   uvicorn server:app --host 0.0.0.0 --port 8000

3. Access the Web Interface: Open your browser and navigate to: http://localhost:8000/

(Note: If you are training from scratch, you can use the provided Jupyter Notebooks like RF-SVM-model.ipynb and audio-text-calssification.ipynb before running the server).