Aura (PsychIT) is a local-first, tri-modal machine learning pipeline designed to detect hidden psychological distress. By analyzing how a person speaks alongside what they say, and fusing that data with physiological telemetry, Aura establishes "Emotional Dissonance" as a quantifiable digital biomarker.
🛑 The Problems Aura Solves
Current digital wellness applications suffer from three critical flaws:
The "Performance Problem": Users frequently mask their true feelings with neutral text. Unimodal text journals cannot detect when a user writes "I'm fine" but sounds deeply distressed.
The "Low Activity Paradox": Wearables (like Google Fit) blindly flag sudden drops in physical activity as depressive lethargy. Without context, a user studying for exams for three days straight will trigger a false-positive depression alert.
Data Privacy: Sending deeply personal audio and health data to cloud-based LLM APIs poses severe biometric security risks.
Aura solves this by comparing acoustic vocal tone against semantic transcriptions locally, and grounding physiological data in qualitative journal themes using a Retrieval-Augmented Generation (RAG) engine.
🏗️ System Architecture (The "Y-Split" Pipeline)
Aura ingests data asynchronously across three distinct streams:
Paralinguistic Stream (Acoustic): Raw audio is dynamically downsampled to 16 kHz and processed by our fine-tuned Wav2Vec 2.0 model to detect the underlying emotional tone.
Semantic Stream (Text): Audio is transcribed verbatim using OpenAI's Whisper. The text is then analyzed by a locally hosted Llama 3 (8B) model to extract thematic drivers (e.g., "Exam Pressure", "Social Isolation").
Physiological Stream (Context): The Google Fit REST API polls daily step counts, while GHQ-12 psychometrics anchor the data to validated clinical baselines.
🔍 Detecting Emotional Dissonance
The Dissonance Engine compares the Acoustic output with the Semantic output. If a user sounds "Sad" or "Anxious" but their text sentiment is transcribed as "Calm" or "Joyful", Aura flags this as a Critical Mismatch (Suppressed Emotion).
Dissonance Mapping
Figure 1: Cross-Modal mapping highlighting active emotional suppression.
🧠 Model Training & Performance Metrics
The core acoustic model is a fine-tuned facebook/wav2vec2-base architecture equipped with a custom Dense + Dropout classification head, trained on the CREMA-D and TESS datasets.
Epochs: 4 (1,152 total steps)
Batch Size: 4
Learning Rate Scheduler: Linear LR
Optimal Checkpoint: Step 1040 (Early Stopping)
Peak Validation F1-Score: 83.0%
Final Evaluation Loss: 0.661
Training Convergence
Figure 2: Dual-axis training convergence. Divergence after step 1040 justifies the early-stopping threshold.
⚙️ Installation & Local Setup
Aura is designed for local inference to ensure zero cloud exposure of biometric data. Due to the inclusion of a 3.5 GB fine-tuned model, Git LFS is required for installation.
Ollama running locally with the Llama 3 model installed (ollama run llama3)
2. Clone the Repository
Initialize Git LFS and clone the repository to download the codebase and model weights:
bash
1git lfs install2git clone [https://huggingface.co/Blackfyred/Aura-Speech-Emotion-Model](https://huggingface.co/Blackfyred/Aura-Speech-Emotion-Model)3cd Aura-Speech-Emotion-Model
43. Install Dependencies
5Create a secure virtual environment and install the required Python packages:
67Bash
8python -m venv venv
910# Activate on Windows:11venv\Scripts\activate
12# Activate on Linux/Mac:13source venv/bin/activate
1415pip install -r requirements.txt
164. Configure Secure Credentials
17To enable the Physiological Stream, you must provide your own Google Fit API credentials.
1819Place your client_secret.json in the src/ directory.
2021Note: For security, client_secret.json and token.json are strictly ignored by .gitignore and will never be pushed to version control.
2223🚀 Usage
24Launch the Streamlit user interface from the root directory:
2526Bash
27streamlit run src/app.py
28Record a Voice Journal: Use the Streamlit UI to record directly or upload a .wav file.
2930Observe the Split Analysis: The application will transcribe your words (Semantic) and analyze your tone (Acoustic) simultaneously.
3132Review Contextual Feedback: The Kai RAG engine will cross-reference your journal themes with your Google Fit activity vector.
3334Example: If your physical activity drops below 1,000 steps, but your extracted theme is "Exam Prep", the RAG engine will successfully recontextualize the low activity as high-focus productivity, thereby resolving the Low Activity Paradox.
3536Figure 3: Resolving a false-positive depressive flag using thematic RAG fusion.
3738📁 Repository Structure
39Plaintext
40├── papers/ # Research figures, diagrams, and evaluation metrics41├── src/ # Core application source code42│ ├── app.py # Streamlit frontend UI43│ ├── audio_analyzer.py # Wav2Vec 2.0 inference pipeline44│ ├── llama_integration.py # Ollama / Llama 3 semantic extraction45│ ├── health_integration.py# Google Fit API telemetry46│ └── report_generator.py # Dissonance engine and RAG fusion47├── my_speech_model/ # 3.5GB Fine-tuned Wav2Vec 2.0 safetensors & config48├── .gitattributes # Git LFS tracking rules49├── .gitignore # Security and environment exclusions50└── requirements.txt # Python dependency manifest51Developed and maintained by the PsychIT Research Team.
