You can deploy this model and start earning money today!

Vighneshkumbhar/
jarvis-v2

Name: jarvis-v2
Availability: InStock

Views

No views yet

🤖 J.A.R.V.I.S v2 — Multi-Agent AI Assistant

     ██╗ █████╗ ██████╗ ██╗   ██╗██╗███████╗
     ██║██╔══██╗██╔══██╗██║   ██║██║██╔════╝
     ██║███████║██████╔╝██║   ██║██║███████╗
██   ██║██╔══██║██╔══██╗╚██╗ ██╔╝██║╚════██║
╚█████╔╝██║  ██║██║  ██║ ╚████╔╝ ██║███████║
 ╚════╝ ╚═╝  ╚═╝╚═╝  ╚═╝  ╚═══╝  ╚═╝╚══════╝

An intelligent multi-agent voice assistant built with Python

Whisper STT • Ollama LLM • Edge-TTS • Multi-Agent Architecture • Plugin System

Python 3.8+

License: MIT

✨ What's New in v2

Feature	v1	v2
Intent Classification	Simple ACTION/CHAT	7 categories: ACTION, CHAT, MEMORY, RESEARCH, UTILITY, SYSTEM_QUERY, META
Conversation Memory	None	4-tier: Working, Episodic, Semantic, Procedural
Commands	3 (open, search, type)	50+ cross-platform commands
Error Handling	Crashes on errors	Graceful degradation at every level
Audio Recording	Fixed 5s	Adaptive VAD (stops on silence)
TTS	Single voice, no cache	10 voices, audio caching, rate control
Plugin System	None	Dynamic plugin registry with auto-discovery
Built-in Plugins	None	Weather, Wikipedia, Web Search, Notes, Reminders, Timer, Calculator, System Monitor, Date/Time, Jokes
Transcription	Basic	Confidence scoring, language detection, model switching
Platform Support	Windows only	Windows, macOS, Linux
CLI Interface	None	Full CLI with 15+ commands
Safety	None	Command blocklist, safe mode, timeout protection
Logging	Print statements	Structured logging with rotation
Configuration	Hardcoded	Environment variables + feature flags

🏗️ Architecture

                          ┌─────────────────────────────┐
                          │        main.py              │
                          │    (Orchestrator/CLI)       │
                          └──────────┬──────────────────┘
                                     │
          ┌──────────────────────────┼──────────────────────────┐
          │                          │                          │
    ┌─────▼─────┐           ┌───────▼───────┐           ┌─────▼─────┐
    │  Listener  │           │     Brain     │           │  Speaker  │
    │  (Audio)   │           │  (Multi-Agent │           │  (TTS)    │
    │  + VAD     │           │   Orchestrator│           │  + Cache  │
    └─────┬─────┘           │   + Memory)   │           └───────────┘
          │                  └───────┬───────┘
    ┌─────▼──────┐                   │
    │ Transcriber │     ┌────────────┼────────────────┐
    │  (Whisper)  │     │            │                │
    └─────────────┘     │            │                │
                   ┌────▼────┐  ┌────▼────┐    ┌─────▼──────┐
                   │Executor │  │ Memory  │    │  Plugins   │
                   │ (50+    │  │ Manager │    │ (10 built- │
                   │commands)│  │ (4-tier)│    │  in + ext) │
                   └─────────┘  └─────────┘    └────────────┘

Intent Classification Flow

User Input → LLM Zero-Shot Classifier (+ keyword fallback)
    │
    ├── ACTION ────────→ Executor (open apps, search, volume, files, etc.)
    ├── CHAT ──────────→ Conversational AI with memory context
    ├── MEMORY ────────→ Store/recall/forget facts
    ├── RESEARCH ──────→ Wikipedia, web search, informational queries
    ├── UTILITY ───────→ Calculator, timer, reminders, translator, notes
    ├── SYSTEM_QUERY ──→ System info, battery, disk, processes
    └── META ──────────→ Jarvis self-commands (voice, stats, help)

📂 Project Structure

jarvis-v1/
├── main.py                  # Entry point — orchestrator + CLI
├── config.py                # Central configuration with env overrides
├── requirements.txt         # Dependencies
├── .env.example             # Environment template
│
├── core/
│   ├── brain.py             # Multi-agent brain — intent routing + LLM
│   ├── listener.py          # Audio input — VAD, adaptive recording
│   ├── transcriber.py       # Whisper STT — confidence, language detect
│   ├── speaker.py           # Edge-TTS — multi-voice, caching
│   └── executor.py          # 50+ cross-platform system commands
│
├── memory/
│   └── manager.py           # 4-tier memory: working, episodic, semantic, procedural
│
├── plugins/
│   ├── __init__.py          # Plugin base class + registry
│   └── builtin.py           # 10 built-in plugins
│
├── agents/                  # Extensible agent modules
├── utils/
│   ├── helpers.py           # Retry, JSON parsing, text cleaning, formatting
│   └── logger.py            # Colored console + rotating file logger
│
├── audio/                   # Audio files + TTS cache
├── data/                    # Notes, reminders, memory persistence
└── logs/                    # Log files

🚀 Quick Start

1. Clone & Setup

bash

git clone https://github.com/vighneshkumbhar004/jarvis-v1.git
cd jarvis-v1

# Create virtual environment
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
# .venv\Scripts\activate         # Windows

# Install dependencies
pip install -r requirements.txt

2. Install & Start Ollama

bash

# Install from https://ollama.ai
ollama serve                     # Start the server
ollama pull llama3.1             # Pull the model

3. Configure (Optional)

bash

cp .env.example .env
# Edit .env with your preferences

4. Run

bash

# CLI mode (default — type commands)
python main.py

# Voice mode (continuous voice input)
python main.py --voice

# Help
python main.py --help

🎯 Features

🎤 Voice Input

Adaptive Recording: Automatically stops when you pause speaking (VAD)
Whisper STT: Multiple model sizes (tiny → large-v3)
Confidence Scoring: Rejects low-confidence transcriptions
Language Detection: Auto-detect spoken language

🧠 Multi-Agent Brain

7-Category Intent Classification: LLM-powered with keyword fallback
Conversation Memory: Full context across turns
Multi-Step Reasoning: Complex queries handled through agent routing
Graceful Degradation: Falls back through classification layers

💾 4-Tier Memory System

Tier	Purpose	Persistence
Working	Current conversation context (last 10 turns)	Session only
Episodic	Timestamped events and interactions	Disk (JSON)
Semantic	User facts and preferences	Disk (JSON)
Procedural	Learned command patterns	Disk (JSON)

⚡ 50+ System Commands

Full Command List (click to expand)

Apps: open, close, kill any app by name Web: search, open URL Input: type text Screen: screenshot, lock screen Volume: up, down, mute/unmute Brightness: up, down Power: sleep (shutdown blocked for safety) Files: open folder, open file, create folder, delete file, list files Clipboard: copy, paste System: system info, disk usage, battery status, list processes, kill process Network: Wi-Fi toggle, Bluetooth toggle, IP address, ping Trash: empty recycle bin Custom: run any shell command (safe mode gated)

🔌 Plugin System (10 Built-in)

Plugin	Features
Weather	Current weather via OpenWeatherMap API
Wikipedia	Article summaries via REST API
Web Search	DuckDuckGo instant answers
System Monitor	CPU, RAM, disk, network, uptime, top processes
Notes	Create, list, search, delete persistent notes
Reminders	Time-based reminders with natural language parsing
Timer	Countdown timers + stopwatch
Calculator	Math, trig, safe eval with natural language
Date/Time	Current time, date, day, countdowns
Jokes	Programming jokes and fun facts

🔊 Text-to-Speech

10 Voice Options: Ryan, Sonia, Jenny, Guy, Aria, Davis, Tony, Emma, Indian, Australian
Audio Caching: Repeated phrases play instantly
Rate & Volume Control: Adjustable speech speed
Auto Cleanup: Temporary files managed automatically

🛡️ Safety & Reliability

Command Blocklist: Dangerous commands blocked (rm -rf /, format, shutdown, etc.)
Safe Mode: File deletion and process killing gated
Timeout Protection: All commands have execution timeouts
Retry Logic: Exponential backoff on API failures
Graceful Degradation: Every pipeline stage has fallbacks

💬 CLI Commands

Command	Description
`/help`	Show help with all features
`/voice`	Switch to voice input for one command
`/voice ryan`	Change TTS voice
`/voices`	List available voices
`/stats`	Show usage statistics
`/memory`	View stored memories
`/clear`	Clear conversation history
`/reset`	Reset all memory
`/plugins`	List loaded plugins
`/apps`	List available apps for your OS
`/config`	Show current configuration
`/history`	Show command execution history
`/cleanup`	Clean temporary audio files
`/quit`	Exit JARVIS

🎤 Usage Examples

You: open chrome
🤖 Jarvis: Launching chrome now.

You: what's the capital of Japan?
🤖 Jarvis: The capital of Japan is Tokyo. It's the most populous...

You: calculate 15 * 24 + sin(3.14)
🔧 Jarvis: The answer is 360.00159...

You: remember my name is Vighnesh
🧠 Jarvis: Got it! I'll remember that.

You: what's my name?
🤖 Jarvis: Your name is Vighnesh.

You: set a timer for 5 minutes
⏱️  Timer set for 5 minutes.

You: tell me about quantum computing
🔍 Jarvis: Quantum computing uses quantum-mechanical phenomena...

You: system info
💻 System: OS: Linux 5.15 | CPU: 12% | RAM: 8GB/16GB (50%)

You: tell me a joke
🤖 Jarvis: Why do programmers prefer dark mode? Because light attracts bugs.

You: volume up
🤖 Jarvis: Turning the volume up.

You: /stats
📊 Session uptime: 15 minutes | Total: 12 | Actions: 4 | Chats: 8

🔧 Configuration

All settings can be overridden via environment variables or .env file:

Variable	Default	Description
`OLLAMA_MODEL`	`llama3.1`	LLM model name
`OLLAMA_BASE_URL`	`http://localhost:11434`	Ollama API URL
`WHISPER_MODEL`	`base`	Whisper model size
`TTS_VOICE`	`en-GB-RyanNeural`	Default TTS voice
`MEMORY_ENABLED`	`true`	Enable memory system
`SAFE_MODE`	`true`	Enable safety guardrails
`LOG_LEVEL`	`INFO`	Logging verbosity
`WEATHER_API_KEY`	(empty)	OpenWeatherMap API key

🔌 Writing Custom Plugins

Create a new file in plugins/ directory:

python

# plugins/my_plugin.py
from plugins import JarvisPlugin

class MyPlugin(JarvisPlugin):
    name = "my_plugin"
    description = "My custom JARVIS plugin"
    version = "1.0.0"

    def get_commands(self):
        return ["my_command", "another_command"]

    def execute(self, command, params=""):
        if command == "my_command":
            return {
                "success": True,
                "result": f"Executed with params: {params}",
                "speak": "Done!"
            }
        return {"success": False, "result": "Unknown command"}

Plugins are auto-discovered and registered at startup.

📋 Dependencies

Package	Purpose	Required
`openai-whisper`	Speech-to-text	✅ (for voice)
`edge-tts`	Text-to-speech	✅ (for voice)
`requests`	Ollama API client	✅
`numpy`	Audio processing	✅
`sounddevice`	Audio recording	✅ (for voice)
`pygame`	Audio playback	✅ (for voice)
`psutil`	System monitoring	Recommended
`pyautogui`	Input automation	Optional
`pyperclip`	Clipboard access	Optional

🐛 Troubleshooting

Issue	Solution
Ollama connection error	Run `ollama serve` and `ollama pull llama3.1`
No audio input	Check microphone permissions and `sounddevice` installation
TTS not working	Install `edge-tts` and check internet connection
Slow responses	Use smaller Whisper model (`tiny`) or faster Ollama model
Commands not working	Check platform-specific commands with `/apps`

🛣️ Roadmap

Wake word detection (pvporcupine / openwakeword)
Streaming TTS for faster responses
Multi-language support
Smart home integration (Home Assistant)
Web UI (Gradio/Streamlit)
Cloud LLM support (GPT-4, Claude, Gemini)
MCP server integration for tool use
Voice cloning (your own voice)
Persistent conversation RAG
Mobile companion app

📝 License

MIT License — see LICENSE for details.

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Made with ❤️ by Vighnesh Kumbhar

"Sometimes you gotta run before you can walk." — Tony Stark

Vighneshkumbhar/jarvis-v2

🤖 J.A.R.V.I.S v2 — Multi-Agent AI Assistant

✨ What's New in v2

🏗️ Architecture

Intent Classification Flow

📂 Project Structure

🚀 Quick Start

1. Clone & Setup

2. Install & Start Ollama

3. Configure (Optional)

4. Run

🎯 Features

🎤 Voice Input

🧠 Multi-Agent Brain

💾 4-Tier Memory System

⚡ 50+ System Commands

🔌 Plugin System (10 Built-in)

🔊 Text-to-Speech

🛡️ Safety & Reliability

💬 CLI Commands

🎤 Usage Examples

🔧 Configuration

🔌 Writing Custom Plugins

📋 Dependencies

🐛 Troubleshooting

🛣️ Roadmap

📝 License

🤝 Contributing

Vighneshkumbhar/
jarvis-v2