Jarvis — Voice-Activated AI Assistant for macOS

A real-time, voice-activated AI assistant inspired by Iron Man's J.A.R.V.I.S., built for macOS. Say "Hey Jarvis" or double-clap to activate — Jarvis listens, thinks, and responds in a natural voice with full tool use.

---

Features

• Wake word detection — say "Hey Jarvis" using openWakeWord (on-device, no API key, <5% CPU)
• Double-clap activation — RMS spike detection opens the full-screen HUD silently
• Streaming speech recognition — OpenAI Realtime API with gpt-4o-transcribe and server-side VAD; mic opens instantly so no words are cut off
• GPT-4o brain — function calling with 12+ tools, 10-turn conversation memory
• Streaming voice — ElevenLabs eleven_flash_v2_5 (~75ms latency) streamed directly to audio output via PCM — no temp files, no wait
• Full-screen Iron Man HUD — live dashboard with weather, calendar, stocks, CPU/RAM/battery, triggered by double clap
• Menu bar app — colored status dot (grey → blue → yellow → green) reflects sleeping / listening / thinking / speaking

---

What Jarvis Can Do

Voice Command	What Happens
"Open Cursor" / "Open Chrome"	Opens any macOS application
"Search for neural networks"	Opens Google in your browser
"What's the weather in Tokyo?"	Live weather via Open-Meteo (no key needed)
"What time is it?"	Current date and time
"Play Blinding Lights on Spotify"	Searches and plays the track
"Pause / Next / Previous"	Controls Spotify playback
"Send a message to Mom saying I'm on my way"	Sends iMessage or SMS
"Set a 10 minute pasta timer"	Countdown timer with spoken alert
"Turn the volume up" / "Mute"	Controls macOS system volume
"What assignments do I have due?"	Fetches from Moodle LMS
"What's my schedule this week?"	Upcoming Moodle calendar events
"Open my most recent file on the Desktop"	Opens latest modified file
"Who are you?"	Plays the Iron Man Jarvis intro
"Clear memory"	Resets conversation history

---

Architecture

"Hey Jarvis" / Double clap
        │
        ▼
  WakeWordDetector     ← openWakeWord neural hotword + RMS clap detection
        │
        ▼
  AudioRecorder        ← PyAudio mic (opens immediately, buffered)
        │              ← OpenAI Realtime API WebSocket + server-side VAD
        │              ← gpt-4o-transcribe
        ▼
  JarvisAgent          ← GPT-4o with function calling + 10-turn memory
        │
        ▼
  Speaker              ← ElevenLabs eleven_flash_v2_5 → PCM stream → PyAudio
        │
  ┌─────┴──────┐
  │            │
 HUD         Menu bar  ← pywebview subprocess + rumps (AppKit)

---

Setup

1. Clone and create environment

git clone https://github.com/AboodFares/Jarvis.git
cd Jarvis
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Install system dependencies

brew install portaudio ffmpeg

3. Configure .env

OPENAI_API_KEY=your_key_here
ELEVENLABS_API_KEY=your_key_here
ELEVENLABS_VOICE_ID=your_voice_id_here

# Your location (used for weather when no city is specified)
DEFAULT_CITY=Montreal
DEFAULT_LAT=45.5017
DEFAULT_LON=-73.5673

# Moodle (optional — for assignment and schedule tools)
MOODLE_URL=https://moodle.yourcollege.ca
MOODLE_TOKEN=your_token_here

4. Run

source venv/bin/activate
python main.py

---

Tech Stack

Layer	Technology
Wake word	openWakeWord — on-device ONNX neural model
Clap detection	PyAudio + NumPy RMS
Speech-to-text	OpenAI Realtime API (gpt-4o-transcribe, server-side VAD)
AI brain	GPT-4o with function calling
Text-to-speech	ElevenLabs eleven_flash_v2_5 — PCM streaming via PyAudio
HUD dashboard	pywebview (standalone subprocess)
Menu bar	rumps (AppKit)
Weather	Open-Meteo API (no key needed)
Stocks	yfinance
System stats	psutil
Calendar / schedule	Moodle REST API

---

Project Structure

Jarvis/
├── main.py              # Entry point — pipeline loop + menu bar
├── config.py            # API keys, audio tuning, thresholds
├── brain/
│   └── agent.py         # GPT-4o agent with tool dispatch
├── wake_word/
│   └── detector.py      # openWakeWord + double-clap detection
├── audio/
│   └── recorder.py      # OpenAI Realtime API streaming recorder
├── voice/
│   └── speaker.py       # ElevenLabs streaming TTS
├── tools/
│   ├── apps.py          # Open macOS applications
│   ├── spotify.py       # Spotify control via AppleScript
│   ├── imessage.py      # Send iMessage / SMS
│   ├── volume.py        # macOS system volume
│   ├── timer.py         # Countdown timers with spoken alerts
│   ├── moodle.py        # Moodle assignments + schedule
│   ├── weather.py       # Open-Meteo weather
│   ├── datetime_tool.py # Current date and time
│   └── ...
├── dashboard/
│   ├── hud_window.py    # pywebview HUD subprocess entry
│   ├── hud_html.py      # Iron Man HUD HTML/CSS/JS
│   └── data_fetcher.py  # Fetches live data for HUD panels
└── ui/
    ├── menu_bar.py      # rumps menu bar app
    └── state.py         # Shared state (status, last heard, last response)

---

Requirements

• macOS 13+ (Apple Silicon or Intel)
• Python 3.9+
• Microphone access
• OpenAI API key (STT + brain)
• ElevenLabs API key (optional — falls back to macOS say)