Echo Mind Orchestrator

AI Avatar Orchestration Service - Coordinates STT, LLM, Memory, and TTS for real-time conversational AI avatars.

🎯 What This Is

The Echo Mind Orchestrator is the central service that coordinates all components needed for a conversational AI avatar:

• Audio Processing: Real-time speech-to-text via Deepgram
• Intelligence: OpenAI GPT-4 with function calling
• Memory: Persistent conversation memory via meMOS + Neo4j
• Avatar: HeyGen integration for realistic avatar responses
• Tools: Web search, time, weather, memory search capabilities

🚀 Quick Start

# 1. Setup environment
cp .env.example .env
# Add your API keys: OPENAI_API_KEY, DEEPGRAM_API_KEY, HEYGEN_API_KEY, etc.

# 2. Start meMOS memory system
cd MemOS/docker && docker compose up -d

# 3. Start orchestrator
python main.py

# 4. Test avatar
open http://localhost:8000/test_avatar_interface.html

🏗️ Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Frontend      │    │   Orchestrator   │    │   Memory        │
│                 │    │                  │    │                 │
│ • React App     │◄──►│ • FastAPI        │◄──►│ • meMOS         │
│ • LiveKit       │    │ • WebSocket      │    │ • Neo4j         │
│ • Microphone    │    │ • Tool System    │    │ • Vector Store  │
└─────────────────┘    └──────────────────┘    └─────────────────┘
          │                         │
          │            ┌────────────┼────────────┐
          │            ▼            ▼            ▼
          │      ┌──────────┬──────────┬──────────┐
          │      │ Deepgram │ OpenAI   │ HeyGen   │
          │      │   STT    │   LLM    │  Service │
          │      └──────────┴──────────┴──────────┘
          │                                   │
          │                  LiveKit WebRTC  │
          └───────────────────────────────────┘
                        Video + Audio

🧠 Memory System

Powered by meMOS + Neo4j:

• Graph Memory: Relationships between entities, conversations, topics
• Vector Search: Semantic similarity matching
• Persistent Storage: Conversations preserved across sessions
• Context Aware: Automatically includes relevant memory in responses

🛠️ Tool System

Available Functions:

• get_current_time - Current time in any timezone
• get_weather - Weather information
• web_search - Real-time web search via Brave API
• search_memory - Search conversation history

📊 System Status

# Check orchestrator health
curl http://localhost:8000/api/v1/health

# Check memory system status  
curl http://localhost:8000/api/v1/memory/status

# Expected: {"memory_type": "memos_chat", "status": "connected"}

🔧 Configuration

Required Environment Variables:

# Core APIs
OPENAI_API_KEY=sk-...
DEEPGRAM_API_KEY=...
HEYGEN_API_KEY=...

# Memory System
MEMOS_REST_URL=http://localhost:8002
USE_MEMOS=true

# Optional Services
BRAVE_API_KEY=...  # For web search
ELEVENLABS_API_KEY=...  # Alternative TTS

Server Configuration:

HOST=0.0.0.0
PORT=8000
DEBUG=false
LOG_LEVEL=INFO

🧪 Development

Test Interface

• URL: http://localhost:8000/test_avatar_interface.html
• Purpose: Test audio pipeline, avatar integration, memory
• Features: Create/start HeyGen session, voice interaction, memory search

API Documentation

• Swagger UI: http://localhost:8000/docs
• OpenAPI: http://localhost:8000/openapi.json

Key Endpoints

• POST /api/v1/heygen/session/new - Create avatar session
• WS /ws/audio - Audio streaming for STT/TTS
• GET /api/v1/memory/status - Memory system status
• POST /api/v1/memory/search - Search conversations

🏃‍♂️ Running Services

Start meMOS Memory System:

cd MemOS/docker
docker compose up -d

Start Orchestrator:

python main.py

Verify Setup:

# All services healthy
curl http://localhost:8000/api/v1/health
curl http://localhost:8002/docs  # meMOS API

🚦 Status

Current State: ✅ Production Ready

• Real-time audio processing working
• meMOS memory integration complete
• HeyGen avatar integration functional
• Multi-language support active
• Tool calling system operational

Next Phase: Migration to Pipecat + SileroVAD for enhanced audio processing

📁 Project Structure

echo-mind-orchestrator/
├── main.py                 # FastAPI orchestrator service
├── streaming_pipeline.py   # Current audio processing pipeline
├── pipecat_pipeline.py     # Pipecat-based pipeline (future)
├── memory_service.py       # meMOS integration
├── heygen_service.py       # Avatar service integration
├── config.py              # Configuration management
├── requirements.txt       # Python dependencies
├── MemOS/                 # Memory system (extract to separate repo)
└── test_avatar_interface.html  # Development test interface

🤝 Contributing

This is the orchestration layer for the Echo Mind AI avatar system. Frontend and other services are separate repositories.

📄 License

Private project - All rights reserved.