Neumann — Local AI Assistant

Overview

Neumann is a local-first AI personal assistant that runs entirely on your machine, providing voice interaction, semantic memory through RAG, and a modular skills system for extensibility. The system supports multiple LLM providers including local models via Ollama and cloud APIs like Claude and Groq.

Problem

Building a local AI assistant that maintains context, supports multiple LLMs, and operates entirely offline while providing voice interaction.

Solution

Created a modular architecture with local LLM inference, semantic memory using ChromaDB RAG, voice pipelines using faster-whisper and Kokoro TTS, and automation via n8n.

Architecture

Local-first architecture with GPU acceleration, vector database for semantic memory, multi-LLM orchestration, and modular skills system.

VOICE INPUT
     ↓
faster-whisper (STT)
     ↓
AI ORCHESTRATOR
     ↓
┌───────────────┐
│ Ollama        │ Local LLMs
│ Claude        │ Cloud API
│ Groq          │ Fast Inference
└───────────────┘
     ↓
MEMORY / RAG
     ↓
ChromaDB (Vector Storage)
     ↓
SKILLS SYSTEM
     ↓
Automation & Tools
     ↓
Kokoro TTS
     ↓
VOICE OUTPUT

AI Pipeline

The AI pipeline handles voice input through faster-whisper for speech-to-text, routes queries through the orchestrator to appropriate LLM providers, maintains context through ChromaDB vector storage, and generates responses with Kokoro TTS for natural voice output.

Memory / RAG

ChromaDB provides semantic memory through Retrieval-Augmented Generation. Conversations and knowledge are stored as vector embeddings, allowing the system to retrieve relevant context and maintain long-term memory across sessions.

Model Orchestration

The system intelligently routes queries to different LLM providers based on task requirements, performance needs, and resource availability. Local models via Ollama provide privacy and offline capability, while cloud APIs offer enhanced capabilities when needed.

Voice Pipeline

Voice interaction is powered by faster-whisper for high-quality speech-to-text and Kokoro TTS for natural text-to-speech. The pipeline is optimized for low latency to enable real-time conversation while maintaining accuracy.

Engineering Decisions

▹Local-first architecture for privacy and offline capability
▹Multi-LLM support for flexibility and redundancy
▹Vector database for semantic memory and context retention
▹Modular skills system for extensibility
▹GPU optimization for real-time performance
▹Voice UI for natural interaction

Challenges

▹Achieving real-time voice interaction with local models
▹Managing memory context windows effectively
▹Orchestrating multiple LLM providers seamlessly
▹Optimizing GPU resource usage

Results

▹Fully local AI assistant with voice capabilities
▹Semantic memory with RAG implementation
▹Multi-LLM support (Ollama, Claude, Groq)
▹Modular skills system for extensibility

Lessons Learned

▹Local AI requires careful resource management
▹RAG dramatically improves context retention
▹Voice UI requires low-latency processing
▹Modular architecture enables rapid iteration

Source Code

View on GitHub