Personal LLM 2.0
// Offline Private Local Model Runtime
Personal LLM 2.0 is an offline-first assistant built for developers and researchers who require total data privacy without sending prompt context to third-party APIs. Eliminates heavy container dependencies (Docker/Ollama) by running a lightweight Python/FastAPI daemon interfacing directly with optimized llama.cpp C++ binaries. Includes local Retrieval-Augmented Generation (RAG) using ChromaDB and sentence-transformers on local disk.
100% Offline
28 Models
Zero Telemetry
[ ARCHITECTURAL INSPECTOR ]
System Design & Code
Architectural Overview & Pipeline
Personal LLM 2.0 is an air-gapped local AI workspace deploying 28 GGUF quantized models without external API calls. Built on an async FastAPI daemon, it manages bidirectional IPC across Electron desktop surfaces and Expo mobile nodes over local mDNS. Context injection relies on a persistent ChromaDB vector store running local embedding quantization with an aggressive LRU CUDA/Metal memory offloader.
Primary Stack:
Engineering Challenges & Solutions
Built an LRU memory offloader with torch.cuda.mem_get_info() probes that dynamically steps down context window sizing from 16k to 8k when host physical memory pressure exceeds 82%.
Swapped recurring UDP broadcast polling with persistent WebSocket peer links backed by automatic Exponential Backoff reconnect workers.