llamaMan

Manager for running multiple llama.cpp and Strata inference servers from one container: spawns sibling GPU containers via the Docker socket, Ollama/OpenAI-compatible API proxy with LRU model eviction, clustering, concurrency gating, multi-vendor GPU monitoring, and a built-in knowledge base with an MCP server.

Docker Infra JavaScript llama.cpp Python Strata

llamaMan is a browser-based UI for launching, monitoring, and managing multiple inference-server instances - llama.cpp and, opt-in, Strata - from inside a Docker container. Includes an Ollama-compatible API proxy so it works as a drop-in replacement for Ollama with Open WebUI, auto-starting models on demand and LRU-evicting them under a configurable cap. Compatible with any OpenAI API compatible client or agent harness.

Setup now via DockerHub

Features

Flexibility and Scaling

Launch and manage multiple llama.cpp server instances from a browser dashboard.
Cluster multiple llamaMan nodes across your network: shared dashboard, cross-node launch and download, and shared-queue load balancing across heterogeneous hardware.
Run models in isolated Docker containers with configurable GPU layers, context size, CPU threads, memory limits, parallel slots, GPU selection, and extra server flags.
Use LlamaMan as a drop-in Ollama-compatible backend for Open WebUI and other Ollama clients.
Serve OpenAI-compatible chat completions, model listings, and embeddings from local GGUF models.
Auto-start models on demand when requests arrive, using saved presets or sensible defaults.
Serve decision models - typed-question classifiers that answer with probabilities instead of text - on a dedicated /v1/systemone route with the same auto-start, routing, and eviction rules; chat endpoints refuse them cleanly.

Resource Management

Keep GPU memory under control with idle sleep, wake-on-request, and configurable model eviction limits (LRU).
Save per-model launch presets, notes, favorites, and proxy-side sampling overrides for convenience and 2-click deployments.
(new) Live preset updates: fields that don't need a relaunch (idle timeout, concurrency gates, sampling overrides, eviction exemption) apply to a running instance without restarting it.
(new) Concurrency gating per instance: max concurrent requests and a bounded queue (429 past it), enforced by a per-instance proxy that also handles wake-on-request and model-name validation.
(new) Exclude-from-cap flag per instance: embedding models, decision models, and anything small can stay up permanently outside the MAX_MODELS cap.

Monitoring

Monitor every running model with live status, logs, load time, tokens per second, time to first token, request counts, CPU/RAM usage, and GPU assignment.
View native GPU telemetry for NVIDIA, AMD, Intel Arc, and (opt-in) Vulkan, including VRAM, utilization, and temperature.
(new) Opt-in request recording - per-request or per-conversation, with retention - feeding a logging dashboard; the same request log rolls up the throughput/TTFT/latency stats.

Downloads

Download GGUF models directly from Hugging Face with progress tracking, pause/resume, retry, cancellation, saved tokens, and per-node speed limits.
(new) Republish detection: compare a local model against its source repo's content hash and atomically re-pull on update - a failed pull never touches the working file.
(new) Model archive: move rarely used models to a second volume and restore them later; moves run as cancellable jobs, are atomic across disks, and presets, favorites, and notes survive the round trip.
(new) Docker image management from the UI: pull newer llama.cpp server-* images (update the engine without touching llamaMan), pull any image by name, and build or delete Strata images.
Download models based on settings from other cluster nodes.

Security

Protect the dashboard and model APIs with first-run admin setup and session login.
API keys support, and optional bearer-token enforcement for all serving endpoints.
(new) Per-API-key scoping for the knowledge base (private topics per key).

Inference Engines

Run llama.cpp and Strata instances side by side: an engine picker per launch, with each instance's engine stored alongside it.
Strata (opt-in, NVIDIA) runs the Qwen3.8-Flash-Next MoE family on consumer NVIDIA GPUs plus system RAM - models that don't fit in VRAM alone. Launched, monitored, proxied, slept, and woken with the exact same machinery as llama.cpp.
Strata images built or pulled from the UI (built for the node's GPU generations); multi-gigabyte shard sets pre-downloaded through llamaMan's own downloader with progress, pause/resume, and retry.
Engine-agnostic by design: the launch path for llama.cpp is snapshot-tested byte-for-byte, so adding an engine never changes existing behavior.

Advanced Serving

Speculative decoding with all five llama.cpp draft families (simple, MTP, dflash, dspark, eagle3) and the advanced acceptance knobs.
MoE CPU offload (--cpu-moe / --n-cpu-moe): shrink big-MoE VRAM usage several-fold by pinning routed experts to system RAM.
Flash attention, KV-cache quantization, reasoning-format, and load-mode (mmap/mlock/dio) as first-class launch fields, with the UI enforcing flag constraints.
Image & PDF input: vision models via --mmproj, and PDFs rasterized page-by-page - working through both OpenAI image_url/file and Ollama images[].
Anti-Loop protection: a DRY sampler at sampling time plus proxy-side output-loop detection that hard-stops a turn when the streamed text starts repeating.
Embedding models flagged at launch, exempt from the concurrent-model cap and never evicted, so RAG clients always have them warm.

Knowledge Base & MCP

Opt-in knowledge base (MariaDB backend): store documents grouped by topic; llamaMan chunks them, embeds them with a local embedding model you already run, and keeps the vectors in your database with a cosine ANN index.
Semantic search exposed to any MCP client (Claude Desktop, Claude Code, agents) at /mcp/knowledge - with a ready-to-paste client config from the UI.
Access modes: a shared pool for everyone, or a private knowledge base per API key alongside a read-only shared pool.
Everything local: embeddings, vectors, and the MCP server all run on hardware you already run.

Storage & Resilience

Persistent state in plain JSON files by default - zero database dependency for single-node installs.
MariaDB/MySQL backend unlocks clustering, the knowledge base, and request recording.
Optional local write-through mirror keeps a node serving through a database outage.