llamaMan Dev Update - What's Old & What's New

Hi there! Today I want to pay a little attention to my not-so-little-anymore side project called llamaMan, which surprisingly has started gaining some traction. It took me a little while to collect and verify everything that has been done since the last article but here we go.

What is llamaMan?

llamaMan is a self-hosted web UI for managing llama.cpp server instances. Point it at a directory of GGUF files, launch models with full control over GPU offload and context size, and get an Ollama-compatible API on port 42069 (nice). Think Ollama but with fewer mystery decisions.

First up - the BIG paradigm change

Up until 0.9.6 release llamaMan ran on top of a baked in GGML llama.cpp docker image. Due to the rapid development of the llama.cpp engine and the daily advancements in LLMs with new architectures, vendor drivers and all that, I decided that maintaining the PHAT app that way was going to be a hurdle even with continuous automated builds, as I'd be constantly chasing THE LATEST thing, which for a side project is way too much.

So I took the opportunity to finally do something I've always wanted to do - lean on Docker to manage Docker. I'd previously referred to this as Docker-in-Docker, and purists will rightly point out that what I built is technically Docker-out-of-Docker (DooD): llamaMan doesn't nest its own daemon, it talks to the host's Docker socket and spawns the llama.cpp images as siblings. So my bad on that.

Anyway, the idea is simple, use llamaMan with settings relative to your hardware for resource monitoring and just pull or build whatever llama.cpp image you need based on your hardware and llamaMan will manage that.

The 0.9.6 release tore out the bundled llama.cpp and rebuilt llamaMan as a pure manager that spawns the official llama.cpp images as sibling containers, which llamaMan would be able to reattach to even if it were to crash. Everything since - across 1.0.0, 1.1.x, and 1.2.x - has been building on top of the manager: observability into what your models are actually doing, fidelity with the Ollama API surface, and ultimately running more than one box as a single cluster - and this is my favorite feature.

Multi-Node Clustering (1.2.1)

While using agents such as Hermes, I kept noticing that they tend to do a lot of async requests to the llamaMan proxy. Sometimes I would see 6 active requests with like 12 more queued up.

As you might imagine, as you stack more and more parallel requests the system eventually starts choking - LLM performance degrades, memory pressure climbs, and once you run out of compute the server is stuck context-switching between all those in-flight requests to keep as many moving at once as it can. That's a serious performance hit, especially if you are hardware limited.

So, since we're already managing sibling llama.cpp containers for each of the LLM deployments, it just hit me - why not use the database backend to make a simple clustering functionality with a proxy entrypoint that balances the queue between nodes? That way we can distribute the load more evenly and increase throughput while relieving the pressure across all nodes. I have a bunch of PCs laying around, so I thought they'd be perfect for that.

So that's what I did.

Several llamaMan deployments can now run as one logical cluster - an aggregated dashboard, cross-node launches and downloads, and a shared inference queue that load-balances requests across nodes.

It is entirely opt-in. Single-node installs are completely unaffected: clustering stays inert unless you set CLUSTER_ENABLED and a shared CLUSTER_SECRET.

Discovery is registry-based, not pairwise. Nodes find each other through a shared registry in the database storage backend rather than each node being configured to point at every other one. Add a node anywhere and it becomes visible to all of them. Every node-to-node call carries the shared secret in a dedicated header - never as a client bearer token - and each node advertises how peers should reach it via CLUSTER_ADVERTISE_URL.

The dashboard becomes cluster-aware. Per-node System and GPU monitoring cards, plus a new Cluster settings tab showing each node's identity, advertise URL, heartbeat age, and an HTTP-reachability badge. Heartbeats are stamped on the database's clock rather than each node's local clock, so wall-clock skew between machines can't flap a healthy node offline.

You can drive any node from any node. A control relay lets the UI run launches, image pulls, and downloads on a remote node, and the launch / images / downloads forms all gained a node selector.

Shared queue and cross-node least-load dispatch

This is the part that makes the cluster more than a shared dashboard. With Share queue with same model enabled, instances of a model across all nodes form a group, and an incoming inference request is routed to the group member with the fewest in-flight requests. A request already sitting in a queue can migrate to a freer peer, with the number of hops bounded by MAX_HOPS and guarded against loops.

Two refinements on top of that:

• Queue group name - pools same-family / different-quant instances under one alias. That alias is also what the instance advertises to clients, so the group looks like a single model.
• Fallback only - marks an instance that should only serve when every non-fallback member of its group is at capacity or unreachable. Useful for a slow-but-always-there CPU instance behind your fast GPU ones.

A subtle but important reliability fix shipped alongside this: the cluster heartbeat runs on its own dedicated thread (configurable via CLUSTER_HEARTBEAT_INTERVAL_S). Forwarded inference lands on the shared web worker, and if the heartbeat shared that worker, a busy node could miss its own heartbeats and flap itself offline mid-load. Splitting it out fixes that class of false-offline.

Per-node settings

Some settings genuinely have to differ per host - the node's local Docker images, the model-cap eviction toggles. Those are now scoped per node instead of being shared cluster-wide. Reads transparently fall back to the legacy shared value, so existing single-node installs upgrade with zero migration.

Upgrade note: as of 1.2.1, LLAMAMAN_NODE_NAME is required for every install, not just clusters. It is the node's stable identity, its per-node settings namespace, and its key in the cluster registry - so a single-node deployment can later join a cluster without orphaning its state. The app now refuses to start (with guidance) if it's unset. Pick it once and keep it; changing it later orphans that node's stored instances and presets.

Observability: Request Logging, Stats, and a Logging Page

The single biggest theme across the 1.x line - it arrived in three waves.

Wave 1 - Request recording (1.0.0)

Optional recording of inference traffic across the Ollama API, the OpenAI-compatible API, and the per-instance proxy. Three modes: off, per-request, and per-conversation, with content-hash conversation grouping and configurable retention. Backed by both storage backends (JSON files and MariaDB). The RECORDINGS_DIR env var lets you relocate the per-conversation records on the JSON backend.

Wave 2 - Per-instance stats + accurate throughput (1.1.8)

Each running instance card gained a Stats button opening a modal that rolls up that instance's recorded traffic: request count (with errors), average and peak throughput (tokens/s), average time-to-first-token, average latency, prompt/completion/total tokens, and the active time span. The numbers persist even after the instance is stopped. When recording is off, the modal shows an empty state prompting you to turn it on.

Wave 3 - The Logging page (1.2.1)

A dedicated Logging page (linked from the dashboard header) that turns the recorded traffic into something you'd actually sit and read: summary tiles (request count with errors, average/peak throughput, average TTFT and latency, token totals), a time-window selector (24h / 7d / 30d / all), a recent-conversations list, and a per-conversation drill-down showing each turn's prompt, response, and metrics.

Ollama API Fidelity (1.1.2, 1.2.2)

A focused release that made llamaMan's responses look much more like real Ollama to clients that introspect the API - the difference between a client guessing your model's capabilities and actually being told them.

The enabling piece is a new full GGUF metadata reader. It pulls in the whole model header but skips the huge tokenizer arrays, so the cheap trailing values - the special token IDs, the chat template - stay reachable without paying for the bulk. Results are cached per file (keyed on path, modification time and size), and the existing layer-fit reader was moved onto the same cache, so the various endpoints that introspect a model now share a single parse per file instead of re-reading it each time.

What that unlocked:

• The running-models list (/api/ps) now reports the effective context each model was launched with, matching real Ollama's shape. A client like Hermes sees a preset-imposed 64K cap instead of inferring the model's trained 256K maximum from the model file.
• Model family and architecture now come from the model file's own metadata instead of being guessed by chopping up the filename.
• Parameter size (e.g. 8B) is populated from that metadata instead of always being empty.
• The reported idle-expiry time now reflects the actual idle-reaper deadline instead of a hardcoded five minutes after launch.
• An estimated VRAM footprint is reported instead of always zero. It's approximate - weights only, no KV cache or scratch buffers - but it's in the spirit of the field.
• /api/show now returns the real model info and chat template instead of a few stub values, with the context figure overridden by the effective cap (running config > saved preset > model default).
• The model list (/api/tags) gets the same metadata-sourced enrichment.

GPU temperature, end to end (1.1.2)

GPU readings now include temperature, wired through every detection path: the NVIDIA management library, a new sysfs reader for AMD/Intel cards (preferring the edge sensor), and nvidia-smi / rocm-smi parsing on the command-line fallbacks. The system info card now stacks each GPU over its live temperature, color-thresholded ≥85°C red / ≥75°C yellow.

context_length on the OpenAI models list (1.2.2)

1.1.2 taught the Ollama surface to report effective context; 1.2.2 does the same for the OpenAI-compatible /v1/models list. Each entry now carries two non-standard fields exposing the model's context window: context_length (OpenRouter's name) and max_model_len (vLLM's name), both set to the same value so different client ecosystems each find the key they look for. Strict OpenAI clients just ignore the extras.

The value follows the same resolution order as /api/show: the running instance's launched context size > a saved preset's context size > the model file's trained maximum. When none of those is knowable - no instance, no preset, unreadable model file - both fields are omitted entirely rather than zeroed. It reuses the same cached metadata the model-listing endpoints already read, so the models list adds no extra file reads.

(OpenWebUI already pulls context from the Ollama /api/show path, so this one is aimed at the vLLM / OpenRouter-style consumers.)

Downloads: Progress Bars and Shard Expansion

Per-file progress, properly (1.1.5)

The per-download "Logs" button became "Progress" and now opens a progress-first modal: an overall bar plus one bar per file. For multipart GGUFs - which llamaMan expands and downloads in full - every shard gets its own bar with bytes / total / percent / live speed. Bars go green when a file's done, freeze grey on pause / failure / cancel, and show an indeterminate stripe while a file's size isn't known yet. A "Raw log" toggle reveals the original text log underneath.

Nested-file and multipart resolution (1.0.0)

The download flow now fetches the repo file list and expands a single shard pick to all shards, and resolves UI-supplied basenames to their nested repo paths before spawning the downloader. So picking one part of an 8-part GGUF, or a file that lives in a subfolder of the repo, just works.

Bare-Metal Deployment (1.1.8)

llamaMan no longer has to run as a container. It can run directly on the host - e.g. as a plain Python process under WSL - and still spawn llama-server containers correctly.

The mode is auto-detected (it checks the usual container markers, then falls back to cgroup inspection), and how spawned instances get reached is now centralized: by container name on the Docker network when containerized, or via localhost on the published ports when bare-metal. It's applied consistently at launch, relaunch, orphan adoption, and state restore. Two new env vars: LLAMAMAN_IN_DOCKER (force the mode) and LLAMA_HOST_ADDR (host address for the published ports, default localhost).

Speculative Decoding (MTP) Toggle (1.1.8)

A new Speculative Decoding section in the launch form runs a model with MTP speculative decoding, with an optional Draft N Max field. It's aimed at models that ship MTP heads (e.g. Qwen3.6); other speculative-decoding flavors can still go through Extra Args. Both settings save in presets and are applied consistently across launch, restart, and proxy auto-start.

In 1.2.1 the blank-field default for Draft N Max was lowered from 3 to 2.

Upgrade Notes

• LLAMAMAN_NODE_NAME is now required (1.2.1). Set it before upgrading; the app won't start without it. Choose a stable value per host and don't change it later.
• New env vars by area: clustering - CLUSTER_ENABLED, CLUSTER_SECRET, CLUSTER_ADVERTISE_URL, CLUSTER_HEARTBEAT_INTERVAL_S, MAX_HOPS; deployment - LLAMAMAN_IN_DOCKER, LLAMA_HOST_ADDR; logging/downloads - RECORDINGS_DIR, HF_PROGRESS_FILE. All optional except LLAMAMAN_NODE_NAME.
• Schema migrations run automatically at startup (timestamp normalization, plus added throughput columns for the request log). They're idempotent and lock-guarded; a failure aborts startup rather than half-migrating. Back up your storage before a major jump if you're on MariaDB, as usual.
• Clustering is fully opt-in - if you don't set CLUSTER_ENABLED, nothing about your single-node setup changes.

Where We Are And Where We're Headed

This project kind of became... well, a lot. I never imagined that this would get so many features, but wanting more convenience out of fast and easy deployments got me here.

I also had no idea that people would like and start supporting the project, which I'm very pleasantly surprised by and extremely thankful for.

GGML recently introduced their own feature with dynamic model loading via ini configs, but I decided against integrating that with llamaMan as it would double up on functionality we already have and would increase complexity significantly.

A few features coming up in the long term are vLLM support + clustering between llama.cpp and vLLM instances, which is a terrible headache to deal with. Nobody asked for this, but I currently have a broken build in my Gitea that will see the light of day soon(tm).

Aside from moving away from the deprecated pynvml package to nvidia-ml-py and resolving other minor issues, the addition of new features will remain mostly paused, unless there is something extremely critical and revolutionary that comes up.

Other than that, the primary focus for this project will be gradually moved to code quality, UI enhancements and some QoL features, especially because I have other important things to take care of (some of which will be announced soon 🫡).

I also whipped up a logo idea for the project via QwenVL as the old mascot was kind of... well - gay. 😆 Not that there's anything wrong with that but I'm over the whole "don't let your memes be dreams" thing and especially since the project has some more serious features, I think it deserves a bit more than a half naked looksmaxxed-out man with furry llama ears.

GitHub issues and contributions are welcome.

Thanks for your support.

• GitHub: github.com/nullata/llamaman
• Docker Hub: hub.docker.com/r/nullata/llamaman

Live long and prosper. 🖖👽