NickScripts NickScripts

llamaMan Dev Update - From 1.2.2 to 2.2.2

ai llm proxy llamaman clustering mcp strata

Hi again! It's been a while since my last llamaMan article, and honestly the project outpaced my ability to write about it. The last update stopped at 1.2.2; we're now at 2.2.2, and the gap in between is... a lot. So instead of a release-by-release dump, this time it is a release-by-release tour, but only the releases with something substantial in them, newest first.

2.2.1 - Decision models (System One)

A weird and wonderful new one: llama.cpp now serves decision models - models that answer typed questions about an input with probabilities instead of generating text. Think "is this email a refund request? which of these five categories?" answered by a small model in a fraction of a second.

llamaMan serves them on POST /v1/systemone, routed by model name like any other endpoint (auto-start, eviction, cluster forwarding, queue - all of it). Chat endpoints refuse them with a 422 because they literally can't generate text. Picking one in the launch form auto-fills sensible settings (parallel slots, batch sizes, exclude-from-max), and llama-server recognizes them straight from the GGUF metadata - no flags needed.

Alongside it came the Exclude from Max Models toggle: small always-on models (embeddings, decision models) no longer count against LLAMAMAN_MAX_MODELS and never get evicted.

2.2.0 - A second inference engine: Strata, and the model archive

Strata

This is the one I'm most excited about. llamaMan always managed llama.cpp instances - but the engine layer I'd been quietly building is engine-agnostic: it spawns containers, monitors them, proxies to them, sleeps them. Nothing about that machinery actually says "llama.cpp".

So llamaMan now also runs Strata as a second inference engine. Strata is a newer engine that runs huge models (think 100B+ parameters) on a single gaming PC by aggressively offloading to system RAM. If you've ever wanted a 125B model on your desktop and you have the RAM for it, this is now a checkbox away.

It's experimental and opt-in (STRATA_ENABLED=true, NVIDIA only for now), and off by default - the llama.cpp path is snapshot-tested byte-for-byte, so nothing changes if you don't turn it on. When you do:

  • The launch form gains an Inference Engine dropdown, and the model library gains a catalogue of virtual Strata models (family × quant size, e.g. strata/qwen-IQ2_XS). Launching, sleeping, waking, eviction - all work exactly like llama.cpp instances.
  • Strata is env-var-only when it comes to launching, so llamaMan parses Strata's own setup to derive per-model flags, pre-downloads model shards through the existing downloader, and watches container logs to track load stages.
  • You can download the Strata repo and docker build the image straight from the UI, built for this node's GPU generations.

The engine abstraction itself (core/engines/) is the part I'm proudest of - additive config keys, no storage migration, missing key = llama.cpp. More engines may come; the groundwork is done.

Model archive

Models you're not using but don't want to re-download can now be archived to a second volume. Archived models stay listed (dimmed, not launchable, hidden from the model-listing APIs), and restoring puts them back at their original path - so presets, favorites and notes survive the round trip. Moves run as cancellable jobs with atomic cross-disk handling; a cancelled or failed move leaves the source untouched.

2.1.0 - Knowledge base + MCP server

llamaMan can now be your agent's memory. Opt-in (MariaDB 11.8+ only), it stores documents grouped by topic, chunks them, embeds them through a llama.cpp instance running in embedding mode, and keeps the vectors in a MariaDB VECTOR column with a cosine-distance index.

One code path, three ways in:

  • A Settings → MCP tab to pick the embedding model, manage topics, test searches, and copy a ready-to-paste mcpServers config block.
  • REST under /api/kb/* for management.
  • An MCP endpoint at POST /mcp/knowledge that any MCP client (Claude Desktop, your coding agents, etc.) can connect to, with search tools always on and ingest/delete tools each behind their own toggle.

The embedding model is picked by name, not by instance - if no live embedding instance exists anywhere in the cluster, llamaMan wakes one, forwards to a peer that has it, or relaunches it on the node that last served it. Your agents get semantic search over your documents without you running a separate RAG stack.

1.8.5 - Anti-Loop

Even with repeat penalties tuned, real models still get stuck - repeating a sentence or a <think> fragment for hundreds of tokens. llamaMan now ships a two-tier defense, per preset, off by default:

  • DRY sampling exposed as first-class launch fields (soft, sampling-time).
  • Output loop detection in the proxy: if a large chunk of output repeats often enough, llamaMan closes the upstream connection and injects a clean terminator into the client stream, so OpenAI clients see a normal finish_reason=stop and Ollama clients see done_reason: "loop_detected".

Detection thresholds are preset-live - tune them without relaunching.

1.8.0 - Image & PDF input

llama-server has no PDF support at all - send it a PDF and you get a decode error. llamaMan now rewrites inbound PDFs before llama-server ever sees the request: either rasterized page-by-page into image blocks (for any vision model with an --mmproj projector), or - when the PDF is born-digital - fast-pathed as its extracted text layer, which completes in milliseconds and spends no context on pixels.

It works on both API surfaces: OpenAI image_url/file blocks and Ollama-native images[]. Per-instance caps on DPI and page count mean a 500-page whitepaper can't monopolize your raster pool. This one quietly made a lot of agent workflows "just work".

1.7.0 - The launch form's power-user release

  • Speculative decoding went to all five draft types llama.cpp accepts (draft-simple, draft-mtp, draft-dflash, draft-dspark, draft-eagle3), plus advanced draft knobs.
  • Flash Attention and KV cache quantization (q8_0, q4_0, ...) as first-class fields - long contexts get a lot cheaper.
  • Multi-GPU placement: split mode and per-GPU tensor split weights, with a live preview auto-filled from each GPU's VRAM.

1.6.1 - Surviving a database outage

If you use MariaDB, the database sits on the critical path of every request - auth reads settings and keys per call. So a DB outage doesn't degrade llamaMan, it stops it.

There's now an optional local mirror: each node keeps a write-through copy of the database in DATA_DIR/db_mirror/, and when the DB goes away a circuit breaker flips the node to mirror mode - inference, launches, downloads, presets, everything keeps working from the local copy. Offline changes to shared rows are journalled as edits and replayed in order on reconnect, so a peer's changes made during the outage survive. Off by default; with it off, behavior is byte-identical to before.

1.5.5 - Keeping models fresh

Authors routinely republish repos under the same filenames - a requant, a fixed chat template - so a model you pulled months ago can be silently stale. llamaMan can now check HuggingFace's content hash against your local file in one HTTP request, and an "update available" re-pulls into a staging folder and atomically swaps files in only once the download completes. Multi-shard models work automatically; it refuses while an instance has the model loaded. There's an opt-in background scan and a local hash-verify for models with no recorded hash.

Also in this release: per-model Display Names, so Open WebUI shows "Qwen 2.5 14B" instead of Qwen2.5-14B-Instruct-Q4_K_M - the friendly name is advertised on the model-listing APIs and accepted as an inbound model name.

1.2.9 -> 1.3.4 - Speculative decoding grows up

It started as an MTP toggle. Then came a Draft Type selector: draft-dflash with a required drafter GGUF (1.2.9), then MTP with an optional separate drafter for the standalone MTP-head GGUFs published alongside main models (1.3.4). Each step flows through presets, restarts, auto-launch, and cluster overrides like everything else.

1.2.6 - A phone layout that makes sense (sort-of)

The dashboard is a fixed two-pane desktop layout, and below ~820px it had no room to breathe. There's now a proper responsive layout: single column, modals near-full-screen, fields stacked. And in portrait it reorders to Running Instances → System → GPU → Launch → Model Library - because from your phone you're checking what's running, not configuring what to launch.

1.2.5 & 1.3.2 - Vision, end to end

Two halves of the same story. The Ollama-compatible API (/api/chat, /api/generate) learned to accept and forward images - Ollama's native base64 images[] array used to be silently dropped before reaching llama-server, so vision was a no-op for any Ollama client, Open WebUI included. Now the translator lifts them into OpenAI image_url blocks with MIME sniffed from magic bytes.

And --mmproj (the multimodal projector GGUF a vision model needs) became a real launch-form field instead of an Extra Args hack. Together: pick a vision model, point at its projector, and images work from both API surfaces.

Where we are

The last article said new features would be "mostly paused". Yeah. About that. 😄 The truth is the manager foundation kept making bigger things possible - engines, knowledge bases, decision models - and I kept saying yes to ideas.

Everything above is opt-in and upgrade-safe; single-node installs that change nothing stay untouched. Full docs live in the repo's docs/ folder (now organized per-language, translations coming), and the changelog on GitHub covers the rest.

GitHub issues and contributions are welcome.

Thanks for your support.

Live long and prosper. 🖖👽

Share this article

Copied!