07 - How to Change or Add LLM Models

Swapping or adding LLM models requires zero agent-server code changes. All model management happens at the Ollama layer, governed by strict VRAM constraints and a three-place documentation update rule. Changing chat models is a pull-and-verify operation, while changing embeddings triggers a full knowledge-base re-index.

The apps/agent-server codebase was deliberately built model-agnostic. When a request arrives, the specified model name passes straight through to Ollama without routing logic, hardcoded defaults, or branching. The outbound request builder (_prepare_upstream_body() in agent_loop.py) handles the model field uniformly. Consequently, adding a new model reduces to pulling it into Ollama and requesting it by name—no server modifications required.

The one architectural exception is the embedding model (nomic-embed-text), which anchors the vector geometry for the entire knowledge base; changing it follows a separate, more intensive process outlined later.

Before pulling any new model, verify hardware compatibility. This Mac provides 77.8 GiB of usable VRAM, with Ollama configured to cap resident models at two (OLLAMA_MAX_LOADED_MODELS=2). Use this conversion for sizing:

Model size in GB ÷ 1.074 = size in GiB (the unit that matters for this math)

Current model pair compatibility:

The co-residency rule: Anything exceeding roughly 55 GiB cannot run alongside the default workhorse model. Selecting a larger model will evict the currently loaded one and trigger a multi-minute cold-load penalty on the next request. This is a hard physical constraint, not a software bug. Deliberately plan which model gets to be large rather than discovering the trade-off by accident.

Adding:

ollama pull <model-name>
ollama list          # Confirm it's there and check its size

A 30–60B model typically downloads in several minutes to over an hour (the original gpt-oss:120b pull took 30–90 minutes). The server does not need a restart; GET /v1/models queries Ollama’s live state (/api/tags for the catalog, /api/ps for resident memory) on demand.

Removing:

ollama rm <model-name>
ollama list          # Confirm it's gone

The critical update rule (applies to both adding and removing): You must update three specific locations, or the new model will silently fail or cause reference errors:

1. Any app's static model list (e.g., the showcase site's picker reads a hardcoded table, not live Ollama state).
2. SOP-LLM-OPERATIONS.md's model table (Part 1) — the source of truth for tracking what runs and why.
3. AGENT_SERVER_GUIDE_v2.md's daily log — records the reasoning behind changes so context survives future retuning.

⚠️ Warning: If a script, app default, or cron job still references a removed model name, Ollama will throw an error rather than gracefully fall back. Audit for hidden callers first.

The trial workflow: Always test in LM Studio first before promoting to production. LM Studio ships with Apple’s MLX runtime, delivering a reported 10–20% speed advantage on Apple Silicon. This allows rapid side-by-side quality/speed benchmarking without touching the live Ollama environment. Once validated, ollama pull it for real so it routes through the existing OpenAI-compatible endpoint.

Default models & routing: Agent-server currently enforces zero default-model logic or server-side routing. Every caller must explicitly specify a model in its request body. Server-side automation ("always route vision-capable requests to the VL model" or "direct short queries to a faster, smaller model") does not yet exist. The closest precedent is ai_chain.py's client-side router that selects a model before calling the API. Implementing server-side routing is a well-scoped extension (see 08_Extending_The_Harness_Agent.md).

Three start scripts control Ollama's serving behavior. Their names differ by only one word, making it easy to edit the wrong file—all three must stay in sync:

• start-ai-stack.command (primary routine)
• start-ai-server.command
• start-ai-server.sh

Current environment defaults:

export OLLAMA_MODELS=/Volumes/AI_DATA/models/ollama
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_KEEP_ALIVE=30m
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_CONTEXT_LENGTH=65536

To modify these: edit all three scripts identically, run stop-ai-stack.command followed by start-ai-stack.command, then verify live variables with:

ps eww $(pgrep -x ollama) | tr ' ' '
' | grep OLLAMA

⚠️ Downtime warning: stop-ai-stack.command also terminates the legacy gateway, briefly taking ~20 older integrations offline. Schedule serving changes during low-traffic windows.

Future watchpoints:

• Monitor qwen3.6 for an official (non-preview) Ollama tag.
• Watch for native video input support in Ollama to replace the current ffmpeg frame-extraction workaround.
• Re-evaluate the model lineup every 3–6 months. Better open models ship constantly; today's "workhorse" is a snapshot-in-time baseline, not a permanent fixture.

nomic-embed-text generates the vectors that align both knowledge-base documents and incoming search queries. Swapping it invalidates geometric alignment immediately. The required process:

1. The index breaks: Old embeddings use the old geometry; new queries would use the new model's. They become mathematically incomparable.
2. Force a full re-index: Run kb_sync.py or Sync-Knowledge-Base.command with a forced flag to reprocess every knowledge bank. This is a slow, bulk operation, not an incremental update.
3. Update the environment variable: Change AGENT_EMBED_MODEL in tools/knowledge_base.py (defaults to nomic-embed-text) so new queries match the re-indexed model.

Do not change the embedding model casually—it triggers a project-wide re-index, treating it as a major architectural operation rather than a config tweak.

• Zero-code swaps: Model changes require no agent-server edits; they rely entirely on Ollama’s live state and three critical documentation updates.
• Hard VRAM limits: The 77.8 GiB Mac limits co-resident pairs to ~55 GiB maximum; oversized models will evict each other and trigger multi-minute cold loads.
• Test first, deploy second: Always benchmark in LM Studio (MLX runtime) before promoting winners to Ollama for production routing.
• Embedding changes break indexes: Swapping nomic-embed-text forces a full knowledge-base re-index; never treat it as a simple config adjustment.
• Sync all startup scripts: Update the three start scripts identically when tuning serving parameters, and schedule swaps during low-traffic windows to avoid legacy gateway downtime.