04 - Operations Cheat Sheet

This guide consolidates daily operations for the local AI infrastructure into a single reference. It covers startup commands, server management, API key provisioning, maintenance schedules, and troubleshooting steps so you can keep the system running smoothly without searching through multiple documents.

Most useful single command:

/Volumes/AI_DATA/Status-AI.command watch

Leave it open in a Terminal tab — refreshes every 5 seconds.

1. Run Status-AI.command — find which service is down.
2. Agent-server down → Restart-Agent-Server.command.
3. Ollama down → stop-ai-stack.command, then start-ai-stack.command.
4. Both look up but requests fail → Status-AI.command logs, read the error.
5. Nothing works at all → check whether the AI_DATA volume is even mounted.

A key from one will not authenticate against the other. If an app suddenly reports "invalid API key," check whether :8443 got dropped from its URL.

# Non-streaming
curl https://llmpowerhouses.taila4b91f.ts.net:8443/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6:35b-a3b","messages":[{"role":"user","content":"hello"}]}'

# Streaming (-N disables curl's buffering)
curl -N https://llmpowerhouses.taila4b91f.ts.net:8443/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6:35b-a3b","stream":true,"messages":[{"role":"user","content":"count to five"}]}'

# List models, with live residency status
curl https://llmpowerhouses.taila4b91f.ts.net:8443/v1/models -H "Authorization: Bearer YOUR_KEY"

Every endpoint requires the Bearer header, /health included — a bare request returning {"detail":"missing bearer token"} is correct behavior, not a fault.

cd /Volumes/AI_DATA/apps/agent-server
venv/bin/python3 keys_admin.py list
venv/bin/python3 keys_admin.py create <app-name> --scope chat,tools --rate 60
venv/bin/python3 keys_admin.py rotate <key_id>
venv/bin/python3 keys_admin.py revoke <key_id>

--rate is requests/minute. --daily-quota is tokens/day (omit for unlimited). Raw keys print exactly once — if lost, rotate; they can never be recovered.

ollama pull <model-name>
ollama rm <model-name>
ollama list

Then update three places or an app will offer a model that doesn't exist: the app's own model table, the SOP's model table, and AGENT_SERVER_GUIDE_v2.md's daily log. Full detail in 07_Switching_LLM_Models_Guide.md. No agent-server restart needed — /v1/models reads Ollama live.

Restart = Restart-Agent-Server.command.

1. Drop files into knowledge-bank/DOD-FM-Knowledge-Bank/ or knowledge-bank/K12-Knowledge-Bank/ (or _inbox/ if unsure where they belong yet).
2. Double-click Sync-Knowledge-Base.command.
3. No restart needed — the index is read live. Files moved to _archive/ are automatically retired from search.

Naming matters for retrieval: FMR-Vol03-Ch08_Unliquidated-Obligations_2025.pdf retrieves far better than document (3).pdf. Pattern: Source_Topic_Date.ext.

Weekly (~5 min): Status-AI.command, Backup-AI.command, glance at free disk space (investigate below 200 GB free), check logs/agents/requests.jsonl size (archive, don't delete, past ~500 MB).

Monthly (~20 min):

cd /Volumes/AI_DATA/apps/agent-server
venv/bin/python3 run_test_questions.py     # quality baseline
venv/bin/python3 build_eval_dashboard.py   # regenerate Eval-Dashboard.html
venv/bin/python3 keys_admin.py list        # revoke anything unused

Quarterly: rotate API keys, check ollama --version for updates (read release notes first — a serving-layer change affects every app at once), copy a backup to a different drive (backups currently live on the volume they protect).

• Don't delete logs/agents/*.jsonl — it's the only latency/quality history that exists.
• Don't delete keys.db or memory.db — both are irreplaceable (keys are hashed and unrecoverable; memory rebuilds only from weeks of real use).
• Don't run restart-ai-stack.command expecting it to restart agent-server — it manages Ollama, Open WebUI, and the legacy gateway only.
• Don't edit files under apps/agent-server/ without backing up first.

⚠️ rm -rf foldername deletes instantly, no confirmation, no Trash. Triple-check the path before running it, and never run it with sudo on a path you didn't type carefully yourself.

cd /Volumes/AI_DATA/apps/llmpowerhouse-site
git pull

That's the only git command that should ever run on this Mac for that folder — all commits happen elsewhere and get pushed to GitHub; the Mac is a read-only mirror refreshed with git pull. Never git push, git add, or git commit here.

If you see "branch and origin/main have diverged":

git fetch origin
git reset --hard origin/main

This is always safe — nothing on the Mac side should ever be a local commit, and gitignored files (gateway.env, api_keys.json, gateway.log) are untouched by it either way.

• Start with Status-AI.command for any anomaly; it gives instant health, resident model status, and latency metrics.
• Never delete logs, keys, or memory databases; they are irreplaceable and contain hashed state that cannot be rebuilt from scratch.
• Agent-server restarts are lightweight; they only require Restart-Agent-Server.command and do not touch Ollama or the gateway.
• Naming conventions drive retrieval accuracy; use the Source_Topic_Date.ext pattern when dropshipping documents into the knowledge banks.
• Separate the two servers mentally; port 8443 serves the current infrastructure, while 8787/443 hosts legacy passthrough apps with distinct key spaces.