Mac Studio 24/7 AI Workhorse Manual Revised

Platform: Mac Studio M3 Ultra, 96 GB unified memory, 2 TB AI_DATA volume
Goal: Operate a dependable private AI service that runs useful scheduled work with minimal attention.
Core principle: Keep the control plane available; load expensive models only when a task requires them.

This system should become more than a local chatbot. It should:

• Serve trusted users and applications.
• Run scheduled jobs and monitor selected sources.
• Maintain searchable knowledge collections.
• Extract, classify, summarize, and reconcile information.
• Route simple work to small models and difficult work to larger models.
• Record jobs, changes, failures, outputs, and required human decisions.

The machine becomes a workhorse when it produces validated outputs while unattended—not when it merely keeps models loaded.

Core services recover after reboot; health checks run continuously; approved data and documents synchronize and index; models are routed by task complexity; and a morning brief reports outputs, failures, insights, and decisions. Heavy work runs only after policy, capacity, and approval checks.

Application → Funnel → authenticated gateway → router → approved service

Do not expose Ollama, MLX, Open WebUI, databases, or the agent server directly to the public internet. Funnel supplies reachability; the gateway supplies identity, authorization, workload policy, and rate control.

• ai-stack: Ollama, Open WebUI, gateway, and Funnel.
• agent-server: FastAPI retrieval, tools, memory, and agent harness.
• AI Local Server: gateway, keys, routing, and heavy-model policy only.

Observe all three in one dashboard, but keep their restart controls independent.

Use scripts for exact work; 9B for classification and extraction; 35B for synthesis, RAG, and most coding; and heavy models only when a quality test or human request justifies them. Require sources for analytical claims and approval for external actions.

The 96 GB memory pool is shared by macOS, model weights, KV cache, applications, and file processing. Protect it at several levels.

• GPU wired-memory limit is approximately 88 GB, while preserving an OS reserve.
• Ollama limits simultaneously loaded models and unloads inactive models through keep-alive settings.
• Heavy requests require X-Confirm-Heavy: yes; otherwise the gateway returns HTTP 428 Precondition Required.
• Large coding, reasoning, and media models load only when needed.
• Heavy-model context should remain at or below 32K until measurements support more.

• Admission control: Check memory, swap, loaded models, queue, active sessions, disk, UPS state, and temperature before heavy work.
• Durable queue: Permit only one heavy-exclusive job at a time.
• Timeouts: Give every job a deadline, retry ceiling, and cancellation path.
• Concurrency classes: Separate limits for light, default, embedding, speech, image, video, and heavy work.
• Circuit breaker: Pause a repeatedly failing model or data source.
• Backpressure: Queue or return HTTP 429 rather than accept unsafe concurrency.
• Disk watermarks: Warn at 20% free and stop nonessential downloads/generation at 10%.

Synchronize USAspending; consolidate Files A, B, and C; reconcile row counts and dollars; and flag expiring funds, pacing risks, missing relationships, and source changes.

Watch approved directories; extract text and metadata; manage duplicates and versions; update embeddings and indexes; and report failed ingestion, stale locks, and unindexed files.

Monitor relevant AI labs, GitHub, Hugging Face, and local runtimes. Use 9B to classify relevance and 35B to explain material impact in a focused brief.

Extract decisions, risks, actions, and contradictions; retrieve evidence; draft briefings; run repository tests and dependency reviews; and keep external actions behind approval.

Convert notes into linked concepts, identify contradictions, generate review prompts, and summarize how understanding changed.

• Create a machine-readable service registry: name, port, commands, health URL, logs, dependencies, owner, and restart policy.
• Add launchd restart controls and wait for AI_DATA and networking before startup.
• Verify graceful shutdown and reboot recovery.
• Fix the wiki-ingest stale lock, rerun the failed FY2026 synchronization, and quarantine obsolete scripts.

Exit: Core services recover automatically, avoid restart loops, and record useful failure reasons.

Build a lightweight status API and SQLite-backed dashboard. Show overall state; services and errors; loaded models, memory, requests, and latency; job status; memory, swap, disk, CPU/GPU, and temperature; data freshness and reconciliation; and authentication, rate-limit, Funnel, and heavy-gate events.

Add a durable job record containing ID, type, priority, resource class, model, input reference, timestamps, timeout, retries, status, structured error, output location, approval flag, and idempotency key.

Use resource classes: light, default, embedding, speech, image, video, and heavy-exclusive.

Generate a Daily AI Operations Brief covering overnight health, job outcomes, USAspending reconciliation, relevant AI releases, knowledge changes, anomalies, and recommended actions.

This brief is the proof of productivity. Running all night without producing decision support is not success.

• Bind backends to localhost or the private Tailnet; expose only the gateway through Funnel.
• Use separate application keys, model permissions, and rate limits; rotate keys and never log secrets.
• Disable Funnel when public access is unnecessary.
• Separate public, personal, work-sensitive, and restricted knowledge collections.
• Allow-list ingestion paths and record source, time, hash, and version.
• Use least-privilege tools, timeouts, retry limits, and output-size limits for every job.
• Preserve source files and transformation logs; treat generated output as draft unless validated.
• Evaluate model upgrades before changing defaults.
• Encrypt backups and test restoration.
• Connect the Mac Studio and storage to the UPS; test safe shutdown and recovery.
• Delay heavy jobs until network, storage, memory, thermal, and power checks pass.

Local hosting improves control but does not by itself establish compliance for organizational or restricted data.

• Fix the wiki-ingest lock and rerun the FY2026 USAspending job.
• Verify reboot-to-healthy and UPS shutdown/recovery.
• Create the service registry.
• Collect service, memory, swap, disk, queue, and power status.

• Build the operator dashboard and durable job schema.
• Add admission control and one heavy-exclusive queue.
• Run the Coder-Next test, vision parity test, and 122B bake-off.
• Add backup verification and disk watermarks.

• Generate the Daily AI Operations Brief automatically.
• Measure uptime, failures, latency, model use, and useful outputs for one week.
• Remove models and services without a demonstrated workload.
• Document normal, batch, heavy, maintenance, and recovery procedures.

The machine already has sufficient model capability. The next advantage comes from orchestration: reliable startup, controlled routing, durable queues, trusted data, observability, evaluation, and repeatable outputs.

The objective is not continuous GPU activity. It is a system that remains available, recognizes useful work, selects the smallest capable resource, preserves evidence, escalates safely, and tells its operator what matters.

Source: Mac_Studio_24x7_AI_Workhorse_Concise_Guide.md