Combines 3 small local LLMs, with reasoning capability rivaling Anthropic Fable 5

This paper introduces a dynamic model fusion method at the logit layer — by running three specialized small LLMs in parallel on a local GPU and weighting their logit outputs together before sampling, it synergizes different models' advantages in grammar, creativity, and logical reasoning. By avoiding modifications to model weights, this approach also addresses data privacy, inference efficiency, and local deployment costs, offering a new paradigm for building high-performance private AI inference systems on consumer hardware.

image

Every time you send a prompt, you pay $0.12 to a centralized API just so a closed-source model can parse simple customer logs — and you constantly worry it might hallucinate. It's slow, expensive, almost absurd. The industry wants you to believe that only through massive expenditures can you access cutting-edge intelligence. But this isn't the case.

Most developers look at lightweight local models like Llama 3 8B or Gemma 2 9B and think they're merely "toys." They run a few standard prompts, see the model occasionally make errors in logical reasoning, and immediately return to expensive cloud APIs. But what if we don't have to choose between the high cost of centralized servers and the performance limits of local devices?

Here's something rarely told: solving highly complex reasoning tasks doesn't require a massive 400-billion parameter model entirely. By fusing the probability distributions generated at the sampler level by three specialized small models, we can achieve near-leading-edge inference precision rivaling multi-agent frameworks like Anthropic Fable 5.

Below, I demonstrate exactly how this works at the low level.

# img

image

Most tutorials stop at simple prompt routing — sending different queries to different API boxes. Don't do that; this approach is essentially just a very slow, fragile workaround. But if we want to truly understand the underlying mechanism of token generation, we must go deeper than token probability distributions. Standard models output probability distributions as logit arrays. If we load three small models simultaneously into your GPU memory, they don't have to execute the same prompt sequentially. Instead, their forward passes run in parallel, and based on task requirements, vector weights are assigned to outputs from different models, with unified token masks applied.

When a model computes its next-token vectors, we intercept the logits before sampling occurs. We apply weighted averaging across their probability matrices — multiplying each model's output vector by specific task-dependent weights and applying a combined token mask. The three models vote on the next character at the hardware level; without modifying any underlying parameters, their grammar and logic strengths are integrated in our fusion mechanism.

# img

image

image

image

image

You've probably already seen static model merging on Hugging Face — using tools like mergekit with spherical linear interpolation (SLERP) or task vectors, permanently fusing different models' weights.

Many tutorials recommend this approach. Don't adopt it so quickly.

The hidden problem is that static weight fusion can be highly destructive — when you permanently fuse the parameter matrices of two neural networks, the mathematical structures that give each model its unique capabilities might be flattened or diluted. Eventually, models may lose their inherent advantages and exhibit subtle "cognitive decay" that standard benchmarks struggle to capture.

Runtime dynamic fusion bypasses this limitation entirely. We retain each model's specialized parameters intact in isolated GPU buffers, allowing every model to evaluate the current context at its native capability. For example: the structured code model handles grammar analysis, the creative model judges narrative style, and the logic model cross-verifies facts. A logit router serves as an arbiter, achieving output precision comparable to top-tier closed-source models without parameter degradation.

90% of people get stuck here: writing a complex Python wrapper that loses milliseconds querying a server pool sequentially. If you want to build a truly responsive local system, we can write a lightweight PyTorch custom sampler and perform the math at the logit layer entirely within local GPU memory.

Next, let's look at a concrete implementation: using Python to fuse probability distributions generated by three local models, enabling high-quality handling of structured technical tasks.

# Dynamic Logit-Level Ensemble Fusion Controller

Wait — before going further, let's examine the math in this loop. It introduces no added latency. On modern consumer GPUs, running three small models in parallel takes less time than invoking a single massive cloud model through a standard network round-trip. The result: absolute data privacy, zero subscription fees, and extremely low-latency local inference performance. 图片 图片

Original title: I Fused 3 Tiny Local LLMs on my Laptop and Matched the Reasoning of Anthropic Fable 5 Original link: https://pub.towardsai.net/i-fused-3-tiny-local-llms-on-my-laptop-and-matched-the-reasoning-of-anthropic-fable-5-4e62930b2bf0