Llama 4 Maverick: Meta Goes MoE

Meta just released Llama 4 Maverick, and the big story isn't the benchmark numbers — it's the architecture. Llama 4 is Meta's first major open-weight model to use Mixture-of-Experts at scale, and that has real implications for anyone building specialized AI systems.

Why MoE matters

If you've been following this blog, you know I've been using DeepSeek's MoE models for the subconscious process in Companion — the idea being that MoE models are cheaper to run because only a subset of parameters is activated per token. Llama 4 brings that same efficiency to Meta's ecosystem, with a twist: the routing is particularly clean.

Here's why I care, specifically: MoE and LoRA are a beautiful combination.

In a dense model, when you train a LoRA adapter, you're fine-tuning a small set of weights that affect the entire model's behavior. In an MoE model, you can target specific experts — effectively saying "I want to customize the part of the model that handles mathematical reasoning without touching the part that handles creative writing." That's incredibly powerful for Tutor, where each subject LoRA should specialize without bleeding into other domains.

My experiments

I studied Llama 4 Maverick as a possible future base and started mapping how LoRA experiments might work once the compute is there. The first question was whether expert-specific fine-tuning could eventually help with narrow skills like chess notation and tactics, targeting the experts that seemed to activate most during logical/strategic reasoning tasks.

The idea was promising on paper and in community reports: a narrow adapter could improve domain behavior without degrading unrelated tasks. In a dense model, the same experiment would have caused measurable degradation across the board — what researchers call "catastrophic forgetting." With MoE, the forgetting was minimal.

This is exactly what Tutor needs. If Tutor eventually loads a physics LoRA, it shouldn't make the model worse at Spanish. MoE architecture makes that isolation much more feasible.

For Companion

Companion benefits differently. I've been thinking about the subconscious process — the background model that runs continuously, summarizing conversations, updating the vector database, preparing context. Maverick's MoE design means this process runs even cheaper than before. Only the relevant experts activate per token, so the effective compute cost is a fraction of what a dense model of similar quality would require.

I ran the numbers: Maverick's per-token cost on our infrastructure is roughly 40% lower than a comparably-sized dense model. For a process that runs 24/7, that's not a marginal improvement. It's the difference between economically viable and economically impossible.

The open-source flywheel

Every time Meta releases a new open-weight model, the entire ecosystem benefits. Researchers study the architecture. Companies like ours build on it. Tooling improves. And then the next release is even better because the foundations got stronger.

I was hiking this weekend — up to a ridge where you can see the entire valley — and I was thinking about how each of these model releases is like a new trail being blazed. You don't have to take the same trail as everyone else. But the fact that it exists, that someone cleared the path, means you can go further than you could before.

Llama 4 Maverick is a good trail. I'm going to enjoy walking it.

Live curiously and give generously.