Llama 3 on Groq: Open Source at Ludicrous Speed

Meta released the Llama 3 family in April, and the 70B model is, by broad consensus, one of the most important open-weights releases so far. That alone would be a story. But the thing that actually rearranged my brain this month is where we ran it.

Groq. I need to talk about Groq.

The Speed

If you have not tried Llama 3 on Groq yet, the first experience is almost comic. You press enter and the answer arrives so quickly that your normal rhythm for reading model output breaks. We are used to waiting for language models: the small pause, the first token, the gradual stream. On Groq, the stream feels less like typing and more like opening a tap.

The first time we tested it seriously, I laughed. Not because the answer was funny, but because the latency was.

Groq achieved this not by throwing another ordinary GPU at the problem but by building hardware specifically for inference. Their LPU is designed around the memory and sequencing patterns that large language models need at runtime. Whether that architecture becomes the default is a separate question. What mattered in the moment was simple: Llama 3 was fast enough that conversation felt different.

Why Llama 3 Matters

I should not bury the model itself. Llama 3 was good enough to take seriously in a way that earlier open models often were not. It could hold a conversation, follow instructions reasonably well, and sustain a personality experiment without immediately collapsing into nonsense.

That last part mattered because we were testing it with a personality we named Luna.

Luna was not a finished product. She was one of the experimental personalities we used to test memory, tone, system instructions, and model behavior across different providers. The point was not to ask, "Can Llama 3 answer trivia?" The point was to ask, "What does this model feel like when it is asked to carry a conversational identity for a while?"

On Groq, Luna felt startlingly alive in the narrow, practical sense of conversational speed. The back-and-forth was quick enough that the usual pause between thought and response almost disappeared. That changes the texture. You can test ideas quickly. You can refine tone. You can push and observe without waiting long enough to lose the thread.

The Follow-Up Question Loop

But speed exposed problems too.

One behavior showed up repeatedly: the model would get stuck asking a follow-up question almost every time it responded. It did not matter whether the question was useful. It did not matter whether the conversation needed one. Luna would answer, then pivot into another question, as if every exchange had to end by handing the turn back in the same formulaic way.

At first, this sounded like friendliness. Then it became a pattern. Then it became hard not to see.

We tried changing the system instructions. We told the model directly what it was doing. We explained that it should not always end with a follow-up question, that sometimes a complete answer should simply end. The behavior persisted more than I expected. It was one of those moments where prompting felt less like steering a car and more like arguing with a river.

That was instructive. A model can be open, fast, and capable, while still having conversational grooves that are difficult to remove. The system prompt matters, but it is not magic. Some behaviors seem baked into the model's training style or alignment habits deeply enough that direct confrontation only helps a little.

Local Experiments

We also tested Llama locally. That was part of the appeal of open weights: the idea that we could run the model ourselves, study it, and control the environment without depending entirely on someone else's API.

The reality was more mixed. Even with GGUF builds and quantized variants, local inference was slow on our hardware compared with Groq. Usable for experiments, yes. Useful for understanding the deployment tradeoffs, yes. But not the same conversational experience. The speed difference was not a small optimization detail. It changed whether Luna felt responsive or sluggish.

That made the tradeoff very clear: local gives control, Groq gives speed. For a companion-style system, speed matters because latency is emotional. A delayed answer feels distant. A fast answer feels present.

What This Implies

The lesson from Llama 3 on Groq was not simply "open source caught up" or "fast inference is nice." It was more specific than that.

Open models were becoming good enough to participate in serious product experiments. Specialized inference hardware could make them feel immediate. But model behavior still mattered, sometimes in stubborn ways. Luna could be quick, engaging, and useful, then get trapped in a conversational habit that no simple instruction seemed to fix.

That is the actual engineering problem: not just finding a model that can answer, but finding a model that behaves well inside the shape of the product. Memory, personality, tools, latency, local deployment, API deployment, and conversational habits all interact.

Llama 3 on Groq showed us a possible future where open models are fast enough to feel present. Luna showed us that presence alone is not enough. The personality has to breathe without getting stuck in a loop.

Live curiously and give generously.