Running Models Locally: Workstation Inference Experiments

Saturday morning, coffee, GPU fans rising and falling under my desk. Time to see what we can actually run ourselves.

Still no Proxmox server available, so the workstation is doing the work for now: an RTX 3090, a pile of model downloads, and a question: how much of the Companion stack can we test locally before paying someone else for every token?

The answer, spoiler: enough to learn from, not enough to replace the frontier APIs.

The Hardware

The test box is my workstation with a single RTX 3090. The important number is 24GB of VRAM. That sounds like a lot until you start loading models and context windows and discover how quickly "a lot" becomes "almost enough."

The 3090 is still the right card for the phase we are in. It lets us test small models, quantized models, embedding workflows, and local inference wrappers without waiting on cloud capacity or worrying about per-token cost. It is not a production inference server. It is a laboratory.

That distinction matters.

What We Tested

We tried the usual suspects: small Llama-family models, DeepSeek variants, uncensored Hugging Face models, and GGUF builds that could run through local tooling. The point was not to win a benchmark. The point was to understand the tradeoffs.

Local models give us privacy and control. They also give us latency.

Some prompts are fine. Short summaries, extraction tasks, small memory-cleanup jobs, and narrow classification work are usable. Longer reflective conversations are a different story. The model can run, but it does not feel alive in the way GPT-4o or Groq-hosted Llama does. There is a difference between "this completed" and "this feels conversational."

We feel that difference immediately.

GGUF Helped, But It Did Not Fix Physics

GGUF builds were useful because they made local testing practical. Quantization let us fit models that would otherwise be annoying or impossible to run on the workstation. It reduced memory pressure and made experiments less painful.

But it did not turn a workstation into a datacenter.

Even with quantized models, local inference was often slow compared with hosted APIs. That was especially obvious after seeing Llama 3 on Groq. Once you experience absurdly fast tokens, a local model that technically works but makes you wait starts to feel like talking through syrup.

That does not make local inference useless. It just clarifies the role.

Where Local Won

Local won for development tasks where privacy and repeatability mattered more than speed:

• testing memory summarization prompts
• experimenting with uncensored models for background reflection
• checking whether a small model could extract useful facts from logs
• running rough personality experiments without sending every attempt to an external provider
• understanding the operational shape of model hosting before we bought server hardware

It also helped us think about the eventual Proxmox build. Local testing showed what mattered: VRAM, storage speed, power, thermals, and the emotional reality of latency. A model that responds slowly feels different, no matter how good the answer is when it finally arrives.

Where APIs Still Won

The hosted models were simply better for high-quality conversation. GPT-4o, Claude 3 Sonnet, and Groq-hosted Llama were faster, stronger, or both. For the parts of Companion that needed to feel fluid and emotionally present, the API models were still the better tool.

That is not disappointing. It is useful. The workstation experiments are telling us which workloads can move local first and which ones still need frontier or specialized-hosted inference.

Local for privacy and background work. APIs for quality and velocity. Build the architecture so either can be swapped later.

What This Taught Us

This is the phase where we learn what not to assume.

Do not assume open weights mean cheap if the hardware cannot keep up. Do not assume local means better if latency kills the experience. Do not assume quantization is free. Do not assume a model is useful for Companion just because it can answer a benchmark prompt.

The RTX 3090 is pulling its weight. It is teaching us the shape of the problem. That is enough for November.

Coffee's cold. The GPU is still warm.