Planning the GPU Server: The Expensive Road to Local Fine-Tuning

Ella writes about personality and learning loops. I write about the part where the spreadsheet turns red.

By August, the conclusion was obvious: if Sorren.ai was going to do serious local fine-tuning and inference, the workstation experiments were not enough. The RTX 3090 had been useful. It taught us what local models felt like, what GGUF could and could not solve, and where latency became a product problem instead of a benchmark number.

But a workstation GPU is not an infrastructure plan.

So I started pricing the server we would actually need.

The 5090 Temptation

The obvious consumer-hardware fantasy was simple: put two RTX 5090s in a box. Or four, if I wanted to turn the power bill into a character-building exercise.

On paper, the idea made sense. Consumer cards offer a lot of raw performance for the money. If the goal is to get as much GPU compute as possible into a rackmount ATX case, the 5090 path is tempting. Two cards could handle a serious amount of experimentation. Four cards would be absurd in the way infrastructure plans become absurd when you stare at spec sheets too long.

Then reality arrived, carrying a calculator.

The cards were expensive. The motherboard requirements were expensive. The power and cooling requirements were expensive. The case airflow question was not theoretical. The 1600 watt power supply I had bought suddenly looked less like overkill and more like the beginning of a negotiation.

The cost was the prohibitive issue. Not desire. Not use case. Money.

Why the Pro Cards Stayed in the Conversation

The other path was RTX Pro 6000-class hardware. More expensive in some ways, but better aligned with the kind of server I was trying to build: lots of VRAM, workstation/server reliability, fewer gymnastics around multi-GPU fitment, thermals, and long-running workloads.

For AI work, VRAM keeps coming back as the hard constraint. Training adapters, running larger local models, keeping context windows useful, and avoiding constant offloading all push you toward cards that are not merely fast, but spacious.

That is why the RTX Pro 6000 option stayed on the board. It was not the cheap path. It was the path that looked less likely to become a pile of compromises later.

What We Actually Had

At this point, the server build was still staged.

The rackmountable ATX case was there. The 1600 watt power supply was there. The old Dell servers were still doing their jobs. My workstation was still handling local experiments. I had not magically built the final Proxmox server yet.

That matters because infrastructure posts have a way of making everything sound more complete than it was. In reality, this was the planning phase: price the GPUs, understand the power envelope, decide what storage should be SSD and what could be old magnetic drives, and figure out how to get from "parts on a table" to "Proxmox host I trust."

The Storage Split

The storage plan stayed consistent:

• SSDs for active workloads
• magnetic drives for backups and cold storage

Fine-tuning, inference, databases, vector indexes, and running VMs should not be waiting on old spinning disks. Backups can. Cold file storage can. The trick is not to spend NVMe money on data that only needs to sit still and survive.

That is the kind of decision you make when funds are limited. Spend where latency matters. Save where it does not.

The Honest State

No, this was not the week I built the final LoRA training rig.

This was the week the requirements became unavoidable. We needed local GPU compute. We needed Proxmox. We needed enough VRAM to make the work meaningful. We needed a machine that could grow into Companion, Tutor, background memory work, and model experiments without turning every new idea into a cloud bill.

And we needed to pay for it somehow.

That last line is not romantic, but it is infrastructure. The technical plan can be sound and still have to wait for the budget to catch up.

The spreadsheet remains rude.