Instrumenting Everything: Our Monitoring Stack

There's a moment every infrastructure engineer knows. You're asleep, it's 3 AM, and your phone buzzes. Something is broken, or maybe something is about to be broken. You sit up, grab your laptop, and spend the next 40 minutes figuring out what happened — when the actual failure started two hours ago and you could have caught it at the first sign if you'd been watching.

I got tired of that moment. So I instrumented everything.

The Stack

Three tools, each with a specific job:

Prometheus  — time-series metrics, scraping, alerting
Grafana     — dashboards, visualization, the thing I stare at
Loki        — log aggregation, because metrics tell you something's wrong
              but logs tell you why

All three run as Docker containers on the Proxmox server. Prometheus scrapes metrics endpoints every 15 seconds. Grafana renders them into dashboards I can actually read. Loki ingests structured logs from every service via Promtail.

The whole stack cost zero dollars in licensing and about six hours of setup time.

What I'm Watching

Here are the dashboards I actually care about, in order of how often I look at them:

Inference Latency. Every model call is timed. p50, p95, and p99 percentiles, broken down by model and endpoint. If Companion's p95 response time creeps above 800ms, something is wrong with the inference pipeline and I want to know before it hits 2 seconds.

GPU Utilization. nvidia-smi metrics exported via dcgm-exporter, scraped by Prometheus. Memory usage, compute utilization, temperature, power draw. The Proxmox GPU server reports in. If a GPU is at 99% utilization during what should be a quiet period, either a training job is running or something is stuck in a loop.

Vector DB Query Times. Companion's memory experiments can run a lot of vector queries when testing. If query latency spikes, the entire experience degrades — slow memory retrieval means slow responses and bad conclusions. I track query time, recall accuracy, and index size.

Error Rates. Every 500-level error from every service hits Prometheus within 15 seconds. Error rate above 1% for any service triggers an alert. I'd rather investigate a false alarm than miss a real one.

Disk and Memory. Boring. Essential. A full disk will take down everything and the failure mode is ugly.

The Alerting Setup

Dashboards are for humans who want to look. Alerts are for humans who need to know. The difference is whether the system wakes you up.

# prometheus/alerts.yml (excerpt)
groups:
  - name: inference
    rules:
      - alert: HighInferenceLatency
        expr: histogram_quantile(0.95,
              rate(inference_request_duration_seconds_bucket[5m])) > 1.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p95 inference latency above 1.5s"

      - alert: GPUThermalThrottle
        expr: gpu_temperature_celsius > 85
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} above 85C"

Alerts route through Alertmanager to a Discord webhook and my phone. Critical alerts wake me up. Warnings wait until morning. I tuned the thresholds over about three weeks — too sensitive and you get alert fatigue, too loose and you miss real failures.

The Logging Side

Metrics tell you that something is wrong. Logs tell you why. Loki aggregates structured JSON logs from every service — the inference API, the RAG pipeline, the vector database, the subconscious process. When an alert fires, I can jump from the Grafana metric panel directly to the correlated logs for that time window.

# Structured logging in the inference service
import structlog
log = structlog.get_logger()

log.info("inference_request",
    model="deepseek-7b",
    module="tutor-calculus",
    tokens_in=42,
    tokens_out=186,
    latency_ms=412,
    user_id=hash(user_id),
)

Every log line is structured, correlatable, and queryable in Loki with LogQL. When inference latency spikes, I filter to the affected model, sort by latency, and look at the slowest requests. Usually the answer is obvious: a long context window, a cold cache, an expensive retrieval path, or a slow model response.

The Goal

The dream of monitoring is simple: know about problems before users do.

I'm not fully there. There are still failure modes I only find by testing manually — edge cases in the RAG pipeline, weird local-model behavior, the occasional database or disk issue that does not look serious until it is. But the gap is closing. More issues now show up on a Grafana panel before they become a lost evening.

That's the difference between reactive infrastructure and something you can actually trust in production. The stack doesn't make problems go away. It makes them visible. And visible problems get fixed.

Invisible ones fester until 3 AM.