Lab
Inside my AI lab.
I run a five-machine homelab for local models, retrieval, and agents. Operating it means dealing with deployments and dependencies myself, monitoring what happens, and weighing when to run something locally or use a hosted service.
GPU workstation
A Ryzen 9950X3D and RTX 5090 run local models, detailed evaluations, and multiple agents without metered API calls.
Apple Silicon node
A Mac mini M4 Pro runs inference around the clock and hosts the production agents that need to stay up.
Ubuntu server
This server runs the reverse proxy, monitoring, and storage that let me see what's happening across the lab.
What runs on it
- Local LLM serving (vLLM, Ollama)
- Vector search (Qdrant)
- Self-hosted speech-to-text
- Single sign-on across supported web services
What I weigh when running models locally
Local inference lets me control the model version and test environment. To reproduce a run, I also need the runtime, configuration, and recorded inputs. Fixed weights alone aren't enough.
Running locally gives me control over where data goes. I still need to track what happens to it in tools, logs, and backups, and check any external services the workflow uses.
Local inference avoids metered API charges, but I pay for the hardware and power and handle capacity and maintenance. Those costs, along with model quality and latency, determine where I run a workload.