AI Models
Open Weights Moved the Buy-versus-Build Line
The argument about self-hosting is not ideology, it is arithmetic. Find your volume before you pick a side.
The argument about open-weight models is usually framed as ideology. It is arithmetic. For a fixed workload there is a volume below which an API is cheaper and a volume above which a GPU is. The useful work is finding your volume, not picking a side.
What self-hosting actually costs
The model weights are free. Everything around them is not: a GPU that bills by the hour whether or not a request arrives, a serving stack (vLLM or llama.cpp, both excellent and both something you now operate), quantisation choices that trade quality for memory, monitoring, an upgrade path, and the person who gets paged at 2am when the container runs out of VRAM.
- Idle cost — a dedicated GPU is a fixed monthly bill. An API is zero when nobody is using the product.
- Throughput cost — at high utilisation, self-hosted tokens are dramatically cheaper per million.
- People cost — the most expensive line item, and the one missing from every comparison blog post.
A small product with spiky traffic almost never wins on self-hosting. A product with a steady queue, a privacy constraint, or a fine-tune that must not leave the building often does.
The three cases where open weights win outright
- Data residency — the input cannot legally cross a border. No API contract fixes that; a GPU in your own rack does.
- A narrow, repeated task — classification, extraction, embedding. A 7B model fine-tuned on 5,000 labelled examples can beat a general frontier model on that one task, at a fraction of the price.
- Latency floor — a local model on the same machine as the application removes a network hop and a rate limit from the critical path.
The hybrid that usually wins
Run the cheap, high-volume, well-defined work on a local model and send the rare hard case to an API. The router decides by task type, not by confidence score — a model's self-reported confidence is not a calibrated probability, and building a control flow on it is how you get a system that escalates everything or nothing.
const route = (task) =>
task.kind === "extract" || task.kind === "classify"
? localModel // steady volume, narrow task
: frontierModel; // rare, open-ended, worth the priceBuy the frontier. Own the commodity. Route between them by task, not by vibes.
One more thing worth writing down: open weights are a continuity guarantee. If a provider changes its terms or retires a model, a self-hosted checkpoint keeps working. That is not a cost saving; it is insurance, and it is worth paying for when the product depends on it.