Running an open-weight language model on your own infrastructure sounds simple: download the weights, start a server, point your application at it. In practice the decision touches licensing, hardware, security and operations. This checklist walks through the questions in order, so you commit to self-hosting only when it fits your use case.
It is written for CTOs, founders and engineering leads. It avoids model names, version numbers and prices on purpose, because those change monthly while the decision process stays the same. For an example of how to read a current open-weight release, see our article What is Qwen 3.8.
Open-weight is not open source: read the license first
An open-weight model publishes its trained parameters so you can run them yourself. That is different from open source in the traditional sense, where code, data and a permissive license are all available. Many open-weight releases ship under custom licenses, and the label on a download page does not tell you what is allowed.
Read the license before any engineering work. Check these points:
- Is commercial use allowed, and are there limits based on user count or revenue?
- Are there restrictions on fields of use, or on using outputs to train other models?
- Are there attribution or naming requirements, and must fine-tuned derivatives carry the same terms?
- Does the license cover the weights, the code or both, and is it the version you actually downloaded?
If the license is unclear for your case, ask legal counsel before building. A model you cannot ship to customers is not a candidate, however well it performs.
Define the use case and build an evaluation set
Choose the model after you define the task, not before. Write down what the system must do, what an acceptable answer looks like, what latency users tolerate and what a wrong answer costs.
Then build a small evaluation set from your own data: real questions or documents, with expected outputs or scoring rules written by someone who knows the domain. Keep it separate from prompting or fine-tuning data, and include hard and ambiguous cases. Public leaderboards cannot show how a model behaves on your documents, language and edge cases.
Run the same set against a hosted API model as a baseline and against each open-weight candidate. Re-run it whenever you change the model, the quantization level, the prompt or the serving software. Without this set, every later decision in this checklist is a guess.
Plan hardware, memory and quantization
Memory is usually the first constraint, not compute. Plan for three consumers of accelerator memory: the model weights, the key-value cache that grows with context length and with concurrent requests, and overhead from the serving software. A model that fits at idle can still fail under real traffic.
Work backwards from your workload: longest prompt, simultaneous users and response length. Measure on a test machine instead of trusting a formula, and keep headroom for peaks.
Quantization stores weights with fewer bits, which reduces memory and often speeds up generation. The trade-off is a possible loss of quality that varies by model, task and method. Treat it as a variable to test:
- Run your evaluation set at each precision level you consider.
- Watch structured output and less common languages, where degradation is more likely to show.
- Choose the lowest precision that still passes your quality threshold.
Splitting one model across several accelerators adds communication overhead and operational complexity, so use it only when a single device cannot hold the model.
Choose the serving layer and decide where data lives
The serving layer is the software that loads the model and exposes it as an API. Choose by requirements, not popularity: support for your model architecture and quantization format, continuous batching, streaming, a compatible API, observability hooks and active maintenance.
Self-hosting is often chosen for privacy, so make the privacy claim concrete. Map where prompts, outputs, logs, caches and backups live, who can read them and how long they are kept. A model running inside your network can still leak data through verbose logs or a shared vector store.
Data residency adds a second question: in which country and under which legal regime the machines sit. If you rent GPUs from a provider, the data lives in that provider's jurisdiction, so check contract terms and regulations such as GDPR or KVKK. Put authentication, network isolation and rate limits in front of the model endpoint from day one.
Monitor, secure and update the deployment
A self-hosted model is a production service and needs the same care as a database. Monitor these signals:
- Time to first token and total request time, as percentiles rather than averages.
- Throughput, queue length and accelerator memory use.
- Error rates and timeouts.
- Output quality, using a sample of real traffic scored against your criteria or reviewed by people.
Security has model-specific parts. Treat model output as untrusted input to the rest of your system: validate structured responses, never let generated text run commands or queries unchecked, and defend against prompt injection when the model reads external documents. Download weights only from sources you trust, verify checksums and prefer formats that cannot execute code on load.
Plan updates before launch. Pin exact model and software versions, test every change against the evaluation set in staging, roll out gradually and keep the previous version ready for rollback. Decide who owns this work.
Compare cost with a hosted API honestly
A cost comparison fails when it counts only the hourly price of a GPU. Build it from the same workload on both sides: requests per day, average input and output length and peak concurrency.
On the self-hosted side include accelerator rental or purchase, power and hosting if you own hardware, engineering time for setup and on-call, monitoring, storage, redundancy for failover and the idle capacity you pay for between peaks. On the hosted side include per-token charges at your real volume, minimum commitments and the impact of rate limits or outages.
Self-hosting tends to make sense when usage is steady and high, when data cannot leave your boundary or when you need control over model behavior. A hosted API tends to win for spiky or low volume, fast prototypes and teams without operations capacity. A hybrid is common: sensitive workloads in-house, the rest hosted. Recalculate regularly, since options on both sides change.
Frequently Asked Questions
Is an open-weight model free to use commercially? Not automatically. The license sets the terms, and some limit commercial use or large deployments. Read it and confirm with legal counsel.
Do I need a GPU to run an open-weight model? Larger models generally need accelerators with enough memory for the weights and the cache. Small or heavily quantized models can run on modest hardware, but speed and quality vary, so test with your own workload.
Does quantization always hurt quality? It can, but the effect depends on the model, the method and the task. Some tasks barely change while others degrade noticeably. Measure with your evaluation set instead of assuming.
When is a hosted API the better choice? When volume is low or unpredictable, when you need to prototype quickly or when you lack a team to operate and secure a model service. Keep your application independent of any one provider so you can move later.
If you are weighing self-hosting against a hosted API, Detartech can help with evaluation, architecture and rollout through its AI solutions service. You can also describe your project through the quick quote form.
Related Services
