Local model vs hosted AI API for a production feature

In short: for a production feature, use a hosted AI API when you need strong general ability fast, and a model you run yourself when the task is narrow, repeatable and measurable, or when data cannot leave your infrastructure. Whichever you choose, put the model behind an interface you control and cap the spend, so changing provider is a configuration change, not a rewrite. Below are two real contrasting designs.

Two designs, two reasons

A narrow classifier you run yourself

For a recruitment platform I built, one task was classifying each CV by specialty domain. It was narrow and repeatable, so I fine-tuned a classifier on 5,617 real CVs exported from the client's applicant tracking system, split into training, validation and test sets. It reached a 0.795 micro-average F1 on the held-out test set, at roughly 136 ms inference time per CV. Years of experience, which is arithmetic over dates, was handled by a deterministic engine instead. See rules engine or machine learning and the case study. The platform also ran on shared hosting with no root access, which favoured small self-contained services.

An open-ended task through a hosted API

For a meeting summariser handling calls that mix Darija, French and English, sometimes within one sentence, I used a hosted model (Gemini 2.5 Flash) for transcription and summarisation. The transcription engine sits behind a swappable interface, with another provider (Groq Whisper large-v3) as an alternative, plus a maximum meeting length and a monthly AI-spend ceiling. See MeetScribe.

A comparison

	Hosted API	Model you run
Best for	Open-ended, language-heavy tasks; fast start	Narrow, repeatable tasks with labelled data
Cost shape	Pay per use, grows with volume	Fixed infrastructure and engineering time
Data	Leaves your infrastructure (check terms)	Stays with you
Quality control	Provider changes can alter behaviour	Version is frozen; you measure it
Operations	Little	Hosting, updates, monitoring
Latency	Network round trip	Local, predictable

Five questions to decide

1. Can you measure quality? If you have labelled data and a metric, a model you run can be tuned and tested. If quality is judged by reading, a hosted model is simpler.
2. Can the data leave? Confidentiality, contracts and regulation may decide this for you.
3. What happens at ten times the volume? Per-use cost can overtake a fixed one.
4. What if the provider changes the model or the price? You need a fallback.
5. Who operates it? A model you run needs an owner.

Guardrails that apply to both

• An interface in front of the model, so the provider can be swapped.
• A spend ceiling and a length cap, so a bug or a long input cannot produce a surprise bill.
• A human review step before anything is written back to a system of record. In the recruitment project, nothing is written back before an administrator reviews it.
• A baseline and a held-out test set drawn from real data, not examples you wrote.
• Honest scope: say which parts are built and which are roadmap.

Get an independent view

If you are deciding where AI belongs in a product or a workflow, the Architecture Power Hour is one prepared hour and a one-page decision record. For a team workflow rather than a product feature, see AI-Augmented Delivery Enablement.