An internal LLM gateway is a policy boundary between product code and model providers. It centralizes provider selection, typed outputs, budgets, logging, retries, and shutdown controls so each feature does not invent a different safety model. The gateway helps, but it does not replace use-case-level permissions or human review.
What contract should the gateway expose?
Expose the smallest interface the product needs, such as a structured completion method that accepts a versioned prompt and an output schema. Return the actual provider and model used, token or cost measurements when available, latency, and a validation verdict. Keep provider-specific response shapes behind the adapter.
Why are typed outputs important?
Model output is untrusted input. Parse it, validate it against a strict schema, and reject unexpected fields before the result reaches a database, workflow, browser, or tool. A valid shape still does not make the content true; factual checks and business rules remain separate gates.
Where should human review be required?
Require an explicit human decision before content reaches a real person, a public page, or a live system when the consequence matters. The model may draft or classify within a bounded workflow, but code owns permissions and deterministic decisions. A prompt instruction is not an authorization control.
How should fallbacks work?
The system should run without a model key when the use case permits it. A deterministic fallback can preserve structure, mark the model stage as skipped, and cap the result below any publication threshold. Provider failures should not silently switch to a behavior with different privacy, cost, or quality assumptions.
What belongs in the budget?
Set limits for requests, tokens, cost, latency, retries, and tool calls per job or user action. Record which limit stopped the operation. Unbounded retries turn an availability problem into a cost problem; unbounded context turns irrelevant data into privacy and injection risk.
What should be tested?
- Malformed and adversarial model output.
- Provider timeout and rate-limit behavior.
- Fallback selection with no credentials.
- Budget exhaustion and kill-switch behavior.
- Duplicate delivery and idempotent retries.
- Human-review enforcement.
- Provenance from source through published draft.
Grafiesto applies this bounded approach in its AI engineering work: the model can help produce an answer, but deterministic code decides what the system is allowed to do.