Developer tools decision
LLM Gateway
A competent technical user can build a useful self-hosted gateway using existing open-source proxies ( cited prior art ) and managed provider APIs; the main durable advantage the vendor has is its SOC 2 compliance and hosted SLAs which you would forgo or need to validate separately.
Visit website↗Open-source builds that already do this
Every project below is open source and already does this job today. Fork one, self-host it, or take the parts you need - the build prompt further down assumes an empty file, and this is the shortcut past that. Licences differ; check the one on each card before you ship. All LLM Gateway alternatives, with the arithmetic →
What a replacement has to do
- Accept an OpenAI-compatible API request, route it to a selected provider (BYOK or pooled), apply guardrails, record cost/latency/tokens, and return the provider response to the client.
What it still won’t have
- Hosted SLAs, uptime guarantees and managed enterprise support
- Out-of-the-box SOC 2 attestation and vendor compliance evidence
- Any proprietary integrations, paid billing/credit flow and negotiated volume discounts
- Polished analytics dashboards and built-in model directory
What remains hard
- Compliance and regulation
LLM Gateway Is Now SOC 2 Type II Certified
- Compliance and regulation
SOC 2 Type II certified
First-year cost
No published price
LLM Gateway does not publish a price we could read, so there is nothing to compare against. What building costs is below.
Money you would actually spend
Time you would spend
—
What you would spend
What we assumed
The verdict above measures whether you could build it. This one is only about money.
Runnable build prompt
Build a self-hosted LLM gateway in Node.js (TypeScript) with Fastify for the HTTP API, Postgres for metadata, Redis for cache/state, and Sequelize/Prisma for DB access. Implement: (1) an OpenAI-compatible proxy endpoint that accepts client requests and preserves request/response shape, (2) provider adapter layer for OpenAI and one other provider (Anthropic or Google) with routing and failover, (3) secure BYOK storage (encrypt keys at rest) and per-project API key handling, (4) guardrails module with PII/secret regex checks and prompt-injection heuristics (configurable rules), (5) observability: log tokens, latency, provider, and compute per-request cost; include a minimal web UI to view recent requests and cost charts, (6) tests for adapter routing, guardrails blocking, and DB persistence, and (7) error handling, retries, and configurable timeouts. Out of scope: full multi-provider catalog, enterprise billing UI, SOC2 attestation, and advanced analytics. Provide Docker Compose and Kubernetes manifests for deployment, health checks, and a README with setup and post-install verification steps.
How we checked
How the score was reached
- Partly verdict base52
- An open-source build was found+5
- 5 cited sources+3
- 3/3 assessment runs agreed+4
- Hard moats found in the evidence-3
- Evidence score61
The base comes from the verdict. Everything under it is a check that either happened or did not, and each one is a fact frozen in this record rather than a judgement made at render time - so the same evidence always produces the same number.
How scoring works →Cited sources · 5
Every page the run actually retrieved.
- official productLLM Gateway - Unified API for Multiple LLM Providers
- official pricingLLM API Pricing — Provider Rates, Free BYOK | LLM Gateway
- official docsLLM Guardrails | LLM Gateway
- open sourceBerriAI/litellm
- open sourcecoaidev/coai
Integrity checks
What held up, and what did not.
