AI Gateway
Self-hostable · On-prem · Compliance-ready

One API in front of
every LLM provider.

An enterprise LLM gateway built on top of LiteLLM Proxy — a single OpenAI-compatible endpoint with governance, virtual keys, budgets, usage & cost tracking, guardrails, and audit. Runs on your laptop with zero external services.

  • OpenAI-compatible
  • BYOK
  • SQLite → Postgres
  • Docker & Helm
chat/completions
# Same OpenAI wire shape — just point at the gateway
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role":"user","content":"hi"}]
  }'

# → routed, metered, priced, audited, attributed
#   to your org / team / user / app.
<15ms p50 gateway overhead
99.9% data-plane availability
5k+ concurrent streams / fleet
0deps external services for local dev
Overview

A deliberate two-plane split

LiteLLM is consumed as a pinned dependency, never forked. It owns the fast request path; our FastAPI control plane owns the product. The two are kept honest by one invariant.

Data plane

LiteLLM Proxy

The hot path: the OpenAI-compatible /v1/* surface, provider adapters, routing, and fallback. Stateless and autoscaled.

Extended only through four documented seams: custom-auth callable, callback / CustomLogger, config schema, and the OpenAI wire shape.

Control plane

Governance API

The product: orgs / teams / users / keys, model registry, budgets, usage & billing, RBAC, and audit — a FastAPI service backed by SQLite or Postgres.

Compiles the registry into derived LiteLLM YAML; secrets emitted as environment references, never plaintext.

Core invariant

Our database is the source of truth for virtual keys and spend — not LiteLLM's key store. LiteLLM authenticates every request by calling back into our DB via the custom-auth hook, which is what keeps a single SQLite file viable end-to-end.

Capabilities

Everything a team needs to govern LLM traffic

One API, many providers

A single OpenAI-compatible endpoint in front of OpenAI, Anthropic, Gemini, Bedrock, Azure, and self-hosted vLLM / Ollama.

BYOK & virtual keys

Bring your own provider keys; consumers get scoped virtual keys. Only the hash + a display prefix are stored — plaintext is shown exactly once.

Usage & cost accounting

Every request is metered, priced, and attributed to an org / team / user / app. Invoices, CSV export, and rate cards included.

Budgets & rate limits

Attach budgets and policies at any level. Most-specific-wins resolution: key › app › user › team › org.

Guardrails

Pluggable input / output guardrails and logging sinks run in-proc on the proxy via the callback hook.

RBAC & audit

Org → team → user hierarchy with roles (org-admin, team-admin, developer, billing-viewer, auditor). Every mutation appends an immutable audit event.

Reliability

Automatic fallback and retry when a primary model or provider fails or degrades — inherited from the LiteLLM router.

Self-hostable

All request/response data can stay inside your VPC or on-prem. Docker Compose for small deployments, Helm + HPA for scale.

Architecture

Data plane hot path, control plane alongside

The data plane must be fast and highly available; the control plane can tolerate brief downtime without stopping inference.

The hooks package (aigw-hooks) depends on the control plane so the proxy process shares our DB models. The reverse never holds — the control plane must not import hooks.

Setup

Up and running in four commands

Prerequisites: uv and Node 20+. No external services, no real API keys — a bundled stub provider, a SQLite file, and an in-process cache stand in for everything.

1

Install & seed

uv sync --all-packages          # install Python deps
make migrate                    # create local SQLite schema
make seed                       # demo org/team/model/key (prints a virtual key)
cd admin-ui && npm install      # install UI deps
2

Run the dev stack

Migrates, seeds, then starts the stub provider (:9099), governance-api (:8080), and the Vue UI (:5173) — all hot-reloading.

make dev
3

Sign in

Open localhost:5173 and sign in with the dev principal — user admin, the org id printed by make seed, role org-admin. API docs (Swagger) live at localhost:8080/docs.

Note — make dev does not require the LiteLLM proxy. The control plane, UI, and tests never route real inference. Run make proxy to install the litellm[proxy] extra and exercise real /v1/chat/completions calls on :4000.
Run

One target per job

CommandWhat it does
make devMigrate + seed + stub provider (:9099) + governance-api (:8080) + Vue UI (:5173)
make apiBackend only — uvicorn hot reload on :8080
make uiFrontend only — Vite on :5173, proxies /api → :8080
make proxyInstall litellm[proxy] extra + run the LiteLLM data plane on :4000
make docker-upBuild + start the whole stack via Compose, detached, and leave it running
make testUnit tests — pytest (backend) + vitest (UI)
make e2eEnd-to-end — boots the real server and drives the API over HTTP
make e2e-dockerFull-system QA — Compose up, ~34 assertions across both planes, then teardown
make lintruff check + mypy (strict)
make smokeShell smoke — migrate → seed → API → authenticated request

Full stack including the data plane, in one Docker command:

make docker-up       # control :8080 · proxy :4000 · admin-ui :8081 · stub :9099
make docker-down     # stop it when you're done
Deploy

From a laptop to a cluster

Docker Compose

On-prem / small deployments

cd deploy/docker-compose
docker compose up --build
# --profile scale adds Postgres + Redis

Kubernetes / Helm

Scale — proxy autoscales via HPA

helm upgrade --install ai-gateway \
  deploy/helm/ai-gateway ...

Swap SQLite for Postgres by changing AIGW_DATABASE_URL only — same models, same migrations. Operations, upgrades, secrets, and incident response live in the runbook.

Stack

Boring, local-first, well-tested

BackendPython · FastAPI · uv · SQLAlchemy 2.0 · Alembic
DatastoreSQLite by default · Postgres by connection-string swap
FrontendVue 3 · Vite · TypeScript · Pinia
Data planeLiteLLM Proxy (pinned) + custom-auth & callback hooks
Testingpytest + vitest · TDD · local-first, no external services
DeployDocker Compose · Kubernetes / Helm + HPA

Ready to run it?

Clone the repo and you'll have a governed, OpenAI-compatible gateway on your machine in minutes — no accounts, no cloud, no keys.