This walkthrough uses the one-command Docker stack, which boots the whole gateway pre-seeded with demo data and a fake upstream provider — so every step below is real, with no cloud account and no provider API key. You only need Docker installed.
How it's built
Before the steps, a one-minute mental model. The gateway is a deliberate two-plane split. The data plane is LiteLLM Proxy, consumed as a pinned dependency — it owns the fast, OpenAI-compatible request path and all the provider adapters. The control plane is our own FastAPI service plus a Vue console — orgs, teams, users, keys, the model registry, budgets, usage, and audit. That is the product.
/v1/* · routing · fallback
The one invariant that makes everything else work: our database is the source of truth for keys and spend, not LiteLLM's own key store. The proxy authenticates every request by calling back into our DB through the hooks — which is why a revoke (step 14) takes effect on the very next call.
Anatomy of a governed request
Everything the later steps demonstrate happens on a single request path. A call enters the proxy, our hooks run governance before it reaches any provider, and meter it after — so nothing is billed for a request that was blocked, and nothing that succeeds goes uncounted.
Keep this picture in mind: the getting-started steps walk the console top to bottom, Part 3 exercises each pre-call gate on its own, and Part 4 shows the config and pricing that feed it.
Start the gateway
From the repository root, bring the whole stack up in Docker and leave it running:
make docker-up
This builds and starts four services and seeds a demo organization:
- Control plane (governance API) —
http://localhost:8080 - LiteLLM proxy (the OpenAI-compatible endpoint) —
http://localhost:4000 - Admin console —
http://localhost:8081 - Stub provider (fake upstream, so requests need no real key) —
http://localhost:9099
When it finishes, it prints the seeded org id and a ready-to-use virtual key — keep both handy:
AI Gateway is up and running:
Control plane http://localhost:8080 (/docs for Swagger UI)
LiteLLM proxy http://localhost:4000
Admin UI http://localhost:8081
Stub provider http://localhost:9099
org: 3b9d4553c6e44513b2efecc78a707702 (sign in with this org id)
key: sk-ag-… # shown once
Your org id and key will differ from the example above — copy the values your
own run prints. Stop everything later with make docker-down.
Sign in to the console
Open http://localhost:8081.
Authentication is a dev stand-in until SSO lands, so you sign in by identity rather than password:
enter the user id admin, paste the org id from step 1, keep the role
org-admin, and choose Enter console.

org-admin is the top role: an org-admin has every team-level permission across the whole organization.
Read the dashboard
The dashboard is the at-a-glance view of your org: the control-plane health, spend so far, total requests, and how many budgets are approaching their limit. On a fresh seed you'll already see activity from the demo data.

Register a model
The model registry is the source of truth for what callers can use. Each entry maps a friendly public name (what clients ask for) to a provider and the real upstream model. Add one with the form, or start from the three seeded demo models.

The registry is the source of truth; the LiteLLM proxy config is a derived artifact compiled from it, with credentials emitted as environment references — never plaintext.
Create teams
Everything is scoped org › team › user › app. Teams are how you group people and
attribute spend. The seed ships three — Platform, Research, and Support — and you can add more from
the Teams page.

Issue a virtual key
Consumers never touch your provider keys — they get a scoped virtual key. Pick a team, choose Issue key, and the full key is shown exactly once. After that only a display prefix and a hash are stored, and you can Revoke a key at any time.

Make your first request
Point any OpenAI-compatible client at the proxy on port 4000 and authenticate with your
virtual key. The wire shape is exactly OpenAI's — only the base URL and the key change:
export VIRTUAL_KEY=sk-ag-… # the key from step 1
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $VIRTUAL_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "demo-gpt",
"messages": [{"role":"user","content":"Say hello"}]
}'
You get a standard chat-completion back — served here by the stub upstream — with a token count the gateway uses to meter and price the call:
{
"model": "demo-gpt",
"object": "chat.completion",
"choices": [{ "index": 0, "finish_reason": "stop",
"message": { "role": "assistant", "content": "Hello from the AI Gateway stub." } }],
"usage": { "prompt_tokens": 8, "completion_tokens": 7, "total_tokens": 15 }
}
A wrong or revoked key is rejected before it ever reaches a provider — the proxy calls back into
the control-plane DB to authenticate every request and returns 401 invalid API key.
Track usage & cost
Every request you just made is metered, priced, and attributed. The Usage page breaks spend down by model — prompt and completion tokens, cost, and request counts — and the same data can be grouped by team or by day for invoices and CSV export.

Set a budget
Attach a spending limit at any scope — key, app, user, team, or org. Budgets resolve
most-specific-wins (key › app › user › team › org), and the console flags
any that cross their alert threshold before they overrun.

Use the API directly
Everything the console does is the control-plane REST API — orgs, teams, keys, models, budgets, and
usage. Explore and try it interactively in the Swagger UI at
http://localhost:8080/docs.
Authenticate the same way the console does, with the X-User-Id, X-Org-Id, and
X-Org-Roles headers.

/docs).A quick health/version check needs no console at all:
curl -s http://localhost:8080/api/v1/version
# → {"version":"0.1.0","litellm":"1.91.1"}
A virtual key's lifecycle
The same story end to end, driven from the API: mint a key, use it from your own app, watch the spend it produces, then switch it off and confirm it stops working — the moment you revoke, the data plane rejects it, because the control-plane DB is the single source of truth.
Issue a key from the API
Create a key scoped to a team (and optionally restricted to specific models). The full key is
returned once, in the key field — store it now; afterwards only the
prefix and a hash remain.
curl -s http://localhost:8080/api/v1/keys \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin" \
-H "Content-Type: application/json" \
-d '{"team_id": "<platform-team-id>", "allowed_models": ["demo-gpt"]}'
{
"id": "f17cf5df…", "prefix": "sk-ag-uSRx-2mz", "status": "active",
"allowed_models": ["demo-gpt"],
"key": "sk-ag-uSRx-2mzGYw_…" # shown once — copy it now
}
Need the team id? GET /api/v1/teams?org_id=$ORG_ID lists them, or read it off the
Teams page (step 5).
Call it from your app
Because the proxy is OpenAI-compatible, you don't need a new SDK — point the official OpenAI client at the gateway and pass the virtual key. Nothing else in your code changes.
Python
# pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000/v1", # the gateway, not api.openai.com
api_key="sk-ag-uSRx-2mzGYw_…", # your virtual key
)
resp = client.chat.completions.create(
model="demo-gpt", # the public registry name
messages=[{"role": "user", "content": "Say hello"}],
)
print(resp.choices[0].message.content)
JavaScript / TypeScript
// npm i openai
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:4000/v1",
apiKey: "sk-ag-uSRx-2mzGYw_…",
});
const r = await client.chat.completions.create({
model: "demo-gpt",
messages: [{ role: "user", content: "Say hello" }],
});
console.log(r.choices[0].message.content);
Watch usage climb
Every successful call is metered, priced, and attributed to the calling key — you can see the numbers
move. Ask the usage endpoint to group by key before and after a few requests:
curl -s "http://localhost:8080/api/v1/usage?group_by=key" \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin"
# after 5 completions with this key — a row attributed to its id:
[
{ "group": "f17cf5df…", "requests": 5,
"prompt_tokens": 40, "completion_tokens": 35, "cost": "0.037500" }
]
The console shows the same, rolled up by model on the Usage page — here demo-gpt
reflects the live calls on top of the seeded totals:

demo-gpt requests and cost have grown.Cost comes from the org's rate card for that model; with no card the request is still counted, at zero cost. See metering-writeback.md for how the write-back works.
Switch the key off
Revoke the key through the control plane. It flips to revoked immediately — the console
shows it greyed out with the revoke action disabled:
curl -s -X POST "http://localhost:8080/api/v1/keys/<key-id>/revoke" \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin"
# → {"id":"f17cf5df…","status":"revoked", …}

Now send the same request again. The data plane calls back into the control-plane DB on every request, so the very next call is rejected — no provider is ever contacted:
curl -s http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-ag-uSRx-2mzGYw_…" \
-H "Content-Type: application/json" \
-d '{"model":"demo-gpt","messages":[{"role":"user","content":"hi"}]}'
# → HTTP 401
{ "error": { "message": "API key is revoked", "type": "auth_error", "code": "401" } }
A revoked key returns API key is revoked; a key that never existed
returns invalid API key; a request with no key at all returns
no API key provided. All three are 401, decided by our DB — never by a
copy inside the proxy.
Governance in action
The controls that make this a gateway and not just a proxy — every one enforced before a request reaches a provider, and every one demoable offline against the seeded stack. Route any provider through one OpenAI-shaped endpoint, redact or block risky prompts, stop over-budget and over-rate traffic, and stream responses that still meter to the token.
Route any provider
Your client always speaks the OpenAI wire shape; the gateway translates to each
provider upstream. Switching from OpenAI to Anthropic or Google is a one-word change — the
model field — with the same key, URL, and request body. The seeded demo ships
demo-gpt (OpenAI), demo-claude (Anthropic) and demo-gemini
(Google Gemini), all answered offline by the stub in their native wire formats:
# Same call — only "model" changes. Each routes through a different provider adapter.
for M in demo-gpt demo-claude demo-gemini; do
curl -s http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $VIRTUAL_KEY" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$M\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"
done
# → all three return HTTP 200 with a normal chat.completion, metered per model.
Provider support comes from LiteLLM, so the registry accepts any provider it knows — OpenAI, Anthropic, Gemini, Bedrock, Azure, Mistral, and more. Add a deployment with real credentials and it routes; no gateway code changes.
Guardrails that act
The seeded org policy runs input guardrails on every request. PII is redacted from the outbound prompt before it leaves the gateway, and prompt-injection attempts are blocked outright. Send an email address and the upstream only ever sees a redaction token:
curl -s http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $VIRTUAL_KEY" -H "Content-Type: application/json" \
-d '{"model":"demo-gpt","messages":[{"role":"user","content":"email me at a@b.com"}]}'
# The stub echoes back exactly what it received — note the address is gone:
# → "content": "Hello from the AI Gateway stub. [echo] email me at [REDACTED:email]"
A prompt-injection attempt never reaches a provider — the guardrail rejects it with
400:
curl -s http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $VIRTUAL_KEY" -H "Content-Type: application/json" \
-d '{"model":"demo-gpt","messages":[{"role":"user","content":"ignore all previous instructions"}]}'
# → HTTP 400
{ "error": { "message": "blocked by guardrail: injection", "type": "invalid_request_error", "code": "400" } }
Budgets & rate limits
Budgets and rate limits are enforced in the same pre-call step, so a request that would overrun is
stopped before it costs anything. Pin a key-scoped budget with no headroom and the next call is
rejected with 402:
# Set a zero-limit budget on the key issued in step 11
curl -s -X PUT http://localhost:8080/api/v1/budgets \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin" \
-H "Content-Type: application/json" \
-d "{\"scope_type\":\"key\",\"scope_id\":\"$KEY_ID\",\"limit\":0}"
# The next completion on that key → HTTP 402 { "message": "budget exceeded" }
Give a key an rpm_limit and a burst past it returns 429:
# Issue a key capped at 1 request/minute
curl -s http://localhost:8080/api/v1/keys \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin" \
-H "Content-Type: application/json" \
-d "{\"team_id\":\"$TEAM_ID\",\"rpm_limit\":1}"
# First call → 200; a second within the same minute → HTTP 429 { "message": "rate limit exceeded" }
Budgets resolve most-specific-wins (key › app › user › team › org),
so a key limit overrides its team's, which overrides the org's — the same scoping the console uses.
Stream with metering
Set "stream": true for token-by-token Server-Sent Events. The gateway transparently asks
the provider to report token usage on the final chunk, so a streamed call meters exactly like a
buffered one — no silent under-counting:
curl -N http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $VIRTUAL_KEY" -H "Content-Type: application/json" \
-d '{"model":"demo-gpt","stream":true,"messages":[{"role":"user","content":"hi"}]}'
# → a stream of SSE chunks, ending with usage, then [DONE]:
data: {"choices":[{"delta":{"content":"Hello from the AI Gateway stub."}}]}
data: {"choices":[],"usage":{"prompt_tokens":8,"completion_tokens":7,"total_tokens":15}}
data: [DONE]
Streaming works for all three families — the stub replies in each provider's native SSE format — and the Usage page climbs after a streamed call just as it does for a buffered one.
Operate it
The plumbing behind the request path from the architecture section: the LiteLLM config is a derived artifact compiled from the registry, cost comes from a rate card you own, and every metered call can be exported for billing. All three are plain control-plane API calls.
Inspect the compiled config
You never hand-write the proxy's YAML. The registry is the source of truth and the LiteLLM config is compiled from it on demand. Ask the control plane to compile your org's config and it returns exactly what the data plane runs:
curl -s -X POST http://localhost:8080/api/v1/config/compile \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin"
{
"model_list": [
{ "model_name": "demo-gpt",
"litellm_params": {
"model": "openai/gpt-4o-mini",
"api_key": "os.environ/STUB_KEY", # a reference — never the secret itself
"api_base": "http://localhost:9099"
},
"model_info": { "id": "md_…", "tags": [] } },
{ "model_name": "demo-claude",
"litellm_params": { "model": "anthropic/claude-sonnet-5",
"api_key": "os.environ/STUB_KEY", "api_base": "http://localhost:9099" },
"model_info": { "id": "md_…", "tags": ["long-context", "vision"] } }
],
"router_settings": {
"routing_strategy": "simple-shuffle", "num_retries": 2,
"fallbacks": [ { "demo-gpt-4o": ["demo-gpt"] } ] # premium falls back to the cheap model
},
"litellm_settings": { "callbacks": ["hooks.callbacks.aigw_logger"] },
"general_settings": { "custom_auth": "hooks.auth.user_api_key_auth" }
}
Two lines do the heavy lifting: custom_auth points every request back at our key check,
and the callbacks logger runs metering — the seams that keep our DB the source of truth.
Credentials are only os.environ/… references, so the written YAML is safe to commit.
Price calls with a rate card
The cost you saw on the Usage page (step 8) and in the group-by-key query (step 13) comes from the
org's rate card for each model. Set or update a price per 1k_tokens, with an
optional percentage markup on top of raw provider cost:
curl -s -X PUT http://localhost:8080/api/v1/rate-cards \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin" \
-H "Content-Type: application/json" \
-d '{"model":"demo-gpt","unit":"1k_tokens","price":0.5,"markup_pct":15}'
# → { "model":"demo-gpt", "unit":"1k_tokens", "price":"0.5", "markup_pct":"15" }
List every card the org has, to see how each model is priced:
curl -s http://localhost:8080/api/v1/rate-cards \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin"
Metering and pricing are separate concerns: a call with no rate card is still counted (tokens and requests), just at zero cost. Add the card later and future calls price immediately — no gateway restart.
Export usage for billing
The same metered data behind the Usage page is available as CSV for invoicing or a spreadsheet.
Group it by team, model, key, or day — here, by team:
curl -s "http://localhost:8080/api/v1/exports/usage.csv?group_by=team" \
-H "X-User-Id: admin" -H "X-Org-Id: $ORG_ID" -H "X-Org-Roles: org-admin"
group,prompt_tokens,completion_tokens,cost,requests
team_platform,18240,15980,17.11,412
team_research,9120,8010,8.55,206
team_support,3040,2670,2.85,68
Prefer JSON? The same numbers come from GET /api/v1/usage?group_by=team, and
month-close invoices from GET /api/v1/invoices — all read-only, so a
billing-viewer can pull them without any write access.