Model Platform
Route LLM traffic to multiple providers through one OpenAI-compatible router.
The Chalk Model Gateway is a managed, OpenAI-compatible router for LLM traffic, hosted alongside your Chalk deployment. Point any OpenAI-compatible client at it and it forwards your request to the right provider (OpenAI, Anthropic, or Google Gemini) behind a single API. Because the Model Gateway sits in front of every provider, it is also where you centralize the things you do not want scattered across application code: API keys and budgets, rate limits, automatic fallback between models, and provider credentials.
The same endpoint also serves the Anthropic Messages API, so Anthropic SDKs and Claude Code route through the Model Gateway without a translation layer.
The Model Gateway runs as a service in your Chalk environment, alongside the engine, and the Chalk API routes each request to it. On a dedicated or self-hosted deployment, running there means your LLM traffic stays inside your own cloud.
The Model Gateway is not deployed by default. Open the Cloud Resource Configuration pane in the Chalk dashboard, add Model Gateway to a resource group from the Add Config menu, where it is listed under Advanced, set its resource request, and choose Save and Apply Service. See Resource Configuration for how services and resource groups work.
Until the service is added, the endpoint below is unavailable in that environment.
The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header.
The Model Gateway exposes the standard OpenAI-compatible paths under this base URL:
| Path | Purpose |
|---|---|
/chat/completions | Chat completions (streams via Server-Sent Events). |
/embeddings | Text embeddings. |
/images/generations | Image generation. |
/models | List the models available to your key. |
The same base URL also serves the Anthropic Messages API. Anthropic SDKs append their own /v1
segment to the base URL they are given, so those paths read with the extra segment:
| Path | Purpose |
|---|---|
/v1/messages | Messages (streams via Server-Sent Events). |
/v1/messages/count_tokens | Exact token count for a request. |
/v1/models | List the models available to your key. |
Use whichever format your client already speaks: the Model Gateway resolves the requested model against your configured providers either way, and a key’s restrictions are enforced on both. See Use with Claude Code for a client configured this way.
If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own
API server host. You can find it in the API Server row of chalk config, or as
apiServer.value in chalk config --format json. Requests are routed to the calling
environment’s Model Gateway, selected by the X-Chalk-Env-Id header described below.
Every request carries an issued Model Gateway API key as a bearer token, plus the environment to route to:
| Header | Value |
|---|---|
Authorization | Bearer <ROUTER_API_KEY> |
X-Chalk-Env-Id | The environment to route the request to |
Model Gateway API keys are issued and revoked from the dashboard or the CLI. See API keys below. A key is shown only once when it is issued, so copy it immediately.
Requests made with an API key are billed to your provider credentials. Treat a key like any other secret: scope it to the connections and models it needs, set a usage budget, and revoke any key you suspect is compromised.
The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header.
from openai import OpenAI
client = OpenAI(
base_url="https://<your-chalk-api-host>/v1/router",
api_key="<YOUR_ROUTER_API_KEY>",
default_headers={"X-Chalk-Env-Id": "<env-id>"},
)
resp = client.chat.completions.create(
model="openai/gpt-5.6-luna",
messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)Model IDs are always prefixed with their provider, as in openai/gpt-5.6-luna or
anthropic/claude-sonnet-5. Run chalk router model list to see every
model your configured providers expose.
The same request with curl:
curl https://<your-chalk-api-host>/v1/router/chat/completions \
-H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \
-H "X-Chalk-Env-Id: <env-id>" \
-H "Content-Type: application/json" \
-d '{"model": "openai/gpt-5.6-luna", "messages": [{"role": "user", "content": "Hello"}]}'The model you request is resolved against the providers you have configured. A key’s connection
scopes, usage pool, rate limits, and usage budgets are all enforced on every request.
An Anthropic client is configured the same way (the base URL, the key, and the environment header) and calls the Messages API:
from anthropic import Anthropic
client = Anthropic(
base_url="https://<your-chalk-api-host>/v1/router",
api_key="<YOUR_ROUTER_API_KEY>",
default_headers={"X-Chalk-Env-Id": "<env-id>"},
)
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
)
print(resp.content[0].text)If a request does not work, two CLI commands report what the Model Gateway sees.
chalk router status shows whether the Model Gateway is reachable and your
harness configuration, and chalk router doctor diagnoses the Model
Gateway, the model you asked for, and the harness together.
Because the Model Gateway is OpenAI-compatible, you can also call it from inside feature computation
with chalk.functions. F.openai_complete issues a chat completion while a
query runs. Route it through the Model Gateway rather than OpenAI in one of two ways:
api_server to the Model Gateway base URL, https://api.chalk.ai/v1/router.OPENAI_BASE_URL environment variable on the execution host and omit api_server.import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame
df = DataFrame({"questions": ["Recommend some movies like High and Low by Akira Kurosawa"]})
(
df.with_columns(
F.openai_complete(
prompt=_.questions,
model="openai/o4-mini",
service_tier="flex",
api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway
).alias("result")
)
.run()
.to_arrow()
)F.openai_complete takes the following arguments:
| Argument | Description |
|---|---|
prompt | The prompt text to send to the model. |
model | The model to use. Defaults to gpt-3.5-turbo when omitted. |
api_server | Base URL of an OpenAI-compatible endpoint; /chat/completions is appended. Falls back to the OPENAI_BASE_URL env var, then to the default OpenAI endpoint. Point it at https://api.chalk.ai/v1/router to use the Model Gateway. |
api_key | API key for authentication. Falls back to the OPENAI_API_KEY env var, so the secret does not have to be threaded through feature data. |
max_tokens | Maximum number of tokens to generate. |
temperature | Sampling temperature between 0 and 2. |
service_tier | Optional OpenAI service tier: "flex" (cheaper, higher-latency), "priority", or "auto". "flex" is only supported on reasoning models (o3, o4-mini, gpt-5-class) and uses a longer request timeout; passing it with an unsupported model returns null. |
It returns a struct with the completion text plus prompt_tokens, completion_tokens,
total_tokens, model, finish_reason, and the upstream ratelimit_remaining_tokens /
ratelimit_remaining_requests headers.
LLM calls are blocking and metered, so throttle them with the policy modifiers chained onto the
expression. with_rate_limit caps how often the call may run across every expression that shares
its key:
import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame
df = DataFrame({"questions": ["Recommend some movies like Buzzard by Joel Potrykus"]})
(
df.with_columns(
F.openai_complete(
prompt=_.questions,
model="openai/o4-mini",
service_tier="flex",
api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway
)
.with_rate_limit(rate=3, key="openai", per="minute")
.alias("result")
)
.run()
.to_arrow()
)The same modifier works on a feature defined in a @features class: chain .with_rate_limit(...)
before selecting the .completion field.
with_rate_limit takes:
| Argument | Description |
|---|---|
rate | Number of calls allowed per window. |
per | Window length: "second" (default), "minute", or "hour". |
key | Bucket name; all expressions sharing a key draw from the same budget. |
enforce_globally | When True, enforce the limit across all workers rather than per-worker. Defaults to False. |
These policy modifiers compose: chain with_concurrency, with_rate_limit, and with_retry on
the same expression to bound in-flight calls, cap the call rate, and retry transient failures with
backoff:
(
F.openai_complete(
prompt=_.questions,
model="openai/o4-mini",
service_tier="flex",
api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway
)
.with_concurrency(max_concurrent=4, key="my_api")
.with_rate_limit(rate=100, key="my_api")
.with_retry(max_retries=3, key="my_api")
)Reusing one key ties the policies to the same logical resource, so every expression that calls
"my_api" shares a single budget: here, at most 4 concurrent calls and 100 calls per second
(per defaults to "second"), with each failed call retried up to 3 times.
Exercise the Model Gateway from the dashboard under Model Gateway → Playground, which
authenticates with your Chalk session rather than a separate API key. It covers chat completions,
structured Decisions requests, embeddings, and image generation. The Decisions tab provides
Text/JSON state editors, Noul/Choice/Score question builders, example presets, answer summaries,
token usage, cost when the provider reports it, and the raw response. Its model picker is populated
from the Model Gateway’s /models endpoint and also accepts a model ID entered manually.
A coding agent points at the Model Gateway the same way any other client does, so the provider credential, connection scopes, budgets and rate limits already configured apply to it.
Claude Code speaks the Anthropic Messages API, so the Model Gateway works as its LLM gateway: model requests route through Chalk, where the provider credential, connection scopes, usage budgets, and rate limits already live, and each developer holds a revocable Model Gateway API key instead of a provider key.
The CLI configures this for you, using the credentials you are already logged in with:
chalk router claude onPass --model <model-id> to pin a model. To start a single session without changing Claude Code’s
configuration at all, use chalk router claude launch instead: it
writes the Model Gateway settings to a temporary file, hands them to Claude Code with
--settings, and removes the file when the session ends, so ~/.claude/settings.json is never
read or rewritten. chalk router claude off restores whatever the CLI changed, and
chalk router claude status shows the current state.
The rest of this section covers the same setup by hand, which is worth reading if you are scripting it or debugging a connection.
Set the base URL and the key in your shell:
export ANTHROPIC_BASE_URL=https://<your-chalk-api-host>/v1/router
export ANTHROPIC_AUTH_TOKEN=<YOUR_ROUTER_API_KEY>
export ANTHROPIC_CUSTOM_HEADERS="X-Chalk-Env-Id: <env-id>"Use ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY: the Model Gateway reads the key from
the Authorization header, and ANTHROPIC_API_KEY sends it in x-api-key. The Model Gateway
accepts either, but ANTHROPIC_API_KEY also needs a one-time approval prompt in Claude Code
before it takes effect.
Confirm the Model Gateway answers before starting Claude Code, so a failure points at the Model Gateway rather than at your configuration:
curl -X POST "$ANTHROPIC_BASE_URL/v1/messages" \
-H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \
-H "X-Chalk-Env-Id: <env-id>" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{"model": "claude-sonnet-4-6", "max_tokens": 1, "messages": [{"role": "user", "content": "."}]}'A body starting with {"id":"msg_ means the URL, key, and environment header all work. An
unknown-model error also confirms them, since the Model Gateway authenticated the request before
rejecting the model.
To make the configuration apply everywhere Claude Code runs, including background agents, move the
values into the env block of ~/.claude/settings.json:
{
"env": {
"ANTHROPIC_BASE_URL": "https://<your-chalk-api-host>/v1/router",
"ANTHROPIC_AUTH_TOKEN": "<YOUR_ROUTER_API_KEY>",
"ANTHROPIC_CUSTOM_HEADERS": "X-Chalk-Env-Id: <env-id>"
}
}Run /status in Claude Code to confirm: the Status tab shows an Anthropic base URL line with
your Chalk host and an Auth token line naming the variable you set.
Two things to know before rolling this out:
ANTHROPIC_AUTH_TOKEN is set, a
saved claude.ai subscription is unused and its limits do not apply; usage bills against the
provider credential the Model Gateway forwards. Features that need a claude.ai identity, such as
Remote Control and voice dictation, are unavailable while it is set.CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 so the
/model picker is populated from the Model Gateway’s /v1/models response.The CLI covers four more harnesses. Each has on, off and status:
| Harness | Command | Also supports |
|---|---|---|
| Codex | chalk router codex | launch to start a session without changing its configuration, and resume <session-id> to continue one whichever provider recorded it |
| Cursor | chalk router cursor | |
| OpenCode | chalk router opencode | |
| VS Code | chalk router vscode |
Cursor is set up by hand. chalk router cursor on prints the settings to apply under Cursor
Settings → Models → OpenAI, and off prints the steps to undo them; neither edits Cursor’s
configuration. For Codex, OpenCode and VS Code, on writes the base URL and key into that
harness’s own configuration and off restores whatever it changed.
Run chalk router status to see every harness at once, including whether it
is installed and whether its routing is currently on.
Routing decides which upstream model serves a request, and what happens when one is unavailable. Configure all of it under Model Gateway → Routing in the Chalk dashboard.
A connection holds the credential for one provider along with the settings that decide which of its models the Model Gateway offers. Create and edit connections under Model Gateway → Routing → Connections.
You can create several connections for the same provider, each with its own credential, base URL, and model prefix. An API key is scoped to the connections it may use; see API keys.
The Model Gateway supports these provider kinds:
| Kind | Notes |
|---|---|
openai | |
anthropic | |
gemini | |
vertex | The stored credential is a GCP service account key rather than a token. |
bedrock | |
vllm | |
voyage | |
cohere | |
mistral | |
qwen | |
xai | |
groq | |
fireworks | |
together | |
openrouter | |
typesafe | |
openai-compatible | Any endpoint that speaks the OpenAI API. Set a base URL. |
Each connection also carries:
/v1/models lists the
connection’s models as <prefix>/<model>./v1/models. Dynamic lists every
model the provider reports. Unlisted keeps those models reachable but hidden, so a caller has
to know the ID already. Explicit serves nothing from the provider’s catalog, so callers reach
models only through a model route. Unspecified means no policy is stored, and
the connection falls back to the default.Credentials apply at runtime, so changing one needs no redeploy.
A model route maps the model ID a caller asks for to a model on a specific connection. Manage routes under Model Gateway → Routing → Routes.
Each route sets:
Give the upstream ID in its bare form, without the connection’s prefix. /v1/models presents a
connection’s models as <prefix>/<model>, and the Model Gateway strips that one leading segment
before dispatching, so a route holding the prefixed form reaches the provider as an unknown model.
An upstream ID may contain slashes of its own, and those segments stay.
Publish a route to make it callable. Each route reports a diagnostic alongside its status: ready, unconfigured, disabled, unreadable, or missing its connection.
A fallback policy keeps requests succeeding when a primary model is unavailable. For a primary model you define an ordered list of fallbacks; if a request against the primary fails, the Model Gateway retries the fallbacks in order. For example:
openai/gpt-5.6-luna → [openai/gpt-4o-mini, anthropic/claude-3-5-sonnet]
If openai/gpt-5.6-luna is unavailable, the Model Gateway tries openai/gpt-4o-mini, then
anthropic/claude-3-5-sonnet, before giving up. Edit fallback rules under Model Gateway →
Routing → Fallback policy.
A judge policy lets a scoring model decide when a request can be served by a cheaper one. The judge model scores each request, and the score selects a rung on a ladder of models. Configure the policy under Model Gateway → Routing → Judge policy, and override it for a single route from the Judge tab on that route’s detail page.
A ladder is the model the caller requested plus the cheaper targets you list. Boundaries divide the 0 to 1 score range into one band per rung. Leave the boundaries empty to split the range into equal bands. Otherwise supply exactly one fewer value than there are rungs, each strictly between 0 and 1 and each larger than the one before it. A target that repeats the requested model is ignored.
Two settings control rollout:
Scope the policy per key when you issue it with chalk router api-key create:
--downgradable: allow this key’s requests to be downgraded.--model-floor: the cheapest model this key may be downgraded to.Keys that set neither flag fall back to the policy’s own defaults for both.
An edit to the policy reaches every replica within about a minute. A judge call that times out, or that arrives while the judge is at its concurrency limit, leaves the request on the model the caller asked for, so a slow or unavailable judge never fails a request.
Access decides who may call the Model Gateway and how much they may spend. Configure all of it under Model Gateway → Access in the Chalk dashboard.
Issue and revoke Model Gateway API keys from Model Gateway → Access → API keys in the Chalk
dashboard, or from the CLI with chalk router api-key (create,
list, and revoke). Scope each key at issue time so that a leaked or over-eager client cannot
do more than you intend:
| Setting | Effect |
|---|---|
| Description | A human-readable label for the key. |
| Connection scopes | The provider connections the key may use. Each scope covers all models on that connection, or the specific models you select. |
| Usage pool | Associates the key with a usage pool, so pool-scoped rate limits apply. |
| Labels | Custom key/value metadata. |
Each key tracks its total token usage, and revoking a key takes effect immediately.
A usage pool groups API keys so you can manage and limit access by pool rather than key by key.
Create a pool, then assign keys to it at issue time. Manage pools from the CLI with
chalk router usage-pool (create, list, and delete). Scope a
rate limit to a pool so every key in it shares one budget. Deleting a
pool leaves its keys working and removes their pool association.
A usage budget caps how many tokens a key or a pool may consume in a fixed window. Create budgets under Model Gateway → Access → Usage budgets.
Each budget sets three things:
You can disable a budget rather than delete it. Each one reports the tokens counted so far in the current window and the time of the next reset. That counter decides admission rather than serving as a usage ledger, so read it as how close a scope is to being cut off, and use Access → Usage for historical accounting.
Rate limit policies cap throughput and protect against runaway spend. Each policy has a limit type, an optional target, and a scope:
Enable or disable a policy without deleting it.
Token-based limits — tokens per minute, and any usage budget — need a conservative token estimate before the Model Gateway forwards a request. A request carrying images, audio or video, encrypted content, or a reference to a stored conversation cannot be estimated. The Model Gateway rejects that request with HTTP 429 without contacting the provider. Send the conversation history inline rather than referencing a stored conversation to avoid this.
The Model Gateway records a trace for each request. Read traces for one key from the Traces tab on that key’s detail page, and choose where the Model Gateway exports them under Model Gateway → Telemetry.
A sampling policy decides which traces the Model Gateway keeps, and whether it records prompt and completion content. Edit it under Model Gateway → Routing → Sampling policy.
The policy is an ordered list of rules. The first rule whose target matches a request supplies both the keep rate and the content setting for that request, and requests matching no rule fall to the policy defaults. A rule matches on one of:
x-chalk-request-tag header the caller sends.Keep rate and content logging are independent. A trace the policy keeps may carry no content, and a request a rule marked for content logging may still be sampled away.
Outcome rules run once the response is known. Each reads a field from the response body and compares it to a value, and the first one that matches replaces the keep rate chosen at request time.
Some behavior applies regardless of which rule matches: