Model Gateway

The Chalk Model Gateway routes LLM requests to OpenAI, Anthropic, and Google Gemini through one OpenAI-compatible API. Manage provider credentials, API keys, budgets, rate limits, and fallback policies in the Chalk dashboard.

The same endpoint also serves the Anthropic Messages API, so Anthropic SDKs and Claude Code route through the Model Gateway without a translation layer.

The Model Gateway runs in your Chalk environment. On a dedicated or self-hosted deployment, it uses your own cloud infrastructure.

Getting started

Open Model Gateway → Routing → Connections in the Chalk dashboard and choose Add connection. Enter your provider credentials and save the connection.
Open Model Gateway → Playground, select a model, and send a prompt.

Chalk sets up your gateway automatically when you add your first connection. The playground uses your Chalk session, so you can try it immediately.

To use the gateway from your application, create an API key and follow Making a request below.

Sending requests

The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header.

Endpoint

POST

https://api.chalk.ai/v1/router

The Model Gateway exposes the standard OpenAI-compatible paths under this base URL:

Path	Purpose
`/chat/completions`	Chat completions (streams via Server-Sent Events).
`/responses`	OpenAI Responses creation (streaming or non-streaming).
`/embeddings`	Text embeddings.
`/moderations`	Text moderation.
`/images/generations`	Image generation.
`/models`	List the models available to your key.

The same base URL also serves the Anthropic Messages API. Anthropic SDKs append their own /v1 segment to the base URL they are given, so those paths read with the extra segment:

Path	Purpose
`/v1/messages`	Messages (streams via Server-Sent Events).
`/v1/messages/count_tokens`	Exact token count for a request.
`/v1/models`	List the models available to your key.

The same base URL also serves the native Gemini Interactions API, preserving its /v1beta segment:

Path	Purpose
`/v1beta/interactions`	Create a Gemini interaction with `POST`, with or without native Server-Sent Events streaming.

The full endpoint is https://api.chalk.ai/v1/router/v1beta/interactions. Use a configured Gemini connection prefix or a published route alias in model, and provide the native input field. Set stream: true for streaming. Agent/background jobs and interaction retrieval, deletion, and cancellation are not supported and return HTTP 501.

The same base URL also serves the Decisions and System One APIs:

Path	Purpose
`/alpha/decisions`	Submit a Decisions request with `POST`.
`/v1/systemone`	Submit a System One request with `POST`.

The full endpoints are https://api.chalk.ai/v1/router/alpha/decisions and https://api.chalk.ai/v1/router/v1/systemone. Both accept model, state, and questions in the JSON request body. Select a model on a connection that supports the corresponding API. The gateway preserves System One’s explicit /v1 segment and adds /v1 when forwarding /alpha/decisions to the router.

Use whichever supported format your client speaks: the Model Gateway resolves the requested model against your configured providers and enforces the key’s restrictions across these APIs. See Use with Claude Code for an Anthropic client configuration.

If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own API server host. You can find it in the API Server row of chalk config, or as apiServer.value in chalk config --format json. Requests are routed to the calling environment’s Model Gateway, selected by the X-Chalk-Env-Id header described below.

Text moderation

Call client.moderations.create to classify a string or an array of strings:

result = client.moderations.create(
    model="openai/omni-moderation-latest",
    input="Text to check",
)
print(result.results[0].flagged)

Use your connection’s configured prefix in the model name. If you omit model, the gateway resolves omni-moderation-latest through the configured routes and OpenAI connection. The resolved connection and model must be permitted by your API key. The endpoint forwards the provider’s verdict and returns provider errors when moderation fails.

Moderation dispatches once to the resolved model. It does not use chat fallback, the judge, or prompt caching. Request-rate and concurrency limits apply, including limits scoped to the connection and model. Moderation does not report token usage, so token budgets and tokens-per-minute limits do not apply. Successful calls appear in usage tracking as one request with zero tokens. This endpoint accepts text input only.

Authentication

Every request carries an issued Model Gateway API key as a bearer token, plus the environment to route to:

Header	Value
`Authorization`	`Bearer <ROUTER_API_KEY>`
`X-Chalk-Env-Id`	The environment to route the request to

Model Gateway API keys are issued and revoked from the dashboard or the CLI. See API keys below. A key is shown only once when it is issued, so copy it immediately.

Treat API keys as secrets

Requests made with an API key are billed to your provider credentials. Treat a key like any other secret: scope it to the connections and models it needs, set a usage budget, and revoke any key you suspect is compromised.

Making a request

The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header.

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-chalk-api-host>/v1/router",
    api_key="<YOUR_ROUTER_API_KEY>",
    default_headers={"X-Chalk-Env-Id": "<env-id>"},
)

resp = client.chat.completions.create(
    model="openai/gpt-5.6-luna",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Use a connection-prefixed model ID, such as openai/gpt-5.6-luna or anthropic/claude-sonnet-5, or a published model route such as chalk/assistant. Run chalk router model list to see every model your configured providers expose.

The same request with curl:

curl https://<your-chalk-api-host>/v1/router/chat/completions \
  -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \
  -H "X-Chalk-Env-Id: <env-id>" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-5.6-luna", "messages": [{"role": "user", "content": "Hello"}]}'

The model you request is resolved against the providers you have configured. A key’s connection scopes, usage pool, rate limits, and usage budgets are all enforced on every request.

An Anthropic client is configured the same way (the base URL, the key, and the environment header) and calls the Messages API:

from anthropic import Anthropic

client = Anthropic(
    base_url="https://<your-chalk-api-host>/v1/router",
    api_key="<YOUR_ROUTER_API_KEY>",
    default_headers={"X-Chalk-Env-Id": "<env-id>"},
)

resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.content[0].text)

If a request does not work, two CLI commands report what the Model Gateway sees. chalk router status shows whether the Model Gateway is reachable and your harness configuration, and chalk router doctor diagnoses the Model Gateway, the model you asked for, and the harness together.

Calling the Model Gateway from Chalk

Because the Model Gateway is OpenAI-compatible, you can also call it from inside feature computation with chalk.functions. F.openai_complete issues a chat completion while a query runs. Route it through the Model Gateway rather than OpenAI in one of two ways:

Set api_server to the Model Gateway base URL, https://api.chalk.ai/v1/router.
Set the OPENAI_BASE_URL environment variable on the execution host and omit api_server.

import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame

df = DataFrame({"questions": ["Recommend some movies like High and Low by Akira Kurosawa"]})
(
    df.with_columns(
        F.openai_complete(
            prompt=_.questions,
            model="openai/o4-mini",
            service_tier="flex",
            api_server="https://api.chalk.ai/v1/router",  # route through the Model Gateway
        ).alias("result")
    )
    .run()
    .to_arrow()
)

F.openai_complete takes the following arguments:

Argument	Description
`prompt`	The prompt text to send to the model.
`model`	The model to use. Defaults to `gpt-3.5-turbo` when omitted.
`api_server`	Base URL of an OpenAI-compatible endpoint; `/chat/completions` is appended. Falls back to the `OPENAI_BASE_URL` env var, then to the default OpenAI endpoint. Point it at `https://api.chalk.ai/v1/router` to use the Model Gateway.
`api_key`	API key for authentication. Falls back to the `OPENAI_API_KEY` env var, so the secret does not have to be threaded through feature data.
`max_tokens`	Maximum number of tokens to generate.
`temperature`	Sampling temperature between 0 and 2.
`service_tier`	Optional OpenAI service tier: `"flex"` (cheaper, higher-latency), `"priority"`, or `"auto"`. `"flex"` is only supported on reasoning models (`o3`, `o4-mini`, `gpt-5`-class) and uses a longer request timeout; passing it with an unsupported model returns `null`.

It returns a struct with the completion text plus prompt_tokens, completion_tokens, total_tokens, model, finish_reason, and the upstream ratelimit_remaining_tokens / ratelimit_remaining_requests headers.

Consuming rate limits from a resolver

LLM calls are blocking and metered, so throttle them with the policy modifiers chained onto the expression. with_rate_limit caps how often the call may run across every expression that shares its key:

import chalk.functions as F
from chalk.features import _
from chalkdf import DataFrame

df = DataFrame({"questions": ["Recommend some movies like Buzzard by Joel Potrykus"]})
(
    df.with_columns(
        F.openai_complete(
            prompt=_.questions,
            model="openai/o4-mini",
            service_tier="flex",
            api_server="https://api.chalk.ai/v1/router",  # route through the Model Gateway
        )
        .with_rate_limit(rate=3, key="openai", per="minute")
        .alias("result")
    )
    .run()
    .to_arrow()
)

The same modifier works on a feature defined in a @features class: chain .with_rate_limit(...) before selecting the .completion field.

with_rate_limit takes:

Argument	Description
`rate`	Number of calls allowed per window.
`per`	Window length: `"second"` (default), `"minute"`, or `"hour"`.
`key`	Bucket name; all expressions sharing a `key` draw from the same budget.
`enforce_globally`	When `True`, enforce the limit across all workers rather than per-worker. Defaults to `False`.

These policy modifiers compose: chain with_concurrency, with_rate_limit, and with_retry on the same expression to bound in-flight calls, cap the call rate, and retry transient failures with backoff:

(
    F.openai_complete(
        prompt=_.questions,
        model="openai/o4-mini",
        service_tier="flex",
        api_server="https://api.chalk.ai/v1/router",  # route through the Model Gateway
    )
    .with_concurrency(max_concurrent=4, key="my_api")
    .with_rate_limit(rate=100, key="my_api")
    .with_retry(max_retries=3, key="my_api")
)

Reusing one key ties the policies to the same logical resource, so every expression that calls "my_api" shares a single budget: here, at most 4 concurrent calls and 100 calls per second (per defaults to "second"), with each failed call retried up to 3 times.

Playground

Exercise the Model Gateway from the dashboard under Model Gateway → Playground, which authenticates with your Chalk session rather than a separate API key. It covers chat completions, structured Decisions requests, embeddings, and image generation. The Decisions tab provides Text/JSON state editors, Noul/Choice/Score question builders, example presets, answer summaries, token usage, cost when the provider reports it, and the raw response. Its model picker is populated from the Model Gateway’s /models endpoint and also accepts a model ID entered manually.

Coding agents

A coding agent points at the Model Gateway the same way any other client does, so the provider credential, connection scopes, budgets and rate limits already configured apply to it.

Claude Code

Claude Code speaks the Anthropic Messages API, so the Model Gateway works as its LLM gateway: model requests route through Chalk, where the provider credential, connection scopes, usage budgets, and rate limits already live, and each developer holds a revocable Model Gateway API key instead of a provider key.

The CLI configures this for you, using the credentials you are already logged in with:

chalk router claude on

Pass --model <model-id> to pin a model. To start a single session without changing Claude Code’s configuration at all, use chalk router claude launch instead: it writes the Model Gateway settings to a temporary file, hands them to Claude Code with --settings, and removes the file when the session ends, so ~/.claude/settings.json is never read or rewritten. chalk router claude off restores whatever the CLI changed, and chalk router claude status shows the current state.

The rest of this section covers the same setup by hand, which is worth reading if you are scripting it or debugging a connection.

Set the base URL and the key in your shell:

export ANTHROPIC_BASE_URL=https://<your-chalk-api-host>/v1/router
export ANTHROPIC_AUTH_TOKEN=<YOUR_ROUTER_API_KEY>
export ANTHROPIC_CUSTOM_HEADERS="X-Chalk-Env-Id: <env-id>"

Use ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY: the Model Gateway reads the key from the Authorization header, and ANTHROPIC_API_KEY sends it in x-api-key. The Model Gateway accepts either, but ANTHROPIC_API_KEY also needs a one-time approval prompt in Claude Code before it takes effect.

Confirm the Model Gateway answers before starting Claude Code, so a failure points at the Model Gateway rather than at your configuration:

curl -X POST "$ANTHROPIC_BASE_URL/v1/messages" \
  -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \
  -H "X-Chalk-Env-Id: <env-id>" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-4-6", "max_tokens": 1, "messages": [{"role": "user", "content": "."}]}'

A body starting with {"id":"msg_ means the URL, key, and environment header all work. An unknown-model error also confirms them, since the Model Gateway authenticated the request before rejecting the model.

To make the configuration apply everywhere Claude Code runs, including background agents, move the values into the env block of ~/.claude/settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://<your-chalk-api-host>/v1/router",
    "ANTHROPIC_AUTH_TOKEN": "<YOUR_ROUTER_API_KEY>",
    "ANTHROPIC_CUSTOM_HEADERS": "X-Chalk-Env-Id: <env-id>"
  }
}

Run /status in Claude Code to confirm: the Status tab shows an Anthropic base URL line with your Chalk host and an Auth token line naming the variable you set.

Two things to know before rolling this out:

The Model Gateway API key replaces a claude.ai login. While ANTHROPIC_AUTH_TOKEN is set, a saved claude.ai subscription is unused and its limits do not apply; usage bills against the provider credential the Model Gateway forwards. Features that need a claude.ai identity, such as Remote Control and voice dictation, are unavailable while it is set.
Serve the Model Gateway’s model names. If a model route exposes a name that is not one of Claude Code’s built-in models, set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 so the /model picker is populated from the Model Gateway’s /v1/models response.

Other harnesses

The CLI covers four more harnesses. Each has on, off and status:

Harness	Command	Also supports
Codex	`chalk router codex`	`launch` to start a session without changing its configuration, and `resume <session-id>` to continue one whichever provider recorded it
Cursor	`chalk router cursor`
OpenCode	`chalk router opencode`
VS Code	`chalk router vscode`

Cursor is set up by hand. chalk router cursor on prints the settings to apply under Cursor Settings → Models → OpenAI, and off prints the steps to undo them; neither edits Cursor’s configuration. For Codex, OpenCode and VS Code, on writes the base URL and key into that harness’s own configuration and off restores whatever it changed.

Run chalk router status to see every harness at once, including whether it is installed and whether its routing is currently on.

Routing

Routing decides which upstream model serves a request, and what happens when one is unavailable. Configure all of it under Model Gateway → Routing in the Chalk dashboard.

Provider connections

A connection holds the credential for one provider along with the settings that decide which of its models the Model Gateway offers. Create and edit connections under Model Gateway → Routing → Connections.

You can create several connections for the same provider, each with its own credential, base URL, and model prefix. An API key is scoped to the connections it may use; see API keys.

The Model Gateway supports these provider kinds:

Kind	Notes
`openai`
`anthropic`
`gemini`
`vertex`	The stored credential is a GCP service account key rather than a token.
`bedrock`
`vllm`
`voyage`
`cohere`
`mistral`
`qwen`
`xai`
`groq`
`fireworks`
`together`
`openrouter`
`typesafe`
`openai-compatible`	Any endpoint that speaks the OpenAI API. Set a base URL.

Each connection also carries:

Prefix: the segment the Model Gateway puts in front of a model ID, so /v1/models lists the connection’s models as <prefix>/<model>.
Exposure: whether the connection’s models appear in /v1/models. Dynamic lists every model the provider reports. Unlisted keeps those models reachable but hidden, so a caller has to know the ID already. Explicit serves nothing from the provider’s catalog, so callers reach models only through a model route. Unspecified means no policy is stored, and the connection falls back to the default.
State: whether the credential is configured, readable, and healthy.

Credentials apply at runtime, so changing one needs no redeploy.

Model routes

A model route gives callers a public model ID beginning with chalk\, such as chalk/assistant, and controls which connection and upstream model serve it. Manage routes under Model Gateway → Routing → Routes. Choose one of two routing behaviors when creating or editing a route:

Simple: use one provider connection and upstream model.
Custom: set a default connection and upstream model, then optionally add fallback chains and/or a judge policy in the route’s decision grid.

The custom route’s default target serves requests when judging is off, in shadow mode, or cannot produce a usable classification. That target can have its own fallback chain. A custom route can also have fallback chains without a judge policy.

Give each upstream model ID exactly as the provider expects it, without the connection’s routing prefix. For example, select the OpenAI connection and upstream model gpt-4o. An upstream model ID may contain slashes of its own. All judge targets, outcome models, and fallbacks name concrete connections and upstream models; another route alias cannot be a target.

Publish a route to make it callable. Each route reports a diagnostic alongside its status: ready, unconfigured, disabled, unreadable, or missing its connection. Editing one route’s policy affects requests to that route; in-flight requests retain the configuration they started with.

Fallback policies

Configure fallback chains inside a custom route. Each chain identifies a selected connection and model, followed by an ordered list of alternatives. For example, a route could select gpt-4o on your OpenAI connection and fall back to gpt-5.6-luna on that connection, then claude-sonnet-4-6 on your Anthropic connection.

Each judge outcome can also choose from the following fallback options: Use selected model’s route chain, Stop after selected model, or Use outcome-specific chain. This lets two outcomes select the same model with different recovery behavior. The Model Gateway chooses one chain using this precedence:

A matching request-specific fallback entry, for Chat Completions.
The selected judge outcome’s explicitly configured fallback list.
The route’s fallback entry for the selected model.
No alternatives.

An omitted outcome list inherits the route’s chain. An explicit empty list disables fallback for that selection. Lists replace one another; they are not combined. When judging selects an outcome that uses the default target, that outcome’s fallback setting still applies.

The Model Gateway chooses the chain once and tries its alternatives in order. Reaching an alternative does not expand that model’s chain or run the judge again. Exhausting an outcome or request chain does not resume the route’s chain. Each list can contain at most 16 alternatives, with no repeated target or entry equal to the starting model. Each route or request fallback map can contain at most 128 entries.

Request-specific fallbacks

Chat Completions accepts a fallbacks object mapping selected models to lists of alternatives. This overrides fallback behavior for that request without changing the stored route. The judge policy itself remains configured on the route.

Keys and alternatives use connection_id/upstream_model_id or connection_name/upstream_model_id. Connection names match exactly, including case; they are separate from routing prefixes. If both an ID and a name key identify the selected model, the ID-keyed entry wins. Connection IDs remain stable when you rename a connection.

For a route whose judge can select gpt-4o on the connection named openai-prod, this request replaces that selection’s alternatives with gpt-5.6-luna:

curl 'https://<your-chalk-api-host>/v1/router/chat/completions' \
  -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \
  -H 'X-Chalk-Env-Id: <env-id>' \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON'
{
  "model": "chalk/assistant",
  "messages": [{"role": "user", "content": "What is water in five words?"}],
  "fallbacks": {
    "openai-prod/gpt-4o": ["openai-prod/gpt-5.6-luna"]
  }
}
JSON

Replace the host, environment, route alias, and connection name with your configuration. If the judge selects a different model, this entry does not match, so the outcome or route configuration applies. Valid unmatched entries do not affect routing. The entire fallback map must pass validation, including entries that are not selected; an invalid entry returns HTTP 400. Setting "openai-prod/gpt-4o": [] instead explicitly disables fallback when that target is selected. Request overrides also work without a judge, using the initially resolved target as the lookup key. These request overrides are supported only on Chat Completions.

Failures and limits

Fallback runs for upstream connection failures and timeouts, HTTP 429, and HTTP 5xx responses. A candidate’s connection or model rate limit can also cause the next candidate to be tried. Other upstream HTTP errors, unreadable responses, refusals, and answer-quality problems do not trigger fallback. Once the Model Gateway commits a streaming response, a subsequent error ends that stream without switching models. For background Responses requests, an uncertain dispatch failure stops retries because the upstream job may already be running.

Access to a custom chalk/* route authorizes its judge-selected model and route-configured fallbacks under that route’s public model ID. Each destination must still satisfy the key’s provider and connection restrictions and the request’s capability requirements. Request-supplied alternatives require direct-model access to each destination and cannot use route-only (Explicit) connections. Model-scoped rate limits use the requested route ID for route-owned destinations and the direct, connection-prefixed model ID for request-supplied alternatives.

Missing, disabled, unauthorized, or incompatible candidates are skipped. Admission to the requested route must still succeed before the policy can run. Shared key or usage-pool quota exhaustion stops the whole request; fallback cannot bypass it. The Model Gateway reserves quota once for the generation plan using the most conservative eligible candidate estimate and checks each attempted connection’s limits separately.

The router stops starting additional fallback attempts after 300 seconds. This limit does not cancel an in-progress foreground request. Background Responses submissions are bounded by the deadline; the accepted background job can continue running. Other endpoint and provider timeouts still apply.

Route fallbacks apply to Chat Completions, Messages, Responses creation, and Decisions/System One. Gemini Interactions supports route fallback only when you explicitly set store: false and send a fresh request without connection-bound history, such as previous_interaction_id, cached content, thought signatures, or tool-history items. Omitting store disables fallback, even for a new interaction. This restriction avoids retrying a potentially stored interaction on another connection after an uncertain failure. Operations such as Messages token counting and Responses retrieval or compaction do not walk fallback chains.

Judge policy

A route’s judge classifies a request and selects a configured outcome. In the route editor, enable Judge policy, choose a backend and mode, then configure each outcome’s model and fallback behavior in the Decision grid. Models are explicit choices; the Model Gateway does not infer a price or capability ordering, or restrict a choice to a downgrade.

Classification and outcomes

Choose one of two classification types:

Classification	Default outcomes	Meaning
Difficulty	`0`, `1`	`0` covers straightforward retrieval, rewriting, summaries, extraction, and small self-contained changes. `1` covers sustained reasoning, ambiguity, complex debugging, mathematics, and system design.
Task	`coding`, `summarization`, `extraction`, `other`	Choose by the main deliverable: code or technical guidance, a condensed account of source material, specific information from a source, or another kind of request.

The editor supplies descriptions for these categories and starts new outcomes at the route’s default target with inherited fallbacks. Assign a connection and model to each outcome. Remote judges use the descriptions to distinguish categories; customize them for your workload. Existing descriptions are preserved unless you edit them or choose Restore defaults. Restoring defaults resets categories and descriptions while preserving models and fallbacks for matching labels.

Remote judges support custom task labels or a difficulty scale with consecutive integers starting at 0, ordered from easiest to hardest. Each judge supports up to 32 outcomes. Blank descriptions for standard task labels or the two-level difficulty scale use the default criteria. Supply your own criteria for custom labels or a larger difficulty scale.

Judge backends

Backend	Configuration
Local heuristic	Uses text rules without a model call. Requires difficulty outcomes `0` and `1`.
Completion model	Select a connection and upstream model that support completion-based classification, plus a timeout.
Decisions model	Select a connection and upstream model that support Decisions-based classification, plus a timeout.

The editor checks remote judge compatibility before saving. A judge’s backend determines how classification runs; it does not change the endpoint format used by callers.

For Local heuristic, add or remove hard-task keywords, choose whether code blocks count as hard, and set the easy-input length cutoff. Keywords match user text as case-insensitive substrings. Code fences in any message role count as hard by default. The length rule counts UTF-8 text bytes across message roles; the default cutoff is 6,000 bytes, and 0 disables that rule. Any enabled hard signal selects 1; otherwise the result is 0. Outcome descriptions are retained but do not affect heuristic classification.

Remote backend timeouts are in milliseconds, from 1 to 30,000. An omitted timeout or 0 uses the 1,000 ms default. If classification times out, is unavailable, or returns a malformed or unknown label, the route uses its default target and that target’s applicable fallback chain. Remote judging consumes token budgets and is subject to the judge connection’s limits, with model-scoped limits evaluated against the requested route ID. Shared token-budget exhaustion can reject a live request instead of proceeding to generation. If the remote judge’s concurrency limit is reached, classification is skipped and the request uses the default target and its applicable fallbacks.

Modes and endpoint support

Mode	Serving behavior
Off	Uses the default target and its applicable fallbacks without classifying.
Shadow	Serves through the default target and its applicable fallbacks. After serving admission succeeds, classifies asynchronously and records the proposed outcome, model, and chain without sending generation requests to those proposed targets.
Live	Classifies before the first generation attempt and uses the selected outcome and its applicable fallbacks.

Remote shadow classification still consumes judge tokens. To inspect a recorded decision in exported traces, look for chalk.router.judge.classification, chalk.router.judge.selected_model, chalk.router.judge.fallbacks, and chalk.router.judge.shadow. These follow your trace export and sampling settings.

Judging applies to Chat Completions, Anthropic Messages, and self-contained OpenAI Responses creation, including streaming requests. Responses requests using previous_response_id, conversation, or a stored prompt use the default target because the judge does not fetch that external context. Their fallback eligibility remains subject to connection affinity. Other operations, including Decisions and Gemini Interactions, do not invoke a route judge.

Successful remote classifications are cached for 60 seconds. Local heuristic classifications are computed on every request. The cache distinguishes the request endpoint, key, request content, and judge configuration and backend. Changing the judge configuration invalidates cached classifications. Cached labels are applied to the current request’s routing configuration, so a change to the route fallback map can reuse a classification while applying the updated chain. Remote judges receive a bounded text view of the request; classification does not provide multimodal understanding of embedded images, audio, or video.

Access

Access decides who may call the Model Gateway and how much they may spend. Configure all of it under Model Gateway → Access in the Chalk dashboard.

API keys

Issue and revoke Model Gateway API keys from Model Gateway → Access → API keys in the Chalk dashboard, or from the CLI with chalk router api-key (create, list, and revoke). Scope each key at issue time so that a leaked or over-eager client cannot do more than you intend:

Setting	Effect
Description	A human-readable label for the key.
Connection scopes	The provider connections the key may use. Each scope covers all models on that connection, or the specific models you select.
Usage pool	Associates the key with a usage pool, so pool-scoped rate limits apply.
Labels	Custom key/value metadata.

Each key tracks its total token usage, and revoking a key takes effect immediately.

Usage pools

A usage pool groups API keys so you can manage and limit access by pool rather than key by key. Create a pool, then assign keys to it at issue time. Manage pools from the CLI with chalk router usage-pool (create, list, and delete). Scope a rate limit to a pool so every key in it shares one budget. Deleting a pool leaves its keys working and removes their pool association.

Usage budgets

A usage budget caps how many tokens a key or a pool may consume in a fixed window. Create budgets under Model Gateway → Access → Usage budgets.

Each budget sets three things:

Scope: an individual API key, or a usage pool.
Token limit: the tokens allowed in one window.
Period: daily, weekly, or monthly.

You can disable a budget rather than delete it. Each one reports the tokens counted so far in the current window and the time of the next reset. That counter decides admission rather than serving as a usage ledger, so read it as how close a scope is to being cut off, and use Access → Usage for historical accounting.

Configuring rate limits

Rate limit policies cap throughput and protect against runaway spend. Each policy has a limit type, an optional target, and a scope:

Limit type: tokens per minute, requests per minute, or concurrent requests.
Target (optional): narrow the policy to a specific provider or model. Leave it unset to apply across all traffic.
Scope: apply the limit per individual key (token) or shared across a usage pool.

Enable or disable a policy without deleting it.

Token-based limits — tokens per minute, and any usage budget — need a conservative token estimate before the Model Gateway forwards a request. A request carrying images, audio or video, encrypted content, or a reference to a stored conversation cannot be estimated. The Model Gateway rejects that request with HTTP 429 without contacting the provider. Send the conversation history inline rather than referencing a stored conversation to avoid this.

Observability

The Model Gateway records a trace for each request. Read traces for one key from the Traces tab on that key’s detail page, and choose where the Model Gateway exports them under Model Gateway → Telemetry.

Sampling policy

A sampling policy decides which traces the Model Gateway keeps, and whether it records prompt and completion content. Edit it under Model Gateway → Routing → Sampling policy.

The policy is an ordered list of rules. The first rule whose target matches a request supplies both the keep rate and the content setting for that request, and requests matching no rule fall to the policy defaults. A rule matches on one of:

The x-chalk-request-tag header the caller sends.
The calling key.
The calling key’s usage pool.

Keep rate and content logging are independent. A trace the policy keeps may carry no content, and a request a rule marked for content logging may still be sampled away.

Outcome rules run once the response is known. Each reads a field from the response body and compares it to a value, and the first one that matches replaces the keep rate chosen at request time.

Some behavior applies regardless of which rule matches:

A request that returns an error is always exported.
Conversation rate floor sets a keep rate, between 0 and 1, that a session-keyed trace gets whatever the rules decided.
A deployment-wide setting turns content logging off entirely, overriding every rule that would record it.

Getting started

Sending requests

Endpoint

Text moderation

Authentication

Making a request

Calling the Model Gateway from Chalk

Consuming rate limits from a resolver

Playground

Coding agents

Claude Code

Other harnesses

Routing

Provider connections

Model routes

Fallback policies

Request-specific fallbacks

Failures and limits

Judge policy

Classification and outcomes

Judge backends

Modes and endpoint support

Access

API keys

Usage pools

Usage budgets

Configuring rate limits

Observability

Sampling policy

See also

On this page

​Getting started

​Sending requests

​Endpoint

​Text moderation

​Authentication

​Making a request

​Calling the Model Gateway from Chalk

​Consuming rate limits from a resolver

​Playground

​Coding agents

​Claude Code

​Other harnesses

​Routing

​Provider connections

​Model routes

​Fallback policies

​Request-specific fallbacks

​Failures and limits

​Judge policy

​Classification and outcomes

​Judge backends

​Modes and endpoint support

​Access

​API keys

​Usage pools

​Usage budgets

​Configuring rate limits

​Observability

​Sampling policy

​See also

On this page

Getting started

Sending requests

Endpoint

Text moderation

Authentication

Making a request

Calling the Model Gateway from Chalk

Consuming rate limits from a resolver

Playground

Coding agents

Claude Code

Other harnesses

Routing

Provider connections

Model routes

Fallback policies

Request-specific fallbacks

Failures and limits

Judge policy

Classification and outcomes

Judge backends

Modes and endpoint support

Access

API keys

Usage pools

Usage budgets

Configuring rate limits

Observability

Sampling policy

See also