# Model Gateway source: https://docs.chalk.ai/docs/model-gateway ## Route LLM traffic to multiple providers through one OpenAI-compatible gateway. The Chalk Model Gateway is a managed, OpenAI-compatible gateway for LLM traffic, hosted alongside your Chalk deployment. Point any OpenAI-compatible client at it and it forwards your request to the right provider — OpenAI, Anthropic, or Google Gemini — behind a single API. Because the router sits in front of every provider, it is also where you centralize the things you do not want scattered across application code: API keys and budgets, rate limits, automatic fallback between models, and provider credentials. The same endpoint also serves the Anthropic Messages API, so Anthropic SDKs and Claude Code route through the router without a translation layer. The router runs as a service in your Chalk environment, alongside the engine, and the Chalk API routes each request to it. To deploy it, open the Cloud Resource Configuration pane, add LLM Router to a resource group from the Add Config menu, where it is listed under Advanced, set its resource request, and choose Save and Apply Service. See Resource Configuration for how services and resource groups work. On a dedicated or self-hosted deployment, running there means your LLM traffic stays inside your own cloud. ### Endpoint The router exposes the standard OpenAI-compatible paths under this base URL: | Path | Purpose | | --------------------- | -------------------------------------------------- | | `/chat/completions` | Chat completions (streams via Server-Sent Events). | | `/embeddings` | Text embeddings. | | `/images/generations` | Image generation. | | `/models` | List the models available to your key. | The same base URL also serves the Anthropic Messages API. Anthropic SDKs append their own /v1 segment to the base URL they are given, so those paths read with the extra segment: | Path | Purpose | | --------------------------- | ------------------------------------------ | | `/v1/messages` | Messages (streams via Server-Sent Events). | | `/v1/messages/count_tokens` | Exact token count for a request. | | `/v1/models` | List the models available to your key. | Use whichever format your client already speaks — the router resolves the requested model against your configured providers either way, and a key's restrictions are enforced on both. See Use with Claude Code for a client configured this way. If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own API server host. You can find it in the API Server row of chalk config, or as apiServer.value in chalk config --format json. Requests are routed to the calling environment's router, selected by the X-Chalk-Env-Id header described below. ### Authentication Every request carries an issued router API key as a bearer token, plus the environment to route to: | Header | Value | | ---------------- | --------------------------------------- | | `Authorization` | `Bearer ` | | `X-Chalk-Env-Id` | The environment to route the request to | Router API keys are issued and revoked from the dashboard or the CLI — see API keys below. A key is shown only once when it is issued, so copy it immediately. Router API keys spend against your configured providers. Treat them like any other secret, scope them with a model allow-list and daily budget, and revoke any key you suspect is compromised. ### Using the router The router speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header. ``` from openai import OpenAI client = OpenAI( base_url="https:///v1/router", api_key="", default_headers={"X-Chalk-Env-Id": ""}, ) resp = client.chat.completions.create( model="openai/gpt-5.6-luna", messages=[{"role": "user", "content": "Hello"}], ) print(resp.choices[0].message.content) ``` Model IDs are always prefixed with their provider, as in openai/gpt-5.6-luna or anthropic/claude-sonnet-5. Run chalk router model list to see every model your configured providers expose. The same request with curl: ``` curl https:///v1/router/chat/completions \ -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \ -H "X-Chalk-Env-Id: " \ -H "Content-Type: application/json" \ -d '{"model": "openai/gpt-5.6-luna", "messages": [{"role": "user", "content": "Hello"}]}' ``` The model you request is resolved against the providers you have configured. A key's provider restriction, model allow-list, usage pool, and daily token budget are all enforced on every request. An Anthropic client is configured the same way — the base URL, the key, and the environment header — and calls the Messages API: ``` from anthropic import Anthropic client = Anthropic( base_url="https:///v1/router", api_key="", default_headers={"X-Chalk-Env-Id": ""}, ) resp = client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, messages=[{"role": "user", "content": "Hello"}], ) print(resp.content[0].text) ``` If a request does not work, two CLI commands report what the router sees. chalk router status shows router availability and your harness configuration, and chalk router doctor diagnoses the router, the model you asked for, and the harness together. ### Use with Claude Code Claude Code speaks the Anthropic Messages API, so the router works as its LLM gateway: model requests route through Chalk, where the provider credential, model allow-list, daily budget, and rate limits already live, and each developer holds a revocable router key instead of a provider key. The CLI configures this for you, using the credentials you are already logged in with: ``` chalk router claude on ``` Pass --model to pin a model. To start a single session without changing Claude Code's configuration at all, use chalk router claude launch instead: it writes the router settings to a temporary file, hands them to Claude Code with --settings, and removes the file when the session ends, so ~/.claude/settings.json is never read or rewritten. chalk router claude off restores whatever the CLI changed, and chalk router claude status shows the current state. The rest of this section covers the same setup by hand, which is worth reading if you are scripting it or debugging a connection. Set the base URL and the key in your shell: ``` export ANTHROPIC_BASE_URL=https:///v1/router export ANTHROPIC_AUTH_TOKEN= export ANTHROPIC_CUSTOM_HEADERS="X-Chalk-Env-Id: " ``` Use ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY: the router reads the key from the Authorization header, and ANTHROPIC_API_KEY sends it in x-api-key. The router accepts either, but ANTHROPIC_API_KEY also needs a one-time approval prompt in Claude Code before it takes effect. Confirm the router answers before starting Claude Code, so a failure points at the router rather than at your configuration: ``` curl -X POST "$ANTHROPIC_BASE_URL/v1/messages" \ -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \ -H "X-Chalk-Env-Id: " \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model": "claude-sonnet-4-6", "max_tokens": 1, "messages": [{"role": "user", "content": "."}]}' ``` A body starting with {"id":"msg_ means the URL, key, and environment header all work. An unknown-model error also confirms them, since the router authenticated the request before rejecting the model. To make the configuration apply everywhere Claude Code runs, including background agents, move the values into the env block of ~/.claude/settings.json: ``` { "env": { "ANTHROPIC_BASE_URL": "https:///v1/router", "ANTHROPIC_AUTH_TOKEN": "", "ANTHROPIC_CUSTOM_HEADERS": "X-Chalk-Env-Id: " } } ``` Run /status in Claude Code to confirm: the Status tab shows an Anthropic base URL line with your Chalk host and an Auth token line naming the variable you set. Two things to know before rolling this out: - The router key replaces a claude.ai login. While ANTHROPIC_AUTH_TOKEN is set, a saved claude.ai subscription is unused and its limits do not apply; usage bills against the provider credential the router forwards. Features that need a claude.ai identity, such as Remote Control and voice dictation, are unavailable while it is set. - Serve the router's model names. If a router model route exposes a name that is not one of Claude Code's built-in models, set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 so the /model picker is populated from the router's /v1/models response. ### Calling the router from Chalk Because the router is OpenAI-compatible, you can also call it from inside feature computation with chalk.functions. F.openai_complete issues a chat completion while a query runs. To route it through the Model Gateway rather than calling OpenAI directly, set api_server to your Chalk API host — or set the OPENAI_BASE_URL environment variable on the execution host and omit the argument. ``` import chalk.functions as F from chalk.features import _ from chalkdf import DataFrame df = DataFrame({"questions": ["Recommend some movies like High and Low by Akira Kurosawa"]}) ( df.with_columns( F.openai_complete( prompt=_.questions, model="o4-mini", service_tier="flex", api_server="https://api.chalk.ai", # route through the Model Gateway ).alias("result") ) .run() .to_arrow() ) ``` F.openai_complete takes the following arguments: | Argument | Description | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `prompt` | The prompt text to send to the model. | | `model` | The model to use. Defaults to `gpt-3.5-turbo` when omitted. | | `api_server` | Base URL of an OpenAI-compatible endpoint; `/chat/completions` is appended. Falls back to the `OPENAI_BASE_URL` env var, then to the default OpenAI endpoint. Point it at your Chalk API host to use the Model Gateway. | | `api_key` | API key for authentication. Falls back to the `OPENAI_API_KEY` env var, so the secret does not have to be threaded through feature data. | | `max_tokens` | Maximum number of tokens to generate. | | `temperature` | Sampling temperature between 0 and 2. | | `service_tier` | Optional OpenAI service tier — `"flex"` (cheaper, higher-latency), `"priority"`, or `"auto"`. `"flex"` is only supported on reasoning models (`o3`, `o4-mini`, `gpt-5`-class) and uses a longer request timeout; passing it with an unsupported model returns `null`. | It returns a struct with the completion text plus prompt_tokens, completion_tokens, total_tokens, model, finish_reason, and the upstream ratelimit_remaining_tokens / ratelimit_remaining_requests headers. ### Throttling with rate limits LLM calls are blocking and metered, so throttle them with the policy modifiers chained onto the expression. with_rate_limit caps how often the call may run across every expression that shares its key: ``` import chalk.functions as F from chalk.features import _ from chalkdf import DataFrame df = DataFrame({"questions": ["Recommend some movies like Buzzard by Joel Potrykus"]}) ( df.with_columns( F.openai_complete(prompt=_.questions, model="o4-mini", service_tier="flex") .with_rate_limit(rate=3, key="openai", per="minute") .alias("result") ) .run() .to_arrow() ) ``` The same modifier works on a feature defined in a @features class — chain .with_rate_limit(...) before selecting the .completion field. with_rate_limit takes: | Argument | Description | | ------------------ | ---------------------------------------------------------------------------------------------- | | `rate` | Number of calls allowed per window. | | `per` | Window length — `"second"` (default), `"minute"`, or `"hour"`. | | `key` | Bucket name; all expressions sharing a `key` draw from the same budget. | | `enforce_globally` | When `True`, enforce the limit across all workers rather than per-worker. Defaults to `False`. | These policy modifiers compose — chain with_concurrency, with_rate_limit, and with_retry on the same expression to bound in-flight calls, cap the call rate, and retry transient failures with backoff: ``` ( F.openai_complete(prompt=_.questions, model="o4-mini", service_tier="flex") .with_concurrency(max_concurrent=4, key="my_api") .with_rate_limit(rate=100, key="my_api") .with_retry(max_retries=3, key="my_api") ) ``` Reusing one key ties the policies to the same logical resource, so every expression that calls "my_api" shares a single budget — here, at most 4 concurrent calls and 100 calls per second (per defaults to "second"), with each failed call retried up to 3 times. ### API keys Issue and revoke router keys from Model Gateway → Access → API keys in the Chalk dashboard, or from the CLI with chalk router api-key (create, list, and revoke). Each key can be scoped at issue time so that a leaked or over-eager client cannot do more than you intend: | Setting | Effect | | ---------------------- | ------------------------------------------------------------------------------------------------ | | **Description** | A human-readable label for the key. | | **Usage pool** | Associates the key with a [usage pool](#usage-pools), so pool-scoped rate limits apply. | | **Provider** | Restricts the key to a single provider (for example, `openai`). | | **Model allow-list** | Restricts the key to specific models (for example, `openai/gpt-5.6-luna`, `openai/gpt-4o-mini`). | | **Daily token budget** | Caps the tokens the key may consume per day. | | **Labels** | Custom key/value metadata. | | **Cost tags** | Tags used to attribute spend for billing and reporting. | Each key tracks its total token usage, and revoking a key takes effect immediately. ### Usage pools A usage pool groups API keys so you can manage and limit access by pool rather than key by key. Create a pool, then assign keys to it at issue time. Pools are also managed from the CLI with chalk router usage-pool (create, list, and delete). Rate limits can be scoped to a pool so every key in it shares one budget. Deleting a pool leaves its keys working and removes their pool association. ### Rate limits Rate limit policies cap throughput and protect against runaway spend. Each policy has a limit type, an optional target, and a scope: - Limit type: tokens per minute, requests per minute, or concurrent requests. - Target (optional): narrow the policy to a specific provider or model. Leave it unset to apply across all traffic. - Scope: apply the limit per individual key (token) or shared across a usage pool. Policies can be enabled or disabled without deleting them. ### Fallback policies A fallback policy keeps requests succeeding when a primary model is unavailable. For a primary model you define an ordered list of fallbacks; if a request against the primary fails, the router retries the fallbacks in order. For example: ``` openai/gpt-5.6-luna → [openai/gpt-4o-mini, anthropic/claude-3-5-sonnet] ``` If openai/gpt-5.6-luna is unavailable, the router tries openai/gpt-4o-mini, then anthropic/claude-3-5-sonnet, before giving up. Edit fallback rules under Model Gateway → Routing → Fallback policy. ### Providers Configure provider credentials under Model Gateway → Routing → Connections. The router supports: | Provider | ID | | ------------- | ----------- | | OpenAI | `openai` | | Anthropic | `anthropic` | | Google Gemini | `gemini` | For each provider you set an API key and, optionally, a base URL override to point at a custom or OpenAI-compatible endpoint (for example, Gemini's OpenAI-compatible API). Credentials are applied at runtime — no redeploy is required — and each provider shows a connected or not-configured status. ### Playground Model Gateway → Playground is an in-dashboard tester for the router. It authenticates with your Chalk session — no separate API key needed — and lets you exercise the router across three tabs: - Chat: send streaming chat completions with a configurable system prompt, temperature, and max tokens. - Embeddings: generate embeddings and inspect their dimensions and token counts. - Images: generate images with configurable size, quality, and count. The model picker is populated from the router's /models endpoint, so it reflects the providers you have configured. ### See also - Authentication — service credentials and RBAC for the Chalk API. - MCP Gateway — govern the external MCP servers your agents reach. - LLM Toolchain — building AI features and agents on Chalk.