Running Your Own Open-Source Models

​

Summary

The entrypoint is vLLM’s own OpenAI server, unmodified. Chalk supplies the GPU, the container, and the URL — every flag after api_server is vLLM’s, so swap --model to serve any other HuggingFace model.

from chalkcompute import ScalingGroup, Secret

sg = ScalingGroup(
    image="vllm/vllm-openai:latest",
    name="gemma-vllm",
    gpu="nvidia-l4:1",
    port=8000,
    entrypoint=[
        "python3", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "google/gemma-3-4b-it",
        "--port", "8000",
    ],
    secrets=[Secret.from_chalk_env("VLLM_API_KEY")],
).deploy()

print(sg.web_url)

• gpu="nvidia-l4:1" is the "<type>:<count>" GPU request. Without it, no nvidia.com/gpu request is emitted and your replicas land on CPU nodes, where the CUDA-built vLLM image won’t start.
• web_url is public and unauthenticated by default. Passing VLLM_API_KEY makes vLLM require it as a bearer token on the /v1 API — but not on every path. See Securing the endpoint.

​

Paved path

Deploy the upstream vllm/vllm-openai:latest image as a Scaling Group, override the entrypoint with vLLM’s own OpenAI API server, and request the GPU through the gpu argument. Do not build a custom image and do not write a serving wrapper — vLLM already ships the server you want, and the entrypoint override is how you configure it.

All four decisions belong in the first deploy:

Secret.from_chalk_env refers to a Chalk-level secret — a runtime environment variable you manage under Integrations → Secrets & Variables in the Chalk dashboard. Chalk injects it into every replica at deploy time.

Skipping the last one leaves a public, unauthenticated GPU endpoint on the internet. See Securing the endpoint.

​

Deploy the model

Create deploy_vllm.py:

from chalkcompute import ScalingGroup, Secret

sg = ScalingGroup(
    image="vllm/vllm-openai:latest",
    name="gemma-vllm",
    gpu="nvidia-l4:1",
    cpu="4",
    memory="16Gi",
    port=8000,
    entrypoint=[
        "python3", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "google/gemma-3-4b-it",
        "--port", "8000",
        "--max-model-len", "4096",
        "--dtype", "bfloat16",
        "--gpu-memory-utilization", "0.90",
    ],
    secrets=[
        Secret.from_local_env("HF_TOKEN"),
        Secret.from_chalk_env("VLLM_API_KEY"),
    ],
)

sg.deploy(ready_timeout=600.0)
print(sg.web_url)

$ export HF_TOKEN=hf_...
$ uv run python deploy_vllm.py

deploy() creates the group and waits for it to report ready. GPU nodes take longer to provision than CPU nodes, so ready_timeout is raised from its 300-second default.

deploy() returns once a replica is ready, but this deploy sets no readiness_probe, so “ready” means the container started, not that vLLM finished loading weights. Poll {web_url}/health before sending your first request.

​

Requesting a GPU

gpu takes a "<type>:<count>" string. The type selects the node pool, and the count sets the per-replica GPU request and limit. "nvidia-l4:1" gives each replica one L4, which serves a 4B parameter model at bfloat16.

Available types depend on the node pools configured in your cluster. See GPU support for the type table and for clusters where you pass a bare count instead.

​

Serving a different model

Swap one string:

    entrypoint=[
        "python3", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "Qwen/Qwen3-8B",
        "--port", "8000",
        "--max-model-len", "8192",
    ],

Every flag vLLM’s server accepts works here — Chalk passes the list through as the container’s entrypoint. Size the GPU, --max-model-len, and --gpu-memory-utilization to the model you pick; a larger model on an L4 may fail at startup with an out-of-memory error during weight loading.

​

When you need HF_TOKEN

HF_TOKEN is required for gated models. The Gemma and Llama families are gated: you must accept the license on the model’s HuggingFace page with the account that owns the token before any download will succeed. Ungated models need no token at all — you can drop the HF_TOKEN entry from secrets entirely.

The example above delivers the token with Secret.from_local_env("HF_TOKEN"), reading it from your shell at deploy time and storing it as a Chalk secret. Passing it through env= instead also works, but writes the token into the container spec in plaintext.

Skip this step and the group deploys fine, then vLLM exits during weight download with an authorization error from huggingface.co. Check the replica logs if a group never becomes ready.

​

Query the endpoint

vLLM exposes an OpenAI-compatible API, so any OpenAI client works. Point it at {web_url}/v1 and pass the same key you deployed:

import os

from openai import OpenAI

client = OpenAI(
    base_url=f"{sg.web_url}/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemma-3-4b-it",
    messages=[{"role": "user", "content": "Explain feature stores in two sentences."}],
)

print(response.choices[0].message.content)

The model value must match the --model string from the entrypoint exactly — vLLM serves it under that name.

From a shell, using the URL printed at deploy time:

$ export VLLM_URL=https://your-scaling-group-url
$ curl "$VLLM_URL/v1/chat/completions" \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "google/gemma-3-4b-it",
      "messages": [{"role": "user", "content": "Hello!"}]
    }'

​

Securing the endpoint

A scaling group’s web_url is public by default. The gateway fronting it is the cluster’s internet-facing load balancer, so anyone with the URL can reach your GPU.

Private routing is per scaling group. chalk scaling-group create --internal sets routing=PRIVATE, placing the group behind the cluster’s private gateway and deriving web_url from it instead of the public one.

Public or private, give the group a key on the first deploy.

​

Set a key with VLLM_API_KEY

vLLM’s OpenAI server reads VLLM_API_KEY from its own process environment and enforces it as a bearer token. You do not need to add a CLI flag — deliver the variable and vLLM picks it up:

    secrets=[Secret.from_chalk_env("VLLM_API_KEY")],

See Paved path for what Secret.from_chalk_env does. If the value only lives in your shell, use Secret.from_local_env("VLLM_API_KEY") instead, which upserts it as a Chalk secret at deploy time. Either way, callers authenticate with Authorization: Bearer $VLLM_API_KEY, as shown above.

​

What the key does and does not cover

The API key protects only a fixed set of path prefixes: /v1, /v2, /inference, and /cohere. Everything outside them bypasses it even when the key is set — including POST /invocations, which runs the same inference as /v1/chat/completions with no key at all. vLLM’s own guidance says the same thing: “Do not rely exclusively on --api-key for securing access to vLLM.”

The key keeps scanners and leaked-URL traffic out. It is not a production access-control boundary. If you need one, terminate authentication in a service you control and place it in front of the group.

​

Options

• Replicas. min_replicas and max_replicas control autoscaling, exactly as described in Scaling Groups. Note that each replica loads its own copy of the weights.
• Scale to zero. min_replicas defaults to 1, so a deployed group keeps one replica — and its GPU — running while idle. Set min_replicas=0 to have the group scale to zero when traffic stops and scale back up on the next request.
• Weight caching. If repeated cold-start downloads are the bottleneck, mount a Volume at the HuggingFace cache directory so replicas reuse already-downloaded weights.