Model Inference with vLLM

​

Overview

Running your own inference endpoint gives you full control over model selection, GPU allocation, and cost. Chalk Compute supports scaling groups with GPU access — deploy a vLLM server once and let Chalk scale it across multiple replicas.

This tutorial deploys Gemma 4 using vLLM with autoscaling and persistent model caching.

​

Cache model weights with a volume

Large model files (multi-GB) should live in a Volume so they persist across scaling-group replicas. This avoids re-downloading weights every time a replica starts.

from chalkcompute import Volume

vol = Volume(name="gemma4-weights")

On first boot, vLLM downloads the model into the Hugging Face cache directory. By mounting the volume at that path, subsequent replicas start serving immediately.

​

Define the scaling group

Create deploy_gemma4.py:

from chalkcompute import Image, ScalingGroup, Volume

vol = Volume(name="gemma4-weights")

image = (
    Image.base("vllm/vllm-openai:latest")
    .run_commands(
        "pip install huggingface_hub",
    )
)

sg = ScalingGroup(
    image=image,
    name="gemma4-vllm",
    env={
        "HF_TOKEN": "hf_...",                    # Hugging Face access token
        "HUGGING_FACE_HUB_TOKEN": "hf_...",
    },
    port=8000,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=1,
    max_replicas=4,
    entrypoint=[
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "google/gemma-3-27b-it",
        "--host", "0.0.0.0",
        "--port", "8000",
        "--tensor-parallel-size", "1",
        "--max-model-len", "8192",
        "--dtype", "auto",
    ],
).deploy().wait_ready()

print(f"Inference endpoint: {sg.web_url}")

​

Key parameters

​

Deploy it

python deploy_gemma4.py
# Inference endpoint: https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai

​

Query the endpoint

vLLM exposes an OpenAI-compatible API. Point any OpenAI client at your scaling-group URL:

from openai import OpenAI

client = OpenAI(
    base_url="https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai/v1",
    api_key="not-needed",  # no auth required within Chalk
)

response = client.chat.completions.create(
    model="google/gemma-3-27b-it",
    messages=[
        {"role": "user", "content": "Explain feature stores in two sentences."},
    ],
)

print(response.choices[0].message.content)

Or with curl:

curl https://c9d4e71a-5f23-48b6-a0e3-7824bc19d5f6.compute.chalk.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-3-27b-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

​

Scaling behavior

Chalk monitors the scaling group and adds replicas as traffic increases. Each new replica pulls model weights from the shared volume instead of downloading them again. When traffic drops, Chalk scales back down to min_replicas.

# Dev configuration — scale to zero when idle
sg = ScalingGroup(
    image=image,
    name="gemma4-dev",
    port=8000,
    volumes=[("gemma4-weights", "/root/.cache/huggingface")],
    min_replicas=0,
    max_replicas=2,
    target_cpu_utilization_percentage=70,
    entrypoint=[...],
).deploy().wait_ready()