Aug 13: Context layer webinar
Chalk home page
  1. Functions and Scaling Groups
  2. Tutorial: Serving a Model on a GPU

A scaling group runs a container behind a URL, and it can request a GPU for each replica. In this tutorial you will serve a small open model on a GPU with vLLM:

  • Deploy the model on a GPU
  • Ask it a question
  • Delete the scaling group, which frees the GPU

​
Before you start

You need:

  • A Chalk account and an environment to work in.
  • A role that can create scaling groups: the deploy.create permission. If you created your Chalk team yourself, you already have it. Otherwise, ask an admin for a role that includes it. See Permissions.
  • A GPU your environment can use. The example requests one NVIDIA T4 (nvidia-tesla-t4). If your environment has a different GPU type, use that type instead. See Resources and GPUs for the type names, or ask an admin which types your environment has.

You do not need a Chalk project or chalk init, and you do not deploy anything with chalk apply. The model is Qwen2.5 0.5B Instruct, which anyone can download without a Hugging Face account.

Set up the Python environment that will run the tutorial:

  • If you followed the Setup Guide, chalk init already installed chalkcompute in your project. Run the tutorial’s files from your project folder with uv run python.

  • Otherwise, install the Chalk CLI, run chalk login, and install chalkcompute in a virtual environment:

    python3 -m venv .venv
source .venv/bin/activate
pip install chalkcompute

    On Windows, activate the environment with .venv\Scripts\activate instead. Run the tutorial’s files with python, from the folder where you ran chalk login, so that chalkcompute finds your login. See Logins and environments.

​
Step 1: Deploy the model on a GPU

Save this file as deploy.py:

import chalkcompute

group = chalkcompute.ScalingGroup(
    name="tutorial-llm",
    image="vllm/vllm-openai:latest",
    gpu="nvidia-tesla-t4:1",
    cpu="4",
    memory="16Gi",
    port=8000,
    entrypoint=[
        "vllm", "serve", "Qwen/Qwen2.5-0.5B-Instruct",
        "--port", "8000",
        "--max-model-len", "4096",
        "--dtype", "half",
    ],
    readiness_probe=chalkcompute.ReadinessProbe.http("/health"),
    authenticated=True,
    min_replicas=1,
    max_replicas=1,
)
group.deploy(ready_timeout=1800.0)
print(group.web_url)

Run it with python deploy.py. While the group starts, the script prints its progress, and then the group’s URL:

https://<your-environment>-tutorial-llm.<your-domain>

deploy() returns once a replica is ready to answer. That takes a few minutes: Chalk starts a GPU node if none is free, the node downloads the vLLM image, and vLLM downloads the model and loads it onto the GPU. When we ran this step on a T4, it took between three and eight minutes, depending on whether a GPU node was already running.

What each setting does:

  • image and entrypoint: the container is vLLM’s published image, and vllm serve starts its OpenAI-compatible server for the model you name. You do not write any serving code.
  • gpu="nvidia-tesla-t4:1": each replica gets one T4. --dtype half runs the model at 16-bit precision, which every GPU type supports.
  • readiness_probe: Chalk sends traffic to a replica only after vLLM’s /health endpoint responds, which happens once the model is loaded.
  • authenticated=True: every request needs a Chalk access token. Without it, anyone with the URL could use your GPU. See Authentication.
  • min_replicas=1 and max_replicas=1: one replica runs, on one GPU, until you delete the group.

​
Step 2: Ask the model a question

vLLM serves an OpenAI-compatible API. Save this file as ask.py:

import chalkcompute

group = chalkcompute.ScalingGroup.from_name("tutorial-llm")
response = group.call(
    "/v1/chat/completions",
    json={
        "model": "Qwen/Qwen2.5-0.5B-Instruct",
        "messages": [{"role": "user", "content": "In one sentence, what is a GPU?"}],
        "temperature": 0,
        "max_tokens": 60,
    },
    timeout=120.0,
)
print(response.json()["choices"][0]["message"]["content"])

Run it with python ask.py:

A GPU (Graphics Processing Unit) is a specialized computer processing unit designed to handle complex mathematical and scientific computations, rendering graphics, and other demanding tasks efficiently.

Your answer may differ slightly, depending on the GPU type.

ScalingGroup.from_name(...) attaches to the running group, and call(...) sends a request to its URL. Because the group is authenticated, call adds your Chalk access token to the request. Other clients, such as curl or the OpenAI Python client, send the token themselves; see Model Inference.

The model value matches the model name in vllm serve, because vLLM serves the model under that name.

​
Step 3: Clean up

The group keeps its GPU while it runs, even when no requests arrive. Save this file as cleanup.py and run it with python cleanup.py:

import chalkcompute

chalkcompute.ScalingGroup.from_name("tutorial-llm").delete()

delete() removes the group and releases its GPU.

To keep a model deployed without holding a GPU while it is idle, set min_replicas=0, and the group scales to zero when traffic stops. The first request after that waits while a replica starts. See Autoscaling.

​
Next steps