# Tutorial: Serving a Model on a GPU
source: https://docs.chalk.ai/docs/scaling-groups/gpu-tutorial

## Deploy a small open model on a GPU with vLLM, ask it a question, and delete it.

A scaling group runs a container behind a URL,
and it can request a GPU for each replica. In this tutorial you will serve a
small open model on a GPU with vLLM:

- Deploy the model on a GPU
- Ask it a question
- Delete the scaling group, which frees the GPU

### Before you start

You need:

- A Chalk account and an environment to work in.
- A role that can create scaling groups: the deploy.create permission. If
you created your Chalk team yourself, you already have it. Otherwise, ask an
admin for a role that includes it. See
Permissions.
- A GPU your environment can use. The example requests one NVIDIA T4
(nvidia-tesla-t4). If your environment has a different GPU type, use that
type instead. See Resources and GPUs
for the type names, or ask an admin which types your environment has.

You do not need a Chalk project or chalk init, and you do not deploy anything
with chalk apply. The model is
Qwen2.5 0.5B Instruct,
which anyone can download without a Hugging Face account.

Set up the Python environment that will run the tutorial:

- If you followed the Setup Guide, chalk init already
installed chalkcompute in your project. Run the tutorial's files from your
project folder with uv run python.
- Otherwise, install the Chalk CLI,
run chalk login, and install chalkcompute in a virtual environment:python3 -m venv .venv
source .venv/bin/activate
pip install chalkcomputeOn Windows, activate the environment with .venv\Scripts\activate instead.
Run the tutorial's files with python, from the folder where you ran
chalk login, so that chalkcompute finds your login. See
Logins and environments.

### Step 1: Deploy the model on a GPU

Save this file as deploy.py:

```
import chalkcompute

group = chalkcompute.ScalingGroup(
    name="tutorial-llm",
    image="vllm/vllm-openai:latest",
    gpu="nvidia-tesla-t4:1",
    cpu="4",
    memory="16Gi",
    port=8000,
    entrypoint=[
        "vllm", "serve", "Qwen/Qwen2.5-0.5B-Instruct",
        "--port", "8000",
        "--max-model-len", "4096",
        "--dtype", "half",
    ],
    readiness_probe=chalkcompute.ReadinessProbe.http("/health"),
    authenticated=True,
    min_replicas=1,
    max_replicas=1,
)
group.deploy(ready_timeout=1800.0)
print(group.web_url)
```

Run it with python deploy.py. While the group starts, the script prints its
progress, and then the group's URL:

```
https://<your-environment>-tutorial-llm.<your-domain>
```

deploy() returns once a replica is ready to answer. That takes a few minutes:
Chalk starts a GPU node if none is free, the node downloads the vLLM image, and
vLLM downloads the model and loads it onto the GPU. When we ran this step on a
T4, it took between three and eight minutes, depending on whether a GPU node was
already running.

What each setting does:

- image and entrypoint: the container is vLLM's published image, and
vllm serve starts its OpenAI-compatible server for the model you name.
You do not write any serving code.
- gpu="nvidia-tesla-t4:1": each replica gets one T4. --dtype half runs the
model at 16-bit precision, which every GPU type supports.
- readiness_probe: Chalk sends traffic to a replica only after vLLM's
/health endpoint responds, which happens once the model is loaded.
- authenticated=True: every request needs a Chalk access token. Without it,
anyone with the URL could use your GPU. See
Authentication.
- min_replicas=1 and max_replicas=1: one replica runs, on one GPU, until
you delete the group.

### Step 2: Ask the model a question

vLLM serves an OpenAI-compatible API. Save this file as ask.py:

```
import chalkcompute

group = chalkcompute.ScalingGroup.from_name("tutorial-llm")
response = group.call(
    "/v1/chat/completions",
    json={
        "model": "Qwen/Qwen2.5-0.5B-Instruct",
        "messages": [{"role": "user", "content": "In one sentence, what is a GPU?"}],
        "temperature": 0,
        "max_tokens": 60,
    },
    timeout=120.0,
)
print(response.json()["choices"][0]["message"]["content"])
```

Run it with python ask.py:

```
A GPU (Graphics Processing Unit) is a specialized computer processing unit designed to handle complex mathematical and scientific computations, rendering graphics, and other demanding tasks efficiently.
```

Your answer may differ slightly, depending on the GPU type.

ScalingGroup.from_name(...) attaches to the running group, and call(...)
sends a request to its URL. Because the group is authenticated, call adds
your Chalk access token to the request. Other clients, such as curl or the
OpenAI Python client, send the token themselves; see
Model Inference.

The model value matches the model name in vllm serve, because vLLM serves
the model under that name.

### Step 3: Clean up

The group keeps its GPU while it runs, even when no requests arrive. Save this
file as cleanup.py and run it with python cleanup.py:

```
import chalkcompute

chalkcompute.ScalingGroup.from_name("tutorial-llm").delete()
```

delete() removes the group and releases its GPU.

To keep a model deployed without holding a GPU while it is idle, set
min_replicas=0, and the group scales to zero when traffic stops. The first
request after that waits while a replica starts. See
Autoscaling.

### Next steps

- Model Inference: serve larger or gated
models, and query them with the OpenAI client or curl
- Resources and GPUs: GPU types
and counts
- Autoscaling: add replicas under load,
and scale to zero
- Revisions and Rollback: deploy
a new version of a group and roll back to an earlier one