# Scaling Groups
source: https://docs.chalk.ai/docs/compute/scaling-groups

## Deploy autoscaling HTTP services with routing.

A scaling group is a replicated, HTTP-fronted service managed by Chalk. Each
group is backed by a Kubernetes Deployment with rolling updates, automatic
service discovery, and CPU-utilization-based autoscaling between configurable
min and max replica counts. Use scaling groups for ML inference servers,
internal APIs, agent backends, and any long-lived service that needs to be
reachable from outside the cluster.

For workloads that don't need replication or HTTP fronting, use a
Sandbox instead. For serverless, function-shaped
invocations, use Functions.

For registered models served through Chalk's Arrow inference contract, use the
Model Deployments APIs. They expose typed deployment objects,
readiness waiting, revision history, updates, and rollback.

You can drive scaling groups two ways: the Python SDK (chalkcompute.ScalingGroup)
or the chalk scaling-group CLI. Both surfaces target the same underlying
service, so groups created one way are visible and manageable from the other.

### Quick start (SDK)

Save a small Flask app as app.py:

```
from flask import Flask

app = Flask(__name__)


@app.get("/health")
def health():
    return {"ok": True}


app.run(host="0.0.0.0", port=8080)
```

Then deploy it from a script in the same directory:

```
import chalkcompute

img = (
    chalkcompute.Image.debian_slim("3.12")
    .pip_install(["flask"])
    .add_local_file("./app.py", "/app/app.py")
    .entrypoint(["python", "/app/app.py"])
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="hello-api",
    port=8080,
    min_replicas=1,
    max_replicas=3,
    startup_probe=chalkcompute.StartupProbe.http("/health"),
).deploy()

resp = sg.call("/health", method="GET")
print(resp.status, resp.text())

sg.delete()
```

ScalingGroup(...).deploy() builds the image, uploads any local files declared
via add_local_file / add_local_dir, creates the scaling group, and waits
until at least one replica is ready and available. With startup_probe set,
deploy() also waits until the group's web_url responds at /health.

### Quick start (CLI)

```
$ chalk scaling-group create \
    --image=my-registry/my-inference-server:v1 \
    --name=inference \
    --max-replicas=1 \
    --port=8080
```

This creates a Kubernetes Deployment with a single replica running your image,
a headless Service, and an HTTPRoute that exposes the service over HTTPS.

You can tune replicas and resource limits at creation time:

```
$ chalk scaling-group create \
    --image=my-registry/my-inference-server:v1 \
    --name=inference \
    --port=8080 \
    --min-replicas=3 \
    --max-replicas=3 \
    --cpu=4 \
    --memory=8Gi
```

### Revisions and rollback

Each deploy records an immutable revision of the group's spec. The scaling
group keeps a stable ID and name; later deploys append a revision and serve
it, rather than creating a second group.

Redeploying with the same name from ScalingGroup(...).deploy() or
chalk scaling-group create records a new revision and routes all traffic
to it. That is how you roll a new image or spec onto a running group.

To roll back, select an earlier revision so the group serves that spec
again. A group serves one revision at a time. Revision history is on the
scaling group's Revisions tab in the dashboard.

### Replicas and autoscaling

The SDK exposes the autoscaler controls directly on the ScalingGroup
constructor:

| Parameter                           | Default                 | Description                                                        |
| ----------------------------------- | ----------------------- | ------------------------------------------------------------------ |
| `min_replicas`                      | `1`                     | Lower bound for the autoscaler. Set to `0` to allow scale-to-zero. |
| `max_replicas`                      | `1`                     | Upper bound for the autoscaler.                                    |
| `target_cpu_utilization_percentage` | unset (cluster default) | Average CPU utilization the autoscaler targets.                    |
| `scaling_interval_seconds`          | unset                   | How often the autoscaler re-evaluates.                             |
| `shutdown_delay_seconds`            | unset                   | Grace period when removing replicas, to drain in-flight requests.  |

```
sg = chalkcompute.ScalingGroup(
    image=img,
    name="inference",
    port=8080,
    min_replicas=2,
    max_replicas=20,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=30,
).deploy()
```

From the CLI, --min-replicas and --max-replicas set the replica range, and
--target-cpu sets the CPU utilization target for autoscaling. --max-replicas
is required. To run a fixed number of replicas, set both flags to the same value.

### CPU and memory resources

The --cpu and --memory flags (CLI) and the cpu / memory constructor
arguments (SDK) control the resource requests and limits for each replica.
When specified, both the Kubernetes request and limit are set to the same
value, guaranteeing your workload the resources it asks for.

```
$ chalk scaling-group create \
    --image=my-registry/my-app:v1 \
    --name=my-service \
    --max-replicas=1 \
    --port=8080 \
    --cpu=2 \
    --memory=4Gi
```

If you omit CPU and memory, each replica gets 250m CPU and 512Mi memory, set as both
the request and the limit.

CPU values follow Kubernetes conventions: 1 means one full core, 500m means
half a core. Memory values use standard suffixes: Mi (mebibytes) and Gi
(gibibytes).

### GPU support

Scaling groups support GPU-accelerated workloads. The SDK and CLI both accept a
type:count value (or just count) for the GPU request:

```
sg = chalkcompute.ScalingGroup(
    image=img,
    name="gpu-inference",
    port=8080,
    cpu="8",
    memory="32Gi",
    gpu="nvidia-l4:1",
    min_replicas=1,
    max_replicas=4,
).deploy()
```

```
$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-t4:1 \
    --cpu=4 \
    --memory=16Gi
```

The type portion selects the GPU node the replicas are scheduled on, and the
count sets the number of GPUs each replica gets (the nvidia.com/gpu request
and limit). Chalk adds the nvidia.com/gpu toleration automatically, so replicas
can schedule onto GPU-tainted nodes.

Write type as a GKE accelerator name on both GCP and AWS:

- GCP: the value is used as the cloud.google.com/gke-accelerator node
selector, so any accelerator in your cluster's node pools works.
- AWS: Chalk maps the value to Karpenter's karpenter.k8s.aws/instance-gpu-name
label. The types in the table below are supported.

| GPU type                | GPU                 |
| ----------------------- | ------------------- |
| `nvidia-tesla-t4`       | NVIDIA T4           |
| `nvidia-a10g`           | NVIDIA A10G         |
| `nvidia-l4`             | NVIDIA L4           |
| `nvidia-l40s`           | NVIDIA L40S         |
| `nvidia-tesla-v100`     | NVIDIA V100         |
| `nvidia-tesla-a100`     | NVIDIA A100         |
| `nvidia-a100-80gb`      | NVIDIA A100 80GB    |
| `nvidia-h100-80gb`      | NVIDIA H100 80GB    |
| `nvidia-h100-mega-80gb` | NVIDIA H100 Mega    |
| `nvidia-h200-141gb`     | NVIDIA H200         |
| `nvidia-rtx-pro-6000`   | NVIDIA RTX PRO 6000 |
| `nvidia-b200`           | NVIDIA B200         |

Which types you can schedule depends on the GPU node pools configured in your
cluster.

To run on any available GPU node without selecting a type, pass only the count:

```
$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=1
```

To request multiple GPUs, increase the count:

```
$ chalk scaling-group create \
    --image=my-registry/my-training-server:v1 \
    --name=multi-gpu \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-a100:4
```

### DNS and routing

Every scaling group is automatically assigned a TLS-terminated DNS hostname
through the cluster's gateway. The pattern is:

```
<environment-id>-<scaling-group-name>.<gateway-domain>
```

For example, a scaling group named inference in environment abc123 with
gateway domain gw.chalk.ai would be reachable at:

```
https://abc123-inference.gw.chalk.ai
```

From the SDK, read it off the deployed group:

```
print(sg.web_url)
# https://abc123-inference.gw.chalk.ai
```

From the CLI, the URL is shown as Web URL in the output of chalk scaling-group get and chalk scaling-group list. Traffic arriving at this
hostname is routed to the port you specified with --port (or the port
constructor argument).

Because the hostname includes the scaling group name, you can reference it by
a stable, human-readable URL rather than an opaque UUID.

### Invoking from the SDK

The .call() helper sends a request to a path under web_url. The response it
returns has status, headers, and content attributes, plus text() and
json() methods. Use .call() for ad-hoc requests from the SDK, and send
production traffic to web_url directly.

```
resp = sg.call(
    "/predict",
    method="POST",
    json={"text": "the cat sat on the mat"},
    timeout=60.0,
)
print(resp.json())
```

### Environment, secrets, and volumes

These use the same environment, secret, and volume configuration patterns as
sandboxes:

```
import chalkcompute

sg = chalkcompute.ScalingGroup(
    image=img,
    name="api",
    port=8080,
    env={"LOG_LEVEL": "INFO"},
    secrets=[chalkcompute.Secret.from_chalk_env("OPENAI_API_KEY")],
    volumes=[("training-data", "/data")],
    min_replicas=1,
    max_replicas=3,
).deploy()
```

Secrets are resolved at deploy time and injected as environment variables in
every replica. Volume mounts are shared across all replicas, so use them for
read-mostly data — if multiple replicas write to the same volume path
concurrently, the last writer wins on the next sync.

### Managing scaling groups

### Listing

```
$ chalk scaling-group list
```

Displays a table of all scaling groups in your environment with their ID,
name, selected revision, image, status, replica counts, URL, and creation time.

```
from chalk.client import ChalkClient

client = ChalkClient()
response = client.list_scaling_groups()

for scaling_group in response.scaling_groups:
    print(f"Scaling group: {scaling_group.name}")
    print(f"  Status: {scaling_group.status}")
```

### Inspecting

```
$ chalk scaling-group get --name=inference
```

Shows the group's ID, name, selected revision, status, image, port, minimum and
maximum replicas, web URL, and creation time.

```
sg = chalkcompute.ScalingGroup.from_name("inference")
# or
sg = chalkcompute.ScalingGroup.from_id("scaling-group-id")

info = sg.refresh()
print(info.status)             # 'Available', 'Pending', 'ScaledToZero', ...
print(info.ready_replicas)
print(info.available_replicas)
print(sg.web_url)
```

A scaling group reports one of these statuses:

| Status         | Meaning                                                                              |
| -------------- | ------------------------------------------------------------------------------------ |
| `Pending`      | No replica is ready.                                                                 |
| `Available`    | All desired replicas are ready.                                                      |
| `Degraded`     | Some replicas are ready, but not all desired replicas are, or a rollout has stalled. |
| `ScaledToZero` | `min_replicas` is `0` and the autoscaler has scaled the group to zero replicas.      |
| `Unhealthy`    | No replica is ready and a replica is failing, or a startup probe failed.             |
| `Error`        | Chalk could not apply the group's spec.                                              |
| `Unknown`      | Chalk has not reported a status for the group.                                       |
| `Deleted`      | The group was deleted.                                                               |

### Deleting

```
$ chalk scaling-group delete --name=inference
```

```
sg.delete()
```

Both surfaces remove the Kubernetes Deployment, Service, and HTTPRoute
associated with the scaling group. The SDK additionally cleans up any
temporary volumes created from add_local_file uploads. The database record
is soft-deleted so the group still appears in history.

### Tags

You can attach arbitrary key-value tags to a scaling group for organization
and filtering:

```
$ chalk scaling-group create \
    --image=my-registry/my-app:v1 \
    --name=my-service \
    --max-replicas=1 \
    --port=8080 \
    --tags="team=ml,version=2.1"
```

Tags are applied as Kubernetes labels with the prefix chalk.ai/tag-, making
them visible in standard Kubernetes tooling.

### Entrypoint override

To override the image's default entrypoint, pass entrypoint from the SDK or
--entrypoint from the CLI:

```
sg = chalkcompute.ScalingGroup(
    image="python:3.12",
    name="custom-server",
    port=8000,
    entrypoint=["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"],
).deploy()
```

```
$ chalk scaling-group create \
    --image=python:3.12 \
    --name=custom-server \
    --max-replicas=1 \
    --port=8000 \
    --entrypoint="python,-m,uvicorn,main:app,--host,0.0.0.0,--port,8000"
```

The CLI takes comma-separated arguments; the SDK takes a list.

### Full example

A GPU-accelerated inference service that loads model weights from a Chalk
Volume, with custom resource limits, multiple replicas, autoscaling, and tags:

```
$ chalk scaling-group create \
    --image=my-registry/llm-serving:v3 \
    --name=llm-inference \
    --port=8080 \
    --min-replicas=2 \
    --max-replicas=4 \
    --gpu=nvidia-l4:1 \
    --cpu=8 \
    --memory=32Gi \
    --volume=chalk://llama3-weights=/models/llama3 \
    --tags="model=llama3,team=ml-platform" \
    --entrypoint="python,-m,vllm.entrypoints.openai.api_server,--model,/models/llama3"
```

After creation, the service is reachable at its assigned DNS name and can be
referenced by resolvers or other services running in the same Chalk
environment.

### When to use a Scaling Group vs other compute primitives

See Choosing the right primitive
on the Compute overview for the full decision table.