Scaling Groups

---

A scaling group is a replicated, HTTP-fronted service managed by Chalk. Each group is backed by a Kubernetes Deployment with rolling updates, automatic service discovery, and CPU-utilization-based autoscaling between configurable min and max replica counts. Use scaling groups for ML inference servers, internal APIs, agent backends, and any long-lived service that needs to be reachable from outside the cluster.

For workloads that don’t need replication or HTTP fronting, use a Sandbox instead. For serverless, function-shaped invocations, use Functions.

For registered models served through Chalk’s Arrow inference contract, use the Model Deployments APIs. They expose typed deployment objects, readiness waiting, revision history, updates, and rollback.

You can drive scaling groups two ways: the Python SDK (chalkcompute.ScalingGroup) or the chalk scaling-group CLI. Both surfaces target the same underlying service, so groups created one way are visible and manageable from the other.

---

​

Quick start (SDK)

Save a small Flask app as app.py:

from flask import Flask

app = Flask(__name__)


@app.get("/health")
def health():
    return {"ok": True}


app.run(host="0.0.0.0", port=8080)

Then deploy it from a script in the same directory:

import chalkcompute

img = (
    chalkcompute.Image.debian_slim("3.12")
    .pip_install(["flask"])
    .add_local_file("./app.py", "/app/app.py")
    .entrypoint(["python", "/app/app.py"])
)

sg = chalkcompute.ScalingGroup(
    image=img,
    name="hello-api",
    port=8080,
    min_replicas=1,
    max_replicas=3,
    startup_probe=chalkcompute.StartupProbe.http("/health"),
).deploy()

resp = sg.call("/health", method="GET")
print(resp.status, resp.text())

sg.delete()

ScalingGroup(...).deploy() builds the image, uploads any local files declared via add_local_file / add_local_dir, creates the scaling group, and waits until at least one replica is ready and available. With startup_probe set, deploy() also waits until the group’s web_url responds at /health.

---

​

Quick start (CLI)

$ chalk scaling-group create \
    --image=my-registry/my-inference-server:v1 \
    --name=inference \
    --max-replicas=1 \
    --port=8080

This creates a Kubernetes Deployment with a single replica running your image, a headless Service, and an HTTPRoute that exposes the service over HTTPS.

You can tune replicas and resource limits at creation time:

$ chalk scaling-group create \
    --image=my-registry/my-inference-server:v1 \
    --name=inference \
    --port=8080 \
    --min-replicas=3 \
    --max-replicas=3 \
    --cpu=4 \
    --memory=8Gi

​

Revisions and rollback

Each deploy records an immutable revision of the group’s spec. The scaling group keeps a stable ID and name; later deploys append a revision and serve it, rather than creating a second group.

Redeploying with the same name from ScalingGroup(...).deploy() or chalk scaling-group create records a new revision and routes all traffic to it. That is how you roll a new image or spec onto a running group.

To roll back, select an earlier revision so the group serves that spec again. A group serves one revision at a time. Revision history is on the scaling group’s Revisions tab in the dashboard.

---

​

Replicas and autoscaling

The SDK exposes the autoscaler controls directly on the ScalingGroup constructor:

Parameter	Default	Description
min_replicas	1	Lower bound for the autoscaler. Set to 0 to allow scale-to-zero.
max_replicas	1	Upper bound for the autoscaler.
target_cpu_utilization_percentage	unset (cluster default)	Average CPU utilization the autoscaler targets.
scaling_interval_seconds	unset	How often the autoscaler re-evaluates.
shutdown_delay_seconds	unset	Grace period when removing replicas, to drain in-flight requests.

sg = chalkcompute.ScalingGroup(
    image=img,
    name="inference",
    port=8080,
    min_replicas=2,
    max_replicas=20,
    target_cpu_utilization_percentage=70,
    shutdown_delay_seconds=30,
).deploy()

From the CLI, --min-replicas and --max-replicas set the replica range, and --target-cpu sets the CPU utilization target for autoscaling. --max-replicas is required. To run a fixed number of replicas, set both flags to the same value.

---

​

CPU and memory resources

The --cpu and --memory flags (CLI) and the cpu / memory constructor arguments (SDK) control the resource requests and limits for each replica. When specified, both the Kubernetes request and limit are set to the same value, guaranteeing your workload the resources it asks for.

$ chalk scaling-group create \
    --image=my-registry/my-app:v1 \
    --name=my-service \
    --max-replicas=1 \
    --port=8080 \
    --cpu=2 \
    --memory=4Gi

If you omit CPU and memory, each replica gets 250m CPU and 512Mi memory, set as both the request and the limit.

CPU values follow Kubernetes conventions: 1 means one full core, 500m means half a core. Memory values use standard suffixes: Mi (mebibytes) and Gi (gibibytes).

---

​

GPU support

Scaling groups support GPU-accelerated workloads. The SDK and CLI both accept a type:count value (or just count) for the GPU request:

sg = chalkcompute.ScalingGroup(
    image=img,
    name="gpu-inference",
    port=8080,
    cpu="8",
    memory="32Gi",
    gpu="nvidia-l4:1",
    min_replicas=1,
    max_replicas=4,
).deploy()

$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-t4:1 \
    --cpu=4 \
    --memory=16Gi

The type portion selects the GPU node the replicas are scheduled on, and the count sets the number of GPUs each replica gets (the nvidia.com/gpu request and limit). Chalk adds the nvidia.com/gpu toleration automatically, so replicas can schedule onto GPU-tainted nodes.

Write type as a GKE accelerator name on both GCP and AWS:

• GCP: the value is used as the cloud.google.com/gke-accelerator node selector, so any accelerator in your cluster’s node pools works.
• AWS: Chalk maps the value to Karpenter’s karpenter.k8s.aws/instance-gpu-name label. The types in the table below are supported.

GPU type	GPU
nvidia-tesla-t4	NVIDIA T4
nvidia-a10g	NVIDIA A10G
nvidia-l4	NVIDIA L4
nvidia-l40s	NVIDIA L40S
nvidia-tesla-v100	NVIDIA V100
nvidia-tesla-a100	NVIDIA A100
nvidia-a100-80gb	NVIDIA A100 80GB
nvidia-h100-80gb	NVIDIA H100 80GB
nvidia-h100-mega-80gb	NVIDIA H100 Mega
nvidia-h200-141gb	NVIDIA H200
nvidia-rtx-pro-6000	NVIDIA RTX PRO 6000
nvidia-b200	NVIDIA B200

Which types you can schedule depends on the GPU node pools configured in your cluster.

To run on any available GPU node without selecting a type, pass only the count:

$ chalk scaling-group create \
    --image=my-registry/my-gpu-model:v1 \
    --name=gpu-inference \
    --max-replicas=1 \
    --port=8080 \
    --gpu=1

To request multiple GPUs, increase the count:

$ chalk scaling-group create \
    --image=my-registry/my-training-server:v1 \
    --name=multi-gpu \
    --max-replicas=1 \
    --port=8080 \
    --gpu=nvidia-tesla-a100:4

---

​

DNS and routing

Every scaling group is automatically assigned a TLS-terminated DNS hostname through the cluster’s gateway. The pattern is:

<environment-id>-<scaling-group-name>.<gateway-domain>

For example, a scaling group named inference in environment abc123 with gateway domain gw.chalk.ai would be reachable at:

https://abc123-inference.gw.chalk.ai

From the SDK, read it off the deployed group:

print(sg.web_url)
# https://abc123-inference.gw.chalk.ai

From the CLI, the URL is shown as Web URL in the output of chalk scaling-group get and chalk scaling-group list. Traffic arriving at this hostname is routed to the port you specified with --port (or the port constructor argument).

Because the hostname includes the scaling group name, you can reference it by a stable, human-readable URL rather than an opaque UUID.

​

Invoking from the SDK

The .call() helper sends a request to a path under web_url. The response it returns has status, headers, and content attributes, plus text() and json() methods. Use .call() for ad-hoc requests from the SDK, and send production traffic to web_url directly.

resp = sg.call(
    "/predict",
    method="POST",
    json={"text": "the cat sat on the mat"},
    timeout=60.0,
)
print(resp.json())

---

​

Environment, secrets, and volumes

These use the same environment, secret, and volume configuration patterns as sandboxes:

import chalkcompute

sg = chalkcompute.ScalingGroup(
    image=img,
    name="api",
    port=8080,
    env={"LOG_LEVEL": "INFO"},
    secrets=[chalkcompute.Secret.from_chalk_env("OPENAI_API_KEY")],
    volumes=[("training-data", "/data")],
    min_replicas=1,
    max_replicas=3,
).deploy()

Secrets are resolved at deploy time and injected as environment variables in every replica. Volume mounts are shared across all replicas, so use them for read-mostly data — if multiple replicas write to the same volume path concurrently, the last writer wins on the next sync.

---

​

Managing scaling groups

​

Listing

$ chalk scaling-group list

Displays a table of all scaling groups in your environment with their ID, name, selected revision, image, status, replica counts, URL, and creation time.

from chalk.client import ChalkClient

client = ChalkClient()
response = client.list_scaling_groups()

for scaling_group in response.scaling_groups:
    print(f"Scaling group: {scaling_group.name}")
    print(f"  Status: {scaling_group.status}")

​

Inspecting

$ chalk scaling-group get --name=inference

Shows the group’s ID, name, selected revision, status, image, port, minimum and maximum replicas, web URL, and creation time.

sg = chalkcompute.ScalingGroup.from_name("inference")
# or
sg = chalkcompute.ScalingGroup.from_id("scaling-group-id")

info = sg.refresh()
print(info.status)             # 'Available', 'Pending', 'ScaledToZero', ...
print(info.ready_replicas)
print(info.available_replicas)
print(sg.web_url)

A scaling group reports one of these statuses:

Status	Meaning
Pending	No replica is ready.
Available	All desired replicas are ready.
Degraded	Some replicas are ready, but not all desired replicas are, or a rollout has stalled.
ScaledToZero	min_replicas is 0 and the autoscaler has scaled the group to zero replicas.
Unhealthy	No replica is ready and a replica is failing, or a startup probe failed.
Error	Chalk could not apply the group’s spec.
Unknown	Chalk has not reported a status for the group.
Deleted	The group was deleted.

​

Deleting

$ chalk scaling-group delete --name=inference

sg.delete()

Both surfaces remove the Kubernetes Deployment, Service, and HTTPRoute associated with the scaling group. The SDK additionally cleans up any temporary volumes created from add_local_file uploads. The database record is soft-deleted so the group still appears in history.

---

​

Tags

You can attach arbitrary key-value tags to a scaling group for organization and filtering:

$ chalk scaling-group create \
    --image=my-registry/my-app:v1 \
    --name=my-service \
    --max-replicas=1 \
    --port=8080 \
    --tags="team=ml,version=2.1"

Tags are applied as Kubernetes labels with the prefix chalk.ai/tag-, making them visible in standard Kubernetes tooling.

---

​

Entrypoint override

To override the image’s default entrypoint, pass entrypoint from the SDK or --entrypoint from the CLI:

sg = chalkcompute.ScalingGroup(
    image="python:3.12",
    name="custom-server",
    port=8000,
    entrypoint=["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"],
).deploy()

$ chalk scaling-group create \
    --image=python:3.12 \
    --name=custom-server \
    --max-replicas=1 \
    --port=8000 \
    --entrypoint="python,-m,uvicorn,main:app,--host,0.0.0.0,--port,8000"

The CLI takes comma-separated arguments; the SDK takes a list.

---

​

Full example

A GPU-accelerated inference service that loads model weights from a Chalk Volume, with custom resource limits, multiple replicas, autoscaling, and tags:

$ chalk scaling-group create \
    --image=my-registry/llm-serving:v3 \
    --name=llm-inference \
    --port=8080 \
    --min-replicas=2 \
    --max-replicas=4 \
    --gpu=nvidia-l4:1 \
    --cpu=8 \
    --memory=32Gi \
    --volume=chalk://llama3-weights=/models/llama3 \
    --tags="model=llama3,team=ml-platform" \
    --entrypoint="python,-m,vllm.entrypoints.openai.api_server,--model,/models/llama3"

After creation, the service is reachable at its assigned DNS name and can be referenced by resolvers or other services running in the same Chalk environment.

---

​

When to use a Scaling Group vs other compute primitives

See Choosing the right primitive on the Compute overview for the full decision table.