Functions and Scaling Groups
Set replica ranges, scaling targets, scale to zero, and scheduled replica floors for scaling groups and functions.
Autoscaling applies to scaling groups and functions. A function runs on a scaling group and passes these arguments to it. Functions can also scale on their own request queue depth.
| Parameter | Applies to | Description |
|---|---|---|
min_replicas | Both | Lower bound for the autoscaler. Set to 0 to allow scale to zero. On a scaling group, defaults to 1. |
max_replicas | Both | Upper bound for the autoscaler. On a scaling group, defaults to 1. |
target_cpu_utilization_percentage | Both | Average CPU utilization the autoscaler targets, from 1 to 100. The autoscaler adds replicas when the average exceeds it and removes them when the average is below it. |
target_gpu_utilization_percentage | Both | Average GPU utilization the autoscaler targets, from 1 to 100. Requires gpu and min_replicas of at least 1. |
target_queue_depth | Functions | Target pending-request queue depth per replica. When set, the autoscaler scales on the function’s own queue depth. |
scaling_interval_seconds | Both | How often the autoscaler scrapes metrics and makes scaling decisions. Defaults to 60s on the server. |
shutdown_delay_seconds | Both | How long an idle group with min_replicas=0 waits before it scales to zero. See Idle time before scaling to zero. |
scale_from_zero_request_policy | Both | How requests to a scaling group are handled while it scales up from zero. See Scale to zero. |
cron_scaling_trigger | Both | Replica floors for scheduled time windows. See Scheduled replica floors. |
A scaling group:
import chalkcompute
img = chalkcompute.Image.debian_slim("3.12").entrypoint(
["python", "-m", "http.server", "8080"]
)
sg = chalkcompute.ScalingGroup(
image=img,
name="inference",
port=8080,
min_replicas=2,
max_replicas=20,
target_cpu_utilization_percentage=70,
).deploy()
From the CLI, --min-replicas and --max-replicas set the replica range, and
--target-cpu sets the CPU utilization target for autoscaling. --max-replicas
is required. To run a fixed number of replicas, set both flags to the same value.
A function:
import chalkcompute
@chalkcompute.function(
min_replicas=1,
max_replicas=8,
scaling_interval_seconds=30,
target_cpu_utilization_percentage=70,
gpu="nvidia-l4",
)
def infer(text: str) -> str:
...
These two signals apply to functions:
target_cpu_utilization_percentage): good for compute-bound
functions, such as model inference or data transformation, where CPU is the
bottleneck.target_queue_depth): good for I/O-bound or variable-latency
functions, or when you want to scale on demand rather than resource
saturation. Scales up when calls pile up regardless of CPU.Both can be set together; the autoscaler takes the higher signal.
Set min_replicas=0 to let a scaling group scale down to zero replicas when it is not
receiving traffic. A function also accepts min_replicas=0. The group reports the ScaledToZero status while it has no
replicas.
A request that arrives while the group has no replicas starts a scale-up. By
default, Chalk responds to that request with HTTP 503, and your client has to
retry it. To hold the request until a replica is ready to serve it, set
scale_from_zero_request_policy to ScaleFromZeroRequestPolicy.HOLD:
import chalkcompute
img = chalkcompute.Image.debian_slim("3.12").entrypoint(
["python", "-m", "http.server", "8080"]
)
sg = chalkcompute.ScalingGroup(
image=img,
name="on-demand-api",
port=8080,
min_replicas=0,
max_replicas=5,
scale_from_zero_request_policy=chalkcompute.ScaleFromZeroRequestPolicy.HOLD,
).deploy()
From the CLI, pass --min-replicas=0 and --scale-from-zero-request-policy=hold.
shutdown_delay_seconds sets how long, in seconds, a group with min_replicas=0 stays
idle before it scales to zero. A new request during that time resets the timer.
When the group goes idle depends on how it scales:
shutdown_delay_seconds defaults to 30 seconds.target_queue_depth) or a positive scheduled floor (cron_scaling_trigger): the
group goes idle when all scaling signals are inactive, including HTTP activity
when the group supports HTTP wake-up.
shutdown_delay_seconds defaults to 300 seconds.import chalkcompute
@chalkcompute.function(
min_replicas=0,
max_replicas=4,
shutdown_delay_seconds=600,
)
def infer(text: str) -> str:
return text.upper()
On a scaling group, pass the same arguments to ScalingGroup.
shutdown_delay_seconds applies only when min_replicas=0. With a higher min_replicas, it
has no effect, and a group in the host compute class returns an error.
To keep replicas running during set hours, pass a CronScalingTrigger. Each
CronScalingWindow sets a replica floor from its start cron expression to its
end cron expression, evaluated in the trigger’s IANA timezone. During a
window the group runs at least desired_replicas replicas. Outside its windows
the group scales between min_replicas and max_replicas.
A window with desired_replicas=0 adds no replica floor. It does not force the
group to zero or disable HTTP wake-up. For a group that scales on HTTP requests,
a schedule containing only zero floors preserves that behavior.
HTTP requests can also wake a group outside its scheduled windows. A positive scheduled floor keeps replicas warm during its window; CPU utilization can scale running replicas above that floor. CPU utilization alone cannot wake a group from zero.
For a group with a positive scheduled floor, HTTP wake-up occurs on the next
autoscaling poll. Use scale_from_zero_request_policy=HOLD to hold the request
while the group starts, or retry the HTTP 503 response with the default policy.
This group runs at least three replicas from 8:00 to 18:00 New York time on weekdays, and can scale to zero the rest of the time:
import chalkcompute
img = chalkcompute.Image.debian_slim("3.12").entrypoint(
["python", "-m", "http.server", "8080"]
)
sg = chalkcompute.ScalingGroup(
image=img,
name="business-hours-api",
port=8080,
min_replicas=0,
max_replicas=10,
cron_scaling_trigger=chalkcompute.CronScalingTrigger(
timezone="America/New_York",
windows=[
chalkcompute.CronScalingWindow(
start="0 8 * * 1-5",
end="0 18 * * 1-5",
desired_replicas=3,
),
],
),
).deploy()
A function takes the same cron_scaling_trigger argument. Scheduled replica floors are set from the SDK. The CLI has no flag for them.