# Host Pools source: https://docs.chalk.ai/docs/compute/host-pools ## Configure the hosts that back Chalk sandboxes, and size them so sandbox workloads can be scheduled. ### Overview A host pool is a group of hosts that Chalk sandboxes run on. Each host is sized with a fixed amount of CPU and memory, and Chalk packs sandbox containers onto the available hosts. Host pools back workloads that use the host compute class, which is the default for Sandboxes, along with interactive sessions and outbound managed SSH connections from a sandbox. Scaling Groups, Functions, model deployments, and notebook kernels run on Kubernetes directly and do not use host pools. ### Where host pools are configured Host pools can be configured at two scopes, both from the Chalk dashboard: - Cluster: the pool serves every environment in the cluster. This is the recommended scope, because one pool can be shared cluster-wide. To edit these pools, click Clusters in the team navigation, select the cluster, and click its Settings tab. The pools are in the Compute Host Pools section. - Environment: the pool serves a single environment, and cannot be shared with other environments. To edit these pools, open the environment's Settings and click Compute. For each sandbox, Chalk uses cluster pools when at least one matches its GPU requirement: a CPU-only sandbox matches a CPU-only pool, and a GPU sandbox matches a pool with the requested GPU type and count. Otherwise, Chalk uses the environment's pools. Once Chalk selects a scope, it does not fall back to the other scope when a matching pool has no available capacity. Sandbox isolation is enforced the same way regardless of scope, so workloads from different environments can share a cluster pool safely. Editing host pools requires the deploy.create permission, held by the Admin, Owner, and Team Manager roles. Every change is recorded in the audit log. Host pools are not available in serverless environments. ### The default host pool Until you configure a pool of your own, Chalk manages an implicit pool for you. It is created the first time a host-class workload runs in an environment, and it appears in the dashboard marked System managed with its fields locked: | Setting | Default | | --------------- | -------- | | Minimum hosts | 0 | | Maximum hosts | 2 | | CPU per host | 3 | | Memory per host | 12Gi | | Idle timeout | 1 minute | A sandbox requesting more than 3 CPU or more than 12Gi of memory does not fit on these hosts and fails to schedule. To run larger sandboxes, configure a pool. Creating your own host pool removes the system-managed default pool. Your pools then serve every host-class workload in scope, so size them to cover your smallest and largest sandboxes, not only the large one you added the pool for. Deleting your last pool restores the default. ### Configuring a host pool The cluster and environment host pool settings work the same way, with one row per pool. Click the add (+) button to create a pool, fill in its fields, and click Save changes. To delete a pool, click the remove icon on its row and save. The system-managed default pool appears with a System managed badge, and its fields cannot be edited. Each pool has the following fields: | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Name** | A unique name within the scope. Use 3 to 63 characters: lowercase letters, numbers, and hyphens, starting with a letter. | | **Minimum hosts** | Set to `0` for on-demand capacity, or to the same value as **Maximum hosts** for fixed capacity. Values in between are rejected. | | **Maximum hosts** | How many hosts the pool runs when it is active. Must be at least 1. | | **Idle timeout** | How long the pool sits empty before scaling back down, in minutes. Required for on-demand capacity, minimum 1 minute, and unavailable for fixed capacity. | | **CPU per host** | Cores per host, written as a core count such as `2`, `1.5`, or `500m`. Minimum `100m`. | | **Memory per host** | Memory per host, written as a byte quantity such as `512Mi`, `12Gi`, or `1T`. Minimum `512Mi`, maximum `1T`. | | **GPU per host** | Optional. The GPU type each host has. See the GPU details below. | | **Container storage** | Where containers keep their images, runtime state, checkpoints, and scratch files: **Standard disk** or **Local SSD**. Local SSD is available only on GCP, and the choices offered depend on the pool's CPU, memory, GPU, and region. You can't change it after the pool is created. To switch, create a new pool. | CPU and memory are set per host, not for the pool as a whole. A pool of 2 hosts at 8 CPU each can run two sandboxes of 8 CPU, not one sandbox of 16. Chalk selects host hardware that satisfies the CPU and memory you set, so there is no machine type to choose. On AWS and GKE, a pool can have GPUs. Choose a type in GPU per host, or set gpu to a supported :1 value, such as nvidia-l4:1, through the HostPoolService API. GKE host pools support T4, P4, V100, A100 (40 GB and 80 GB), L4, RTX PRO 6000, H100 (High and Mega), H200, and B200. Your GKE cluster needs GPU autoscaling limits, quota, and capacity for the selected type. GB200 and GB300 are not available through GPU host pools. GPU sandboxes use a pool with the matching GPU type; CPU-only sandboxes use CPU-only pools. Your edits take effect when you save. Some saves can interrupt running sandboxes, such as changing CPU, memory, or GPU per host (which replaces the pool's hosts), lowering minimum or maximum hosts, or deleting a pool. For those, Chalk lists what will happen and asks you to confirm with Apply changes. ### Maintenance window Chalk applies its own host updates, such as a new host software image or platform configuration, only during a pool's maintenance window. To set the window, expand the pool's Advanced settings and choose a Mode: - Inherit cluster: use the cluster's maintenance window. New pools start with this mode. - Default: use Chalk's default maintenance window. - Custom: set your own Schedule, as a cron expression, and Duration, in minutes. - Unrestricted: Chalk can update hosts at any time. - Disabled: Chalk doesn't update hosts until you run maintenance yourself. Effective window shows the window that applies to the pool now. While an update waits, the pool shows "An update is pending and will be applied during the next maintenance window." To apply pending updates right away, click Run maintenance now and confirm. This opens a maintenance window immediately, for the Custom duration or 30 minutes, and running sandboxes may be interrupted. The button appears after you save a mode other than Inherit cluster or Unrestricted, and it's unavailable while the pool has unsaved changes. ### On-demand and fixed capacity Setting Minimum hosts to 0 gives you on-demand capacity. The pool runs no hosts while it is empty, starts all of its hosts the first time a sandbox needs one, and returns to zero once it has been empty for the idle timeout. The idle timeout is measured from the moment the last sandbox leaves the pool, not from host CPU idleness. Setting Minimum hosts equal to Maximum hosts gives you fixed capacity. The hosts stay running, so sandboxes do not wait for a host to start. On-demand pools scale as a unit: they run either zero hosts or the full Maximum hosts count. ### Troubleshooting A sandbox that fails to be placed on a host reports one of three errors, depending on the cause. ### A sandbox fails with "no configured host pool can satisfy" ``` Unschedulable: no configured host pool can satisfy the resource and placement constraints (requested cpu=8000m, memory=8.00 GiB). Please contact support if you need additional capacity. ``` No pool in scope is large enough to hold this sandbox. Chalk compares the request against the CPU and memory of a single host in each pool, so a request larger than any one host fails even when the pool's hosts add up to more than the request. The values in the message are the sandbox's request: 8000m is 8 cores, and memory appears in a unit such as GiB, so 8.00 GiB is 8Gi. A GPU request appears as gpu= x , for example gpu=1 x nvidia-h100-80gb. To resolve it: - Raise CPU per host or Memory per host on an existing pool so a host can hold the sandbox. - Add a pool sized for the larger workload. Chalk places each sandbox on the smallest pool that fits it, so adding a large pool alongside a small one does not waste the small one. - Reduce the sandbox's cpu or memory request. Size hosts somewhat larger than the biggest sandbox you intend to run. A host reserves part of its memory for its own overhead, so a sandbox requesting exactly the host's memory may still fail to be placed. ### A sandbox fails with "no eligible registered host has available capacity" ``` Unschedulable: no eligible registered host has available capacity (requested cpu=8000m, memory=8.00 GiB). Please contact support if you need additional capacity. ``` The pools are sized correctly, but their hosts are full. Raise Maximum hosts, or wait for running sandboxes to finish. ### A sandbox reports "no host with sufficient capacity" ``` Unschedulable for at least 600s: no host with sufficient capacity (requested cpu=8000m, memory=8.00 GiB). Please contact support if you need additional capacity. ``` Chalk waited for a host to become available and none did. This usually means the pool is still waiting on the cloud provider for a host. If it persists, check whether the pool's CPU per host and Memory per host are larger than the hardware available in your region.