# Volumes source: https://docs.chalk.ai/docs/compute/volumes ## Persistent, versioned file storage for compute workloads. ### Overview Volumes provide persistent, versioned file storage that sandboxes mount as a reloadable directory. Each sandbox works on its own copy of the volume. Its writes stay inside the sandbox until they are committed as a new version, and another sandbox sees them only after it reloads the volume or starts after the commit. See Syncing a mounted volume for how commits and reloads work. Volumes do not support live coordination between running sandboxes, such as a SQLite database or a message inbox that two sandboxes read and write at the same time. Volumes are file-level storage, backed by a Rust-based FUSE driver that mounts directly into the container's filesystem. Workloads use volumes through normal filesystem operations, and the driver handles tiered caching and replication to object storage. Volumes can be managed through the Python SDK, the CLI, or the web dashboard. ### Creating and using volumes ### CLI ``` # Create a volume chalk volume create chalk://my-data # Upload a file chalk volume cp ./local-model.bin chalk://my-data/models/latest.bin # Upload a directory chalk volume cp ./models/ chalk://my-data/models --recursive # List contents chalk volume ls chalk://my-data # Download a file chalk volume cp chalk://my-data/models/latest.bin ./downloaded-model.bin ``` ### Chalk Volume URIs A Chalk Volume URI starts with chalk://VOLUME, where VOLUME is the volume name. Everything after the next / is a path inside that volume. ``` chalk://my-data/models/latest.bin └─────┘ volume name ``` | Chalk Volume URI | Meaning | | ----------------------------------- | ------------------------------ | | `chalk://my-data` | Volume named `my-data` | | `chalk://my-data/models/latest.bin` | File inside `my-data` | | `chalk://my-data:experiment` | Fork `experiment` of `my-data` | | `chalk://my-data@3` | Version `3` of `my-data` | ### Python SDK ``` import chalkcompute vol = chalkcompute.Volume("my-data") # Upload files vol.put_file("models/latest.bin", model_bytes) # Batch upload for efficiency with vol.batch_upload() as batch: batch.put("data/train.csv", train_csv) batch.put("data/eval.csv", eval_csv) # Read files data = vol.read_file("models/latest.bin") # List files for f in vol.listdir("data/"): print(f"{f.path} {f.size} bytes") ``` ### Mounting into sandboxes Create a sandbox with a volumes parameter to mount that volume before starting it. The mounted volume is available as an ordinary directory inside the sandbox: ``` import chalkcompute vol = chalkcompute.Volume("my-data") vol.put_file("inputs/message.txt", b"hello from the volume\n") sandbox = chalkcompute.Sandbox( image=chalkcompute.Image.debian_slim("3.12"), name="volume-reader", cpu="1", memory="2Gi", volumes=[("my-data", "/data")], ).run() result = sandbox.exec( "python3", "-c", "print(open('/data/inputs/message.txt').read())", ) print(result.stdout_text) sandbox.terminate() ``` ### Syncing a mounted volume Writes to a mounted volume stay inside the sandbox until they are committed. Each commit saves a new immutable version of the volume. A mount commits its writes when: - You run the commit control file, /data/.chalk-volume/commit - You run sync /data, or a program calls fsync on a file in the mount - The mount holds enough uncommitted changes that it commits them on its own - The sandbox terminates To commit and wait until the new version is saved, run the commit control file. It prints the number of the new version: ``` /data/.chalk-volume/commit ``` ``` published Chalk Volume version 3 ``` sync and fsync return right away, and the new version is saved in the background a few seconds later. The .chalk-volume directory is not saved in the volume. When a sandbox terminates, its mount commits the remaining writes before the volume is detached. That final commit has about 10 seconds to finish. To make sure that a sandbox's writes are saved, run the commit control file before you terminate the sandbox. If the mount itself stops unexpectedly, the volume keeps its last committed version, and later writes are not saved. A sandbox that already has the volume mounted keeps the version it mounted. To switch the mount to the latest committed version, run the reload control file: ``` /data/.chalk-volume/reload ``` A sandbox that starts after a commit mounts the latest committed version. ### Web browser Volumes are also browsable from the Chalk dashboard. You can navigate the file tree, preview file contents, and upload or download files directly from the browser. ### Architecture ### FUSE driver The volume mount is implemented as a Rust-based FUSE (Filesystem in Userspace) driver that runs outside your workload's container. The driver exposes the volume as a standard POSIX directory, so workloads don't need special libraries or APIs to read and write files. Any tool that works with the filesystem works with a volume. The FUSE driver handles: - Tiered caching. Frequently accessed files are cached on the sandbox's local disk. Cold data is fetched from object storage on demand. - Transparent replication. All persisted data is durably stored in object storage (S3 or GCS, depending on your environment). Local cache is ephemeral and rebuilt automatically. ### Copy-on-write semantics Volumes use batch copy-on-write. When a sandbox writes to a mounted volume, the writes are buffered locally on the sandbox's filesystem. Other sandboxes do not see these writes until they are committed and the other sandboxes reload, as described in Syncing a mounted volume. ``` # Writes inside the sandbox are local until committed sandbox.exec("cp", "output.parquet", "/volumes/my-data/output.parquet") ``` This design avoids partial-write visibility: other consumers of the volume see a consistent snapshot of committed files. For when a mount commits, including when its sandbox terminates, see Syncing a mounted volume. ### Versioning and fork semantics Every commit creates an immutable snapshot of the volume's state. Past versions are retained and can be retrieved by version ID: ``` # List available versions versions = vol.versions() # Open a previous version (read-only) old = vol.at_version(versions[-2].id) data = old.read_file("models/latest.bin") ``` Because prior versions are immutable and cheaply addressable, volumes support fork semantics. You can spawn multiple sandboxes from the same volume version, let them diverge independently, and commit their results into separate version lineages without copying the underlying data. This is particularly useful for coding agents that need to explore multiple solution paths in parallel: ``` import chalkcompute base_version = vol.latest_version() # Fork two sandboxes from the same starting state sandbox_a = chalkcompute.Sandbox( image=chalkcompute.Image.debian_slim(), volumes=[vol.at_version(base_version.id).fork("sandbox_a")], ).run() sandbox_b = chalkcompute.Sandbox( image=chalkcompute.Image.debian_slim(), volumes=[vol.at_version(base_version.id).fork("sandbox_b")], ).run() # Each sandbox writes independently, with no interference sandbox_a.exec("python", "approach_a.py") sandbox_b.exec("python", "approach_b.py") sandbox_a.terminate() sandbox_b.terminate() ``` Each sandbox's writes are isolated until they are committed, and the original version remains available regardless of what the forks produce. ### Setting up a volume bucket To use volumes on your own EKS or GKE cluster, link an object storage bucket and bind it to the cluster. Chalk volumes are backed by that bucket. The volume service manages file metadata, versions, and signed upload and download URLs. File contents live in your bucket. Grant bucket access to the cloud identity of the service handling volume requests: | Volume backend | Identity to use | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Dataplane API server | The account associated with the `dataplane-api-server` Kubernetes service account in namespace `chalk-dataplane-api-server` | | Legacy cluster manager | The account associated with the cluster-manager pod's Kubernetes service account, configured through background persistence | Chalk routes versioned-volume requests to the dataplane API server when its endpoint is available. Older deployments can serve supported operations through the cluster manager. The two services can have different cloud identities. A running dataplane API pod still needs a volume bucket binding and a reachable endpoint before it can serve volume operations. ### Bucket permissions Apply these permissions to the volume backend's identity for each environment. GCP: - Grant roles/storage.objectAdmin on the bucket to the volume service's Google service account. - Ensure that account has iam.serviceAccounts.signBlob on its signing service account, typically itself. Existing IAM grants may already provide this permission. A custom role containing iam.serviceAccounts.signBlob, or roles/iam.serviceAccountTokenCreator scoped to the signing service account, can provide signing access. See Google's signed URL permissions. AWS: The volume service obtains AWS credentials through its pod's IRSA role. The role needs s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, and s3:GetBucketCors on the bucket. ### Find the volume service identity Select the environment you are configuring. Repeat the lookup for other environments because their service accounts can differ. For the dataplane API server: - Open Kubernetes → Service accounts. - Set Filter by namespace to chalk-dataplane-api-server. - Open dataplane-api-server. Copy GCP Service Account for Google Cloud or AWS Role ARN for AWS. For the legacy background-persistence deployment: - Open Kubernetes → All pods. In Pod visibility, enable Show non environment-scoped pods and clear the namespace filter if needed. - Open the cluster-manager pod, such as go-metrics-bus-writer. - Expand Service Account and copy GCP Service Account or AWS Role ARN. The background-persistence setting Service Account Name identifies a Kubernetes service account. For GCP IAM grants, use the Google service account email shown in GCP Service Account. You can also find a pod's Kubernetes account by name and namespace under Kubernetes → Service accounts. ### CORS policy The Chalk dashboard uploads and downloads volume files directly from the bucket using signed URLs, so the bucket needs a CORS rule that allows: - Origin: the URL you open the dashboard at, for example https://chalk.ai - Methods: GET and PUT - Headers: Content-Type Without it, dashboard uploads and downloads are proxied via the metadata server instead of going directly to the bucket, and the cluster's volume health check reports a CORS error. To add the rule, follow Set up and view CORS configurations for GCS or Using cross-origin resource sharing for S3. ### 1. Create a cloud storage reference Create the bucket first if you haven't already. In the team navigation (click on the Chalk logo in the top left of the page), go to Cloud storage and click Link Storage. Fill in: | Field | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Storage Type | The bucket's provider: **Google Cloud Storage**, **AWS S3**, or **Azure Blob Storage**. | | URI | The bucket URI, optionally with a prefix (for example `gs://my-bucket/volumes`, `s3://my-bucket/volumes`, or `https://myaccount.blob.core.windows.net/my-container`). | | Cloud Account | The saved cloud connection associated with the bucket. The volume service also needs the runtime permissions listed above. | ### 2. Bind the storage reference to the cluster In the team navigation click on Clusters and select the cluster to open the Settings tab. Under Volume Bucket Binding, select the cloud storage reference created in step 1 and click Save Binding. This sets the bucket the cluster's volume service uses. To change or remove the binding later, select a different reference and save, or click Delete Binding. ### Verify volume access For a dataplane API deployment, open Settings → Connections in the environment and inspect the Dataplane API server checks for bucket-access or CORS errors. Then list volumes through the normal Chalk API, for example with chalk volume ls after selecting the environment with chalk login. To verify writes and signed URLs, upload a small file to a test volume and download it again. Finding a service account in Kubernetes confirms its identity; the health checks and volume operations verify that the configured service can reach and use the bucket.