Infrastructure
Build a Chalk engine image and push it to a registry you own without creating a deployment. Mirror it into a separate cloud partition such as AWS GovCloud.
chalk deployment build builds the engine image for your project and pushes it to container
registries that you name. It does not deploy anything. chalk deploy build is an alias for the
same command. It requires Chalk CLI v1.54.1 or later.
The command does not:
:latest tag on your environment’s engine repository.The result is an image in a registry you control. Your active deployment keeps serving the same version it served before the build.
Use chalk deployment build when you want the image as an artifact to handle yourself:
If you just want to deploy, use chalk apply.
chalk deployment build --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1Repeat --destination to push the same image to several registries in one build:
chalk deployment build \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25 \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidateThe build pushes to every destination in a single pass.
On success the command prints the build’s identifier and the references it pushed to:
Build ID: chalk-build-argo-8f3c21d4
↗ 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25
↗ 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidate
Run the command from your project directory, as you would chalk apply. It archives the same
source tree and honors .chalkignore.
Every destination must be a fully-qualified reference with an explicit tag:
<registry-host>/<repository>:<tag>
| Rule | Accepted | Rejected |
|---|---|---|
| Registry host | 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 | myorg/engine:v1 (Docker Hub shorthand) |
| Explicit tag | .../engine:v1 | .../engine |
| Tag, not digest | .../engine:v1 | .../engine@sha256:… |
| Supported registry | Amazon ECR, Google Artifact Registry, GCR | Azure Container Registry, other hosts |
Chalk rejects an untagged repository instead of tagging it :latest, because writing latest
in a registry you own can overwrite an image your other systems already pull.
ACR destinations are rejected. The builder authenticates only to the environment’s own ACR, so it cannot push to a second registry. The command fails before the build starts.
Chalk pushes with the build’s own cloud identity. It never accepts, stores, or transmits registry credentials.
That means you grant Chalk’s build identity push access to the destination before running the command. What that identity is depends on your cloud:
This model implies two limits on AWS:
Cross-account works. A destination in a different AWS account is fine, as long as that repository’s policy grants your environment’s management role.
Cross-region does not. An ECR authorization token is only valid in the region that issued it, so the destination must be in the same region as your environment’s own registry. A destination elsewhere is rejected before the build starts:
image destination "123456789012.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1"
is in region us-gov-west-1 but this environment's registry is in us-east-1;
an ECR authorization token is only valid in the region that issued it
Destinations must also be on the same cloud as your environment, because the build pod holds one cloud identity. A GCP environment cannot push to ECR, and an AWS environment cannot push to Artifact Registry or GCR.
Because there is no deployment record, this build does not appear in
chalk deployment list, the deployments page, or chalk deployment build-steps. Those
surfaces are all keyed on a deployment.
To find out when the build has finished, watch for the image to appear at its destination:
crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1or equivalently aws ecr describe-images. The GovCloud example below builds this wait into
the mirroring workflow.
To automate this, poll the destination registry. The build ID that chalk deployment build
prints identifies the build workflow, not a deployment, so deployment endpoints do not find it.
The image is the same engine image chalk apply produces: Chalk’s platform software and the
resolved Python virtualenv for your requirements.
Your project source is not baked in. The engine downloads it at boot, from the source
bundle the deploy uploaded to your environment’s bucket. One image therefore serves every
deployment whose requirements are unchanged. This is also what makes
chalk apply --from-image possible.
Runtime configuration is not baked in. When Chalk deploys an image it supplies environment variables pointing at your online store, offline store, and graph artifacts. An image you run outside Chalk’s control plane needs equivalent configuration supplied by whatever runs it.
AWS GovCloud is a separate AWS partition (aws-us-gov). This is a stronger boundary
than a region:
sts:AssumeRole does not cross partitions, so you cannot assume a GovCloud role from a
commercial account.So when your Chalk environment runs in the commercial partition, Chalk cannot push into
GovCloud directly, and --destination will reject a GovCloud reference on the region rule
above. The split is:
Step 2 is your code, running in your account, holding your GovCloud credentials. Chalk is not involved and never sees them. The example below shows one way to do it.
Because role assumption cannot cross partitions, the copy needs a credential issued in GovCloud. Create an IAM user in your GovCloud account with permission to push to the destination repository, and store its keys as a Kubernetes secret:
kubectl create secret generic govcloud-ecr \
--from-literal=access_key_id="$GOVCLOUD_ACCESS_KEY_ID" \
--from-literal=secret_access_key="$GOVCLOUD_SECRET_ACCESS_KEY"Partition boundaries rule out short-lived, federated credentials. Scope the IAM user to ecr:PutImage and the related upload actions on the single
destination repository, rotate it on a schedule, and keep it out of the commercial account’s
blast radius.
The commercial side needs no secret. Give the workflow’s service account an IRSA role with read access to the source repository.
This WorkflowTemplate logs into both registries, copies the image, and verifies that the
digest survived the copy. All three containers share one pod so they can share a docker
config through an emptyDir.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: mirror-chalk-engine-to-govcloud
spec:
entrypoint: mirror
# IRSA role with ECR read on the commercial source repository.
serviceAccountName: chalk-image-mirror
arguments:
parameters:
- name: source-image
value: 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1
- name: destination-image
value: 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1
- name: source-region
value: us-east-1
- name: destination-region
value: us-gov-west-1
templates:
- name: mirror
volumes:
- name: docker-config
emptyDir: {}
containerSet:
volumeMounts:
- name: docker-config
mountPath: /docker
containers:
- name: login
image: amazon/aws-cli:2
env:
- name: GOVCLOUD_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: govcloud-ecr
key: access_key_id
- name: GOVCLOUD_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: govcloud-ecr
key: secret_access_key
command: [sh, -c]
args:
- |
set -eu
SRC='{{workflow.parameters.source-image}}'
DST='{{workflow.parameters.destination-image}}'
SRC_HOST="${SRC%%/*}"
DST_HOST="${DST%%/*}"
# Commercial partition: the pod's own IRSA role.
SRC_PW=$(aws ecr get-login-password \
--region '{{workflow.parameters.source-region}}')
# GovCloud is a different partition, so this must use the static
# credential. `env -u` drops the IRSA variables so the SDK cannot
# prefer web identity over the keys we are handing it.
DST_PW=$(env -u AWS_ROLE_ARN -u AWS_WEB_IDENTITY_TOKEN_FILE \
AWS_ACCESS_KEY_ID="$GOVCLOUD_ACCESS_KEY_ID" \
AWS_SECRET_ACCESS_KEY="$GOVCLOUD_SECRET_ACCESS_KEY" \
aws ecr get-login-password \
--region '{{workflow.parameters.destination-region}}')
SRC_AUTH=$(printf 'AWS:%s' "$SRC_PW" | base64 -w0)
DST_AUTH=$(printf 'AWS:%s' "$DST_PW" | base64 -w0)
umask 077
cat > /docker/config.json <<EOF
{"auths":{
"$SRC_HOST":{"auth":"$SRC_AUTH"},
"$DST_HOST":{"auth":"$DST_AUTH"}
}}
EOF
echo "wrote docker config for $SRC_HOST and $DST_HOST"
- name: copy
image: gcr.io/go-containerregistry/crane:debug
dependencies: [login]
env:
- name: DOCKER_CONFIG
value: /docker
command: [crane]
args:
- copy
- '{{workflow.parameters.source-image}}'
- '{{workflow.parameters.destination-image}}'
- name: verify
image: gcr.io/go-containerregistry/crane:debug
dependencies: [copy]
env:
- name: DOCKER_CONFIG
value: /docker
command: [sh, -c]
args:
- |
set -eu
SRC_DIGEST=$(crane digest '{{workflow.parameters.source-image}}')
DST_DIGEST=$(crane digest '{{workflow.parameters.destination-image}}')
echo "source: $SRC_DIGEST"
echo "destination: $DST_DIGEST"
if [ "$SRC_DIGEST" != "$DST_DIGEST" ]; then
echo "digest mismatch; the copy did not preserve the image" >&2
exit 1
fi
echo "digests match"Run it with:
argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \
-p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
-p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1crane copycrane copy transfers the manifest and its layers as-is, so the image keeps its digest
across partitions. The verify step compares the two digests, and a match shows that the
image in GovCloud is byte-identical to the one Chalk built. A docker pull and docker push
through an intermediate host can rewrite the manifest and change the digest.
skopeo copy --all is an equivalent choice if it fits your tooling better.
# 1. Chalk builds and pushes to your commercial repository.
chalk deployment build \
--destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1
# 2. Wait for the image to land (there is no deployment to poll).
until crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
>/dev/null 2>&1; do sleep 30; done
# 3. Mirror it into GovCloud.
argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \
-p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \
-p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1Tag each build with an immutable value, such as a git SHA or a release version, instead of reusing one tag. The digest check in step 3 shows that the copy matches its source. An immutable tag records which build was authorized.
Once an image exists in a registry your environment’s cluster can pull from, deploy it with
chalk apply --from-image. This flag and the --with-* flags below require Chalk CLI v1.54.0 or
later. They do not appear in chalk apply --help.
chalk apply --from-image \
210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256:<digest>This creates a normal deployment, with a deployment record, a graph, and a rollout, but it
builds nothing. The reference you give becomes the deployment’s engine image verbatim, and the
Python source uploaded by that same chalk apply is what the engine downloads and serves.
Pin the image by its digest instead of a tag. The digest is the identifier the mirroring step verified, so pinning it makes the deployment run the image that was authorized.
Nothing checks that the image matches the source you are deploying. The image carries the
virtualenv for the requirements it was built from; if your source imports something that
virtualenv does not have, the failure appears when the engine boots, not when you run
chalk apply. Deploy from source whose requirements.txt produced the image.
Because no image is built, Chalk rejects any option that describes how to build one: a base
image override or a build profile cannot be combined with --from-image. --from-image also
cannot be combined with --branch.
A deployment created from a pre-built image cannot be rebuilt or redeployed. To change its image,
run chalk apply --from-image again with the image you want.
An image build also produces artifacts besides the image: the source bundle it was built from, the virtualenv tarball, and the graph export an indexing job wrote. Three optional flags serve those instead of producing them again:
| Flag | What it replaces |
|---|---|
--with-source-bundle | The source bundle the engine downloads at boot, in place of the archive chalk apply just uploaded |
--with-export | The deployment’s export.pb, in place of the one an indexing job would have written |
--with-venv | Seeds your environment’s virtualenv cache, so a later indexing or branch build reuses it instead of resolving the requirements again |
Each takes an object that is already in your environment’s source bundle bucket, given as a
bare object key or a full s3://, gs:// or az:// URI naming that same bucket. Chalk copies
within that one bucket; it does not read from anywhere else, and an object in another bucket
is rejected. Copy the artifacts across the partition boundary yourself, the same way you
copied the image.
--with-venv must keep the filename the cache uses, shallow_copy_<hash>.tar.zst, because
that name is the cache key. --with-export is optional in a different sense: without it the
engine builds its graph from the source bundle at boot, which is slower but correct.
chalk apply \
--from-image 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256:<digest> \
--with-source-bundle s3://my-chalk-bundles/authorized/build-v1.tar.gz \
--with-export s3://my-chalk-bundles/authorized/build-v1-export.pb \
--with-venv s3://my-chalk-bundles/authorized/shallow_copy_<hash>.tar.zstchalk apply Works: the deploying counterpart, and what the
image build stage does in detail.