Tunnels - Chalk

Overview

A sandbox is a long-running container you create with chalk sandbox create or with the Sandbox class in chalkcompute. It accepts no inbound connections. To reach a server running in one, use chalk session proxy, which connects a TCP port inside the sandbox to its own standard input and output over an authenticated stream to the Chalk API. Set it as your ssh ProxyCommand to open a shell in a sandbox, copy files, forward ports, or attach a remote IDE.

Reaching a sandbox takes both a Chalk token and an SSH key. Chalk authenticates the stream, and the SSH connection inside it is encrypted end to end between your machine and the sandbox’s sshd.

Requirements

You need Chalk CLI v1.54.6 or later. To update, run chalk update. The sandbox must meet four requirements.

Host-backed. Sandboxes run on the host compute class by default. Sandboxes on the k8s compute class, including GPU sandboxes, are not supported.

An image with sshd installed. Install the SSH server when you build the image, and start it with --entrypoint when you create the sandbox.

No files in /root/.ssh. Chalk rejects an image that has files in /root/.ssh, so keep the SSH server’s authorized keys somewhere else, as the example below does.

A shell and a connector. The image needs sh and one of socat, nc, or bash. If it has none of the three, add socat.

By default, a host-backed sandbox has no network access in either direction, so install everything the tunnel depends on (the SSH server, the connector, and a shell) when you build the image.

Building an image with sshd

Install the SSH server and your public key when you build the image. This Debian example installs openssh-server, socat, and rsync, makes sure host keys exist, stores your public key in /etc/ssh/authorized_keys/root, and binds sshd to loopback. The same steps apply to other base images.

FROM debian:trixie-slim

RUN apt-get update \
 && apt-get install -y --no-install-recommends openssh-server socat rsync ca-certificates \
 && rm -rf /var/lib/apt/lists/*

# sshd needs a host key to present. -A generates any host key type that is missing
# and leaves existing keys in place.
RUN ssh-keygen -A

# Chalk rejects images with files in /root/.ssh, so authorized keys live under
# /etc/ssh. Only the tunnel can reach sshd, so it listens on loopback only.
RUN mkdir -p /run/sshd /etc/ssh/authorized_keys /workspace \
 && printf '%s\n' \
      'ListenAddress 127.0.0.1' \
      'PermitRootLogin prohibit-password' \
      'PasswordAuthentication no' \
      'AuthorizedKeysFile /etc/ssh/authorized_keys/%u' \
      > /etc/ssh/sshd_config.d/chalk.conf

COPY id_ed25519.pub /etc/ssh/authorized_keys/root
RUN chmod 644 /etc/ssh/authorized_keys/root

CMD ["/usr/sbin/sshd", "-D", "-e"]

Push the image to a registry your Chalk environment can pull from, or build the same Dockerfile through Chalk with Image.from_dockerfile (see Images). Then create a sandbox from it. Pass --entrypoint to start sshd, because a sandbox otherwise keeps itself running with its own process and the image’s CMD does not run. Pass --lifetime too, because a sandbox’s lifetime defaults to forever:

chalk sandbox create \
  --image my-registry/dev-box:latest \
  --name my-box \
  --entrypoint "/usr/sbin/sshd -D -e" \
  --lifetime 8h

Generate a key pair for development sandboxes only, and use its public key when you build the image. Every copy of the image carries that public key, so to revoke access you generate a new pair and rebuild the image.

Configuring ssh

Run chalk sandbox configure-ssh to add a block to ~/.ssh/config that routes hosts named chalk-sbx-<sandbox> through chalk session proxy. Pass --dir with the directory your Chalk credentials belong to: the directory where you ran chalk login, or your Chalk project. The CLI chooses credentials by directory, so without --dir the tunnel works only when you start ssh from that directory. To see the block without writing it, add --dry-run:

chalk sandbox configure-ssh --dir ~ --dry-run

# >>> chalk sandbox configure-ssh >>>
# Managed by 'chalk sandbox configure-ssh'. Edits between these markers are
# overwritten on the next run; delete the markers too to stop managing it.
#
# 'ssh chalk-sbx-<sandbox name or id>' tunnels to a Chalk sandbox over the Chalk API.
# The sandbox must be host-backed ('--compute-class host') and running an sshd.
Host chalk-sbx-*
  User root
  ProxyCommand sh -c 'exec "/Users/you/.chalk/bin/chalk-latest-1" session proxy --dir "/Users/you" "${0#chalk-sbx-}"' %n
  # Sandboxes are ephemeral and regenerate their host keys with them, so
  # recording those keys would only produce false mismatch warnings. The
  # connection is authenticated by Chalk before ssh ever sees it.
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  LogLevel ERROR
# <<< chalk sandbox configure-ssh <<<

To write the block, run the command without --dry-run:

chalk sandbox configure-ssh --dir ~

The block points at the specific chalk binary installed when you ran the command, so re-run the command after chalk update, after you move the chalk binary, or when you switch projects. It rewrites its block in place, between the markers. It also keeps the block at the top of the file: ssh uses the first value it finds for each option, so a Host * block above it would override its settings.

Connect to a sandbox with the chalk-sbx- prefix followed by the sandbox’s name or ID:

ssh chalk-sbx-my-box
ssh chalk-sbx-54b4aa4c-6adf-46d0-a3ed-a92a24f19052

The prefix keeps the Host pattern from matching real hosts, and the sh wrapper removes it before the name reaches chalk session proxy.

If your key is not one ssh offers by default, such as a key you generated for development sandboxes, add an IdentityFile in a separate block after the markers. configure-ssh overwrites everything between them:

Host chalk-sbx-*
  IdentityFile ~/.ssh/chalk_sandbox_ed25519

To change the host prefix, the login user, or the sshd port, use --prefix (default chalk-sbx-), --user (default root), and --port (default 22). To write to a file other than ~/.ssh/config, use --ssh-config:

chalk sandbox configure-ssh --prefix box- --user dev --port 2222

Using the connection

With ssh configured, you can use any tool that connects over ssh. To copy files:

scp ./train.py chalk-sbx-my-box:/workspace/
rsync -av ./src/ chalk-sbx-my-box:/workspace/src/

To reach a server in the sandbox from your machine, such as a notebook, a profiler UI, or a debugger, forward its port:

ssh -N -L 8888:127.0.0.1:8888 chalk-sbx-my-box

Editors that read ~/.ssh/config, such as VS Code Remote-SSH and JetBrains remote clients, can connect to chalk-sbx-my-box too. In VS Code, run Remote-SSH: Connect to Host and type the host name. Do not use Remote-SSH: Add New SSH Host: it rewrites the configure-ssh block and breaks its quoting. If that happens, run chalk sandbox configure-ssh --dir ~ again.

Interactive latency is the network round trip to your Chalk region, plus about a millisecond for the tunnel.

Tunnelling other services

chalk session proxy carries any TCP connection. Use --port to reach a database, a debugger, or an inference server in the sandbox the same way. To connect to an address other than 127.0.0.1 inside the sandbox, use --target-host.

For example, to connect to Postgres running in the sandbox on port 5432:

chalk session proxy my-box --port 5432

This connects the service to the command’s standard input and output. To use a client such as psql, run a local listener that starts the proxy for each connection. With socat on your machine:

socat TCP-LISTEN:15432,reuseaddr,fork \
  EXEC:'chalk session proxy my-box --port 5432'

psql -h 127.0.0.1 -p 15432 -U postgres

Run these commands from the directory where you ran chalk login, or pass --dir to chalk session proxy, because the CLI chooses your credentials by directory.

Each connection to the listener starts its own proxy and its own connector process in the sandbox, and both exit when the connection closes.

If the sandbox runs an sshd, forward the ports with ssh instead. A single ssh process carries every forwarded port over one tunnel, and your machine needs nothing except ssh:

ssh -N -L 15432:127.0.0.1:5432 -L 8888:127.0.0.1:8888 chalk-sbx-my-box

How it works

When you set a ProxyCommand, ssh runs that command and sends the SSH protocol over its standard input and output instead of opening a TCP connection. chalk session proxy provides that transport:

It resolves the sandbox and opens an authenticated, bidirectional stream to the Chalk API over an outbound HTTPS connection from your machine.
Chalk starts a connector process in the sandbox (socat, nc, or bash), attached to that stream. The connector opens a TCP connection to 127.0.0.1:22, or to the address and port set with --target-host and --port.
Bytes flow in both directions between ssh and the sandbox’s sshd.

Because ssh sees an ordinary byte stream, authentication, channels, port forwarding, and SFTP all work unchanged.

chalk session proxy writes nothing to standard output except the proxied bytes, because standard output is the connection. When the local side closes, the proxy kills the connector in the sandbox, so a closed connection leaves no process running. A process started with chalk session exec keeps running after its stream ends.

Troubleshooting

The CLI reports chalk session proxy or chalk sandbox configure-ssh as an unknown command: your CLI is older than v1.54.6. Run chalk update.

sandbox image … contains non-empty /root/.ssh; configure all SSH access through the managed SSH API: the image has files in /root/.ssh. Store authorized keys under /etc/ssh/authorized_keys and point AuthorizedKeysFile there, as the example image does. Managed SSH covers connections from a sandbox to other hosts, so it does not replace an sshd for connecting in.

ssh fails with no valid token found and client credentials are incomplete: the proxy ran from a directory with no Chalk credentials. Run chalk sandbox configure-ssh --dir with the directory where you ran chalk login.

container sessions require a host-backed container: the sandbox was created with --compute-class k8s. You cannot change the compute class of an existing sandbox, so create a new one without that flag.

chalk session proxy: no socat, nc, or bash in the container: the image has no connector. Add socat to the image.

Connection closed by UNKNOWN port 65535, after socat reports Connection refused and connector exited with code 1: the proxy reached the sandbox, but no server answered on the port. Check that sshd is running. The list of process names should include sshd:

chalk session exec my-box sh -- -c 'cat /proc/[0-9]*/comm'

If it does not, check the sandbox’s entrypoint with chalk sandbox get --name my-box. An entrypoint of sleep infinity means the sandbox was created without --entrypoint.

User <name> not allowed because account is locked: sshd rejects key authentication for an account whose password field in /etc/shadow starts with !, when sshd runs without PAM. Alpine’s sshd is built without PAM, and adduser -D creates accounts in this state. In the image, set the account’s password field to *, which allows key login and keeps password login disabled: run usermod -p '*' <name> on Debian or Ubuntu, or echo '<name>:*' | chpasswd -e on Alpine. Do not use passwd -u: on Alpine it empties the password field, which leaves the account with no password.

Authentication fails with the right key: run ssh -v to see which key ssh offered, and check that /etc/ssh/authorized_keys/<user> in the image contains that key. sshd ignores the file if a group or other users can write to it.

Sandbox: creating sandboxes, and the isolation model they run under.
Images: building the image that carries your sshd.
Volumes: persistent storage, for work that should outlive a sandbox.
CLI reference: chalk session proxy, chalk session exec, and chalk sandbox configure-ssh.

​Overview

​Requirements

​Building an image with sshd

​Configuring ssh

​Using the connection

​Tunnelling other services

​How it works

​Troubleshooting

​Related

On this page