Benchmarking

After setting up your features, resolvers, and queries, you’ll want to confirm that your deployment meets your latency and throughput requirements. Chalk ships a benchmarking tool that runs load against a query host from inside your cluster and reports latency percentiles, throughput, and per-request data.

To see the most up-to-date options, run chalk benchmark --help (or chalk benchmark run --help) in your terminal or navigate to our CLI docs.

​

Command Overview

The benchmark tool is exposed as a group of subcommands:

Every run produces a server-generated benchmark id. The CLI prints it on submission:

✓ Benchmark workflow submitted with ID 7f3c1e0a-...

That id is the key to retrieving your results — result artifacts are named with it as a prefix (see Retrieving results).

​

Best Practices

​

Statically Compile Resolvers

To get the fastest latencies, ensure your resolvers are statically compiled.

• Static DataFrame resolvers — Resolvers defined with @online(static=True) are symbolically compiled into chalkdf / libchalk expressions and executed as a vectorized, native plan. See the Static Resolvers Tutorial.
• Feature expressions — Use Chalk’s underscore (_) and F.* expressions (for example F.json_value, F.http_post, arithmetic on features). These compile to native operators.
• Python resolvers — Chalk will attempt to statically optimize all python resolvers. See Static Resolver Optimization for more detail and best practices.

​

Named Queries

When interested in a particular query, it’s best practice to benchmark against Named Queries.

• Simplifies benchmark command - Named Queries provide a single name to represent a repeatable and potentially large number of input and output features
• Production encouraged - Named Queries are planned on engine boot which is encouraged for production to prevent first query latency spikes on new deployments.
• Plan cache hashing - Queries with a larger number of requested inputs and outputs will need to be hashed to grab the pre-computed plan from the cache. Named Queries simplify the hashing and cache retrieval.

​

Scale

It’s important to isolate and scale your resources to match the level of benchmark you are performing. The larger the QPS, the larger and/or more machines you’ll need. With more cached features, you may also need to increase your online store’s capacity. In the case of Valkey/Redis, this could mean increasing your cluster size and/or node capacity. The engine layer scales roughly linearly with replicas. Other services will possibly need to be scaled to keep pace (online store, data sources, envoy). You can read more about adjusting these resources on the Resource Configuration page.

​

Tutorial: Benchmarking with a Custom Input Distribution

If wanting to benchmark against a specific distribution of keys, this can be done by constructing an input parquet file and using the --in-file flag.

​

Step 1: Build the Input Parquet File

Each column in the Parquet file is a fully-qualified input feature name (the same string you would pass to --in, e.g. user.id). Each row is the input for one query. The benchmark client reads rows from this file to drive the load.

Below, we generate one million user.id values drawn from a distribution rather than a uniform range — here a Zipfian-style skew so that a small set of “hot” users appears far more often, which exercises your cache the way production traffic does.

import numpy as np
import pandas as pd

N_ROWS = 1_000_000
N_USERS = 200_000

# Skewed (Zipf-like) sampling: a few users are very hot, most are rare.
rng = np.random.default_rng(seed=42)
raw = rng.zipf(a=1.3, size=N_ROWS)
user_ids = (raw % N_USERS) + 1  # map into the valid id range [1, N_USERS]

df = pd.DataFrame(
    {
        # Column name MUST be the fully-qualified input feature name.
        "user.id": user_ids.astype("int64"),
    }
)

df.write_parquet("user_inputs.parquet")

If your query takes multiple inputs, add one column per input feature:

df = pd.DataFrame(
    {
        "user.id": user_ids,
        "merchant.id": merchant_ids,
    }
)

​

Step 2: Upload the Input File

Upload the file to your environment’s benchmark storage. The CLI uses the file’s base name as the input name you’ll reference later.

chalk benchmark upload --file-path user_inputs.parquet

✓ Input files uploaded under the names [user_inputs.parquet]

Only .parquet and .json files are accepted. You can pass --file-path multiple times to upload several files at once.

Confirm it’s available (optionally filtering by a name prefix):

chalk benchmark list-inputs
chalk benchmark list-inputs --prefix user_

​

Step 3: Run the Benchmark

Point the run at the uploaded file with --in-file, choose your outputs, and set the load shape.

chalk benchmark run \
  --in-file user_inputs.parquet \
  --out user.email \
  --out user.risk_score \
  --qps 500 \
  --duration 5m \
  --warmup-qps 200 \
  --warmup-duration 1m \
  --percentile 50 --percentile 95 --percentile 99 \
  --result-target html --result-target parquet

The CLI prints a configuration table, asks for confirmation (skip it with --force), then submits the run and prints the benchmark id:

✓ Benchmark workflow submitted with ID 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f

Note the id — you’ll use it as the prefix to find your results.

​

Step 4: List the Result Files

Results are written once the run completes. List them, filtering by your benchmark id prefix so you only see artifacts from this run:

chalk benchmark list-results --prefix 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f

Name                                              Updated At                     Size
7f3c1e0a-..._result.tar.gz                        Mon, 12 Jun 2026 10:04:11 PDT  18234
7f3c1e0a-..._result.parquet                       Mon, 12 Jun 2026 10:04:11 PDT  402114

Result artifacts are named <benchmark-id>_result.<ext>:

• *_result.tar.gz — the HTML report bundle (always produced).
• *_result.parquet — per-request latencies and metadata (when --result-target parquet).
• *_result.json — a structured summary report (when --result-target json).

​

Step 5: Retrieve the Result File

Download by full name (including the extension) into the current directory:

chalk benchmark download \
  --name 7f3c1e0a-..._result.tar.gz \
  --name 7f3c1e0a-..._result.parquet

✓ Result file(s) downloaded to current directory

Unpack the HTML bundle to view the report, or load the Parquet file to analyze per-request latency yourself:

tar -xzf 7f3c1e0a-..._result.tar.gz

import pandas as pd

df = pd.read_parquet("7f3c1e0a-..._result.parquet")

​

Cancelling a Run

If you need to stop an in-flight benchmark:

chalk benchmark kill --id 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f

​

Path to Success

Benchmarking is most useful as an iterative loop: characterize a single engine first, then scale out. A reliable progression:

​

1. Get Your Pipeline Accelerated

Convert resolvers to static=True and push logic into feature expressions (see Statically Compile Resolvers).

​

2. Benchmark One Engine in Isolation

Configure your gRPC resources to a single engine instance to understand the baseline performance of your specific query:

chalk benchmark run \
  --in-file user_inputs.parquet \
  --out user.email \
  --dedicated-engine \
  --dedicated-engine-replicas 1 \
  --qps 100 \
  --duration 3m \
  --warmup-qps 50 --warmup-duration 1m \
  --percentile 50 --percentile 95 --percentile 99

​

3. Run on the Latest Compute-Optimized Instances

For the most latency-sensitive applications, schedule the engine on current-generation compute-optimized instances — for example AWS c8i (or the newest c-family generation available in your region), or the equivalent compute-optimized SKUs on GCP/Azure. The newest generations give the best single-core performance and memory bandwidth. Depending on your cost vs. latency preferences, using an older compute optimized instance can work as well - for example AWS c7i.

​

4. Find the Optimal QPS for One Engine

Sweep QPS upward, holding everything else constant, and watch your target percentile. Increase QPS until p99 starts to climb past your SLO or throughput stops scaling linearly with offered load — that knee is the sustainable QPS for one engine. Record the engine configuration (CPU/memory, env vars, instance type) and that per-engine QPS.

​

5. Scale Out Roughly Linearly with Engines

Once you know a single engine sustains, say, 500 QPS at an acceptable p99, you can scale to your target throughput by adding engines roughly linearly: to serve ~2,500 QPS, run ~5 replicas of the same engine configuration. Verify with a larger run rather than assuming perfect linearity:

chalk benchmark run \
  --in-file user_inputs.parquet \
  --out user.email \
  --dedicated-engine \
  --dedicated-engine-replicas 5 \
  --qps 2500 \
  --duration 5m \
  --warmup-qps 1000 --warmup-duration 1m \
  --percentile 50 --percentile 95 --percentile 99

If p99 holds, you’ve validated your scaling factor. If it degrades, a shared dependency — such as the online store or envoy — is likely the new bottleneck; see Scale.