# Chalk CLI Reference: Benchmark
source: https://docs.chalk.ai/cli#benchmark

Run performance benchmarks against your feature pipelines. Measure query latency and throughput under various load conditions.

## `chalk benchmark run`
source: https://docs.chalk.ai/cli?command=benchmark_run

```bash
chalk benchmark run
```

Use the `benchmark run` command to kick off a run to benchmark queries - this command is in alpha.

Provide the desired input features, output values, qps, and duration. Default QPS and warmup QPS is 1. Default duration and warmup duration is 1m.

**Flags**

- `--qps <qps>` (int, default `1`)
  Queries per second.
- `--duration <duration>` (string, default `1m0s`)
  Duration to benchmark in seconds.
- `--in <[feature]=[value]>` (string[], default `[]`)
  Known feature value.

  ```bash
  chalk benchmark run --in user.id=1232 --out user.email
  ```
- `--out <feature>` (string[], default `[]`)
  Feature or feature namespace to compute.

  ```bash
  chalk benchmark run --in user.id=1232 --out user.name
  ```

  ```bash
  chalk benchmark run --in user.id=1232 --out user
  ```

  ```bash
  chalk benchmark run --in user.id=1232 --out user.organization.name
  ```
- `--query-name <query-name>` (string)
  The name of the named query to run with. Can provide instead of outputs.
- `--query-name-version <query-name-version>` (string)
  Version of the named query to run.

  ```bash
  chalk benchmark run --in user.id=1232 --query-name my_named_query --query-name-version 1.1.0
  ```
- `--warmup-duration <warmup-duration>` (string, default `1m0s`)
  Warmup duration before benchmark measurements begin.
- `--warmup-qps <warmup-qps>` (int, default `1`)
  Queries per second during the warmup for benchmarking.
- `--in-file <in-file>` (string)
  Name of an uploaded input file to use for the benchmark run.
- `--batch-size <rows>` (int, default `1`)
  Number of input rows in each `query_bulk` request. Supported by the Go benchmark runner.
- `--percentile <percentile>` (intArray, default `[]`)
  Include selected latency percentiles in the over-time graph (histogram/distribution views are unchanged). Accepted values: `50`, `95`, `99`, `999`. If `999` is provided, p99.9 is added to both calculations and graph output.

  ```bash
  chalk benchmark run --in feature_class.id=123 --out feature_class.score --percentile 50 --percentile 99
  ```
- `--image-override <image-override>` (string)
  Override the benchmark runner image in Kubernetes (usually only needed with Chalk support guidance).
- `--memory-request <memory-request>` (string, default `8Gi`)
  Memory request for the benchmark container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--memory-limit <memory-limit>` (string, default `16Gi`)
  Memory limit for the benchmark container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--cpu-request <cpu-request>` (string, default `2`)
  CPU request for the benchmark container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--cpu-limit <cpu-limit>` (string, default `4`)
  CPU limit for the benchmark container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--warmup-memory-request <warmup-memory-request>` (string, default `4Gi`)
  Memory request for the warmup container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--warmup-memory-limit <warmup-memory-limit>` (string, default `8Gi`)
  Memory limit for the warmup container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--warmup-cpu-request <warmup-cpu-request>` (string, default `2`)
  CPU request for the warmup container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--warmup-cpu-limit <warmup-cpu-limit>` (string, default `4`)
  CPU limit for the warmup container. Valid units: [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
- `--force <force>` (bool, default `false`)
  If `true`, skip confirmation before creating a benchmark run.
- `--nodepool <nodepool>` (string)
  Schedule the benchmark pod on a specific Karpenter nodepool. Only use this if you've already setup a nodepool to run the benchmark in.
- `--trace-sample-rate <trace-sample-rate>` (duration, default `0`)
  Sample rate for trace collection (0.0-1.0). When set, each request will randomly enable tracing based on this probability.
- `--data-sample-rate <data-sample-rate>` (duration, default `0`)
  Sample request and response payloads for display in HTML report.
- `--insecure <insecure>` (bool, default `false`)
  If `true`, bypass TLS verification between the benchmark client and query host.
- `--benchmark-query-host <benchmark-query-host>` (string)
  Specify query host for the benchmark client to target.
- `--result-target <result-target>` (string[], default `[html]`)
  Which result artifact(s) to save to the results bucket. Repeatable. Allowed values: `html`, `parquet`, `json`. Defaults to `html` only when not specified. `html` provides the report file of latencies. `parquet` provides a parquet of all resulting queries, latencies, and errors. `json` provides a json of all resulting queries, latencies, and errors. For longer and higher QPS runs resulting in more queries, `parquet` is preferred over `json`.

  ```bash
  chalk benchmark run --in user.id=1 --out user.email --result-target html
  ```

  ```bash
  chalk benchmark run --in user.id=1 --out user.email --result-target html --result-target parquet --result-target json
  ```
- `--dedicated-engine <dedicated-engine>` (bool, default `false`)
  Run the benchmark against a temporary dedicated engine deployment.
- `--dedicated-engine-replicas <count>` (string)
  Replica count for the temporary dedicated engine. Defaults to 1 when a dedicated engine is enabled.
- `--dedicated-engine-env <KEY=VALUE>` (string[], default `[]`)
  Environment variable for the temporary dedicated engine in `KEY=VALUE` form. Repeatable.
- `--dedicated-engine-memory-request <8Gi>` (string)
  Memory request for the temporary dedicated engine container.
- `--dedicated-engine-memory-limit <16Gi>` (string)
  Memory limit for the temporary dedicated engine container.
- `--dedicated-engine-cpu-request <2>` (string)
  CPU request for the temporary dedicated engine container.
- `--dedicated-engine-cpu-limit <4>` (string)
  CPU limit for the temporary dedicated engine container.

**Example**

```
$ chalk benchmark run --in user.id=1 --in user.id=2 --out user.email --qps 80 --duration 100s --warmup-qps 50 --warmup-duration 60s
	$ chalk benchmark run --in-file query_abc_input_set_1.parquet --out user.email --batch-size 100 --percentile 99
```

## `chalk benchmark list_inputs`
source: https://docs.chalk.ai/cli?command=benchmark_list_inputs

```bash
chalk benchmark list-inputs
```

Use the `benchmark list-inputs` command to view the availabile input file names for your run.

Provide a prefix to filter the input files names by.

**Flags**

- `--prefix <prefix>` (string)
  The prefix of file(s) you would like to filter by.

**Example**

```
$ chalk benchmark list-inputs 
	$ chalk benchmark list-inputs --prefix query_abc_inputs
```

## `chalk benchmark upload`
source: https://docs.chalk.ai/cli?command=benchmark_upload

```bash
chalk benchmark upload
```

Use the `benchmark upload` command to upload input files for use in benchmarking.

Provide the path to the desired file to upload. Will use the name of the file as the input-file in `benchmark run`. Please include the file extension in the file path upload. Only supports the following file types currently: parquet, json.

**Flags**

- `--file-path <file-path>` (string[], default `[]`)
  The local paths to the files to upload.

**Example**

```
$ chalk benchmark upload --file-path path/to/input_file.parquet
```

## `chalk benchmark list_results`
source: https://docs.chalk.ai/cli?command=benchmark_list_results

```bash
chalk benchmark list-results
```

Use the `benchmark list-results` command to view the availabile result file names from past benchmark runs.

Provide a prefix to filter the result file names by.

**Flags**

- `--prefix <prefix>` (string)
  The prefix of file(s) you would like to filter by.

**Example**

```
$ chalk benchmark list-results 
	$ chalk benchmark list-results --prefix 12345
```

## `chalk benchmark download`
source: https://docs.chalk.ai/cli?command=benchmark_download

```bash
chalk benchmark download
```

Use the `benchmark download` command to download result files from benchmarking runs.

Provide the name to the desired file to download. Please include the file extension in the file path upload.

**Flags**

- `--name <name>` (string[], default `[]`)
  The name(s) of the file(s) to download.

**Example**

```
$ chalk benchmark download --name 12345.tar.gz --name 67890.tar.gz
```
