# What is Chalk? source: https://docs.chalk.ai/docs/what-is-chalk ## Chalk is the platform for production AI: context, runtime, and learning for your agents and models. Chalk gives you the infrastructure to build, run, and improve production AI in your own cloud: - Context: serve fresh, real-time data to your agents and models at inference time, in milliseconds - Runtime: deploy and run those agents and models securely and at scale - Learning: train and evaluate on point-in-time correct history, so your agents and models improve from exactly what production saw Chalk delivers these through the Context Engine and the Chalk Compute runtime, both built on the same philosophy: define what you need in Python, and run it on infrastructure you control. ### Why Chalk? Most teams assemble a production AI stack from a feature store, a vector database, retrieval and prompt tooling, an orchestration layer, a sandbox runtime, and a governance story — then fight to keep them all consistent. Chalk replaces that stack with a single platform: - One definition of your data: features, embeddings, LLM outputs, and prompts are all just Python, computed once and served everywhere: training, real-time inference, and agents. - Point-in-time correct: query any value as it existed at any moment, so training data matches production and agent evals replay exactly what production saw. - Fast in production: features served in single-digit milliseconds, so real-time models and agents never wait on data. - In your own cloud: everything runs in your VPC under IAM roles you control, so your security team evaluates the platform once instead of every agent you ship. ### The Context Engine The Context Engine is a real-time feature computation engine that serves as the single source of truth for your features. You define features once in Python, and Chalk computes and serves them directly from your data sources at inference time in milliseconds. Unlike traditional feature stores, which behave like databases or caching layers, Chalk keeps training and serving perfectly in sync and makes feature development faster. It adds built-in monitoring, a branch-based deployment model for feature versioning, and deployment inside your own cloud. This is especially critical for teams working with sensitive data or in regulated industries. Here's a quick example of declarative feature engineering in Chalk: ``` from chalk.features import features from chalk import online @features class User: id: int total_orders: int chargeback_rate: float fraud_score: float @online def score_fraud( orders: User.total_orders, cbr: User.chargeback_rate, ) -> User.fraud_score: return 0.7 * cbr + 0.3 * (orders < 3) ``` You can query features in real time, or as of any point in the past for training and evaluation. For example, a larger fraud pipeline can expose a fraud prediction alongside transaction features: The interactive example shows a recorded fraud query plan. Select an operator to inspect it, or use the zoom controls to explore the graph. ### Chalk Compute Building on that same philosophy, Chalk Compute extends the platform to the context and infrastructure challenges of agents. It provides governed, sandboxed environments where agents securely run and take action on your data — without it ever leaving your VPC. That lets you run scalable agents (or any remote process) quickly, manage data access, and evaluate agent performance against historical time periods. Here's a simple example of spinning up a sandbox: ``` sandbox = Sandbox( cpu="2", memory="4Gi", image=( Image.debian_slim(python_version="3.13").run_commands("pip install claude") ), volumes=[("code", "/code")], ).run() print(sandbox.exec("echo", "hello!").stdout_text) sandbox.terminate() ``` Each sandbox is fully isolated using gVisor, starts with its own scoped cloud identity, and can be replayed to any point in time. Sandboxes boot in less than a second and can scale up to 10,000 isolated instances in under 10 seconds. The same SDK can also be used to run persistent model servers — for example, a GPU-backed inference server: ``` import chalkcompute # a GPU-backed model server, in your own cloud server = chalkcompute.ScalingGroup( image=chalkcompute.Image.base("vllm/vllm-openai:latest"), gpu="nvidia-l4:1", cpu="4", memory="16Gi", port=8000, min_replicas=0, max_replicas=1, entrypoint=["--model", "Qwen/Qwen2.5-7B-Instruct"], ).deploy() ``` ### Get started - Install the SDK. pip install chalkpy for the Context Engine, and pip install chalkcompute to run agents and models. - Connect your cloud. Point Chalk at your account so everything runs in your VPC. Chalk Compute runs on AWS or GCP; the Context Engine also supports Azure. - Define your features. Write resolvers for the data your models and agents need, then chalk apply to deploy them. - Query them in real time, or backfill any feature as of any point in the past. - Run your agents. Decorate a Python function with @chalkcompute.function and invoke it remotely. Next steps: read the docs overview, the feature engineering guide, the Chalk Compute overview, and the API reference. # Platform Architecture source: https://docs.chalk.ai/docs/architecture ## How it all fits together. Chalk’s platform is architected to - retrieve fresh features across heterogeneous sources with the minimum possible latency - orchestrate and execute complex transformations on structured and unstructured data - support online serving, offline training-set generation, and streaming ingestion in a single system - uphold enterprise-grade guarantees for security, observability, and reliability In short, Chalk was built to get the right data from the right place at the right time. ### Online serving architecture Let's examine how the pieces work together to compute and serve features. Suppose that you want to compute a set of features for making a decision about a request from a user: - Your application sends an HTTP request to Chalk's serving API for features - Chalk's query planner generates an optimized execution plan by analyzing feature dependencies and available data sources. - Chalk's compute engineretrieves fresh values from underlying sources with Resolverspulls from Chalk’s low-latency online storage, such as materialized features or cached featuresexecutes the generated query plan - Returns computed features in response - Newly computed feature values are stored and logged for reuse and auditability Chalk architecture diagram The entire online pipeline, from SQL queries and API calls to getting a response, runs in less than 5ms, even with heterogeneous data sources and complex logic. Chalk uses many techniques to reduce latency, such as: - Automatic parallelism across pipeline stages - Vectorization of pipeline stages that are written using scalar syntax - Statistics-informed join planning - Low-latency key/value stores (like Redis or DynamoDB) - Transpilation of Python code into native code with Chalk's symbolic Python interpreter ### Data orchestration Chalk eliminates the complexity of orchestrating data and ETL pipelines by building a dependency graph (DAG) of your features, which are defined using Python. At inference time, Chalk dynamically builds query plans (subgraphs of your feature DAG) without manual configuration, based on the features you request. Write feature definitions in Python, and Chalk automatically - Determines optimal computation and query planning strategies - Handles caching, incremental updates, and backfills - Provides built-in observability and data lineage without additional tooling As a result, Chalk can serve as a drop-in replacement for orchestration tools like Dagster, Airflow, and Prefect while simultaneously providing purpose-built features for production ML workloads. Chalk query plan Declaratively defining features frees up data teams to focus on designing features instead of writing plumbing code. There's no need to write custom glue code because Chalk interfaces directly with underlying data sources, managing all the connections and transformations behind the scenes. Note: Features can also be computed on a recurring basis with scheduled queries or ingested continuously from streaming sources such as Kafka, Kinesis, and Pub/Sub. ### Offline computation and serving Chalk's architecture also supports efficient point-in-time queries to construct model training sets or perform batch offline inference. - Submit a training data request from a notebook client like Jupyter using Chalk's Python SDK - The query planner builds a point-in-time correct plan - The compute engine pulls point-in-time correct feature values from offline storage - Chalk returns a DataFrame of features to you. Chalk integrates with your existing data providers (Snowflake, Delta Lake, or BigQuery) to ingest massive amounts of data from a variety of data sources and query it efficiently. Note that data ingested into the offline store can be trivially made available for use in an online querying context with Chalk's Reverse ETL. There's an exhaustive list of supported ingestion sources in the Integrations section. ### Data persistence and storage Chalk uses different storage technologies to support online and offline use cases. The online store is optimized for serving the latest version of any given feature for any given entity with the minimum possible latency. Chalk can be configured to use Redis or Memorystore for smaller resident data sets with strict latency requirements, or DynamoDB when horizontal scalability is required. The offline store is optimized for storing all historical feature values, serving point-in-time correct queries, and tracking provenance of features. Chalk supports a variety of storage backends depending on data scale and latency requirements. Typically, Chalk uses Snowflake, Delta Lake, BigQuery, Iceberg, or Athena. ### High-performance execution engine Under the hood, Chalk uses Velox, an open source unified execution engine, to deliver high-throughput feature computation. We maintain a fork that's been optimized for low-latency online inference. You can think of Velox as a backend for query engines like Presto (AWS Athena) and Spark i.e. you can't point Velox at a database and pass in a SQL expression. Rather than forcing users to work directly with low-level execution primitives, Chalk provides an ergonomic interface (Chalk Python SDK) for defining features, transformations, and pipelines. Velox architecture diagram This architecture allows us to expose the power of vectorized computation with clean APIs that feel natural (like writing Pandas and Polars) to data scientists and engineers. Users write simple Python decorators and SQL queries, while Velox handles the complex optimizations that make these computations blazingly fast. - Velox's columnar memory layout and vectorized expression evaluation deliver significant performance improvements, with benchmarks showing 6-7x speedups on real analytical workloads. - Native support for structs, maps, arrays, and nested data types makes it ideal for feature engineering workflows. - Features like filter reordering, dynamic filter pushdown, and adaptive column prefetching optimize query execution based on runtime statistics. ### Service architecture We offer both a hosted model (“Chalk Cloud”) and a customer-hosted model (“Customer Cloud”). Most companies choose to run Chalk in their own cloud (VPC) for data residency and compliance. Chalk is traditionally deployed as a Platform as a Service into an isolated account under the management of the customer, with the core components (VPC, Kubernetes Cluster, Message Queues, etc) managed by Chalk. Compute nodes run on Kubernetes (typically EKS on AWS, GKE on GCP, and AKS on Azure). If you have custom needs, we are happy to customize the deployment to fit with your service architecture. ### Metadata Plane The Metadata Plane is responsible for storing and serving non-customer data (like alert and RBAC configurations). It can control many Data Planes, which it manages through the Kubernetes API, enabling tasks such as scaling deployments and running batch jobs. In short, the Metadata Plane handles: - Deployment orchestration - Access control - Monitoring and alerting - Feature discovery It does not have access to customer data. ### Data Plane The Data Plane encompasses the execution environment for feature pipelines along with the storage and serving infrastructure for both online and offline feature stores. A single Data Plane can run many Chalk Environments. Companies often have 2-3 environments, such as qa, stage, and prod. If running in a single data plane, these environments share resources, which helps with cost and ease of setup. However, if you prefer to have stronger isolation between Chalk Environments, each Chalk Environment can run in a separate Data Plane. Typically, you share one Metadata Plane to orchestrate all Data Planes, and deploy the shared Metadata Plane to the most sensitive of the environments. ### Deployment models Chalk offers several deployment options to provide the right level of infrastructure control. ### Chalk-Hosted Deployment In the Chalk-Hosted Deployment, both the Metadata Plane and Data Plane run in Chalk's cloud account. Deployed in this manner, Chalk runs as a SaaS application. There is no infrastructure to manage, and no ability to see inside the cloud account running Chalk. ### Customer Cloud Deployment The Customer Cloud Deployment is Chalk's most common deployment. Most customers choose our Customer Cloud Deployment. In this model, the customer runs the Data Plane in their own cloud, while Chalk runs the Metadata Plane in Chalk's managed cloud environment. This deployment model strikes a good balance between security and ease of maintenance. No one at Chalk will be able to access your data, but the Chalk team can handle upgrades to the underlying resources without your team's involvement. ### Air-Gapped Deployment Chalk offers the option to self-host both the Data Plane and Metadata Plane. In the Customer Cloud Deployment, only the Data Plane runs in your cloud account, whereas in the Air-Gapped Deployment, the Metadata Plane joins the Data Plane in your cloud account. There are two primary reasons that customers will choose to host the Metadata Plane: - Regulatory requirements: highly regulated environments (such as FedRAMP) may require additional security control. - Disaster recovery: deploying both the Metadata Plane and Data Plane into your cloud account means that Chalk will continue to run independent of any Chalk uptime. In this configuration, no service hosted by Chalk needs to talk to your instance. Telemetry can be exported for billing purposes (over topics), but is non-essential to uptime of your instance. In the event of a complete outage in Chalk's cloud accounts, your instance continues to run indefinitely without disruption. ### Data Plane architecture Chalk has a few tiers of concepts within the Data Plane that are important to understand when thinking about how to structure your deployment, such as how many environments you should set up. Each customer is associated with a team. Each team is tied to one or more projects, and each project can have one or more environments. The team, project, and environment concepts are all logical groupings for the underlying Kubernetes resources of Data Plane components. Each environment deploys into its corresponding Kubernetes namespace. An environment has at least one resource group, which is one set of Data Plane components, as well as a metrics database deployment. Several environments can be deployed to the same Kubernetes cluster. Every Chalk-managed Kubernetes cluster has a set of background processes that can be shared across environments deployed to that cluster. These background processes include load balancers, background persistence workers, a ClickHouse deployment for tracing, and a cluster manager. Most customers have a team, a project, and one or more environments deployed across one or more Kubernetes clusters, but the exact structure of your deployment is flexible within the constraints outlined in the previous paragraph. ### Monitoring Native integrations with PagerDuty and Slack ensure teams are immediately alerted to any issues in their feature pipelines. Beyond alerting, every Chalk query is fully instrumented with traces and detailed logs, enabling both broad system-wide monitoring and deep request-level debugging across every stage of computation, down to the root data source. With Chalk, data teams get - Comprehensive logging & debuggingResolver execution detailsFeature computationsCache missesData source failuresFeature stalenessFeature ownership tracking - Metrics and performance monitoringFeatures: request volumes, computation times, and error ratesModels: inference latency, prediction accuracy, and resource utilizationSystem: query throughput, system latency, and pipeline health - End-to-end request tracingTrack the inputs and outputs of every step in any runLineage tracking from data source to final predictions Easily, build your own views and set up custom dashboards to visualize your metrics and configure smart alerts with custom formulas that notify you instantly when thresholds are crossed or anomalies are detected. This flexibility to configure and define your own metrics makes it easy to answer common questions such as how often certain features are computed, how long individual computations take, and what the average value for a feature is. ### Architecting maintainable and scalable systems for enterprise success By both connecting to your data stores directly and computing features post-fetch, Chalk makes it trivial to integrate new data sources from other teams, dramatically increasing predictive accuracy and the context available to your models. Your systems can also bidirectionally integrate with Chalk's underlying infrastructure, which is built on widely-adopted technologies like Redis or DynamoDB, and leverages open standards like Arrow, Parquet, and Iceberg, ultimately maximizing compatibility and unlocking downstream analytical workflows. Together, these architectural choices enable enterprises to build future-proof ML and AI systems that scale with their needs, maintain interoperability, and seamlessly integrate with their existing technology stack. # Choosing Where to Run Code source: https://docs.chalk.ai/docs/compute/overview ## Choose between a Sandbox, a Scaling Group, and a Function for a workload you run on Chalk. ### Overview Every workload you run on Chalk takes one of three shapes: - Sandbox: an isolated environment for a single ephemeral task, torn down when the task finishes. - ScalingGroup: a replicated, HTTP-fronted service that stays running and autoscales. - Function: a Python callable, deployed once and invoked remotely. All three take the same runtime environment settings: an image, CPU, memory and GPU resources, environment variables and secrets, and volumes. ### Choosing the right component | Mental Model | Reach For | Example | | ------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **An ephemeral task** | [`Sandbox`](/docs/compute/sandbox) | Run an agent's generated code, a one-off data backfill, or a per-session agent workspace that is torn down when the session ends | | **An application that lives forever** | [`ScalingGroup`](/docs/compute/scaling-groups) | A vLLM inference server, an internal API, a persistent agent or chat backend, or a hosted MCP server | | **A (potentially long) RPC call** | [`Function`](/docs/compute/functions) | An embedding or re-ranking endpoint, an enrichment call inside a resolver, or a "score this transaction" call that returns a float to the query server | A Sandbox is an ephemeral task. Reach for a sandbox by default. For a long-running process that needs to be reachable over HTTP, use a Scaling Group. A Function deploys your callable onto a Scaling Group and registers it in the Chalk Catalog. Use a Function when the workload needs to be reachable from Chalk SQL, a resolver, the query server, or a DataFrame, or when it needs async invocation, streaming results, nested calls, retries, concurrency caps, rate limits, or request-queue-driven autoscaling. The diagram below shows how these components sit inside a Chalk-managed deployment in your cloud account. Sandboxes and scaling groups run on a gVisor-hardened nodepool; image builds, the function registry, secrets, and volumes are all reachable from the same VPC; clients can hit the API Server directly, the Envoy router for HTTP traffic into a scaling group, or the Function Queue for async invocations. Chalk deployment architecture # Setup Guide source: https://docs.chalk.ai/docs/setup-guide ## Install the Chalk CLI, create a project, connect your data sources, and define, deploy, and query your first features. This guide walks through setting up a Chalk project from scratch: installing the CLI, creating a project, connecting your data sources, and defining, deploying, and querying your first features. ### Installing the CLI and creating your project - If you don't already have a GitHub repository where you will store your Chalk code, create one. Then, pick a local directory in which to work on Chalk code, and clone the GitHub repository there. Then cd inside of the directory. - If you haven't already, install Python. You can do this through Homebrew as follows: ``` # Install Homebrew $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Use Homebrew to install python3.11 $ brew install python@3.11 ``` - Install the Chalk command line tool. The Chalk CLI allows you to create, update, and manage your feature pipelines directly from your terminal ``` curl -s -L https://api.chalk.ai/install.sh | sh ``` - Create a Python virtual environment within your repository root directory and activate it ``` $ python3.11 -m venv .venv $ source .venv/bin/activate ``` You can run source .venv/bin/activate to activate the virtual environment, and deactivate to deactivate. - Log in by typing chalk login. If you're using a dedicated environment, make sure you use the --api-host flag. Type y when prompted and log in through the browser. - Run chalk init to initialize your project files. The command prompts you for two things: - Dependency manager: choose uv to list your project dependencies in pyproject.toml, or pip to list them in requirements.txt. - Project name: enter the name of your Chalk project. chalk init then creates the following files and directories. requirements.txt is created only if you choose pip. ``` root_directory/ ├── .chalkignore ├── .gitignore ├── .helix/ │ └── languages.toml ├── .vscode/ │ ├── extensions.json │ └── settings.json ├── .zed/ │ └── settings.json ├── notebooks/ │ └── queries.ipynb ├── src/ │ ├── __init__.py │ ├── datasources.py │ ├── models.py │ └── resolvers/ │ ├── resolvers.py │ └── sql/ ├── tests/ │ ├── integration_tests.py │ └── unit_tests.py ├── chalk.yaml ├── pyproject.toml ├── README.md └── requirements.txt ``` chalk.yaml stores configuration information about your project: ``` project: {YOUR_PROJECT_NAME} environments: default: runtime: python312 requirements: pyproject.toml ``` If you choose pip, requirements is set to requirements.txt instead. If you choose uv, pyproject.toml lists your project dependencies under [project], and development tools such as pytest and jupyter under [tool.uv]: ``` [project] name = "[company_name]-chalk" version = "0.1.0" description = "Chalk pilot for [company_name]" readme = "README.md" requires-python = ">=3.11" dependencies = [ "chalkpy[postgresql,runtime]", ] ``` Replace [company_name] in the name and description fields with your company name before you install dependencies. A project name can contain only letters, digits, -, _, and .. If you choose pip, requirements.txt lists your project dependencies: ``` requests chalkpy[runtime] ``` With either choice, pyproject.toml also contains [tool.pyright], [tool.mypy], and [tool.ty.rules] settings that stop those type checkers from reporting diagnostics on Chalk code. The .vscode, .zed, and .helix files configure those editors for the Chalk editor plugin. README.md contains some basic commands you can use with the Chalk CLI. .chalkignore works like a .gitignore file: files that match its patterns are excluded from deploys when you run chalk apply. - Install the Chalk plugin for your editor. - Within your virtual environment, install the project dependencies. If you chose uv, run: ``` $ uv sync ``` If you chose pip, run: ``` $ pip3 install -r requirements.txt ``` Now, you should have a virtual environment with all project dependencies installed and a basic project structure upon which we will build in the next step. ### Configure data sources Having set up a basic project directory, next we'll want to configure the data sources from which we will load data to compute our features. We can do so in the Chalk dashboard. - In the same directory from before, log in or sign up with Chalk directly from the command line. The chalk login command will open your browser and create an API token for your local development, as well as redirect you to your dashboard. If you are not redirected you can also find the dashboard at https://chalk.ai/projects or run the chalk dashboard command. You will automatically see all the environments in which the email you used to log in has been provisioned as a user. - Within the dashboard, navigate to Data Sources in the sidebar and add all data sources that you will be working with here. After you have saved a data source, you can use the Test Data Source button in the upper right hand corner of the Data Source configuration view to verify that your connection is valid. - In src/datasources.py, which chalk init created, reference the data sources that you added in the dashboard. ``` root_directory/ ├── src/ │ ├── __init__.py │ └── datasources.py ├── chalk.yaml ├── README.md └── pyproject.toml ``` Say we added a PostgreSQL data source, then our datasources.py might look something like this: ``` from chalk.sql import PostgreSQLSource pg = PostgreSQLSource(name='PG') ``` For more details on setting up data sources, see here. ### Define feature classes and resolvers Next, we'll define our feature classes and resolvers. Each feature class is a Python class of features, and each resolver tells Chalk how to compute the values for different features. Each feature that we write should correspond to a resolver output. We recommend starting with a minimal feature class and building up iteratively to easily test your code along the way. After writing some feature classes, resolvers, and tests, we would expect to see a directory structure like this: ``` root_directory/ ├── src/ │ ├── resolvers/ │ │ ├── .../ │ │ ├── __init__.py │ │ └── pipelines.py │ ├── __init__.py │ ├── datasources.py │ └── feature_sets.py ├── tests/ │ └── ... ├── .chalkignore ├── chalk.yaml ├── README.md └── pyproject.toml ``` You can read more in our docs about the different kinds of features and the different kinds of resolvers that you can write. If you would like guidance on how to structure your feature classes and resolvers, please reach out in your support channel! ### Deploy and query Now, you can deploy the features and resolvers that you wrote! You can deploy to production by using the chalk apply command. During development, we recommend that you use chalk apply --branch {BRANCH_NAME} to deploy to the branch server, which allows multiple people to work concurrently in one environment, and also enables more performant deploys. Once you have deployed your code, then you can query your features directly from the command line using the chalk query command, or by calling one of our Chalk Clients in code. Chalk has a Python client, Go client, Java client, and a Typescript client. This is the primary workflow for iterating on features and resolvers! Write, deploy, and query to verify whether the feature values that you receive are the values you expect. Once you have finalized your feature class and resolver definitions, the final step is frequently orchestration. ### Orchestration Having verified that your feature and resolver definitions are correct, the next step is to determine how you want to use the corresponding feature values within your larger machine learning platform. Some users trigger resolvers and run queries from within other orchestrated pipelines, such as Airflow. Some users define cron schedulesfor their resolvers, and set staleness values on different features to ensure that the data they query falls within their requirements for freshness. The data world is your oyster! But, as always, if you would like guidance on how to configure that oyster, please reach out in your support channel! ### Further resources For a detailed tutorial on how to build a fraud model using Chalk, see here. # Editor setup source: https://docs.chalk.ai/docs/editor-setup ## Install Chalk language support in your editor. Chalk's editor integrations provide diagnostics, navigation, and typechecking for Python features and resolvers, Chalk SQL resolvers, and Chalk configuration files in projects containing a chalk.yml or chalk.yaml file. Run chalk init from your project root before opening the project in your editor. It writes project settings for supported editors and configures command-line typecheckers to not report spurious diagnostics. If you already have existing editor configurations, they are preserved for you to merge manually. ### VS Code Install the Chalk plugin from the Visual Studio Marketplace. The first Chalk project you open downloads the language server for your platform. To update it later, open the Command Palette and run Chalk: Update Language Server. ### JetBrains IDEs The Chalk JetBrains plugin supports IntelliJ IDEA, PyCharm, DataSpell, and other commercial IntelliJ-based IDEs version 2025.3 and newer. Install Chalk from Settings → Plugins → Marketplace. The plugin downloads the language server when you open the first Chalk file. Use Tools → Update Chalk Language Server to update it later. ### Cursor Install the Chalk plugin from Open VSX. The first Chalk project you open downloads the language server for your platform. To update it later, open the Command Palette and run Chalk: Update Language Server. ### Neovim On Neovim 0.12 and newer, add the Chalk Neovim plugin to your init.lua with the built-in plugin manager: ``` vim.pack.add({ { src = "https://github.com/chalk-ai/chalk-nvim" }, }) require("chalk").setup() ``` On Neovim 0.11, install it with lazy.nvim: ``` { "chalk-ai/chalk-nvim", config = function() require("chalk").setup() end, } ``` The plugin downloads the language server the first time it opens a supported file in a Chalk project. Run :ChalkLspUpdate to update it later. ### Zed New projects created by chalk init configure Zed to install the Chalk extension automatically when you open the project. To install it manually, open the Extension Gallery with Cmd+Shift+X on macOS or Ctrl+Shift+X on Linux and Windows, search for Chalk, and select Install. The extension downloads the language server when you open a Chalk project and selects Chalk as the Python language server. ### Python language-server conflicts Chalk is set up to be the only semantic and typechecking language server inside Chalk projects; do not run ty, pyright, or other Python typecheckers/LSPs in the same project, as they will produce spurious diagnostics. Chalk's editor integrations do not affect Python tooling in non-Chalk projects. # File Structure source: https://docs.chalk.ai/docs/configuration ## Configure your Chalk project and organize your features and resolvers. ### Overview Your Chalk project's configuration is shared across the following files: - chalk.yaml (or chalk.yml): Configuration for your project's deployment - .chalkignore: Files to exclude from your project's deployment - A requirements file: Default Python requirements for Chalk to install via pip (can be overridden in chalk.yaml). This can either be requirements.txt or a pyproject.toml (via Poetry or uv). Here's our recommended repository structure: ``` company_chalk/ ├── src/ │ ├── resolvers/ │ │ ├── ... │ │ ├── __init__.py │ │ └── pipelines.py │ ├── __init__.py │ ├── datasources.py │ └── feature_sets.py ├── tests/ │ └── ... ├── notebooks/ │ └── ... ├── .chalkignore ├── chalk.yaml ├── README.md └── requirements.txt ``` When you're first getting started, we recommend putting all your features in a single file. Keeping the features in a single file makes circular references easier to reason about, as they can just be quoted. If you do want to split your features across multiple files, you'll need to use the if TYPE_CHECKING block from the typing module. ### Organizing Feature Definitions Across Files In large projects, it's common to split feature definitions across multiple Python modules. For unidirectional dependencies, this is straightforward. For example, if src/models/user.py imports src/models/profile.py, you can define the User and Profile features in separate files without issues. However, if you have circular dependencies, you may run into problems. Chalk supports this, but circular imports can arise when features reference each other across files. To avoid these issues, use the if TYPE_CHECKING block from the typing module and quote your forward references. Here's an example of how to do this cleanly: ``` # Imports `User` directly, because `src/models/user.py` # doesn't import `src/models/profile.py` from chalk.features import features from src.models.user import User @features class Profile: id: Primary[User.id] username: str ``` ``` from chalk.features import features from typing import TYPE_CHECKING if TYPE_CHECKING: # Imports `Profile` only when type checking # to avoid circular imports from src.models.profile import Profile @features class User: id: str # Profile must be quoted because it is imported # only when type checking profile: "Profile" ``` By quoting imports inside if TYPE_CHECKING, you avoid circular dependency errors while still maintaining type safety and feature linkage. If the relationship to Profile is optional, you can use typing.Optional or the | None syntax, but the entire annotation should be quoted: ``` from chalk.features import features from typing import TYPE_CHECKING if TYPE_CHECKING: from src.models.profile import Profile @features class User: id: str # All of `"Profile | None"` must be quoted, not just the `Profile` part profile: "Profile | None" ``` A similar pattern should be used for DataFrame annotations: ``` from chalk.features import features from typing import TYPE_CHECKING if TYPE_CHECKING: from src.models.profile import Profile @features class User: id: str # All of `"DataFrame[Transaction]"` must be quoted, not just the `Transaction` part transactions: "DataFrame[Transaction]" ``` In a project with features living in different files, we recommend that the schema definition all live in a single folder, separate from the resolvers. In this case, your folder structure will look something like: ``` company_chalk/ ├── src/ │ ├── models/ │ │ ├── user.py │ │ ├── profile.py │ │ ├── __init__.py │ │ └── ... │ ├── resolvers/ │ │ ├── ... │ │ ├── __init__.py │ │ └── pipelines.py │ ├── __init__.py │ ├── datasources.py │ └── feature_sets.py ├── tests/ │ └── ... ├── notebooks/ │ └── ... ├── .chalkignore ├── chalk.yaml ├── README.md └── requirements.txt ``` Keeping the schema definition all in one place (as you might with something like Protobuf or Avro files) helps you keep your schema definitions organized and makes importing them in your resolvers straightforward. ### chalk.yaml Use chalk.yaml to configure your project's Docker environment, Python configuration, and metadata validation for your features and resolvers. ### Schema The name of this Chalk project, which must match your project's name in the dashboard (case-sensitive). Per-environment overrides. See our Docker documentation for more details. The name of this environment, such as prod or qa. Find available environment names in your dashboard. Use default to configure default values to apply to all of this project's environments. The Python version for the Chalk project. Defaults to python311. Python requirements file to install with pip. This can either be a requirements.txt file or a pyproject.toml file. Path to an alternative Dockerfile that will be used to build a base image for your Chalk deploys. Reference a version of the Chalk runtime that you want to use for this environment. Can be either a version tag, like v3.0.0, or a release stream like stable. If not specified, defaults to stable. Releases are documented in the Platform Release Notes. Path to a custom ignore file for this environment. If not specified, defaults to .chalkignore in the project root. Use this to exclude different files from deployment in different environments. Configuration for metadata validation for features and resolvers. See our validation documentation for more details. Validations for all features. A list of dictionaries of metadata attributes and error severity levels when this metadata property is missing. The name of the metadata property. The level of severity to use when a feature is missing this metadata property. Deploys are permitted for info and warning levels, but disallowed for error level. Validations for all resolvers. A list of dictionaries of metadata attributes and error severity levels when this metadata property is missing. The name of the metadata property. The level of severity to use when a feature is missing this metadata property. Deploys are permitted for info and warning levels, but disallowed for error level. ### Sample file Here is a sample chalk.yaml file. In this file, we use a different Dockerfile in production and different chalkignore files per environment. ``` project: my-project-id environments: default: runtime: python312 requirements: requirements.txt dev: chalkignore: .chalkignore.dev prod: dockerfile: ./DockerfileProd chalkignore: .chalkignore.prod validation: feature: metadata: - name: owner missing: error - name: description missing: warning - name: tags missing: info resolver: metadata: - name: owner missing: error ``` ### .chalkignore Your .chalkignore file should include your scripts, notebooks, and tests. Anything that you are not actively using in your deployment should be added so that non-deployment code does not clutter or interfere with your deployment. ### Per-Environment Chalkignore Files You can specify different chalkignore files for each environment in your chalk.yaml file. This is useful when you want to exclude different files from deployment in different environments. For example, you might want to include experimental features in dev but exclude them from production. ``` project: my-project environments: dev: runtime: python312 chalkignore: .chalkignore.dev # Custom ignore file for dev prod: runtime: python312 chalkignore: .chalkignore.prod # Custom ignore file for prod staging: runtime: python312 # No chalkignore specified - defaults to .chalkignore ``` If no chalkignore field is specified for an environment, Chalk will use the default .chalkignore file in your project root. ### Python Requirements Chalk will install the Python requirements for your project as specified in the requirements parameter of your chalk.yaml file. This can either be a requirements.txt file or a pyproject.toml. Chalk supports both Poetry and uv for managing your Python dependencies. You can specify this file's location and type in your chalk.yaml file like below: ``` project: my-project-id environments: dev: runtime: python312 requirements: requirements-dev.txt prod: requirements: pyproject.yaml ``` ### Python Requirement Overrides The requirements file specified in your chalk.yaml file will serve as the default dependency set for your environment. However, if you have different requirements for specific resolvers, you can also define alternative virtual environments under the venvs key in your chalk.yaml file. ``` project: features environments: default: runtime: python311 requirements: ./requirements.txt platform_version: v3.26.8 venvs: model-1: dependencies: - numpy==2.3.2 - pandas==2.3.2 - scikit-learn==1.7.2 - tensorflow==2.20.0 - torch - transformers model-2: requirements: ./requirements-model-2.txt ``` For each virtual environment, you can specify the associated dependencies either by passing in a list of Python requirements under the dependencies key, or by providing a filepath to a requirements.txt or pyproject.toml file under the requirements key. Then, to run a specific resolver in one of these virtual environments, you can reference your virtual environment using the name specified in your chalk.yaml file in the @before_all, @online, and @offline decorators. For your @before_all functions, if there is no venv specified, the function will run in every virtual environment. Otherwise, if there is a venv specified, the function will only run in the specified virtual environment. Otherwise, @online and @offline resolvers will default to run using the default virtual environment, unless a different venv is specified. ``` from chalk import online, offline, Features from chalk.features import before_all from src.models import User # this will run in all venvs @before_all() def before_all_hook(): from services import msvc msvc_client = msvc.Client() # runs **only** in specified venv @before_all(venv="model-1") def before_all_model_1(): from models import Model1 model_1 = Model1.load() # runs **only** in specified venv @before_all(venv="model-2") def before_all_model_2(): from models import Model2 model_2 = Model2.load() # runs using default venv @online def get_user_email_domain( email: User.email ) -> Features[User.email_handle, User.email_domain]: handle, domain = email.split("@") return User(email_handle=handle, email_domain=domain) # runs using model-1 venv @online(venv="model-1") def get_user_risk_score_1( id: User.id, email_domain: User.email_domain, recent_activity_count: User.recent_activity_count["7d"] ) ->User.is_fraud_1: threshold = msvc_client.get_model_threshold("model-1") return model_1.predict(id, email_domain, recent_activity_count).sum() > threshold # runs using model-2 venv @online(venv="model-2") def get_user_risk_score_2( id: User.id, email_domain: User.email_domain, recent_activity_count: User.recent_activity_count["7d"] ) -> User.is_fraud_2: threshold = msvc_client.get_model_threshold("model-2") return model_2.predict(id, email_domain, recent_activity_count).sum() > threshold ``` ### Pinning Chalk Platform Versions By default, when you deploy in your Chalk environment, Chalk will pull the latest version of the Chalk platform. You can choose to pin a Chalk platform version for each environment such that your deployments will only update your Chalk code, and will use the same platform image until you deploy a new pinned platform version. ``` project: my-project-id environments: dev: runtime: python312 requirements: requirements-dev.txt platform_version: v3.27.30 staging: runtime: python312 requirements: requirements-staging.txt platform_version: v3.27.30 prod: runtime: python312 requirements: requirements.txt platform_version: v3.27.30 ``` ### The Chalk Runtime Package chalkruntime is the underlying library that drives Chalk's feature processing — it manages the core mechanics of your environment. While it isn't intended for standalone use, it's useful to know about if you need precise control over dependency versions. Including chalkruntime in your requirements lets you inspect the exact package versions running in your live environment and keep your local setup in sync with the remote platform configuration. Pick the chalkruntime version that matches your pinned platform version — available versions are listed on PyPI. # Best Practices source: https://docs.chalk.ai/docs/best-practices ## Learn our best practices for building and maintaining your Chalk solution With Chalk, the same solution can be implemented in a number of different ways. Below are some guidelines and recommended patterns for building and maintaining your Chalk solution. ### Data sources and integrations ### Test your data source connections Define integrations through the Chalk dashboard and check that they connect correctly with the Test connection button. The data source configuration page describes the form and the test results. ### Avoid naming your data sources by their "type", e.g. do not call your Postgres data source "postgres" Though this works, it can lead to ambiguity in SQL resolvers when you add multiple data sources of the same type. This occurs because Chalk lets you refer to data sources by their type if you have only linked one data source of that type to Chalk. In general, to future-proof your resolvers, you should refer to them by their name. ### Features ### Start by defining your features Features fully specify what you want your data to look like. Once you have an understanding of the inner relations, writing resolvers for your features becomes easier. ### Avoid dataclass feature types, instead use separate feature classes or unpack the data While Chalk allows for dataclass feature types, they should be avoided. They do not always serialize correctly and can cause errors that are hard to debug. We recommend either unpacking the nested class into basic types or defining and joining an additional Chalk feature class if the nested component is truly a separate entity. Defining a separate feature class also makes the underlying data easier to monitor and test. ### Tag and annotate your features We recommend annotating your features as you are developing them. Your feature annotations show up in the Chalk dashboard and are a good way of documenting your code. You can also add tags and owners to your features, which can be used for aggregation and filtering in the Chalk dashboard. ``` from chalk.features import features from datetime import datetime @features class User: id: str # the user's fullname # :owner: mary.shelley@aol.com # :tags: team:identity, priority:high name: str # the user's birthdate # :owner: mary.shelley@aol.com # :tags: team:identity, priority:high birthday: datetime ``` To apply a tag or owner to all features in a feature class, set it in the feature decorator: ``` from chalk.features import features @features(owner="ada.lovelace@aol.com", tags=['group:risk']) class User: id: str # the user's fullname. name: str ``` You can also apply restrictions or enforce feature annotation for your entire project. For instance, you can block deployment if features are not tagged or described. ### Centrally manage feature-permission tags Feature tags drive feature permissions, so the set of tags on your features is your access-control policy. Declaring tags inline next to each feature spreads that policy across your entire codebase, which creates a large surface to audit and makes it easy to add or change a tag without review. override_feature_tags collapses that surface to a single file. Calling it switches your project into override mode: at deploy time every code-defined tag (inline feature(tags=...), class-level @features(tags=...), and # :tags: comments) is discarded, and the only tags that apply are the ones in the central mapping. ``` # feature_tags.py — the one place that defines feature access from chalk.features import override_feature_tags override_feature_tags( { Transaction.raw_amount: ["pii"], Transaction.amount_bucket: ["cleared"], User.email: ["pii"], } ) ``` Because this becomes the only source of tags, a reviewer has exactly one file to read to reason about who can access which features. Pair it with a CODEOWNERS rule requiring sign-off from your security or platform team on that file, and the organization gets a single, reviewed chokepoint for feature access. Without it, any developer could grant a sensitive feature an AllowDownstream tag from any file and self-escalate access. The override is applied over the fully-populated feature registry at deploy time, so it is independent of import order, and you can split the mapping across multiple override_feature_tags(...) calls within that one reviewed file. ### Keep feature definitions separated from resolvers Features should be defined in separate files from resolvers (except underscore features). ### When starting your Chalk implementation, define all your feature classes in the same file Although it can get a bit lengthy, we recommend starting by defining your features in a single file. This makes expressing joins between features easier and prevents circular dependencies. ### Add validations for your features Validations for your features can prevent incorrect data from being written to your offline store. They can provide an even stricter complement to monitoring, ensuring that nothing is going wrong with the feature you are calculating. ### Use implicit join syntax Joins between feature classes can be specified in a number of different ways. We recommend using an implicit join syntax, which we cover in the join section of the docs. ### Resolvers ### Use SQL resolvers to read data from your raw datasets While Python resolvers can be used to read data from your data sources, SQL resolvers are preferred. SQL resolvers allow for direct execution against your data sources and additional optimizations, making them more efficient. ### Prefer Chalk expressions over Python resolvers for simple computations ``` # Preferred: Chalk expression @features class User: id: str age: int is_adult: bool = feature(expression=_.age >= 18) # Avoid: Python resolver for simple computation @online def get_is_adult(age: User.age) -> User.is_adult: return age >= 18 ``` ### Explicitly list columns in a select statement for SQL file resolvers Select statements in SQL resolvers should be explicit. Avoid using the * syntax. ### Give Python resolvers and SQL filenames clear names Resolver names are used in the Chalk dashboard to identify resolvers. They should be clear and concise. ### Make sure your resolvers operate inside a single feature space Resolver inputs and outputs must belong to the same feature class, but joins can allow resolvers to connect data between feature classes. ### Transform your Chalk DataFrame to Pandas or Polars Converting a Chalk DataFrame to Pandas or Polars in a Python resolver is cheap. Chalk, Pandas, and Polars all use Arrow, so moving data from a Chalk DataFrame to either is close to free. ### Querying ### Run simple queries with the Chalk CLI, for more flexibility use one of Chalk's API clients Use the Chalk CLI to run simple online queries. For more complex use cases, use one of Chalk's API clients. ### Run large queries asynchronously When sending large or long-running queries, use run_asynchronously to set up one Kubernetes pod per input, separate from your production and branch servers. You can configure resources for asynchronous offline queries in your dashboard settings. You can also terminate asynchronous queries through the dashboard by terminating the pod or canceling the query within the dashboard, which is not possible for synchronous queries. ### Create named queries for your commonly executed queries With Chalk, you can alias your complex queries using a NamedQuery. Named queries help simplify and document your query patterns. To define a named query, add a NamedQuery object to your Chalk deployment: ``` from chalk import NamedQuery from src.models import User NamedQuery( name="fraud", input=[User.id], output=[ User.email_age_days, User.denylisted, User.credit_report.flags, ], tags=["team:fraud"], staleness={ User.credit_report.flags: "30d" }, owner="jodie@chalk.ai", description="Primary fraud model for signup", ) ``` Running chalk apply makes the query name available as an alias for executing your more complex queries. For instance, running: ``` chalk query --in user.id=1 --query-name fraud ``` is equivalent to running the more complicated: ``` chalk query \ --in user.id=1 \ --out user.email_age_days \ --out user.denylisted \ --out user.credit_report.flags \ --staleness user.credit_report.flags=30d \ --tag team:fraud ``` This feature is also accessible in all of our API clients through the query_name parameter. For instance, in Python, you can run: ``` from chalk.client import ChalkClient ChalkClient().query( input={"user.id": 1}, query_name="fraud", ) ``` To list the named queries in your active deployment, run: ``` chalk named-query list ``` To create multiple versions of similar queries, use the version parameter of the NamedQuery object and the query_name_version parameter of our various clients. ### Deployment ### Customize your Chalk project with configuration files Chalk expects to find a chalk.yaml file in your project's root repository. This file stores configuration for your deployment (such as your Dockerfile and Python requirements). You can find a sample Chalk project repository structure and more details in our configuration documentation, which also describes how to exclude files from your deployment with .chalkignore. ### Test and query code changes on the branch server Use the branch server to test that your deployments and new features are behaving as expected. ### Use the @before_all decorator to configure global variables for your resolvers Global setup for resolvers should be done through a function decorated with the @before_all decorator. This also allows for unique setups for different environments. ### Custom files such as machine learning models are accessed with the TARGET_ROOT environment variable To access files packaged in your chalk deployments, use the TARGET_ROOT environment variable to fully specify the path to your files. For instance, if you have the following directory which you are deploying to Chalk: ``` example/ ├── chalk.yaml ├── features.py ├── model.joblib └── resolvers.py ``` Access the model.joblib file as follows: ``` import os model_file_path=f"{os.environ['TARGET_ROOT']}/model.joblib" ``` ### Observability and testing ### Set up monitoring on your most important features and resolvers early Monitoring helps you catch tricky bugs early and gives you guarantees about the data you are generating and serving. Configure monitoring for your important features and resolvers early in your implementation. ### Unit test your resolvers to make sure they function as expected You can set up unit tests for your resolvers using Pytest or any other Python testing framework. ### Improve CI/CD in your Chalk repository using our GitHub Actions workflow Chalk has a GitHub Actions integration that you can use to create branch deployments or run queries as part of your code development cycle. ### Security ### Generate and use access tokens to set and restrict the permissions for your different users Scope your Chalk user permissions with access tokens. These can be programmatically generated through Chalk clients or in the dashboard. Give your users only the permissions they need. # Common Errors source: https://docs.chalk.ai/docs/debugging-errors ## Learn techniques for debugging different elements of Chalk. We've compiled some common errors that people come across, as well as some approaches to debug these errors and their most common root causes. If you are still struggling to debug, please reach out in your support channel! ### Query Errors These are errors you would encounter when running queries, either manual or orchestrated. ### Unexpected query results If you're receiving either 0 rows when you expect a non-zero query result, or if the feature values that you're seeing in your query results do not match what you're expecting, we recommend that you follow the steps in our guide for debugging queries. This includes some common tips and tricks, such as using store_plan_stages and stepping through the query plan. ### Unexpected Chalk server error with status code 401 The 401 status code means unauthenticated. We'd recommend verifying that the token you're using to run the query is correct. To refresh your token, you can run chalk login from the terminal using our CLI and then chalk config to verify that your token has been provisioned for the right environment. ### Unexpected Chalk server error with status code 500 The 500 status code means internal server error. This should usually come with another message with more details on the root cause of the error. If you are struggling to debug this, please reach out to us! ### Unexpected Chalk server error with status code 502 The 502 status code means bad gateway, which usually indicates timeout or not enough servers. We'd recommend looking at CPU utilization on your cluster to determine if you need to modify the cloud resource configurations. You can do so in the Infrastructure > Resource Configuration page in the Chalk dashboard by modifying and applying resource configuration JSON changes under Advanced Resource Configuration. However, we'd also encourage reaching out to us for discussion on how to optimize resource configuration for your environment! ### Query Timeout / Unclear Failure If you are executing a long-running query from a notebook or somewhere locally, the client's polling for query status might have a timeout shorter than the query's runtime. If the query times out before completion, we'd recommend verifying whether the query is still running in the dashboard by looking at the query run page and validating that there is a pod in the cluster running the query. It's possible that your query may also complete after the polling timeout, in which case you can view the query status in the dashboard. ### Please double check that all features have resolvers and resolvers do not have circular dependencies. The first thing to validate is that every feature can be mapped to a resolver output. If all features have resolvers, then you might see this error message if Chalk is unable to construct a dependency graph for resolving the features that you are querying. This is usually a result of a join that cannot be processed. Please reduce the number of output features in your query until you have identified a minimal set that reproduces the circular dependency error and reach out to us with this context! ### Duplicate Resolver Shortname The first thing to check is whether there actually are duplicate resolvers by the same name. If there are not, this error message can also sometimes be an indicator of a missing import. If you are unable to determine if your recent changes required an import, you can use a linting tool to find out, such as pylint. ### chalk apply Errors These are errors you would encounter when running chalk apply or chalk apply --branch. ### Branch server might still be starting You would usually encounter this error when running chalk apply --branch. In the Chalk dashboard under Branches, you can check the status of the branch server. You can also kill lingering processes on branches, such as long-running queries, by stopping and restarting the branch server in the dashboard. ### Could not connect to server when running chalk apply If you've verified that you have a token and are properly authenticated, then you should also verify what role you have within the Chalk environment. The developer role can only run chalk apply --branch rather than chalk apply. ### Deployed Errors These are errors that you might observe through metrics monitors on deployed and orchestrated features and resolvers, as opposed to errors you would encounter in local development. ### Elevated number of errors (e.g. 502/504's) Typically, these errors are an indication of a misalignment in load and resource configuration. Please reach out to us in your support channels with context on data scale and current resource configurations and we can help you identify mitigation strategies to avoid these errors! ### Returning stale features If you're seeing stale features, we'd recommend ensuring that you have max_staleness set for the features to configure the maximum staleness allowed for the features, so that Chalk will know to recompute fresh values as specified. # Debugging Queries source: https://docs.chalk.ai/docs/debugging-queries ## Learn techniques for debugging online and offline queries Online and offline queries can combine data from multiple sources, and the way data flows through a query can be complex. Chalk has several tools for debugging queries. This page uses them on a few simple features and resolvers. First, we define the feature classes. The example uses a User and a Transaction object, representing users performing financial transactions. We then write a simple aggregation that has a bug in its definition, and use various tools to debug that issue. ``` from datetime import datetime from chalk.features import features, DataFrame @features class Transaction: id: int user_id: "User.id" amount: float ts: datetime @features class User: id: int sum_transaction_amount: float transactions: DataFrame[Transaction] ``` Then, we define simple online resolvers to ingest user and transaction data from a Postgres database: ``` -- resolves: User -- source: postgres select id from users ``` ``` -- resolves: Transaction -- source: postgres select id, user_id, amount, ts from transactions ``` Next, we write a simple (buggy) aggregation resolver to compute the sum of a user's transactions: ``` from chalk import online @online def sum_transaction_amount(txns: User.transactions[Transaction.amount]) -> User.sum_transaction_amount: return txns[Transaction.amount].count() # this 'count' is a bug! we debug it below. ``` This aggregation has a bug. It computes the count of the User's Transactions, rather than the sum of their amounts. Run a query: ``` data = ChalkClient().query(input={User.id: 1}, output=[User.sum_transaction_amount]) print(data.get_feature_value(User.sum_transaction_amount)) # Outputs 100 ``` The output is wrong. From prior knowledge of the data, the transaction sum for User 1 is not 100. To debug this issue, use the query plan visualizer. ### Query plan visualizer The query plan visualizer shows the structure of a query and the data that flows through it. It is on the query's details page. For an online query, go to Online > Online queries, select the query, and open the Plan tab. For an offline query, go to Offline > Offline queries and select the query. To inspect the data at each stage of the query, run the query again with store_plan_stages=True. This stores all of the data that passes through each stage of the query plan. Storing plan stages significantly increases latency, so use it only for debugging. ``` ChalkClient().query(input={User.id: 1}, output=[User.sum_transaction_amount], store_plan_stages=True) # To debug multiple users at once: ChalkClient().offline_query(input={User.id: [1, 2, 3]}, output=[User.sum_transaction_amount], store_plan_stages=True) ``` With the CLI, pass the --store-plan-stages flag to chalk query. Here is the plan for the online query. The plan shows the structure of the query: the transactions resolver runs to fetch the transactions for each user, and then the sum_transaction_amount resolver runs to compute the sum. The number on each edge is the row count, and each operator shows its duration. The visualizer also shows the data that flows through the query. Select the transactions resolver, then open the Outputs tab to see its output. The Stats tab shows summary statistics for the output. The transaction.amount column shows that the sum of the transaction amounts is far larger than 100. The incorrect output 100 is identical to the count summary of the transactions resolver, which is a hint that the aggregation computes the count of the transactions rather than the sum of their amounts. If the match with the count is not obvious, more tools are available. The next section walks through how to execute the aggregation locally using the raw input data, so you can use a debugger. The visualizer displays the first 20 rows of each stage of the query plan. To see more, select Download full output to download the raw parquet file. ### Resolver replay Resolver execution can be examined using the query plan visualizer, but Chalk also allows you to directly execute your resolver on your local computer using the same arguments that were used in your query. To use the resolver_replay functionality: - Run an offline_query with store_plan_stages=True to store the query plan stages, and recompute_features=True to allow resolver execution. - On the returned dataset, call resolver_replay with your resolver function. Example: ``` from chalk.client import ChalkClient ChalkClient().offline_query( input={User.id: [1]}, output=[User.sum_transaction_amount], store_plan_stages=True, recompute_features=True ).resolver_replay(sum_transaction_amount) ``` This will execute the resolver locally, using the same input data that was used in the query. This allows you to debug using your IDE (VS Code, PyCharm, or even pdb): Debugger You can edit the definition of the resolver locally, and re-run resolver_replay to see the results of your changes in your terminal or by stepping through with a debugger. ### Using logs to debug If you cannot use the query plan visualizer or resolver replay, you can also use the logs to debug your query. Use the chalk_logger from the chalk.clogging just like a standard Python logger. This logger will output to the web interface, and to configured log sinks (for example, a Datadog instance). ``` from chalk.clogging import chalk_logger @online def sum_transaction_amount(txns: User.transactions[Transaction.amount]) -> User.sum_transaction_amount: chalk_logger.info(f"Transactions: {txns}") return txns[Transaction.amount].count() # this 'count' is a bug! we debug it below. ``` You can also view these logs using resolver_replay, where they will be emitted to your terminal: Logs output example ### Query error categories In using the debugging techniques above, you may encounter a few different categories of errors that have common root causes: ### Request error Request errors are raised before your resolver code executes. They are often caused by invalid feature names in the input or by requests that cannot be satisfied by the resolvers you have defined. ### Feature error Feature errors are raised when a specific resolver that maps to a feature fails. For this type of error, you will find a feature and resolver attribute in the error type. When a resolver crashes, you will receive null value in the response. To differentiate from a resolver returning a null value and a failure in the resolver, you need to check the error schema. ### Network error Network errors are thrown outside your resolvers. They can be caused by unauthenticated requests or other connection failures. ### Error schema The online query interface for resolvers returns the following schema: ### Response schema The outputs features and any query metadata (discussed in detail at Query Basics.) Errors encountered while running the resolvers. Each element in the list is a ChalkError. If no errors were encountered, this field is empty. ### ChalkError The type of error, matching one of the error codes. The category of the error, given in the type field for the error codes. This will be one of "REQUEST", "NETWORK", and "FIELD". A readable description of the error message. The exception that caused the failure, if applicable. The name of the class of the exception. The message taken from the exception. The stacktrace produced by the code. The fully qualified name of the failing feature, eg. user.identity.has_voip_phone. The fully qualified name of the failing resolver, eg. my.project.get_fraud_score. ### Error codes The query contained features that do not exist. A resolver was required as part of running the dependency graph that could not be found. The query is invalid. All supplied features need to be rooted in the same top-level entity. A feature value did not match the expected schema (eg. incompatible type "int"; expected "str") The resolver for a feature errored. The resolver for a feature timed out. A crash in a resolver that was to produce an input for the resolver crashed, and so the resolver could not run. The request was submitted with an invalid authentication header. The request has credentials that do not provide the required authorization to execute an operation. An unspecified error occurred. # Development with LLMs source: https://docs.chalk.ai/docs/development-with-llms ## Write Chalk code with LLMs using Chalk's agent prompts and skills. Chalk provides specialized prompts to help Large Language Models (LLMs) write effective Chalk code. These prompts are designed to guide AI assistants in understanding Chalk's patterns, best practices, and API conventions. ### Getting started - Run the chalk init agent-prompt command to add the prompt to your local repository. - Provide specific context about your feature requirements. - Review generated code for adherence to your team's conventions. - Test the generated features in your development environment. These prompts encode Chalk's conventions, so an assistant working from them produces more idiomatic Chalk code. ### Agent prompts repository The Chalk team maintains the public repository chalk-ai/agent-prompts with prompts specifically designed for LLM-assisted Chalk development: This repository contains prompts that are tested against LLM providers to help them understand: - Chalk's feature definition patterns - Built-in LLM integration capabilities - Resolver implementation best practices - Data source integration patterns - Template interpolation syntax - Model selection and configuration ### Agent skills Chalk publishes six agent skills for specific tasks, alongside the general-purpose prompt above. Install one with chalk install agent-skill: ``` chalk install agent-skill chalk-notebooks --claude ``` That writes the skill to .claude/skills/ in the current directory. Pass --global to install it at the user level instead, so it applies in every project. The same command serves other agents with --codex or --cursor, and --delete removes a skill you no longer want. | Skill | Use it for | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `writing-online-resolvers` | Writing, refactoring, and debugging `@online` Python resolvers, including input and output signatures. | | `writing-static-chalkdf` | Writing `@online(static=True)` resolvers with chalkdf, and porting pandas or numpy logic to static acceleration. | | `chalk-resolver-acceleration` | Reducing feature-pipeline latency, and deciding whether to accelerate a Python resolver or migrate it to inline expressions. | | `chalk-streaming` | Writing, testing, and tuning stream resolvers against Kafka, Kinesis, or PubSub. | | `migrating-features-to-chalk` | Porting existing features into Chalk from dbt models, Airflow or Prefect pipelines, and Spark jobs. | | `chalk-notebooks` | Creating, running, and debugging [Chalk Notebooks](/docs/chalk-notebooks), including the CLI and installed dependencies. | Skills live alongside the prompts in that repository, under skills/. # Frequently Asked Questions source: https://docs.chalk.ai/docs/faq ## A collection of questions frequently asked by customers ### Getting support The questions are sorted by category. Don't see your question answered? Please reach out via your support channel and we will be happy to help! ### Data Sources and Infrastructure ### What is the difference between the online store and the offline store? The online store is intended to store features for low-latency retrieval in online query. Typically, the online store is implemented using Redis or DynamoDB. The offline store is intended to store historical logs of all previously ingested or computed features. It is used to compute large historical training sets. It is typically implemented using BigQuery, Snowflake, Iceberg, or other data warehouses. ### Can we do RBAC (Role Based Access Control) within Chalk? Yes! Within the dashboard you can assign roles with different permissions to different users. The default roles available are shown below. ### What are the necessary steps for us to get Chalk in our system? Please reach out via your support channel and we'd be happy to walk you through how to get Chalk setup running on your cloud infrastructure! ### We use Okta SCIM provisioning. Can we import these roles to be used in Chalk? Yes! You can set up Okta to automatically provision and deprovision Chalk users. For more information on how, see here. ### How should I set up secrets to be accessible in our deployed environments? You can configure secrets in the Secrets tab of Settings in your Chalk dashboard, and these secrets would then be available on the next deploy in the environments that you've specified, and on all branches in those environments as Python environment variables in the resolver runtime! ### Features ### Does Chalk have a feature catalog? Yes! You can view all the features for all namespaces deployed in your environments, along with some metadata on recent activity and updates. ### How should I use feature versioning? You can define versions for features if, for example, you are iterating on feature definitions. We'd recommend setting a default feature version for ensuring consistency across feature references. For more information, please see this tutorial on feature versions ### How does FeatureTime work? Should I override FeatureTime with timestamps from existing data sources? Feature time is the time at which a feature was observed. By default, feature time is set to the feature's resolver execution time. You can override feature time and may want to do so if you're ingesting historical data. For more information on how feature times work, see our time documentation. ### How should I structure my windowed features? What kind of windows can I use? We recommend writing windowed features for aggregated computations over different time periods. You can define the different windows in terms of weeks, days, hours, minutes, or even seconds (weeks="w", days="d", hours="h", minutes="m", seconds="s"). Under the hood, we normalize the time representations into durations in seconds, so you would have multiple syntactical options for accessing different windows for features. We recommend setting default values for windowed features in case there are windows with no events during the specified time period. Otherwise, you can write resolvers defining how your windowed features should be computed and pass windowed features as inputs into resolvers just like normal features! ### Can I upload features into the online store with an API endpoint? Yes! In addition to streaming and scheduled bulk ingests of features, you can submit requests using the upload_features SDK endpoints to synchronously ingest features into the online or offline stores using API clients. ### Resolvers ### What is the difference between an online and offline resolver? Not much! The only difference is in which contexts the resolvers are executed. A few key scenarios: - in "online query" (i.e. queries submitted via .query, .query_bulk, multi_query), offline resolvers never execute. - in "offline query" (i.e. queries submitted via .offline_query), offline resolvers are preferred, taking precedence over online resolvers that compute the same features. - in offline query, online resolvers are permitted to execute @offline is intended to be used for resolvers whose backing data sources are too slow or expensive to fulfill online query requests -- i.e. data warehouses or certain API sources, but if your query requests can tolerate the latency of one of these slow sources, you can mark resolvers using those datasources @online to query them on-the-fly. On the other hand, if your query requests can't tolerate the latency of the underlying data store, evaluate using scheduled ingestion or streaming resolvers in order to ensure that fresh data is available in the online store. ### How do I force certain resolvers to execute in my query? Make use of tags=[...]. If a resolver is marked with a tag, it is only eligible to execute in queries that have a matching tag. An even stricter variant of this concept is available with the required_resolver_tags argument to query and offline_query which allows you to force all resolvers to have a particular tag. ### How do I know when my resolver will run and when it will time out? You can set cron schedules for your resolvers, which will be parsed in your local system's timezone. You can also set customized timeouts for resolvers. Chalk currently has a max timeout of 18 hours for resolvers, but if this does not suit your needs, please reach out to us in your support channel with a description of your compute needs for your resolver! ### How do I know if my resolver is still running? The best way to check if your resolver is still running is to see if there are still resources provisioned to run your resolver. For scheduled resolver runs, you can view an overview of Run History as well as a view of Cloud Resources dedicated to scheduled resolver runs under the Runs tab on the menu sidebar. For triggered resolver runs, you can also find which pod is running your resolver on the run page. Finally, if you navigate to Infrastructure > Resource Configuration, you can view all pods currently running in your cluster, whether for resolver runs or queries. ### How should I work with dataframes in my resolvers? You can define resolvers with DataFrame inputs or outputs, which uses the Chalk DataFrame structure. You can do some aggregations and projections using the Chalk DataFrame, but you can also convert the Chalk DataFrames into Pandas or Polars using .to_pandas() and .to_polars() for more data manipulation. ### Querying ### Can I query for two different feature classes in a single query? We recommend creating a "root" feature class that models the interaction between the two entities -- i.e. if need "customer" and "business" features for a transaction fraud model, you might create a class like this: ``` @features class AuthQuery: # A unique ID that represents this interaction; may be randomly generated if you have # no natural ID in your system. id: str # A reference to the relevant customer. customer_id: "Customer.id" customer: Customer # A reference to the relevant business. business_id: "Business.id" business: Business ``` Then, queries can be submitted using: ``` ChalkClient().query( input={ AuthQuery.id: ..., AuthQuery.customer_id: ..., AuthQuery.business_id: ..., }, output=[ AuthQuery.customer, AuthQuery.business, ] ) ``` ### How do I query for multiple entities at the same time? Use the .query_bulk SDK method instead of .query. For example, to query features for multiple Hospital entities, you might use: ``` result = ChalkClient().query_bulk( input={Hospital.id: [1,2,3,4,5]}, output=[Hospital.current_waiting_time, Hospital.has_trauma_bay, Hospital.has_er, Hospital.has_mri] ) df = result[0].to_pandas() ``` ### How do I tell if my offline query is still running if it shows as In Progress in the UI? The first thing to check is whether there is a pod in your cluster running your offline query. You can verify this by viewing the pods running under Infrastructure. If you're running an offline query from a notebook, then the polling may timeout even if the offline query is still running, so the best way to verify the status of your offline query is a combination of checking the query run in dashboard and the status of pods in the cluster. For more visibility, you can also add the run_asynchronously=True argument to ChalkClient.offline_query to explicitly run your offline query on an isolated worker so you can use the worker status as a query status. ### How do I sample instances of a feature class? You can query for random instances of a feature class by running an offline query and specifying the max_samples parameter. For instance, running the example query below will sample ten random users from the User feature class, returning the features specified by the output parameters: [User.id, User.name, User.age]. ``` from chalk.client import ChalkClient from datetime import datetime from src.feature_sets import User client = ChalkClient() dataset = client.offline_query( output=[User.id, User.name, User.age], max_samples=10, recompute_features=True ) df = dataset.get_data_as_pandas() ``` ### Deployment ### How are resources provisioned for my Chalk cluster, and can I modify the configuration? We have default resource configurations for general environments. You can modify the configuration for your project's cloud resources by modifying the specs under Infrastructure > Resource Configuration. You must hit Save and Apply Changes in order for your configuration changes to go through. If you are not sure how you should configure your cloud resources, please reach out to us in your support channel! ### Can I suspend or delete branches when they are not in use? You cannot currently suspend or delete branches, but stale branches do not consume resources, so there is no cost or performance impact from old branches. ### Observability and Testing ### Is my (resolver / query) running? See answers above under Resolvers and Querying respectively! ### How do I setup a sensor so that I can run something else after a resolver run? If you are triggering a resolver run as part of an orchestrated pipeline, we usually see customers using the built-in sensors from the orchestrators (e.g. Airflow or Dagster) to poll for resolver completion. You can also customize this polling using an API to query run status. See here for an example of how to set up Airflow orchestration to trigger and poll a resolver run. ### Roadmap ### What have the people at Chalk been working on? To get a glimpse of recently released features, take a look at our changelog! ### I want to request a feature! Feel free to reach out to us with feature requests or questions about feature requests in your support channel! # Features Overview source: https://docs.chalk.ai/docs/features ## Define features for training and inference. Chalk lets you spell out your features directly in Python. Features are namespaced to a FeatureSet. To create a new FeatureSet, apply the @features decorator to a Python class with typed attributes. A FeatureSet is constructed and functions much like Python's own dataclass. ### Example ``` from datetime import datetime from typing import Optional from chalk.features import features @features class User: id: int full_name: str nickname: Optional[str] email: Optional[str] birthday: datetime fraud_score: float ``` ### Namespacing Features are namespaced by their containing FeatureSet, and then by the name of the variable. In the above example, our features, when rendered as strings, are: | Feature Name | Type | | ---------------- | -------------- | | user.id | Integer | | user.full_name | String | | user.nickname | String \| None | | user.email | String \| None | | user.birthday | Datetime | | user.fraud_score | Decimal | (FeatureSet names are stripped of the suffix "Features", if it exists). ### Overrides Feature names and feature classes can be overridden by supplying the name keyword argument to the feature function or the @features decorator. This practice allows us to evolve our variable names without losing the past history of this feature. Use @features(name=...) to keep the same feature namespace while renaming the Python class: ``` - @features + @features(name="prince") - class Prince: + class TheArtistFormerlyKnownAsPrince: birthday: datetime ``` Use feature(name=...) to keep the same feature name while renaming the Python attribute: ``` from datetime import datetime from chalk.features import feature, features @features class User: - birthday: datetime + date_of_birth: datetime = feature(name="birthday") ``` ### Primary keys Feature sets must all have a primary key. This primary key is used to index storage for features you later resolve in this feature class. Your primary key can have type int or str, given by the type annotation on the field. By default, if you have a feature with the name id, that feature will be the primary key. However, you can override this behavior: ``` from chalk.features import features, Primary @features class User: user_id: Primary[str] ... ``` If you mark an explicit primary key, it will override the default behavior: ``` @features class User: user_id: Primary[str] # Not really the primary key! id: str ``` Alternatively, you can use the feature function to set a feature to primary: ``` from chalk.features import features, feature @features class User: user_id: str = feature(primary=True) ``` ### Versions Chalk versions all of your features with every deployment. However, you can also choose explicit versions for your features. ``` @features class User: ... email_domain: str = feature(version=2) ``` ### Feature time By default, Chalk marks the time a feature was created as the time that its resolver was run. However, you may want to provide a custom value for this time for data sources like events tables. You can inspect the time a feature was created and set the time for when a feature was created by creating a feature of type FeatureTime. To set the time a feature was created, assign the feature when you resolve it: ``` @offline def fn(uid: User.uuid) -> Features[User.name, User.ts]: return User( name="Anousheh Ansari", ts=datetime(month=9, day=12, year=1966) ) ``` Then, when you sample offline data, the name feature will be treated as having been created at the provided date. ### Constructing feature classes To construct a User instance, supply the feature values to the __init__() method ``` User(full_name="Grace Hopper", nickname="Amazing Grace") User(email="grace.hopper@yale.edu") ``` The @features decorator adds a custom __init__(): ``` def __init__( self, uid: int | MISSING = MISSING, full_name: str | MISSING = MISSING, email: Optional[str] | MISSING = MISSING, ... ): self.uid = uid self.full_name = full_name self.email = email ... ``` Note that all fields have a default MISSING value. Therefore, you can construct feature classes with any subset of the fields you would like to use. ### Refactoring After going to production, you may find that you want to change the name of a property on the feature class. You can change the name of a feature property without changing the underlying data using the name override. From the example in the namespacing section, if you initially called a feature birthday, and decided to rename it date_of_birth, you can keep the underlying data the same and rename the property on the class as follows: If you rename the feature class itself, keep the namespace stable with @features(name=...): ### Interplay with auto-id Where the name of the Python property and the name provided to feature(name=...) differ, IDs are auto-assigned based on the name provided to feature(name=...). ### Default feature values For features that can't always be computed, you can pass default to feature or assign a default directly: ``` from chalk.features import features @features class User: id: int + num_purchases: int = 0 + count_logins: int = feature(default=0) ``` # Feature Types source: https://docs.chalk.ai/docs/feature-types ## Define features for training and inference. ### Scalars Features can be any primitive Python type: ``` from enum import Enum class Genre(Enum): FICTION = "FICTION" NONFICTION = "NONFICTION" DRAMA = "DRAMA" POETRY = "POETRY" @features class Book: id: int name: str publish_date: date copyright_ended_at: datetime | None genre: Genre ``` In addition to the primitive Python types (int, float, str, bool), Chalk also supports date, datetime, and the Chalk FeatureTime as scalar feature types. ### Lists and Sets Features can be a list or set of scalar or struct types. ``` @dataclass class Chapter: start_page: int end_page: int @features class Book: authors: list[str] categories: set[str] chapters: list[Chapter] ``` ### Structs Features can be a struct, which is a collection that maps a fixed set of keys to sub-fields. You can use dataclasses, Pydantic models, and attrs classes to represent struct features. Struct features can be used recursively within list features or other struct features. Struct types should be used for objects that don't have ids. If an object has an id, consider using has-one. ### Dataclass You can use any dataclass as a struct feature. ``` @dataclass class JacketInfo: title: str subtitle: str body: str @features class Book: id: int jacket_info: JacketInfo ``` ### Pydantic models Pydantic models can also be used for structs. ``` from pydantic import BaseModel, constr class TitleInfo(BaseModel): heading: constr(min_length=2) subheading: Optional[str] @features class Book: title: TitleInfo ... ``` ### Attrs Alternatively, you can use attrs. ``` import attrs @attrs.define class TableOfContentsItem: foo: str bar: int @features class Book: table_of_contents: list[TableOfContentsItem] ... ``` ### Document Both dataclass and Pydantic structs are implemented using the PyArrow serialization format, a high-performance schema for data serialization. This data is stored "value only", i.e. without keys, so any change to these structs over time will invalidate historical data. To support feature values where the schema changes over time, we introduced the Document struct type. Documents are serialized as JSON and supports changes to schema over time, at the cost of a small performance penalty. ``` from pydantic import BaseModel from chalk import Document class AuthorInfo(BaseModel): first_name: str last_name: str @features class Book: title: Document[AuthorInfo] ... ``` ### Vectors Features can be vectors (fixed sized arrays of floats), such as the output from an embedding model. Unlike list or set features, vector features are compatible with embedding functions and nearest neighbor similarity search. ``` from chalk.features import features, Vector @features class Document: embedding: Vector[1536] # Defines a vector with 1536 dimensions ``` When using the built-in embedding functions, then the vector dimensions don't need to be specified, as Chalk will automatically infer it from the embedding model. ``` from chalk.features import embed, features, Vector @features class Document: content: str # Chalk knows that text-embedding-ada-002 has 1536 dimensions embedding: Vector = embed( input=lambda: Document.content, provider="openai", model="text-embedding-ada-002" ) ``` By default, vectors are persisted as 16 bit floats for efficiency. However, Chalk also supports persisting vectors as 32-bit or 64-bit floats via the keywords "f32" and "f64", respectively. ``` from chalk.features import Vector @features class Document: # Defines a vector with 1536 dimensions that will be persisted with 32 bit precision embedding: Vector["fp32", 1536] ``` ### Custom serializers Finally, if you have an object that you want to serialize that isn't from dataclass, attrs, or pydantic, you can write a custom codec. This custom codec must target a type that can be serialized to a PyArrow data type, which is the underlying serialization format for all features. Consider the custom class below: ``` class CustomStruct: def __init__(self, foo: str, bar: int) -> None: self.foo = foo self.bar = bar def __eq__(self, other: object) -> bool: return ( isinstance(other, CustomStruct) and self.foo == other.bar and self.bar == other.bar ) def __hash__(self) -> int: return hash((self.foo, self.bar)) ``` Here, we use the custom class as a feature, and provide an encoder and decoder. The encoder takes an instance of the custom type and outputs a Python object, and the decoder takes output of the encoder and creates an instance of the custom type ``` @features class Book: custom_field: CustomStruct = feature( encoder=lambda x: dict(foo=x.foo, bar=x.bar), decoder=lambda x: CustomStruct(**x), ) ``` # Discovery source: https://docs.chalk.ai/docs/feature-discovery ## Declare owners and metadata for features. Features can capture metadata to inform alerting, monitoring, and discovery. By default, features are created as any type-annotated variable of a class decorated with @features. However, you can also assign these variables to the result of the chalk.features.feature(...) function to provide metadata. ### Description Descriptions are parsed from the comments preceding the feature definition. For example, you can document a fraud_score feature with information about the values as follows: ``` @features class User: # **0 to 100 score indicating an identity match.** # **Low scores indicate safer users** fraud_score: float ... ``` You can alternatively provide a description of a feature directly in the code. The following example is equivalent to explicitly providing the description as above: ``` @features class User: fraud_score: float = feature(description=""" **0 to 100 score indicating an identity match.** **Low scores indicate safer users** """) ... ``` If both an explicit description and a comment are present, the description is set to the explicit value from feature(description=...). You can programmatically access the description for a feature with the description(...) function: ``` from chalk.features import description print(description(User.fraud_score)) ``` ### Owner You can also specify which person or group is responsible for an individual feature. The owner tag is available in Chalk's web portal. Alerts that do not otherwise have an owner are assigned to the owner of the monitored feature. Owners are parsed from the source code. For example: ``` @features class User: full_name: str # :owner: **katherine.johnson@nasa.gov** fraud_score: float ... ``` You can instead specify an owner via a keyword argument to feature(). For example: ``` @features class User: full_name: str fraud_score: float = feature(owner="**katherine.johnson@nasa.gov**") ... ``` An owner can also be assigned to every feature in a namespace via a keyword argument to @features: ``` @features(owner="**katherine.johnson@nasa.gov**") class User: full_name: str fraud_score: float # :owner: **annie.easley@nasa.gov** email: str ... ``` Here, User.full_name and User.fraud_score assume the owner katherine.johnson@nasa.gov. However, User.email, which specifies an owner at the feature level, assumes the owner annie.easley@nasa.gov. You can programmatically access the owner of a feature with the owner(...) function: ``` from chalk.features import owner assert owner(User.email) == "annie.easley@nasa.gov" ``` ### Tags Tags are a way of adding metadata to features for use in filtering, aggregations, and visualizations. Feature tags do not select resolvers or datasource routes. When used with RBAC, they control access through a permission-tag mapping rather than routing. For example, you can use tags to assign features to a team and find all features for a given team. ``` @features class User: # :tags: **team:identity**, **priority:high** fraud_score: float ... ``` Alternatively, you can specify tags via explicit construction: ``` @features class User: fraud_score: float = feature(tags=[ "**team:identity**", "**priority:high**", ]) ... ``` As with the owner property, tags can be assigned to all features in a namespace: ``` @features(tags="**group:risk**") class User: fraud_score: float # :tags: pii email: str ... ``` Here, User.fraud_score inherits the tag group:risk. The feature User.email also inherits this tag in addition to the tag pii. You can programmatically access the tags for a feature with the tags(...) function: ``` from chalk.features import tags assert tags(User.email) == ["pii", "group:risk"] ``` ### Semantic search Chalk provides two ways to search. Global search matches the text you type against names. Chalk Assistant provides semantic search: you describe what you want in plain language, and the Assistant finds it, even when you do not know the exact name. For example, you can search for “user risk features” or “features updated last week” or “features owned by data team”. ### Global search Press Cmd+K (Mac) or Ctrl+K (Windows/Linux) anywhere in the Chalk dashboard, or select the search bar in the header, to open global search. Each tab searches one kind of result: - All: results from every tab - Pages: dashboard pages - Features: features in the current environment - Resolvers: resolvers in the current environment - Source code: source files - Environments: projects and environments ### Semantic search with Chalk Assistant To use semantic search, send your query from global search to Chalk Assistant: - Type your query in the search box. - Press Shift+Enter. Global search switches to a prompt box that contains your query. - Press Enter to send it. Chalk Assistant opens in the sidebar and answers. In the prompt box, Shift+Enter adds a new line and Esc returns to search. Once you type a query, the footer of the search window shows a Shift ↵ Ask AI hint. The hint and the shortcut appear only when Chalk Assistant is enabled for your environment. See Enabling Chalk Assistant for setup. ### Data lineage Chalk tracks how every feature produced by a SQL File Resolver is computed, including which datasource and table supplied the raw data, which resolver transformed it, and which queries consume it. See Data Lineage for the full reference. # Data Lineage source: https://docs.chalk.ai/docs/data-lineage ## Explore how features are computed and what they depend on, with views by feature, resolver, and source column. Data lineage tracks the complete path from a raw data source to a computed Chalk feature. For every feature produced by a SQL File Resolver, Chalk records which datasource and table supplied the raw data, which resolver transformed it, and which queries consume it as input or output. The Data Lineage page in the dashboard makes that graph browsable. Data lineage is available for features produced by SQL File Resolvers. Features computed by Python resolvers do not appear. To open the page, select Catalog › Data lineage in the project sidebar. ### List view The list view presents lineage as a multi-column drill-down. You start from data sources on the left and progressively narrow to a specific feature on the right. Selecting a feature opens a detail panel that shows: - Resolved from columns: the source column names from the SQL resolver's query - Output of queries: named queries that list this feature as an output - Input to queries: named queries that use this feature as an input - A link to the resolver that produces it Use the Group by dropdown in the toolbar to choose among three views, each of which adds a different intermediate column to the drill-down. ### Feature view Data Sources → Tables → Features The default view. Select a datasource, then a table, then a feature. Use this when you want to answer "given this datasource and table, what Chalk features does it feed?" ### Resolver view Data Sources → Tables → Resolvers → Features Adds a Resolvers column between Tables and Features. Select a resolver to see only the features that particular resolver produces from the selected table. If the selected feature is also produced by other resolvers, those appear in an Other Source Resolvers section of the detail panel. Use this when you want to understand what a specific resolver contributes, or when multiple resolvers read from the same table and you need to separate their outputs. ### Column view Data Sources → Tables → Columns → Features Adds a Columns column that lists the raw source column names from the SQL query. Select a column to see which Chalk features are derived from it. Use this when you need to trace a specific raw database column forward to the Chalk features it affects — for example, to assess the impact of a schema change in a source system, or to document which features carry data from a particular field for compliance purposes. ### Graph view The graph view renders the same lineage data as a directed acyclic graph. Nodes flow left to right: datasources, datasource tables, resolvers, features, and scheduled queries. Selecting a feature scopes the graph to the subgraph reachable from that feature, hiding unrelated nodes. To open the graph for a feature, select it from the Select a Feature dropdown in the top-right corner of the graph tab. Each selected feature opens in its own tab; use the + button to open additional tabs and compare features side by side. Tabs are closeable and persist as long as the page is open. Click any node in the graph to see its detail panel on the right side of the page. If the banner reads N scheduled queries not shown, those queries have no retained query plan from their most recent run and cannot be placed in the graph. This does not affect the list view. ### Use cases Impact analysis — before modifying a resolver or changing a source table's schema, use the Feature or Column view to see which downstream features and queries would be affected. Debugging — trace a feature back through its resolver and source columns to identify where unexpected values may be introduced. Compliance — document which raw data fields feed a given feature, and which queries consume it, to satisfy data provenance requirements. Optimization — identify resolvers that read from the same tables and evaluate whether their queries can be consolidated. # Feature Caching source: https://docs.chalk.ai/docs/feature-caching ## Cache pre-computed feature values in the online store When a feature is expensive or slow to compute, you may wish to cache its value in the online store. Chalk uses the terminology "maximum staleness" to describe how recently a feature value needs to have been computed for the value in the online store to be returned rather than recomputing a fresh feature value by running a resolver. The online store is typically best suited for low-latency reads on a smaller amount of data relative to the offline store. You can specify the maximum staleness for a feature as follows: ``` from chalk.features import feature, features from datetime import timedelta @features class User: id: int # Using text descriptors: expensive_fraud_score: float = feature( max_staleness="1m 30s" ) # Alternatively, using timedelta: expensive_fraud_score: float = feature( max_staleness=timedelta(minutes=1, seconds=30) ) ``` Max staleness durations can be given in natural language, or specified using datetime.timedelta. You can specify a max staleness of "infinity" to indicate that Chalk should cache computed feature values forever. This makes sense for data that never becomes invalid, or for data that you wish to explicitly update using Streaming Updates or Reverse ETL. Staleness can also be assigned to all features in a namespace: ``` from chalk.features import features, feature @features(max_staleness="**1d**") class User: id: int fraud_score: float full_name: str email: str = feature(max_staleness="0s") ... @features(max_staleness="infinity") class UserReport: user_id: Primary[User.id] report_data: str created_at: datetime ... ``` Here, User.fraud_score and User.full_name assume the max-staleness of 1d. However, User.email, which specifies max-staleness at the feature level, assumes the max-staleness of 0s, forcing it to be recomputed on every request. By default, features are not cached, and instead are recomputed for every online request. In effect, you can think of max_staleness as being 0 except where otherwise specified. ### Caching indefinitely If you want to cache a feature indefinitely, you can set the max_staleness to "infinity": ``` from chalk.features import feature, features @features class User: id: int fraud_score: float = feature(max_staleness="infinity") ``` In this example, the latest computed value for User.fraud_score will be cached indefinitely in the online store. When a new value is computed, it will replace the old value in the online store. ### Populating the Online Store Once you have set the max staleness for a feature, there are several ways to populate the online store, depending on whether you want to just cache recently computed feature values or if you want to ensure that your queries utilize the low-latency path of online store lookup. - Online Queries: When you run an online query, the Chalk engine will check the online store for feature values that fall within the max staleness duration for a feature. If a feature value is found then the engine will return that value. Else, the engine will run the associated resolver to compute a fresh feature value and store the newly computed feature value in the online store. - Offline Queries: When you run an offline query with the parameter store_online=True, the feature values computed in the offline query output will be loaded into the online store. - Triggered Resolver Run: You can trigger a resolver run with the parameter store_online=True to populate the online store with the feature values computed in the resolver run. - Scheduled Queries: You can create a ScheduledQuery with the parameter store_online=True to populate the online store with the feature values computed in the scheduled query. - Dataset Ingest: You can ingest a Dataset to the online store using the method dataset.ingest(store_online=True). - ETL Offline to Online: You can set the etl_offline_to_online parameter to True in an @online or @offline scheduled resolver to populate the online store with the feature values computed in the resolver for features with max_staleness != 0. - Stream: You can stream feature values to the online store for features with max_staleness != 0. ### Handling null and default values For features with max_staleness != 0, you can also specify how you want to handle null and default feature values. By default, Chalk will cache the computed feature value, even if it is null or the default value, however if you set the cache_null or cache_default parameter to False, Chalk will not cache the null/default computed feature value. Furthermore, if your online store is Redis or DynamoDB, you can also set cache_nulls="evict_nulls" to evict cached null feature values from the online store, and cache_defaults="evict_defaults" to evict cached default feature values from the online store. Read more about how to handle null and default value caching here. ### Caching Cookbook Given all of these options, which recipe should you follow for what data to cache in the online store and how to load that data into the online store? First, there are a few general principles to keep in mind: - The online store is optimized for low-latency reads, so you should cache data that you want to query quickly, usually for real-time use cases. - The online store is not optimized for huge amounts of data, so you should cache only the data that you need to query quickly. - When you run an online query, if the output feature values are not found in the online store, the online resolver will run. When you run an offline query, if the output feature values are not found in the offline store, an offline resolver will run, or if there is no offline resolver then the online resolver will run. Keeping these principles in mind, here are some common use cases and how to handle them: - If you know the specific primary keys for the data that you would like to query quickly, you canRun offline queries or dataset ingest to load data ad-hoc: Given the set of feature values that you would like to load into the online store, you can run offline queries with store_online=True or ingest a dataset to the online store using dataset.ingest(store_online=True).Schedule or orchestrate queries and ingests: If you would like to regularly update the feature values in the online store, you can also orchestrate the offline queries and dataset ingests to run at specific intervals, or use a ScheduledQuery. - If you would like to cache all possible values for a feature in the online store, you canTrigger a resolver run: A resolver run with store_online=True will populate the online store with all possible feature values for the resolver's output features. - If you would like to cache recently computed data, but do not have a specific concept in mind of what data you would like to cache, you canRun online queries: Running online queries will populate the online store with the feature values that fall within the max staleness duration for the feature.Set etl_offline_to_online in your resolvers: If you have resolvers that compute features with max_staleness != 0, you can set etl_offline_to_online=True to populate the online store with the feature values computed in the resolver run. - If you have a streaming data sourceStream feature values to the online store: You can stream feature values to the online store for features with max_staleness != 0. ### Overriding default caching The max_staleness values provided to the feature function may be overridden at the time of querying for features. See Overriding Default Caching for a detailed discussion. ### Specifying Online and Offline Storage Chalk enables granular configuration about whether and how to store feature values in the online and offline stores. For each feature class, the @features decorator can be used to specify whether the scalar features in the feature class should be cached in the online store based on the max_staleness parameter. However, the feature function enables per-feature specification of whether values computed for that feature within a feature class should be stored in the online and offline stores using the store_online and store_offline parameters. The following example showcases how these parameters can be used in combination to express different storage behavior. ``` from chalk.features import features, feature, DataFrame from datetime import datetime # due to the max_staleness parameter, all scalar features in the Driver feature class will be cached in the online # store with a max staleness of 30 days. In addition--all of the computed feature values will be stored offline by # default. @features(max_staleness="30d") class Driver: id: int # with no overrides, this feature will be stored online with max staleness of 30 days, and stored offline name: str # with an override on store_offline, this feature will be stored online with a max staleness of 30 days, but # will not be stored offline age: int = feature(store_offline=False) # with overrides for store_online and store_offline, computed feature values for location will not be persisted # in either the online or offline store location: str = feature(store_online=False, store_offline=False) # joined features are not scalar features, and hence do not inherit the storage behavior of the feature class. # because neither the Job feature class nor the jobs: "DataFrame[Job]" # because the Record feature class has a max staleness of 30 days, and so does the join, this join # would be cached on the Driver feature class records: "DataFrame[Record]" = has_many(lambda: Record.driver_id == Driver.id, max_staleness="30d") # with no max_staleness set, features in the Job feature class will by default by stored offline, but not online @features class Job: id: int # with no overrides, this feature will be stored offline, but not online driver_id: Driver.id # with an override for store_offline, computed feature values for start_time will not be persisted start_time: datetime = feature(store_offline=False) # with overrides for store_online and store_offline, computed feature values for end_time will be persisted # in the online store, but not the offline store end_time: datetime = feature(store_online=True, store_offline=False) # with max_staleness set, all scalar features in the Record feature class will be cached in the online store with a # max staleness of 30 days and stored offline by default. @features(max_staleness="30d") class Record: id: int # with an override on max_staleness, all other features in the feature class would be cached with a max # staleness of 30 days, but driver_id would be cached with a max staleness of 1 day. driver_id: Driver.id = feature(max_staleness="1d") timestamp: datetime record_details: str ``` ### Removing Cached Feature Values You can remove cached feature values from the online store in two ways: for an entire feature (all primary keys), or for specific primary keys. Both are available from the CLI and from the Python SDK. By default, both operations remove values from the online store and the offline store. If you only want to clear the online cache and keep your historical (offline) data, use the client's retain_offline flag, shown below. ### Removing all cached values for a feature When you change how a feature is computed (for example, its resolver logic), the online store keeps serving the previously cached values until they expire. To force Chalk to recompute fresh values on the next query, remove the feature's cached values so that none of them remain eligible to be served. From the CLI, use chalk migrate-type: ``` chalk migrate-type --features user.risk_score_1 ``` From the Python SDK, use drop_features: ``` from chalk.client import ChalkClient ChalkClient().drop_features( namespace="user", features=["risk_score_1"], ) ``` With either approach, you must redeploy with chalk apply afterward for the removal to take effect. Both operate on the feature you name. If a feature is computed from other features, remove the input feature that it depends on to reset its computed values. To clear only the online cache while keeping your offline history, pass retain_offline=True: ``` ChalkClient().drop_features( namespace="user", features=["risk_score_1"], retain_offline=True, ) ``` ### Removing cached values for specific primary keys If you want to remove an incorrectly computed value for one or a few primary keys instead of the whole feature, target those keys directly. From the CLI, use chalk delete: ``` chalk delete --keys=123 --features user.risk_score_1 ``` From the Python SDK, use delete_features: ``` from chalk.client import ChalkClient ChalkClient().delete_features( namespace="user", features=["risk_score_1"], tags=None, primary_keys=["123"], ) ``` Instead of naming features to delete, you can select them by feature tag. Both chalk delete and delete_features also accept retain_offline and retain_online to scope the deletion to a single store. ### Garbage Collection When a cached feature value expires (i.e. its age exceeds the configured max_staleness), Chalk immediately stops serving it. Subsequent queries will recompute the feature by running the appropriate resolver rather than returning the stale value. However, the expired data is not immediately deleted from the underlying storage — actual cleanup of expired data happens asynchronously according to the garbage collection strategy of each storage backend: | Store | Garbage Collection Strategy | Details | | -------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Redis / Valkey | Periodic background job | Chalk runs a periodic background job that scans for and removes expired feature values. Redis also natively evicts keys with expired TTLs via its lazy and active expiration mechanisms. | | DynamoDB | TTL-based expiration | DynamoDB automatically deletes items with expired TTL attributes. Deletions typically occur within 48 hours of expiration and are performed by a background process at no additional cost. | For Redis and Valkey, the periodic garbage collection job ensures that feature values that are no longer eligible to be served (e.g. after a chalk migrate-type) are cleaned up, freeing memory for active feature data. For DynamoDB, garbage collection is handled natively by the storage engine using TTL and compaction-based mechanisms respectively. ### LRU Caching Chalk also supports tiered caching using an LRU cache in front of the online store. To enable LRU caching, you can create an OnlineStoreConfig with an lru_cache, and then pass that configuration to the feature classes that you want to cache: ``` from chalk.features import features from chalk.stores import LRUCache, OnlineStoreConfig lru_cache_config = OnlineStoreConfig( lru_cache=LRUCache( max_size=10000, ttl="60s", store_cache_misses=True, ) ) @features(online_store_config=lru_cache_config) class User: id: int risk_score: float = feature(max_staleness="1h") ``` In this example, the User feature class is configured to use an LRU cache with a maximum size of 10,000 entries. Every worker process will maintain its own LRU cache. If a feature value is found in the LRU cache, it will be returned directly. If it is not found in the LRU cache, the online store will be queried. If it is not found in the online store, the resolver will be run to compute a fresh value. If the store_cache_misses parameter is set to true, then a cache miss against the online store is stored in the LRU cache: subsequent requests for the same primary key will not hit the online store until the TTL of the LRU cache for the pkey expires. # Has One source: https://docs.chalk.ai/docs/has-one ## Define one-to-one relationships between feature classes. Has-one relationships link a feature to a single instance of another feature. ### Recommended Implementation The simplest way to specify a join for a has-one relationship is implicitly. In the example below, a User is linked to their Profile. ``` from chalk.features import features @features class Profile: id: str user_id: "User.id" email_age_years: float @features class User: id: str profile: Profile ``` With a has-one relationship established, you can reference features on Profile through User. For example: ``` user_email_age = User.profile.email_age_years ``` ### Explicit Join In the following snippet, the has-one join is explicitly defined. This is functionally equivalent to the recommended implementation: ``` from chalk.features import features, has_one, ... @features class Profile: id: str user_id: str email_age_years: float @features class User: id: str uid: str profile: Profile = has_one(lambda: Profile.user_id == User.uid) ``` The lambda solves forward references, letting you reference User before it is defined. ### Composite Join Keys You can also specify a composite join key for a has-one relationship. For example, if a User is linked to a Profile by org and email, you can define the join as follows: ``` from chalk.features import features, has_one from datetime import datetime @features class User: id: str email: str = _.alias + "-" + _.org + _.domain org_domain: str = _.org + _.domain org: str domain: str alias: str # join with composite key posts: DataFrame[Posts] = has_many(lambda: User.email == Post.email) # multi-feature join org_profile: Profile = has_one(lambda: (User.alias == Profile.email) & (User.org == Profile.org)) @features class Workspace: id: str # join with child-class's composite key users: DataFrame[Users] = has_many(lambda: Workspace.id == User.org_domain) ``` ### Back-references ### One-to-one You can also add a back-reference to User from Profile. However, you don't have to explicitly set the join on Profile. Instead, the join condition is assumed to be symmetric and copied over. To complete the one-to-one relationship from our example, add a User to the Profile class: Here you need to use quotes around User to use a forward reference. ### Optional relationships When a has-one relationship is specified, the default behavior is to treat the linked Feature as required. Following the example above, specifying a User without a Profile and querying for a User's profile or using the User.profile in a resolver raises an error. To define optional relationships, use the typing.Optional[...] keyword: Note, resolvers that take optional features as inputs need to handle the None case. This is covered in more detail in the resolver's section of the docs. ### Chained Has-One Joins You can chain has-one joins to traverse multiple relationships. For example, you could define the following features to represent a user's profile and preferences in an application. ``` from chalk.features import features, Primary @features class User: id: str email: str @features class Profile: id: Primary[User.id] username: str @features class Preferences: id: Primary[Profile.id] dark_mode: bool ``` ### Organizing Feature Definitions Across Files In large projects, it's common to split feature definitions across multiple Python modules. For unidirectional dependencies, this is straightforward. For example, if src/models/user.py imports src/models/profile.py, you can define the User and Profile features in separate files without issues. However, if you have circular dependencies, you may run into problems. Chalk supports this, but circular imports can arise when features reference each other across files. To avoid these issues, use the if TYPE_CHECKING block from the typing module and quote your forward references. Here's an example of how to do this cleanly: ``` # Imports `User` directly, because `src/models/user.py` # doesn't import `src/models/profile.py` from chalk.features import features from src.models.user import User @features class Profile: id: Primary[User.id] username: str ``` ``` from chalk.features import features from typing import TYPE_CHECKING if TYPE_CHECKING: # Imports `Profile` only when type checking # to avoid circular imports from src.models.profile import Profile @features class User: id: str # Profile must be quoted because it is imported # only when type checking profile: "Profile" ``` By quoting imports inside if TYPE_CHECKING, you avoid circular dependency errors while still maintaining type safety and feature linkage. If the relationship to Profile is optional, you can use typing.Optional or the | None syntax, but the entire annotation should be quoted: ``` from chalk.features import features from typing import TYPE_CHECKING if TYPE_CHECKING: from src.models.profile import Profile @features class User: id: str # All of `"Profile | None"` must be quoted, not just the `Profile` part profile: "Profile | None" ``` ### Querying for Has-One Relationships You can also query for a feature that is joined through a has-one relationship by referencing the root namespace. For example, to query for the Profile features associated with a User continuing from the example above, you can write: ``` from chalk.client import ChalkClient from src.features import User client = ChalkClient() client.query( input={User.id: "1"}, output=[User.profile.id, User.profile.email_age_years] ) ``` # Has Many source: https://docs.chalk.ai/docs/has-many ## Define one-to-many and many-to-many relationships between feature classes. Has-many relationships link a feature to many instances of another feature. ### Foreign Keys The recommended way to specify a join for a has-many relationship is implicitly. In the example below, a User is linked to potentially multiple Transfers. ``` from chalk.features import features, DataFrame @features class Transfer: id: str # note, the annotation must be a string reference because User is # defined after Transfer. user_id: "User.id" amount: float @features class User: id: str transfers: DataFrame[Transfer] ``` ### Explicit Join The following example, which explicitly sets the join, is equivalent to the above: ``` from chalk.features import has_many, DataFrame @features class Transfer: id: str user_id: str amount: float @features class User: id: str transfers: DataFrame[Transfer] = has_many(lambda: Transfer.user_id == User.id) ``` ### Composite Join Keys You can also specify multiple join keys for a has-many relationship. For example, say hospitals want to compute aggregations over visits to each of their departments. We could write the following feature classes. ``` from chalk import _ from chalk.features import features, DataFrame, has_many import chalk.functions as F from datetime import datetime @features class HospitalVisit: id: str = _.user_id + _.hospital + _.department + F.cast(_.date, str) composite_key: str = _.hospital + "-" + _.department department: str hospital: str user_id: str date: datetime @features class HospitalDepartment: id: int name: str hospital_name: str composite_key_match: str = _.hospital_name + "-" + _.name # multi-feature join visits: DataFrame[HospitalVisit] = has_many( lambda: (HospitalDepartment.hospital_name == HospitalVisit.hospital) & (HospitalDepartment.name == HospitalVisit.department) ) # composite key join visits_with_composite_key: DataFrame[HospitalVisit] = has_many( lambda: HospitalDepartment.composite_key_match == HospitalVisit.composite_key ) ``` You can also join in Has-Many relationships for features that have primary composite keys. ``` from chalk.features import features, DataFrame, has_many @features class SoftwareEngineer: id: int = _.first_name + " " + _.last_name first_name: str last_name: str manager_id: str @features class Manager: id: int direct_reports: DataFrame[SoftwareEngineer] = has_many( lambda: Manager.id == SoftwareEngineer.manager_id ) ``` ### Aggregations on References Having established a has-many relationship, you can now reference the transfers for a user through the user namespace. The has_many feature returns a chalk.DataFrame, which supports many helpful aggregation operations: ``` # Number of transfers made by a user User.transfers.count() # Total amount of transfers made by the user User.transfers[Transfer.amount].sum() # Total amount of the transfers made by the user that were returned User.transfers[ Transfer.status == "returned", Transfer.amount ].sum() ``` To compute an aggregation over one or more time windows, see our docs on windowed aggregations. ### Back-references ### One-to-many In the reverse direction, a one-to-many relation is defined by a has_one relation (following the above example, a user has many transfers but a transfer has a single user). However, you don't have to explicitly set the join a second time. Instead, the join condition is assumed to be symmetric and copied over. To complete the one-to-many relationship from our example, add a User to the Transfer class: Here, you need to use quotes around User to use a forward reference. ### Many-to-many The recommended way to define a many-to-many relationship is through a joining feature class. For instance, to define a many-many relationship between Actors and Movies, you could write the following feature classes: ``` from chalk.features import features, DataFrame @features class Actor: id: int appearances: "DataFrame[MovieRole]" full_name: str # this will be used to demonstrate one of the # ways the joining feature can be populated movie_ids: list[int] @features class Movie: id: int title: str @features class MovieRole: id: str actor_id: Actor.id movie_id: Movie.id movie: Movie ``` Here you need to use quotes around DataFrame[MovieRole] to use a forward reference. This joining feature class can be populated by a SQL file resolver: ``` -- resolves: MovieRole -- source: PG SELECT id, actor_id, movie_id FROM movie_roles; ``` Alternatively, by a DataFrame-returning Python resolver (namespaced to one of the joined feature sets): ``` @online def get_actor_in_movie( a_id: Actor.id, movie_ids: Actor.movie_ids, ) -> Actor.appearances: return DataFrame([ MovieRole( id=f"{a_id}_{m_id}", actor_id=a_id, movie_id=m_id ) for m_id in movie_ids ]) ``` The joining feature class lets you: - query for movie features from the Actor namespace, and - use movie features in downstream Actor resolvers. For example, to get the titles for all the movies that an actor has appeared in, you can run the following query: ``` $ chalk query --in actor.id=1 --out actor.appearances.movie.title Results Name Hit? Value ─────────────────────────────────────────────────────────────────────────────── actor.appearances.movie.title ["The Bad Sleep Well","High and Low",...] ``` # Composite Keys source: https://docs.chalk.ai/docs/composite-keys ## Model composite identifiers and joins in Chalk, including filter pushdown, SQL resolver outputs, and materialized aggregations. A composite key identifies a row or entity using more than one source value, such as workspace_id and user_id. In Chalk, you can model that relationship in two ways: - join on the individual component features, or - compute a single scalar feature that contains the composite key and join on that feature. Both patterns are valid. The important rule is that Chalk joins and filters on features, not on the implied parts of a computed string. If WorkspaceMember.id is computed from workspace_id and user_id, Chalk treats WorkspaceMember.id as its own feature. Chalk does not automatically split that value back into workspace_id and user_id later. This matters for both correctness and performance. The query must have the features needed to make the join possible, and large offline queries must also have join keys that can be pushed down safely so the warehouse can prune the data it scans. ### Component Joins Use a component join when both sides naturally expose the same component columns. This usually gives warehouses the clearest filter predicates because each component can be pushed down independently. ``` from chalk import DataFrame from chalk.features import features, has_many @features class ActivityEvent: id: str workspace_id: str user_id: str amount: float @features class WorkspaceMember: id: str workspace_id: str user_id: str activity_events: DataFrame[ActivityEvent] = has_many( lambda: (WorkspaceMember.workspace_id == ActivityEvent.workspace_id) & (WorkspaceMember.user_id == ActivityEvent.user_id) ) ``` If an offline query starts from WorkspaceMember.workspace_id and WorkspaceMember.user_id, Chalk can push those inputs toward SQL resolvers as filters like: ``` where workspace_id in (...) and user_id in (...) ``` This is often the best shape when the underlying table is clustered, partitioned, or indexed by the component columns. ### Scalar Composite Keys Use a scalar composite key when the rest of the graph needs a single feature to represent the identity. This is common when you want the id of a feature class to be derived from multiple source columns. ``` from chalk import _ import chalk.functions as F from chalk.features import feature, features @features class WorkspaceMember: workspace_id: str user_id: int id: str = feature( expression=_.workspace_id + ":" + F.cast(_.user_id, str), primary=True, ) ``` The expression must be deterministic and must produce exactly the same value everywhere the key is used. When a query provides all of the input features used by the expression, Chalk can synthesize the primary key from those inputs. For example, if the query provides WorkspaceMember.workspace_id and WorkspaceMember.user_id, Chalk can compute WorkspaceMember.id. All inputs to the expression must be available. If the query provides workspace_id without user_id, Chalk cannot partially compute the key and will ask for the missing component. If a child feature class needs to join to this key, define the child-side foreign key as a feature too: ``` from chalk import _ import chalk.functions as F from chalk.features import feature, features, has_one WORKSPACE_MEMBER_ID = _.workspace_id + ":" + F.cast(_.user_id, str) @features class WorkspaceMember: workspace_id: str user_id: int id: str = feature(expression=WORKSPACE_MEMBER_ID, primary=True) @features class ActivityEvent: id: str workspace_id: str user_id: int workspace_member_id: "WorkspaceMember.id" = feature(expression=WORKSPACE_MEMBER_ID) workspace_member: WorkspaceMember = has_one( lambda: _.workspace_member_id == WorkspaceMember.id ) ``` The string annotation "WorkspaceMember.id" declares that workspace_member_id is a foreign key to WorkspaceMember.id. It does not compute the value. The value computation comes from feature(expression=...). ### SQL Resolver Outputs SQL resolvers must output the features that Chalk needs to join or filter on. This is the most common source of confusion with composite keys. Suppose your relationship joins on _.workspace_member_id == WorkspaceMember.id. If the resolver for ActivityEvent only emits workspace_id and user_id, the planner does not have a workspace_member_id column to attach the join or filter to. In an offline query, that can force Chalk to unload or scan far more of the table than expected. Even if the resolver SQL computes a composite value internally, Chalk can only push filters to features that are present in the resolver output and used by the relationship. For a scalar composite key join, emit the scalar key: ``` -- source: snowflake -- resolves: ActivityEvent select id, workspace_id, user_id, workspace_id || ':' || cast(user_id as string) as workspace_member_id, amount from activity_events ``` For a component join, emit the component columns: ``` -- source: snowflake -- resolves: ActivityEvent select id, workspace_id, user_id, amount from activity_events ``` If a query spine provides only workspace_member.id, but the resolver exposes only workspace_id and user_id, Chalk cannot infer the component filters from the scalar id. Either provide workspace_id and user_id directly as inputs, or emit the scalar composite key as a resolver output and join on that feature. There is one useful exception for root resolvers that return rows for a feature class whose primary key has an underscore expression. If the resolver returns every component needed by the primary-key expression, Chalk can synthesize the primary key from those columns. If the primary key does not have an underscore expression, the resolver must output the primary key itself. ### Materialized Aggregations For materialized aggregations over a has_many, Chalk keys the materialized values by the join features used by the relationship. When an aggregation is reached through a has_one prefix, the has_one join keys and the has_many local keys must overlap. This can matter in two-hop graphs: ``` from chalk import DataFrame, Windowed, _, windowed import chalk.functions as F from datetime import datetime from chalk.features import feature, features, has_many, has_one WORKSPACE_MEMBER_ID = _.workspace_id + ":" + F.cast(_.user_id, str) @features class ActivityEvent: id: str workspace_id: str user_id: int workspace_member_id: "WorkspaceMember.id" = feature(expression=WORKSPACE_MEMBER_ID) occurred_at: datetime amount: float @features class WorkspaceMember: workspace_id: str user_id: int id: str = feature(expression=WORKSPACE_MEMBER_ID, primary=True) activity_events: DataFrame[ActivityEvent] = has_many( lambda: _.workspace_member_id == WorkspaceMember.id ) @features class MemberRiskFeatures: id: str workspace_member_id: "WorkspaceMember.id" workspace_member: WorkspaceMember = has_one( lambda: _.workspace_member_id == WorkspaceMember.id ) event_amount: Windowed[float] = windowed( "30d", materialization={"bucket_duration": "1d"}, expression=_.workspace_member.activity_events[ _.amount, _.occurred_at > _.chalk_window, _.occurred_at <= _.chalk_now, ].sum(), ) ``` Here, both relationships use WorkspaceMember.id as the key. That lets the materialization planner line up the has_one prefix with the has_many aggregation. If the has_one used WorkspaceMember.id but the has_many joined on (workspace_id, user_id), the planner may reject the materialization because the join keys do not overlap. ### Best Practices Use the source system's stable identifier when one exists. Use a computed composite key only when the source system does not already provide a durable id. Use component joins when the warehouse can prune efficiently on the component columns. This is especially useful for offline queries over large tables. Use scalar composite keys when the entity identity needs to travel through the graph as one feature, or when a materialized two-hop aggregation needs overlapping join keys. Keep the composite key expression in one place. Define a shared expression constant and reuse it for the parent primary key and any child foreign keys. Choose an encoding that cannot collide. If you concatenate strings, use a delimiter that cannot appear in the component values, escape the components, or use a stable hash of a structured value. Make casts explicit so every resolver and expression produces the same string. Emit every join key from SQL resolvers. If a relationship joins on a scalar composite key, the resolver should output that scalar key. If a relationship joins on components, the resolver should output the components. For root resolvers, either output the primary key or make sure the primary key has an underscore expression and the resolver outputs all of the expression inputs. Treat persisted composite keys as stable cache keys. Persisted results can be read back by the computed key, but changing the key expression changes the identity of cached rows. Test with a small offline query and inspect the query plan or generated SQL. You should see filters on the columns used by the relationship, not a full-table unload followed by a join in Chalk. Check both that the query can run and that the plan is efficient: a valid composite-key model can still be too expensive if the relevant filters are not pushed down. ### Common Pitfalls Assuming Chalk can split a composite id. If an input spine contains values like workspace-7:123, Chalk does not know that workspace-7 is workspace_id and 123 is user_id. Provide the component inputs or model and emit the scalar key directly. Providing only some key components. If WorkspaceMember.id is computed from workspace_id and user_id, Chalk needs both components to synthesize the id. Supplying only one component is not enough. Using a foreign-key annotation as a computation. A type like workspace_member_id: "WorkspaceMember.id" declares the relationship target. It does not populate workspace_member_id. Use feature(expression=...), a Python resolver, a SQL resolver column, or uploaded data to provide the value. Expecting a resolver to omit an uncomputable primary key. A DataFrame or SQL resolver may omit a computed primary key only when Chalk has an underscore expression and the resolver outputs all of that expression's inputs. Otherwise, the resolver must output the primary key feature. Joining on a key that the resolver does not output. If the planner needs activity_event.workspace_member_id, the resolver must return a column for workspace_member_id. Returning only workspace_id and user_id is not equivalent when the relationship joins on workspace_member_id. Mixing scalar and component joins across a two-hop materialization. If the first hop joins on WorkspaceMember.id but the second hop joins on workspace_id and user_id, the materialization planner may not be able to prove that the keys line up. Use the same join feature on both hops, or model the relationship with component joins consistently. Duplicating the key formula by hand. If the parent key changes but the child key expression does not, joins silently stop matching. Reuse a shared expression or, for SQL resolvers, keep the SQL projection in lockstep with the Chalk expression. # Chalk DataFrame source: https://docs.chalk.ai/docs/dataframe ## Describe and fetch rows of features. A Chalk DataFrame is a 2-dimensional data structure similar to pandas.DataFrame, but with richer underlying optimizations. In a Chalk DataFrame, the column headers are instances of chalk.feature. The Chalk DataFrame can be used to describe rows of data and as a wrapper for returning that data. The Chalk DataFrame is parameterized by the feature values it contains. For example: ``` DataFrame[User.name, User.email] ``` The DataFrame type is commutative on its type parameters. This property allows you to write the type parameters of a DataFrame in any order: ``` DataFrame[User.name, User.email] == DataFrame[User.email, User.name] ``` Note: The Chalk DataFrame is used as a feature type for defining has-many relationships between feature classes, but the functions, filters, and aggregations described in this document are meant to be used in Python resolvers or notebooks -- not in Chalk Expressions. For information about the functions and aggregations that you can use in Chalk Expressions, see here. When a Python resolver accepts a DataFrame and returns a transformed DataFrame, preserve Chalk's Index column so each output row remains correlated with its input row. See DataFrame-to-DataFrame resolvers. ### Constructing a DataFrame You're likely to use the Chalk DataFrame primarily as a type, or to handle one as the result of a SQL query. However, you can also construct a DataFrame directly. ### From a dictionary Much like a pandas.DataFrame, a chalk.DataFrame can be constructed from a dictionary. The keys of the dictionary are the headers and the values are the rows under each header. The keys can be specified as either strings ("taco.price") or as their Python types (Taco.price). You can convert from a feature's Python definition to a string with a cast: str(Taco.price) == "taco.price". ``` chalk_df = DataFrame( { Taco.id: ["t_1", "t_2", "t_3"], "taco.price": [1, 2, 10], str(Taco.contains_meat): [True, True, False], } ) ``` In the above example, we use all forms of referencing a feature's name. ### From a list You can create the same chalk.DataFrame by passing a list of feature classes: ``` chalk_df = DataFrame([ Taco(id="t_1", price=1, contains_meat=True), Taco(id="t_2", price=2, contains_meat=True), Taco(id="t_3", price=10, contains_meat=False), ]) ``` ### From a pandas.DataFrame You can also convert to and from a pandas.DataFrame. ``` import pandas as pd DataFrame( pd.DataFrame({ "taco.id": ["t_1", "t_2", "t_3"], # Taco.id or "taco.id" are accepted "taco.price": [1, 2, 10], }) ) ``` The Chalk DataFrame can be converted to a pandas.DataFrame via the method .to_pandas(): ``` df = DataFrame(...) df.to_pandas() ``` ### From a SQL query Chalk's SQL integrations output the type DataFrame. For more information, see the SQL Integration section. ``` pg = PostgreSQLSource() @offline def fn() -> DataFrame[Login.ts, Login.user_id, Login.status]: return pg.query( Login( status=LoginHistorySQL.status, ts=LoginHistorySQL.created_at, user_id=LoginHistorySQL.user_id, ) ).all() ``` ### Aggregations | Method | Description | | ----------------- | ------------------------------------------------------------ | | DataFrame.count() | The number of rows in a `DataFrame` | | DataFrame.mean() | The average for each column in the `DataFrame`† | | DataFrame.sum() | The sum of each column in the `DataFrame`† | | DataFrame.max() | The maximum value for each column in the `DataFrame`† | | DataFrame.min() | The minimum value for each column in the `DataFrame`† | † There will be only one row in the DataFrame after this operation, one for each value. If the DataFrame contained only a single column to start, then the value returned is a scalar. ### Projections You can select columns out of a DataFrame from the set of columns already present to produce a new DataFrame scoped down to those columns. For example: ``` df: DataFrame[User.name, User.email, User.age] = ... projected = df[User.email, User.age] # type(projected) == DataFrame[User.email, User.age] ``` ### Filters In addition to selecting columns, you can filter the rows of a DataFrame. The example below restricts the rows of a DataFrame to only rows where the User is over 21 years old: ``` df: DataFrame[User.name, User.email, User.age] f = df[User.age >= 21] # type(f) == DataFrame[User.name, User.email, User.age] ``` Filtering a DataFrame keeps all the existing columns, but drops rows where the predicate is not met. The Chalk DataFrame supports the standard comparison functions (<, <=, >, >=, in, is, is not, ==, and !=) for features and their matching scalars values. You can compose these filters with the boolean operations and, or, and not. | Operation(s) | Example | Comment | | ------------ | ----------------------------------------------------------- | ------------------------- | | >, <, >=, <= | User.age > 21 | | | ==, != | User.age is None; User.age is not None | AST-only feature.† | | is, is not | User.age is None; User.age is not None | AST-only feature.† | | in, not in | User.age in {17, 18}; User.age not in [17, 18] | AST-only feature.† | | or, and | (User.state == "AL" and User.age >= 17) ; or User.age >= 18 | AST-only feature.† | | not | not (User.state == "AL" and User.age >= 17) | AST-only feature.† | † The marked operations must be provided as direct arguments. They cannot first be assigned to a variable, and later provided as an argument. For example, DataFrame[User.age in {17, 18}] is valid, but defining x = User.age in {17, 18} and then running DataFrame[x] is not valid. The reason for this is that Python does not allow overriding these operations. However, Chalk's aim is to allow developers to inspect and support natural Python syntax. To that end, Chalk parses the AST to override these operations, with the restriction that they are provided as arguments directly. ### Composing Filters To filter on multiple columns, separate your filters by a comma: ``` df[User.age > 21, User.email == "joe@chalk.ai"] ``` Alternatively, use the and keyword between filters: ``` df[User.age > 21 and User.email == "joe@chalk.ai"] ``` Similarly, you can perform or queries with Python's or keyword: ``` df[User.age > 21 or User.email == "joe@chalk.ai"] ``` All of these can be composed: ``` df[ User.age > 21 and ( User.email == "joe@chalk.ai" or User.full_name is None ) ] ``` ### Composing projections and filters Filtering may be combined with projection. For example, you can select all the values for User.email where the user is at least 21 years old as follows: ``` df: DataFrame[User.name, User.email, User.age] projected = df[User.email, User.age > 21] # type(projected) == DataFrame[User.email] ``` If you'd like to filter on a value and also return the value, you need to explicitly select it. You can amend the above example to include the User.age feature like so: ``` df: DataFrame[User.name, User.email, User.age] projected = df[User.email, User.age, User.age > 21] # type(projected) == DataFrame[User.email, User.age] ``` ### Performance (Vectorization, Laziness) Chalk's DataFrame is designed to support efficient operations across a variety of underlying data sources. DataFrame vectorizes scalar operations on data that is kept resident in memory. Chalk calls operations on data that is loaded into memory "strict execution". In coordination with Chalk's Execution Engine, DataFrame is capable of pushing down filtering, projection, and aggregation operations to underlying data sources. Chalk calls this style of execution "lazy execution". Suppose that we use a DataFrame to query a SQL source: ``` @online def get_return_count( transfers: User.transfers[Transfer.status == "returned", after(days_ago=60)] ) -> User.returned_transactions_last_60: return transfers.count() ``` This ultimately results in (approximately) the following SQL query being executed: ``` SELECT COUNT(*) from transfers WHERE status = 'returned' and transfers.ts > current_date - interval '60' day; ``` This "push down" mechanism in lazy computation helps make computationally expensive operations execute quickly, since Chalk only needs to load a single integer from the underlying data source, instead of potentially all Transfer rows. # Validation source: https://docs.chalk.ai/docs/validation ## Validate features and resolvers ### Feature Values Chalk can enforce requirements on your feature values. You can validate the values and length of many primitive types through the keyword arguments min, max, min_length, and max_length. To prevent 'invalid' features from being written to the offline or online store, set strict=True. In this case an error will be thrown when an invalid feature is observed. The validations keyword argument can be used when there are multiple validations, only some of which are strict. ``` from chalk import Validation from chalk.features import feature, features @features class Office: size_sqft: int = feature(min=0, max=100_000_000) street: str = feature(validations=[ Validation(min_length=1, max_length=256, strict=True), Validation(min_length=20, max_length=100) ]) ``` ### Feature validation requirements For some features, it is important to specify metadata such as owner and description. Chalk allows you to enforce metadata requirements easily. In addition, you can specify for each feature the severity at which to raise a missing metadata issue. In chalk.yml, the validation > feature > metadata section specifies settings for metadata validation. In the following example, users will be blocked from deploying features (chalk apply) with an unspecified owner. They are allowed to deploy features with missing description and tags. ``` project: Predict Q2 Spending validation: feature: metadata: - name: owner missing: error - name: description missing: warning - name: tags missing: info environments: default: runtimes: 'python311' requirements: requirements.txt ``` # Versioning source: https://docs.chalk.ai/docs/feature-versions ## Rigorously update your features as their meaning changes. ### What are feature versions Feature versions allow you to manage a feature as its definition changes over time. You can reference versions of a feature when querying, in resolvers, in migrations, and anywhere you use features! The ability to maintain multiple versions of a feature is especially useful as your use cases evolve and the definition of a feature changes. This might be because your old feature had a bug, or maybe you want to improve the feature to be even more useful. Changing the definition of a feature can have unexpected side effects, especially when there might be many consumers of a feature, some of which you may be unaware of. Feature versions allow you to update the feature definition without worrying that you might affect these consumers unknowingly. Try it out yourself with this tutorial. ### Definition To define a versioned feature, supply the version keyword argument to the feature function. This allows you to continue adding versions without affecting past versions. ``` from chalk.features import features, feature @features class Trial: id: int code: str = feature(version=2) ``` ### Querying You can refer to a specific version of a feature using the @ operator. You can query versioned features from any client, including the CLI, Python, etc. ``` $ chalk query --in trial.id=1 --out trial.code@2 ``` ``` from chalk.client import ChalkClient from src.models import Trial client = ChalkClient() result = client.query( input={Trial.id: 1}, output=[Trial.code@2] ) ``` ### Default versions When you define a versioned feature, that feature always has a default version. When you reference a versioned feature without the @ operator, you reference the default version. The default version can be controlled by adding the default_version keyword argument to the feature function. If no default_version is provided, the default version is 1. It is best practice to leave the default as 1 until all consumers have been updated to use a newer version. ``` from chalk.features import features, feature @features class Trial: id: int code: str = feature(version=2, default_version=2) ``` ### Deprecating features As you migrate to newer feature versions or phase out old features, you can mark them as deprecated to signal that they should no longer be used. This helps teams understand which features to avoid and provides a clear migration path. You can deprecate features that aren't versioned to indicate they should be phased out: ``` from chalk.features import features, feature @features class User: id: str # Mark the old feature as deprecated legacy_risk_score: float = feature( deprecated=True, description="Use risk_score_v2 instead" ) # The new feature that should be used instead risk_score_v2: float ``` ### Deprecating specific versions When using multiple versions of a feature, you can mark specific versions as deprecated: ``` from chalk.features import features, feature from chalk import _ @features class User: id: str score: int banned: bool = feature(versions={ 1: feature( deprecated=True, description="Prior thresholds we used", expression=_.score > 900, ), 2: feature( description="New cutoff released by Monica's team", expression=_.score > 300 | (_.score < 100 & _.id == "admin"), ), }) ``` When a deprecated feature is used in queries or resolvers, Chalk will log warnings to help you track and eliminate usage of deprecated features across your codebase. This is particularly useful during gradual migrations when you need to maintain backward compatibility while encouraging adoption of new features. To monitor the migration progress and identify which deprecated features can be safely removed, you can use the Chalk web interface to sort features by usage and see when each feature was last queried. This helps you make data-driven decisions about when it's safe to fully retire deprecated features. ### Resolvers Resolvers also reference versioned features with @. Resolvers can take versions of a feature as input, return them as output, or both. Chalk will always generate the right version of a feature when chaining resolvers that require versioned features. ### Requesting versioned features ``` from chalk.features import online @online def detect_trial_failure(code: Trial.code) -> Trial.failed: # Default version is 1. return code in {'failed', 'err'} @online def detect_trial_failure_v2(code: Trial.code@2) -> Trial.failed: # Version 2 is specifically requested. return code in {'FAILED', 'ERR'} ``` ### Producing versioned features ``` @online def extract_trial_code_v1(d: Trial.raw_data) -> Trial.code @ 1: return d['code'] @online def extract_trial_code_v2(d: Trial.raw_data) -> Trial.code @ 2: return d['data']['code'] ``` ``` -- resolves: Trial@2 -- source: pg_trial select id, code from trial_table; ``` ### Limitations Scalar and windowed aggregation features can be versioned. Other kinds of features cannot be versioned. These include: - has_one and has_many - FeatureTime - Primary # Embeddings source: https://docs.chalk.ai/docs/embeddings ## Automatically calculate embeddings from existing features Embedding models are generally used to calculate a vector feature. Chalk includes built-in support for common embedding models, or you can define your own embedding model through a resolver. ### Built-In Embedding Functions Chalk includes built-in support for both open-source and hosted embedding models via the embedding function. We recommend using this function when possible, as Chalk will automatically handle batching and retries, and you don't need to specify the vector size. The main arguments for this function are: - input (required): A lambda that returns the feature that will be embedded.If you would like to use multiple features as input, you can define a resolver to combine these features into one. Then, reference this combined feature as input. - provider (required): The embedding model provider. The current list of supported providers is sentence-transformers, openai, cohere, and vertexai. - model (required): The name of the model to use. Each provider has a different set of models that are supported. - max_staleness (optional): The duration for which the embedding will be cached. By default, the embedding vector will be cached for the same duration as the feature. If you would like different behavior, you can specify this argument explicitly. For the complete signature, please see the api docs. ### Sentence Transformers Chalk supports the all-MiniLM-L6-v2 model from the sentence-transformers framework. To use a different sentence-transformers model, deploy it with model deployments, or contact Chalk. ``` from chalk.features import embed, features, Vector @features class Document: content: str embedding: Vector = embed( input=lambda: Document.content, provider="sentence-transformers", model="all-MiniLM-L6-v2", ) ``` ### OpenAI Chalk can proxy calls to the OpenAI Embeddings API. It is recommended to use the text-embedding-ada-002 model, though all OpenAI models are supported. If you don't already have an OpenAI account, sign up here, and then create an OpenAI Integration in Chalk. All OpenAI requests will be attributed to your API key. To minimize usage, we highly recommend specifying an appropriate max staleness in Chalk, which will ensure that embeddings are cached. ``` from chalk.features import embed, features, Vector @features class Document: content: str embedding: Vector = embed( input=lambda: Document.content, provider="openai", model="text-embedding-ada-002", max_staleness="infinity", ) ``` ### Cohere Chalk can proxy calls to Cohere Embed. To use this integration, first sign up for a Cohere Account, and then create a Cohere Integration in Chalk. All Cohere requests will be attributed to your API key. To minimize usage, we highly recommend specifying an appropriate max staleness in Chalk, which will ensure that embeddings are cached. ``` from chalk.features import embed, features, Vector @features class Document: content: str embedding: Vector = embed( input=lambda: Document.content, provider="cohere", model="embed-english-v3.0", max_staleness="infinity", ) ``` ### Custom embedding functions If you would like to run your own embedding model, you can define a custom resolver to compute the embedding from existing features in the feature class. For performance, we recommend to store the model weights in an object store (such as AWS S3 or GCS) rather than including them in your source code and to load the model using a boot hook. ``` from chalk.features import before_all, DataFrame, embed, features, online, Vector my_model = MyModel() @before_all def load_my_model(): my_model.initialize("s3://my-bucket/my-checkpoint.pt") @features class Document: content: str # When using a custom embedding function, the size of the vector must be specified. embedding: Vector[1536] @online def my_embedding_function(content: DataFrame[Document.content]) -> DataFrame[Document.embedding]: return my_model.embed(content.to_arrow()['document.content']) ``` Chalk will then call my_embedding_function whenever an embedding is needed. # Nearest Neighbors (Vector Search) source: https://docs.chalk.ai/docs/vector-search ## Find the nearest neighbors across a vector relationship A feature class can be linked to the closest examples of another feature class. This functionality can be useful for search and retrieval applications. Nearest neighbor relationships are only supported for vector features. We recommend to first take a look through Chalk's support for vector features and embedding functions. To illustrate how to use nearest neighbor relationships, we'll walk through an example of how Chalk can power FAQ search. In this example, the SearchQuery feature class represents an incoming request, and the FAQDocument feature class represents our collection of frequently asked questions and answers. Our goal is to return the five most relevant FAQ entries for the given search query. ### Defining nearest neighbor relationships Using the has_many function and the is_near method, we can express a relationship where we want the nearest documents for each query. ``` from chalk.features import DataFrame, embed, features, has_many, Vector @features class SearchQuery: query: str product_version: int embedding: Vector = embed(...) nearest_documents: DataFrame[FAQDocument] = has_many( lambda: SearchQuery.embedding.is_near( FAQDocument.embedding ) ) response: str @features class FAQDocument: question: str product_version: int question_embedding: Vector = embed(...) link: str ``` The lambda solves forward references, letting you reference SearchQuery and FAQDocument before they are defined. ### Distance Measure Nearest neighbor relationships use a distance function to measure closeness. By default, Chalk uses L2 distance, though inner product and cosine similarity are also supported. To change the distance function, use the metric argument: ``` from chalk.features import DataFrame, features, has_many @features class SearchQuery: nearest_documents: DataFrame[Document] = has_many( lambda: SearchQuery.embedding.is_near( FAQDocument.embedding, metric="cos", # or "l2", "ip" ) ) ``` ### Nearest neighbors as resolver inputs It's possible to use this relationship as a has-many resolver input. The resulting documents will be returned as a Chalk DataFrame. Because the search is approximate, the number of documents to return must be specified via a slice expression. ``` from chalk import online @online def generate_response( # Query for the five most relevant documents, and select their links nearest_documents: SearchQuery.nearest_documents[FAQDocument.link, :5], ) -> SearchQuery.response: return "\n".join(nearest_documents[FAQDocument.link]) ``` ### Filtering Inside the input argument signature, we can include filters for more accurate results. The filters will be applied before the limit is applied. When using a nearest neighbor relationship, do not filter within the resolver. Filtering inside the resolver will be performed after the limit is applied, which may filter out all returned neighbors if none of them match the filter expression. Don't filter like this ``` from chalk import online @online def generate_response( version: SearchQuery.product_version, nearest_documents: SearchQuery.nearest_documents[ FAQDocument.link, FAQDocument.product_version, :5, ], ) -> SearchQuery.response: # Don't do this! If the nearest five documents are all for a different product version, # then filtered_nearest_documents will be empty filtered_nearest_documents = nearest_documents[FAQDocument.product_version == version] return "\n".join(filtered_nearest_documents[FAQDocument.link]) ``` Instead, specify the filter conditions in the resolver signature. This will ensure that the filter is applied before the limit, meaning that the nearest five documents that match all of the filters will be returned. Filter like this ``` from chalk import online @online def generate_response( filtered_nearest_documents: SearchQuery.nearest_documents[ FAQDocument.link, FAQDocument.product_version == SearchQuery.product_version, :5, ], ) -> SearchQuery.response: return "\n".join(filtered_nearest_documents[FAQDocument.link]) ``` # Prompts source: https://docs.chalk.ai/docs/prompts ## Define inline LLM completions in feature code. Define LLM completions directly in feature code using chalk.prompts.completion. Messages can use Jinja templates to reference feature values. Stored named prompts and run_prompt are no longer supported. ### Prompt definitions To define a direct prompt to perform sentiment analysis on a comment, you could define the following Chalk features utilizing the chalk.prompts library. ``` import chalk.functions as F import chalk.prompts as P from chalk.features import _, features from pydantic import BaseModel, Field system_message = """You are a sentiment analysis expert. Analyze the user's comment and determine if it's positive, negative, or neutral. Rules: - Be objective and focus on the text's emotional tone - Consider context and nuance - Provide a confidence score based on clarity of sentiment""" user_message = "Analyze this comment: {{User.comment}}" class UserSentiment(BaseModel): sentiment: str = Field(description="positive, negative, or neutral") confidence: float = Field(description="on a 1-100 scale") @features class User: id: str comment: str actual_sentiment: str response: P.PromptResponse = P.completion( model="gpt-5.1", messages=[ P.message(role="system", content=system_message), P.message(role="user", content=F.jinja(user_message)), ], output_structure=UserSentiment, ) predicted_sentiment: str = F.json_value(_.response.response, "sentiment") ``` In the example above, we see how direct prompts integrate with Chalk's feature system. The prompt uses Jinja templating to dynamically inject the user's comment into the prompt through {{User.comment}}, allowing the LLM to analyze the specific content at runtime. The UserSentiment Pydantic model defines a structured output format, ensuring consistent response parsing across all completions. While this approach provides fine-grained control over the prompt configuration, it tightly couples the prompt template, model settings, and output structure to your deployment code. Any changes to the prompt template or model parameters require a new deployment. ### Multimodal Prompts Chalk's chat completion API extends beyond text-only interactions by supporting multimodal inputs, enabling you to process images alongside text in your LLM workflows. This capability opens up new possibilities for visual content analysis, allowing you to pass images directly to vision-capable models like GPT-5.1, combine text and image inputs in a single prompt, and extract structured information from visual content. Here's how you can process a receipt image to extract structured data: ``` @features class Receipt: image_url: str image_response: P.MultimodalPromptResponse = P.completion( model="gpt-5.1", messages=[ P.message( "user", [ {"type": "input_text", "text": "Extract all items, prices, and total from this receipt"}, {"type": "input_image", "image_url": _.image_url}, ], ), ], ) ``` Image inputs work with Chalk's feature system the same way text inputs do. The same structured output capabilities available for text-only prompts apply here, allowing you to define Pydantic models for consistent parsing of visual information extraction tasks. # Windowed Aggregations source: https://docs.chalk.ai/docs/aggregations ## Define features as aggregations of data over sliding time ranges. Given a has-many relationship between two feature classes, Chalk can compute aggregations over the joined data for all rows, filtered by additional field predicates, or filtered by specific windows of time. The has-many can be either directly on the feature class that owns the aggregation, or reached through a single has-one hop (e.g. _.user.transactions.count() on a class that has a user: User has-one and where User.transactions is a has-many). Define general aggregation features with aggregation functions in a Chalk expression, and use the windowed function for aggregations over fixed lookback windows. For example, use windowed features to count the number of login attempts made by a user over the past 10 minutes, or to track the largest purchase amount a cardholder has made in the past 30 days. For high-throughput or low-latency use cases, use Materialized Windowed Aggregations to pre-aggregate data. ### Define an Aggregation Chalk supports a number of out-of-the-box aggregations. Chalk builtins are very performant as they're optimized and run natively in C++ and Rust. Aggregations automatically skip null or None values. The following table lists the supported aggregation functions: | Function | Notes | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sum` | Support for vector and scalar feature aggregations. | | `min` | | | `max` | | | `min_by_n` | Returns a list of the `n` values for the minimum `by` values. | | `max_by_n` | Returns a list of the `n` values for the maximum `by` values. | | `mean` | Feature type must be `Vector`, `float` or float \| None. `None` values are skipped, meaning they are not included in the mean calculation. | | `count` | | | `std` | Standard deviation. Requires at least 2 values. | | `var` | Variance. Same requirements as `std`. | | `skewness` | Skewness of the distribution. Computed using Fisher's moment coefficient of skewness. Accepts an optional `bias` keyword argument which defaults to `True`. Setting `True` produces the biased (population) coefficient, `False` produces the bias-corrected (sample) coefficient. Requires at least 3 values and returns null otherwise. | | `kurtosis` | Kurtosis of the distribution. Computed using Pearson moments. Accepts an optional `bias` keyword argument which defaults to `True`, and an optional `fisher` keyword argument which defaults to `True`. Setting `bias=True` gives the biased (population) excess coefficient, and setting `bias=False` makes the result bias-corrected. Setting `fisher=True` computes excess kurtosis (normal → 0), and setting `fisher=False` computes Pearson (normal → 3). Requires at least 4 values and returns null otherwise. | | `approx_count_distinct` | An approximation of the cardinality of non-null data. | | `approx_top_k` | An approximation of the most common values in a field. Takes in a required `k` parameter, such as `approx_top_k(k=5)` for the five approximately-most common values. Up to `k` values will be returned, sometimes fewer. The optional `counters` parameter sets how many distinct values the underlying sketch tracks, which defaults to `2 * k`; raising it trades memory for accuracy on high-cardinality data. | | `approx_percentile` | An approximation of the value a given percentage of your data falls below. Accepts `quantile` parameter, between `0` and `1`, such as `approx_percentile(quantile=.5)`. | For example, say you want to compute some aggregations over a document's revisions. You could define the features, filters, and aggregations as below. ``` from chalk.features import features, DataFrame, _ from datetime import datetime @features class Document: id: int revisions: "DataFrame[Revision]" # these are aggregations over the revisions DataFrame num_revisions: int = _.revisions[_.id].count() max_revision_size: float = _.revisions[_.size_bytes].max() # you can also filter the rows being aggregated earliest_large_revision_ts: datetime = _.revisions[_.size_bytes > 100_000_000, _.timestamp].min() # some aggregations return a list most_common_sizes: list[float] = _.revisions[_.size_bytes].approx_top_k(k=5) @features class Revision: id: int # this is a foreign key join between Document and Revision document_id: Document.id size_bytes: float timestamp: datetime ``` ### Define a Windowed Aggregation In an aggregation, you can filter and sort by datetime features. In a single feature definition, Chalk enables you to configure a Windowed Aggregation, which computes the same aggregation over multiple time windows. A windowed feature takes in a series of window durations, and an aggregation expression for how to compute the feature outputs. Within the expression, you can reference the timestamp feature on the feature class over which you are aggregating and compare to _.chalk_window and _.chalk_now to filter against the current window's time range. The _.chalk_now keyword references the query time which will default to timestamp.now() but can be overridden when a different input time is specified at query time. This allows you to query for an aggregation for any window duration at any point in time. The _.chalk_window keyword refers to the start timestamp of each configured lookback window relative to _.chalk_now. Here is an example of a windowed feature representing the number of failed logins in the last 10 minutes, 30 minutes, and 1 day: ``` from chalk import Windowed, windowed from chalk.features import DataFrame, _ from datetime import datetime @features class LoginAttempt: id: int user: "User.id" status: str at: datetime @features class User: id: int login_attempts: DataFrame[LoginAttempt] num_failed_logins: Windowed[int] = windowed( "10m", "30m", "1d", max_staleness="10m", expression=_.login_attempts[ _.status=="failed", _.at > _.chalk_window, _.at < _.chalk_now, ].count(), default=0, ) most_common_statuses: Windowed[list[str]] = windowed( "1h", max_staleness="15m", expression=_.login_attempts[ _.status, _.at > _.chalk_window, _.at < _.chalk_now, ].approx_top_k(k=5) ) ``` Windowed features support much of the same functionality as normal features. They are most often used alongside max_staleness and etl_offline_to_online. Windowed features are typically resolved, either: - inline by expressions, - by stream resolvers, or - by materialized aggregates. ### Infinite lookback For aggregating across all features for all time, Chalk offers "all" and "infinity" as keywords for window durations. As a default, Chalk will look back 100 years for the "all" window. ``` from chalk.features import features, DataFrame, _ from chalk.streams import windowed, Windowed @features class TennisPlayer: id: int games: DataFrame[Game] lifetime_wins: Windowed[int] = windowed( "all", expression=_.games[_.winner_id == _.id].count(), default=0, ) ``` ### Referencing windowed features A windowed feature can be referenced in a query or a resolver in Python using the window as defined in the feature definition, but Chalk will convert every windowed feature to a FQN (fully qualified name) that defines the window duration in seconds. So, when referencing the feature for a resolver, you can use the Python syntax. However, when referencing a feature in a query, you can use the Python syntax when querying with the Chalk Python SDK, otherwise the CLI will require the FQN syntax. If you do not specify a window when referencing a windowed feature, Chalk will return all windows. Given the feature definition below, the following showcase the ways to reference the respective windows in either Python syntax or using the FQN. ``` from chalk.features import features from chalk.streams import windowed, Windowed @features class User: id: int ... num_failed_logins: Windowed[int] = windowed( "10m", "1h30m", "1d", expression=..., ) ``` | Feature Definition Window | Python Syntax | Fully Qualified Name | | ------------------------- | ------------------------------- | --------------------------------- | | "10m" | User.num_failed_logins["10m"] | `User.num_failed_logins__600__` | | "1h30m" | User.num_failed_logins["1h30m"] | `User.num_failed_logins__5400__` | | "1d" | User.num_failed_logins["1d"] | `User.num_failed_logins__86400__` | Windowed features can be inputs to resolvers: ``` @online def account_under_attack( failed_logins_30m: User.num_failed_logins["30m"], failed_logins_1d: User.num_failed_logins["1d"] ) -> ...: return failed_logins_30m > 10 or failed_logins_1d > 100 ``` ### Nested windowed feature references Windowed features can also be referenced by other windowed features in the same feature class using expressions and the _.chalk_window operator. For example, we can compute the average transaction amount over different time windows: ``` @features class Transaction: id: int user_id: "User.id" amount: float at: datetime @features class User: id: int transactions: DataFrame[Transaction] sum_transactions: Windowed[float] = windowed( "30d", "60d", "90d", expression=_.transactions[ _.amount, _.at > _.chalk_window, _.at < _.chalk_now, ].sum(), ) count_transactions: Windowed[float] = windowed( "30d", "60d", "90d", expression=_.transactions[ _.at > _.chalk_window, _.at < _.chalk_now, ].count(), ) avg_transaction_amount: Windowed[float] = windowed( "30d", "60d", "90d", expression=( _.sum_transactions[_.chalk_window] / _.count_transactions[_.chalk_window] ) ) ``` ### Materialized Windowed Aggregations Windowed features are typically computed using either raw data or pre-aggregated data. For larger datasets, some systems may pre-aggregate batch data to optimize performance at the cost of real-time accuracy. Chalk supports both modes of operation: - For smaller datasets or lower throughput use cases, Chalk can compute windowed aggregations directly on the raw data and this is the default mode. - To achieve low latency for real time aggregations over large datasets, Chalk provides Materialized Windowed Aggregations through the materialization parameter of windowed features. See Materialized Windowed Aggregations for guidance on when to use them. - To evolve an aggregation's expression or time windows without breaking existing consumers, see Versioned Windowed Aggregations. # Materialized Windowed Aggregations source: https://docs.chalk.ai/docs/materialized_aggregations ## Cache and materialize feature aggregations Machine learning teams often build and maintain separate pre-aggregation pipelines that relay data as standalone features into a feature store to reduce the latency of computing expensive features. With Chalk, developers don’t need to spin up separate pipelines to materialize aggregations. This can be done with just a single line of code without compromising on flexibility: ``` from chalk.features import features, Vector, _ from chalk import DataFrame, Windowed, windowed from datetime import datetime @features class Transaction: id: int amount: float category: int user_id: "User.id" timestamp: datetime embedding: Vector[384] @features class User: id: int transactions: DataFrame[Transaction] total_transaction_amount: Windowed[float] = windowed( "1d", "7d", "30d", materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.amount, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].sum(), ) most_active_categories: Windowed[list[int]] = windowed( "30d", "60d", "all", materialization={"bucket_duration": "1h"}, # The approximate top 10 categories that the user has # spent at in each of the windowed buckets. expression=_.transactions[ _.category, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].approx_top_k(k=10) ) avg_embedding: Windowed[Vector[384]] = windowed( "30d", "60d", "all", materialization={"bucket_duration": "1h"}, # Vector mean will be computed as the element-wise # mean for each component of the vector expression=_.transactions[ _.embedding, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].mean(), default=[0.0] * 384 ) ``` ### Why are materialized aggregations useful? Windowed features are typically computed using either raw data or pre-aggregated data. Raw data is the ground truth, but aggregating it can be slow if you request long time windows or large volumes of data. Some systems improve performance by serving features from pre-aggregated batch data, mitigating high latency with the trade-off of not having access to the freshest data. Chalk balances accuracy and performance by combining both approaches. Chalk aggregates historical data into buckets in the online store and continuously updates those buckets as new data arrives. Chalk automatically rolls new data into the appropriate buckets and reconstructs the aggregations. Buckets that become stale i.e. no longer relevant to any of your active time windows, are automatically cleaned up. For each bucket, Chalk stores the minimal required partial aggregate state, and at query time Chalk computes the requested aggregation by merging the partial aggregate states for every bucket that overlaps the requested time window. Chalk aligns buckets on Unix epoch (ignoring leap seconds) by default. However, if you'd like to customize the bucket starts, you can use the bucket_start field on the MaterializationWindowConfig in your definition. When serving windowed queries, Chalk uses all buckets containing any overlap with the requested time window. Materialized Windowed Aggregations ### What types of aggregations does Chalk support? Chalk supports a number of aggregations out-of-the-box. Chalk builtins are very performant as they're optimized and run natively in C++ and Rust. Aggregations automatically skip null or None values. The following table lists the supported aggregation functions: | Function | Notes | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sum` | Support for vector and scalar feature aggregations. | | `min` | | | `max` | | | `min_by_n` | Returns a list of the `n` values for the minimum `by` values. | | `max_by_n` | Returns a list of the `n` values for the maximum `by` values. | | `set_agg` | Returns the deduplicated set of values in the column as a list. Order is not guaranteed. | | `mean` | Feature type must be `Vector`, `float` or float \| None. `None` values are skipped, meaning they are not included in the mean calculation. | | `count` | | | `std` | Standard deviation. Requires at least 2 values. | | `var` | Variance. Same requirements as `std`. | | `stddev_pop` | Population standard deviation. Requires at least 1 value. | | `var_pop` | Population variance. Same requirements as `stddev_pop`. | | `skewness` | Skewness of the distribution. Computed using Fisher's moment coefficient of skewness. Accepts an optional `bias` keyword argument which defaults to `True`. Setting `True` produces the biased (population) coefficient, `False` produces the bias-corrected (sample) coefficient. Requires at least 3 values and returns null otherwise. | | `kurtosis` | Kurtosis of the distribution. Computed using Pearson moments. Accepts an optional `bias` keyword argument which defaults to `True`, and an optional `fisher` keyword argument which defaults to `True`. Setting `bias=True` gives the biased (population) excess coefficient, and setting `bias=False` makes the result bias-corrected. Setting `fisher=True` computes excess kurtosis (normal → 0), and setting `fisher=False` computes Pearson (normal → 3). Requires at least 4 values and returns null otherwise. | | `approx_count_distinct` | An approximation of the cardinality of non-null data. Uses [Apache DataSketches CPC](https://datasketches.apache.org/docs/CPC/CpcSketches.html). | | `approx_top_k` | An approximation of the most common values in a field. Takes in a required `k` parameter, such as `approx_top_k(k=5)` for the five approximately-most common values. Up to `k` values will be returned, sometimes fewer. The optional `counters` parameter sets how many distinct values the underlying sketch tracks, which defaults to `2 * k`; raising it trades memory for accuracy on high-cardinality data. For a weighted count of the most common values, you can use the parameter `by`, and set `return_total_weight` to True or False, which will return the top k values based on the `by` parameter, optionally with the total weight as well. | | `approx_percentile` | An approximation of the value a given percentage of your data falls below. Accepts `quantile` parameter, between `0` and `1`, such as `approx_percentile(quantile=.5)`. | These aggregations can be applied to DataFrame features that represent a has-many join relationship between two feature classes. These joins are defined using a join key, which can either be implicit like below, or explicit through the has-one function. ``` from chalk.features import features, _ from chalk import DataFrame, Windowed, windowed from datetime import datetime @features class Transaction: id: int amount: float timestamp: datetime user_id: "User.id" @features class User: id: int transactions: DataFrame[Transaction] total_transaction_amount: Windowed[float] = windowed( "1d", "7d", "30d", # materialize your features in a single line materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.amount, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].sum(), ) ``` Materialized aggregations can also be applied to DataFrame features that are defined using a composite join key. ``` from chalk.features import features, has_many, _ from chalk import DataFrame, Windowed, windowed from datetime import datetime @features class Transaction: id: int bank: str user_id: str timestamp: datetime amount: float @features class User: id: int bank: str transactions: DataFrame[Transaction] = has_many( lambda: (Transaction.user_id == User.id) & (Transaction.bank == User.bank) ) total_transaction_amount: Windowed[float] = windowed( "1d", "7d", "30d", # materialize your features in a single line materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.amount, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].sum(), ) top_5_bank_amounts: Windowed[list[str]] = windowed( "7d", "14d", materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.bank, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].max_by_n(_.amount, 5), ) lowest_5_bank_amounts: Windowed[list[str]] = windowed( "7d", "14d", materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.bank, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].min_by_n(_.amount, 5), ) distinct_banks: Windowed[list[str]] = windowed( "7d", "14d", materialization={"bucket_duration": "1d"}, expression=_.transactions[ _.bank, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].set_agg(), ) ``` ### Approximate Count Distinct The approx_count_distinct aggregation provides an efficient way to estimate the number of unique values in your data using the Compressed Probability Counting (CPC) sketch algorithm. ### Why use approximate count distinct? Computing exact distinct counts for large datasets can be memory-intensive and slow, especially for materialized aggregations where you need to track uniqueness across many time buckets. The CPC sketch algorithm provides: - Memory efficiency: Uses significantly less memory than storing all unique values - Mergeable sketches: Partial aggregates from different buckets can be efficiently combined - High accuracy: Provides estimates with low relative error (typically < 2% for reasonable sketch sizes) ### How do I use materialized aggregations with Chalk? Users can materialize a feature aggregation in Chalk by supplying the materialization parameter of the windowed function on the feature you want to aggregate. In the example above, we specify a bucket_duration of "1d". The materialization parameter takes in a dictionary that adheres to the schema outlined by MaterializationWindowConfig: The Duration of each bucket in the window e.g. "1h" A dictionary that specifically maps bucket durations to your window intervals e.g. if you wanted to use "10m" bucket durations for your "1d" window interval and "3d" bucket durations for your "30d" window interval. We recommend explicitly mapping bucket durations to each window if your window intervals have a wide discrepancy e.g. ten minute bucket duration with a one year window interval. Any window intervals not explicitly included in the "bucket_durations" dictionary will use your supplied "bucket_duration" by default. The schedule on which to automatically backfill the aggregation. For example, "* * * * *" or "1h". See CronTab for more details. A tolerance for late-arriving data in scheduled backfills. Each scheduled backfill run normally resumes from the previous run's upper bound; with a lookback duration, the run starts that much earlier instead, re-aggregating the overlapping span and overwriting the previously backfilled buckets. Set this to the maximum lateness of events in your data source, for example "2h". The minimum period of time for which to sample data directly via online query, rather than from the backfilled aggregations. The continuous buffer to use for offline queries, in place of continuous_buffer_duration. When unset, offline queries use continuous_buffer_duration. See A separate buffer for offline queries. A list of tag sets to use when running scheduled backfills. Each inner list is a set of query tags passed to the resolver for one scheduled backfill run. For example, [[], ["source:a"], ["source:b"]] will create three separate scheduled backfill jobs — one with no tags, one tagged source:a, and one tagged source:b. See Running backfills with tags for more details. When you query for a specific window_interval, Chalk aggregates over all buckets that intersect with the window interval. ### Aggregating through a has-one Materialized aggregations can also be defined on a feature class that reaches the events through a single has-one hop, not just directly on the class that owns the has-many. For example, a FraudFeatures class scoped to a user can count that user's transactions even though the transactions has-many lives on User: ``` from chalk.features import features, Primary, _ from chalk import DataFrame, Windowed, windowed from datetime import datetime @features class Transaction: id: Primary[int] user_id: "User.id" amount: float ts: datetime @features class User: id: Primary[int] transactions: DataFrame[Transaction] @features class FraudFeatures: id: Primary[int] user_id: User.id user: User # `_.user.` walks the has-one prefix, then aggregates the # foreign has-many — the materialization itself still lives on the # `Transaction` namespace. txns_30d: Windowed[int] = windowed( "30d", materialization=True, expression=_.user.transactions.count(), ) ``` Only a single has-one is allowed before the has-many. Chains like _.a.b.transactions.count() are rejected at parse time. The has-one's foreign join key must overlap with the has-many's local join key on the intermediate class (e.g. FraudFeatures.user_id ==(has-one) User.id ==(has-many) Transaction.user_id). ### Bucket Duration If you want to use one bucket size for all of your window intervals, you can just use the bucket_duration parameter. If your window intervals have a wide discrepancy (say, 1d and 365d), you can use the bucket_durations parameter to explicitly configure creation of differently-sized buckets tailored for each window interval. For example: ``` materialization={ "bucket_duration": "1d", "bucket_durations": { "1d": ["7d", "30d"], "5d": ["60d", "90d", "180d"], }, } ``` ### Storage Selecting the right bucket duration is a tradeoff between accuracy and storage. Shorter bucket durations will yield more accurate results, but will require more storage. To optimize for accuracy given a storage constraint, you can compute the approximate storage costs by computing the number of buckets needed for your materialized aggregation, and then computing the storage per bucket. Buckets of the same size will be reused across all relevant window interval calculations, but each configured bucket duration creates a separate set of buckets. The storage used per bucket is the sum of the size of the online storage key representation for the primary key, as well as the size of the bucket state. Size of the bucket state depends on the type of aggregation: - 9 bytes for simple aggregations such as sum, min, and max, - 18 bytes for mean, - ~50-200 bytes for more complex aggregations like approx_count_distinct. For more details on determining the tradeoff between accuracy and storage space, please reach out to the Chalk team. ### Backfilling data and managing aggregation Chalk provides multiple ways to backfill and update your windowed aggregations. You can control how your aggregations are backfilled by: - Supplying a cron expression (CronTab) to the backfill_schedule keyword parameter. - Supplying a Duration to continuous_buffer_duration. - Triggering a backfill job using the Chalk CLI with chalk aggregate backfill. ### Cron Schedule Expression ``` from chalk.features import features, _ from chalk import DataFrame, Windowed, windowed from datetime import datetime @features class Transaction: id: int amount: float timestamp: datetime user_id: "User.id" @features class User: id: int transactions: DataFrame[Transaction] total_transaction_amount: Windowed[int] = windowed( "10d", "90d", materialization={ "bucket_duration": "1d", "backfill_schedule": "0 0 * * *", "continuous_buffer_duration": "36h", }, expression=_.transactions[ _.amount, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].sum(), ) ``` In the example above, we supply a cron expression that's set to run daily at midnight ("0 0 * * *") into the backfill_schedule parameter of the materialization config. Now that backfill_schedule has been set, the total_transaction_amount feature will be backfilled every day. ### Continuous Buffer In addition to backfilling your aggregation with the recurring batch job, you can also keep recent windows fresh with live lookups through a continuous buffer duration. Configure this by supplying a Duration to continuous_buffer_duration, for example "36h". Chalk will compute data within your continuous_buffer_duration directly from your online resolvers. Note: Chalk includes all the data contained in the bucket partially overlapping with the continuous buffer duration. We suggest shortening the bucket duration, if the time delta between the end of your window interval and the boundary of the last overlapping bucket exceeds the constraints of your use case. If continuous_buffer_duration is not set, then Chalk will only serve data from the backfilled aggregations. no continuous buffer duration If continuous_buffer_duration is set, and its value is less than the duration between the end of the last bucket and the current time, Chalk will run online resolvers to compute the data for this time window. 12h continuous buffer duration If the continuous_buffer_duration is longer than the duration between the end of the last bucket and the current time, Chalk will run online resolvers to compute the data from the start of the most recently filled bucket to now. 20h continuous buffer duration ### A separate buffer for offline queries Offline queries read the same backfilled aggregations, but their trailing edge lags the online path, so the span that has to be recomputed from raw events is a different one. Supply a Duration to offline_continuous_buffer_duration to size that span independently: ``` @features class User: id: int transactions: DataFrame[Transaction] total_transaction_amount: Windowed[float] = windowed( "1d", "7d", materialization={ "bucket_duration": "1h", "continuous_buffer_duration": "1h", "offline_continuous_buffer_duration": "36h", }, expression=_.transactions[_.amount].sum(), ) ``` Offline queries for total_transaction_amount recompute the trailing 36 hours from transactions; online queries recompute the trailing hour, exactly as they would without this parameter. Aggregate backfills are unaffected — they write the buckets rather than read them. Setting offline_continuous_buffer_duration on its own, with no continuous_buffer_duration, is what makes an aggregation serve online queries purely from the backfilled buckets while offline queries still keep the recent span fresh. ### Triggering a backfill through the Chalk CLI Use chalk aggregate backfill to trigger a backfill aggregation for a windowed feature. This command is useful if you change your feature's time windows or bucket_duration values. To view existing aggregations, use chalk aggregate list. To write pre-aggregated tiles to offline storage (S3) instead of the online store, pass --store-offline. This is required before you can read tiles in offline queries using use_tile_store. --store-offline requires both --lower-bound and --upper-bound as ISO 8601 instant strings. ``` chalk aggregate backfill \ --feature user.total_transaction_amount \ --store-offline \ --lower-bound 2025-01-01T00:00:00Z \ --upper-bound 2026-04-14T00:00:00Z ``` ``` Series Namespace Group Agg Bucket Retention Aggregation Dependent Features ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 transaction user_id merchant_id amount 1d 30d sum user.txn_sum_by_merchant merchant.txn_sum_by_user 1 transaction user_id merchant_id amount 1d 30d count user.txn_count_by_merchant 2 transaction user_id amount 1d 30d sum user.txn_sum ``` The series column shows the unique ID of the timeseries data underlying our aggregation system. Each unique combination of the Namespace, Group (see group_by_windowed), and Agg (the aggregated feature) columns represents a separate timeseries. When possible, Chalk will use the same timeseries data to serve multiple features. Other useful information here is the Bucket column, which shows the current bucket size. The Retention column shows the maximum time window of any feature that depends on the given timeseries. The Dependent Features column lists the features that are served by the given timeseries. ### Running backfills with tags You can use backfill_tags to run multiple scheduled backfills for the same feature with different resolver tags. This is useful when you have multiple resolvers from different sources that get data for different primary keys and want each tagged resolver to independently populate buckets. ``` from chalk.features import features, _ from chalk import DataFrame, Windowed, windowed from chalk.streams import MaterializationWindowConfig from datetime import datetime @features class Transaction: id: int amount: float timestamp: datetime user_id: "User.id" @features class User: id: int transactions: DataFrame[Transaction] total_transaction_amount: Windowed[int] = windowed( "10d", "90d", materialization=MaterializationWindowConfig( bucket_duration="1d", backfill_schedule="0 0 * * *", backfill_tags=[[], ["source:a"], ["source:b"]], ), expression=_.transactions[ _.amount, _.timestamp <= _.chalk_now, _.timestamp > _.chalk_window ].sum(), ) ``` In this example, Chalk will create three separate scheduled backfill jobs — one with no tags, one with source:a, and one with source:b. Each run passes its tag set to the resolver as query tags, allowing you to select different resolvers per run. Resolver tags on their own do not change which backfill or query runs; you still need to pass the matching tags when you trigger a query or backfill manually. See Query Basics, the Chalk Python SDK query reference, and chalk aggregate backfill. If multiple backfills resolve data for overlapping primary keys and time buckets, later writes will overwrite earlier ones. Disjoint set of primary keys are expected, if your resolvers may return data for the same primary key under different tags the results are not additive. ### Materialized aggregations with streaming data For information on updating materialized aggregations with streaming data, see the example in the streaming documentation here. To evolve materialized aggregations across versions using versioned features, see Versioned Windowed Aggregations. ### Updating aggregations via upload_features You can upload child records into materialized aggregations directly from an external pipeline using ChalkGRPCClient.upload_features with update_mataggs=True. Chalk recomputes the affected materialized aggregation buckets from the uploaded records rather than storing raw feature values. The uploaded records must include a FeatureTime column so Chalk can determine which time buckets to update. ``` from datetime import datetime, timezone import pyarrow as pa from chalk.client.client_grpc import ChalkGRPCClient from chalk import DataFrame, FeatureTime, Primary, windowed from chalk.features import _, features @features class Transaction: id: Primary[int] user_id: "User.id" amount: float ts: FeatureTime @features class User: id: Primary[int] transactions: DataFrame[Transaction] total_spend: "Windowed[float]" = windowed( "7d", "30d", materialization={"bucket_duration": "1d"}, expression=_.transactions[_.amount].sum(), ) client = ChalkGRPCClient() now = datetime.now(timezone.utc) batch = pa.RecordBatch.from_pydict({ "transaction.id": [101, 102, 103], "transaction.user_id": [42, 42, 42], "transaction.amount": [10.0, 25.0, 5.0], "transaction.ts": [now, now, now], }) response = client.upload_features(inputs=batch, update_mataggs=True) ``` After this call, User.total_spend["7d"] and User.total_spend["30d"] will reflect the newly uploaded transactions for user 42. Caveats: - update_mataggs is only available on ChalkGRPCClient.upload_features. The HTTP-based ChalkClient.upload_features and ChalkClient.multi_upload_features do not support this option. - The FeatureTime column is required. Uploads without it cannot be routed to the correct aggregation buckets and will not update aggregations correctly. - For high-volume ingestion of raw child records into aggregations, consider ChalkGRPCClient.upload_features_bulk, a dedicated endpoint optimized specifically for this pattern. See Uploading Features for full API reference and additional upload options. ### Materialization with Offline Queries By default, offline queries that request materialized windowed aggregations run resolvers (preferring offline resolvers over online resolvers) to recompute the aggregation expression from raw events. Two planner options let you improve accuracy and performance: - align_offline_chalk_window_with_materialization — rounds each input time's lower bound down to the nearest bucket boundary, perfectly simulating the rounding behavior of the online store. - use_materialized_offline_query — bins input times to bucket boundaries and merges all complete buckets that intersect the window interval, rather than re-scanning raw events. ### Offline Store Tiling When tiles have been backfilled to offline storage with --store-offline (see Triggering a backfill through the Chalk CLI), you can instruct the planner to read directly from those pre-aggregated S3 tiles by also setting use_tile_store: True: ``` from chalk.client import ChalkClient, ResourceRequests from datetime import datetime client = ChalkClient() result = client.offline_query( input={ "user.id": [1, 2, 3] }, output=["user.total_transaction_amount"], run_asynchronously=True, recompute_features=True, planner_options={"use_tile_store": True, "use_materialized_offline_query": True}, input_times=[datetime(2026, 4, 14, 0, 0)], resources=ResourceRequests(cpu=4, memory="16Gi"), ) ``` This reads from pre-aggregated data in cloud storage, instead of recomputing aggregates from raw events, which can be dramatically faster for large historical aggregations. See Training Sets with Materialized Aggregations for a detailed discussion of trade-offs and use cases. Tiles must have been backfilled to cover the requested input_times before using use_tile_store. If tiles are missing for part of the range, the offline query will error. ### Scheduled Aggregate Backfills To keep offline tiles fresh on a recurring basis (for example, computing a new tile every day), use ScheduledAggregateBackfill. This is the recommended approach for automating offline tile backfills, as the backfill_schedule field on MaterializationWindowConfig targets the online store only. ``` from chalk import ScheduledAggregateBackfill, AggregateBackfillTarget from datetime import datetime ScheduledAggregateBackfill( name="daily_total_transaction_amount_offline_backfill", features=["user.total_transaction_amount"], schedule="0 1 * * *", # daily at 1 AM UTC target=AggregateBackfillTarget.OFFLINE, lower_bound=datetime(2025, 1, 1), ) ``` Use targets= when the same scheduled backfill should materialize both storage targets from the same run: ``` from chalk import ScheduledAggregateBackfill, AggregateBackfillTarget from datetime import datetime ScheduledAggregateBackfill( name="daily_total_transaction_amount_backfill", features=["user.total_transaction_amount"], schedule="0 1 * * *", # daily at 1 AM UTC targets=[ AggregateBackfillTarget.ONLINE, AggregateBackfillTarget.OFFLINE, ], lower_bound=datetime(2025, 1, 1), ) ``` A unique identifier for this backfill schedule. Names must be unique across all ScheduledAggregateBackfill instances in your project. The features to backfill. At least one feature is required. How often to run the backfill. Accepts a CronTab expression (e.g. "0 1 * * *") or a Duration (e.g. "1d"). A single storage target to materialize. Use AggregateBackfillTarget.OFFLINE to write pre-aggregated tiles to offline storage, or AggregateBackfillTarget.ONLINE to write aggregate state to the online store. Provide exactly one of target or targets. One or more storage targets to materialize from the scheduled run. Provide exactly one of target or targets. Optional lower bound on the time range to backfill. Converted to UTC automatically. Optional upper bound on the time range to backfill. Converted to UTC automatically. Query tags to pass to resolvers during the backfill run. The resource group to run the backfill against. ScheduledAggregateBackfill with AggregateBackfillTarget.OFFLINE is critical for operationalizing offline tiling — it ensures a fresh tile is computed each day so that training set queries can always read from pre-aggregated data without rescanning raw events. # Training Sets with Materialized Aggregations source: https://docs.chalk.ai/docs/training-sets-materialized-aggregations ## Leverage pre-materialized aggregate tiles to efficiently build training sets for windowed features. Building training sets for windowed aggregation features is one of the most expensive parts of the offline ML pipeline. A naive point-in-time join has to scan every underlying event per spine row, which scales poorly as either your historical spine or your event table grows. Chalk can re-use the same materialized aggregation tiles that back online serving to make offline training set construction dramatically cheaper. By binning historical events into fixed-width partial-aggregate buckets once, Chalk can then answer arbitrarily many windowed queries by merging a small number of pre-computed tiles per row — rather than re-scanning the raw events for every row of the training spine. ### How it works When a feature is defined with a materialization config, Chalk maintains a schedule that periodically backfills partial aggregate state into object storage — one row per (grouping_key, bucket_start) pair, storing the minimal state required to reconstitute the aggregation (e.g. running sum and count for a mean). At offline query time, you can tell the planner to read directly from these tiles rather than re-aggregating the raw event table: ``` from chalk.client import ChalkClient client = ChalkClient() ds = client.offline_query( input={"user.id": big_input_list}, input_times=big_input_time_list, output=[ User.total_transaction_amount["7d"], User.total_transaction_amount["30d"], User.mean_transaction_amount["7d"], ], planner_options={"use_materialized_offline_query": True}, ) ``` With use_materialized_offline_query=True, Chalk bins each spine row's input_time to a bucket boundary, loads the partial aggregate tiles that overlap each requested window, and merges them to produce the final aggregation. ### When this is especially useful ### Adding a new window to an existing aggregation Because the underlying tiles are window-agnostic, adding "14d" to a feature that previously declared "7d" and "30d" is almost free — Chalk merges a different subset of the already-materialized tiles. No new scan of the raw event table is required. ### Deriving new features from existing state Partial aggregate state often contains more information than the feature you originally exposed. For example, if you have materialized sum and count for a column, you can compute mean by merging those two tile streams — again without re-reading any raw events. ### High-time-point-density backfills over heavy-hitter keys Consider computing merchant.count_transactions_last_7d on a historical spine where transaction.timestamp has millions of events per merchant per day. A naive backfill has to, for each spine (merchant, timestamp) pair, scan every transaction within the preceding seven days — a pattern that scales roughly quadratically on heavy-hitter merchants, since both the number of spine rows and the number of events per window grow with activity. Pre-materialized tiles flatten this. Each merchant has at most window / bucket_duration tiles overlapping any given 7-day window, so the work per spine row becomes constant, and the backfill scales linearly in the number of bucket points rather than in the product of spine rows and events per window. ### Trade-off: bucket granularity coarsens accuracy Tiles are aligned on fixed bucket boundaries (see bucket_duration in MaterializationWindowConfig). When an offline query requests a window that does not fall exactly on a bucket boundary, Chalk must decide how to treat the partial bucket at the leading or trailing edge of the window. With use_materialized_offline_query=True, Chalk aggregates over all complete buckets that intersect the window interval, which means: - The effective window for a row may be slightly wider or narrower than the literal window_interval requested. - Two spine rows whose input_time falls in the same bucket will see identical aggregate values. For most training-set use cases this is an acceptable trade-off given the order-of-magnitude improvement in backfill time, but you should pick a bucket_duration that is small relative to your window intervals when accuracy matters. If you need the offline computation to exactly mirror the rounding behavior of online serving, also set align_offline_chalk_window_with_materialization=True. Shorter bucket durations yield more accurate offline results at the cost of more storage and more tiles to merge at query time. For most production windowed features, a bucket duration between 1% and 10% of the shortest window interval is a good starting point. use_materialized_offline_query requires that a materialization schedule has been configured for the feature and that sufficient historical tiles have been backfilled to cover the requested input_times. If tiles are missing for part of the requested range, Chalk will fall back to running resolvers for that range. ### Related reading - Materialized Windowed Aggregations — how tiles are defined, scheduled, and served online. - Offline Queries — other planner options for optimizing aggregations in offline queries. - chalk aggregate backfill — CLI for triggering a backfill of aggregate tiles. # Versioned Windowed Aggregations source: https://docs.chalk.ai/docs/versioned-windowed-aggregations ## Define multiple versions of a windowed aggregation with independent expressions, time windows, and materialization configs. ### What are versioned windowed aggregations Versioned windowed aggregations combine feature versioning with windowed features. They let you evolve a windowed aggregation's expression, time windows, or materialization configuration across multiple versions without affecting existing consumers. Like scalar versioned features, each version is independently addressable with the @ operator. Consumers that don't specify a version continue to see the default (v1), while new consumers can opt in to a newer version explicitly. Use versioned windowed aggregations when: - An aggregation's definition needs to change (e.g., switching from sum to mean), but existing consumers depend on the current behavior. - You want to A/B test a new aggregation definition alongside the existing one before promoting it broadly. - Different time windows are appropriate for different use cases, and you want both available under one feature name. ### Definition Supply a versions dict to feature(), where each value is a windowed() call: ``` from chalk import Windowed, windowed from chalk.features import features, feature, DataFrame, _ from datetime import datetime @features class Transaction: id: int user_id: "User.id" amount: float timestamp: datetime @features class User: id: int transactions: DataFrame[Transaction] total_spend: Windowed[float] = feature(versions={ 1: windowed( "30d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].sum(), materialization={"bucket_duration": "1d"}, ), 2: windowed( "30d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].mean(), materialization={"bucket_duration": "1d"}, ), }) ``` Each version can independently specify its expression, set of time windows, materialization config, and other windowed() parameters. ### Querying versioned windowed features Use the @ operator to request a specific version. Without it, the default version (v1 unless overridden with default_version) is returned. ### Python SDK ``` from chalk.client import ChalkClient from src.models import User client = ChalkClient() # Default version (v1) result = client.query( input={User.id: 42}, output=[User.total_spend["30d"]], ) # Explicitly request version 2 result = client.query( input={User.id: 42}, output=[User.total_spend["30d"] @ 2], ) ``` ### CLI In the CLI, windowed features use the FQN format .____. Non-default versions get an @ suffix. ``` # Default version (v1) — no suffix $ chalk query --in user.id=42 --out user.total_spend__2592000__ # Version 2 — @2 suffix $ chalk query --in user.id=42 --out user.total_spend__2592000__@2 ``` See Referencing windowed features for how window durations map to their FQN representations. ### Changing buckets per version Each version can define a different set of time windows. This is useful when a newer version of an aggregation needs to support a longer lookback or a different granularity: ``` @features class User: id: int transactions: DataFrame[Transaction] spend: Windowed[float] = feature(versions={ 1: windowed( "30d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].sum(), materialization={"bucket_duration": "1d"}, ), 2: windowed( "30d", "90d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].sum(), materialization={"bucket_duration": "1d"}, ), }) ``` Here v1 exposes only a 30d window, while v2 adds a 90d window. The new window is addressable as User.spend["90d"] @ 2 in Python, or user.spend__7776000__@2 in CLI/FQN form. ### Changing the default version Once all consumers have migrated to a newer version, you can promote it to the default using default_version: ``` total_spend: Windowed[float] = feature( default_version=2, versions={ 1: windowed( "30d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].sum(), materialization={"bucket_duration": "1d"}, ), 2: windowed( "30d", expression=_.transactions[ _.amount, _.timestamp > _.chalk_window, _.timestamp < _.chalk_now, ].mean(), materialization={"bucket_duration": "1d"}, ), }, ) ``` It is best practice to leave default_version=1 until all consumers have been updated to the newer version. ### Backfilling a specific version Use chalk aggregate backfill with the @N version suffix to backfill a specific version's materialized aggregation: ``` chalk aggregate backfill --feature user.total_spend@2 ``` Without the @N suffix, Chalk backfills the default version. See Materialized Windowed Aggregations for full backfill options. ### Limitations - All entries in the versions dict must be windowed() calls. You cannot mix windowed() and feature() values in the same dict. - The @ operator selects the version, not the time window. To specify both, use User.feature["30d"] @ 2. # Streaming with Materialized Aggregations source: https://docs.chalk.ai/docs/windowed-streaming ## Computing aggregate functions on streams. Windowed features represent aggregated data pertaining to discrete time buckets. Using materialized aggregations in concert with streams, you can update aggregations on the fly as new streaming data arrives. Materialized aggregations are a powerful tool to compute aggregations over time windows, combining representations of historical data with incoming real-time data to produce accurate and fresh results quickly. If you have a stream resolver that outputs a feature that is related to a feature that uses a materialized aggregation (more on this later), the materialized aggregations will be updated as new messages arrive. ### Setting up a materialized aggregation Setting up features for materialized aggregation with streaming is almost identical to the method described in the materialized_aggregations docs. ``` from datetime import datetime from chalk.features import features, has_many, DataFrame, _ from chalk.streams import Windowed, windowed @features class Transaction: id: str user_id: 'User.id' amount: float ts: datetime @features class User: id: str name: str transactions: DataFrame[Transaction] sum_transaction_amount: Windowed[float] = windowed( "30d", expression=_.transactions[_.amount].sum(), materialization=True, default=0.0, ) mean_transaction_amount: Windowed[float] = windowed( "30d", expression=_.transactions[_.amount].mean(), materialization=True, default=0.0, ) ``` Now, set up your streaming resolvers referencing these features using a stream resolver: ``` from pydantic import BaseModel from chalk.features import _ from chalk.features.resolver import make_stream_resolver from chalk.streams import KafkaSource class TransactionMsg(BaseModel): id: str user_id: str amount: float process_transaction_topic = make_stream_resolver( name="process_transaction_topic", source=KafkaSource(name="transactions"), message_type=TransactionMsg, output_features={ Transaction.id: _.id, Transaction.user_id: _.user_id, Transaction.amount: _.amount, }, ) ``` That's it! When a message arrives on the stream, Chalk identifies which materialized aggregates the resolver's output features can update, and applies the new row to those aggregates' bucket state. For the resolver above: Drives the _.transactions traversal from User, so the new row lands in the right user's bucket. Feeds both User.sum_transaction_amount and User.mean_transaction_amount. Each Kafka message is therefore folded into the buckets of both aggregates for the matching User.id. A few notes on what counts as "relevant": - Version match required. An aggregate whose aggregate_on column is a different version of the column the resolver actually writes is skipped — see Per-version isolation below. - Opt-out per resolver. Set updates_materialized_aggregations=False on the stream resolver to skip mat-agg updates for that resolver entirely. ### Per-version isolation When a versioned scalar backs versioned aggregates, a stream resolver's writes only flow into the aggregates sourced from the version that resolver actually writes. An aggregate whose aggregate_on column is a different version of the freshly written column is skipped for that batch. ``` from chalk.features import DataFrame, FeatureTime, Primary, feature, features from chalk.features import _ from chalk.features.resolver import make_stream_resolver from chalk.streams import Windowed, windowed @features class Transaction: id: Primary[str] user_id: "User.id" amount: float = feature( versions={1: feature(), 2: feature()}, default_version=1, ) ts: FeatureTime @features class User: id: Primary[str] transactions: DataFrame[Transaction] total_spend: "Windowed[float]" = feature( versions={ 1: windowed( "30d", expression=_.transactions[_.amount].sum(), materialization=True, default=0.0, ), 2: windowed( "30d", expression=_.transactions[_.amount @ 2].sum(), materialization=True, default=0.0, ), }, default_version=1, ) class TransactionMsg(BaseModel): id: str user_id: str amount: float # Stream resolver writes amount @ 2 (the parity column). make_stream_resolver( name="process_transaction_topic", source=transaction_source, message_type=TransactionMsg, output_features={ Transaction.id: _.id, Transaction.user_id: _.user_id, Transaction.amount @ 2: _.amount, # noqa: F722 }, ) ``` For each message, only User.total_spend["30d"] @ 2 is updated. User.total_spend["30d"] (v1, sourced from _.amount) is left untouched, even though the new Transaction row exists at the same primary key — its v1 column wasn't part of this resolver's output. The isolation only kicks in when the source feature (the column referenced inside the aggregation expression) is itself versioned. If the underlying scalar is unversioned, every version of an aggregate that references it updates normally on every write. For a full guide to defining, querying, and backfilling versioned windowed aggregations, see Versioned Windowed Aggregations. ### Online Storage Similar to materialized aggregations that are backfilled from a non-streaming data source, when the Chalk streaming server receives a message on a stream, if the streaming resolver is used to compute a materialized aggregation, then the relevant data will be used to update the correct bucket for the materialized aggregation. When a stream resolver is used to compute a materialized aggregation, the minimum amount of data required to compute the aggregation will be stored for each bucket. For example, for a sum aggregation, the sum of all values in the bucket will be stored. Then when new messages arrive, the appropriate bucket based on the timestamp of the message will be updated with the new value. Because this data is stored online, you can access the values through both online and offline queries. # Stream Resolvers source: https://docs.chalk.ai/docs/streams ## Define streaming resolvers with Chalk expressions. Stream resolvers consume messages from a streaming source — such as Kafka, Kinesis, or Pub/Sub — and write Chalk features in real time. Resolvers are defined with statically-typed expressions that are compiled and executed as vectorized C++, enabling high-throughput processing. For source configuration (credentials, topic names, late-arrival policies), see Stream Sources. For local and end-to-end testing of resolvers, see Testing Stream Resolvers. ### Overview Stream resolvers are defined using the make_stream_resolver() function. The definition takes a message type and a mapping from Chalk features to expressions. In production, raw streaming messages are first converted into the given message type. Then, the output features of the resolver are computed based on the given expressions. ### Basic Example Let's start with a simple example that processes user data from a Kafka stream: ``` from chalk.features import Features, Primary, features, _ from chalk.streams import KafkaSource from chalk.features.resolver import make_stream_resolver from chalk import functions as F from pydantic import BaseModel import pyarrow as pa import datetime as dt # Define the Kafka stream source kafka_source = KafkaSource(name="user_updates") # Define the feature class @features(max_staleness="30d", etl_offline_to_online=True) class User: user_id: Primary[int] full_name: str email_address: str | None updated_at: dt.datetime | None # Define the message schema class ContactInfo(BaseModel): phone_number: str email_address: str | None class UserMessage(BaseModel): id: int # maps to user_id name: str updated_at: int # epoch microseconds contact_info: ContactInfo # Create a stream resolver users_streaming_resolver = make_stream_resolver( name="get_users_stream", # The resolver name source=kafka_source, # Your Kafka stream source message_type=UserMessage, output_features={ User.user_id: _.id, User.full_name: _.name, User.email_address: _.contact_info.email_address, User.updated_at: F.from_unix_seconds(_.updated_at / 1_000_000), }, ) ``` Here, stream messages will come in as JSON strings whose structure matches the definition of the UserMessage model. The output features are defined as simple field accesses into the JSON struct. Note that nested access, such as _.contact_info.email_address, works as expected. Similar to feature definitions, scalar functions can be used to process the raw message fields. Here, we convert a raw integer timestamp in microseconds into a UTC datetime. Note that the expressions here are essentially the same as those used in feature definitions. However, the semantics are slightly different: The "root" here (_), refers not to a namespace of features, but rather a table column whose datatype is based on the stream resolver's message_type. Thus, _.amount does not mean "get me Chalk feature amount in the current namespace" but rather "access struct field amount on this column." In addition, since these expressions represent projections on single columns, dataframe operations such as _.transactions[_.amount, _.amount > 100].sum() don't apply here. ### Example Stream Messages Here's what an incoming message might look like: ``` { "id": 123, "name": "John Smith", "updated_at": 1700000000000000, "contact_info": { "phone_number": "555-555-5555", "email_address": "john.smith@gmail.com" } } ``` This message will get parsed into the following data: ``` User( id=123, full_name="John Smith", email_address="john.smith@gmail.com", updated_at=datetime.datetime(2023, 11, 14, 22, 13, 20, tzinfo=datetime.timezone.utc) ) ``` ### Supported message types The following message types are supported: - Pydantic BaseModels or Python @dataclasses: Incoming string messages will be deserialized from JSON into structs. - Protocol buffer messages: The incoming binary messages will be deserialized into structs with the various fields. - str or bytes: The incoming messages can also be treated as simple strings or bytes. In this case, your feature expressions can't access struct fields on the message, but they can use scalar string/binary functions: ``` { Event.timestamp: _.timestamp, # INVALID -- strings don't have a 'timestamp' field Event.raw_data_cleaned: F.trim(F.lower(_)), # Here, `_` refers to the string message as whole. Event.timestamp: F.json_value(_, "$.metadata.timestamp") # This parses the input string as JSON and extracts a single field. } ``` Note: Not all pydantic or protobuf types are supported. The message type needs to be converted into a well-defined PyArrow schema. Types that are recursive and can be arbitrarily nested will fail to be parsed. Similarly, Pydantic types with type annotations such as Any or complex union types are also not supported. To process these types in a stream resolver, use a custom parse function to only extract specific fields with well-known datatypes. ``` class PhoneInfo(BaseModel): number: str area_code: str class EmailInfo(BaseModel): address: str class Parent(BaseModel): id: int parent: "Parent | None" class UsersMessage(BaseModel): id: int account_id: int | str # INVALID -- 'account_id' field's datatype is not well-defined card_id: int | None # Nullable types OK contact: PhoneInfo | EmailInfo # INVALID -- complex union types not supported parent: Parent | None # INVALID -- even if finitely nested in practice, the schema of this can't be determined ahead of time since it contains arbitrarily many sub-columns. ``` ### Custom Parse Functions By default, the message parsing logic is inferred automatically from the message_type. For Pydantic types or dataclasses, JSON parsing is used. For protobuf message classes, these are parsed from the protobuf's binary representation. However, it is also possible to provide custom parsing logic. In addition to handling custom message formats, this also allows the stream resolver to skip a subset of messages. For instance, one might have multiple message types on the same Kafka topic, or perhaps a stream of events where only events of type "CREATE" are relevant. The make_stream_resolver() function has an optional parse argument that takes an expression. If provided, this expression will be used to convert raw message bytes into the expected message type. If the expression returns None, that message will be skipped. For example, consider the following protobuf definition: ``` message User { enum UserType { USER_TYPE_UNSPECIFIED = 0; USER_TYPE_GUEST = 1; USER_TYPE_MEMBER = 2; } message ContactInfo { optional string phone_number = 1; optional string email_address = 1; } uint32 id = 1; string name = 2; UserType user_type = 3; optional ContactInfo contact_info = 4; uint64 updated_at = 5; } ``` Let's say our system produces protobuf messages w/ an unusual format (a 6-byte prefix is added). We would like to strip the prefix before parsing and also ignore messages for users of type "GUEST". We can use a custom parse expression to do this: ``` from messages.user_pb2 import UserMessage # Import the generated protobuf classes from chalk import functions as F from chalk.features.resolver import make_stream_resolver # Strip first 6 bytes, then parse the protobuf message parsed_message = F.proto_deserialize(F.substr(_, 6, None), UserMessage) # Exclude any UserMessages whose user_type == GUEST parse_expression = F.if_then_else( parsed_message.user_type == UserMessage.UserType.GUEST, None, parsed_message ) users_streaming_resolver = make_stream_resolver( name="get_users_stream", source=kafka_source, message_type=UserMessage, parse=parse_expression, # Custom parse expression provided here output_features={ User.user_id: _.id, User.full_name: _.name, User.email_address: _.contact_info.email_address, User.updated_at: F.from_unix_seconds(_.updated_at / 1_000_000), }, ) ``` ### Fan-out: one message to many rows By default a stream resolver writes one feature row per message. When a single message carries a collection — a JSON array, or a repeated protobuf field — you may want one output row per element instead. To do this, write a parse expression that returns a list: Chalk explodes the list and runs output_features once per element. Two things change relative to the 1:1 case: - Set message_type to list[Element] — the list type is what tells the engine to explode. Element describes one output row's worth, and the parse expression produces the list of them. (A singular message_type=Element would not fan out.) - output_features expressions run per exploded element, so _ refers to one element at a time. An empty (or absent) list produces zero rows for that message — the natural way to express "this message contributes nothing." To skip a message entirely, a parse expression may also return None (see Custom Parse Functions). ### Fan out over a JSON array Use F.json_extract_array to pull the array out of the message, then F.array_transform to build one struct per element. Inside the transform you can reference both the current element and outer fields of the message, so per-element derived keys are straightforward: ``` class UserTagRow(BaseModel): user_id: str tag: str user_tags_resolver = make_stream_resolver( name="process_user_tags", source=kafka_source, message_type=list[UserTagRow], # list[Element] -> engine fans out one row per element parse=F.array_transform( # json_extract_array yields large_list; cast the whole array to # large_list so the lambda's element type unifies. F.cast(F.json_extract_array(F.bytes_to_string(_, "utf-8"), "$.tags"), pa.large_list(pa.large_string())), lambda tag: F.struct_pack( { # json_value also yields an arrow.json extension type — cast before string ops like `+`. "user_id": F.cast(F.json_value(F.bytes_to_string(_, "utf-8"), "$.user_id"), pa.large_string()), "tag": tag, } ), item_type=pa.large_string(), # type of each array element ), output_features={ # these run once per exploded element UserTag.id: _.user_id + ":" + _.tag, UserTag.user_id: _.user_id, UserTag.tag: _.tag, }, ) ``` A message with "tags": [] produces no rows. ### Fan out over a repeated protobuf field Suppose the message is an Order carrying a repeated items field: ``` message Order { string id = 1; repeated LineItem items = 2; } message LineItem { string id = 1; string sku = 2; } ``` Deserialize the message with F.proto_deserialize and transform its repeated items field. Each element is itself a struct, so array_transform needs an explicit item_type — pass the element's protobuf message class: ``` from messages.order_pb2 import Order, LineItem # your generated protobuf classes class LineItemRow(BaseModel): order_id: str item_id: str sku: str order_items_resolver = make_stream_resolver( name="process_order_items", source=kafka_source, message_type=list[LineItemRow], # list[Element] -> engine fans out per element parse=F.array_transform( F.proto_deserialize(_, Order).items, # repeated child field lambda item: F.struct_pack( { "order_id": F.proto_deserialize(_, Order).id, # parent field copied into each row "item_id": item.id, "sku": item.sku, } ), item_type=LineItem, # the element's protobuf message class ), output_features={ OrderLineItem.id: _.order_id + ":" + _.item_id, OrderLineItem.sku: _.sku, }, ) ``` For testing fan-out resolvers locally, see Testing a fan-out resolver. ### Filtering Multiplexed Topics with Message Headers A single Kafka topic often carries multiple message types — for example, a User event and a Transaction event distinguished by a header like X-Chalk-MessageType. The recommended way to route each type to the right resolver is the message_header_filters argument on make_stream_resolver. Each resolver declares the (header_name, header_value) pairs it wants to consume, and the streaming server drops any message whose Kafka headers don't match before it is deserialized. ``` users_streaming_resolver = make_stream_resolver( name="get_users_stream", source=kafka_source, message_type=UserMessage, output_features={ User.user_id: _.id, User.full_name: _.name, }, message_header_filters=[("X-Chalk-MessageType", b"User")], ) transactions_streaming_resolver = make_stream_resolver( name="get_transactions_stream", source=kafka_source, # same topic message_type=TransactionMessage, output_features={ Transaction.id: _.id, Transaction.amount: _.amount, }, message_header_filters=[("X-Chalk-MessageType", b"Transaction")], ) ``` A few things to keep in mind: - Header values are matched as raw bytes, not strings — use b"User", not "User". - Multiple entries in the list are combined with logical AND: a message must match every (name, value) pair to be processed. - The source must support headers. Kafka does; the validator will raise at deployment time if you attach filters to a source that doesn't. Use message_header_filters instead of inspecting headers inside a custom parse expression. Header filtering runs before deserialization, so messages destined for other resolvers are discarded cheaply without ever being parsed. If your filtering logic needs to look at the message body — not just headers — fall back to a parse expression that returns None for messages you want to skip (see Custom Parse Functions). You can also combine both: pre-filter on headers, then run additional body-level filtering inside parse. ### Performance Considerations Stream resolvers compile to vectorized C++, so a single CPU core can handle thousands of messages per second, and processing is parallelized across cores automatically. Expressions are validated at compile time, so schema and type mismatches surface before deployment rather than at runtime. Stream resolvers are ideal for: - High-volume data ingestion (>1000 messages/second) - Low-latency feature computation (<10ms) - Simple to moderate transformations - Stateless processing For complex business logic that requires external API calls or elaborate error handling, a regular Python @online resolver downstream of a stream resolver is usually a better fit. ### Writing to a Sink Stream resolvers can write computed features to downstream Kafka topics using the Sink parameter. This enables chaining of stream processing stages, where one resolver's output becomes another resolver's input. This pattern is particularly useful for: - Multi-stage processing pipelines: Break complex transformations into discrete stages - Feature routing: Direct specific features to specialized processing streams - Model inference workflows: Send features to GPU-accelerated resolvers for heavy computation The sink specifies which output features should be written to the destination stream. These features are computed by the resolver or by downstream resolvers that depend on the initial resolver's outputs. ### Example: Item Embedding Pipeline This example demonstrates a two-stage pipeline where items are first processed through a stream resolver, then routed to a GPU-accelerated resolver for embedding generation: ``` from chalk.streams import KafkaSource from chalk.features.resolver import Sink, make_stream_resolver from chalk.features import _ from chalk import online from pydantic import BaseModel from src.models import Item import numpy as np class ItemMessage(BaseModel): id: int name: str price: float description: str image_url: str new_items = KafkaSource( name="new_items", ) embedding_stream = KafkaSource( name="item_embeddings", ) make_stream_resolver( name="process_new_items", source=new_items, message_type=ItemMessage, output_features={ Item.id: _.id, Item.name: _.name, Item.price: _.price, Item.description: _.description, Item.image_url: _.image_url, }, sink=Sink( send_to=embedding_stream, output_features=[Item.id, Item.embedding], ), ) @online(resource_hint="gpu") def run_item_embedding_model( price: Item.price, description: Item.description, state: Item.state, image_url: Item.image_url, ) -> Item.embedding: """Mock generate item embedding using a pre-trained model.""" # In a real implementation, this function would load a pre-trained model # and generate an embedding based on the input features or make an API # call to an external service. return np.random.rand(128).tolist() ``` In this example: - The process_new_items resolver consumes messages from the new_items Kafka source and extracts basic item features - The sink configuration specifies that Item.id and Item.embedding should be written to the item_embeddings stream - When a message is processed, Chalk computes Item.embedding by calling the run_item_embedding_model resolver, which runs on a GPU-enabled instance - The computed features are then written to the item_embeddings topic This pattern allows the lightweight stream resolver to handle high-throughput message parsing while delegating compute-intensive embedding generation to specialized GPU resources. The sink acts as a bridge, automatically triggering the downstream computation and routing results to the appropriate stream. Messages can be encoded as either Arrow IPC streams or as JSON. The fully qualified feature names are used as the column names in the output stream or as the JSON keys. ### Deduplication Stream resolvers can drop duplicate messages by passing a Deduplication policy to make_stream_resolver. For each incoming message, Chalk evaluates the on expression to compute a deduplication key. If that key has already been seen within the within retention window, the message is skipped — no features are written and no downstream computation is triggered. ``` from chalk.features.resolver import Deduplication, make_stream_resolver transactions_streaming_resolver = make_stream_resolver( name="get_transactions_stream", source=kafka_source, message_type=TransactionMessage, output_features={ Transaction.id: _.id, Transaction.amount: _.amount, Transaction.submitted: F.from_unix_seconds(_.submitted / 1_000_000), }, deduplication=Deduplication(on=_.id, within="24h"), ) ``` The on expression is a regular underscore expression over the parsed message, so the dedup key can be any field or computed value — for example _.id, a composite like F.concat(_.user_id, "-", _.event_id), or a hash. The within argument accepts a Chalk duration string ("24h", "7d") or a datetime.timedelta. ### Performance Overhead Deduplication is not free: every message incurs an extra round-trip to a key-value store (Redis Lightning or DynamoDB, depending on your online store configuration) to check for and record the dedup key. This adds latency per message and consumes storage proportional to the volume of unique keys seen within the retention window. Consider the tradeoff before enabling deduplication on very high-throughput streams, and keep the within window as short as your duplicate-tolerance requirements allow. Deduplication is currently supported only when the online store is Redis Lightning or DynamoDB. ### Skip Writing to the Offline Store If you already store a historical log of your stream messages in a data warehouse, you can disable stream persistence to the Chalk offline store. This can be done by setting the skip_offline parameter of make_stream_resolver: ``` users_streaming_resolver = make_stream_resolver( name="get_users_stream", source=kafka_source, message_type=UsersMessage, output_features={ User.user_id: _.id, User.full_name: _.name, User.email_address: _.contact_info.email_address, User.updated_at: F.from_unix_seconds(_.updated_at / 1_000_000), }, skip_offline=True, # don't write resolved events from stream to offline store ) ``` ### Environment-specific Stream Resolvers A stream resolver can be restricted to a specific environment through the environment parameter of make_stream_resolver: ``` @features(max_staleness="30d", etl_offline_to_online=True) class Transaction: transaction_id: Primary[int] submitted: dt.datetime amount: int class TransactionMessage(BaseModel): id: int # maps to transaction_id submitted: int # epoch microseconds amount: int # new resolver to experiment with new Transaction class experimental_streaming_resolver = make_stream_resolver( name="new_transaction_stream_resolver", source=kafka_source, message_type=TransactionMessage, output_features={ Transaction.transaction_id: _.id, Transaction.submitted: F.from_unix_seconds(_.submitted / 1_000_000), Transaction.amount: _.amount, }, environment=["dev"], # only deploy resolver in "dev" environment ) ``` ### Migrating from @stream Earlier versions of Chalk used the @stream decorator to define streaming resolvers in Python. Existing @stream resolvers continue to run, but new resolvers should use make_stream_resolver — it compiles to vectorized C++ and offers significantly better performance for high-throughput streams. Most @stream resolvers translate directly to make_stream_resolver by replacing the decorated function body with an output_features mapping: ``` from chalk.features.resolver import make_stream_resolver from chalk import stream, Features # Old: Python @stream decorator @stream(source=user_stream) def process_users(message: UsersMessage) -> Features[Users]: return Users( user_id=message.id, has_been_evicted_bool=message.qualification.has_been_evicted, monthly_income_min=message.qualification.monthly_income.min, # ... more processing ) # New: make_stream_resolver process_users_native = make_stream_resolver( name="process_users_native", message_type=UsersMessage, output_features={ Users.user_id: _.id, Users.has_been_evicted_bool: _.qualification.has_been_evicted, Users.monthly_income_min: _.qualification.monthly_income.min, # ... same mappings using underscore expressions }, source=user_stream, ) ``` Both resolvers produce identical results. # Stream Sources source: https://docs.chalk.ai/docs/stream-sources ## Consuming Kafka streams. Chalk supports Kafka, Kinesis, and Pub/Sub as streaming sources for @stream resolvers. ### Configuration options You can configure the Kafka-, Kinesis-, or Pub/Sub-specific options in two places: through the Chalk dashboard as a named integration, or explicitly in code on the source object. See Choosing where to configure for the trade-offs. ### Named integration If configuring through the dashboard, your Kafka/Kinesis integration must be given a name. You can then reference this name in your stream configuration. The name of the integration as configured in your Chalk dashboard. ### Explicit configuration You can instead choose to configure the source in your code directly for the following source types. The Kafka broker's host name without the security protocol. You can specify multiple brokers by passing a list of strings. Example: "localhost:9092" or ["localhost:9092", "localhost:9093"] The topic or topics to subscribe to. An S3 or GCS URI that points to the keystore file that should be used for brokers. You must configure the appropriate AWS or GCP integration in order for Chalk to be able to access these files. Optionally, you may specify a prefix that will be used for client ids generated by @stream resolvers that consume this source. Optionally, you may specify a prefix that will be used for group ids generated by @stream resolvers that consume this source. Security protocol passed directly to Kafka. Defaults to 'PLAINTEXT'. Authentication mechanism when security_protocol is configured for SASL_PLAINTEXT or SASL_SSL. Defaults to 'PLAIN'. Username for SASL PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 authentication. Defaults to null. Password for SASL PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 authentication. Defaults to null. The name of your stream. Either this or the stream_arn must be specified. The ARN of your stream. Either this or the stream_name must be specified. The AWS region, e.g. "us-east-2". The AWS access key id credential, if not already set in the environment. The AWS secret access key credential, if not already set in the environment. The AWS session token credential, if not already set in the environment. An optional endpoint to hit the Kinesis server. An optional role ARN for the consumer to assume. The project id that your Pub/Sub source resides in. The subscription id of your Pub/Sub source. ### Choosing where to configure Both approaches resolve to the same source configuration at runtime, so the choice is about where the values live and how they are managed, not about capability. | | Named integration (dashboard) | Explicit configuration (code) | | ---------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | Where values live | Stored in your environment and backed by your cloud secret manager | In your source code / repository | | Secrets (passwords, keys) | Encrypted at rest, never committed to the repo | Encrypted at rest as a secret; Can manage state via repo or agent OR through the [environment Secret page](/docs/env-vars) | | Changing a data source value | Edit in the dashboard and `chalk apply` | Edit code and `chalk apply` | | Discoverability | Lives under data sources in the environment | Lives in repo and/or [Secret page](/docs/env-vars) in UI | | Referenced in code as | `KafkaSource(name="...")` | `KafkaSource(bootstrap_server=..., topic=..., ...)` | ### When to use each named integration in the dashboard (recommended) - The source carries secrets (SASL password, SSL keystore, AWS keys). The dashboard stores them in your cloud secret manager and injects them into the running containers, so credentials never live in the repo. - Configuring the named integration through the dashboard allows all information pertaining to the Kafka data source to live with the definition of the data source in the UI. The code base only needs to reference by name and topic, keeping environment secret management out of the codebase itself. For more information on configuring data sources and previewing messages from a stream, checkout Configuring Data Sources. ``` from chalk.streams import KafkaSource transaction_stream_source = KafkaSource( name="transaction_stream", # name of Kafka data source in UI ) ``` ### When to configure explicitly in code - You want to manage secrets through your code base and upload via chalk secret command. - You want agents to be able to rewrite secrets. - Legacy users of streams. ``` import os from chalk.streams import KafkaSource transaction_stream_source = KafkaSource( name="transaction_stream", bootstrap_server=os.getenv("CHALK_KAFKA_BOOTSTRAP_SERVER"), sasl_username=os.getenv("CHALK_KAFKA_SASL_USERNAME"), sasl_password=os.getenv("CHALK_KAFKA_SASL_PASSWORD"), sasl_mechanism=os.getenv("CHALK_KAFKA_SASL_MECHANISM"), security_protocol=os.getenv("CHALK_KAFKA_SECURITY_PROTOCOL"), topic="transactions.v1", ) ``` ### Example ``` from pydantic import BaseModel from chalk import stream, feature from chalk.streams import KafkaSource from chalk.features import features, Features @features class User: id: str favorite_color: str = feature(max_staleness='30m') class UserUpdateBody(BaseModel): user_id: str favorite_color: str kafka_source = KafkaSource(name="user_favorite_color_updates") @stream(source=kafka_source) def fn(message: UserUpdateBody) -> Features[User.id, User.favorite_color]: return User( id=message.value.user_id, favorite_color=message.value.favorite_color ) ``` ### Autoscaling Chalk can scale the streaming server on message backlog with KEDA: consumer-group lag for Kafka sources, and undelivered messages for Pub/Sub sources. Kafka sources that authenticate with MSK IAM are not supported. Enable it per stream resolver in the dashboard under Infrastructure > Resources, in the Streaming Autoscaling panel of the streaming server that runs the resolver. The service's Min Instances and Max Instances bound the scaling range; see Resource Configuration for those settings. Backlog autoscaling requires a named integration. The scaler reads its connection configuration from the integration: bootstrap servers, topic, credentials, and group ID prefix for Kafka; project and subscription for Pub/Sub. A source configured only in code cannot be autoscaled. This includes named integrations that have overrides in code, such as the kafka_source example below. ``` kafka_source = KafkaSource( name="kafka_source", group_id_prefix="bad-overridden-group-id-prefix", # this is incorrectly configured, and will not deploy the correct KEDA scaler ) ``` ### Additional Configurations For Kafka data sources, Chalk supports the following Kafka configuration values for your integration. You can set these configurations within your Kafka Data Source either by constructing a dict of {"your.kafka.config": "config_value"} and setting that under the ADDITIONAL_ENGINE_ARGUMENTS field, or by passing in KafkaSource(additional_kafka_args=...) in your code-based data source configuration. ### Supported Kafka Configurations Fetch & Polling Configuration - fetch.min.bytes - Minimum amount of data the server should return for a fetch request - fetch.max.bytes - Maximum amount of data the server should return for a fetch request - fetch.max.wait.ms - Maximum time the server will block before answering the fetch request - max.partition.fetch.bytes - Maximum amount of data per-partition the server will return - max.poll.records - Maximum number of records returned in a single call to poll() - max.poll.interval.ms - Maximum delay between invocations of poll() Timeout Configuration - session.timeout.ms - Timeout used to detect consumer failures - heartbeat.interval.ms - Expected time between heartbeats to the consumer coordinator - request.timeout.ms - Maximum amount of time the client will wait for the response of a request - consumer.timeout.ms - Timeout for consumer operations - poll.timeout - User-friendly alias for consumer.timeout.ms - connections.max.idle.ms - Close idle connections after this many milliseconds Retry & Reliability - retry.backoff.ms - Time to wait before attempting to retry a failed request - reconnect.backoff.ms - Base amount of time to wait before attempting to reconnect - reconnect.backoff.max.ms - Maximum amount of time to wait when reconnecting Metadata & Discovery - metadata.max.age.ms - Period of time before forcing a refresh of metadata Message Processing - check.crcs - Automatically check the CRC32 of the records consumed - isolation.level - Controls how to read messages written transactionally - exclude.internal.topics - Whether internal topics should be exposed to the consumer Protocol Settings - receive.buffer.bytes - Size of the TCP receive buffer (SO_RCVBUF) - send.buffer.bytes - Size of the TCP send buffer (SO_SNDBUF) # Testing Stream Resolvers source: https://docs.chalk.ai/docs/streaming-testing ## Locally verify stream resolver parsers and end-to-end scenarios. Chalk provides three tools for testing stream resolvers, each with a different scope. Pick the one that matches what you want to verify: | Tool | Scope | When to use | | -------------------------------------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------- | | [`check_stream_parsing`](#local-parser-checks-with-check-stream-parsing) | Parser only, local | Fast iteration on the `parse` expression and `output_features` mapping | | [`check_stream_scenario`](#end-to-end-scenarios-with-check-stream-scenario) | Full pipeline against a branch, multi-step | Verify materialized windowed aggregates, or behavior that depends on prior online-store state | | [`test_streaming_resolver`](#testing-against-a-deployed-branch-with-test-streaming-resolver) | Full resolver against a deployed branch | Sanity-check a deployed resolver against real bytes; inspect the resulting Arrow table | ### Local parser checks with check_stream_parsing The parsing logic of a stream resolver can be tested locally via chalk.testing.check_stream_parsing. This verifies that your parse expression and output_features mappings produce the expected results, without deploying to a branch or running the full resolver pipeline. ``` import datetime import json import pyarrow as pa from pydantic import BaseModel from chalk import functions as F from chalk.features import _ from chalk.features.resolver import make_stream_resolver from chalk.testing import StreamMessage, check_stream_parsing class UserMessage(BaseModel): id: int name: str updated_at_us: int users_streaming_resolver = make_stream_resolver( name="get_users_streaming", source=kafka_source, message_type=UserMessage, parse=F.struct_pack( { "id": F.cast(F.json_value(F.bytes_to_string(_, "utf-8"), "$.id"), pa.int64()), "name": F.json_value(F.bytes_to_string(_, "utf-8"), "$.name"), "updated_at_us": F.cast( F.json_value(F.bytes_to_string(_, "utf-8"), "$.updated_at_us"), pa.int64() ), } ), output_features={ User.user_id: _.id, User.full_name: _.name, User.updated_at: F.from_unix_seconds(_.updated_at_us / 1_000_000), }, ) check_stream_parsing( users_streaming_resolver, [ StreamMessage( message=json.dumps( {"id": 123, "name": "John Smith", "updated_at_us": 1_700_000_000_000_000} ).encode(), parsed=User( user_id=123, full_name="John Smith", updated_at=datetime.datetime( 2023, 11, 14, 22, 13, 20, tzinfo=datetime.timezone.utc ), ), ), StreamMessage( message=json.dumps( {"id": 456, "name": "Alice Liddell", "updated_at_us": 1_710_000_000_000_000} ).encode(), parsed=User( user_id=456, full_name="Alice Liddell", updated_at=datetime.datetime( 2024, 3, 9, 16, 0, 0, tzinfo=datetime.timezone.utc ), ), ), ], ) ``` Each StreamMessage contains the raw message bytes and the expected parsed feature output(s). The parsed field controls what is checked: - parsed=Feature(...) — assert the message produces exactly one row matching these fields. - parsed=[Feature(...), Feature(...)] — for a fan-out resolver (list-returning parse), assert it produces exactly these rows, in emission order. - parsed=[] — assert the message produces zero rows (for example an empty array or a filtered message). Use parsed=[], not parsed=None, when you want to verify that a message was dropped. ### Testing a fan-out resolver For a resolver whose parse returns a list, pass a list of expected rows — one per emitted element. The empty case is asserted with parsed=[]. The user_tags_resolver below is the JSON-array fan-out resolver from the stream resolver docs: ``` check_stream_parsing( user_tags_resolver, [ # one message with two tags -> two rows StreamMessage( message=json.dumps({"user_id": "u1", "tags": ["red", "blue"]}).encode(), parsed=[ UserTag(id="u1:red", user_id="u1", tag="red"), UserTag(id="u1:blue", user_id="u1", tag="blue"), ], ), # tag-less message -> zero rows ([] asserts emptiness; None would skip the check) StreamMessage( message=json.dumps({"user_id": "u2", "tags": []}).encode(), parsed=[], ), ], ) ``` ### End-to-end scenarios with check_stream_scenario check_stream_parsing verifies the parser in isolation, but it does not exercise the rest of the streaming pipeline: feature uploads, materialized windowed aggregates, or any computation that depends on prior online-store state. To test those behaviors, use ChalkClient.check_stream_scenario, which runs the full pipeline against a branch. The method accepts a sequence of step objects that are processed left-to-right: - StreamMessage(message=..., stream_resolver=..., timestamp=...) — parses message locally using the resolver's parse expression (the same path as check_stream_parsing), then uploads the resulting features against branch with materialized-aggregate updates enabled. Set timestamp to control the feature time of the uploaded row (useful for out-of-order or windowed tests). - UploadFeatures({Feature: value_or_list, ...}) — primes the branch online store directly without going through a resolver. Each value can be a scalar (uploaded as one row) or a list (bulk upload, one row per element). All features in a single UploadFeatures step must agree on row count. This path does not refresh materialized aggregates. - FeatureAssertion(input=..., output=...) — runs an online query whose inputs are the set fields of input and whose requested outputs are the set fields of output, then compares each returned value against the expected value. A mismatch raises AssertionError. For Windowed[...] features, pass a {window: value} dict on either side, e.g. transaction_count={"1d": 1, "30d": 7}, and each bucket is checked independently. check_stream_scenario parses each StreamMessage locally via the same chalkdf-backed path as check_stream_parsing, so the chalkdf package must be installed in the test environment. Uploads and assertions run against a Chalk branch, so the target environment must have a branch online store configured. Without it, the uploads produced by each StreamMessage and UploadFeatures step have nowhere to land and the subsequent FeatureAssertion queries will not see them. The example below seeds three merchant rows, then drives a transaction stream and a user-creation stream, and finally asserts that the windowed transaction_count aggregate reflects the single transaction processed: ``` from datetime import datetime from chalk.client import ChalkClient from chalk.testing import FeatureAssertion, StreamMessage, UploadFeatures ChalkClient().check_stream_scenario( UploadFeatures({ Merchant.id: [501, 502, 503], Merchant.category: ["grocery", "restaurant", "fuel"], }), StreamMessage( timestamp=datetime(2026, 1, 1, 12, 20), message=b'{"id": 1, "user_id": 1, "merchant_id": 501, "amount": 10.0}', stream_resolver=transaction_stream, ), StreamMessage( timestamp=datetime(2026, 1, 1, 12, 20), message=b'{"id": 1, "name": "Bartleby"}', stream_resolver=create_user, ), FeatureAssertion( input=User(id=1), output=User(name="Bartleby", transaction_count={"1d": 1}), ), branch="my-branch", ) ``` For the common case where you just want to prime the online store before the first stream message, pass seed_online_store= instead of writing an explicit UploadFeatures step. It is equivalent to prepending one UploadFeatures step at the head of the scenario: ``` ChalkClient().check_stream_scenario( StreamMessage( message=b'{"user_id": 1, "amount": 10.0}', stream_resolver=transaction_stream, timestamp=datetime(2026, 1, 1, 12, 20), ), FeatureAssertion( input=User(id=1), output=User(transaction_count={"1d": 1}), ), seed_online_store={User.id: [1, 2, 3], User.name: ["Alice", "Bob", "Carol"]}, branch="my-branch", ) ``` Other useful parameters: - show_table=True — print the Chalk Feature Value Check Table for every FeatureAssertion step, not only on failure. On a mismatch the table is always printed, regardless of this flag. - float_rel_tolerance / float_abs_tolerance — tolerances applied to float comparisons inside FeatureAssertion checks (defaults 1e-6 and 1e-12). Values pass if either tolerance is satisfied. - environment — pin the scenario to a specific Chalk environment; otherwise the client's default is used. ### Testing against a deployed branch with test_streaming_resolver To test against a deployed branch, use ChalkClient.test_streaming_resolver: ``` from chalk.client import ChalkClient sample_messages = [ '{"id": 123, "name": "John Smith"}', '{"id": 456, "name": "Alice Liddell"}', ] client = ChalkClient() # 1. Test messages against production resolver: result = client.test_streaming_resolver(resolver="get_users_streaming", message_bodies=sample_messages) # 'test_streaming_resolver' returns a signed URL to a parquet file in cloud storage -- `result.features` downloads this and extracts the table. print(result.features) # 2. Test messages against a stream resolver on a branch: result = client.test_streaming_resolver(resolver="get_users_streaming", message_bodies=sample_messages, branch="new_users_stream_logic") print(result.features) # 3. Create a new resolver and test it: new_users_resolver = make_stream_resolver( source=..., name="get_users_streaming_new", message_type=..., output_features={...}, ) result = client.test_streaming_resolver(resolver=new_users_resolver, message_bodies=sample_messages) print(result.features) ``` # Resolvers Overview source: https://docs.chalk.ai/docs/resolver-overview ## Create resolvers to compute feature values. While feature classes define what your data looks like (e.g. the types of your features, their relationships, and their properties), resolvers define how your feature values are computed. If you haven't read the section on feature classes, you are strongly encouraged to do so. Understanding feature classes will help you understand resolvers. Resolvers have four main components: inputs, outputs, code, and run conditions. The first three of these are rather straightforward. The inputs to a resolver are the features it uses to compute its output. The outputs of a resolver are the features that it computes. The code of the resolver is the logic that transforms your input features into your output features. The last component, run conditions, is a little more nuanced. At a high level, Chalk is optimized for two different access patterns: online and offline. In online access, information about a single entity is being requested and features must be computed quickly and cached. In offline access, a large chunk of historical feature data is being requested (you can think of this as creating a dataset for model training or running an analytic query). Resolvers are always specified as either online or offline. We go into more detail about what this means in the next section where we talk about run conditions, but for now it is important to note that: 1). every resolver is either online or offline, and 2). different resolvers run depending on how you are requesting data from Chalk. ### Writing Resolvers Chalk allows you to write resolvers in three different ways: - as SQL queries, with comments specifying data sources and output features. - as Python functions, with type annotations specifying the input and output features. - as an expression, inline in your feature definitions. Below is an example of how to write the three kinds of resolvers in Chalk for a User feature class. In this example, we assume that a Postgres data source called pg_users has been configured and connected to Chalk. First, let's define our User feature class: ``` from chalk.features import features @features class User: id: str name: str age: int hobbies: list[str] is_runner: bool # computed feature athleticism_score: float # computed feature ``` Now we define our resolvers: We resolve the id, name, age, and hobbies features for our users from the users table of our pg_users data source. When reading data from your data sources, you should use a SQL resolver: ``` -- The features given to us by the user. -- resolves: user -- type: online -- source: pg_users select id, name, age, hobbies from users; ``` To compute the is_runner feature for users, we simply check whether running is in the list of hobbies, which we can do with an expression. Chalk expressions utilize low-latency C++ for efficient execution. Chalk expressions can use any of the chalk.functions to express a variety of computations. ``` import chalk.functions as F from chalk.features import features, _ @features class User: id: str name: str age: int hobbies: list[str] is_runner: bool = F.contains(_.hobbies, "running") athleticism_score: float # computed feature ``` We now use a Python resolver to compute the athleticism_score feature for our users. Python resolvers are useful for integrating existing libraries, and executing more complex logic, such as API calls. ``` from chalk import online from src.user import User from utils.athleticism import is_athletic @online def get_is_runner( hobbies: User.hobbies ) -> User.athleticism_score: num_athletic_hobbies = sum([is_athletic(hobby) for hobby in hobbies]) return num_athletic_hobbies / len(hobbies) ``` ### When Do Resolvers Run A question that often crops up at this point is, "I have defined a resolver, but how do I make it run?" Just like features, the idea behind resolvers is that they are declarative. Resolvers don't run when you define them, and they don't run exhaustively over your data. They run only in response to a request for the features that they are responsible for computing. In practice, this means that they run in response to "queries". Resolvers can also be run on a schedule or triggered. Either approach causes them to 1). execute, in batch-job fashion, across all data they find in your upstream data sources, and 2). write computed features to your offline store. A query is just a request for data. Chalk takes a query and determines which resolvers it needs to run, optimizes the execution order, and provides you with a response. For now, we'll hold off on discussing queries further, but feel free to skip ahead and get a sense of what Chalk queries are and how they work before diving deeper into resolvers. # Online / Offline source: https://docs.chalk.ai/docs/resolver-online-offline ## Eliminate train-serve skew with shared logic. Chalk supports two primary execution contexts: online (for inference) and offline (for training). Online contexts handle real-time prediction requests with millisecond latency requirements, while offline contexts process historical data for model training and analysis. Chalk enables you to write resolvers that can run in either or both contexts, helping eliminate train-serve skew while optimizing for the different performance characteristics of each environment. ### Online and Offline Resolvers Online resolvers are eligible to run in online and offline queries. However, in an offline context, Chalk will prefer to run an offline resolver to an online resolver if both are available for the same feature. Offline resolvers can run only in offline queries, and can never be used as part of an online query. There are three types of resolvers: expressions, Python resolvers, and SQL resolvers. Expressions are online resolvers, while Python and SQL resolvers can be specified as either offline or online. ### Expressions Chalk expressions are online resolvers. However, an expression can operate on data that is fetched from either an online or offline resolver. Expressions are written in terms of the data model, and thus are portable between online and offline contexts. For example, consider the following feature definition: ``` from chalk import _ from chalk.features import features @features class User: id: int first_name: str last_name: str full_name: str = _.first_name + " " + _.last_name ``` The full_name feature is defined as an expression that concatenates the first_name and last_name features. No matter how we compute first_name and last_name, the expression for full_name remains the same. If we have a PostgreSQL online resolver that computes first_name and last_name from a low-latency database, we can use that resolver in an online query. If we have a Snowflake offline resolver that computes first_name and last_name from a data warehouse, we can use that resolver in an offline query. Scalar expressions like this one are also time-independent. So long as the underlying features are temporally consistent, the expression will be as well. If your expression depends on time-varying data, such as a DataFrame aggregation, you can use _.chalk_now or _.chalk_window (in the case of a windowed aggregation) to ensure temporal consistency. ``` from datetime import timedelta, datetime from chalk import _ from chalk import windowed, Windowed from chalk.features import features, DataFrame @features class Transaction: id: int user_id: "User.id" amount: float at: datetime @features class User: id: int transactions: DataFrame[Transaction] total_amount_last_30d: float = _.transactions[ _.at >= _.chalk_now - timedelta(days=30), _.amount ].sum() average_transaction_amount: Windowed[float] = windowed( "1d", "30d", "365d", expression=_.transactions[_.amount, _.at > _.chalk_window].mean(), ) ``` Now, total_amount_last_30d and average_transaction_amount will compute the same values in both online and offline contexts, regardless of how the transactions for the User are fetched. At inference time, _.chalk_now will be the current time, and _.chalk_window will be the time of the query less the window size. At training time, _.chalk_now will be the point-in-time of the query, and _.chalk_window will be the point-in-time of the query less the window size. ### Python To define a Python resolver as online or offline, use the @online or @offline decorator. ``` from chalk.features import online, offline, DataFrame from src.features import User, Transaction @online def get_email_username(email: User.email) -> User.email_username: username = email.split("@")[0] if "gmail.com" in email: username = username.split("+")[0].replace(".", "") return username.lower() @offline def get_transactions() -> DataFrame[ Transaction.id, Transaction.amount, Transaction.date, ]: return DataFrame.read_parquet(...) ``` Online Python resolvers can run in both online and offline queries, while offline Python resolvers can run only in offline queries. Typically, almost all Python resolvers will be online. The most common use-case for offline Python resolvers is to load parquet files or other data files that represent historical data. If your online Python resolver is time-dependent, you can pull in the point-in-time of the query to ensure temporal consistency. ``` from chalk import Now, online @online def get_age(birthdate: User.birthdate, now: Now) -> User.age: return (now.date() - birthdate).days // 365 ``` At inference time, chalk.Now will be the current time, and at training time, chalk.Now will be the point-in-time of the query. ### SQL To define a SQL resolver as online or offline, use the type field in the resolver comment. ``` -- resolves: User -- source: postgres + -- type: online select id, name, age from users ``` With SQL resolvers, you're likely to want to define an online and offline variant. You would never want to run a query against, for example, Snowflake in an online context. Snowflake, and other data warehouses like BigQuery, Databricks, and Redshift, are not designed for low-latency queries. Instead, you would want to run a query against a low-latency database like Postgres or MySQL in an online context. In contrast, with an offline query, you might want to run a query against Snowflake to get a large amount of data, and you might not have a Postgres instance that contains all the data you need or can handle unloading terabytes of data. ``` -- resolves: User + -- source: snowflake + -- type: offline select id, name, first_name, last_name, age from users ``` ``` -- resolves: User + -- source: postgres + -- type: online select id, name, age from users ``` Note that these two resolvers can have different schemas. The offline Snowflake resolver above returns first_name and last_name in addition to name, while the online Postgres resolver returns only name. ### Eliminating train-serve skew Introducing different online and offline resolvers leads to an opportunity to introduce different means of computing a feature. Online and offline SQL resolvers are a necessary source of skew. Data warehouses cannot stand up to production traffic request volumes, and transactional databases cannot store the nearly unlimited data volumes where warehouses excel. However, we should aim to minimize this skew as much as possible. For example, consider these aggregation features for user transaction statistics: ``` @features class User: id: int count: int avg_amount: float amount_stddev: float ``` Implementing the same feature logic twice in different SQL dialects invites subtle bugs and inconsistencies. These SQL resolvers look nearly identical, but they produce subtly different results due to differences in how Postgres and Snowflake handle standard deviation calculations and rounding. The STDDEV_POP() in Postgres computes population standard deviation, while Snowflake's STDDEV() defaults to sample standard deviation. Additionally, the rounding methods (::DECIMAL vs ROUND()) can produce different results for edge cases. These small numerical differences can compound and degrade model performance in production. Implementation for Postgres online context ``` -- resolves: User -- source: postgres -- type: online select user_id as id, count(*) as count, avg(amount)::decimal(10,2) as avg_amount, stddev_pop(amount)::decimal(10,2) as amount_stddev from transactions where created_at >= current_date - interval '30 days' group by user_id ``` Now you need to implement the same logic again for the offline context, but adapted for Snowflake's SQL dialect. Notice how even though the intent is identical, the implementation details differ — different function names, different date arithmetic syntax, and different rounding behavior. Separate implementation for Snowflake offline context ``` -- resolves: User -- source: snowflake -- type: offline select user_id as id, count(*) as count, round(avg(amount), 2) as avg_amount, round(stddev(amount), 2) as amount_stddev from transactions where created_at >= dateadd('day', -30, current_date()) group by user_id ``` Write the aggregation logic once using Chalk expressions, and it runs identically in both online and offline contexts. Instead of computing these aggregations differently in each database, you can fetch the raw transaction data and compute the statistics using Chalk expressions. This ensures the exact same computation logic runs in both online and offline contexts, completely eliminating this source of train-serve skew. The expression syntax is simple and declarative — you just specify what you want to compute, and Chalk handles the execution optimally for each environment. Using expressions to compute derived features consistently ``` from chalk import _ from chalk.features import features, DataFrame @features class User: id: int transactions: DataFrame[Transaction] count: int = _.transactions.count() avg_amount: float = _.transactions[_.amount].mean() amount_stddev: float = _.transactions[_.amount].std() ``` ### Storage Chalk includes an online and an offline store. The online store is used to cache data to make realtime data requests extremely fast, while the offline store is a data warehouse that stores all the features that you've computed — enabling monitoring and dataset generation for training. Your online store holds data that you want to cache according to the cache policies you set on your features or on your queries. At inference time, you need exceptionally fast reads to serve data at low latency. However, it's not important to keep every value of every historical feature value around for low-latency access. This access pattern aligns with the performance characteristics of Redis or DynamoDB, which Chalk uses to store your online data. The offline store holds far more data than the online store. It keeps a record of all online runs and indexes all data brought in from your offline data sources. Chalk integrates with a number of different data warehouse systems for our large-scale offline storage depending on customer needs and deployment type, including BigQuery, Snowflake, and Redshift. In addition, offline queries write their output to a parquet file in cloud storage (S3/GCS), whereas online queries write their results to the database. | writes to the offline store | writes to the online store | | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | an online query writes all freshly computed features (those not read from the online store) | an online query writes all freshly computed features with `max_staleness != 0` | | A triggered resolver run with `store_offline=True (default behavior)` | A triggered resolver run with `store_online=True (default behavior)` | | scheduled queries with `recompute_features=True` and `store_offline=True (default behavior)` | scheduled query with `store_online=True (default behavior)` | | Ingesting a dataset to the offline store: `dataset.ingest(store_offline=True)` | Ingesting a dataset to the online store: `dataset.ingest(store_online=True)` | | offline queries with `ChalkClient.offline_query(store_offline=True)` | offline queries with `ChalkClient.offline_query(store_online=True)` | | An `@online` or `@offline` scheduled resolver | An `@online` or `@offline` scheduled resolver that computes features with `max_staleness!=0` and `etl_offline_to_online=True` | | A streaming resolver: `@stream` | A streaming resolver: `@stream` | ### Querying Every request you make to Chalk for data is done through a query, and every query you make is either an online or an offline query. Online queries are used to receive information about a single entity. For example, you might be looking to compute the features of a credit model for a single user, or decide what products to suggest to a customer. Thus, online queries are designed to be as quick as possible— within milliseconds. You can use our API client to run queries. Offline queries are used to sample historical data about many entities at specific points in time for model training or investigation. When you execute an offline query, Chalk will kick off a job that acquires the requested data for every primary key/timestamp combination presented. This could take a few seconds! Since offline queries often lookup data for thousands of rows, they are not designed to be used to make millisecond-level decisions. See our guide on offline queries for a more in-depth treatment. | | online query | offline query | | ---------------- | ---------------------------------- | --------------------------------------------------------------------------------------- | | online resolver | `@online` resolver will run | `@online` resolver will run if there is no `@offline` resolver with the same definition | | offline resolver | `@offline` resolver will never run | `@offline` resolver will run | When an online query requests a feature, the planner looks for either an @online resolver, or if the feature has max_staleness configured, a value in the online store cache that is fresh enough to return. If neither an online resolver nor caching is configured, the planner will raise an error as there is no way to resolve the feature. When an online query requests a feature that has no @online resolver — because only an @offline resolver exists, or because no resolver exists at all — the planner treats both cases identically. @offline resolvers are excluded from the valid candidate set for online queries, so from the planner's perspective, "only an offline resolver" and "no resolver" look the same. When an @offline resolver was found but excluded, the error message will name the resolver and suggest running an offline query instead. To allow @offline resolvers to run in an online query — useful for local testing — pass include_offline_resolvers via planner_options: ``` client.query( input={User.id: 1}, output=[User.score], planner_options={"include_offline_resolvers": True}, ) ``` This is intended for development and testing only. Offline resolvers are not designed for low-latency online serving. ### Online/Offline Interaction ### Online-to-Offline After an online resolver runs, its values are copied into the offline store. When you query the offline store, you will receive data from both records of online runs and offline-specific resolvers. Which data you receive depends on which data was closest to the point-in-time that you queried. For more information, see temporal consistency. ### Offline-to-online In contrast, data from the offline store does not reach the online store by default. However, you can choose to ETL the data from an offline resolver into the online store. This can be helpful, for example, when you tolerate stale data in online inference and have a data source in the offline store that doesn't have a direct replacement in the online store. More details are provided in the section Reverse ETL. ### Summary | Online query | Offline query | | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Runs only `@online` resolvers | Runs both `@online` and `@offline` resolvers | | Returns one row of data about one entity | Returns a DataFrame of many rows of historical data corresponding to multiple entities at historical time points | | Designed to return data in milliseconds | Blocks until computation is complete, not designed for millisecond-level computation | | Queries the online store and calls `@online` resolvers for quick retrieval | Queries the offline store which stores all data from online queries, unless `recompute_features=True`, in which case `@offline` and `@online` resolvers are used to resolve the outputs | | Writes output data to online store database and offline store database | Writes output to a parquet file containing results to cloud storage. Only writes to online store or offline store if specified. | # SQL Resolvers source: https://docs.chalk.ai/docs/sql-resolvers ## Define feature resolvers as a SQL query. SQL resolvers enable you to load and transform data from your data sources as feature values. ### Defining SQL Resolvers There are two ways to define SQL resolvers. The first is to write a SQL query in a file with the .chalk.sql extension, and the second is to use the make_sql_file_resolver function, which will generate a SQL file resolver based on the function arguments at build time. ### SQL File Resolvers To define a SQL resolver in a file, create a file with the .chalk.sql extension, add comments to specify the data source the SQL query will run on and the feature class whose features it will resolve, and write the SQL query. ``` -- source: pg_users -- resolves: User select id, name, age, hobbies from users ``` The output column names in the SQL query must match the feature names in the feature class specified in the resolves comment. The data source specified in the source comment must also match the name of the data source configured in the Chalk dashboard. Each SQL file resolver must only resolve features within the same feature class, but a resolver does not have to resolve all features within the feature class. ### Generating SQL File Resolvers To generate SQL file resolvers at build time, use the make_sql_file_resolver function. This function enables you to define one or many SQL file resolvers at a time that may have slight variations, for example querying a different data source in a different environment. The following code would generate an equivalent SQL file resolver to the one above: ``` from chalk import make_sql_file_resolver make_sql_file_resolver( name="get_user", source="pg_users", resolves="User", sql="select id, name, age, hobbies from users" ) ``` ### SQL Resolver Configurations As with Python resolvers, SQL resolvers can be configured through many optional parameters. For generating SQL file resolvers, these parameters are passed as arguments to the make_sql_file_resolver function. For SQL file resolvers, these parameters are specified in the comments at the top of the file. ``` -- These parameters are required in all SQL file resolvers -- source: pg -- resolves: User -- These parameters are optional in all SQL file resolvers -- type: online <-- default is "online," but can also be "offline" or "streaming" -- incremental: <-- read more about incremental resolvers here: https://docs.chalk.ai/docs/sql#incremental-queries -- mode: row -- lookback_period: 60m -- incremental_column: updated_at -- count: 1 <-- the number of rows the resolver should return. This can be set to 1, 'one', 'one_or_none', or 'all' -- timeout: 5m <-- the maximum time the resolver should run before timing out -- cron: 0 0 * * * <-- a cron expression to schedule the resolver to run at a specific time -- owner: 'helly@lumonindustries.com' <-- the email of the owner of the resolver. Alerts can be routed to owners. -- tags: ['user', 'profile'] <-- tags to help organize resolvers in the Chalk dashboard -- environment: 'production' <-- the environment in which the resolver should run -- total: True <-- whether this resolver returns all ID's of a given namespace. select id, name, age, hobbies, updated_at from users ``` For the full list of configurations, see here. ### Testing To test your SQL resolvers, we recommend setting up integration tests or iterating on a branch. # Expressions source: https://docs.chalk.ai/docs/expression ## Using Chalk expressions to define features ### Overview Chalk Expressions let you define features declaratively, using symbolic computation over your data. While you write expressions in idiomatic Python, they are compiled and executed as vectorized C++, enabling low-latency computation at serve time and high-throughput processing at train time. Expressions support a wide range of operations, including arithmetic, filtering, aggregations, and built-in functions like . For example, in a Transaction feature class, we can compute the subtotal of a transaction as the difference between its total and sales tax: ``` from chalk.features import features, Primary from chalk import _ @features class Transaction: id: int total: float sales_tax: float subtotal: float = _.total - _.sales_tax ``` The _ symbol refers to the current scope (here, the feature class Transaction) and is used to reference other fields on the same instance. Expressions like _.total - _.sales_tax are compiled into native execution plans that run efficiently in production. In addition to referencing fields on the same object, you can traverse relationships. If each Transaction is associated with a User, for example, you can compute a string similarity between the user’s name and the transaction memo: ``` from chalk.features import _, features + import chalk.functions as F @features class Transaction: ... amount: float memo: str user: "User" + name_match_score: float = F.jaccard_similarity( + _.user.name, _.memo + ) ``` Here, _.user.name follows the foreign key relationship from Transaction to User. The function F.jaccard_similarity is one of many built-in Chalk functions that operate on symbolic expressions. Expressions can also perform aggregations over related records. In a User feature class, we can compute aggregates like the number of large transactions or the total amount spent: ``` from chalk import _ from chalk.features import DataFrame, features @features class User: id: int name: str transactions: DataFrame[Transaction] # Total spend is nullable because the sum of an empty DataFrame is null total_spend: float | None = _.transactions[_.total].sum() # The count is never null, because an empty DataFrame has count 0 num_large_txns: int = _.transactions[_.total > 1000].count() ``` In this context, _ refers to the User instance when referring to _.transactions. But when you apply a filter, like _.transactions[_.total > 1000], the expression inside the brackets is evaluated in the context of each individual Transaction. That means _.total refers to the total field on each Transaction, not on the User. This scoped evaluation makes it easy to filter, project, and aggregate over related data. All expressions are statically analyzed, optimized to eliminate redundant computation, and executed as high-performance C++ at runtime. ### Scalar Functions Chalk expressions support a wide range of built-in functions for manipulating data, performing calculations, and transforming features. These functions can be used in expressions to operate on feature values, DataFrames, and other data types. ``` - from chalk import features, online + from chalk.features import _, features @features class Transaction: id: int total: float sales_tax: float + subtotal: float = _.total - _.sales_tax - @online - def get_subtotal(total: Transaction.total, sales_tax: Transaction.sales_tax) -> Transaction.subtotal: - return total - sales_tax ``` ### Infix Operators Chalk expressions support a variety of infix operators for arithmetic, conditions, and boolean logic. | Operator | Description | Example | | -------- | --------------------- | ----------------------------- | | `+` | Addition | `_.total + _.sales_tax` | | `-` | Subtraction | `_.total - _.sales_tax` | | `*` | Multiplication | `_.quantity * _.price` | | `/` | Division | `_.total / _.quantity` | | `>` | Greater than | `_.total > 1000` | | `>=` | Greater than or equal | `_.total >= 1000` | | `<` | Less than | `_.total < 1000` | | `<=` | Less than or equal | `_.total <= 1000` | | `==` | Equal | `_.status == "completed"` | | `!=` | Not equal | `_.status != "completed"` | | `&` | Boolean and | `_.is_active & _.is_verified` | | \| | Boolean or | _.is_active \| _.is_verified | | `~` | Boolean not | `~_.is_active` | Python does not allow these operators to be overridden, so they will not work with Chalk's expressions. Instead, use the infix operators &, |, and ~ for boolean logic, and use == and != for equality comparisons. ### Builtin Functions The chalk.functions module exposes several helpful functions that can be used in combination with expression references to transform features. These functions are meant to be used in expressions and are not available as standalone functions. To view all available functions, see our SDK docs. ### Structs Expressions can be used to access nested attributes from other features in a feature class, whether these other features are struct dataclasses, features, or DataFrames. ``` import chalk.functions as F from chalk.features import features from dataclasses import dataclass @dataclass class LatLon: lat: float | None lon: int | None @features class User: id: int home: LatLon work: LatLon commute_distance: float = F.haversine( lat1=_.home.lat, lon1=_.home.lon, lat2=_.work.lat, lon2=_.work.lon, ) ``` ### Custom Functions You can create custom functions to encapsulate complex logic or reusable computations in your expressions. For example, if you wanted to apply consistent windows across many features, you could define a custom function like this: Use helper functions to create expressions ``` from chalk import _ from chalk.features import features, DataFrame def count_where(*filters): return _.transactions[_.created_at > _.chalk_window, *filters].count() @features class User: id: int transactions: DataFrame[Transaction] num_large_transactions: int = count_where(_.total > 1000) num_small_transactions: int = count_where(_.total < 100) ``` Expressions are not supported in Python resolvers, so you cannot use Chalk functions like F.jaccard_similarity in a Python resolver. Don't use expressions in Python resolvers ``` @features class User: id: int name: str email: str name_email_match_score: float @online def get_score( name: User.name, email: User.email, ) -> User.name_email_match_score: # Don't do this!! Expressions don't run in Python resolvers return F.jaccard_similarity(name, email) ``` Instead, use expressions to define the feature directly in the feature class: Use expressions in feature classes ``` @features class User: id: int name: str email: str name_email_match_score: float = F.jaccard_similarity( _.name, _.email ) ``` ### DataFrame Functions ### Conditions and filters DataFrame features can be filtered with expressions. Extending our Transaction example, we can create a User feature class with a has-many relationship to Transaction. Then, we can define a feature representing the number of large purchases by referencing the existing User.transactions feature: ``` from chalk.features import _, features, DataFrame @features class Transaction: id: int + user_id: "User.id" total: float sales_tax: float subtotal: float = _.total - _.sales_tax + @features + class User: + id: int + # implicit has-many relationship with Transaction due to `user_id` above + transactions: DataFrame[Transaction] + num_large_transactions: int = _.transactions[_.total > 1000].count() ``` The object referenced by _ changes depending on its current scope. In this code, the _ in _.transactions references the User object. Within the DataFrame filter, the _ in _.total references each Transaction object as each one is evaluated. The count aggregation is covered in the next section. ### Projections and aggregations DataFrame features support projection with expressions, which produce a new DataFrame scoped down to the referenced columns. DataFrames can be aggregated after eligible columns are selected. With our Transaction example, we already saw a count aggregation for counting the number of large transactions. We can add another aggregation for computing the user's total spend: ``` from chalk.features import _, features, DataFrame @features class Transaction: id: int user_id: "User.id" sales_tax: float subtotal: float total: float = _.subtotal + _.sales_tax @features class User: id: int transactions: DataFrame[Transaction] num_large_transactions: int = _.transactions[_.total > 1000].count() + total_spend: float = _.transactions[_.total].sum() ``` To compute User.total_spend, we needed to create a projection of the User.transactions DataFrame limited to only the Transaction.total column so that the sum aggregation could work. In contrast, no projection was needed for the num_large_transactions aggregation because count works on DataFrames with any number of columns. For computing low-latency aggregations over high volumes of data, Chalk also offers materialized windowed aggregations that uses materialization of buckets of data to compute large aggregations efficiently. ### Aggregation functions Aggregation functions have varying behavior when handling None values and empty DataFrames. If an aggregation function says None values are skipped in the table below, it will consider a DataFrame with only None values as empty. | Function | `None` values | Empty DataFrame | Notes | | ----------------------- | ------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sum` | Skipped | Returns `0` | | | `min` | Skipped | Returns `None` | | | `max` | Skipped | Returns `None` | | | `mean` | Skipped | Returns `None` | Feature type must be `float` or `float \| None`. `None` values are skipped, meaning they are not included in the mean calculation. | | `count` | Included | Returns `0` | | | `any` | Skipped | Returns `False` | | | `all` | Skipped | Returns `True` | | | `std` | Skipped | See notes | Standard deviation. Requires at least 2 values. For DataFrames with less than 2 values, returns `None`. ; Aliases: `stddev`, `stddev_sample`, `std_sample`. | | `var` | Skipped | See notes | Variance. Same requirements as `std`. ; Alias: `var_sample`. | | `approx_count_distinct` | Skipped | Returns `0` | | | `approx_percentile` | Skipped | Returns `None` | Takes one argument, `quantile`, expected to be a float in the range `[0, 1]`. ; Example: `approx_percentile(0.75)` returns a value approximately equal to the 75th percentile of the not-`None` values in the DataFrame. | | `approx_top_k` | Skipped | Returns `None` | Takes one argument, `k`, expected to be a positive integer, as a keyword argument, and an optional `counters` argument sizing the underlying sketch (defaults to `2 * k`). ; Example: `approx_top_k(k=25)` returns the 25 or fewer approximately-most common values, and `approx_top_k(k=25, counters=1000)` does the same while tracking 1000 distinct values for better accuracy. | For aggregations that can return None, either mark the feature as optional (for example, by setting the feature type to float | None) or use coalesce to fall back to a default value. ### Run conditions and Execution To specify tags and versions for a feature that is resolved through an expression, you can use the feature function and pass in the expression as an argument. ``` from chalk.features import Primary, features, feature @features class User: id: int purchases: DataFrame[Purchase] # Uses a default value of 0 when one cannot be computed. num_purchases: int = feature( expression=_.purchases.count(), default=0, tags=["fraud", "credit"], version=1, ) ``` You may also want to define expressions that are only meant to be used for the offline pathway, say if you would like to compute values or aggregations dependent on offline resolvers. To do so, you can specify a distinct offline_expression as an offline resolver for your feature. ``` from chalk.features import features, feature @features class Merchant: id: int transactions: DataFrame[Transaction] merchant_risk_score: float = feature( expression=_.transactions[_.status=="approved"].count() / _.transactions.count(), offline_expression=(0.50*_.transactions[_.is_chargeback==True].count()+0.25*_.transactions[_.is_refund==True].count()+0.25*_.transactions[_.status=="approved"].count())/_.transactions.count() ) ``` ### Testing To test your expressions, we recommend setting up integration tests or iterating on a branch. ### Dynamic Expressions In some cases, you may want to build expressions dynamically. For example, if you have a rules engine that generates expressions and stores them in a database, you can load those expressions at runtime and ask Chalk to compute their values. Consider this User and Transaction model, which would be checked in to the code: ``` from chalk.features import features, DataFrame @features class Transaction: id: int user_id: "User.id" user: "User" total: float sales_tax: float @features class User: id: int transactions: DataFrame[Transaction] name: str email: str ``` Using the Golang or Python SDKs, you can compute both scalar and aggregate features on demand. For example, in Golang, if we wanted to compare the lowercased name and email by their Jaccard similarity, we could build the expression dynamically and ask Chalk to compute it: ``` package main import ( "testing" "github.com/zeebo/assert" "github.com/chalk-ai/chalk-go/expr" "github.com/chalk-ai/chalk-go" ) func TestQueryingExpressions(t *testing.T) { // Picks up the ambient credentials from the `chalk login` run on the CLI. client, err := chalk.NewGRPCClient(t.Context()) assert.NoError(t, err) result, err := client.OnlineQueryBulk( t.Context(), chalk.OnlineQueryParams{}. WithInput("user.id", []int{1}). WithOutputs("user.name"). WithOutputs("user.email"). WithOutputExprs( expr.FunctionCall( "jaccard_similarity", expr.FunctionCall("lower", expr.Col("name")), expr.FunctionCall("lower", expr.Col("email")), ). As("user.name_email_sim"), ), ) assert.NoError(t, err) row, err := result.GetRow(0) assert.NoError(t, err) for feature, value := range row.Features { t.Logf("Feature: %s, Value: %+v", feature, value.Value) } } ``` This program produces output like this: ``` === RUN TestChalkClient chalk_test.go:46: Feature: user.name, Value: Nicole Mann chalk_test.go:46: Feature: user.email, Value: nicoleam@nasa.gov chalk_test.go:46: Feature: user.name_email_sim, Value: 0.5714285714285714 --- PASS: TestChalkClient (0.43s) ``` The feature user.name_email_sim was computed using the expression jaccard_similarity(lower(name), lower(email)), which was built dynamically using the expr package. For a complete list of functions, see our SDK docs. All the functions available in expressions are also available in the expr package. You can also compute aggregations dynamically. Using the Golang SDK, we can compute the number of large transactions for a user: ``` result, err := client.OnlineQueryBulk( t.Context(), chalk.OnlineQueryParams{}. WithInput("user.id", []int{1}). WithOutputExprs( expr.DataFrame("transactions"). Filter(expr.Col("amount").Gt(expr.Float(0.))). Agg("count"). As("user.positive_transaction_count"), ), ) ``` For this program, the output would look like this: ``` === RUN TestChalkClient chalk_test.go:42: Feature: user.positive_transaction_count, Value: 33 --- PASS: TestChalkClient (0.45s) ``` In a Python expression, the above SDK call is equivalent to: ``` User.positive_transaction_count: int = _.transactions[_.amount > 0].count() ``` Both the scalar functions and aggregations can be computed in Python SDK. See this guide for more details. # Python Resolvers source: https://docs.chalk.ai/docs/python-resolvers ## Specify dependencies and results of feature resolvers. Python resolvers are Python functions marked with either an @online or @offline decorator, with Python type annotations declaring the dependencies on other features, as well as the output feature(s). ### Inputs ### Scalar dependencies To depend on features from a feature class, label your resolver arguments with your features as the types. You can then use those arguments in the body of your resolver to compute your output features. If you're running our editor plugin, your editor will see the type of each variable as the type of the underlying scalar. ``` from chalk.features import features, online @features class User: id: str email: str email_domain: str @online def get_domain(email: User.email) -> User.email_domain: # type(email) == str return email.split('@')[1].lower() ``` You can require multiple features in a resolver. However, all feature dependencies in a single resolver need to originate in the same root namespace: Requiring features from the same root namespace ``` @online def get_name_sim(a: User.email, b: User.name) -> User.email_name_match: return len(set(a) & set(b)) / len(set(a) | set(b)) ``` Here, we incorrectly request features from the root namespaces of Transfer and User: Requiring features from different root namespaces ``` @online def txn_email_sim( email: User.email, memo: Transfer.memo, ) -> Transfer.email_in_memo: return email in memo ``` If you want to require features from different namespaces, you can use has-one or has-many relationships. Requiring features from different root namespaces using a relationship ``` @online def txn_email_sim( email: Transfer.user.email, memo: Transfer.memo, ) -> Transfer.email_in_memo: return email in memo ``` ### Has-one dependencies ### Scalar has-one You can also require features joined to a feature class through has-one relationships. For example, if users in your system have bank accounts, and you wanted to compare the name on the user's bank account to the user's name, you could require the user's name and the account's title through the user: ``` @online def name_sim(title: User.account.title, name: User.full_name) -> User.name_sim: return len(set(title) & set(name)) / len(set(title) | set(name)) ``` You can also require all scalars on the user: ``` from chalk import online, Now @online def get_txn_name_match( memo: Transaction.memo, user: Transaction.user, ) -> User.profile.age: return ( len(set(memo) & set(user.full_name)) / len(set(memo) | set(user.full_name)) ) ``` Chalk will materialize all scalar features on the user before calling this function. If you want to pull only a few features of the user, require each directly: ``` @online def get_txn_name_match( memo: Transaction.memo, full_name: Transaction.user.full_name, ) -> User.profile.age: return ( len(set(memo) & set(full_name)) / len(set(memo) | set(full_name)) ) ``` ### Optional has-one Has-one relationships can also be declared as optional. You may also require feature through optional relationships, but the types for all of those optional features will become optional. Consider the below example: ``` from chalk.features import features, online @features class Account: id: int user_id: "User.id" balance: float # Non-optional balance @features class User: id: int account: Account | None has_high_balance: bool @online def has_high_bal(balance: User.account.balance) -> User.has_high_balance: # Balance will be "float | None", because User.account is optional if balance is None: return False return balance > 1000 ``` The resolver in this example receives an optional float, even though balance is not an optional field on Account. The optional is added because the user may not have an account, in which case the resolver will receive None for the balance. ### Nested has-one You can also traverse nested has-one relationships in the same manner as requiring a single has-one. Consider a schema where users have a feature class of profile information, and the user's profile has an identity feature class, which in turn has the age of the user's email. You can require the email age feature as below: ``` @online def email_score( email_age: User.profile.identity.email_age ) -> User.email_score: return ( 1 if email_age < 30 else 0.5 if email_age < 60 else 0 ) ``` However, you cannot access nested relationships without explicitly asking for them. Accessing a transitive relationship from a dependency. ``` @online def fn(acct: User.account) -> ...: acct.balance # Ok acct.institution.name # Error! ``` Instead, you can require the nested relationship directly and access any of its scalar features. Directly requiring the transitive relationship. ``` @online def fn(ins: User.account.institution, acct: User.account) -> ...: acct.balance # Ok ins.name # Ok ``` The semantics of optional has-one dependencies carry over to nested has-one dependencies. If you traverse an optional relationship, then all downstream attributes will become optional. ### Has-many dependencies ### Scalar has-many You can also require has-many relationships as inputs to your resolver: ``` @online def txn_count(transfers: User.transfers) -> User.count_transfers: return len(transfers) ``` You receive a Chalk DataFrame, which supports projections, filtering, and aggregations, among other operations. ### Projections By default, Chalk will materialize all scalar features on the Transfer feature class before calling your resolver. As an optimization hint, you can specify which features from the transfers that you'd like Chalk to materialize before calling the function. For example, if there were expensive features to compute on the transfer, you could scope the features to only the set you need: ``` @online def fn(transfers: User.transfers[Transfer.amount, Transfer.memo]) -> ...: transfers[Transfer.amount].sum() # Ok transfers[Transfer.from_institution] # Error: filtered out above ``` The error above is surfaced statically by our editor plugin. ### Filtering You can apply filters to the has-many inputs of resolvers: ``` @online def fn(transfers: User.transfers[Transfer.amount > 100]) -> ...: ``` Filters can be composed with projections following the semantics of the Chalk DataFrame. ``` @online def fn(transfers: User.transfers[Transfer.amount > 100, Transfer.memo]) -> ...: ``` Filters can also be used on features through has-one relationships. ``` @online def fn(transfers: User.transfers[Transfer.amount, Transfer.bank.location == "USA"]) -> ...: ``` More on projection and filtering here ### Has-many through has-one Has-many relationships can be required through has-one relationships: ``` @online def fn(transfers: User.account.transfers) -> ...: ``` As with scalar has-many dependencies, you can scope down the scalar features on the transfer to only those required: ``` @online def fn(transfers: User.account.transfers[Transfer.amount]) -> ...: transfers[Transfer.amount].sum() # Ok transfers[Transfer.from_institution] # Error: filtered out above ``` ### Has-many through optional-has-one If the has-one relationship that you're traversing is optional, then the transfers argument in the example above will either be None or a Chalk DataFrame. ### Has-one through has-many You can also select columns through nested has-one relationships that would not normally materialize. ``` @online def fn(transfers: User.transfers[Transfer.amount, Transfer.bank.name]) -> ...: ``` In the above example, if there was a has-one relationship between Transfer and Bank, we can fetch any scalar feature from Bank as well for the DataFrame. Note that the Bank.name column would not materialize with the simple transfers: User.transfers typing, since this typing only materializes scalar features in the root namespace. ### Has-many through has-many Has-many relationships can be required through other has-many relationships. For example, consider the following feature definitions for User, Account, and Transaction, where a user can have many accounts, each with many transactions. ``` from chalk.features import features, has_many, DataFrame @features class Transaction: id: str account_id: "Account.id" amount: float @features class Account: id: str user_id: "User.id" transactions: DataFrame[Transaction] @features class User: id: str total_spent: float accounts: DataFrame[Account] ``` We can resolve the total_spent feature on User by computing the sum of transaction amounts across all of a user's accounts, as shown below. ``` @online def get_total_spent( txns: User.accounts.transactions[Transaction.amount] ) -> User.total_spent: return txns.sum() ``` ### Time dependencies To set a time-based dependency in your resolver, you can use the Now keyword. Resolvers with a dependency on Now can use the input times passed in at query-time to compute either scalar or DataFrame outputs. ``` from chalk import online, DataFrame, Now from chalk.features import features from datetime import datetime @features class Account: id: str balance: float updated: datetime @online def get_daily_balance(now: Now) -> DataFrame[Account]: return DataFrame([ Account(id="1", balance=100, updated=now), Account(id="2", balance=200, updated=now), ]) ``` Read more about time-based dependencies here. ### Outputs Python resolvers can output either scalar features or a DataFrame of features. ### Scalar output ### Single output To return a single feature from a resolver, set the return type annotation to the feature you want to resolve: ``` from chalk.features import features @features class User: id: int name: str employer: str @online def resolve(u: User.id) -> User.name: return "Jennifer Doudna" ``` Equivalently, you can wrap the return value in the User class: ``` @online def resolve(u: User.id) -> User.name: - return "Jennifer Doudna" + return User(name="Jennifer Doudna") ``` ### Multiple outputs To return multiple features, return an instance of the feature class. In the type signature, specify the Features[...] class, parameterized by the features that you pass to the feature class. ``` @online def resolve(u: User.id) -> Features[User.name, User.employer]: return User( name="Jennifer Doudna", employer="University of California, Berkeley" ) ``` You only need to pass a subset of the features to the constructor for the feature class. The editor plugin will check that the type annotation you assign to the resolver matches the subset of features passed to the constructor of the feature class. ### All features To return all features of a class, use Features[...] around the feature class. ``` @online def get_user(u: User.id) -> Features[User]: return User( name="Jennifer Doudna", employer="University of California, Berkeley" ) ``` If your resolver takes input features, those features are not considered as part of the output features. Note that the id feature is not returned from the function. This definition is equivalent to: ``` @online - def get_user(u: User.id) -> Features[User]: + def get_user(u: User.id) -> Features[User.name, User.employer]: return User( name="Jennifer Doudna", employer="University of California, Berkeley" ) ``` However, you may want to return almost all features of a class. Writing out all the features can be tedious and error-prone. You can subtract features from a feature class using the - operator: ``` from chalk.features import Features, ... @online def get_all_users(id: User.id) -> Features[User] - User.name: return User(employer="University of California, Berkeley") ``` Here, both the id feature and the name feature are not returned, which leaves only the employer feature. ### DataFrame output You can also output many instances of a feature class from a resolver by specifying a DataFrame as the return type of the function: ``` @offline def get_events() -> DataFrame[Transfer.uuid, Transfer.amount, Transfer.ts]: return DataFrame.read_csv(...) ``` Say you wanted to return many instances of a feature class, including nested features, from a resolver, then you can use the DataFrame class for your return type and in your resolver definition. ``` @online def get_user_employer_information(id: User.id) -> DataFrame[ User.id, User.name, User.employer.name, User.employer.category ]: return DataFrame([ User( id="1", name="Jennifer Doudna", employer=Employer( name="University of California, Berkeley", category="Education" ) ) ]) ``` For more info on how to load batch data, see the Data Sources sections. DataFrame-returning resolvers don't need inputs. ### All features To return all features of a class in a DataFrame, use DataFrame[...] class around the feature class: ``` @online def get_all_users() -> DataFrame[User]: return DataFrame([ User( name="Jennifer Doudna", employer="University of California, Berkeley" ) ]) ``` ### DataFrame-to-DataFrame resolvers Imagine a scalar feature you'd like to backfill over many thousands of primary keys and historical times. DataFrame-returning resolvers can dramatically reduce the computation time due to its vectorized handling. ``` from chalk import DataFrame, Index, offline from chalk.features import _ @offline def get_new_feature_as_dataframe( df: DataFrame[Index, Transaction.id, Transaction.amount], ) -> DataFrame[Index, Transaction.id, Transaction.is_large]: return df.with_columns( {Transaction.is_large: _.amount > 100} )[Index, Transaction.id, Transaction.is_large] ``` The above resolver runs faster on a thousand rows than the equivalent scalar resolver ran a thousand times. Index is a unique identifier that Chalk assigns to each row in the input batch. When a resolver transforms one input DataFrame into an output DataFrame in the same feature namespace, include Index in both type annotations and return the same Index values. This lets Chalk match each output to the exact input row that produced it, even when multiple rows have the same primary key and query timestamp. For example, an offline query can contain the same entity more than once: | `Index` | `User.id` | Query timestamp | | ------- | --------- | --------------- | | `0` | `123` | `2025-01-01` | | `1` | `123` | `2025-01-01` | The primary key and query timestamp do not uniquely identify these requests, but Index does. Now provides the point-in-time value for a computation; it is not a substitute for row identity. Preserve Index through the entire transformation. An operation such as explode repeats the parent row's Index onto the expanded rows. Aggregate those rows back to one output per Index: ``` # Incorrect: combines every request for the same user. grouping_keys = [str(User.id)] # Correct: keeps each input request independent. grouping_keys = [str(User.id), str(Index)] result = expanded_events.agg(grouping_keys, ...) ``` Similarly, when joining intermediate tables derived from the same input batch, include Index in the join keys. Operations may reorder rows, but they must not regenerate or modify Index values. Index correlation currently supports exactly one DataFrame input and a DataFrame output in the same root namespace. It does not apply to scalar resolvers or no-input DataFrame resolvers that load rows from a source. ### Relationship-returning resolvers Chalk also supports relationship-returning resolvers that enable users to return a DataFrame belonging to a has-many relationship. ``` @offline def relationship_returning_resolver( df: User.transactions[Transaction.id, Transaction.amount, Transaction.description], user_type: User.type ) -> User.transactions[Transaction.id, Transaction.transaction_type]: return ... ``` Just make sure that the return DataFrames do not have duplicate rows. That means no two rows should have the same primary key, or primary key & timestamp combinations if the feature time is also returned. ### How often a resolver runs The shape of a resolver's inputs decides how many times Chalk calls it. - A resolver whose inputs are scalar runs once per row. A query over 300 rows calls it 300 times, and Chalk runs those calls concurrently rather than one after another. - A resolver whose input is a DataFrame runs once for the whole batch, receiving every row in a single call. - A SQL file resolver also runs once for the whole batch. Chalk pushes your inputs into the query's WHERE clause, so one statement serves every row. For a resolver that reads a database, scalar inputs mean one connection per row, all at the same time. That can exhaust a connection pool on a query that looks routine. A scalar resolver is the right shape for a single online query, where there is one row and so one call. The cost appears in bulk and offline queries, where the same resolver runs once for every row in the request. One query per row: ``` from chalk import online from chalk.sql import PostgreSQLSource pg = PostgreSQLSource(name="PG") @online def user_txn_count(uid: User.id) -> User.txn_count: # A query over 300 users runs this 300 times, all at once. return pg.query_string( "select count(*) as txn_count from txns where user_id = :uid", args=dict(uid=uid), ).first() ``` One query for the whole batch: ``` from chalk.features import DataFrame @online def user_txn_counts(users: DataFrame[User.id]) -> DataFrame[User.id, User.txn_count]: # One call, one query, however many users. ids = users.to_pyarrow().column(0).to_pylist() return pg.query_string( "select user_id as id, count(*) as txn_count from txns " "where user_id = any(:ids) group by user_id", args=dict(ids=ids), ).all() ``` For reading raw data, a SQL file resolver does the same thing with no Python: ``` -- source: PG -- resolves: User select user_id as id, count(*) as txn_count from txns group by user_id ``` ### Testing To test your Python resolvers, you can set up unit tests, construct integration tests or iterate on a branch. ### Examples ### MLflow Python resolvers can also be used to load and run ML models. An example using MLflow is shown below. This example loads a model from the MLflow registry and uses it to make predictions. Using the @before_all decorator, the model is loaded once before any online requests are made. The model is then used in the get_prediction function to make predictions based on the input features. ``` import mlflow from chalk import online from chalk.features import before_all from src.models import FeatureClass import ExampleModelClass model: ExampleModelClass | None = None @before_all def load_model(): mlflow.set_registry_uri("EXAMPLE_REGISTRY_URI") mlflow.set_tracking_uri("EXAMPLE_TRACKING_URI") global model model = mlflow.load_model("models:/mymodel@production") @online def get_prediction( model_input: FeatureClass.model_input, ) -> FeatureClass.model_output: x_in = [model_input] return model.predict(x_in) ``` # Environments source: https://docs.chalk.ai/docs/resolver-environments ## Change resolver behavior by deployment. Environments are used to trigger behavior in different deployments such as staging, production, and local development. For example, you may wish to interact with a vendor via an API call in the production environment, and opt to return a constant value in a staging environment. ### Specifying environments You can choose to scope resolvers to a restricted set of environments. Resolvers optionally take a keyword argument named environment that can take one of three types: - Unassigned (default) - The resolver will be a candidate to run in every environment. - String value - The resolver will run only in this environment. - List of strings - The resolver will run in any of the specified environments and no other environments. ### Example Say your fraud models needed to interact with a fraud vendor that you wanted to mock out in staging. We can scope the environments as follows: ``` @online(environment="**production**") def fraud_score_prod(email: User.email, phone: User.phone) -> User.fraud_score: return api_vendor.fraud_score(email) @online(environment=["**staging**", "**dev**"]) def fraud_score_staging(email: User.email) -> User.fraud_score: if email == "fraud_user@chalk.ai": return 10 return 90 ``` Resolvers in different environments don't need to take the same arguments. In the above example, the production version of the resolver takes a phone number and an email, while the staging version of the resolver takes only the email. # Routing tags source: https://docs.chalk.ai/docs/resolver-tags ## Select eligible resolvers and SQL source routes within an environment. Routing tags control resource selection within an environment. A query's routing tags make matching tagged resolvers eligible to run and select a matching route in a tagged SQL source group. Both the routing tags and environment need to match for a resolver to be a candidate to execute. Routing tags do not grant access to the selected resource. Permission tags authorize access to datasources and features after routing; feature tags do not participate in routing. You might consider using tags, for example, to change out whether you want to use a sandbox environment for a vendor, or to bypass the vendor and return constant values in a staging environment. ### Specifying tags Tags can be either a string value, or a key-value pair. As a best practice (and fitting with recommendations from other services like Datadog), we recommend using key-value pairs, but the choice is yours. There are two ways to specify tags on resolvers: - "key" - As a single scalar string. - "key:value" - As a string value representing a key-value pair. ### Python resolver tags Resolvers take one or many tags, all of which need to match for the resolver to run. For example, you may want to test with a sandboxed vendor, and also be able to set a constant value for a particular feature. ``` @online(tags=["**api:mock**", "**fraud**"]) def simulate_fraud(id: User.id) -> User.fraud_score: return 100 @online(tags="**api:mock**") def simulate_no_fraud(id: User.id) -> User.fraud_score: return 10 @online(tags="**api:sandbox**") def sandbox_score( name: User.name, email: User.email, ) -> User.fraud_score: return sandbox.fraud_score(name, email).score @online def real_score( name: User.name, email: User.email, ) -> Features[User.fraud_score, User.fraud_tags]: r = prod.fraud_score(name, email).score return User(fraud_score=r.score, fraud_tags=r.tags) ``` In the above example, the resolver that is chosen to compute the User.fraud_score feature will depend on the tags provided at query time. The table below shows which resolver will be chosen for a given set of tags. | Query tags | Resolver | | ----------------- | ----------------- | | [api:mock, fraud] | simulate_fraud | | api:mock | simulate_no_fraud | | api:sandbox | sandbox_score | | | real_score | Note that these resolvers don't need to take the same set of inputs, and don't need to return the same types. ### SQL resolver tags SQL resolvers can also specify flags via the tags comment. For example, if you wanted to use Snowflake for User features in a fraud context and use Redshift for the same features in a signup context, you could write two different SQL file resolvers with different tags: ``` -- type: offline -- resolves: User -- source: snowflake -- tags: ["fraud"] select id, name, fraud_status from users ``` ``` -- type: offline -- resolves: User -- source: redshift -- tags: ["signup"] select id, name, fraud_status from users ``` ### When tagged resolvers run Like Environments, tags control when resolvers run based on the Online Context or Training Context matching the tags provided to the resolver decorator. Resolvers optionally take a keyword argument named tags that can take one of three types: - Unassigned (default) - The resolver will be a candidate to run for every set of tags. - String value - The resolver will run only if this tag is provided. - List of strings - The resolver will run only if all the specified tags match. If your resolver is tagged only by a key (not a key-value pair), and the request context contains a key-value pair such that the resolver's tag (a key only) matches they key of a key-value pair in the context, the resolver will be eligible to run. For example: | Resolver Tag | Request Context | Matches? | | ---------------------- | ---------------------- | -------- | | api | api | Yes | | api | api:live | Yes | | [api:live, mock-phone] | [api:live, mock-phone] | Yes | | [api:live, mock-phone] | api:live | No | | api:live | api | No | | api:fixture | api:live | No | ### Example Frequently, you'll want to combine tags and Environments, as below. This span uses a constant value in staging when the tag api takes the value fixture, uses the sandboxed fraud vendor in staging when the tag api takes the value live, and uses the production fraud vendor in production. ``` @online(environment="staging", tags="**api:fixture**") def fraud_score_fixture(email: UserFeatures.email) -> UserFeatures.fraud_score: if email == "elliot@chalk.ai": return 100 return 50 @online(environment="staging", tags="**api:live**") def fraud_score_sandbox(email: UserFeatures.email) -> UserFeatures.fraud_score: return api_vendor_sandbox.fraud_score(email, profile="dev") @online(environment="production") def fraud_score_prod(email: UserFeatures.email) -> UserFeatures.fraud_score: return api_vendor_live.fraud_score(email) ``` # Errors source: https://docs.chalk.ai/docs/resolver-errors ## Surface and handle failed feature values. Resolvers can raise errors in the course of executing, and handle errors in resolving other feature values. ### Raising errors If your resolver fails by raising an exception, the error will be surfaced as a ChalkError in query responses. Clients will receive a null value for the feature, and by default, downstream resolvers won't run. ### Handling errors By default, if there was an error in computing one of the inputs to a resolver, that resolver will also fail with the error code UPSTREAM_FAILED. To handle failures in the input features, take as input the feature value, but wrapped in typing.Optional: ``` from typing import Optional @online def fn(score: Optional[User.fraud_score]) -> ...: ``` The argument score will receive the value None in the case of a failure. If you'd like to provide a default value to use in the case of a failure, you can use Python's syntax for assigning a default value to an argument: ``` @online def fn(score: Optional[User.fraud_score] = -1) -> ...: ``` # Scheduling source: https://docs.chalk.ai/docs/resolver-cron ## Automate resolver runs. You can schedule resolvers to run on a pre-determined schedule via the cron argument to resolver decorators. ``` @online(cron="**90d**") def fn() -> DataFrame[User.id, User.name]: return ... ``` The cron keyword arguments takes a duration to determine the schedule on which to run. For more fine-grained control, you can alternatively specify a crontab in the UTC timezone. ``` @online(cron="***/5 * * * ***") def region_average( houses: DataFrame[ House.city, House.rent_price ] ) -> DataFrame[Region.name, Region.average_price]: return houses.group_by( group={Region.name: House.city}, agg={ Region.average_price: op.mean(House.rent_price), }, ) ``` In SQL resolvers you can also use the cron keyword in the comments. ``` -- type: offline -- resolves: user_seller_features -- source: snowflake -- owner: disco -- tags: ["team:discoml","type:simulated"] -- cron: "0 0 * * *" ``` ### Scheduling with arguments Scheduled resolvers can also take arguments. Chalk gives you control over which arguments should be provided for each run of the schedule. ### All examples (default) Chalk calls the resolver with all unique argument names that could have called the function. Consider a scheduled resolver that takes in Feature 1 and Feature 2: In this example, we compute two points to sample: (1, 5) and (1, 7). These two values will be given to the resolver on the specified cron schedule. ### Filtering examples You can choose to filter down the set of all examples to run. The cron keyword argument alternatively takes a Cron class. By specifying a function from features to boolean, you can tell Chalk which of the default examples to run, and which to skip: ``` def filter_examples(bank_id: User.bank_account.balance) -> bool: return balance > 100 @online(cron=Cron(schedule="1d", filter=filter_examples)) def fn(balance: User.bank_account.balance) -> ...: ``` The arguments to the filter function all need to be rooted in the same entity as the features in the scheduled resolver, but there is no requirement that the filter function take a subset of the scheduled resolver's arguments: ``` def filter_examples(status: User.bank_account.status) -> bool: return status == "opened" @online(cron=Cron(schedule="1d", filter=filter_examples)) def fn(balance: User.bank_account.balance) -> ...: ``` ### Custom examples Finally, you can pick exactly the examples that you'd like to run. ``` def pull_examples() -> DataFrame[User.id]: return DataFrame.read_csv(...) @online(cron=Cron(schedule="1d", samples=pull_examples)) def fn(uid: User.id) -> ...: ``` In the above example, we provide all arguments to the resolver function. However, you may also choose to provide only a subset of the arguments, and Chalk will sample the other arguments. ``` def pull_examples() -> DataFrame[User.id]: return DataFrame.read_csv(...) @online(cron=Cron(schedule="1d", samples=pull_examples)) def fn(uid: User.id, balance: User.account.balance) -> ...: ``` # Reverse ETL source: https://docs.chalk.ai/docs/reverse-etl ## Move data from offline to online. Reverse ETL is the process of moving data from a data warehouse into operational systems. In the context of Chalk's architecture, our data warehouse is the offline data store (Timescale or BigQuery) and our operational system is the online data store (Redis, Cloud Memorystore, or DynamoDB). Chalk's API client can be used to query the online data store and the offline data store. one our API clients, while the offline data store can be queried by our bulk API. Data from online resolvers is always loaded into the offline store and made available for training. In contrast, data from offline resolvers is not loaded into online stores by default. To enable offline data to reach the online environment, use the keyword argument etl_offline_to_online on the feature you wish to ETL. ``` from chalk.features import features, feature, offline from chalk import DataFrame @features class User: ... favorite_color: str = feature(etl_offline_to_online=True) @offline def fn(...) -> DataFrame[User.favorite_color, ...]: ... ``` When this argument is present in the feature declaration, Chalk copies this feature into the online environment. Reverse ETL can also be assigned to all features in a namespace: ``` from chalk.features import features, feature @features(etl_offline_to_online=True) class User: fraud_score: float full_name: str email: str = feature(etl_offline_to_online=False) ... ``` Here, User.fraud_score and User.full_name will be reverse ETL'd into the online environment. However, User.email, which specifies the ETL parameter at the feature level, will not be reverse ETL'd. ### Interplay with max staleness When data from an offline store reaches an online store, it is necessarily somewhat stale. The data may have come from an events table, where it could be arbitrarily old, or it could be a snapshot that was live when the snapshot was taken, but takes non-zero time to migrate online. Therefore, you will only receive offline data from queries in the online environment when you your queries tolerate maximum staleness via features or maximum staleness via queries. # Triggered Runs source: https://docs.chalk.ai/docs/runs ## Execute resolver runs via API In addition to scheduling resolver executions, Chalk allows you to trigger resolver executions programmatically via the Chalk Client, CLI and REST API. ### Trigger a Run via Chalk Client Using the Chalk Client, you can trigger a resolver run using the trigger_resolver_run command. ``` from chalk.client import ChalkClient ChalkClient().trigger_resolver_run(resolver_fqn="load_user_data", store_online=True) ``` To read more about how to specify parameters like upper_bound, lower_bound, store_online, store_offline, and idempotency_key, check out our API documentation. ### Trigger a Run via CLI The chalk trigger command allows you to trigger a resolver execution via the CLI. ``` $ chalk trigger --resolver my.module.fn ID: j-2qtwuxpskm2pbg Status: Received URL: https://chalk.ai/runs/j-2qtwuxpskm2pbg ``` To read more about how to specify flags like deployment, persist-online, persist-offline, and idempotency-key, check out our CLI documentation. ### Trigger a Run via API Using the trigger API endpoint allows you to build custom integrations with other data orchestration tools like Airflow. The trigger endpoint will return within 300 seconds, even if the triggered run takes longer to execute. If the returned status is received, you should poll the v1/runs/{id} endpoint until status transitions to succeeded or failed. It is also possible to manually trigger resolver executions via the Chalk dashboard, on the details page for the resolver you would like to run. The same input parameters are available in the dashboard. Note: if you use a user-scoped token (i.e. one that's derived from chalk login) you will need to specify the environment in which to trigger your resolver via the X-Chalk-Env-id header. ### Trigger a run The fully qualified name of the resolver to trigger. Example: 'neobank.resolvers.offline_get_transactions'. The lower bound timestamp to use when sampling input features to run this resolver on. Example: '2021-01-01T00:00:00Z'. The upper bound The maximum number of samples to pull from the offline store when running this resolver. Example: 1000. ### Response ID of the relevant run. One of 'received', 'succeeded', or 'failed' ### Example ``` curl -XPOST -H "Authorization: Bearer $(chalk token)" \ -H "X-Chalk-Env-Id: " \ -H "Content-Type: application/json" \ https://api.chalk.ai/v1/runs/trigger \ -d '{ "resolver_fqn": "neobank.resolvers.offline_get_transactions" }' # Returns { "id": "", "status": "succeeded" } ``` ### Query run status URL Param. ID of the relevant run. ### Response ID of the relevant run. One of 'received', 'succeeded', or 'failed' ### Example ``` curl -H "Authorization: Bearer $(chalk token)" \ -H "X-Chalk-Env-Id: " \ https://api.chalk.ai/v1/runs/{run_id} ``` # Timeout source: https://docs.chalk.ai/docs/timeout ## Maximum execution time for resolvers Computing features associated with third-party services can be unpredictably slow. Chalk helps you manage such uncertainty by specifying a resolver timeout duration. Chalk will instantaneously (±1ms) cease waiting for a resolver whose runtime has exceeded the specified timeout duration. Simultaneously, via parallel execution, Chalk will return other crucial features whose resolvers did not time out. To specify a timeout, use the timeout keyword argument to either @online or @offline: ### Example ``` from chalk.features import online, offline @online(timeout="200ms") def resolve_australian_credit_score( driver_id: User.driver_id_aus, ) -> User.credit_score_aus: return experian_client.get_score(driver_id) @offline(timeout=timedelta(milliseconds=50)) def resolve_canadian_credit_score( user_si_number: User.social_insurance_number, ) -> User.credit_score_can: return equifax_client.get_score(user_si_number) ``` See Duration for supported durations. ### Behavior upon timeout ### Direct timeout If a query's output feature is directly resolved by a resolver that timed out, Chalk will return an error with the error code RESOLVER_TIMED_OUT, and its message will contain the name of the timed-out resolver. ### Upstream timeout If a query's output feature is resolved by a resolver whose upstream resolver has timed out, Chalk will return an error with the error code UPSTREAM_FAILED. The resolver that directly outputs the requested feature will not be run (unless it marks its inputs as optional). ### Multiple output features If your query contains multiple output features, Chalk will still return the other features whose resolver did not time out. # Integrations Overview source: https://docs.chalk.ai/docs/integrations ## Integrate any API, 3rd-party client or data source without needing to orchestrate data pipelines Chalk connects to your underlying systems and queries your data sources directly, without an ETL step. Querying sources directly: - alleviates the need to move data across multiple systemssingle source of truth (define once and use everywhere)prevents data drift (same feature logic for offline and online workloads)reduces data duplication and storage costs - optimizes compute by only ever fetching exactly what you need when you need it (dynamic query planning) - enables real-time data delivery by satisfying strict (under 5ms) latency requirements ### Cloud Platforms Anywhere that you can run Kubernetes, you can run Chalk -- Chalk is cloud-agnostic. Chalk deploys into your VPC co-located with your data sources for the lowest latency and cost. Multi-cloud deployments for high availability and disaster recovery. - AWS (Amazon Web Services) - GCP (Google Cloud Platform) - Azure Cloud (Microsoft) ### SQL data sources and data warehouses Chalk has native drivers and integrations with a variety of SQL data sources and query engines, and provide a unified interface for adding new data sources. Adding a new SQL source is as simple as providing a connection string and a few configuration options through your Chalk dashboard. Once it's been added to your Chalk deployment, you can start querying it right away with SQL Resolvers. ``` -- resolves: User -- source: postgres select id, name, from users ``` The features in a feature class can be hydrated from multiple SQL sources -- we can pull a user's social security number from a different database that has stricter access controls. ``` -- resolves: User -- source: restricted_postgres select id, ssn from sensitive_user_data ``` In addition, Chalk can reverse ETL features from your data warehouses into Chalk's online store for low-latency access. Chalk integrates natively (C++ integration) with the following data sources and pushes down filters and projections into SQL queries for more efficient data fetching. Data Warehouses - Snowflake - Databricks Native: - MySQL - PostgreSQL - ClickHouse - Presto / Trino - DuckDB AWS: - Redshift - Athena - DynamoDB GCP: - BigQuery - Spanner - Alloy Azure: - Microsoft SQL Server (MSSQL) / Azure SQL Database - Database for PostgreSQL - Database for MySQL ### Streaming / Real-Time Data Systems We provide stream resolvers for integrating Kafka compatible data sources. - Kafka compatibleConfluentRedpanda - Kinesis (AWS) - Pub/Sub (GCP) - Event Hubs (Azure) Streams can also be filtered, processed, and materialized as a step in Chalk's feature computation pipelines. ``` @stream(source=KafkaSource(name='transactions_stream')) def process_transaction_topic( value: TransactionMsg, ) -> Features[Transaction.id, Transaction.user_id, Transaction.amount]: return Transaction( id=value.id, user_id=value.user_id, amount=value.amount, ) ``` ### Feature caching with expensive features with Redis/Valkey, Memcached, DynamoDB, and more Chalk makes it easy to cache features for low-latency access with the max_staleness keyword argument. These features skip expensive API calls and are fetched from the online store. ``` @feature class User: id: int name: str ssn: int credit_score: int = feature(max_staleness="30d") ``` We support a variety of caching backends: - Redis / Valkey - Memcached - DynamoDB - Amazon ElastiCache - Google Cloud Memorystore - Azure Cache for Redis - Azure Cosmos DB ### APIs & Microservices Call internal APIs, third-party services, and microservices with built-in retry logic and circuit breakers: ``` @online def get_credit_score(ssn: User.ssn) -> User.credit_score: response = requests.get( f"https://api.creditbureau.com/score/{ssn}", headers={"Authorization": f"Bearer {API_KEY}"}, timeout=2.0 ) return response.json()["score"] ``` Chalk's Symbolic Python Interpreter supports accelerating libraries like requests, and so this function gets run in C++. ### Object Storage and Iceberg Catalogs AWS (Amazon Web Services) - S3 (Amazon Simple Storage Service) - Glue Catalog GCP (Google Cloud) - Google Cloud Storage - Cloud Data Catalog - BigLake Microsoft - Azure Blob Storage - Microsoft Purview - Azure Data Lake Chalk is Iceberg native and can write to your underlying object storage and catalog directly from offline queries. ``` from chalk.integrations import GlueCatalog catalog = GlueCatalog( name="aws_glue_catalog", aws_region="us-west-2", catalog_id="123", aws_role_arn="arn:aws:iam::123456789012:role/YourCatalogueAccessRole", ) results.write_to(destination="database.table_name", catalog=catalog) ``` ### AI & ML Services Access traditional machine learning functions like Scikit, XGBoost, and your own models directly within feature definitions using Chalk Expressions: - Sci-kit functions - ONNX Models (Open Neural Network Exchange) through Chalk's model registry Integrating unstructured data with LLMs (large language models) or computing embeddings is straightforward with Chalk's built-in integrations. Easily conduct Evals, switch out different models and providers, and reference the features you need in your prompts without having to configure complex pipelines. - OpenAI - Anthropic - AWS Bedrock - Azure OpenAI - Google Vertex - Any OpenAI compatible chat completion modelCerebrasGroqOllama CloudTogether.ai You can override the base URL and API key to connect to any OpenAI compatible endpoint. ``` @features class Item: id: int title: str description: str llm: P.PromptResponse = P.completion( model="gpt-5.1-2025-11-13", messages=[ P.message( role="user", content=F.jinja( """ Classify the following item category using its title and description: Item title: {{ Item.title }} Item description: {{ Item.description }} """, ), ), ], output_structure=StructuredOutput, ) ``` You can just as easily compute embeddings for items, users, or any other entity using built-in integrations: ``` @features class VectorSearch: q: Primary[str] # from chalk.features import embed vector: Vector = embed( input=lambda: VectorSearch.q, provider="vertexai", model="text-embedding-005", ) query_type: str = "vector" results: "DataFrame[ItemDocument]" ``` ### Get started today With dozens of native integrations across cloud platforms, databases, streaming systems, caching layers, and AI services, Chalk eliminates the complexity of building and maintaining production machine learning systems. Whether you're pulling user data from PostgreSQL, processing real-time events from Kafka, caching expensive feature computations in Redis, or extracting features from unstructured data with LLMs — Chalk's unified platform handles it all. The result? Faster time to production, lower operational overhead, and consistent feature logic across your entire ML stack. # Configuring Data Sources source: https://docs.chalk.ai/docs/configuring-data-sources ## Configure connections between Chalk and your data sources using the Chalk dashboard. Chalk connects to a variety of data sources, so you can call external APIs, query production databases, and more directly from Chalk. You can configure data sources through the data source API or through the Chalk dashboard. This page covers the dashboard. ### Adding data sources through the dashboard Configure connections in the Chalk dashboard under Integrations > Data sources. Select Add data source, choose a data source type, and fill in the connection information, such as credentials and host details. The Integrations section lists the supported data sources. ### Example: adding a Kinesis data source This example connects Amazon Kinesis to Chalk as a streaming data source. After you select Kinesis as the data source type, fill in the connection information and the workload role that needs permission to access your stream. Before you save, select Test connection to check that the configuration works. Chalk attempts to connect to your data source with the information you provided and reports any issues. To also see sample messages from a Kinesis stream, select Preview test message before you test. A successful test lists the messages on the Messages tab: ### Using your data source After you add a data source and test the connection, you can reference it in your resolvers. The Integrations section and the page for your data source include examples. # Docker source: https://docs.chalk.ai/docs/docker ## Customize the base image for your pipelines. Chalk runs your feature pipelines on Docker images. If your code relies only on your resolvers and data source definitions, you may use Chalk's base image with no modification. If you require additional dependencies, you can specify installation and management instructions. You can specify build options on a per-environment basis, and you can provide a default set of configuration for environments that do not have explicit configuration values. ### Extra pip dependencies Chalk can install extra dependencies with pip. By default, Chalk looks for a file named requirements.txt in the base of your project. However, you can override this location or specify requirements files per-environment in the chalk.yaml file located at the base of your repo. ``` project: my-project-id environments: default: runtime: python312 requirements: ./requirements.txt dockerfile: Dockerfile prod: requirements: ./requirements-prod.txt ``` ### Building your own image You may have dependencies that cannot be expressed through pip. For those dependencies, you need to customize the base image Chalk uses to run your pipelines. Chalk allows you to specify a custom Dockerfile that will be used to build a base image for your deploys. ``` project: my-project-id environments: default: dockerfile: ./Dockerfile prod: dockerfile: ./DockerfileProd ``` You can build any Debian-based image that you like, with a few requirements: - python must be on the path - pip must be on the path - The Python version can be python311, python312, python313, or python314 - There should be no folder or file at /app Chalk is responsible for mounting your code into the image, and for installing your requirements with pip. ### Python Version The Python version for a Chalk project is set at the project level. The version is set with the chalk.yml file in your project repo. Chalk supports python311, python312, python313, and python314 as values for runtime. If you omit runtime, Chalk uses python311: ``` project: my-project-id environments: default: runtime: python312 prod: runtime: python311 ``` # Amazon Web Services source: https://docs.chalk.ai/docs/aws ## Setup your AWS integration AWS recommends the use of temporary security credentials instead of long-term access keys for enhanced security. It is strongly recommended to use IAM workload roles with access to your AWS resources rather than access keys where possible. Chalk has integration with AWS Services such as Athena, Redshift, and S3. You can use these integrations to connect Chalk to your data sources in AWS and reference them in your resolvers. ### Authorization To enable access to your AWS resources, you will need to ensure that the IAM Role defined in your cluster deployment has the necessary permissions to access the relevant AWS services. You can view the IAM workload role that requires access to your data source when setting up your integration in the Chalk dashboard at Integrations > Add a data source. # S3 / Object Storage source: https://docs.chalk.ai/docs/s3 ## Integrate with S3 compatible data sources. ### Reading .csv files Chalk parses .csv files from S3 or the local file system with the function DataFrame.read_csv(...). If your .csv has headers, you can tell Chalk which columns to parse into which features by providing a mapping in the columns keyword argument: ``` DataFrame.read_csv( path="s3://your-bucket/path/to/file.csv", columns={"uid": User.uid, "fraud score": User.socure_score}, ) ``` The first line of the file is taken to be the header row, and subsequent lines are parsed into the provided features. The returned value matches the type: ``` DataFrame[User.uid, User.socure_score] ``` Alternatively, you can map the file into features by column number in the .csv, and optionally skip rows that might contain header fields or summary stats: ``` DataFrame.read_csv( path="s3://your-bucket/path/to/file.csv", columns={0: User.uid, 1: User.socure_score}, has_header=True, ) ``` ### Reading .parquet files Reading .parquet files works just like reading .csv files. As with .csv, Chalk parses .parquet files from S3 or the local file system, this time with the function DataFrame.read_parquet(...). Otherwise, the functionality provided is the same as provided above for reading .csv files. ### Authentication Chalk can connect to AWS S3 or GCP Cloud Storage for reading .csv and .parquet files. Chalk will use the application credentials that you set up in the AWS and GCP integrations sections. # Google Cloud Platform source: https://docs.chalk.ai/docs/gcp You can provide Chalk with GCP application credentials to use integrations with GCP (like BigQuery and Cloud Storage). By navigating to Integrations > Add a data source and selecting GCP, you'll find a form where you can input information about your GCP integration. You are encouraged to use different service accounts for different environments, although they can be shared. ### Creating credentials Credentials are created through GCP directly. You'll want to make a new service key for Chalk with the permissions you require. Credentials are created through GCP's dashboard. You can create a key and download it from your console here. ### BigQuery permissions If you plan to add a BigQuery data source to your Chalk deployment, then you should ensure that the permissions set includes bigquery.tables.create and bigquery.jobs.create. When querying your BigQuery data source, Chalk will push down filters on top of your queries to optimize the amount of data read from your tables. For larger queries, rather than interpolating values directly in the SQL string for the query, which has length limits in BigQuery, Chalk will use a table to temporarily hold the values against which to query. # SQL Integration source: https://docs.chalk.ai/docs/sql ## Integrate with SQL-like sources. Chalk can ingest data using a SQL interface from any of our supported SQL data source integrations. The full list of supported SQL data sources can be found in our API reference. ### Basic queries Chalk supports running SQL from files or from strings. When you run queries that invoke your SQL resolver, Chalk automatically pushes your input parameters into the WHERE clause of your query. For more details, see our section on push-down filters. The examples on this page use our PostgreSQL data source, but can be generalized to any of our other SQL data sources. ### SQL file resolvers SQL file resolvers are Chalk's recommended method for writing SQL resolvers. The .chalk.sql extension marks a file as a SQL file resolver. Chalk SQL is a separate feature: a SQL dialect for querying across a Chalk environment. A SQL file resolver's query is written in Chalk SQL only when the file declares source: chalksql, as described in Chalk SQL resolvers. ``` -- type: online -- resolves: user -- source: PG select id, email, full_name from users ``` Here, we define an online resolver that returns some features for the User feature class from the users table in the PostgreSQL source PG. The comments are yaml-parsed to provide other metadata for Chalk to decide how to design the resolver. SQL file resolvers can return multiple rows for aggregation operations, offline queries, and more. It's also possible to use SELECT * in a SQL file resolver, but be careful! ``` -- type: online -- resolves: user -- source: PG select * from users ``` Implicitly, this query tries to align every scalar feature from the User feature class to a column name in users. If a feature name is misnamed or absent from the table, you'll get a "missing columns" error. To programmatically generate SQL file resolvers, check out Generated SQL file resolvers. ### Parameterized feature inputs Like other resolvers, SQL file resolvers can take features as input. In this example, we want our resolver to require EmailUser.id as input: ``` -- type: online -- resolves: user -- source: PG select id, email, full_name from users where id = ${user.id} ``` Use ${} with snake case to reference the desired feature. Use ${now} in your query as a special argument representing the time of the query. For more details, see our time documentation. ### SQL strings You can run SQL queries either as Python strings or from .sql files. When the name of the column matches the name of the feature with non-alphanumeric characters removed, the mapping from column to feature is automatic. ``` pg = PostgreSQLSource(name='PG') @online def get_user(uid: User.id) -> Features[User.full_name, User.email]: return ( pg.query_string("select full_name, email from users where id=1") .first() ) ``` get_user's return type indicates that it expects features named full_name and email, which are returned as columns from the SQL query. If the column names don't align exactly, you can include the parameter fields to specify the mapping from the query to the fields. ``` pg = PostgreSQLSource(name='PG') @online def get_user(uid: User.id) -> Features[User.full_name, User.email]: return ( pg.query_string( "select name, email from users where id=1", fields=dict(name=User.full_name), ) .first() ) ``` Here, the email column of the query automatically aligns with the expected User.email feature, but the name column of the query is explicitly mapped to the User.full_name feature. ### Parameterizing string queries You can parameterize queries to pass variables. Parameterize names with a colon, and pass a dictionary from parameter name to parameter value: ``` pg.query_string( query="select * from users where user_id=**:user_id**", args=dict(user_id="**uid123**"), ) ``` Use a backslash to escape any literal : characters you need to use in your query: ``` pg.query_string( query="select * from users where user_id=**:user_id** and name=**'\:colon'**", args=dict(user_id="uid123"), ) ``` ### SQL string files Instead of passing a string directly into your Python code, you can load the SQL content from a file. You can use the query_sql_file function. For example, here is a query.sql file containing the same query from above: ``` select * from users where user_id=:user_id ``` You can reference this file in a Python resolver, either using the absolute path from the root of your project or relative to the resolver's file. For example, if the snippet below lived in the same directory as query.sql, we could refer to it as follows: ``` pg = PostgreSQLSource(name='PG') @online def get_user(uid: User.id) -> Features[User.full_name, User.email]: return ( pg.query_sql_file( path="query.sql", args=dict(user_id=uid) ) .first() ) ``` Auto-mapping of column name to feature name also applies for the query_sql_file method. Parameterization also works the same way. ### Push-down filtering Chalk automatically adds query inputs to the WHERE clause of your SQL queries. We recommend relying on our push-down filtering logic rather than parameterizing your queries by hand. The following example will show a transaction feature and how we use push-down filtering to retrieve transactions belonging to a specific category. Here's the feature class: ``` pg = PostgreSQLSource(name='PG') @features class Transaction: id: int category: string amount: float ``` And the SQL file resolver: ``` -- type: online -- resolves: transaction -- source: PG SELECT id, merchant_category AS category, amount FROM txn_table; ``` Finally, here is our query for grocery transactions: ``` client = ChalkClient() client.query( input={ Transaction.id: [1, 2, 3, 4], Transaction.category: "Groceries", }, output=[ Transaction.id, Transaction.amount, ], ) ``` When this query is run, Chalk adds each input parameter to the WHERE clause, effectively running this query against your database: ``` SELECT id, merchant_category AS category, amount FROM txn_table + WHERE + id IN (1, 2, 3, 4) + AND merchant_category = "Groceries"; ``` If your database column names differ from your feature names, the column name must be aliased in the SELECT clause of your query, as shown above with the merchant_category column. ### Configuration ### Supported SQL file resolver comment keys The following are supported keys that may be included in .chalk.sql file comments. Describes the namespace to which the outputs belong. In the above example, user.email and user.full_name are the outputs. Describes the database by name, as in the above example, or the type if there is only one database of that type. Thus, if you have one PostgreSQL source, you can also write source: postgresql. The reserved value chalksql binds no database at all: see the Chalk SQL resolvers section below. The type of resolver. If not specified, online is the default. Parameters for incremental queries. For more information, see the below section on incremental queries. The incrementalization mode decides how to ingest new data. Defaults to "row". The length of the window from the last observed row that Chalk will re-ingest, e.g. 1h. Defaults to 0. The timestamp column in the underlying table to use as the basis for incrementalization. Must be supplied in row and group modes. The timestamp used for timedelta calculations. Defaults to feature_time. Returns one. Equivalent to the common query finalizer .one(). The maximum time to wait before timing out the query. See Duration for more details. The user tags associated with this resolver. For online and offline resolvers. The environments associated with this resolver. The schedule for a cron run, e.g. 1h. The max staleness for the resolver, e.g. 24h. For each primary key, the longest it may go without a new row, as measured by each row's FeatureTime. Chalk uses it to bound the query from below instead of reading all history to find each key's latest version. Set it on snapshot tables; see the section below. The owner of the resolver. An optional mapping from SQL column to Chalk feature. For example, with SELECT name AS arbitrary_column_name, we can map the arbitrary_column_name to a Chalk feature belonging to the namespace described by the resolves field with the mapping arbitrary_column_name: chalk_feature_name. A list of features that must be unique for each row of the output. This enables unique optimizations in the resolver execution. Only applicable to resolvers that return a DataFrame. A list of features that correspond to partition keys in the data source. This field indicates that this resolver executes its query against a data storage system that is partitioned by a particular set of columns. This is most common with data-warehouse sources like Snowflake, BigQuery, Databricks, or Iceberg. Whether this resolver returns all ids of a given namespace. To have this annotation, the resolver must take no arguments and return a DataFrame. Conditions under which the resolver re-runs its query. See the below section on retry policies. Re-run the query when an attempt returns no rows. Takes a single backoff strategy, exp, with the keys factor, n_retries, and base_ns. Re-run the query when an attempt fails with a server-side statement timeout. Takes the same exp backoff as if_not_found. ### Bounding history on snapshot tables When a resolver returns a FeatureTime feature, each row is assumed to be valid from that point in time going forward; there can be multiple rows with the same primary key but different FeatureTime values, which allows the resolver to describe entities that change over time. When querying the resolver, Chalk will pick the newest version of a row at-or-before the as-of input time. This means the Chalk query engine automatically places an upper bound on the returned FeatureTime feature. However, it is possible that the most-recent version of an entity is arbitrarily old, which means that Chalk cannot automatically place a lower bound on returned rows, forcing potentially unbounded historical lookback to discover the most-recent version of each row. A resolver's author might know that unbounded historical lookback is not necessary to get the most-recent version of an entity. For example, if a table contains daily snapshot rollups, where every entity has a new row computed every day, it's never necessary to look back more than one day before encountering the most-recent row. The option max_row_version_lookback can be used to supply this fact to the Chalk query planner, allowing for bounded lookback relative to the as-of input query time: ``` -- type: offline -- resolves: account -- source: PG -- max_row_version_lookback: 1d select account_id as id, balance, record_date as updated_at from daily_account_balances ``` Here Account.updated_at is the feature class's FeatureTime, so record_date is what tells Chalk when each balance became true. The resolver must select its namespace's FeatureTime, since that is the column being bounded. The precondition that has to hold when max_row_version_lookback is set to 1d is: for any as-of input time t, if a primary key has any row at-or-before t, then its most recent such row is at-or-after t - 1d. t is any as-of input time, not now. Every input row carries its own, and in an offline query they are usually historical, so the precondition has to hold at all of them. Chalk turns that precondition into a lower bound on the query. For a query as of 2025-01-04, it may skip every row before 2025-01-03, because the precondition says the row it wants is not among them. Match the value to the table's refresh cadence plus whatever lateness you want to survive: the 1d above assumes the job never runs late; 2d buys one missed run. A wider window causes more rows to be scanned, while too narrow a window returns nulls for keys whose latest row fell just outside it. ### When it does not apply Chalk applies the bound when you query the namespace directly or reach it through a has_one. It skips the bound, silently and with the result unchanged, in every other case: - when the resolver's rows are the elements of a DataFrame relationship; - when the namespace is a DataFrame input to another resolver; - when the resolver takes a parameter, such as ${user.id}, and so runs once per input row. In each of these the query stays correct and simply is not accelerated. If you set the directive and see no change in how much your table is scanned, this list is the first thing to check. Rows whose FeatureTime is null are dropped by the bound. Elsewhere Chalk treats a null FeatureTime as always current, so a key whose only row has one resolves today and will stop resolving once this directive is set. Do not set it on a table with null FeatureTime values. ### Retrying queries Chalk always retries native SQL failures that look transient (a dropped or reset connection, a deadlock, throttling) on a fresh connection, and never retries failures that are deterministic, such as a SQL syntax error or a permission failure. A resolver can additionally opt in to two more retry conditions with retry_policy: ``` -- resolves: user -- source: PG -- count: one -- retry_policy: -- if_not_found: -- exp: -- factor: 2.0 -- n_retries: 3 -- base_ns: 50000000 -- if_timeout: -- exp: -- factor: 2.0 -- n_retries: 1 -- base_ns: 250000000 select id, email from users where id = ${user.id} ``` if_not_found re-runs a query that succeeded but returned no rows, for a resolver reading a store that may not yet reflect a very recent write, such as a read replica behind its primary. An attempt counts as "not found" when it returns no rows, which includes a count: one or count: first query rejecting an empty result. After n_retries retries the last result is returned as-is: a row that never appears still resolves to "not found", and no error is raised. if_timeout re-runs a query that the database gave up on because it exceeded a server-side statement time limit (for example Postgres statement_timeout). Without the condition, a statement timeout is not retried, because re-running a statement that already burned its time budget usually just multiplies the latency. For each condition, the retry after attempt i (0-based) waits base_ns * factor^i nanoseconds, so the if_not_found example above waits 50ms, then 100ms, then 200ms. Because the retries sleep, budget base_ns * (factor^n_retries - 1) / (factor - 1) against the resolver's timeout and your query's overall latency target. retry_policy is honored only by native SQL execution, and only for queries that take at least one parameter. A parameterless query streams its result, so its outcome is not known in time to retry; a resolver in that shape logs a warning at query time and runs without the policy. ### Proper comment formatting All comments must be inserted before the body of the SQL query. Each comment line is parsed as either a yaml-formatted line describing the resolver or a docstring. Below, the last comment will appear as a docstring since it is not in key:value format. ``` -- type: online -- resolves: user -- source: PG -- This comment is not in yaml format, so it will be parsed as a docstring select * from users ``` For field values that can be lists or dictionaries, such as tags or incremental settings, we can either enumerate the values inline or with an extra indentation. Remember, if your values include a colon, you must use quotes around your value in order for the line to have valid YAML format. Both of the following formats are valid and equivalent. ``` -- type: online -- resolves: user -- source: PG -- tags: ["single:1", "double:2"] select * from users ``` ``` -- type: online -- resolves: user -- source: PG -- tags: -- - single:1 -- - double:2 select * from users ``` ### Chalk SQL resolvers Instead of naming an integration, a SQL file resolver can declare the reserved source chalksql. The query is then written in Chalk SQL and compiled by the Chalk engine directly into the query plan, exactly like the input_sql of an offline query. This is the only case in which a .chalk.sql file contains Chalk SQL. No external database is contacted, and everything the Chalk SQL catalog exposes (files in object storage, attached catalogs, the offline store) is available to the query. ``` -- source: chalksql -- resolves: thermometer -- type: offline select id, degrees_fahrenheit, measured_at from read_parquet('s3://my-bucket/thermometers/*.parquet') ``` Because the query is compiled once, before any rows exist, a Chalk SQL resolver must be a root resolver: it takes no ${...} parameters, returns every row it selects (no count: one), and does not support incremental settings. Filters that the query planner can push down, for instance the observed-at bounds of an offline query, are applied to the compiled plan. If the query is a select *, every scalar feature in the resolves feature class becomes an output of the resolver, so the query has to return a column for each one. A feature computed somewhere else (by a Python resolver, or by an underscore expression) then fails to plan with does not select a column for output feature .... List the columns explicitly whenever the feature class holds anything the query does not return. ### File lists are fixed at deployment time Chalk compiles each Chalk SQL resolver's query once per deployment and reuses the compiled plan. A query that enumerates files, such as read_parquet over a wildcard, resolves that wildcard at compile time, so files added to the prefix afterwards are not picked up until the next deployment. Write Chalk SQL resolvers against data that is complete when you deploy, and redeploy to pick up new files. To read a dataset that grows continuously, use a SQL resolver against a datasource integration, or an offline query with input_sql, which is compiled per query. ### Streaming SQL file resolvers Chalk supports streaming with SQL file resolvers: ``` -- source: kafka -- resolves: store_features -- type: streaming select store_id as id, sum(amount) as purchases from topic group by 1 ``` ### SQL linting configuration If you are using SQLFluff or another SQL Linter, you may need to set configurations to accept the variable pattern. For SQLFluff, set the templater to placeholder and add the following to your config file. ``` # Allows sqlfluff to correctly parse # ${arbitrary_placeholder_name} [sqlfluff:templater:placeholder] param_regex =\$\{[^}]*\} 1 = 'arbitrary_placeholder_name' ``` ### Generated SQL file resolvers You can programmatically generate SQL resolvers with make_sql_file_resolver. This function is useful if you have many SQL tables and want to automate management of their resolvers. ``` from chalk import make_sql_file_resolver from chalk.sql import PostgreSQLSource pg = PostgreSQLSource(name='PG') definitions = [ { resolver_name: "get_user_features", entity_name: "User", table: "users", pkey_column: "id", features: ["feature1", "feature2"], }, ... ] for definition in definitions: targets = ", ".join(definition.features) make_sql_file_resolver( name=definition.resolver_name, sql=f"select {definition.pkey_column}, {targets} from {definition.table}", source=pg, # "PG" is also acceptable resolves=definition.entity_name, ) ``` make_sql_file_resolver adds this resolver to your registry as if it were a SQL file resolver, but without creating the .chalk.sql file. All SQL file resolvers require source and resolves. These values can be provided as SQL comments within your sql value or as parameters. If comments and parameters are both provided, parameters will take precedence. This function call is equivalent to the previous example: ``` make_sql_file_resolver( name=definition.resolver_name, sql=f""" -- source: PG -- resolves: {definition.entity_name} select {definition.pkey_column}, {targets} from {definition.table} """, ) ``` ### Incremental queries ### Overview The first time that a resolver with an incremental query is executed, Chalk ingests all data from the source. On subsequent runs of the resolver, Chalk only looks for new rows in the table to ingest. Using incremental queries limits the amount of data that Chalk needs to ingest, lowering latency for updates and reducing costs. Incremental queries are useful for ingesting immutable tables or queries, like event tables or logs. This type of data is frequently found in the offline context, as it represents logs of real-time events. Incremental queries use the incremental_column parameter to page over the underlying table. ### Incremental queries with SQL file resolvers Imagine you have a login events table, where you keep track of login attempts to your website. You can ingest this table with a SQL file resolver as follows: ``` -- type: offline -- resolves: login -- source: PG -- incremental: -- mode: row -- lookback_period: 60m -- incremental_column: attempted_at select attempted_at, status, user_id from logins ``` Configuration for incremental resolvers can be passed in the incremental dictionary of the file's comments. For more details, see the configuration reference above. ### Incremental queries with SQL strings To use incremental mode with SQL string resolvers, use .incremental along with the incremental_column parameter. ``` pg = PostgreSQLSource(name='PG') @offline def fn() -> DataFrame[Login.attempted_at, Login.user_id, Login.status]: return ( pg.query_string("select attempted_at, status, user_id from logins") .incremental(incremental_column="attempted_at") ) ``` ### Handling late-arriving messages If your underlying data source has "late arriving records", you may need to use the lookback_period argument to incremental. When lookback_period is specified, Chalk subtracts the lookback_period from the "maximum observed timestamp" that it uses as a lower-bound for ingestion. Concretely, if your resolver body looks like this: ``` ( db.query_string("SELECT * FROM payments") .incremental(incremental_column="updated_at", lookback_period="30m") ) ``` then Chalk will rewrite your SQL query to: ``` SELECT * FROM payments WHERE updated_at >= ( - ) ``` This means that rows that arrive up to 30 minutes late will be properly ingested. The trade-off is that Chalk will re-ingest some redundant data. ### Managing incremental resolvers Each incremental resolver tracks the timestamp of its latest ingested row in an internal property called chalk_incremental_timestamp. This property can be referenced within your SQL query when using parameter as your incrementalization mode. The Chalk CLI provides several commands for managing your incremental resolvers: - chalk incremental status shows the status and latest timestamp of the given resolver. - chalk incremental set allows you to directly modify the internal timestamps the resolver uses to track its state. - chalk incremental drop clears the resolver's tracking state so that it will restart data ingestion on its next run. ### Incrementalization modes The default incrementalization mode for .incremental is mode='row'. Three modes are supported: - row: Chalk ingests features from all rows whose incremental_column is newer than the previously observed max timestamp. - group: Chalk ingests features from all groups who are aggregating a row that has been added or changed since the previously observed max timestamp. - parameter: Chalk passes the chalk_incremental_timestamp value (including lookback_period) to your query, and leaves your query unchanged. ### "Group" incremental mode Group mode incremental ingestion is appropriate when you are aggregating rows in a table in order to compute features. For example, if you are running the following query: ``` SELECT business_id, SUM(amount) as sum_payments_amount, COUNT(*) as count_payments, max(updated_at) as ts FROM payments GROUP BY 1 ``` to ingest features for this feature class: ``` @features class Business: id: int sum_payments_amount: float count_payments: int ts: FeatureTime ``` then you can specify the following resolver: ``` @offline(...) def resolve_aggregate_payment_features() -> DataFrame[Business]: query = """ SELECT business_id, SUM(amount) as sum_payments_amount, COUNT(*) as count_payments, max(updated_at) as ts FROM payments GROUP BY 1 """ return db.query_string(query, fields={"business_id": Business.id}) \ .incremental(incremental_column="updated_at", mode="group") ``` and Chalk will automatically rewrite your query into this form: ``` SELECT business_id, SUM(amount) as sum_payments_amount, COUNT(*) as count_payments, max(updated_at) as ts FROM payments WHERE business_id IN ( SELECT DISTINCT(business_id) FROM payments WHERE updated_at >= :chalk_incremental_timestamp ) GROUP BY 1 ``` This means that if you have a payments table like this: ``` | id | business_id | amount | updated_at | | 1 | 1 | 10 | 2022-11-01T00:00:00.000Z | | 2 | 1 | 5 | 2022-11-15T00:00:00.000Z | | 3 | 2 | 7 | 2022-11-15T00:00:00.000Z | | 4 | 3 | 17 | 2022-10-01T00:00:00.000Z | ``` and your query had previously run on 2022-11-07, then Chalk would return the following aggregate values: ``` | business_id | sum_payments_amount | count_payments | ts | | 1 | 15 | 2 | 2022-11-01T00:00:00.000Z | | 2 | 7 | 1 | 2022-11-15T00:00:00.000Z | ``` Both business 1 and 2 are present, because they have at least one payment after 2022-11-07. Business 3 is excluded, since it has no payments after 2022-11-07. ### "Parameter" incremental mode In parameter incremental mode, Chalk leaves your query untouched. Chalk will simply pass the max incremental timestamp to your query as a bind parameter named chalk_incremental_timestamp. Concretely, if you write: ``` @offline(...) def parameter_incremental_mode_resolver() -> DataFrame[...]: return ( db.query_string("SELECT * FROM payments WHERE updated_at >= :chalk_incremental_timestamp") .incremental(mode="parameter") ) ``` Then Chalk will execute your query verbatim, and will keep track of the appropriate value for chalk_incremental_timestamp between executions of your resolver. ### Incremental interaction with FeatureTime When Chalk executes an incremental query, it has to update the "max timestamp" value that it will use as the lower bound for the next query. By default, Chalk sets this value to the time at the start of the query. If your resolver returns a FeatureTime feature, Chalk will update the "max timestamp" value to the "max" FeatureTime value that is returned instead. This allows you to control the incremental behavior more precisely. ### Tagged SQL sources Chalk supports applying routing tags to SQL sources. This allows you to define a single resolver that routes traffic to multiple different backing databases depending on tags supplied at query time. This is useful for limiting the blast-radius of traffic from different use-cases. Routing tags select a route; they do not authorize access to its datasource. Use datasource permission tags to authorize the selected datasource. First, define a SQL source group: ``` from chalk.sql import SQLSourceGroup, PostgreSQLSource sql_group = SQLSourceGroup( name='primary_group', default=PostgreSQLSource(name="default_replica"), tagged_sources={ 'risk': PostgreSQLSource(name='risk_replica'), 'analytics': PostgreSQLSource(name='analytics_replica'), } ) ``` Then, define a resolver that uses the group: ``` @online def users() -> DataFrame[User]: return sql_group.query_string("select id, name from users").all() ``` Then, when you submit queries, the routing tags control which data source is used to execute the query: ``` client = ChalkClient() # This query uses the risk datasource client.query(input={User.id: 1}, output=[User.name], tags=['risk']) # This query uses the analytics data source client.query(input={User.id: 1}, output=[User.name], tags=['analytics']) # This query uses the default datasource client.query(input={User.id: 1}, output=[User.name]) ``` ### Additional query options ### SQLAlchemy Chalk supports SQLAlchemy: ``` pg = PostgreSQLSource(name='PG') @online def get_user(uid: User.id) -> Features[User.email, User.full_name]: return ( pg.query(User(email=UserSQL.email, full_name=UserSQL.name)) .filter_by(id=uid) .first() ) ``` In the .query(...) call, you map the target columns of the SQL statement to the feature namespace. Here, we assign User.email to UserSQL.email and User.full_name to UserSQL.name. ### Incremental queries with SQLAlchemy To create an incremental SQLAlchemy query, use .incremental. Chalk will page over the underlying table using the column mapped to FeatureTime. ``` pg = PostgreSQLSource(name='PG') @offline def fn() -> DataFrame[Login.ts, Login.attempted_at, Login.user_id, Login.status]: return pg.query( Login( ts=LoginHistorySQL.created_at, attempted_at=LoginHistorySQL.attempted_at, user_id=LoginHistorySQL.user_id, status=LoginHistorySQL.status, ) ).incremental() ``` # Custom Data Source source: https://docs.chalk.ai/docs/generic ## Integrate with any data source you use. Chalk doesn't need to officially support a data source or API vendor for you to include it in your feature pipelines. Chalk has a mechanism for initializing objects at boot time, and then you can use those objects within your resolver. ### Initializing To initialize your custom data source, use the @before_all decorator. The decorated function will run before any of your resolvers are called: ``` import os from chalk.features import before_all vendor_client: VendorClient = VendorClient(...) @before_all def initialize(): key = os.getenv("MY_VENDOR_KEY") vendor_client.api_key = key ``` This function has access to all Secrets configured for your environment as Python environment variables. Then, in your resolver, you can use the vendor_client knowing it has been correctly initialized. ``` @online def get_vendor_score(...) -> ...: vendor_client.do_something(...) ``` You can alternatively supply environment-specific initialization functions. The before_all decorator takes an optional keyword argument environment that accepts a single environment name or a list of applicable environment names: ``` @before_all(environment="**production**") def initialize(): ... @before_all(environment=["**staging**", "**development**"]) def initialize(): ... ``` ### Initialization order The order in which setup hooks run is not guaranteed and may differ across instances of your service. Your setup hooks should not rely on being called in a specific order. ### Scopes You should be careful when trying to assign a variable outside the local scope. In the below example, the vendor_client isn't initialized as it may appear to be. Instead, a new local variable that shadows the name in the outer scope is initialized, and the outer variable doesn't receive a value. Careful with assigning outside of the local scope. ``` vendor_client: VendorClient @before_all def initialize(): vendor_client = VendorClient(...) ``` Instead, try modifying the client inside the initialization function: Modify the outer scope from the inner scope. ``` vendor_client: VendorClient = VendorClient(...) @before_all def initialize(): vendor_client.api_key = os.getenv("MY_VENDOR_KEY") ``` If you really want to assign a variable in the outer scope, you can use Python's global keyword. Use the global keyword to modify outside of local scope (not recommended, but correct) ``` vendor_client: VendorClient @before_all def initialize(): global vendor_client vendor_client = VendorClient(...) ``` However, if you do this, you need to make sure that access to vendor_client happens through the module. Global is discouraged because of this footgun. ``` import myproject from myproject import vendor_client @online def fn(...) -> ...: vendor_client.api_call(...) # Error! vendor_client is None myproject.vendor_client.api_call(...) # Correct! ``` ### Cleaning up If there is clean up work you need to do when Chalk suspends your runtime, you can specify that work in a function decorated with @after_all. ``` from chalk.features import after_all @after_all def tear_down(): ... ``` As with before_all, you can optionally supply a filter on the environment: ``` @after_all(environment=["**production**"]) def tear_down(): ... @after_all(environment=["**staging**"]) def tear_down(): ... ``` # Datasource Management API source: https://docs.chalk.ai/docs/datasource-api ## Programmatically manage data sources using the Python SDK The Chalk Python SDK provides a programmatic API for managing data sources, allowing you to create, list, update, test, and delete integrations without using the dashboard UI. All data source operations are available through client.api.datasources on a ChalkClient instance. The API communicates with the Chalk API server over gRPC and reuses the same authentication as the query client. Data sources created through this API are environment-level configuration — they persist across deployments and appear in the dashboard, just like sources added through the UI. This is an administrative API intended for infrastructure-as-code scripts, CI/CD pipelines, or notebooks, not for use inside resolver code. ``` from chalk.client import ChalkClient, IntegrationKind client = ChalkClient() ``` ### Listing data sources List all integrations configured in the current environment. ``` sources = client.api.datasources.list() for s in sources: print(f"{s.kind} {s.name} id={s.id}") ``` Returns a list of Datasource objects with id, name, kind, environment_id, created_at, and updated_at fields. ### Creating a data source Use client.api.datasources.create() to add a new integration. The kind parameter accepts either an IntegrationKind enum or a string. Data source names must only contain letters, numbers, and underscores. The created source will immediately appear in the dashboard and be available for use after clicking Redeploy in the dashboard. ### BigQuery ``` import base64 with open("path/to/service-account-key.json") as f: sa_key_b64 = base64.b64encode(f.read().encode()).decode() created = client.api.datasources.create( kind=IntegrationKind.BIGQUERY, name="my_bigquery_source", config={ "BQ_PROJECT": "my-gcp-project", "BQ_DATASET": "my_dataset", "BQ_CREDENTIALS_BASE64": sa_key_b64, }, ) ``` ### Snowflake ``` created = client.api.datasources.create( kind=IntegrationKind.SNOWFLAKE, name="my_snowflake_source", config={ "SNOWFLAKE_USER": "admin", "SNOWFLAKE_PASSWORD": "secret", "SNOWFLAKE_ACCOUNT": "xy12345.us-east-1", "SNOWFLAKE_DATABASE": "PROD_DB", "SNOWFLAKE_WAREHOUSE": "COMPUTE_WH", "SNOWFLAKE_SCHEMA": "PUBLIC", "SNOWFLAKE_ROLE": "ANALYST_ROLE", }, ) ``` ### PostgreSQL ``` created = client.api.datasources.create( kind=IntegrationKind.POSTGRESQL, name="my_postgres_source", config={ "PGHOST": "db.example.com", "PGPORT": "5432", "PGDATABASE": "mydb", "PGUSER": "admin", "PGPASSWORD": "secret", }, ) ``` ### Linking existing secrets Config values can be either literal strings or references to secrets synced from an external cloud secret manager (e.g., AWS Secrets Manager or GCP Secret Manager). Use LinkedSecretRef to point at a secret by its ID instead of inlining the value. ``` from chalk.client import ChalkClient, IntegrationKind, LinkedSecretRef client = ChalkClient() created = client.api.datasources.create( kind=IntegrationKind.SNOWFLAKE, name="my_snowflake_source", config={ "SNOWFLAKE_USER": "admin", "SNOWFLAKE_PRIVATE_KEY_B64": LinkedSecretRef("snowflake-prod-key"), "SNOWFLAKE_ACCOUNT": "xy12345.us-east-1", "SNOWFLAKE_DATABASE": "PROD_DB", "SNOWFLAKE_WAREHOUSE": "COMPUTE_WH", "SNOWFLAKE_SCHEMA": "PUBLIC", "SNOWFLAKE_ROLE": "ANALYST_ROLE", }, ) ``` LinkedSecretRef works with update() as well — pass it for any config key you want to reference from your cloud secret manager rather than providing a literal value. A linked secret may be referenced either by its provider-local secret name, or by a same-store full identifier: an AWS Secrets Manager ARN in the same account and region, a GCP Secret Manager resource name (projects/.../secrets/...) in the same project and region, or an Azure Key Vault secret URL (https://.vault.azure.net/secrets/...) in the same vault. Chalk normalizes full identifiers to the provider-local secret name and always reads the latest version. Cross-account, cross-region, cross-project, and cross-vault identifiers are not supported for injection, and explicit non-latest versions are rejected. ### Getting, testing, updating, and deleting data sources Fetch a data source by ID, test its connectivity, update its configuration, or remove it. ``` # Get a data source by ID source = client.api.datasources.get(id=created.id) # Test connectivity result = client.api.datasources.test(id=created.id) print(f"Success: {result.success}, Message: {result.message}") # Update config or name (only provided keys are changed) updated = client.api.datasources.update( id=created.id, config={"BQ_DATASET": "new_dataset"}, name="my_updated_source", ) # Delete client.api.datasources.delete(id=created.id) ``` ### Supported integration kinds The IntegrationKind enum supports the following data source types: | Kind | Value | | ---------------------------- | -------------------- | | `IntegrationKind.ATHENA` | AWS Athena | | `IntegrationKind.BIGQUERY` | Google BigQuery | | `IntegrationKind.CLICKHOUSE` | ClickHouse | | `IntegrationKind.DATABRICKS` | Databricks | | `IntegrationKind.DYNAMODB` | Amazon DynamoDB | | `IntegrationKind.KAFKA` | Apache Kafka | | `IntegrationKind.KINESIS` | Amazon Kinesis | | `IntegrationKind.MSSQL` | Microsoft SQL Server | | `IntegrationKind.MYSQL` | MySQL | | `IntegrationKind.POSTGRESQL` | PostgreSQL | | `IntegrationKind.PUBSUB` | Google Pub/Sub | | `IntegrationKind.REDSHIFT` | Amazon Redshift | | `IntegrationKind.SNOWFLAKE` | Snowflake | | `IntegrationKind.SPANNER` | Google Spanner | | `IntegrationKind.TRINO` | Trino | # Test Integrations Locally source: https://docs.chalk.ai/docs/local-development ## Execute resolvers that connect to external systems ### Introduction Many resolvers generate features by connecting to external, production-like systems (e.g. databases and APIs). It can be challenging to validate these resolvers since they involve communication between several systems. This guide will help you execute resolvers that communicate with those external systems on your local environment. ### Setup Environment Variables Chalk's datasource integrations operate by assuming certain environment variables exist which provide integration information (such as URLs, passwords, etc.) to the Python code. When executing these resolvers locally, you'll need to replicate those environment variables. For the following guide let's assume a single Postgres datasource named "PG". - The first env var lists all your datasources by name. ``` _CHALK_AVAILABLE_INTEGRATIONS=["PG"] ``` - Then you should set environment variables for the datasource. These environment variables should be prefixed with the datasource name. You should have an env var for each field. ``` PG_PGHOST=database-1.myproject.us-west-1.rds.amazon.com PG_PGPORT=5432 PG_PGDATABASE=db PG_PGUSER=postgres PG_PGPASSWORD=password ``` ### Exercise your resolvers Once the environment variables are setup correctly, you can execute your resolver like any Python code. For example if the file chalkproject/resolvers.py includes a resolver like get_transactions_from_snowflake, you can execute it in any Python environment. ``` $ source .venv/bin/activate $ (.venv) python >>> from chalkproject.resolvers import get_transactions_from_snowflake >>> get_transactions_from_snowflake() ``` # Secrets and Engine Configuration Variables source: https://docs.chalk.ai/docs/env-vars ## Manage secrets and engine configuration variables for your Chalk deployments. Chalk provides two distinct settings mechanisms that inject key-value pairs into your deployments that are designed for two distinct purposes, secrets and engine configuration variables: - Secrets are designed to inject values into your resolver runtime - the runtime environment that is exposed to your Python and SQL resolvers when they run in Chalk's system. These values are stored encrypted in your own cloud account's Key Management Service. This is the correct tool for providing passwords, API keys, tokens, connection strings, and certificates to your resolvers. - Engine configuration variables are designed to inject values into the Chalk execution engine runtime. These variables are an escape hatch designed for detailed and advanced tuning of the operation of the execution engine, including threadpool sizes, advanced query planner behavior, and other behavioral changes, and are not intended for the purpose of injecting values into your resolver runtime. Secrets are injected into the resolver runtime and read the same way in your resolver or in a setup hook: ``` import os value = os.getenv("MY_VARIABLE") ``` ### Secrets Use a secret for any value you would not want to check into source control — SASL passwords, AWS/GCP keys, third-party API tokens, database connection strings, or SSL material. Secrets are stored in your cloud secret manager and injected into your deployment containers, where they are made available to resolvers via os.getenv (as Python environment variables). ### Managing secrets in the dashboard Under Integrations > Secrets, you can view, add, edit, and delete secrets. When creating or editing a secret, provide the name and value and select the environments it should apply to. This is the recommended way to set a secret whose value you do not want to type on the command line. ### Managing secrets with the CLI The chalk secret command lets you upsert, inspect, and delete secrets. ### Setting a secret chalk secret set upserts one or more secrets. There are three ways to provide values: ``` # Interactive mode $ chalk secret set # From stdin $ cat key.pem | chalk secret set TLS_CERT $ base64 -i chalk.p12 | chalk secret set PKCS12_CERT # Key-value pairs $ chalk secret set MY_SECRET_KEY=s3cr3t $ chalk secret set MY_OTHER_SECRET_KEY=5ecret PORT=9000 ``` ### Inspecting and deleting secrets ``` # List the secrets available in this environment $ chalk secret list # Get a single secret by name $ chalk secret get --name CHALK_KAFKA_SASL_USERNAME # Delete one or more secrets $ chalk secret delete --name CHALK_KAFKA_SASL_PASSWORD ``` Changing a secret is the same operation as creating one — re-run chalk secret set (or edit it on the Secrets page) with the new value. ### Engine Configuration Variables Use engine configuration variables to inject configuration that adjusts how your services behave — feature flags, tuning knobs, and other runtime configuration. They can be applied at two scopes, plus the CLI. Engine configuration variables are used to configure the Chalk engine's own runtime. Any value you need to read from your resolvers — and especially anything sensitive — should be set as a Secret instead. ### Environment-wide (Config variables page) Under Integrations > Config variables, you can view and edit the global engine configuration variables for your environment. Values set here apply across the whole environment and map to the environment's additional_env_vars. ### Per-service (Resource page) To scope an engine configuration variable to a specific service (for example, only the engine-grpc or streaming server in a particular resource group), set it on the Resource page for that service under Advanced > Engine Config Variable Overrides. These values apply only to the targeted service rather than the whole environment. ### With the CLI You can read more about programmatic management of Chalk configuration variables under Programmatic Management ### Chalk environment variables Chalk also injects some special environment variables into the resolver runtime for use in your resolvers: | Name | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------- | | `TARGET_ROOT` | The root directory of the application. This variable is set for both branch and standard deployments. | | `CHALK_DEPLOYMENT_ID` | The ID of the deployment | | `CHALK_TEAM_ID` | The ID of the team | | `CHALK_PROJECT_ID` | The ID of the project | | `CHALK_ACTIVE_ENVIRONMENT` | The ID of the active environment (e.g. `"9d0oj902"`) | | `CHALK_ENVIRONMENT_NAME` | The name of the active environment (e.g. `"prod"`) | # Unit Tests source: https://docs.chalk.ai/docs/unit-tests ## Unit tests for Chalk resolvers Chalk lets you specify your feature pipelines using idiomatic Python. You can unit test individual resolvers and combinations of resolvers as you would normal Python functions. Read more about how to write integration tests for resolvers here. ### Testing resolvers Consider the following features and resolvers: ``` from chalk.features import features, Features from chalk import online @features class Home: id: str address: str price: int sq_ft: int @online def get_address(hid: Home.id) -> Home.address: return "Bridge Street" if hid == 1 else "Filbert Street" @online def get_home_data( hid: Home.id, ) -> Features[Home.price, Home.sq_ft]: return Home( price=200_000, sq_ft=2_000, ) ``` You can test them just like normal Python functions using any unit testing framework: ``` def test_single_output(self): assert get_address(2) == "Filbert Street" def test_multiple_output(self): result = get_home_data(2) assert result.price == 200_000 assert result.sq_ft == 2_000 assert result == Home( price=200_000, sq_ft=2_000, ) ``` ### DataFrame inputs If you specify projections or filters in DataFrame arguments of resolvers, Chalk will automatically project out columns and filter rows in the input data. ### Filters Consider if we extend the Home class to include a rooms field: ``` + @features + class Room: + id: str + home_id: "Home.id" + name: str @features class Home: id: str address: str price: int sq_ft: int + rooms: DataFrame[Room] + num_bedrooms: int + + @online + def get_num_bedrooms( + rooms: Home.rooms[Room.name == 'bedroom'] + ) -> Home.num_bedrooms: + return len(rooms) ``` The get_num_bedrooms resolver filters the rooms argument to only include bedrooms. We can test this by passing in a list of rooms, some of which are bedrooms and some of which are not: ``` def test_get_num_rooms(): # Rooms is automatically converted to a `DataFrame` rooms = [ Room(id=1, name="bedroom"), Room(id=2, name="kitchen"), Room(id=3, name="bedroom"), ] # The kitchen room is filtered out assert get_num_bedrooms(rooms) == 2 # `get_num_bedrooms` also works with a `DataFrame` assert get_num_bedrooms(DataFrame(rooms)) == 2 ``` Note that although we passed in a list of three rooms, only two of them were bedrooms, so the resolver returns 2. Furthermore, we didn't need to convert the list of rooms to a DataFrame. In this case, we passed in a list of Room objects, but Chalk automatically converts it to a DataFrame for us. We also could have passed in a polars.DataFrame or any other valid constructor for DataFrame. ### Projections Chalk will also automatically project the input data as specified by the DataFrame argument. So, if you specify which columns you need in the resolver body, but accidentally use an extra column in your resolver body, your unit tests will fail. For example, if we wrote a resolver for summing the square footage of all rooms in a home using the Room.sq_ft feature, but accidentally excluded the Room.sq_ft column in the DataFrame argument: ``` @online def get_sqft( rooms: Home.rooms[Room.name] ) -> Home.sq_ft: # This will fail because we used `Room.sq_ft` return rooms[Room.sq_ft].sum() ``` Then our unit tests will fail: ``` def test_get_sqft(): rooms = [ Room(id=1, name="bedroom", sq_ft=100), Room(id=2, name="kitchen", sq_ft=200), Room(id=3, name="bedroom", sq_ft=300), ] # This will fail because we used `Room.sq_ft`, # which is excluded by the `Home.rooms[Room.name]` # argument assert get_sqft(rooms) == 400 ``` ### after(...)/before(...) and time filtering Some DataFrame filters are implicitly resolved relative to "now". In offline_query and online_query, this value is controlled by input_times= and now= parameters, respectively. In unit tests, the value defaults to datetime.now(), but can be explicitly set with the chalk.freeze_time context manager: ``` from chalk import freeze_time @online def get_p30d_transactions_count(txns: User.transactions[after(days_ago=30)]) -> User.recent_txn_count_30d: """ Count the number of transactions that have occurred in the past 30 days """ return txns.count() NOW = datetime.now(tz=timezone.utc) transactions = DataFrame([ Transaction(id=1, created_at=NOW), Transaction(id=2, created_at=NOW - timedelta(days_ago=30)), Transaction(id=3, created_at=NOW - timedelta(days_ago=32)) ]) # There is a single transaction in the range (30, 0] days ago assert get_p30d_transactions_count(transactions) == 1 # There are two transactions in the range (45, 15] days ago with freeze_time(at=NOW - timedelta(days_ago=15)): assert get_p30d_transactions_count(transactions) == 2 ``` Note that the at= parameter must be timezone-aware. # Integration Tests source: https://docs.chalk.ai/docs/integration-tests ## Integration tests for Chalk resolvers A Chalk resolver is a callable Python function that you can unit test like any other Python function, making assertions on the expected output. Chalk also provides two features that make integration testing really easy. - branch deployments, - the check method of the Python Chalk client. With branch deployments, you can test your changes in an isolated environment before shipping your code into production. With Chalk check, you can set up simple integration tests that assert on the expected outputs of your queries. ### Branch deployments Chalk allows you to create an unlimited number of branch deployments. Branch deployments run all of your resolvers in the same way that they run in production. A branch can change features and resolvers, except stream resolvers. Everything else, such as secrets and data sources, comes from your mainline deployment, and a branch does not write feature values to the online or offline store. See Branch limitations for the full list. ### Creating a branch deployment You can create a branch deployment in the same way that you create a full deployment by passing the flag --branch . ``` # --branch creates a branch deployment > chalk apply --force --await --branch ``` The --await flag means that the deployment will be live by the time that the command returns. ### Querying a branch deployment from CLI You can quickly check your branch deployment using chalk query --branch on the command line to pull feature values: ``` # Example of making a query directly > chalk query --branch \ --in user.id=1 \ --out user.id \ --out user.email ``` The flag --branch tells this query to target the branch set during chalk apply. ### Querying a branch deployment using ChalkClient You can also target a branch from the Chalk API client. Using this client, you can write integration tests: ``` import pytest from chalk.client import ChalkClient @pytest.fixture(scope="module") def chalk_client(): client = ChalkClient(branch=True) return client def test_get_features(chalk_client): resp = chalk_client.query( input={User.id: 1}, output=[ User.credit_report.fico_score, User.email, User.name, User.dob, ], ) assert resp.get_feature_value(User.credit_report.fico_score) == 700 assert resp.get_feature_value(User.email) == "katherine.johnson@nasa.gov" ``` Branch deployments are a great way to quickly test whether the changes you have made to features or resolvers are behaving as expected when they are composed and executed in response to Chalk queries. However, if you want to write concrete integration tests, we recommend using the check method of the Chalk Python client. ### Writing integration tests using ChalkClient.check() Suppose you have the following features defined in Chalk: ``` from chalk.features import DataFrame, features, FeatureTime, _ from chalk.streams import Windowed, windowed @features class Transaction: id: int user_id: "User.id" amount: float ts: FeatureTime @features class User: id: int name: str transactions: DataFrame[Transaction] transaction_count: Windowed[int] = windowed( "1d", "3d", expression=_.transactions[_.ts > _.chalk_window].count(), ) transaction_mean: Windowed[float] = windowed( "1d", "3d", expression=_.transactions[_.amount, _.ts > _.chalk_window].mean(), ) ``` You can write integration tests with the ChalkClient by leveraging the check method. The check method allows assertion on values, errors, and cache hits. Mismatches between expected and resolved data will be printed in a table. Note that the service token being used by the ChalkClient must have permission to Query online features to be able to use the check method. ``` from chalk.client import ChalkClient from chalk.features import DataFrame from src.feature_sets import Transaction, User import datetime as dt import pytest @pytest.fixture(scope="session") def client(): return ChalkClient(branch=True, local=True) # deploys your local changes to a branch named after your current git branch def test_transaction_aggregations(client): now = dt.datetime.now() result = client.check( input={ User.id: 1, User.transactions: [ Transaction(id=1, amount=10, ts=now - dt.timedelta(days=1)), Transaction(id=2, amount=20, ts=now - dt.timedelta(days=2)), Transaction(id=3, amount=30, ts=now - dt.timedelta(days=3)), Transaction(id=4, amount=40, ts=now - dt.timedelta(days=4)), Transaction(id=5, amount=50, ts=now - dt.timedelta(days=5)), Transaction(id=6, amount=60, ts=now - dt.timedelta(days=6)), Transaction(id=7, amount=70, ts=now - dt.timedelta(days=7)), Transaction(id=8, amount=80, ts=now - dt.timedelta(days=8)), Transaction(id=9, amount=90, ts=now - dt.timedelta(days=9)), ] }, assertions={ User.transaction_count["3d"]: 60, User.transaction_count["1d"]: 10, User.transaction_mean["1d"]: 10, User.transaction_mean["3d"]: 21, # This test will fail because this value is incorrect! } ) ``` You can then run pytest -s tests/test_transaction_aggregations.py, where you will see a result like this: ``` $ pytest tests/test_transaction_aggs.py -s ==================================================== test session starts ===================================================================== platform darwin -- Python 3.11.14, pytest-8.3.3, pluggy-1.5.0 rootdir: /Users/chalk/deployment/ configfile: pyproject.toml plugins: anyio-4.4.0 collected 1 item tests/test_transaction_aggs.py Chalk Feature Value Check Table ┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓ ┃ Kind ┃ Name ┃ Value ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩ │ Match │ user.transaction_count[1d] │ 1 │ │ Match │ user.transaction_count[3d] │ 3 │ │ Match │ user.transaction_mean[1d] │ 10.0 │ │ Expect │ user.transaction_mean[3d] │ 21.0 │ │ Actual │ user.transaction_mean[3d] │ 20.0 │ └────────┴────────────────────────────┴───────┘ {'user.transaction_count__259200__': 3, 'user.transaction_count__86400__': 1, 'user.transaction_mean__86400__': 10.0, 'user.transaction_mean__259200__': 20.0} F ``` ### Testing for errors You can also use the check method to test for errors. This might look like the following: ``` from chalk.client import ChalkClient from chalk.features import DataFrame from chalk.client import ChalkClient, ErrorCode, ErrorCodeCategory, ChalkError from src.feature_sets import Transaction, User import pytest @pytest.fixture(scope="session") def client(): return ChalkClient(branch=True, local=True) # deploys your local changes to a branch named after your current git branch def test_user_null_error(client): result = client.check( input={ User.id: 1, User.name: None, }, query_errors=[ ChalkError( code=ErrorCode.PARSE_FAILED, category=ErrorCodeCategory.REQUEST, message="Received a null input for feature 'user.name', which is non-nullable (with type 'str').", ) ], assertions={ User.name: ... }, ) ``` ### Integration testing using tagged resolvers In addition to the check method, you can also use tagged resolvers to test changes to resolvers. If you have a resolver that you want to update and then test, you could add resolver tags to two versions of the resolver -- one tagged "v1.0.0" and one tagged "v1.0.1", for example. Then, you can run two queries with the same input data but the different tags and compare the query outputs to ensure that the new resolver is working as expected. ### GitHub Action You can also run integration tests as part of your CI/CD pipeline using the Chalk GitHub Action. You will need to create a Chalk token from the settings page of your dashboard with the necessary permissions and store the resulting client ID and secret as GitHub Secrets. ``` name: Chalk Integration Test on: push jobs: chalk-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Python uses: actions/setup-python@v4 with: python-version: '3.11' cache: 'pip' - name: Install dependencies run: pip install -r requirements.txt - uses: chalk-ai/deploy-action@v2 with: client-id: ${{secrets.CHALK_CLIENT_ID}} client-secret: ${{secrets.CHALK_CLIENT_SECRET}} # Deploys Chalk to a branch environment branch: ${{ GITHUB_REF_NAME }} # Waits for the deployment to succeed (Optional, default false) await: true - name: Runs a test query against the branch run: | # Example of making a query directly against a branch chalk query --branch ${{ GITHUB_REF_NAME }} \ --in user.id=1 \ --out user.id \ --out user.email \ --json \ --include-meta # Alternatively, run the integration test suite, which should make queries with the branch parameter. pytest -s ./tests ``` # Snapshot Tests source: https://docs.chalk.ai/docs/snapshot-tests ## Snapshot testing with Chalk Snapshot testing is an effective way of ensuring that your features are not dramatically changing between deployments. You can set up snapshot testing in Chalk by leveraging Datasets. Typically, snapshot testing will be part of a CI (continuous integration) workflow. Pull requests will automatically trigger tests which verify that code changes have not caused feature distribution shifts. In this section, we'll walk through the process of creating snapshot tests using Pytest and GitHub Actions. ### Overview To set up snapshot testing, we'll write a couple Pytest fixtures, a test, and a helper script. The fixtures will pull your current snapshot data and create a new snapshot based on your current code. The test will then compare the distributions of the features in your old and new datasets. The helper script will update the main snapshot and will be run when a PR is merged. In this example we'll be snapshot testing the features on the following feature class: ``` from chalk.features import features import datetime as dt @features class User: id: int full_name: str birthday: dt.datetime age: int ``` In this example, we resolve the full_name and birthday of our Users from a Postgres database. The age feature is calculated using an online resolver: ``` from chalk import online from chalk import Now @online def get_age_in_years(bday: User.birthday, now: Now) -> User.age: return now.year - bday.year - ((now.month, now.day) < (bday.month, bday.day)) ``` Though this is a rather trivial example, let's set some expectations for our feature distribution shifts between snapshots: - We expect a User's birthday to never change, - We expect a User's full_name to change in a maximum of 0.001% of our users (1/100,000), - We expect a User's age to change by a maximum amount of 2 years (in reality, average age will increase by precisely the amount of time between invocations of our snapshot test). ### Writing the Test and Helper Script First we'll write two fixtures: the first sets up a Chalk Client and the second creates your new snapshot (using the previous snapshot to make sure that the exact same User IDs are being pulled). ``` from chalk.client import ChalkClient from chalk.features import DataFrame from src.feature_sets import Transaction, User import datetime as dt import pytest import pandas as pd @pytest.fixture(scope="session") def client(): return ChalkClient(branch=True) # Uses your current git branch @pytest.fixture(scope="session") def snapshots(client: ChalkClient): # The dataset name will be set to the latest commit on the branch main_snapshot = client.get_dataset(dataset_name="user_snapshot") main_snapshot_df = snapshot.to_pandas().set_index(User.id) branch_snapshot = client.offline_query( input={User.id: main_snapshot_df.index.to_list()}, recompute_features=True, wait=True, run_asynchronously=True ) branch_snapshot_df = branch_snapshot.to_pandas().set_index(User.id) return active_snapshot_df, branch_snapshot_df ``` These fixtures serve to output two Pandas DataFrames: the main_snapshot_df (which is stored in a dataset called user_snapshot) and the branch_snapshot_df (which is generated on the fly with an offline query). Next, we will use those snapshots in a test: ``` # Note, we skip this test locally since we only want to run it in CI @pytest.mark.skipif(os.getenv("CI") is None, reason="Skipping on local machine") def test_snapshot__user_feature_drift(snapshot_dfs: tuple[pd.DataFrame, pd.DataFrame]): main_snapshot, branch_snapshot = snapshot_dfs # Birthdays should not change assert main_snapshot[User.birthday].equals(branch_snapshot[User.birthday]) # Full names should not change in more than 0.001% of users assert (main_snapshot[User.full_name] != branch_snapshot[User.full_name]).mean() < 0.00001 # Age should not change by more than 2 years between runs assert (main_snapshot[User.age] - branch_snapshot[User.age]).max() <= 2 ``` In addition to the tests, we'll create a script which will run when we merge pull requests — this script will update the user_snapshot dataset with the one generated with our latest code. ``` import os from chalk.client import ChalkClient branch = os.environ["GITHUB_REF_NAME"] client = ChalkClient(branch=branch) dataset = client.get_dataset(dataset_name="user_snapshot") dataset.recompute_features(branch=branch, wait=True, run_asynchronously=True) ``` ### Writing the GitHub Action We will write two GitHub actions: one will run when we create a pull request, the other will run when we merge code into our main branch. The first GitHub Action sets up the environment and then runs pytest. The GitHub Action is set to run on pull, which means it will run when we open a pull request against main: ``` name: Chalk Integration Test on: pull_request: branches: [main] jobs: chalk-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Python uses: actions/setup-python@v4 with: python-version: '3.11' cache: 'pip' - name: Install dependencies run: pip install -r requirements.txt - uses: chalk-ai/deploy-action@v2 with: client-id: ${{secrets.CHALK_CLIENT_ID}} client-secret: ${{secrets.CHALK_CLIENT_SECRET}} # Deploys Chalk to a branch environment branch: ${{ GITHUB_REF_NAME }} # Waits for the deployment to succeed (Optional, default false) await: true - name: Runs the Snapshot Test run: | pytest -s ./tests ``` The second GitHub Action runs our update script to update the user_snapshot dataset with our latest snapshot. It runs whenever we push code to the main branch. ``` on: push: branches: [main] jobs: update-snapshot: steps: - uses: actions/checkout@v4 - name: Setup Python uses: actions/setup-python@v4 with: python-version: '3.11' cache: 'pip' - name: Install dependencies run: pip install -r requirements.txt - uses: chalk-ai/cli-action@main with: client-id: ${{ secrets.CHALK_CLIENT_ID }} client-secret: ${{ secrets.CHALK_CLIENT_SECRET }} environment: ${{ secrets.CHALK_ENVIRONMENT }} - name: Update snapshot run: python3 scripts/update_user_snapshot.py ``` ### Bootstrapping the GitHub Action Before this works fully, we need to bootstrap the snapshot test. If you tried to run the action above, you'd get an error when running client.get_dataset(...): the error would indicate that the dataset was not found. To bootstrap your snapshot, you'll want to run an offline_query to create a dataset named user_snapshot. ``` from src.feature_sets import User client.offline_query( output=[User], # This can be however many you want to include in your snapshot max_samples=200_000, recompute_features=True, run_asynchronously=True, dataset_name="user_snapshot" ) ``` With your initial snapshot created, future pull requests will be able to automatically execute a snapshot test against your main deployment. # Time Overview source: https://docs.chalk.ai/docs/time ## Manage feature timestamps and point-in-time query timestamps. ### Overview Chalk uses two main timestamps you should be aware of as you build your Chalk project: - Feature time (FeatureTime): The time at which a feature was observed, which is used in query time filters and aggregation time windows. - Query time (Now): The time a query should assume is "now" when retrieving data. Features whose feature time comes after a given query time will never be returned for those queries, in online or offline contexts. Feature time is returned in the __observed_at__ column in Chalk query results. Query time is returned in the __ts__ column. ### Feature time Feature time is the time at which a feature was observed. By default, Chalk sets a feature's time to the feature's resolver execution time. The feature time can be overridden for a feature class, accessed from resolver parameters, and requested in query inputs and outputs. You may have multiple timestamps associated with your data. It's important to set the feature time to the value that most closely represents when your system would have accessed the data in production. For example, in an asynchronous streaming system, you may have one timestamp for when an event was added to your task queue and another timestamp for when the event was removed from the queue and processed. We recommend using the latter timestamp as your feature time to make your training most closely resemble production. If you use the timestamp for when an event was added to your task queue, you would be training your system with data it would not have been able to access in production. ### Setting feature time for a feature class Each individual feature has its own feature time, which is used to retrieve point-in-time correct data for temporal consistency. To access the latest time associated with any feature in a feature class, use the special FeatureTime feature throughout Chalk. By default, if it is present, Chalk will treat a feature named ts with datetime.datetime type as the FeatureTime value. Otherwise, you can use the FeatureTime type annotation to set a different name: ``` from chalk.features import features, FeatureTime @features class User: id: int name: str timestamp: FeatureTime # datetime.datetime under the hood ``` Using your FeatureTime feature, you can access and override feature time for the whole feature class. You can directly set the FeatureTime value by returning it from a resolver: ``` @offline def fn(...) -> Features[User.name, User.timestamp]: return User( name="Maryam Mirzakhani", ts=datetime(2014, 8, 12, tzinfo=timezone.utc) ) ``` You can include the FeatureTime feature in query output. Its value will be set to the maximum timestamp across all features in its feature class. ### Time filtering in resolvers has-many features create DataFrames. These DataFrames can be filtered with before and after. Regardless of which time filters you use, Chalk will never return features where the feature time is strictly greater than the current query time (Now), in order to maintain temporal consistency. ### Examples To compute the number of transfers a user made in the last seven days, use after: ``` from chalk.features import after, ... @online def fn(transfers: User.transfers[after(days_ago=7)]) -> ...: return transfers.count() ``` To compute the number of transfers a user made more than seven days ago, use before: ``` from chalk.features import before, ... @online def fn(transfers: User.transfers[before(days_ago=7)]) -> ...: return transfers.count() ``` Combine before and after to retrieve transfers made 1-2 weeks ago: ``` from chalk.features import before, after, ... @online def fn(transfers: User.transfers[after(days_ago=14), before(days_ago=7)]) -> ...: return transfers.count() ``` All of these examples can be used in combination with other DataFrame projections and filters. You may also find windowed aggregations useful. ### Interaction with the online store Features with overridden observation timestamps are treated specially when inserted into the online store. In particular, Chalk will always check for existing "newer" feature values in the online store before inserting historically dated feature values. This means that you can safely ingest large quantities of backdated features without accidentally ingesting stale data into the online store. Additionally, once features are inserted into the online store, Chalk tracks the source observation timestamps when these feature values are returned as part of online queries. Chalk uses these source timestamps to compute the "feature staleness" metric. Staleness in this context is defined as "query time - observation time". ### Interaction with the offline store Features with overridden observation timestamps are inserted into the offline store with the timestamp that you specify. The observation timestamp works like an "effective as of" timestamp, so if you insert something like this: ``` | id | feature | value | timestamp | |---------------------------------------------| | 1 | age | 7 | 2022-02-01T00:00:00Z | ``` into an offline store that already contained these observations: ``` | id | feature | value | timestamp | |---------------------------------------------| | 1 | age | 6 | 2022-01-01T00:00:00Z | | 1 | age | 8 | 2022-03-01T00:00:00Z | ``` then the observation will be interleaved "in between" the existing observations, and you would see the following query results: ``` id, age, <= 2022-02-01 output: 7 id, age, <= 2022-03-02 output: 8 id, age, <= 2022-01-02 output: 6 ``` ### Enforcing a TTL Features in the offline store have an optional TTL (time to live). When a feature has a TTL value, it will never be returned at any time later than FeatureTime + TTL. For example, you may not want to consider credit scores which were retrieved more than a year ago. Setting offline_ttl will make credit_score return None if the last observed credit score is more than one year old in comparison to the current query time. ``` @features class User: id: str credit_score: int = feature(offline_ttl=timedelta(days=365)) ``` ### Query time Query time is the time treated as "now" within a query context. For online queries, Now is equal to datetime.now(). For offline queries, you can pass one or more timestamps that will be used as the query time for each input row. In Chalk Expressions, you can reference the query time with _.chalk_now. ### Setting query time In training, you will likely want to retrieve data as if you are at a point in the past to create the most accurate predictions. We cover this idea in greater detail in our temporal consistency documentation. To set the query's "now" time, pass input_times as either a single timestamp or as a list corresponding to the Now times to use for each entry in input: ``` from datetime import timezone ChalkClient().offline_query( # Pass id 1 multiple times because we want to # request it with multiple input_times input={User.id: [1, 1, 1]}, input_times=[ datetime.now(tz=timezone.utc) - timedelta(days=365 * 10), datetime.now(tz=timezone.utc) - timedelta(days=365), datetime.now(tz=timezone.utc) - timedelta(days=0), ], output=[User.age_in_years], ) ## Output: # | id | age_in_years | # | 1 | - 10 | # | 1 | - 1 | # | 1 | - 0 | ``` ### Accessing query time in resolvers To access the query time in your resolvers, you can reference a special feature called Now, which is a datetime.datetime object. You can pass Now in Python resolvers: ``` from chalk import Now @online def get_age_in_years(birthday: User.birthday, now: Now) -> User.age_in_years: return (now - birthday).years ``` Now can be used in DataFrame resolvers as well in order to compute bulk values: ``` from chalk import DataFrame, Index, Now @online def batch_get_age_in_years( df: DataFrame[Index, User.id, User.birthday, Now], ) -> DataFrame[Index, User.id, User.age_in_years]: return ( df.to_polars() .select( pl.col(str(Index)), pl.col(User.id), pl.col(str(User.birthday) - pl.col(str(Now))).alias(str(User.age_in_years)) ) ) ``` Now supplies the point-in-time value for each computation, while Index identifies the input row that produced each output. See DataFrame-to-DataFrame resolvers for guidance on preserving Index through transformations. You can also reference ${now} in SQL file resolvers. ``` -- source: sql_file_resolver_temp_db -- resolves: tv_episode select id, name, season_no, episode_no, show_name, air_date from tv_episodes where air_date < ${now} and id = ${tv_episode.id} ``` You can also reference _.chalk_now in Chalk Expressions, particularly in windowed aggregations: ``` from chalk.features import features, _ from chalk.streams import windowed, Windowed @features class Store: id: int visits: DataFrame[Visit] num_recent_visits: Windowed[int] = windowed( "1d", "7d", "30d", expression=_.visits[ _.ts < _.chalk_now, _.ts > _.chalk_window ].count(), default=0, ) ``` ### Accessing query time in query results Chalk Datasets return the query time in the __ts__ column. When converting Chalk Datasets to Polars or Pandas DataFrames, you may want to include the query time column. To do so, pass output_ts to your to_polars or to_pandas calls. You may pass a column name to output_ts to set the name of the query time column. If you pass True, the query time column name will be __chalk__.CHALK_TS. Be careful to not mix up ts and ts. ts represents the query time, or the time the query treats as "now" during query execution. ts is a common name for the feature representing FeatureTime, the time at which a feature was observed. ### Timezone handling for naive datetimes Chalk stores UTC as the timezone for naive datetime objects. Additionally, Chalk assumes UTC if retrieving naive datetimes from data stores. We recommend that you include timezone information on all datetime objects you work with to avoid ambiguity. ### When to use FeatureTime vs Now Chalk enables different levels of temporal consistency, which also have different impacts on performance. When a resolver does not have a FeatureTime or Now as an input or output, there is no temporal pushdown filtering, so the resolver will be fully time independent, which will also be more performant. If the resolver has a FeatureTime as an input, then Chalk will apply a temporal pushdown filter. Then, if a resolver takes Now as an input, the resolvers can fully interact with time, which will provide the most accuracy for point-in-time feature resolution but also be the least performant. # Temporal Consistency source: https://docs.chalk.ai/docs/temporal-consistency ## Point-in-time queries for training data sets. ### Introduction Temporal consistency is crucial for ensuring your model's training performance is reflective of production. When creating datasets for model training, your system must consistently retrieve data as it would appear at a specific point in time. If your model trains on data that was retrieved after the point where it would have made a prediction, it will not perform consistently in production. Chalk performs point-in-time lookups on your training data so you can train your models knowing they won't receive data from the future, even across complex relationships. ### Scenario 1: Historical loan decisions In this scenario, we will consider a machine learning model for determining whether to issue loans to businesses based on their financial performance. For each business, we track sales and COGS (cost of goods sold) over time in millions of dollars. We define a Business feature class with sales and cogs features: ``` from chalk.features import features, FeatureTime @features class Business: id: int sales: float cogs: float ts: FeatureTime ``` After collecting historical data on the performance of businesses we previously considered, we want to retrain our model. For a business with id=123, we gave loans at t1 and t2: In training, you want to know the observed gross profit and COGS for the business at the time you made the loan without knowing the future values of those features. For example, at t1, when we issued the loan, we had observed sales of $1.3M. It would be inconsistent for this model to train with the $1M data point because this data would not have existed at t1. At t1 and t2, the temporally consistent values of sales and cogs would be as follows: | Feature | Value at `t1` | Value at `t2` | | -------------- | ------------- | ------------- | | Business.sales | 1.3 | 1 | | Business.cogs | 0.5 | 0.4 | Each of these values occurred at or before the sample time and is valid to use in training. ### Sample code Chalk allows you to control the time your query considers to be "now", which is covered in greater detail in our time documentation. To set the query's "now" time, pass input_times to your offline queries: ``` from chalk.client import ChalkClient t1 = datetime.now() - timedelta(days=365) t2 = datetime.now() - timedelta(days=30) dataset = ChalkClient().offline_query( input={ # Sample business 123 twice because we have two input_times Business.id: [123, 123], }, # Each element of `input_times` corresponds to the element # with the same index in `input` input_times=[t1, t2], # Sample all features of business. # Alternatively, sample only the features you need: # output=[Business.sales, Business.cogs] output=[Business], ) ``` This query will return a Dataset with temporally consistent values for each input time. ### Scenario 2: Backfilling new features Temporal consistency is especially difficult when you want to build new features. Building on the first scenario, imagine you've observed Business.sales and Business.cogs many times in the past, and for each of the businesses that you track, these values have changed over time. Now, you want to compute a new feature, Business.gross_profit, which is the difference between Business.sales and Business.cogs. We can add an expression to compute this new feature: ``` - from chalk.features import features, FeatureTime + from chalk.features import _, features, FeatureTime @features class Business: id: int sales: float cogs: float ts: FeatureTime + gross_profit: float = _.sales - _.cogs ``` With the same historical data from the previous scenario, we can determine the correct inputs for computing Business.gross_profit: Business.gross_profit at t1 and t2 would be computed with the following values: | Query time | `Business.sales` | `Business.cogs` | `Business.gross_profit` | | ---------- | ---------------- | --------------- | ----------------------- | | `t1` | 1.3 | 0.5 | 0.8 | | `t2` | 1.0 | 0.4 | 0.6 | In your code, you can compute historical values for updated features by running an offline query with recompute_features=True. You may also consider one of our other backfill options for other use cases. As you start layering more resolvers or using has-many relationships, temporal consistency becomes even more difficult to reason about. Chalk manages the complexity under the hood so that you can write any query and expect temporally consistent results. # Batch Backfilling source: https://docs.chalk.ai/docs/backfilling-data ## Chalk makes it easy to batch ingest historical feature data from datasets and bulk data sources. ### Backfilling from datasets The most common way to backfill features is to ingest them from a dataset. This is useful when you have historical data ready or when you want to compute and backfill new features based on existing data. Run an offline query to obtain a Dataset of the feature values you want to ingest and then call .ingest to store online or offline: ``` from chalk.client import ChalkClient # Query for historical data dataset = ChalkClient().offline_query( input={ User.id: [i for i in range(1000)], }, output=[ User.favorite_color, User.purchase_count, User.last_active ], ) # Ingest into the offline store for training dataset.ingest(store_offline=True) # Or ingest into the online store for serving # dataset.ingest(store_online=True) ``` ### Backfills from bulk sources When ingesting historical feature data from external bulk data sources (such as a data warehouse or S3), you can override feature time with timestamps from the data source. Chalk uses feature time to determine when a feature should be included in your point-in-time queries for consistency with production. First, define a FeatureTime feature for the relevant feature class: ``` @features class User: id: int ... backfilled_feature: str ... ts: FeatureTime ``` Next, define a resolver to ingest data from your data source. Here's an example of a SQL file resolver for ingesting data from Snowflake: ``` -- type: offline -- resolves: User -- source: snowflake SELECT id, backfilled_feature, updated_at AS ts FROM users ``` Chalk assumes your timestamp column is in UTC. You may return many values of backfilled_feature for the same id, but they should have different FeatureTime values. Then, after running chalk apply, you can trigger this resolver to run once using the Chalk Dashboard or the chalk trigger command: ``` chalk trigger --resolver your.module.path.ingest_historical_data ``` You must manually trigger the resolver because it has no cron schedule specified. Once your resolver run completes, your data will be available in the offline store with effective times specified by the values returned for the FeatureTime (in this example, updated_at from users). If the feature tolerates staleness and etl_offline_to_online=True, then Chalk will also insert feature values into the online store if they are newer than existing values. ### Re-ingesting incremental resolvers Resolvers that use incremental ingestion don't re-process data from before their "max observed timestamp" by default, even if the query is changed. Chalk lets you reset the maximum observed timestamp of incremental resolvers to a specific timestamp, or re-ingest all historical data. Chalk uses offline queries to perform this operation. Suppose you want to add a new column, favorite_color, to this existing batch SQL resolver: ``` -- type: offline -- resolves: User -- source: snowflake -- cron: "0 * * * *" -- incremental: -- incremental_column: updated_at SELECT id, favorite_food, favorite_color, updated_at FROM preferences ``` If you have been running this resolver in production for a long time, then simply adding favorite_color and running chalk apply will not ingest historical color preferences because the incremental timestamp lower bound will prevent the query from returning "old" rows which include these historical favorite_color observations. Instead, run an offline query to obtain a Dataset of the feature values you want to ingest and then call .ingest to either store online or offline. ``` from chalk.client import ChalkClient dataset = ChalkClient().offline_query( input={ User.id: [i for i in range(1000)], }, output=[ User.favorite_color ], ) dataset.recompute(features=['user.favorite_color']) dataset.ingest(store_offline=True) ``` ### Backfilling Python resolvers Chalk can also backfill feature data for resolvers that take arguments to compute feature values. This is useful for generating historically accurate training datasets using new resolvers that are derived from features where we have historical observations. Suppose we have the following feature class: ``` @features class User: id: int name: str reversed_name: str # a feature we want to backfill ``` with the following newly added resolver: ``` @online def reverse_name(name: User.name) -> User.reversed_name: return name[::-1] ``` Chalk can automatically compute this feature using historically observed values of User.name. To do so, you can either use an offline query with recompute_features=True and ingest the computed feature values as we did above. Alternatively, you can use chalk trigger to trigger a resolver run: ``` chalk trigger --resolver your.module.path.reverse_name --lower-bound 2024-05-05T12:00:00+00:00 --persist-offline=True ``` In this example, Chalk will query the offline store to sample all combinations of user.id, user.name, and the corresponding FeatureTime for which FeatureTime is later than the ISO8601 timestamp passed in as the lower_bound. For each sample, Chalk will invoke your resolver and write the results back to the offline store. Chalk will also update the online store if the new feature is marked with non-zero max_staleness and etl_offline_to_online=True. # Feature Development Lifecycle source: https://docs.chalk.ai/docs/lifecycle ## Developing, testing, deploying, and backfilling a new feature Chalk is the fastest way to deploy new features and feature pipelines to production. This demo covers creating a new feature. You'll develop, unit test, integration test, and deploy a resolver for this feature. Then you'll see how to backfill that feature to Chalk's offline store so that data scientists can use it in historical training sets. ### Develop The first step is to write a new feature and resolver for that feature. Imagine you wanted to create a new feature called user.name_email_match_score. This feature should capture the similarity between a user's name and email. Put another way: andy and andy@gmail.com should produce a high score, and emily and andy@gmail.com should produce a low score. The first step is to add the new feature, and a resolver for that feature. Our new resolver name_email_match_scorer takes a dependency on the user's email and full name in the argument list. Then, it declares that it returns the User.name_email_match_score in the return type signature. In the body of the function, we compute the Jaccard index between the email without the domain and the full name. ### Test Now that you've written the new feature and resolver, it's time to validate your change. Chalk supports both unit testing and integration testing of your feature pipelines. ### Unit test You can unit test resolvers just like normal Python functions using any unit testing framework: ``` from pytest import approx def test_name_email_match_scorer(): assert approx(0.60) == name_email_match_scorer( "katherine.johnson@nasa.gov", "Katherine Johnson", ) assert approx(0.39) == name_email_match_scorer( "katherine.johnson@nasa.gov", "Eleanor Roosevelt", ) ``` Here, the Jaccard index for katherine.johnson@nasa.gov is higher with the name Katherine Johnson than with the name Eleanor Roosevelt, as expected. ### Deploy branch Now that the unit tests have passed, you can create a Branch Deploy with the new changes. Chalk allows you to create an unlimited number of branch deployments. Branch deployments run all of your resolvers in the same way that they run in production. A branch can change features and resolvers, except stream resolvers. Everything else, such as secrets and data sources, comes from your mainline deployment, and a branch does not write feature values to the online or offline store. See Branch limitations for the full list, and When to use a new branch or a mainline deployment for when to move on to the Deploy step. You can create a branch deployment with the following command: ``` chalk apply --branch test ``` ### Integration test With the branch deployment created, you can run integration tests on your changes. ``` import pytest from chalk.client import ChalkClient @pytest.fixture(scope="module") def chalk_client(): client = ChalkClient(branch="test") return client def test_get_features(chalk_client): resp = chalk_client.query( input={User.id: 1}, output=[ User.credit_report.fico_score, User.email, User.name, User.dob, ], ) assert resp.get_feature_value(User.credit_report.fico_score) == 700 assert resp.get_feature_value(User.email) == "katherine.johnson@nasa.gov" ``` Here, the Chalk API client is configured to query the test branch created in the previous step. ### Deploy Once you've tested your changes, it's time to deploy! This step looks much like the preview deployment, but this time without the --branch flag: ``` chalk apply ``` Now, your production environments can request the new user.name_email_match_score feature. ``` chalk query --in user.id=1 --out user.name_email_match_score ``` ### Backfill The user.name_email_match_score feature is live! But historically, this feature did not exist, and you won't be able to sample its values at previous times until you backfill the values, which you can do with the chalk trigger command. ``` chalk trigger --resolver example.name_email_match_scorer --lower-bound 2020-05-05T12:00:00+00:00 --persist-offline=True ``` This command will backfill historical values for the user.name_email_match_score feature. ### Offline training With the feature backfilled, you can query the historical value: That's how you deploy a new feature with Chalk! Let a member of our technical team know if we can be helpful. ### Delete Mistakes are inevitable, but Chalk provides the tools you need to quickly and easily recover and keep going. ### Drop a feature If you need to change a feature's type (for example from string to int), or if you want to drop all the data for a feature value, the Chalk CLI has you covered with chalk migrate-type. Simply execute the command from the CLI and you'll have a fresh start to recreate the feature and its data as necessary. ### Delete a row Sometimes you just need to fully remove a record from your systems, whether because of a GDPR mandated "right to forget" request or due to a business requirement. Chalk provides the chalk delete command to meet this need. # Working With Branches source: https://docs.chalk.ai/docs/branches ## Make rapid changes and explore features with branches. A branch deployment lets you try changes to features and resolvers without changing your mainline deployment, the deployment that chalk apply creates without the --branch flag. When working in a branch, you can: - have Chalk watch your files and deploy any changes - deploy updates to your project in seconds - adjust and create features and resolvers inside a Jupyter notebook - recompute datasets to see the effects of changes - keep the same branch name across deployments A branch does not persist results to the online or offline store, and some settings always come from the mainline deployment. See Branch limitations. ### Create a branch deployment To create a branch, run chalk apply --branch . Chalk creates a new branch in the environment that you can interact with. You can then make queries against your branch with chalk query --branch.If you're using the Python API, you can set the branch when creating your client, and all subsequent commands will execute against your branch. The environment needs a mainline deployment before you can create a branch. If you have not deployed to the environment yet, run chalk apply without --branch first. ### Continuously deploy changes You can ask the Chalk CLI to continuously deploy changes to your branch using the watch flag. Chalk will automatically keep the branch up to date as you make changes locally. ### Branch limitations ### What a branch can change A branch deployment can add or change: - features - resolvers, except stream resolvers Code on a branch can refer by name to data sources and Chalk Catalog entries that already exist. Everything else comes from the active mainline deployment, including: - environment variables - secrets - the Chalk platform version - data sources - model namespaces in the model registry To change any of these, run chalk apply without --branch. ### What a branch cannot do A branch deployment cannot: - write feature values to the online store, so resolvers run on every query even when a feature sets max_staleness, - add a new materialized aggregation, - write feature values to the offline store, - write to the vector store, - record feature metrics, - add a new data source, - add a new model namespace to the model registry, - run stream resolvers, or - run scheduled resolvers on their schedule. Chalk still records each query run against a branch, tagged with the branch name. By default, online queries against a branch read values from the mainline deployment's online store. ### Branch online store To give branches their own online store, go to Infrastructure > Resource Configuration > Branch Server in the dashboard and turn on Deploy Branch Online Store. The branch online store is separate from the mainline deployment's online store. With it turned on: - online queries against a branch read from the branch online store instead of the mainline online store, - values you upload to a branch with upload_features are stored for that branch and returned by later queries against it, - query results are still not stored, so resolvers run on every query, and - Chalk clears the stored values each time you deploy to the branch and each time the branch server restarts. ### When to use a new branch or a mainline deployment Each chalk apply --branch replaces the code on that branch. To keep one version running while you try another, deploy the second version to a new branch name. Calling ChalkClient(branch=True) uses the name of your current git branch, with any / replaced by -, which gives each git branch its own Chalk branch. Run chalk apply without --branch when: - production queries need the change, - you need feature values written to the online or offline store, for example to backfill a feature or to add a materialized aggregation, - you changed an environment variable, a secret, or the platform version, - you added a data source or a model namespace, or - you need stream resolvers to run, or scheduled resolvers to run on their schedule. Deploying to mainline does not change your existing branches. Each branch keeps the code you deployed to it, and you do not need to deploy it again. See After a mainline deployment. ### Develop in a notebook Once you've deployed a branch, you can iteratively edit features and resolvers and see the effects of these updates in a dataset. Create a Chalk client and set the branch parameter equal to your branch. Then, any time you execute a cell that contains a class annotated with @features it will automatically be updated in your branch. Similarly, executing cells that contain a function annotated with @online or @offline will automatically deploy the resolver to your branch. ``` @features class NewFeatures: id: int name: str greeting: str @online def new_resolver(name: NewFeatures.name) -> NewFeatures.greeting: return f"Hello {name}!" ``` ### Recompute datasets As you adjust features and resolvers, you can iteratively see how your changes affect your feature values with the Dataset.recompute method. Just pass any features you want to be re-computed as arguments to the features parameter, and Chalk will generate a new Dataset revision using the latest definitions of features and resolvers. When used in conjunction with the ability to adjust your features and resolvers in the notebook, this tool allows developers and data scientists to rapidly experiment and productionize their work. ### Example In the following example we add an oversize feature to an existing dataset of loans. We start out with a dataset. We may have produced this dataset manually, by calling an external API, or by executing offline_query. ``` dataset shape: (786, 3) ┌───────────────┬──────────────┬─────────────────────────┐ │ loan.amount ┆ loan.id ┆ loan.event_time │ │ --- ┆ --- ┆ --- │ │ f64 ┆ str ┆ datetime[μs, UTC] │ ╞═══════════════╪══════════════╪═════════════════════════╡ │ 165435.647396 ┆ l_G3Mc6bi9y4 ┆ 2023-04-08 10:03:09 UTC │ │ 405006.796909 ┆ l_O9OK3us7t2 ┆ 2022-02-07 20:46:08 UTC │ │ 680377.427817 ┆ l_L0Gg0Bd1v3 ┆ 2023-02-16 16:48:29 UTC │ │ 562678.545344 ┆ l_D1Rb5Jq4U5 ┆ 2022-06-21 06:15:19 UTC │ │ … ┆ … ┆ … │ │ 750583.279013 ┆ l_W4ZY9OK5N7 ┆ 2021-12-16 21:40:27 UTC │ │ 71698.15609 ┆ l_H9tG2yJ5B6 ┆ 2023-01-04 11:33:54 UTC │ │ 488697.890372 ┆ l_L4fd4xu4w2 ┆ 2022-08-22 03:55:16 UTC │ │ 769665.198436 ┆ l_k4Dq8bl7b3 ┆ 2022-10-31 07:15:52 UTC │ └───────────────┴──────────────┴─────────────────────────┘ ``` In our Jupyter notebook, we can execute this in a cell to set up our notebook to point to our branch, then update our branch with the new feature and resolver. ``` client = ChalkClient(branch="") # We want to introduce an `oversize` detection feature @features class Loan: id: str event_time: datetime status: str amount: float oversize: bool # Detect any loans bigger than $250,000 @online def is_oversize(amt: Loan.amount) -> Loan.oversize: return amt > 250000 ``` Finally, we can recompute our dataset, telling it to calculate Loan.oversize, to get a dataset back with our new feature values. ``` # Recomputing the dataset will add the oversize column to our result dataset.recompute(features=[Loan.oversize]) shape: (786, 4) ┌───────────────┬───────────────┬──────────────┬─────────────────────────┐ │ loan.oversize ┆ loan.amount ┆ loan.id ┆ loan.event_time │ │ --- ┆ --- ┆ --- ┆ --- │ │ bool ┆ f64 ┆ str ┆ datetime[μs, UTC] │ ╞═══════════════╪═══════════════╪══════════════╪═════════════════════════╡ │ true ┆ 165435.647396 ┆ l_G3Mc6bi9y4 ┆ 2023-04-08 10:03:09 UTC │ │ true ┆ 405006.796909 ┆ l_O9OK3us7t2 ┆ 2022-02-07 20:46:08 UTC │ │ true ┆ 680377.427817 ┆ l_L0Gg0Bd1v3 ┆ 2023-02-16 16:48:29 UTC │ │ true ┆ 562678.545344 ┆ l_D1Rb5Jq4U5 ┆ 2022-06-21 06:15:19 UTC │ │ … ┆ … ┆ … ┆ … │ │ true ┆ 750583.279013 ┆ l_W4ZY9OK5N7 ┆ 2021-12-16 21:40:27 UTC │ │ true ┆ 71698.15609 ┆ l_H9tG2yJ5B6 ┆ 2023-01-04 11:33:54 UTC │ │ true ┆ 488697.890372 ┆ l_L4fd4xu4w2 ┆ 2022-08-22 03:55:16 UTC │ │ true ┆ 769665.198436 ┆ l_k4Dq8bl7b3 ┆ 2022-10-31 07:15:52 UTC │ └───────────────┴───────────────┴──────────────┴─────────────────────────┘ ``` ### The branch server All branches in an environment run on one server, the branch server. The branch server is separate from the servers that handle queries against your mainline deployment, and adding branches does not add servers. See Billing for how the branch server is billed. ### Start and stop the branch server The branch server does not start when a query arrives. These tools start the branch server if it is not running, and wait for it to be ready before they deploy or query: - chalk apply --branch - chalk query --branch - ChalkClient in chalkpy - the query forms in the dashboard Queries sent over gRPC, for example with ChalkGRPCClient, do not start the branch server. Before you send gRPC queries to a branch, start the branch server with ChalkGRPCClient.await_branch_server_start or one of the tools above. You can also start and stop the branch server from the Branches page of your environment in the dashboard, with Start Branch Server and Stop Branch Server. ### Auto-shutdown The branch server shuts down after a period with no requests. The default period is 1 hour. To change it, go to Infrastructure > Resource Configuration > Branch Server in the dashboard and set Auto Shutdown Period to one of these values: - No shutdown - 15 minutes - 30 minutes - 1 hour - 2 hours - 6 hours Queries sent over gRPC do not count as activity, so the branch server can shut down while you are sending gRPC queries to it. If you query a branch only over gRPC, set a longer Auto Shutdown Period, or call await_branch_server_start before each set of queries. ### After a mainline deployment Each mainline deployment restarts the branch server. Your branches keep the code you deployed to them, and you do not need to deploy them again. Each branch reloads on its first query after the restart, so that query can take around 30 seconds. Later queries return at their usual speed. # Static Resolver Optimization source: https://docs.chalk.ai/docs/static-resolver-optimization ## Accelerate Python resolvers using Chalk's symbolic interpreter for columnar execution. Chalk's static resolver optimization transforms Python resolvers into high-performance columnar operations through symbolic execution, eliminating Python's runtime overhead while maintaining its developer-friendly syntax. ### The Performance Challenge Python resolvers provide flexibility and ease of use for feature engineering, but traditional Python execution introduces significant performance bottlenecks in real-time ML workloads: - Row-by-row processing: Traditional Python resolvers process data one row at a time - Global Interpreter Lock (GIL): Prevents true parallel execution across multiple CPU cores - Dynamic typing overhead: Runtime type checking adds computational cost - Limited vectorization: Cannot leverage SIMD operations for numerical computations These limitations can make Python resolvers unsuitable for latency-sensitive applications requiring sub-millisecond response times. ### Symbolic Execution Approach Chalk addresses these performance challenges by converting Python resolver functions into optimized Velox-native expressions at query-plan time. This transformation happens automatically without requiring changes to your resolver code. ### How It Works - Symbolic Analysis: At query planning stage, Chalk analyzes your Python resolver function - Type Tracking: Maintains both Python and Velox types during symbolic interpretation - Expression Tree Building: Executes the function symbolically with "symbolic values" that represent computation trees - Columnar Translation: Transforms Python logic into efficient columnar operations - Automatic Fallback: Gracefully falls back to subprocess execution for unsupported functions For more details on the implementation, check out our blog post. Static optimization is automatically applied to compatible Python resolvers. No code changes or special decorators are required. ### Supported Operations The symbolic interpreter supports a wide range of Python operations commonly used in feature engineering: ### Basic Operations - Arithmetic operations (+, -, *, /, //, %, **) - Comparison operations (<, >, <=, >=, ==, !=) - Logical operations (and, or, not) - String operations (basic manipulation and formatting) ### Data Types - Numeric types (int, float) - Strings (str) - Booleans (bool) - None values and null handling ### Control Flow - Conditional expressions (if/else) - Simple list comprehensions - Basic function calls ### Performance Benefits Static resolver optimization delivers significant performance improvements: ### Parallel Execution - Eliminates Python's GIL restrictions - Enables true multithreaded processing - Scales linearly with available CPU cores ### Vectorized Operations - Leverages SIMD (Single Instruction, Multiple Data) instructions - Processes multiple data points simultaneously - Optimizes memory access patterns ### Reduced Overhead - Removes dynamic typing checks at runtime - Eliminates Python interpreter overhead - Minimizes memory allocations ### Example Performance Impact ``` @online def compute_risk_score( transaction_amount: Transaction.amount, user_avg_transaction: Transaction.user.avg_transaction_amount, merchant_risk_factor: Transaction.merchant.risk_factor ) -> Transaction.risk_score: if transaction_amount > user_avg_transaction * 10: return merchant_risk_factor * 2.5 else: return merchant_risk_factor * 0.5 ``` This resolver, when optimized, can process millions of transactions per second compared to thousands with traditional Python execution. ### Best Practices To maximize the benefits of static resolver optimization: ### Keep Resolvers Simple Focus on computational logic rather than complex control flow: ``` # Good: Simple mathematical computation @online def calculate_ratio(numerator: Feature.a, denominator: Feature.b) -> Feature.ratio: return numerator / denominator if denominator != 0 else 0 # Less optimal: Complex nested logic @online def complex_calculation(a: Feature.a, b: Feature.b, c: Feature.c) -> Feature.result: result = a for i in range(10): # Loops may not optimize if b > i: result = custom_function(result, c) # External functions may not optimize return result ``` ### Use Native Operations Stick to Python's built-in operations when possible: ``` # Good: Uses native operations @online def normalize_value(value: Feature.value, mean: Feature.mean, std: Feature.std) -> Feature.normalized: return (value - mean) / std if std > 0 else 0 # Less optimal: External library calls @online def normalize_with_library(value: Feature.value) -> Feature.normalized: import numpy as np # External libraries may not optimize return np.log1p(value) ``` ### Leverage Type Annotations Ensure all inputs and outputs have proper type annotations: ``` @online def typed_resolver( amount: Transaction.amount, # Explicit feature type rate: Transaction.currency.exchange_rate # Explicit feature type ) -> Transaction.converted_amount: # Explicit return type return amount * rate ``` ### Monitoring Optimization You can verify whether your resolvers are being optimized by checking the query execution logs or by viewing the query plan in the dashboard. Optimized resolvers will be marked as accelerated, and if your Python resolver is not optimized the query plan node will also provide information about why it was not optimized. Optimized resolvers will show significantly lower execution times and higher throughput. ### Limitations While powerful, static optimization has some limitations: - Complex control flow: Deeply nested loops or recursive functions may not optimize - Dynamic code: eval(), exec(), or dynamic attribute access patterns When the optimizer encounters unsupported operations, it automatically falls back to standard Python execution, ensuring your resolvers always work correctly. ### Summary Static resolver optimization in Chalk bridges the gap between Python's ease of use and the performance requirements of real-time ML systems. By automatically converting Python resolvers into optimized columnar operations, Chalk enables you to write maintainable feature engineering code that runs at production scale without sacrificing performance. # Tasks source: https://docs.chalk.ai/docs/tasks ## Run arbitrary Python scripts in your Chalk deployment Tasks let you run arbitrary Python code in your Chalk deployment environment with full access to your resolvers, dependencies, and infrastructure. Tasks execute in the same environment as your resolvers, making them ideal for: - Data backfills: Recompute historical features or migrate data - One-off computations: Run ad-hoc analyses, reports, or maintenance tasks - ML training: Execute training scripts with access to your feature infrastructure - Testing and validation: Run data quality checks or integration tests All tasks are tracked and monitored through the Chalk dashboard with full logs, metrics, and status tracking. ### Quick Start The fastest way to run a task is using the Chalk CLI. First, create a simple Python script: ``` echo 'print("Hello from Chalk tasks!")' > script.py ``` Then, run it with the Chalk CLI: ``` chalk task run script.py ``` The CLI will output a link to view your task in the dashboard: ``` ✓ View task at: https://chalk.ai/projects/my-project/environments/prod/tasks/550e8400... ``` Click the link to see your script's output, status, and execution metrics. ### CLI Reference ### Basic Usage The chalk task run command executes Python scripts in your deployment: ``` chalk task run [target] [flags] ``` ### Target Formats The target supports multiple formats: ### Running a File Execute an entire Python script: ``` chalk task run scripts/data_migration.py ``` ### Running a Function Execute a specific function from a file: ``` chalk task run scripts/backfill.py::process_batch ``` ### Running a Module For installed Python modules, use the -m / --module flag. Modules require --branch for access to your deployment. See Branch Execution for details. ``` # Module execution chalk task run -m mypackage.scripts.backfill --branch my-branch # Module function chalk task run -m mypackage.scripts.backfill::main --branch my-branch ``` ### Passing Arguments CLI arguments are always passed as strings. Your function should handle type conversion as needed. ### Arguments for Functions When running a specific function (with ::function), arguments are passed directly to your function as *args and **kwargs: ``` chalk task run script.py::backfill --arg 2024-01-01 --arg 2024-12-31 --arg dry_run=true ``` ``` def backfill(start_date: str, end_date: str, dry_run: str = "false"): """ Positional arguments: start_date, end_date Keyword arguments: dry_run """ is_dry_run = dry_run == "true" print(f"Backfilling from {start_date} to {end_date} (dry_run={is_dry_run})") if not is_dry_run: # Perform actual backfill pass ``` ### Arguments for Whole Files When running a whole file (without ::function), arguments are passed as command-line arguments. You can use sys.argv or an argument parser to access them: Using sys.argv: Direct access to raw command-line arguments as a list: ``` chalk task run script.py --arg 2024-01-01 --arg 2024-12-31 ``` ``` #!/usr/bin/env python3 import sys if len(sys.argv) >= 3: start_date = sys.argv[1] end_date = sys.argv[2] print(f"Processing data from {start_date} to {end_date}") ``` Using argparse: Provides automatic parsing, help messages, and type handling: ``` chalk task run process.py --arg users.csv --arg --dry-run --arg output_dir=./results ``` ``` #!/usr/bin/env python3 import argparse parser = argparse.ArgumentParser() parser.add_argument('input_file', help='Input CSV file') parser.add_argument('--dry-run', action='store_true', help='Run without making changes') parser.add_argument('--output_dir', default='./output', help='Output directory') args = parser.parse_args() print(f"Processing {args.input_file}") print(f"Dry run: {args.dry_run}") print(f"Output directory: {args.output_dir}") ``` ### Batch Execution Execute the same script multiple times with different parameters using a JSONL file. This is useful for hyperparameter sweeps, batch processing, or running experiments across different data segments. ### JSONL Format Each line is a JSON object with kwargs (and optionally args): ``` {"kwargs": {"learning_rate": "0.001", "batch_size": "32", "epochs": "10"}} {"kwargs": {"learning_rate": "0.005", "batch_size": "64", "epochs": "10"}} {"kwargs": {"learning_rate": "0.01", "batch_size": "128", "epochs": "5"}} ``` ### Running Batch Tasks ``` chalk task run scripts/train_model.py::train --args-file hyperparams.jsonl ``` This creates a separate task for each line in the file. The CLI will show progress and provide a link to view all tasks in the dashboard. ### Generating JSONL Files From Python: ``` import json params = [ {"kwargs": {"user_id": i, "region": "us-west"}} for i in range(1000, 2000) ] with open("batch_params.jsonl", "w") as f: for param in params: f.write(json.dumps(param) + "\n") ``` From a database query: ``` import json import psycopg2 conn = psycopg2.connect(...) cursor = conn.cursor() cursor.execute("SELECT user_id, region FROM users WHERE needs_backfill = true") with open("users_to_backfill.jsonl", "w") as f: for user_id, region in cursor: f.write(json.dumps({ "kwargs": {"user_id": user_id, "region": region} }) + "\n") ``` ### Branch Execution When you're making changes across multiple files, updating dependencies, or modifying resolvers that your script uses, it's important to test everything together before merging to production. Branch deployments let you run tasks in an isolated environment that includes all your changes: ``` chalk task run scripts.py --branch feature/new-algorithm ``` This automatically: - Runs chalk apply --branch feature/new-algorithm - Waits for the branch deployment to be ready - Submits the task to the branch environment You can skip the automatic apply if the branch is already deployed: ``` chalk task run scripts/hello.py --branch feature/new-algorithm --skip-apply ``` See Working with Branches for more information. ### CLI Flags Reference ### Module flag -m, --module - Treat the target as a module import path instead of a file path. ``` chalk task run -m mypackage.analytics.report ``` ### Branch flags --branch BRANCH_NAME - Execute code from a specific git branch. See Branch Execution. --skip-apply - Skip the automatic chalk apply when running with --branch. Use this if the branch is already deployed. ### Argument flags --arg KEY=VALUE or --arg VALUE - Pass arguments to your script. Can be specified multiple times. See Passing Arguments. --args-file FILE.jsonl - Execute script multiple times with different arguments from a JSONL file. See Batch Execution. ### Execution flags --max-retries N - Set number of retry attempts on failure. Defaults to 0 (no retries). ``` chalk task run flaky_api_call.py --max-retries 3 ``` ### Monitoring and Dashboard Tasks appear in the Chalk dashboard where you can monitor their execution and debug failures. ### Accessing the Dashboard After submitting a task, the CLI outputs a link to the dashboard: ``` $ chalk task run scripts/hello.py ✓ View task at: https://chalk.ai/projects/my-project/environments/prod/tasks/550e8400... ``` You can also view all tasks at the tasks overview page, which shows aggregate CPU and memory utilization charts across all running tasks. ### Task Details The task detail page displays: - Execution timeline - Visual timeline showing queue time, execution attempts, and retries - Status - Current state (QUEUED, WORKING, COMPLETED, FAILED, CANCELED) - Logs - Full stdout and stderr output with search and filtering - Metrics - CPU and memory utilization for the task - Source code - View and download the uploaded script - Task metadata - Branch, deployment ID, resource group, start/end times, and author ### Task Statuses - QUEUED - Task is waiting to execute - WORKING - Task is currently running - COMPLETED - Task finished successfully - FAILED - Task encountered an error or non-zero exit code - CANCELED - Task was manually canceled ### Additional Usage ### Logging in Tasks Task logs are captured and displayed in the Chalk dashboard. You can use standard print() statements which will appear in your task logs. You can also use chalk_logger in your script for structured logging: ``` from chalk import chalk_logger def process_data(batch_size: str): batch_size_int = int(batch_size) chalk_logger.info(f"Starting data processing with batch_size={batch_size_int}") # Your processing logic for i in range(batch_size_int): chalk_logger.debug(f"Processing batch {i+1}/{batch_size_int}") # ... process data ... chalk_logger.info("Data processing completed successfully") ``` chalk_logger provides: - Structured log levels (debug, info, warning, error) - Better integration with Chalk's logging infrastructure - Consistent formatting with resolver logs ### Running Class Methods You can execute class methods using the double-colon syntax: ``` # File-based class method chalk task run scripts/processor.py::DataProcessor::run # Module-based class method chalk task run mypackage.processors::ETLProcessor::execute --module ``` ### Troubleshooting ### Task Stuck in QUEUED If a task remains in QUEUED status: - Verify Kubernetes pods are running - Check for image pull errors or pod crashloops - Ensure the job queue is processing tasks ### Import Errors ``` ModuleNotFoundError: No module named 'mypackage' ``` Solutions: - Ensure the package is in requirements.txt or pyproject.toml - Verify the package is installed in your active deployment - For modules, ensure the module path is correct ### Argument Issues If arguments aren't being passed correctly: - Remember CLI args are always strings - convert types in your function - Use key=value format for kwargs, plain values for positional args - Test locally first with the same argument format ### Next Steps - Learn about Working with Branches for development workflows - Explore Model Training for ML-specific tasks # GitHub Actions source: https://docs.chalk.ai/docs/github-actions ## Deploy feature pipelines in GitHub Actions Chalk provides official support for GitHub Actions. You can install the Chalk CLI and create deployments (preview and production) from pre-built GitHub Actions. ### Installing the Chalk CLI You will need to create a Chalk token from the settings page of your dashboard and store the resulting client ID and secret as GitHub Secrets. This step supports the following inputs: - client-id: The Chalk Client ID from the tokens page in your settings. - client-secret: The Chalk Client Secret from the tokens page in your settings. - version (optional): The version of chalk to install, defaulting to latest. - api-host (optional): If you're using a self-hosted deployment, the API host where Chalk is hosted. - environment (optional): The Chalk environment to use. Your token is typically scoped to a single environment, and you won't need to use this parameter. ### Example ``` - uses: chalk-ai/cli-action@v2 with: client-id: ${{secrets.CHALK_CLIENT_ID}} client-secret: ${{secrets.CHALK_CLIENT_SECRET}} # Optional: Version of the Chalk CLI to install. Defaults to `latest` version: latest # Optional: Environment to use. Optional for environment-scoped tokens. environment: # Optional: Used for Hybrid Cloud deployments api-host: https://custom.deployment.com/ ``` For more information, see the chalk-ai/cli-action on the GitHub Marketplace, or see a complete example at chalk-ai/examples. ### Deploying to Chalk You can deploy to Chalk by first installing the CLI, as in the section above. Or, you can use the chalk-ai/deploy-action step. This step supports the following inputs: - client-id: The Chalk Client ID from the tokens page in your settings. - client-secret: The Chalk Client Secret from the tokens page in your settings. - branch (optional): By default, Chalk will deploy to your production environment. With branch, your pipelines will deploy to a Chalk branch. - await (optional): Should this step block until it completes? Defaults to true. - version (optional): The version of chalk to install, defaulting to latest. - api-host (optional): If you're using a self-hosted deployment, the API host where Chalk is hosted. - environment (optional): The Chalk environment to use. Your token is typically scoped to a single environment, and you won't need to use this parameter. A production deploy checks your features and resolvers before deploying, and the check runs your project's Python code. The job needs Python and your project's dependencies, including chalkpy, installed before chalk apply runs. Use the Python version that matches the runtime in your project's chalk.yaml. Add these steps before the deploy step. ### Example ``` - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" # Your project's dependencies, including chalkpy - run: pip install -r requirements.txt - uses: chalk-ai/deploy-action@v2 with: client-id: ${{secrets.CHALK_CLIENT_ID}} client-secret: ${{secrets.CHALK_CLIENT_SECRET}} # Optional: Version of the Chalk CLI to install. Defaults to `latest` version: latest # Optional: Environment to use. Optional for environment-scoped tokens. environment: # Optional: Used for Hybrid Cloud deployments api-host: https://custom.deployment.com/ ``` # GitLab CI/CD source: https://docs.chalk.ai/docs/gitlab-cicd ## Deploy feature pipelines in GitLab CI/CD Chalk provides official support for GitLab CI/CD. You can install the Chalk CLI and create deployments (preview and production) using GitLab CI/CD jobs. ### Installing the Chalk CLI You will need to create a Chalk token from the settings page of your dashboard and store the resulting client ID and secret as GitLab CI/CD Variables. The Chalk CLI reads its credentials and target from environment variables, so a job needs no separate login step. Define these variables: - CHALK_CLIENT_ID: The Chalk Client ID from the tokens page in your settings (stored as CI/CD variable). - CHALK_CLIENT_SECRET: The Chalk Client Secret from the tokens page in your settings (stored as masked CI/CD variable). - CHALK_VERSION (optional): The version of chalk to install, defaulting to latest. - CHALK_API_SERVER (optional): If you're using a self-hosted deployment, the API host where Chalk is hosted. - CHALK_ENVIRONMENT (optional): The Chalk environment to use. Your token is typically scoped to a single environment, and you won't need to use this parameter. ### Example ``` install-chalk: stage: setup image: ubuntu:latest before_script: - apt-get update && apt-get install -y curl script: - curl -s -L https://api.chalk.ai/install.sh | sh -s -- "${CHALK_VERSION:-latest}" - export PATH="$HOME/.chalk/bin:$PATH" - chalk version variables: # Optional: Version of the Chalk CLI to install. Defaults to `latest` CHALK_VERSION: latest ``` The install script puts the CLI in $HOME/.chalk/bin. ### Deploying to Chalk You can deploy to Chalk by installing the CLI, as in the section above, and running chalk apply in the same job. --force skips the interactive confirmation, which a CI job can't answer. See chalk apply for its other flags. A production deploy checks your features and resolvers before deploying, and the check runs your project's Python code. The job needs Python and your project's dependencies, including chalkpy, installed before chalk apply runs. Use the Python version that matches the runtime in your project's chalk.yaml. The example uses a Python image for this. This job supports the following variables: - CHALK_CLIENT_ID: The Chalk Client ID from the tokens page in your settings (stored as CI/CD variable). - CHALK_CLIENT_SECRET: The Chalk Client Secret from the tokens page in your settings (stored as masked CI/CD variable). - CHALK_BRANCH (optional): By default, Chalk will deploy to your production environment. With this variable, your pipelines will deploy to a Chalk branch. - CHALK_AWAIT (optional): Should this job block until deployment completes? Defaults to true. - CHALK_VERSION (optional): The version of chalk to install, defaulting to latest. - CHALK_API_SERVER (optional): If you're using a self-hosted deployment, the API host where Chalk is hosted. - CHALK_ENVIRONMENT (optional): The Chalk environment to use. Your token is typically scoped to a single environment, and you won't need to use this parameter. ### Example ``` stages: - deploy deploy-chalk: stage: deploy image: python:3.12 before_script: # Your project's dependencies, including chalkpy - pip install -r requirements.txt - curl -s -L https://api.chalk.ai/install.sh | sh -s -- "${CHALK_VERSION:-latest}" - export PATH="$HOME/.chalk/bin:$PATH" script: - | # Deploy to Chalk DEPLOY_ARGS="--force --no-spinners" if [ -n "$CHALK_BRANCH" ]; then DEPLOY_ARGS="$DEPLOY_ARGS --branch=$CHALK_BRANCH" fi if [ "$CHALK_AWAIT" = "false" ]; then DEPLOY_ARGS="$DEPLOY_ARGS --await=false" fi chalk apply $DEPLOY_ARGS variables: # Optional: Version of the Chalk CLI to install. Defaults to `latest` CHALK_VERSION: latest # Optional: Environment to use. Optional for environment-scoped tokens. # CHALK_ENVIRONMENT: # Optional: Used for Hybrid Cloud deployments # CHALK_API_SERVER: https://custom.deployment.com/ # Optional: Deploy to a specific branch instead of production # CHALK_BRANCH: feature-branch # Optional: Wait for deployment to complete (default: true) # CHALK_AWAIT: "false" ``` # Queries Overview source: https://docs.chalk.ai/docs/query-overview ## Fetch feature values via queries. To request or compute data with Chalk, you'll use queries. In general, when you run a Chalk query, you are either requesting the most up-to-date values for your feature classes, requesting a set of historical data for your feature classes, or running a backfill or batch job. The first use case is accomplished through online queries, which try to return values for a single feature class as quickly as possible, taking advantage of caching and distributed execution. The latter use cases are accomplished through offline queries which use the offline store and can return multiple instances of a feature class for multiple primary keys or timepoints. ### Running queries At a high level, a query specifies input features and output features. Inputs differ slightly for online queries and offline queries, but in both cases the input must contain the primary key of your requested feature class. ### Running online queries Online queries can be run using query: ``` chalk query --in user.id=1 --out user.name ``` or one of our API Clients: ``` from chalk.client import ChalkClient client = ChalkClient() client.query( input={'user.id': 1}, output=['user.name'], # branch='test', # run against a branch ) ``` ### Running offline queries Offline queries can be run with one of our API Clients: ``` from chalk.client import ChalkClient from datetime import datetime client = ChalkClient() client.offline_query( input={'user.id': [1,2,3,4]}, output=['user.name'] # branch='test', # run against a branch # recompute_features=True, # recompute features # run_asynchronously=True, # run in separate pod from active deployment # max_samples = 10, # max of 10 samples # lower_bound=datetime(2024, 10, 12), # sample computed after 10.12.2024 # upper_bound=datetime(2024, 10, 20), # sample computed before 10.20.2024 ) ``` ### Running queries using gRPC If you have a gRPC query server active in your environment, you can also run queries using the gRPC client: ``` from chalk.client.client_grpc import ChalkGRPCClient grpc_client = ChalkGRPCClient() grpc_client.query( input={'user.id': 1}, output=['user.name'] ) ``` ### Scheduled and triggered resolver runs Specific resolvers can also be scheduled or triggered (for instance, as part of pipelines like Airflow). Specific queries can also be scheduled with ScheduledQuery. Triggers and schedules are useful for pulling data from "slow" data sources into your offline and online store. ### Query side effects Chalk queries can also write data. This is an essential part of Chalk: every time you compute a feature through an online query, the output is written down in the offline store. This makes it easy to: - create datasets from your previously computed features, - monitor and track your computed features over time. Though not the default, offline queries can write data to both the offline store and the online store using etl_offline_to_online. This can be useful when backfilling data from slow data sources or when performing expensive feature computation that would otherwise significantly impact the latency of your online queries. ### Online and offline query differences | Online query | Offline query | | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Runs only @online resolvers | Runs both @online and @offline resolvers | | Returns one row of data about one entity | Returns a DataFrame of many rows of historical data corresponding to multiple entities point-in-time | | Designed to return data in milliseconds | Blocks until computation is complete, not designed for millisecond-level computation | | Queries the online store and calls `@online` resolvers for quick retrieval | Queries the offline store which stores all data from online queries, unless `recompute_features=True`, in which case `@offline` and `@online` resolvers are used to resolve the outputs | | Writes output data to online store database and offline store database | Writes output to a parquet file containing results to cloud storage. Only writes to online store or offline store if specified. | # Online Queries source: https://docs.chalk.ai/docs/query-online ## Fetch feature values via online queries Online queries access or compute feature values for a single feature set in real time. The term "real time" means that responses should seem practically instantaneous, even if they need to compute features on new data. However, online queries don't only perform data retrieval, they also store the results of the features that they compute. This provides visibility and long-term tracking into the features you are generating. In this section, we provide a high level overview of how online queries compute and record their outputs. ### How Do Online Queries Work? Chalk responds to online queries by getting and executing a query plan. ### Getting a Query Plan A query plan is a sequence of tasks, some of which can be executed in parallel, that will produce a target output. Although a simplification, your resolvers are a subset of these tasks. Consider the following feature class and resolvers: ``` from chalk.features import features, online @features class User: id: int name: str is_palindrome: bool is_short: bool is_short_palindrome: bool @online def get_is_palindrome(name: User.name) -> User.is_palindrome: return name == name[::-1] @online def get_is_short(name: User.name) -> User.is_short: return len(name) < 3 @online def get_is_short_palindrome(is_short: User.is_short, is_palindrome: User.is_palindrome) -> User.is_short_palindrome: return is_short and is_palindrome ``` If you were to begin assembling a dependency graph for the features. You would wind up with something like the following: ``` ┌───────────┐ │name │ └┬─────────┬┘ ┌▽───────┐┌▽────────────┐ │is_short││is_palindrome│ └┬───────┘└┬────────────┘ ┌▽─────────▽────────┐ │is_short_palindrome│ └───────────────────┘ ``` When you run an online query, such as: ``` chalk query --in user.id=1 --in user.name=bob --out is_short_palindrome ``` Chalk constructs a plan for how to "solve" this query. This query plan is viable for other queries with the same input and output features. Running chalk query with the --explain flag outputs your query plan. The following query can reuse the plan generated by the one above: ``` chalk query --in user.id --in user.name=bartholomew --out is_short_palindrome ``` Even though both the input name and the output of the query are different, the query plan remains valid. ### Executing a Query Plan As illustrated above, a query plan is not a linear sequence of tasks that must be executed one after another: a lot of work can often be performed in parallel. After getting a query plan, Chalk distributes subtasks to workers, applies a number of optimizations on your resolvers/datasource connections, and computes the target outputs of your query. These outputs are then returned. There are multiple ways to run an online query. You can use the Chalk API Client, the Chalk CLI, or the REST API. For example the following calls all trigger the same online query: With the Chalk Python API Client: ``` from chalk.client import ChalkClient from src.models import User client = ChalkClient() results = client.query( input={ User.id: 1, }, output=[User.is_short_palindrome, User.name], ) ``` With the Chalk CLI: ``` chalk query --in user.id=1 --out user.is_short_palindrome --out user.name ``` With the REST API: ``` curl -X POST {YOUR_ENGINE_URL}/v1/query/online \ -H "Authorization: Bearer $access_token" \ -H 'x-chalk-env-id: {YOUR_ENV_ID}' \ -H 'x-chalk-deployment-type: engine' \ -H 'content-type:application/json' \ -d '{"inputs": {"user.id": 1}, "outputs":["user.is_short_palindrome", "user.name"]}' ``` ### Caching and Writing Online queries write computed values to two places: the offline store and the online store. However, computed features are only written to the online store if they have a caching policy. The online store is used to circumvent recomputation of expensive features that are either unlikely to have changed or can tolerate slightly stale values. Properly configuring the caching policies for your features can make your online queries significantly more efficient. If you haven't specified a caching policy, Chalk recomputes the values for a feature each time it is requested. We go into more depth on query caching here. # Uploading Features source: https://docs.chalk.ai/docs/upload-features ## Write feature values directly to Chalk's online store. upload_features lets you write feature values directly to the online store and, optionally, update materialized windowed aggregations. This is useful for bootstrapping the online store from an external pipeline, pushing real-time events, or syncing data computed outside of Chalk. ### gRPC upload (recommended) ChalkGRPCClient.upload_features is the recommended way to upload features. It is a synchronous, blocking call that accepts bulk data in a single request and offers significantly higher throughput than the HTTP API. ``` import pyarrow as pa from chalk.client.client_grpc import ChalkGRPCClient from chalk.features import features, Primary @features class Transaction: id: Primary[str] amount: float merchant: str client = ChalkGRPCClient() # Option 1: mapping of feature → list of values response = client.upload_features( inputs={ Transaction.id: ["txn_1", "txn_2"], Transaction.amount: [49.99, 12.50], Transaction.merchant: ["Acme", "Globex"], } ) # Option 2: Arrow Table or RecordBatch batch = pa.RecordBatch.from_pydict({ "transaction.id": ["txn_3"], "transaction.amount": [7.00], "transaction.merchant": ["Initech"], }) response = client.upload_features(inputs=batch) ``` The inputs parameter accepts: - A mapping from a feature (or feature name string) to a list of values - An Arrow Table or RecordBatch - A Polars or Chalk DataFrame Additional parameters: | Parameter | Type | Description | | ----------------- | --------------------------- | ---------------------------------------- | | `inputs` | mapping / DataFrame / Arrow | Feature data to upload | | `request_timeout` | `float \| None` | Network-level timeout in seconds | | `headers` | mapping / sequence | Extra headers forwarded with the request | The call blocks until the server acknowledges the write and returns an UploadFeaturesResponse containing any errors that occurred. ### Upload options By default, upload_features writes to the online store only. You can control this behaviour — as well as materialized aggregation updates — via keyword arguments: | Parameter | Default | Description | | ---------------- | ------- | ----------------------------------------- | | `write_online` | `True` | Write feature values to the online store | | `write_offline` | `False` | Write feature values to the offline store | | `update_mataggs` | `False` | Update materialized windowed aggregations | ### Updating materialized aggregations When update_mataggs=True, Chalk will recompute materialized windowed aggregation buckets from the uploaded child records rather than just storing the raw feature values. This is the primary way to keep pre-aggregated features (e.g. sum, count, approx_top_k over a time window) up to date without running a full resolver. Chalk recomputes every materialized aggregation eligible for the uploaded namespace, with one exception: aggregations whose aggregate_on column is a different version of a feature in the upload are skipped. This isolates per-version writes when both the source feature and the aggregate are versioned — uploading Transaction.amount (v1) won't pollute an aggregate sourced from Transaction.amount @ 2. Aggregates whose aggregate_on column matches the uploaded version (or is unversioned) update normally. The uploaded records must include a FeatureTime column so Chalk knows which aggregation buckets to update. ``` from datetime import datetime, timezone import pyarrow as pa from chalk.client.client_grpc import ChalkGRPCClient from chalk import DataFrame, FeatureTime, Primary, windowed from chalk.features import _, features @features class Transaction: id: Primary[int] user_id: "User.id" amount: float ts: FeatureTime @features class User: id: Primary[int] transactions: DataFrame[Transaction] total_spend: "Windowed[float]" = windowed( "7d", "30d", materialization={"bucket_duration": "1d"}, expression=_.transactions[_.amount].sum(), ) client = ChalkGRPCClient() now = datetime.now(timezone.utc) batch = pa.RecordBatch.from_pydict({ "transaction.id": [101, 102, 103], "transaction.user_id": [42, 42, 42], "transaction.amount": [10.0, 25.0, 5.0], "transaction.ts": [now, now, now], }) response = client.upload_features(inputs=batch, update_mataggs=True) ``` After this call, User.total_spend["7d"] and User.total_spend["30d"] will reflect the newly uploaded transactions for user 42. ### Other upload methods ### HTTP upload ChalkClient.upload_features sends a single record over HTTP and writes to both the online and offline stores. It is lower throughput than gRPC and does not support the UploadFeaturesOptions described above. ``` from chalk.client import ChalkClient ChalkClient().upload_features( input={ Transaction.id: 104, Transaction.amount: 22.00, } ) ``` For multiple records over HTTP, use ChalkClient.multi_upload_features, which accepts a list of mappings, a column-oriented mapping, or a pandas/polars/chalk DataFrame. Both HTTP methods write to the online and offline stores. ### gRPC bulk upload (materialized aggregations) ChalkGRPCClient.upload_features_bulk is a separate endpoint designed specifically for updating materialized windowed aggregations. Use it when you need to push raw child records that feed into pre-aggregated bucket values rather than uploading finished feature values directly. # Scheduled Query source: https://docs.chalk.ai/docs/scheduled-query ## Run feature pipelines on a schedule When features are coming from a single SQL file, or a single resolver, you can use resolver crons to keep your online and offline stores up to date. However, when features are chained together, or when you need to run a feature pipeline on a schedule, you can use scheduled queries. Scheduled queries let you run an offline query on a schedule, and persist the results in the online and/or offline feature stores. ### Create a scheduled query To create a scheduled query, make a ScheduledQuery object somewhere in your code. Crontabs can be used to specify the schedule in the UTC timezone. ``` from chalk import ScheduledQuery ScheduledQuery( name="enrich-transactions", schedule="0 0 * * *", output=[Transaction.clean_name, Transaction.category], store_online=True, store_offline=True, ) ``` When you run chalk apply, Chalk creates the scheduled query. In the dashboard, open Offline > Schedules. The Definitions tab lists your scheduled queries with each one's schedule, latest run, and next run, and the Scheduled query runs tab lists the runs themselves. Click a run's ID to open the run, and use its Logs tab to see its logs. Click a scheduled query's name to open its page, which has Overview, Run history, Scheduled run configuration, and Debug tabs. ### Incrementalization By default, scheduled queries use incrementalization to only ingest data which has been updated since the last run. You can also set a resolver to use as the source of the incrementalization. For example, if you were enriching financial transaction data, you might use the transactions.chalk.sql resolver as the source of incrementalization. ``` from chalk import ScheduledQuery ScheduledQuery( name="enrich-transactions", schedule="0 0 * * *", output=[Transaction.clean_name, Transaction.category], store_online=True, store_offline=True, incremental_resolvers="transactions", ) ``` ### Monitoring To get notified when a scheduled query fails, create a metric chart monitor with these settings: - Series: Cron Runs, filtered to Resolver Status failure - Threshold: Above 0 - Alert channels: the Slack, PagerDuty, or incident.io channels to notify See Monitors for how monitors, incidents, and alert channels work. Additionally, you can export metrics about scheduled query executions to your observability tools. Use scheduled_query_count (scheduled_query_count_total on /v2/metrics/export) to count executions by query_name and query_status. Filter on query_status="failure" and group by query_name to alert on failures for each scheduled query. You can also export scheduled_query_success_ratio for the fraction of executions that succeeded per query. Both metrics cover executions since midnight UTC. See the metrics export documentation for labels and counter reset behavior. ### Resources By default, scheduled query runs are picked up by the Job Queue Consumer. To see your Job Queue Consumer resource configurations, check the Job Queue Consumer service under Resources in the dashboard. To customize the resources allocated to runs for a specific scheduled query, set the resource configuration under the Scheduled run configuration tab on the scheduled query's page in the dashboard. # Named Queries source: https://docs.chalk.ai/docs/named-query ## Track and manage named queries in Chalk With Chalk NamedQuery objects, you can define and version your common query patterns in code. This provides several advantages: - queries are preplanned on engine boot, reducing first-query latency, - query outputs and parameters don't need to be hardcoded, reducing boilerplate code and ensuring consistency between your queries, - queries are grouped together on the web, making them easier to track, monitor, and debug. ### Relationship to models Named queries typically map to specific models that you're running in production. While feature classes model your domain objects (users, transactions, accounts) and may contain hundreds of features for reuse across different models, a named query selects only the specific subset of features needed for a particular model. For example, you might have a User feature class with 100+ features capturing everything about a user: their profile, behavior metrics, transaction history aggregations, and risk signals. However, your fraud detection model might only need 15 specific features, while your recommendation model needs a different set of 30 features. Named queries let you define these model-specific feature sets, ensuring each model gets exactly what it needs without unnecessary computation. This separation allows different models to access the same domain objects through feature classes while only requesting the features they need, improving performance and making it easier to track which features each model depends on. ### Defining Named Queries To define a named query, add a NamedQuery object to your Chalk deployment: ``` from chalk import NamedQuery from src.models import User NamedQuery( name="fraud", input=[User.id], output=[ User.email_age_days, User.denylisted, User.credit_report.flags, ], tags=["team:fraud"], owner="jodie@chalk.ai", description="Primary fraud model for signup" ) ``` Running chalk apply makes the named query available in your deployment. ### Using Named Queries Named queries can then be leveraged through any of our clients by specifying the query_name parameter. Using the Chalk CLI tool, this looks something like: ``` chalk query --in user.id=1 --query-name fraud ``` Because a named query has been specified, you don't need to explicitly pass in the tags and outputs for your query. The above command is equivalent to running the more complicated: ``` chalk query \ --in user.id=1 \ --out user.email_age_days \ --out user.denylisted \ --out user.credit_report.flags \ --tag team:fraud ``` This feature is also accessible in all of our API clients through the query_name parameter. For instance, in Python, you can run: ``` from chalk.client import ChalkClient ChalkClient().query( input={"user.id": 1}, query_name="fraud", ) ``` You can also run a named query offline, provided that all outputs have offline resolvers. ``` from chalk.client import ChalkClient ChalkClient().offline_query( input={"user.id": 1}, query_name="fraud", recompute_features=True, ) df = dataset.get_data_as_pandas() ``` To see all the named queries you've defined in your current active deployment, you can run: ``` $ chalk named-query list ``` ### Versioning Named Queries If you want to create multiple versions of a similar query, you can use the version parameter of the NamedQuery object and the query_name_version parameter of our various clients. Note, when executing a named query both the query name and the query version must match. This means that if you've defined two named queries in your codebase: ``` from chalk import NamedQuery from src.models import User NamedQuery( name="fraud", input=[User.id], output=[User.denylisted], ) NamedQuery( name="fraud", version="1.1.0", input=[User.id], output=[ User.email_age_days, User.denylisted, User.credit_report.flags, ], ) ``` And you run the following query: ``` chalk query --in user.id=1 --query-name fraud ``` We will return User.denylisted since the first named query has no version and no version was passed through query-name-version. To access a version named query, the version must be explicitly passed. For example: ``` chalk query --in user.id=1 --query-name fraud --query-name-version 1.1.0 ``` ### Scoping Preplanning by Resource Group By default, every query server preplans every named query on boot. If you run multiple resource groups—for example, a default group for general traffic and a gpu-workers group for model-serving — you can use the resource_groups parameter to restrict which servers preplan a named query, keeping boot times down and plan caches focused on the queries each group actually serves: ``` from chalk import NamedQuery from src.models import User NamedQuery( name="fraud", input=[User.id], output=[User.denylisted], resource_groups=["gpu-workers"], ) ``` With this configuration, only query servers whose CHALK_RESOURCE_GROUP is gpu-workers preplan the fraud query. Named queries without resource_groups keep the default behavior and are preplanned everywhere. You can also scope resource groups per environment by passing a mapping from environment id to resource groups. Resource groups under the special key "__all__" apply in every environment, while resource groups under an environment id apply only in that environment. Passing a plain list, as above, is shorthand for {"__all__": [...]}. ``` NamedQuery( name="fraud", input=[User.id], output=[User.denylisted], resource_groups={ # Preplanned by 'default' servers in every environment. "__all__": ["default"], # Additionally preplanned by 'gpu-workers' servers in production. "cmproduction123": ["gpu-workers"], }, ) ``` An environment with no matching entry does not preplan the query at all: in the example above, a staging environment's gpu-workers servers would skip the fraud query. ### Caching Ad-hoc Query Plans Defining NamedQuery objects is the recommended way to ensure that your queries will be pre-planned on start-up, so that their planning time will not impact your query latency. By default, the environment variable CHALK_PRE_PLAN_NAMED_QUERIES=1 should be set to enable this. However, sometimes defining NamedQuery objects is not ergonomic or possible. For example, if you are a platform team serving multiple teams, you may not want to define a NamedQuery object for every query that your users run. In this case, you can cache ad-hoc query plans by setting the following engine configuration variables: ``` CHALK_STORE_ADHOC_QUERIES=true CHALK_PLAN_ADHOC_QUERIES=3 ``` The first environment variable will enable writing down query requests to the database. Setting the second environment variable to 3 will make the engine pod plan up to 3 of the most recently saved ad-hoc queries. These ad-hoc queries are re-planned at boot, so code or platform changes will be reflected in the query plan. With ad-hoc query caching enabled, you can cache the sketches of your most frequent queries without defining the queries in code. ### Durable Plan Cache The downside of caching and pre-planning ad-hoc query plans is that the pre-planning a large number of query plans during boot can take a lot of time. To help alleviate this, the Durable Plan Cache can be used. Query plans (as opposed to query requests) can be serialized and written down to the Durable Plan Cache. These plans will persist across pods and can be pre-loaded (as opposed to planned) into the in-memory plan cache on next pod startup. Since pre-loading is faster than planning, pod startups will take less time. You can configure the Durable Plan Cache with the following engine configuration variables: ``` CHALK_PERSIST_DURABLE_PLAN_CACHE=true CHALK_PREPOPULATE_DURABLE_PLAN_CACHE=10 CHALK_PREPOPULATE_DURABLE_PLAN_CACHE_DURATION=259200 ``` CHALK_PERSIST_DURABLE_PLAN_CACHE enables writing down query plans to the Durable Plan Cache. Then, you can choose how to load query plans on new pods using either CHALK_PREPOPULATE_DURABLE_PLAN_CACHE=k, which will load the top k most recent query plans written, or CHALK_PREPOPULATE_DURABLE_PLAN_CACHE_DURATION={DURATION_IN_SECONDS} which will load all query plans written within the specified duration since now. Query plans written by a resource group in a deployment will only be valid for that resource group in the same deployment. # Authentication source: https://docs.chalk.ai/docs/online-authentication ## Access the Chalk API. Chalk implements OAuth for authentication to the online query interface. Two kinds of credentials can be used to access Chalk resources: - Personal credentials: Full access to all resources on your account. A client with these credentials can act as you. - Service credentials: Access scoped to a specific project and environment. Machines use these credentials to talk to Chalk, and you generate them in the web dashboard. Both personal and service credentials can be used to query Chalk, and potentially to modify your Chalk deployment's settings. This means that these credentials are sensitive and must be kept secret. You can create and manage service credentials in the Chalk dashboard or using the Chalk CLI. When you use the CLI to create credentials, you will be asked to authenticate yourself on Chalk's web dashboard. Then, you will receive a client_id and client_secret. Once generated, client_id cannot be changed. However, client_secret can be rotated if your security practices require this or if you suspect that client_secret has been compromised. Once you have generated your client_id and client_secret, you can make authenticated requests to Chalk. ### Authenticating an API client Chalk has published API client libraries for several languages. These libraries handle exchanging a client_id and client_secret for an access_token which can be used to access Chalk. ### Authenticating curl We recommend using the Chalk CLI to authenticate a curl request. You can use chalk token to acquire an access_token that is suitable for use as a Bearer token: ``` curl -H "Authorization: Bearer $(chalk token)" \ https://api.chalk.ai/v1/who-am-i ``` ### Fetching an access token If you are implementing a custom API client for a language that Chalk has not published a library for, you may need to fetch an access_token using the OAuth Client Credentials grant flow. You can use the token endpoint in Chalk's API to execute this flow: ### Request Your client_id Your client_secret The grant_type field must always be "client_credentials". ### Response The access_token that you should use in the Authorization header for authenticated requests. Number of seconds until the access_token expires. This field will always be "Bearer" ### Creating a service token in the dashboard To create a service token in the Chalk dashboard, go to Settings > Access tokens and select New token. Name the token and select its permissions. You can grant only the permissions that you hold yourself. The Routing tags and Permission tags sections are collapsed until you open them. Routing tags select the resources that the token's queries use, and permission tags set the token's datasource and feature access. ### Authenticating a request using an access token Use the token obtained from the Client Credentials grant flow in the Authorization: Bearer header that your client sends along all authenticated requests. For example: ``` curl -H "Authorization: Bearer " https://api.chalk.ai/v1/who-am-i ``` will return a 200 response and a JSON object containing a short description of the requesting user. This is convenient for verifying that you are using a valid access_token. ### Role-based access control (RBAC) Use role-based access control to limit which data an agent can use when it queries Chalk. Access control has two parts: - Datasource permissions gate access to the datasources used by a query. - Feature permissions gate access to the features computed and returned by a query. Both systems are tag-based. Use datasource permissions for broad boundaries such as warehouses, databases, vendors, regions, or product areas. Use feature permissions when sensitive and non-sensitive values live in the same datasource and need feature-level control. ### Shared tag model Datasources and features carry tags, and roles or service tokens map those tags to permissions. - Features use the standard feature tags argument:from chalk.features import feature, features @features class User: id: int ssn: str = feature(tags=["pii"]) - Datasources use permission_tags:from chalk.sql import SnowflakeSource signups = SnowflakeSource( name="SIGNUPS", permission_tags=["warehouse:signups"], ) The same tag-permission map is shared by datasource permissions and feature permissions. For example, a role or service token can map pii to AllowInternal and warehouse:signups to Allow. When Chalk checks a resource, tags that are not configured on the role or token are ignored. If multiple tags have explicit permissions, the most restrictive wins: ``` Deny > AllowInternal > Allow > AllowDownstream ``` ### Default permission If none of a datasource or feature's tags have an explicit permission, Chalk uses the role or service token's default permission. The default does not participate in the most-restrictive comparison when at least one tag has an explicit permission. You can set the default to Allow, AllowInternal, or Deny, but not AllowDownstream: - Default Allow: Features are returnable and datasources are accessible unless their tags specify otherwise. This is the default. - Default AllowInternal: Features can be computed and used as intermediate inputs, but cannot be returned. Datasources are accessible unless a tag explicitly maps to Deny or AllowDownstream. - Default Deny: Features cannot be computed or returned, and datasources cannot be accessed, unless their tags explicitly grant the required access. For Online and Offline queries, a query is permitted only if the requesting agent has access to every datasource and feature needed by the query plan. Datasource permissions are checked for each selected datasource. Feature permissions are checked for requested outputs and for the intermediate features needed to compute them. To keep policies easy to audit, avoid reusing the same tag name for unrelated datasource and feature policies. For example, prefer separate names such as warehouse:signups and feature:pii when the tags represent different access boundaries. ### Datasource permissions Datasource permissions control whether a query can use a datasource at all. A datasource permission tag grants access when it maps to Allow or AllowInternal. Datasource permission tags currently apply only to SQL datasources. They are enforced for Chalk SQL queries; feature permissions are not currently enforced for Chalk SQL. Dashboard-defined tags are available for the supported SQL integrations shown on the Data sources page. The datasource outcomes are allow and deny. Allow and AllowInternal grant datasource access. Deny and AllowDownstream do not grant datasource access. If a datasource has no tags with explicit permissions on the role or token, Chalk uses the default permission. Default Allow and AllowInternal grant access to that datasource. You can define datasource permission tags in code: ``` from chalk.sql import SnowflakeSource signups = SnowflakeSource( name="SIGNUPS", permission_tags=["warehouse:signups"], ) ``` For supported integrations, we recommend defining datasource permission tags in the Chalk dashboard. Changing a dashboard integration requires permission to manage datasources, so the dashboard provides a permission-gated access-policy boundary. It is also a natural place for the policy when it owns the full datasource configuration, such as account IDs, authentication, or warehouse details. A dashboard-defined datasource still needs to be referenced by name in code when it is used by a Chalk app. Use an explicit datasource name and make sure the dashboard integration name and ChalkPy datasource name match exactly, including case: ``` from chalk.sql import SnowflakeSource signups = SnowflakeSource(name="SIGNUPS") ``` Changes to datasource permission tags in the dashboard take effect only after the next deployment. Chalk resolves the effective tags and injects them into the query servers at deploy time, so redeploy the environment after adding, changing, or removing dashboard-defined permission tags for those changes to take effect. When code and the dashboard both define permission tags for the same datasource, the complete non-empty dashboard tag set takes precedence. Code-defined tags are used only when that datasource has no dashboard-defined tags. Consequently, removing all tags in the dashboard exposes any code-defined fallback tags after the next deployment; remove those code tags too if the datasource should remain untagged. Datasource permission tags are different from routing tags. Routing tags, shown in the dashboard as Routing Tags: Resolver and Datasource Selection, control resource selection for a query: they make matching tagged resolvers eligible to run and select a matching route in a tagged SQL source group. They do not authorize access. Datasource permission tags authorize the selected datasource, and feature permission tags authorize the features that can be computed or returned. The permission-tag mapping is encoded on the requesting agent through its assigned role or service token. Feature tags do not have routing capabilities. Clients can also provide routing tags directly on a query. The dashboard setting adds routing tags from the requesting agent's JWT, including for service tokens and SCIM groups; it does not restrict the other routing tags a client can provide. ### Feature permissions Feature permissions control which features can be computed and returned. You tag your features, then a service token or role maps each tag to one of four permissions. | Permission | Usable | Returnable | Declassifies | | --------------------- | ------ | ---------- | ------------ | | **`Deny`** | No | No | No | | **`AllowInternal`** | Yes | No | No | | **`Allow`** | Yes | Yes | No | | **`AllowDownstream`** | Yes | Yes | Yes | - Usable: The feature can be computed and used as an input to other features, that is, it can appear in the query plan. - Returnable: The feature can be returned to the query's caller. - Declassifies: The feature clears the features derived from it, making them returnable too. Allow, AllowInternal, and Deny each apply to the single feature they tag: - Allow: The feature can be computed and returned to the caller. - Deny: The feature cannot be computed or used at all. Any query that needs it, whether as an output or as an intermediate input, fails. - AllowInternal: The feature can be computed and used as an input to other features, but is never returned to the caller. Use it for sensitive raw values that should stay inside the query. These three apply to that one tagged feature only. The permission does not flow to features computed from it. AllowDownstream is the exception: a feature tagged AllowDownstream is returnable like Allow, but it also acts as a declassification boundary, clearing the features derived from it. Because that behavior spans the feature graph, it has its own section below. A query is permitted only if every requested output feature resolves to Allow or AllowDownstream; an AllowInternal or Deny output fails the query. ### AllowDownstream: the declassification boundary AllowDownstream is the one permission that crosses the feature graph. A feature tagged AllowDownstream is returnable, and it declassifies features derived from it under a strict rule: a derived feature is cleared (resolves to AllowDownstream, and so becomes returnable) only when every one of its inputs is AllowDownstream. Declassification therefore propagates along chains whose entire input set is cleared, and stops the moment a non-cleared input is mixed in. This clearance applies only to feature outputs. It does not override datasource permissions: every datasource selected to compute the query must independently have a matching Allow or AllowInternal permission. AllowDownstream never authorizes datasource access. That same all-inputs rule is what lets AllowDownstream override a Deny: a feature tagged Deny is still cleared if all of its inputs are AllowDownstream, because the output is then a pure derivative of already-cleared data. The canonical use case is an aggregation over sensitive data: the individual records are private, but a statistic over them is safe to expose. Here the raw transaction amounts are PII (AllowInternal, never returned), and the per-user average over them is declassified: ``` from chalk import _ from chalk.features import features, feature, DataFrame @features class Transaction: id: int user_id: "User.id" # AllowInternal: usable to compute other features, but never returned on its own. amount: float = feature(tags=["pii"]) @features class User: id: int transactions: DataFrame[Transaction] # AllowDownstream: an aggregate over sensitive amounts, safe to expose. avg_transaction_amount: float = feature( tags=["cleared"], expression=_.transactions[_.amount].mean(), ) ``` With a token mapping pii -> AllowInternal and cleared -> AllowDownstream: ``` $ chalk query --in user.id=1 --out user.avg_transaction_amount # ok: cleared by AllowDownstream $ chalk query --in transaction.id=1 --out transaction.amount # rejected: AllowInternal, not returnable ``` The override does not apply when the inputs are mixed. Take a feature computed from three inputs that resolve to Allow, AllowDownstream, and Deny: - The Deny input taints the result: any Deny in the lineage forces the output to Deny, so it is blocked regardless of the AllowDownstream input. - Even setting the Deny input aside, a mix of Allow and AllowDownstream is not all-AllowDownstream, so the result is not cleared: it falls back to its own tag (or the default) and does not declassify features further downstream. In short, AllowDownstream clears a derivative only when the entire lineage feeding it is cleared. ### Best practices - Use datasource permissions for coarse access boundaries, such as a warehouse, database, vendor, region, or product area. - Use feature permissions when sensitive and non-sensitive data live in the same datasource or when policy depends on the meaning of a specific feature. - Prefer defining datasource permission tags in the dashboard. Dashboard changes are permission-gated and take precedence over code-defined tags; use code tags only as intentional fallbacks. - Keep datasource and feature tag names distinct unless you intentionally want one role/token entry to control both. - AllowInternal grants datasource access and retains its feature-level behavior for feature permissions. AllowDownstream does not grant datasource access. - To keep feature policies easy to audit, prefer declaring feature permission tags in one central place. See Centrally manage feature-permission tags. # Query Caching source: https://docs.chalk.ai/docs/query-caching ## Override default max-staleness for individual requests. Features may specify a default caching strategy through the maximum staleness parameter (see Feature Caching for a more in-depth discussion). However, a request may have freshness requirements that differ from the default caching period specified at the feature level. You can specify maximum staleness requirements for any set of features, even features you do not wish to return. This facility may be useful if a feature value is used as an intermediate result for an explicitly requested feature. ### Example Consider the following feature definition: ``` @features class User: fico_score: int = feature(max_staleness="**30d**") ... ``` For most models, it may be acceptable to use a thirty-day old version of the FICO score. However, if you were running a model for issuing a new line of credit, you may require a more updated version of the feature. By default, our models will receive a FICO score computed in the last 30 days, but we can override this behavior at the time of requesting features for the credit model: ### Handling null and default values in the cache You can specify whether to cache null or default feature values using the cache_nulls and cache_defaults parameters in the feature definition. Additionally, customers with a DynamoDB or Redis online store can specify feature(cache_nulls="evict_nulls") to evict any entry that would have been null from the cache, and similarly feature(cache_defaults="evict_defaults") to evict any entry that would have been a default value from the cache if it exists. For more information on how to configure these parameters for your use case, check out our API documentation. ### Cache busting Cache busting is a special case of overriding the maximum staleness at the time of making a query. To bypass the cache, simply provide "0s" as the max-staleness. # Offline Queries source: https://docs.chalk.ai/docs/query-offline ## Fetch offline feature values. Offline queries can pull data from the offline store, or run offline and online resolvers to compute feature datasets, that can be used for model training, backfilling, and experimentation. Offline queries are highly configurable, and can be used for most offline workflows. In an offline query, you can guarantee temporal consistency by providing input_times, to fetch the latest values for each feature observed before the input time. You can also set recompute_features to compute the freshest feature values for some or all output features, running offline and online resolvers. For offline queries, the query planner will preference an offline resolver if present, but otherwise will still run online resolvers to compute feature values. Offline queries can be used directly for computing feature values to be stored in the online or offline store, as well as creating datasets that can be used for model training. Providing a dataset_name in an offline query creates versioned revisions of Datasets, including metadata about each offline query used to create each revision of a dataset. If an offline query should be run on a schedule, then the same query can be defined and deployed in code as a Scheduled Query. Chalk supports a number of clients for offline queries. In this section, our examples use our Python client, which integrates with Jupyter notebooks. ### Making offline queries As mentioned earlier, offline queries can be made through one of Chalk's API clients. ``` from chalk.client import ChalkClient from datetime import datetime client = ChalkClient() dataset = client.offline_query( input={'user.id': [1,2,3,4]}, # Input output=['user.name'], # Output # tags=['test'], # Environment # branch='branch', # recompute_features=True, # Run Resolvers # run_asynchronously=True, # Run Configuration # max_samples = 10, # lower_bound=datetime(2024, 10, 12), # Bounds # upper_bound=datetime(2024, 10, 20), ) df = dataset.get_data_as_pandas() ``` Offline queries return Chalk Datasets, which we cover in more detail in the next section. At a high level, Chalk Datasets are flexible wrappers for the results of your offline query. They can be converted to pandas or polars DataFrames for ease of use in downstream ML tasks. ### Input Offline queries can receive input from a DataFrame, mapping, or SQL query. Regardless of the format, the primary key feature of the feature class must be included in the input. ### DataFrames and mappings as input The input parameter accepts chalk.DataFrame, pandas.DataFrame, or a mapping (like a Python dictionary). DataFrames should include one column for each known feature in the input. Mappings should map from each feature to a list of its values: ``` input={ User.id: ['id1', 'id2'], User.age: [23, 40] } ``` ### Input times Timestamps can be passed in the input_times argument. If none are provided, Chalk will use the current time. ``` input={ User.id: ['id1', 'id1'], } input_times=[datetime.now() - timedelta(days=1), datetime.now() - timedelta(days=2)] ``` ### SQL queries as input offline_query can also take input_sql as an alternative input parameter for computing input rows dynamically with Chalk SQL. Use input_sql when you want to build a complex input spine from catalog tables, data sources, or zero-argument resolvers instead of materializing the input in Python first. input_sql is recommended in the following scenarios: - You want to retrieve data for multiple rows at once. Chalk will compute an efficient query plan for loading multiple rows of data at once. Chalk will also reuse data between rows where appropriate. - You want to query from Chalk or another registered data source to build your offline query input. You can reduce unnecessary back-and-forth requests by having Chalk execute the SQL query and handle the result rows directly. - You want to request features from multiple feature namespaces for each row of output. input_sql queries accept features from multiple feature namespaces as columns of the result. Each column must either correspond to an existing feature or be included in the output list. For each referenced feature namespace, the feature namespace's primary key must be included as a column. The __ts__ column is interpreted as the query execution time for the row. See our documentation on temporal consistency for more details, or the SQL as Input guide for more input_sql examples. ``` # This input_sql queries the table `transactions` through the registered Snowflake datasource. cutoff = (now - timedelta(days=30)).strftime("%Y-%m-%d %H:%M:%S") output = chalk_client.offline_query( input_sql=f""" SELECT * FROM datasource_query( 'my_snowflake', ' SELECT t.txn_time AS "__ts__", t.seller_id AS "seller.id", t.buyer_id AS "buyer.id", t.amount AS "txn.amount", t.payment_type AS "txn.payment_type" FROM transactions AS t WHERE t.update_at >= TO_TIMESTAMP_NTZ(''{cutoff}'') ' ) """, output=[ Seller.id, Buyer.id, Buyer.account_created_date, Txn.payment_type, # Computed in the seller namespace from the 'seller.id' spine feature. Seller.recent_transactions_volume, # Computed in the buyer namespace from the 'buyer.id' spine feature. Buyer.total_spent_last_30d, # Passed through from the SQL query Txn.amount, ], ) ``` ### Output This argument describes a list of features to sample. ``` output=[ User.returned_transactions_last_60, User.user_account_name_match_score, User.socure_score, User.identity.has_verified_phone, User.identity.is_voip_phone, User.identity.account_age_days, User.identity.email_age, ] ``` ### Recompute features Users can request that features (or a subset of features) be recomputed by resolvers at query time instead of sampled from the offline store. For more information, see recompute_features. ### Timebounds In some cases, users may not have a list of primary keys to sample with, and instead would like to see results within a period of time. The user can then leave the inputs argument empty and supply a lower bound and an upper bound along with the requested output features. The parameter max_samples can also be used to limit the number of rows returned. ``` dataset: Dataset = ChalkClient().offline_query( output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], lower_bound=datetime.now() - timedelta(days=7), upper_bound=datetime.now(), ) ``` ### Environment The user can specify tags, environment, or branch as parameters to offline_query in the same fashion as online query. ### Debugging offline queries Like online queries, offline queries can be debugged using the explain flag, which logs the query execution plan, and the store_plan_stages flag, which stores the inputs and outputs of each of the plan stages, which will be visible in the web dashboard's query detail view. In addition, the offline query output is a Dataset object that you can inspect within the dashboard using the Output Explorer. With the Output Explorer you can run SQL queries on the output to do more testing and validation. More on these and other tips here. ### Running large offline queries Offline queries that run on a large number of rows require their own compute resources and infrastructure to ensure they complete and do not contend with resources from other workloads like the engine or branch server. Chalk handles this by launching a Kubernetes job that runs asynchronously on dedicated compute nodes. To do so, set the run_asynchronously flag to True in the offline_query method. ``` from chalk.client import ChalkClient client = ChalkClient() client.offline_query( input={'user.id': range(1_000_000)}, output=['user.name'], run_asynchronously=True, ) ``` If you'd like to adjust the compute or memory resources for the job, set the resources parameter to a custom ResourceRequests object. Ensure that the request values are compatible with the resources available on Kubernetes cluster. By default, asynchronous offline queries will be picked up by a Job Queue Consumer worker, so the resource requests can override the Job Queue Consumer configurations. Another option to consider is to parallelize the computation of the offline query by dividing the input into shards. If the num_shards parameter is set, Chalk will split the input into num_shards shards and run each shard in a separate pod. The num_workers parameter sets the maximum number of pods that can run concurrently. If the number of shards is large compared to the size of the Kubernetes cluster, set the num_workers parameter to a smaller number to ensure that Chalk can launch enough pods to run the query. ``` from chalk.client import ChalkClient client = ChalkClient() client.offline_query( input={'user.id': range(1_000_000)}, output=['user.name'], run_asynchronously=True, num_shards=50, num_workers=10, ) ``` For the above example, Chalk will split the input spine into 50 shards of 20,000 rows each. It will then launch jobs for each shard, while not exceeding 10 concurrent pods at once. If you lose track of the state of a long-running offline query, you'll always be able to retrieve the outputs later. First, check the offline query dashboard: you'll be able to download all shards of the output, individually or as a group, there. You can also retrieve the outputs via Python with the ChalkClient.get_dataset method. If you pass in a dataset_name when running the offline query, then you can use the dataset name, otherwise the revision_id for your dataset will use the Query Id from the offline query dashboard which you can also use to retrieve the dataset in your function call. ``` from chalk.client import ChalkClient client = ChalkClient() dataset = client.get_dataset(revision_id="21d9ceb7-c1ef-4c08-a39c-f8381e825a66") # your query id df = dataset.get_data_as_pandas() ``` ### Disk spilling for memory-intensive queries When an offline query processes more data than can fit in memory, the execution engine can spill intermediate results to local disk rather than failing with an out-of-memory error. The engine reads the data back as needed to complete the query, trading some I/O overhead for the ability to handle larger workloads. Spilling works best on async offline-query workers backed by local NVMe SSDs. For the end-to-end setup, including node pools, resource sizing, the required spill-mode environment variable, and optional Iceberg scan caching, see Local SSDs for spilling and scan caching. At a high level, enable spilling on those workers by setting: ``` CHALK_VELOX_QUERY_SPILLING_MODE=Always ``` After enabling it, verify that spilling is actually active by checking the query's performance summary for spill_enabled=true and nonzero spilled_bytes when the query completes. Spilling only applies to offline queries. Online queries always run entirely in memory to preserve low-latency serving. ### Optimizing aggregations in offline queries When computing aggregations over large datasets in offline queries, your offline query may have to perform computationally expensive joins. To optimize these joins, Chalk provides a planner option use_materialized_offline_query that you can specify that follows similar bucketing logic to Materialized Windowed Aggregations used in online queries. When this option is specified, the Chalk engine will bin the events over which you are aggregating into appropriately sized buckets based on timestamps and feature groupings, compute the bucket-level aggregations, and then merge the bucket-level aggregations to produce the final result. This approach can significantly reduce the amount of data that needs to be processed during joins, leading to faster offline query execution times, but the performance gain is a tradeoff with some loss of precision in the calculation of the aggregation. ``` from chalk.client import ChalkClient client = ChalkClient() ds = client.offline_query( input={'user.id': big_input_list}, input_times=big_input_time_list, output=[ User.recent_transactions, User.total_spent_last_30d, User.recent_spending_score ], planner_options={ 'use_materialized_offline_query': '1' }, ) ``` # Datasets source: https://docs.chalk.ai/docs/datasets ## Persist and evolve offline queries over time The Chalk Dataset class governs metadata related to offline queries, supports revisions to queries over time, and enables the easy retrieval of data from the cloud. Evaluations use datasets of their own, which share the name but hold plain tabular data rather than feature values. The readers on this page do not apply to them. ### Datasets from offline query Dataset instances are obtained by calling ChalkClient.offline_query() which computes feature values from the offline store. If inputs are given, the method returns the values corresponding to those inputs. Otherwise, the method returns a random sample according to the parameter max_samples, or features from within timebounds specified by lower_bound and upper_bound. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) sample_dataset: Dataset = ChalkClient().offline_query( output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], max_samples=10, lower_bound=datetime.now() - timedelta(days=7), upper_bound=datetime.now(), dataset_name='my_sample' ) ``` Here, we attach a unique name to the Dataset. Whenever we send additional queries with the same name, a new DatasetRevision instance will be created and attached to the existing dataset. If a dataset_name is not given, the output data won't be retrievable beyond the current session. A Dataset's revisions can be inspected in Dataset.revisions: they hold useful metadata relating to the offline query job and the data itself. Be sure to check out Dataset.errors for any errors upon submitting the query. ### Retrieving output data Since offline queries are not realtime, the Dataset instance returned is not guaranteed to have the outputs of the query instantaneously. Thus, loading the data may take some time. The data can be accessed programmatically by calling Dataset.get_data_as_pandas(), Dataset.get_data_as_polars(), or Dataset.get_data_as_dataframe(). If the offline query job is still running, the Dataset will poll the engine until the results are completed. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) pandas_df: pd.DataFrame = dataset.get_data_as_pandas() polars_df: pl.LazyFrame = dataset.get_data_as_polars() chalk_df: chalk.features.DataFrame = dataset.get_data_as_dataframe() ``` The file outputs of the query themselves can also be downloaded to a specified directory. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) dataset.download_data('my_directory') ``` By default, Dataset instances fetch the output data from their most recent revision. A specific DatasetRevision's output data can be fetched using the same methods. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) for revision in dataset.revisions: print(revision.get_data_as_pandas()) ``` ### Dataset Inputs Dataset objects also store the inputs for each revision. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) df = dataset.get_input_dataframe() ``` ### Managing Datasets ### Renaming Datasets You can rename a dataset from the Chalk dashboard to better organize your datasets or reflect changes in your workflow. To rename a dataset: - From the Datasets page under the Offline section in the sidebar, click on a dataset to open its detail page - Click the Edit button in the page header next to the dataset name - Enter the new name and click Save, or press Enter The dataset name is used when retrieving datasets via the API, so update any references in your code after renaming. ### Archiving Dataset Revisions Archiving a revision cannot be undone. Archived revisions will no longer be accessible via the API. As datasets accumulate revisions over time, you can archive older or buggy revisions to keep your dataset history organized. Some metadata about your revision will still be accessible through the dashboard but revisions will be permanently inaccessible after archiving. To archive a revision: - From the Datasets page under the Offline section in the sidebar, click on a dataset to open its detail page - Select the Revisions tab - Select the checkboxes next to all revisions you want to archive - An Archive button will appear in the actions bar at the top of the page. Click this button and confirm the action in the dialog. To view archived revisions, check the Include archived revisions checkbox above the revisions table. Datasets are stored as parquet files in the storage buckets in your cloud. Archiving a revision does not delete the data there — to conserve storage, you can set an expiration policy directly on those buckets. ### Recompute a Dataset Datasets expose a recompute method that enables users to see the results of updates to resolvers/features in the context of this dataset. recompute takes a list of features as an argument to be recomputed, and any other required input features are sampled from the offline store. ### Promote a Dataset to Iceberg A dataset revision's output is stored as Parquet. promote_to_iceberg generates standard Apache Iceberg metadata that references those Parquet files in place — no data is rewritten. The Iceberg metadata.json and manifests are written to a sibling iceberg/ prefix next to the revision's outputs and reference the existing Parquet files. ``` from chalk.client import ChalkClient dataset = ChalkClient().get_dataset(dataset_name="my_dataset") result = dataset.promote_to_iceberg() # gcs:///job_/query_/iceberg/metadata/00001-.metadata.json print(result.metadata_location) print(result.num_rows, result.num_data_files, result.snapshot_id) ``` The returned metadata_location points at the table's metadata.json. The table is not registered in any Iceberg catalog — you register metadata_location in your own catalog (Snowflake, Glue, Polaris, BigLake, …) for name-based discovery. Because it is a standard Iceberg table, any Iceberg-aware client (Snowflake, Spark, DuckDB, PyIceberg, Trino) can read it. ### Example: reading a promoted dataset from Snowflake Snowflake reads the promoted table through an object-store catalog integration plus an external volume pointing at the dataset storage bucket. One-time setup (Snowflake, ACCOUNTADMIN): ``` CREATE CATALOG INTEGRATION IF NOT EXISTS chalk_iceberg CATALOG_SOURCE = OBJECT_STORE TABLE_FORMAT = ICEBERG ENABLED = TRUE; CREATE EXTERNAL VOLUME IF NOT EXISTS chalk_datasets ALLOW_WRITES = FALSE STORAGE_LOCATIONS = ( (NAME = 'chalk' STORAGE_PROVIDER = 'GCS' STORAGE_BASE_URL = 'gcs:///') ); -- Copy the STORAGE_GCP_SERVICE_ACCOUNT value from the output: DESC EXTERNAL VOLUME chalk_datasets; ``` Grant that service account read access to the bucket (GCP side): ``` SA="" gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:$SA" --role="roles/storage.objectViewer" gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:$SA" --role="roles/storage.legacyBucketReader" ``` Per promoted dataset, create an Iceberg table from the metadata file. METADATA_FILE_PATH is the metadata_location returned by promote_to_iceberg() with the gcs:/// prefix stripped: ``` CREATE ICEBERG TABLE my_dataset CATALOG = 'chalk_iceberg' EXTERNAL_VOLUME = 'chalk_datasets' METADATA_FILE_PATH = 'job_/query_/iceberg/metadata/00001-.metadata.json'; SELECT "USER.ID", "USER.SCORE" FROM my_dataset LIMIT 10; ``` Output columns are the dataset's feature fully-qualified names, uppercased and quoted (e.g. "USER.ID"). For an S3-backed dataset bucket, use STORAGE_PROVIDER = 'S3' in the external volume and grant the vended IAM principal read access instead of a GCP service account. ### Integrate Datasets into PyTorch Datasets expose methods create_torch_map_dataset and create_torch_iter_dataset, which create PyTorch datasets from the results of the Chalk dataset. To learn more, see Chalk's PyTorch Integration with Datasets. # Pytorch Integration with Datasets source: https://docs.chalk.ai/docs/pytorch-datasets ## Create PyTorch Datasets from Chalk Datasets Chalk Datasets expose the following methods, which create PyTorch datasets from the Chalk dataset's contents. - Dataset.create_torch_map_dataset(...), which creates a map-based PyTorch dataset. - Dataset.create_torch_iter_dataset(...), which creates an iterable Pytorch dataset. These functions can be used to integrate the results of your Chalk query directly with your PyTorch workflows. Check out the MNIST Dataset Example for example code that can do this. These functions will create datasets that return values in pydict format: {"column_name": torch_tensor}. These column names will by default be the column names of the dataset, but can be mapped by specifying the columns= kwarg in the above function calls. Note that when using a data loader with a batch_size >= 1, each batch when iterating over the data loader will give a mapping from column names to tensors of size batch_size. ### Map-based vs Iterable Datasets In PyTorch, the map-based torch.utils.data.Dataset and iterable torch.utils.data.IterableDataset are the two different types of datasets that can be used in DataLoaders. Map-based datasets are optimized for random access on any given row. In Chalk, Dataset.create_torch_map_dataset(...) will materialize the entire dataset upon the underlying operation's completion. However, if you do not need random access on your dataset, it may be more appropriate to use Dataset.create_torch_iter_dataset(...), which will only materialize one row group of the Chalk dataset at a time by default. # Metaplanning source: https://docs.chalk.ai/docs/metaplanning ## Automated offline query sharding Metaplanning is a feature that automates shard assignments for select offline queries. This feature is automatically enabled by default for scheduled queries that have num_shards set. Contact our support team to enable metaplanning for all scheduled queries in your environments, including ones without an explicit num_shards set. Any async offline query can be metaplanned by setting use_metaplaner=True in the API invocation. ### How It Works Offline queries selected for metaplanning go through a metaplanning workflow: Example: A query with 100,000 rows and the default target of 10,000 rows per shard creates 10 parallel shard jobs. ### Configuration The shard size can be controlled via the CHALK_AUTOSHARDER_TARGET_ROWS_PER_SHARD environment variable (default: 10,000 rows per shard). For scheduled queries: - If num_shards is set, metaplanning is automatically engaged. - If num_shards is not set, metaplanning can still be engaged as an environment setting. For async offline queries: - If use_metaplanner=True, the query will be metaplanned. ### Scheduled query example ``` ScheduledQuery( name="daily_user_scores", output=[User.id, User.score], schedule="0 0 * * *", ) ``` With metaplanning enabled, this query will: - Compute shard keys to find all User IDs (since no input is specified) - Autoshard the IDs based on the configured target - Execute all shards in parallel ### Async offline query examples ``` client.offline_query( input_sql='SELECT "user.id" FROM "chalk.resolvers.list_users"', output=[User.id, User.score], recompute_features=True, run_asynchronously=True, use_metaplanner=True, ) ``` This query will: - Compute shard keys to find all User IDs by executing the input_sql. - Autoshard the IDs based on the configured target - Execute all shards in parallel ``` client.offline_query( max_samples=1000, output=[User.id, User.score], recompute_features=True, run_asynchronously=True, use_metaplanner=True, ) ``` This query will: - Compute shard keys to find 1000 User IDs. - Autoshard the IDs based on the configured target - Execute all shards in parallel ### See Also - Query Scheduling - Creating scheduled queries - ScheduledQuery - API reference - SQL as Input - Using SQL as input # SQL Interface source: https://docs.chalk.ai/docs/sql-interface ## Querying Chalk with SQL. Chalk is testing a direct SQL query interface for your feature values and data sources in your environment. In the dashboard, under Data Explorer, you can find several interfaces that you can use to query your data. The Online Query Explorer and Offline Query Explorer enable you to execute online queries and offline queries, respectively, from the dashboard. In addition, you can use the SQL Query Interface to query against previously computed feature values from your offline store as well as data sources in your environment. Additionally, you can use the GRPC client's run_sql() endpoint. ### Query Syntax Chalk's SQL interface uses a standard SQL dialect, and all queries are processed through a parser that expects familiar SQL syntax. Currently, Chalk SQL only supports making SELECT queries, with the following restrictions: - SELECT list:Column references cannot be qualified yet. If a column in a table has a . delimiter, you may reference it with double quotes, e.g. SELECT "column.with.dot" FROM table.A subset of Chalk's expressions are automatically exposed as top-level functions in the SQL interface. - FROM clause:You can query tables that are exposed by the Chalk catalog. See SQL Interface Sources for more details.You can also query against the following table-valued functionsread_parquet(path: str): Reads a Parquet file from the given path and returns a table. Path must be a literal string.query_values_from_operation_ids(operation_id: str, ...): Returns a table consisting of the values queried in the given operation id(s). The operation id(s) provided must correspond to queries that have the same output schema. Values are available for online and cron queries run while the deployment persists query values (CHALK_PLANNER_PERSIST_VALUES_OFFLINE_STORE=true). Offline query runs do not write these value tables; use get_dataset_givens to view an offline query's inputs.get_dataset_givens(revision_id: str): Returns a table of the inputs (givens) that a dataset revision's offline query ran with — inputs only, with no output columns mixed in. The revision id must be a literal string. - WHERE clause:Column references cannot be qualified yet. As such, JOINs are not possible yet.You can use any of the supported expressions available in Chalk SQL as part of your WHERE clause.A best-effort attempt of pushdown is made for the expressions used in the WHERE clause, but not all expressions are guaranteed to be pushed down to the underlying data source. - GROUP BY clause:At present, you may only use direct column references in the GROUP BY clause. - LIMIT clause:You may use the LIMIT clause to limit the number of rows returned by your query. The value must be a literal integer. - ORDER BY clause:At present, you may only use direct column references in the ORDER BY clause. ### SQL Interface Sources In the SQL Interface, there exist various schema providers, which all exist under the chalk catalog: - chalk.datasources: directly query against the data sources configured in your environment (e.g. Postgres, Snowflake, BigQuery, Iceberg, etc.). Table names are tables within the data sources themselves.Schema: Exactly that of the table in the data source. - chalk.historical_values: query against historically observed feature values with table names of the form feature_namespace.feature_name from your offline store, or query for the freshest values for each primary key for a specific feature namespace with table names feature_namespaceSchema for feature_namespace.feature_name:observation_id, string, non-nullable: The unique identifier for the observation of the feature value.pkey, (type of feature namespace's pkey), non-nullable: The value of the primary feature associated with this observationvalue, (type of feature namespace's value): The observed value of the feature.value_json, string, DEPRECATED: The observed value of the feature, as JSON, for backwards compatibility.observed_at, timestamp, non-nullable: The time at which the feature value was observed.inserted_at, timestamp, non-nullable: The time at which the feature value was inserted into the offline store.resolver_fqn, string, nullable: The resolver that computed this feature.deployment_id, string, nullable: The deployment id where the observation was madeSchema for feature_namespace:__pkey__: (type of feature namespace's pkey), non-nullable: The value of the primary feature associated with this row, (type of feature), for each feature: The freshest observed value of the feature for the primary key..__observed_at__, timestamp, for each feature: The time at which the freshest value was observed. - chalk.query_log: A schema provider consisting of one table, data, which is the query log table. This table provides a list of query metadata for each operation id, including which tables in the offline store correspond to values queried in that operation id. Schema for data:operation_id, string, non-nullable: The unique identifier for the operation.environment_id, string, non-nullable: The environment id where the operation was run.deployment_id, string: The deployment id where the operation was run.operation_kind, string, The kind of operation that was run (inference, streaming, etc.).query_timestamp, timestamp: The value of now() that was used by the query.execution_started_at, timestamp: The time at which the operation started executing. Null if there were errors during planning.execution_finished_at, timestamp: The time at which the operation finished executing. Null if there were errors during execution.query_status, string: The status of the query (success, error, running, etc.).query_name, string: The query name for named queries, null otherwise.query_name_version, string: The version of the query for named queries, null otherwise.branch_name: string: The branch the query was run on, null if the query was not run on a branch.correlation_id, string: The correlation id for the query, null if the query was not run with a correlation id.value_tables: array of strings: The list of tables in the offline store that correspond to result values from this query. - chalk.resolvers: Trigger a resolver run and return the outputs of a specific resolver, with additional filtering based on the SQL query provided. A table exists for every eligible resolver. To run a resolver this way, it must not take any inputs and return a dataframe consisting of only scalar outputs. Schema: Exactly that of the resolver. Chalk also offers a chalk.information_schema that you can query (similar to many other database offerings), and queries such as SHOW TABLES and SHOW SCHEMAS can also be used. ### Examples If you had the following feature class: ``` @features class User: id: int name: str ``` You could query observed feature values for User.name with user.id = 1 by running the following query: ``` SELECT value FROM "chalk.historical_values.user.name" WHERE pkey = 1 ``` # Chalk Clients source: https://docs.chalk.ai/docs/query-basics ## Fetch feature values via online query. Chalk maintains several client libraries (gRPC) and a REST API for fetching feature values. ### Library support Chalk maintains libraries in several major languages for fetching online feature values. If you need support for a language that we don't support, let us know! We also support a rest API if you'd like to build your own. ### REST API Chalk supports a REST API for querying online features and exposes this endpoint in several API clients. When you execute an online query, resolvers will execute to produce the requested data. Online query will prioritize running online resolvers over offline resolvers to compute features if both are possible. The following endpoint can also be hit with the python ChalkClient by using its query method. For information on how to authenticate the ChalkClient, check out the section on authentication. Read more about the parameters to this method here. ### Request Input features and values are provided at the time of request. For example, primary key-value pairs often designate the subset of data returned. Feature inputs are provided by fully qualified path. Has many features are input as lists, and struct features are input as JSON. An example of passing a user with two credit cards as input: Outputs are the features that you'd like to compute from the inputs. Maximum staleness overrides for any output features or intermediate features. See query caching for more information. The context object controls the environment and tags under which a request should execute resolvers: The environment in which to run the resolvers. Like resolvers, API tokens can be scoped to an environment. If no environment is specified in the query, but the token supports only a single environment, then that environment will be taken as the scope for executing the request. The tags used to scope the resolvers. The preview deployment id. See Preview Deployments for more information. The query name. See NamedQuery for more details. If specified, routes to the relevant branch. See Branches for more information. More information on parameters is available here ### Response The outputs features and any query metadata The name of the feature requested, eg. user.identity.has_voip_phone . The value of the requested feature. If an error was encountered in resolving this feature, this field will be empty. The error code encountered in resolving this feature. If no error occurred, this field is empty. Metadata pertaining to the feature, including the resolver run and whether the result was a cache hit. Errors encountered while running the resolvers. Each element in the list is a ChalkError. If no errors were encountered, this field is empty. Metadata related to the query. Returned if include_meta or explain is set to True. The time, expressed in seconds, that Chalk spent executing this query. The id of the deployment that served this query. The id of the environment that served this query. The short name of the environment that served this query. For example: "dev" or "prod". A unique ID generated and persisted by Chalk for this query. All computed features, metrics, and logs are associated with this ID. Your system can store this ID for audit and debugging workflows. At the start of query execution, Chalk computes 'datetime.now()'. This value is used to timestamp computed features. Deterministic hash of the 'structure' of the query. Queries that have the same input/output features will typically have the same hash; changes may be observed over time as we adjust implementation details. An unstructured string containing diagnostic information about the query execution. Only included ifexplain is set toTrue. ### Query Explanation Chalk offers support for the user for when queries don't work. The first step is always to check to see the response contains any errors. Often, the error message will directly point to the failure. In the case of more complicated queries, queries can be sent with explain=True. This will return a representation of the query plan in the meta return attribute. The user can use this information to verify the resolvers and operators ran during execution. Beware, this will result in slower execution times. Some queries that involve multiple operations might need additional tracking. Users can supply store_plan_stages=True to store intermediate outputs at all operations of the query. This will dramatically slow things down, so use wisely! These results are visible in the dashboard under the "Queries" page. For more information, read the ChalkClient docs here. ### Online Query Bulk Compute feature values for many rows of inputs using online resolvers. This endpoint is similar to the online query endpoint, but takes in lists of inputs and produces one output per row of inputs. This is appropriate when you want to fetch the same set of features for many different input primary keys. The following endpoint can be accessed with the Python ChalkClient by using its query_bulk method. ### Request The request body should be a binary payload containing: - A magic string identifier - Serialized query metadata - Feature data in Apache Feather format When using the ChalkClient, this serialization is handled automatically. For direct HTTP usage, the structure is: - Request inputs are provided as mappings of feature names to lists of values - Each list should have the same length, representing multiple rows of data Input features and lists of values. Each key is a feature name (e.g., "user.id") and each value is a list of values for that feature. All lists must have the same length, where each element represents one row. Has-many features are input as lists within each row element, and struct features (has-one) are input as JSON objects within each row element. Example with simple, has-one, and has-many features: ``` { "user.id": [1, 2], "user.name": ["Alice", "Bob"], "user.profile": [ { "age": 30, "city": "NYC" }, { "age": 25, "city": "SF" } ], "user.cards": [[{ "id": "card1" }, { "id": "card2" }], [{ "id": "card3" }]] } ``` The features that you'd like to compute from the inputs. Same as online query. List of timestamps (ISO format) for each row. If provided, the list must match the length of the input value lists. Each timestamp represents the query time for the corresponding row. Maximum staleness overrides for any output features or intermediate features. See query caching for more information. The context object controls the environment and tags under which requests execute: The environment in which to run the resolvers. The tags used to scope the resolvers. If specified, all required_resolver_tags must be present on a resolver for it to be eligible to execute. The semantic name for the query, e.g., "loan_application_model". See NamedQuery. The version of the named query to execute. A globally unique ID for the query, used in logs and web interfaces. If specified, routes to the relevant branch. If specified, routes to the relevant preview deployment. If true, returns query execution plan in the response metadata. Makes the query slower. If true, stores intermediate outputs at all query plan stages. Dramatically impacts performance. Arbitrary key-value pairs to associate with the query. An immutable context accessible from Python resolvers. See ChalkContext. ### Response The response is a binary payload in Apache Feather format containing the results. A list of query results, where each result contains: A DataFrame containing the scalar feature values for all rows. Each row corresponds to an input row. Columns are feature names, and values are the computed feature values. A map of feature names to DataFrames for has-many features. Errors encountered while running the resolvers. Metadata about query execution including execution duration, deployment ID, query ID, etc. See the online query response documentation for QueryMeta details. ### Offline Query Submit an offline query to compute feature values from the offline store or by running offline/online resolvers. Offline queries are typically used for generating training datasets and run asynchronously. The following endpoint can be accessed with the Python ChalkClient by using its offline_query method. See the offline query documentation for more information. ### Request The request body is a binary payload containing: - Serialized query plan (protobuf) - Query metadata (JSON) - Input data (Apache Feather format) When using the ChalkClient, this serialization is handled automatically. The features for which there are known values. Can be: - A mapping of feature names to lists of values (similar to bulk query format) - A DataFrame with input data - A URI pointing to input data in cloud storage When using a mapping, has-many features are input as lists within each row, and struct features (has-one) are input as JSON objects within each row. See the bulk query input format above for examples. Timestamps for point-in-time correctness. If a list, must match the length of input rows. See temporal consistency. The features to compute or sample. If a feature was never computed for a sample, its value will be null. Features that must exist in each row. Rows where a required output was never stored will be skipped. Controls whether resolvers run to compute features: - If true, all output features are recomputed by resolvers - If false, all output features are sampled from the offline store - If a list, features in the list are recomputed, others are sampled The environment in which to run resolvers. A unique name for the dataset. If provided, the dataset will be saved and can be retrieved later. Maximum number of samples to include in the result. If not specified, all samples are returned. Only query data observed after this timestamp. Accepts ISO 8601 format strings. Only query data observed before this timestamp. Accepts ISO 8601 format strings. Override the feature whose values are filtered against lower_bound and upper_bound. By default, the bounds filter against the FeatureTime feature of each output's namespace. Must reference a scalar or FeatureTime feature. The tags used to scope the resolvers. If specified, routes to the relevant branch. A globally unique ID for the query, used in logs and web interfaces. The name of the query. If provided, creates a named query or fills in missing parameters from a preexisting execution. If true, runs the query in separate Kubernetes pods. Useful for large datasets and long-running jobs. If true, stores the query output in the online store. If true, stores the query output in the offline store. If specified, splits the input across this many shards for parallel processing. Override resource requests (CPU, memory) for the offline query job. ### Response The response contains information about the submitted offline query job. A unique identifier for the offline query job. Use this to check the job status. If a dataset_name was provided, this is the ID of the created dataset. ### Check Offline Query Status Check the status of an offline query job. Offline queries run asynchronously, so you need to poll this endpoint to determine when the job is complete and the results are ready. ### Request The job ID returned from the offline query request (also called revision_id). The name of the dataset, if one was provided in the offline query request. The ID of the dataset. If true, returns results even if some errors occurred during execution. Default is false. If true, skips failed shards and returns results from successful shards only. Default is false. You must provide at least one of: job_id, dataset_name, or dataset_id. ### Response Whether the offline query job has completed. Poll this endpoint until this field is true. A list of short-lived, authenticated URLs to download the query results. These URLs point to data files in cloud storage (S3 or GCS) containing the feature values in a columnar format. Only populated when is_finished is true. Errors encountered during query execution, if any. Version number representing the format of the data. The client uses this to properly decode and load the query results into DataFrames. Current version is 1. Once is_finished is true, you can download the data from the provided URLs and load it into a DataFrame. The ChalkClient's Dataset object handles this automatically. # Model Registry source: https://docs.chalk.ai/docs/model_registry ## Learn how to track, version, and run ML models in Chalk With Chalk you can easily register and load machine learning models into your deployments. You can then run inference on these models. This guide covers how to integrate models into your Chalk applications, including loading existing models and running inference. Chalk has a few key model concepts: - Model Name: A namespace for a model (which might have multiple implementations). - Model Version: A model that has been registered with a specific version number — these are models that can be loaded into Chalk deployments. - Model Artifact: The actual files that make up a model, such as weights, configuration files, and tokenizers. - Model Reference: A reference to a specific model version. Used to load models into deployments and track model performance and feature distributions for a deployed model. Model names must start and end with an alphanumeric character (a-z, A-Z, 0-9) and can only contain alphanumeric characters, hyphens, underscores, and dots in between. Names must be at least 2 characters long. A "latest" alias is automatically added to every new model version. Generally running inference on models in Chalk involves three steps: - Registering models in Chalk - Including models in Chalk deployments - Connecting your models to features for inference ### Registering Models in Chalk To add a model to the Chalk model registry, you can either register an existing model or run training code to create a new model. Once registered, models can be versioned and tracked over time. Registering models through training jobs is covered in the Chalk model training docs. To add a new model namespace to the registry, you can use the client.register_model_namespace method. This namespace can then be used when registering a model version to assign it to this namespace. ``` from chalk.client import ChalkClient client = ChalkClient() client.register_model_namespace( name="RiskScoreModel", description="Risk score model developed in PyTorch" ) ``` ### Registration Modes There are four ways to register a model version, depending on how your model is stored and where you want to run it: | Mode | When to use | Key parameters | | ----------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | **Python object** | Iterative development; Chalk auto-serializes and infers type, encoding, and (for some frameworks) schema and dependencies | `name`, `model` | | **File paths** | Pre-serialized models on disk or in cloud storage | `name`, `model_paths`, `model_encoding`, `input_schema`, `output_schema` | | **Docker image** | Containerized models deployed to scaling groups | `name`, `model_image`, `input_schema`, `output_schema` | | **Chalk Image** | Chalk builds and manages the container image for you | `name`, `model_image`, `input_schema`, `output_schema` | ### Registering From Python Model Objects Models can be registered directly from Python objects, such as scikit-learn or PyTorch models. When you pass a Python model object, Chalk automatically infers the model type and encoding. Schema and dependency inference are partial: input_schema and output_schema are inferred for scikit-learn, PyTorch, XGBoost, and CatBoost, and dependencies for scikit-learn, PyTorch, and XGBoost. Pass these explicitly when you use another framework, or whenever you want named output columns or pinned versions. ``` from chalk.client import ChalkClient from sklearn.ensemble import RandomForestClassifier client = ChalkClient() rfc = RandomForestClassifier() rfc.fit(X_train, y_train) client.register_model_version( name="RiskScoreModel", aliases=["v1.0.0"], model=rfc, metadata={ "framework": "sklearn", "training_date": "2025-07-29", "accuracy": 0.89, "precision": 0.91 }, input_features=[User.age, User.income], output_features=[User.churn_risk], ) ``` For the frameworks listed above, input_schema and output_schema are inferred from the model. Inferred schemas are positional (no column names), so pass them explicitly when you want named columns or when you use a framework that isn't inferred. ### Registering From Local Files To register a model from local files, provide the path to the model files using model_paths. When registering from files, you must explicitly specify the model_encoding and provide input_schema and output_schema since these cannot be inferred. ``` from chalk.client import ChalkClient from chalk.ml import ModelType, ModelEncoding client = ChalkClient() client.register_model_version( name="FraudDetector", model_paths=["./fraud_model.pkl"], model_type=ModelType.SKLEARN, model_encoding=ModelEncoding.PICKLE, input_schema={ "transaction_amount": float, "merchant_category": str, }, output_schema={ "fraud_probability": float, }, aliases=["v1.0", "production"], metadata={"accuracy": 0.96, "training_date": "2025-03-15"}, ) ``` If your model requires additional files for inference, such as tokenizers or configuration files, include them with additional_files. You can also pin the dependencies needed to load and run the model with the dependencies parameter: ``` client.register_model_version( name="SentimentModel", model_paths=["./sentiment_model.pt"], model_type=ModelType.PYTORCH, model_encoding=ModelEncoding.PICKLE, additional_files=["./tokenizer.json", "./config.json"], input_schema={"text": str}, output_schema={"sentiment_score": float}, dependencies=["torch==2.7.1", "transformers==4.40.0"], ) ``` ### Registering From Cloud Storage You can register models stored in cloud storage by providing cloud URIs in model_paths along with a source_config for authentication. From S3: ``` from chalk.ml import S3SourceConfig, ModelType, ModelEncoding client.register_model_version( name="FraudDetector", model_paths=["s3://my-bucket/models/fraud_model.pkl"], model_type=ModelType.SKLEARN, model_encoding=ModelEncoding.PICKLE, input_schema={"transaction_amount": float}, output_schema={"fraud_probability": float}, source_config=S3SourceConfig( aws_profile="my-profile", aws_region="us-east-1", ), ) ``` From GCS: ``` from chalk.ml import GCSSourceConfig client.register_model_version( name="FraudDetector", model_paths=["gs://my-bucket/models/fraud_model.pkl"], model_type=ModelType.SKLEARN, model_encoding=ModelEncoding.PICKLE, input_schema={"transaction_amount": float}, output_schema={"fraud_probability": float}, source_config=GCSSourceConfig(gcp_project="my-project"), ) ``` ### Registering With the @model_handler Decorator For the fastest path to a deployable model, decorate a class with @model_handler, write a predict method, and pass a trained model object — Chalk builds the serving image, ships your class, and mounts the serialized model for you. See The @model_handler decorator in the Model Deployments guide. ### Registering With a Docker Image Models can also be registered as Docker images for deployment to scaling groups. This mode does not require model serialization — Chalk runs your container as-is. You must provide input_schema and output_schema to define the model's data contract. See the Model Deployments guide for details on building images and deploying. ``` client.register_model_version( name="ner-model", input_schema={"text": str}, output_schema={"entities": str}, model_image="ghcr.io/my-org/ner-model:latest", ) ``` ### Registering With a Chalk Image Instead of building and pushing a Docker image yourself, you can submit a chalkcompute Image object and Chalk will build the compute image for you when you register your model. You must provide input_schema and output_schema to define the model's data contract. See the Model Deployments guide for details on deploying. ``` client.register_model_version( name="ner-model", input_schema={"text": str}, output_schema={"entities": str}, model_image=chalkcompute.Image().debian_slim("3.11") .pip_install(["chalk-remote-call-python"]) .add_local_file(HANDLER_FILE_PATH, "/app/handler.py", strategy="copy") .env({"PYTHONPATH": "/app"}) .workdir("/app") .entrypoint( [ "chalk-remote-call", "--handler", "handler.handler", "--port", "8080", ] ) ) ``` ### Connecting Models to Features When registering a model version, there are two related but distinct ways to describe the model's inputs and outputs: - input_schema / output_schema: Define the model's raw data contract — what columns or tensors the model expects and produces. Use a dict mapping column names to types for tabular data (float, int, str, bool, or PyArrow types), or a list of (shape, dtype) tuples for tensor data. - input_features / output_features: Map the model's inputs and outputs to Chalk features. This is used by make_model_resolver to wire features into the model. Pass feature references like [User.age, User.income] or strings like ["user.age", "user.income"]. ``` client.register_model_version( name="ChurnPredictor", model=trained_model, input_schema={ "age": int, "income": float, }, output_schema={ "churn_probability": float, }, input_features=[User.age, User.income], output_features=[User.churn_risk], ) ``` input_schema and output_schema are always required, but for scikit-learn, PyTorch, XGBoost, and CatBoost objects Chalk infers them so you can omit them; input_features / output_features are optional. When registering from file paths, Docker images, Chalk images, or any other framework, provide input_schema and output_schema explicitly. ### register_model_version Parameter Reference Unique name for the model. Must start and end with an alphanumeric character, and can contain hyphens, underscores, and dots. Used to group versions under the same model namespace. If no model with this name exists, one is created automatically. A Python model object for direct registration (e.g., a trained scikit-learn estimator or PyTorch module). Chalk serializes the model and infers its type and encoding; schema and dependency inference are partial (see those fields). Mutually exclusive with model_paths. Paths to pre-serialized model files. Accepts local paths, S3 URIs (s3://...), or GCS URIs (gs://...). When using cloud URIs, also provide a source_config. Requires model_encoding to be set. Mutually exclusive with model. The ML framework used by the model. Auto-inferred when registering from a Python object. See ModelType for supported values. The serialization format of the model files. Required when using model_paths; auto-determined when registering from a Python object. See ModelEncoding for supported values. The type of ML task. Auto-inferred for XGBoost and CatBoost models. See ModelClass for supported values. Version aliases such as ["v1.0", "production"]. A "latest" alias is automatically added to every version. Use aliases with ModelReference.from_alias() to load specific versions in deployments. Defines the model's expected input format. Use a dict mapping column names to types for tabular data, or a list of (shape, dtype) tuples for tensor data. Python native types are automatically converted: float → pa.float64(), int → pa.int64(), str → pa.string(), bool → pa.bool_(). PyArrow types can also be used directly for more specific control (e.g., pa.large_string()). Always required, but auto-inferred for scikit-learn, PyTorch, XGBoost, and CatBoost objects; pass explicitly otherwise. Defines the model's output format. Same structure as input_schema. Always required, but auto-inferred for scikit-learn, PyTorch, XGBoost, and CatBoost objects; pass explicitly otherwise. Arbitrary key-value pairs for tracking model provenance, training metrics, or other information (e.g., ). Chalk feature references used as model inputs. Accepts feature references like [User.age] or strings like ["user.age"]. Used by make_model_resolver to wire features to the model. Chalk feature references produced by the model. Same format as input_features. Extra files needed for inference, such as tokenizer files, configuration files, or vocabulary files. These are uploaded alongside the model artifact. Credentials for accessing model files in cloud storage. Use S3SourceConfig for S3, GCSSourceConfig for GCS, or HFSourceConfig for Hugging Face. Defaults to local filesystem access. Python package dependencies required to load and run the model, specified as pip requirement strings (e.g., ["torch==2.7.1", "numpy==1.26.4"]). Auto-inferred only for scikit-learn, PyTorch, and XGBoost objects; pass explicitly for other frameworks, or the serving-image build will fail. A Docker image reference string for scaling group deployment (e.g., "ghcr.io/my-org/model:latest"), or a chalkcompute.Image object to have Chalk build the image for you. Images must have chalk-remote-call-python installed. When set without model or model_paths, triggers image-based registration mode. See Model Deployments. When True, skip uploading model artifacts to a volume during registration. Use this when your model artifacts are already baked into the container image. Defaults to False. ### Types and Enums See the API reference for full details on ModelType, ModelEncoding, ModelClass, and SourceConfig. ### Retrieving Model Information Registered models are tracked in the Chalk Model Registry. They can be loaded through the client and queried for metadata such as versions and performance metrics. They can also be viewed in the Chalk Dashboard. If they've been included in a deployment, you will also see feature distributions and model performance over time. ``` # Get model metadata model = client.get_model(name="RiskScoreModel") print(f"Latest version: {model.latest_model_version}") # Get specific version details model_v1 = client.get_model(name="RiskScoreModel", version=1) print(f"Performance: {model_v1.metadata['training_metrics']}") ``` ### Including Models in Chalk Deployments To include models in your Chalk deployment, use ModelReference objects. ModelReferences are code-defined objects which should be included in your Chalk code: they are used to connect model versions to model deployments. ``` from chalk.features import features, _ from chalk.ml import ModelReference from chalk import functions as F from chalk import make_model_resolver # Load a model into the deployment churn_risk = ModelReference.from_version( name="ChurnModel", version=2, ) @features class User: id: int age: int income: float churn_risk: float # define the prediction feature resolver = make_model_resolver( name="churn_model_prediction", model=churn_risk, input=[User.age, User.income], output=[User.churn_risk], ) ``` Once the model is deployed, metrics for the model will be tracked in the Chalk dashboard, including feature distributions and performance over time. ### Running Inference on Models Once models are registered and loaded into your deployment, you can run inference on your models: ``` from chalk.features import feature, features, _ from chalk.ml import ModelReference from chalk import functions as F from chalk import make_model_resolver # Load a model into the deployment using alias risk_model_latest = ModelReference.from_alias( name="RiskScoreModel", alias="latest", ) # Load a model into the deployment using version risk_model_v1_0_0 = ModelReference.from_version( name="RiskScoreModel", version=1, ) # Reference the loaded model(s) in inference @features class User: id: int age: int income: float risk_score: float = feature(versions=2) make_model_resolver( name="user_risk_score_v1_prediction", input=[User.age, User.income], output=[User.risk_score@1] ) make_model_resolver( name="user_risk_score_v2_prediction", input=[User.age, User.income], output=[User.risk_score@2] ) ``` # Model Deployments source: https://docs.chalk.ai/docs/model_deployments ## Learn how to deploy and manage machine learning models in Chalk With Chalk, you can deploy machine learning models as isolated services in dedicated scaling groups. Your models get their own compute resources, autoscaling policies, and lifecycle management, so you can scale and update them independently of the Chalk engine. Model deployments host your models as standalone services that you can call from feature resolvers or external applications. You can use model inference in your feature pipeline while managing model serving separately from feature computation. ### When to Use Model Deployments Model deployments are ideal when you want to: - Isolate model resources: Give models their own CPU, memory, and GPU resources independent of the engine - Scale models independently: Auto-scale models based on inference demand without affecting other services - Version and update models separately: Deploy new model versions without redeploying your entire Chalk system - Run containerized models: Deploy models as Chalk images or Docker images without converting to Python objects - Enable high-throughput inference: Run multiple replicas of your model in parallel ### The @model_handler decorator The fastest way to deploy a model is the @model_handler decorator. You write a single class with a predict method, hand a trained model object to register_model_version, and Chalk builds the serving image for you, ships your class as source, serializes and mounts the model, and wires up the runtime — no Dockerfile, no hand-written Arrow plumbing, no chalkcompute.Image. ``` import numpy as np import pandas as pd from sklearn.ensemble import RandomForestRegressor from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class HousePriceModel: def predict(self, df): return pd.DataFrame({"price": self.model.predict(df.to_pandas())}) rng = np.random.default_rng(0) X_train = rng.normal(size=(200, 2)) # columns: sqft, rooms y_train = X_train @ [150.0, 50.0] # price rf = RandomForestRegressor().fit(X_train, y_train) client = ChalkClient() result = client.register_model_version( name="house_price", model=HousePriceModel(model=rf), input_schema={"sqft": float, "rooms": float}, output_schema={"price": float}, dependencies=["scikit-learn", "pandas", "chalkdf"], ) deployment = client.create_model_deployment( name="house-price", model_name="house_price", model_version=result.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` Pass input_schema, output_schema, and dependencies explicitly. Schemas are required — naming your columns is what makes the output line up with what predict returns — and dependencies is the list of pip packages your predict needs at runtime. Schema values can be plain Python types — float, int, str, and bool map to pa.float64(), pa.int64(), pa.string(), and pa.bool_() respectively, so you don't need to import pyarrow for ordinary tabular models. Reach for a PyArrow type only when you need one outside those four (for example pa.float32(), pa.large_string(), or a timestamp type). The decorated class is a normal Python class with three Chalk-managed attributes injected: - model: your trained model object. Chalk serializes it with the framework's native serializer at registration time, uploads it to the model's artifact volume, and deserializes it back into self.model inside the container before predict runs. The same attribute is available on both sides. - files: a list of local file paths (files=["./scaler.pkl"]) that Chalk uploads to the artifact volume. At runtime self.files is a {basename: Path} mapping, so self.files["scaler.pkl"] resolves to the mounted path in the container (and to your local path when testing). Use this for tokenizers, scalers, encoders, lookup tables, etc. - artifact_path: the mounted artifact volume directory. ### predict and return types predict(self, df) receives a chalkdf.DataFrame built from the request. Call df.to_pandas() (or df.to_arrow()) to get the shape your model expects. The return value can be any of the following — Chalk coerces it to the output columns for you: | Return type | Becomes | | ---------------------------------------- | ---------------------------- | | `pandas.DataFrame` | columns by name | | `polars.DataFrame` / `chalkdf.DataFrame` | columns by name | | `pyarrow.RecordBatch` / `pyarrow.Table` | columns by name | | `numpy.ndarray` (1-D) | a single `prediction` column | | `numpy.ndarray` (2-D) | `col_0`, `col_1`, … columns | Returning a named frame (pandas/polars/Arrow) is recommended so your output columns are explicit and match your output_schema. ### Supported frameworks The model= object can be any of: scikit-learn, PyTorch, XGBoost, LightGBM, CatBoost, TensorFlow/Keras, or ONNX. The framework is auto-detected and the right serializer is used. ### Custom loading with load_model For models Chalk can't serialize (a custom Python class, a sentence encoder, a lookup table), leave model=None, ship the artifacts via files=[...], and own the loading in load_model, which runs once per replica at startup: ``` @model_handler class CategoryEnricher: def load_model(self): import pickle with open(self.files["table.pkl"], "rb") as f: self.table = pickle.load(f) def predict(self, df): ids = df.to_pandas()["category_id"] return pd.DataFrame({"name": [self.table.get(i, "unknown") for i in ids]}) ``` When you do have a model= object and also need extra setup (device placement, .eval(), auxiliary files), call self.default_load_model() inside your override to populate self.model the default way, then add your own steps: ``` @model_handler class UserEmbedder: def load_model(self): self.default_load_model() # restores self.model self.model.to("cuda").eval() ``` ### Local testing Because the decorated class is plain Python, you can exercise it locally with no container plumbing — construct it, call load_model() (a no-op outside the container when model= is already set), and call predict with a chalkdf.DataFrame: ``` import pyarrow as pa from chalkdf import DataFrame m = HousePriceModel(model=rf) m.load_model() out = m.predict(DataFrame.from_arrow(pa.RecordBatch.from_pydict({"sqft": [1000.0], "rooms": [3.0]}))) ``` The predict path hands your code a chalkdf.DataFrame, so the serving image installs chalkdf, which requires Python below 3.13 (Chalk pins the image to 3.12 automatically). The container also installs the same chalkpy version you registered from, so the version you run locally must be a released version that includes @model_handler predict support. ### Examples by model type Working @model_handler examples for each supported framework. The pattern is identical — write predict, hand over a trained model, register, deploy — but note that the object you get back as self.model in the container is whatever that framework's loader returns, which is often not the class you trained (see the comments in each snippet). Switch tabs to see each one: ``` import numpy as np import pandas as pd from sklearn.ensemble import RandomForestRegressor from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class SklearnRF: def predict(self, df): X = df.to_pandas()[["f0", "f1", "f2", "f3"]] return pd.DataFrame({"prediction": self.model.predict(X)}) rng = np.random.default_rng(0) X_train = rng.normal(size=(200, 4)) y_train = X_train @ [3.0, -2.0, 1.0, 0.5] rf = RandomForestRegressor().fit(X_train, y_train) client = ChalkClient() v = client.register_model_version( name="sklearn_rf", model=SklearnRF(model=rf), input_schema={f"f{i}": float for i in range(4)}, output_schema={"prediction": float}, dependencies=["scikit-learn", "pandas", "chalkdf"], ) deployment = client.create_model_deployment( name=f"sklearn-rf-{v.model_version}", model_name="sklearn_rf", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ``` import numpy as np import pandas as pd import xgboost as xgb from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class XGBoostClf: def predict(self, df): # Reloaded as a raw xgboost.Booster in the container — there is no # predict_proba; use the Booster API. binary:logistic returns P(class=1). X = df.to_pandas()[["f0", "f1", "f2", "f3"]].to_numpy() p = self.model.predict(xgb.DMatrix(X)) return pd.DataFrame({"p_negative": 1 - p, "p_positive": p}) rng = np.random.default_rng(0) X_train = rng.normal(size=(200, 4)) y_train = (X_train[:, 0] > 0).astype(int) # binary label clf = xgb.XGBClassifier().fit(X_train, y_train) client = ChalkClient() v = client.register_model_version( name="xgboost_clf", model=XGBoostClf(model=clf), input_schema={f"f{i}": float for i in range(4)}, output_schema={"p_negative": float, "p_positive": float}, dependencies=["xgboost", "pandas", "chalkdf"], ) deployment = client.create_model_deployment( name=f"xgboost-clf-{v.model_version}", model_name="xgboost_clf", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ``` import pandas as pd import torch import torch.nn as nn from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class TorchReg: def load_model(self): self.default_load_model() # restores the TorchScript module into self.model self.model.eval() def predict(self, df): X = df.to_pandas()[["f0", "f1", "f2", "f3"]].to_numpy().astype("float32") with torch.no_grad(): out = self.model(torch.from_numpy(X)).numpy().ravel() return pd.DataFrame({"prediction": out}) # Must be torch.jit-scriptable — chalkpy serializes via torch.jit.script. model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1)).eval() client = ChalkClient() v = client.register_model_version( name="torch_reg", model=TorchReg(model=model), input_schema={f"f{i}": float for i in range(4)}, output_schema={"prediction": float}, dependencies=["torch", "pandas", "chalkdf"], ) deployment = client.create_model_deployment( name=f"torch-reg-{v.model_version}", model_name="torch_reg", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ``` import numpy as np import pandas as pd from catboost import CatBoostClassifier from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class CatBoostClf: def predict(self, df): # Reloaded as a generic catboost.CatBoost — no predict_proba; use # predict(..., prediction_type="Probability"). X = df.to_pandas()[["f0", "f1", "f2", "f3"]].to_numpy() probs = self.model.predict(X, prediction_type="Probability") return pd.DataFrame({"p_negative": probs[:, 0], "p_positive": probs[:, 1]}) rng = np.random.default_rng(0) X_train = rng.normal(size=(200, 4)) y_train = (X_train[:, 0] > 0).astype(int) # binary label clf = CatBoostClassifier(verbose=False).fit(X_train, y_train) client = ChalkClient() v = client.register_model_version( name="catboost_clf", model=CatBoostClf(model=clf), input_schema={f"f{i}": float for i in range(4)}, output_schema={"p_negative": float, "p_positive": float}, dependencies=["catboost", "pandas", "chalkdf"], ) deployment = client.create_model_deployment( name=f"catboost-clf-{v.model_version}", model_name="catboost_clf", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ``` import numpy as np import pandas as pd from skl2onnx import to_onnx from sklearn.linear_model import LinearRegression from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class OnnxReg: def predict(self, df): # Reloaded as an onnxruntime.InferenceSession — call .run(), not .predict(). X = df.to_pandas()[["f0", "f1", "f2", "f3"]].to_numpy().astype("float32") out = self.model.run(None, {self.model.get_inputs()[0].name: X})[0] return pd.DataFrame({"prediction": np.asarray(out).ravel()}) rng = np.random.default_rng(0) X_train = rng.normal(size=(200, 4)).astype("float32") y_train = X_train @ np.array([1.5, -0.5, 2.0, 0.25], dtype="float32") skl = LinearRegression().fit(X_train, y_train) onnx_model = to_onnx(skl, X_train[:1]) client = ChalkClient() v = client.register_model_version( name="onnx_reg", model=OnnxReg(model=onnx_model), input_schema={f"f{i}": float for i in range(4)}, output_schema={"prediction": float}, dependencies=["onnxruntime", "numpy", "pandas", "pyarrow", "chalkdf"], ) deployment = client.create_model_deployment( name=f"onnx-reg-{v.model_version}", model_name="onnx_reg", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ``` import pickle import pandas as pd from chalk.client import ChalkClient from chalk.ml import model_handler from chalk.scalinggroup import ScalingGroupResourceRequest @model_handler class CustomLookup: def load_model(self): # model=None, so own the loading — no default_load_model() to call. with open(self.files["table.pkl"], "rb") as f: self.table = pickle.load(f) def predict(self, df): ids = df.to_pandas()["category_id"] return pd.DataFrame({"risk": [self.table.get(c, 0.0) for c in ids]}) # Build and pickle the lookup table this model serves (category_id -> risk). table = {"groceries": 0.1, "travel": 0.3, "gambling": 0.9} with open("table.pkl", "wb") as f: pickle.dump(table, f) client = ChalkClient() # chalkpy can't serialize an arbitrary object — leave model=None, ship artifacts via files=. v = client.register_model_version( name="custom_lookup", model=CustomLookup(files=["./table.pkl"]), input_schema={"category_id": str}, output_schema={"risk": float}, dependencies=["pandas", "pyarrow", "chalkdf"], ) deployment = client.create_model_deployment( name=f"custom-lookup-{v.model_version}", model_name="custom_lookup", model_version=v.model_version, resources=ScalingGroupResourceRequest(cpu="1", memory="2Gi"), ) ``` ### Container images When you need full control over the serving container — a custom inference handler, extra system packages, or a pre-built image — register a model with a container image instead of a decorated class or a Python model object. Every model image runs the chalk-remote-call-python shim, which routes requests and handles PyArrow serialization. You supply a handler and an entrypoint that points chalk-remote-call at it, then register either a chalkcompute.Image (Chalk builds it) or a pre-built Docker image (you build it). ### Writing a handler The handler receives a dict of PyArrow Arrays — one per input_schema column — and returns a PyArrow Array. Optionally define on_startup to load resources once when the container starts; artifacts mounted via a volume live at /app/artifacts/ (see Automatic Volume Upload for Model Artifacts). ``` import json import pyarrow as pa import pyarrow.compute as pc model = None def on_startup(): global model with open("/app/artifacts/model.json") as f: model = json.load(f) def handler(event: dict[str, pa.Array], context: dict) -> pa.Array: factor = model["factor"] return pc.multiply(event["x"], pa.scalar(factor, type=pa.float64())) ``` Point the entrypoint at the handler (and optional startup hook): ``` chalk-remote-call --handler model.handler --on-startup model.on_startup --port 8080 ``` ### Building a Chalk image Pass a chalkcompute.Image and Chalk builds and manages the container for you. Install chalk-remote-call-python, add your handler file, and set chalk-remote-call as the entrypoint: ``` from chalk.client import ChalkClient import chalkcompute client = ChalkClient() image = ( chalkcompute.Image.debian_slim("3.11") .pip_install(["chalk-remote-call-python", "pyarrow"]) .add_local_file("model.py", "/app/model.py", strategy="copy") .env({"PYTHONPATH": "/app"}) .workdir("/app") .entrypoint(["chalk-remote-call", "--handler", "model.handler", "--port", "8080"]) ) client.register_model_version( name="my-model", input_schema={"x": float}, output_schema={"y": float}, model_image=image, ) ``` Useful chalkcompute.Image methods: - .base(image): use a custom base Docker image - .debian_slim(python_version): slim Debian base with the given Python version - .pip_install(packages): install Python packages - .run_commands(commands): run shell commands during the build - .add_local_file(src, dest, strategy) / .add_local_dir(src, dest, strategy) — copy files in - .env(vars) / .workdir(path) / .entrypoint(command) — set env vars, working dir, entrypoint ### Using a pre-built Docker image If you build and push the image yourself, register it by string reference. The image must meet the same requirements: install chalk-remote-call-python, define a handler (and optional on_startup), and run chalk-remote-call as the entrypoint. ``` client.register_model_version( name="my-model", input_schema={"x": float}, output_schema={"y": float}, model_image="my-model-image:latest", ) ``` ``` FROM python:3.11-slim WORKDIR /app RUN pip install --no-cache-dir chalk-remote-call-python pyarrow COPY model.py /app/model.py ENV PYTHONPATH=/app EXPOSE 8080 ENTRYPOINT ["chalk-remote-call", "--handler", "model.handler", "--port", "8080"] ``` ``` docker build --platform linux/amd64 -t my-model:latest . docker push my-model:latest ``` ### Automatic Volume Upload for Model Artifacts When your model files are large (e.g. multi-gigabyte weight files), baking them into the container image is impractical — it slows down builds, increases image pull times, and wastes storage. Instead, Chalk automatically uploads model artifacts to a volume that gets mounted into your container at runtime. If your model artifacts are already baked into the image and you want to skip this automatic upload, pass skip_volume_upload=True during registration: ``` client.register_model_version( name="my-large-model", input_schema={"x": float}, output_schema={"y": float}, model_image="my-large-model-image:latest", skip_volume_upload=True, ) ``` The uploaded artifacts are mounted at /app/artifacts/ inside the container. Load them in on_startup exactly as shown in Writing a handler — open /app/artifacts/model.json once at startup and reference it from your handler. ### Creating a model deployment Use ChalkClient.create_model_deployment to deploy a registered model version. It returns a ModelDeployment with the deployment's ID, name, selected revision_id, status, replica counts, and public web_url. ``` from chalk.client import ChalkClient from chalk.scalinggroup import AutoScalingSpec, ScalingGroupResourceRequest client = ChalkClient() deployment = client.create_model_deployment( name="my-model-sg", model_name="my-model", model_version=1, handler="model.handler", scaling=AutoScalingSpec( min_replicas=1, max_replicas=2, target_cpu_utilization_percentage=70, ), resources=ScalingGroupResourceRequest(cpu="2", memory="4Gi"), ) print(deployment.id, deployment.status, deployment.web_url) ``` Creation waits for readiness by default, with a wait_timeout of 300 seconds. Set wait_ready=False to return before the deployment is ready, then call deployment.wait_ready(timeout=600) when you need to wait. deployment.refresh() returns a new snapshot of the deployment's state. To change an existing deployment, use update_model_deployment. Creating with an existing name also records and selects a new revision of that deployment. ### Auto-Scaling Configuration Control how your model deployment scales based on demand using AutoScalingSpec. ``` from chalk.scalinggroup import AutoScalingSpec # Configure auto-scaling behavior scaling = AutoScalingSpec( min_replicas=1, # Minimum number of replicas max_replicas=5, # Maximum number of replicas target_cpu_utilization_percentage=70, # Target CPU utilization (optional) ) ``` Chalk automatically scales the number of replicas based on inference request load and CPU utilization, staying within your min/max bounds. This ensures your models handle traffic spikes efficiently without wasting resources during quiet periods. ### Resource Configuration Specify CPU, memory, and GPU resources for each replica of your model using ScalingGroupResourceRequest. ``` from chalk.scalinggroup import ScalingGroupResourceRequest # Request resources per replica resources = ScalingGroupResourceRequest( cpu="2", # CPU allocation per replica memory="4Gi", # Memory allocation per replica gpu="nvidia-tesla-t4:1", # Optional: GPU type and count ) ``` Each replica gets the specified resources. When Chalk scales from 1 to 3 replicas, total resource usage is multiplied accordingly (e.g., 3 replicas × 2 CPU = 6 CPU total). ### Calling Deployed Models You can call a deployed model three ways: directly from Python with remote() or defer()/get(), from a feature resolver with F.catalog_call, or from SQL. In every case the argument order must match your model's input_schema. ### Directly from Python: remote() and defer()/get() Get a deployment by exactly one of its id or name, then call it directly: ``` from chalk.client import ChalkClient client = ChalkClient() deployment = client.get_model_deployment(name="house-price") price = deployment.remote(sqft=1000.0, rooms=3.0) price = deployment.remote(1000.0, 3.0) handle = deployment.defer(sqft=1000.0, rooms=3.0) price = handle.get(timeout=30) ``` remote() sends one row of feature values and returns the first output value when the model responds. Pass values by name or position, using the model's input_features order for positional arguments. Calls use the deployment's selected revision, so updates and rollbacks apply to subsequent calls. defer() enqueues the same inputs on the selected deployment's stored queue and returns a ModelCallHandle immediately. Pending calls are consumed by the revision selected when they run. Older revisions can retain a shared, model-wide queue until redeployed; selecting an older deployment does not isolate its deferred calls from other consumers of that queue. handle.get() raises ModelRemoteError if the call fails, or TimeoutError if its timeout expires before a result is available. ### Gather multiple model calls Submit several calls with defer(), then collect their results with chalk.gather(): ``` from chalk import gather from chalk.client import ChalkClient deployment = ChalkClient().get_model_deployment(name="house-price") handles = [ deployment.defer(sqft=1000.0, rooms=3.0), deployment.defer(sqft=1500.0, rooms=4.0), deployment.defer(sqft=2000.0, rooms=5.0), ] prices = gather(handles) ``` gather() waits for the results concurrently and returns a list in the same order as the handles. In async code, use prices = await gather(handles). If any call fails, gather() raises the first exception it encounters without returning partial results. Other remote calls continue running. ### Selecting a deployment through a model version get_model_version also exposes remote() and defer() on a DeployedModelVersion. Select a deployment explicitly when a model version is served by several deployments: ``` model = client.get_model_version("house_price", deployment_name="house-price") price = model.remote(sqft=1000.0, rooms=3.0) print(model.deployment.id) model = client.get_model_version( "house_price", version=model.version, deployment_id=model.deployment.id, ) ``` Use deployment_id or deployment_name to choose a deployment; version defaults to the version it serves. Without a deployment selector, get_model_version uses the requested version (or the latest registered version) and requires a single active deployment for inference. Direct calls require the chalkcompute transport. Install it with pip install 'chalkpy[compute]'. ### From Resolvers Address a model deployment by the name passed to create_model_deployment, referenced as model.{deployment_name}. Calls follow the deployment's selected revision. ### From a Feature Resolver — F.catalog_call Call the model as part of feature computation, so its output is just another feature: ``` from chalk.features import features, _ from chalk import functions as F @features class MyModel: id: int x_1: float x_2: float y: float = F.catalog_call( "model.my-model-sg", _.x_1, _.x_2, ) ``` F.catalog_call is evaluated during chalk query, so the call only takes effect once the feature graph is applied to your environment with chalk apply. ### Asynchronous Calls — F.catalog_call_async F.catalog_call_async takes the same arguments as F.catalog_call and produces the same result, but the engine enqueues the call on the scaling group and polls for the result instead of holding a blocking connection to the model server for the duration of the call. Use it for long-running or heavy inference, such as LLM generation or large batch scoring, where blocking calls would pin engine connections while the model works: ``` y: float = F.catalog_call_async( "model.my-model-sg", _.x_1, _.x_2, ) ``` ### From SQL Resolver — chalksql In a SQL resolver, invoke the model with the catalog_call('model.{deployment_name}', ...) function, passing the model's qualified name as the first argument: ``` select id, catalog_call('model.my-model-sg', x_1, x_2) as y from my_table ``` A deployed model only becomes available to SQL after you redeploy your Chalk deployment with chalk apply. Registering the model and deploying it to a scaling group is not enough on its own — the redeploy is what makes the model's qualified name resolvable by catalog_call. If you get an "unknown function" or unresolved-name error in a SQL resolver right after deploying a model, run chalk apply and try again. ### Asynchronous SQL calls catalog_call_async takes the same arguments as catalog_call and is the SQL counterpart of F.catalog_call_async: we enqueue and poll for a result instead of holding a connection open for the duration of the call. This is preferred for longer-running inference. Only model.* and function.* targets can be called like this. ``` select id, catalog_call_async('model.my-model-sg', x_1, x_2) as y from my_table ``` ### Named arguments After the call's positional arguments, catalog_call and catalog_call_async accept named arguments that control how the call is made. They may be combined, and each may be given at most once. ``` select id, catalog_call_async( 'model.my-model-sg', x_1, x_2, timeout => interval '90 seconds', on_error => 'struct' ) as y from my_table ``` - timeout: how long a single request may take. Write it as an interval literal, or as a bare number of seconds (timeout => 90). Rows reach the model a batch at a time: one request per batch, and one request per distinct metadata value within it. Each of those requests gets the whole budget, so a slow request cannot spend another's. A request that runs out of time fails; on_error decides what that does to the query. The deadline stops the query waiting, but does not abort the work: the model is not told, and finishes the request unread. - on_error ('fail' / 'struct', default 'fail') — what a failed request does. 'fail' propagates it and fails the query, which is what catalog_call has always done. 'struct' reports it in the result instead: the call returns struct, where a successful row has its value and a null error, and a failed row has a null value and the failure's message. Only the rows that failed are affected: with catalog_call_async that is the failing row alone, and otherwise the rows of the request that failed. Every other row, and the query itself, carry on. started_at and ended_at are UTC timestamps of when the row was sent (for catalog_call_async, when it was queued, so queue wait is included) and when its result or failure came back. Note that a model whose deployment configures retries does not retry under 'struct': the failure is reported rather than raised, so the retry policy never sees it. ``` -- Keep the rows the model answered, and see why the rest did not. select id, y.value as score, y.error as failure from ( select id, catalog_call('model.my-model-sg', x_1, on_error => 'struct') as y from my_table ) ``` ### Serving an LLM with vLLM create_model_deployment runs the image with the chalk-remote-call entrypoint (Chalk's Arrow request/response runtime), so you can't deploy a stock vLLM image directly — its own OpenAI HTTP server never starts. Instead, put the Chalk shim in front of vLLM: ship a handler that loads vLLM at startup and answers each Arrow request by generating with it. The model then deploys through the normal registration and model deployment path and is called like any other Chalk model (F.catalog_call or SQL). This is exactly how curated models are built. ``` import os import pyarrow as pa _llm = None def on_startup(): global _llm from vllm import LLM _llm = LLM(model=os.environ.get("VLLM_MODEL", "Qwen/Qwen2.5-0.5B-Instruct")) def handler(event, context): from vllm import SamplingParams rb = pa.Table.from_pydict(event).combine_chunks().to_batches()[0] prompts = rb.column("prompt").to_pylist() outs = _llm.generate(prompts, SamplingParams(max_tokens=64)) texts = [o.outputs[0].text for o in outs] return {"completion": pa.array(texts, type=pa.large_string())} ``` ``` import pyarrow as pa import chalkcompute from chalk.client import ChalkClient from chalk.scalinggroup import AutoScalingSpec, ScalingGroupResourceRequest # vLLM base + the chalk-remote-call runtime + our handler. image = ( chalkcompute.Image.base("vllm/vllm-openai:latest") .pip_install(["chalk-remote-call-python", "pyarrow"]) .add_local_file("handler.py", "/app/handler.py", strategy="copy") .workdir("/app") .env({"PYTHONPATH": "/app"}) ) client = ChalkClient() result = client.register_model_version( name="vllm-qwen", model_image=image, input_schema={"prompt": pa.large_string()}, output_schema={"completion": pa.large_string()}, ) deployment = client.create_model_deployment( name="vllm-qwen-sg", model_name="vllm-qwen", model_version=result.model_version, scaling=AutoScalingSpec(min_replicas=1, max_replicas=2), resources=ScalingGroupResourceRequest(gpu="nvidia-tesla-t4:1"), # vLLM needs a GPU handler="handler.handler", env_vars={"PYTHONPATH": "/app"}, ) ``` Then call it like any Chalk model — over the Arrow contract, not raw HTTP. For example, from a feature resolver: ``` from chalk.features import features, _ from chalk import functions as F @features class Doc: id: int prompt: str completion: str = F.catalog_call("model.vllm-qwen-sg", _.prompt) ``` Curated models are Chalk's prepackaged version of this idea: a maintained catalog of open-weight models (e.g. gemma-3-4b-it, mistral-7b-instruct, qwen3-embedding-0-6b, chronos-2) that serve the same Arrow contract and deploy with one call — no image to build or maintain. They are not built on this vLLM-in-a-Python-handler recipe, though: the text models serve through Chalk's native Rust chalk_model_runtime, with -cpu and -gpu image variants selected at deploy time. The recipe above is how you serve your own vLLM model when a curated one doesn't fit. If you instead want vLLM's raw OpenAI-compatible HTTP API — to point an OpenAI client straight at it — deploy it as a chalkcompute.ScalingGroup, which runs the image's own entrypoint and exposes the HTTP endpoint directly. See Model Inference with vLLM. ### Managing deployments Use the model deployment APIs to inspect a deployment, select a new revision, or delete it. ### Listing deployments list_model_deployments returns one page. Filter by model_name and, optionally, model_version. Pass next_cursor back as cursor until it is None: ``` cursor = None while True: page = client.list_model_deployments(model_name="my-model", cursor=cursor, limit=50) for item in page.deployments: print(item.id, item.name, item.model_version, item.status) cursor = page.next_cursor if cursor is None: break ``` ### Updating a deployment Register the new model version, then replace the deployment's serving spec with update_model_deployment. ``` import chalkcompute from chalk.client.model_deployment import ModelDeploymentSpec new_version = client.register_model_version( name="my-model", input_schema={"x": float}, output_schema={"y": float}, model_image=( chalkcompute.Image.debian_slim("3.11") .pip_install(["chalk-remote-call-python", "pyarrow"]) .add_local_file("model_v2.py", "/app/model.py", strategy="copy") .env({"PYTHONPATH": "/app"}) .workdir("/app") .entrypoint(["chalk-remote-call", "--handler", "model.handler", "--port", "8080"]) ), ) current = client.get_model_deployment(name="my-model-sg") previous_revision_id = current.revision_id deployment = client.update_model_deployment( id=current.id, spec=ModelDeploymentSpec( model_version=new_version.model_version, scaling=current.scaling, resources=current.resources, handler="model.handler", ), ) ``` ModelDeploymentSpec replaces the entire serving configuration. Supply its required model_version and scaling, and every optional setting you want to keep. Omitted resource requests, environment variables, secrets, probes, and workload identity take their defaults; they are not copied from the current revision. The example preserves the scaling and resources from the creation example. For a deployment with additional settings, include those settings from your deployment configuration as well. Update and rollback wait for readiness by default, and accept wait_ready=False and wait_timeout just like creation. ### Inspecting revisions and rolling back list_model_deployment_revisions also returns one page at a time. Use next_cursor to read all revisions. Each revision has a stable id, a model_version, and a selected flag. After rollback, the selected revision can be older than the newest revision. ``` cursor = None while True: page = client.list_model_deployment_revisions( deployment_id=deployment.id, cursor=cursor, limit=50 ) for revision in page.revisions: print(revision.id, revision.model_version, revision.selected) cursor = page.next_cursor if cursor is None: break previous = client.get_model_deployment_revision( previous_revision_id, deployment_id=deployment.id ) deployment = client.rollback_model_deployment( id=deployment.id, revision_id=previous.id, ) ``` Rollback selects an existing revision's already-built spec without rebuilding its image. The endpoint and deferred calls follow the selected revision. ### Deleting a deployment Delete the serving deployment with delete_model_deployment: ``` deleted = client.delete_model_deployment(id=deployment.id) archived = client.get_model_deployment(id=deleted.id, include_deleted=True) print(archived.deleted_at) ``` Deletion removes the serving endpoint and leaves archived metadata available with include_deleted=True. That option is also supported by deployment lists and revision lookups/lists. Deleting a deployment does not delete the registered model version. ### Structuring Your Model Deployment Code Model registration and deployment should be controlled manually and separately from your feature definitions. Either: - Add to .chalkignore to prevent them from running during chalk apply. - Run in a separate repository dedicated to model management, keeping it independent from your Chalk feature code. Your chalk apply will fail if it tries to run model registration and deployment code. Organize your project to keep model management separate from feature definitions: ``` my-chalk-project/ |- models/ # Model deployment code (add to .chalkignore) | |- model.py | `- deploy_model.py # Registration + deployment script | |- features/ # Feature definitions (synced with chalk apply) | |- __init__.py | `- user_features.py | |- .chalkignore `- chalk.yaml ``` Put the following line in your .chalkignore so chalk apply skips everything under models/. ``` models/ ``` # Model Training source: https://docs.chalk.ai/docs/model_training ## Define a training function once, then run it locally or on managed containers with GPU resources. ### Overview Chalk Model Training is a declarative Python API for training models on your Chalk data. You define a training function once, then run it in-process while developing, or on a managed container with the CPU, memory, and GPU you ask for. The API covers: - Declaring data, resources, and container configuration with a decorator - Running remotely on managed containers, or locally in-process - Checkpointing to managed storage that survives container restarts - Logging metrics that feed the training dashboard - Registering trained models in the Model Registry The API spans two packages. The @training decorator and the run handle come from chalkcompute, and the helpers you call from inside a run (checkpoint, last_checkpoint_path, log_metrics, log_rollout) come from chalkpy. Install both where you write and launch your training code: ``` pip install --upgrade chalkcompute 'chalkpy[runtime,training]' ``` Your training container needs the same two packages, and there it is worth pinning exact versions. A container image is cached and reused, so an unpinned image can go on running an SDK version you have long since upgraded past locally. ### Defining a training function Use the @training decorator to declare what the run needs. The function body is ordinary PyTorch. ``` import os from dataclasses import dataclass import polars as pl import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.data import DataLoader from chalk.ml.chalk_train import log_metrics from chalkcompute import Image, training @dataclass class SpamConfig: lr: float = 1e-3 epochs: int = 10 batch_size: int = 64 class SpamNet(nn.Module): def __init__(self): super().__init__() self.layers = nn.Sequential(nn.LazyLinear(64), nn.ReLU(), nn.Linear(64, 2)) def forward(self, x): return self.layers(x) @training( data="sms_spam_v3", cpu="8", memory="32Gi", gpu="nvidia-a10g", image=Image.debian_slim("3.12").pip_install([ "chalkcompute==2.11.10", "chalkpy[runtime,training]==2.157.22", "torch", "torchvision", "safetensors", "polars", ]), max_retries=2, ) def train_spam(df, config: SpamConfig): if hasattr(df, "collect"): df = df.collect() loader = DataLoader( df.to_torch("dataset", label="label", dtype=pl.Float32), batch_size=config.batch_size, shuffle=True, ) device = "cuda" if torch.cuda.is_available() else "cpu" model = SpamNet().to(device) opt = torch.optim.Adam(model.parameters(), lr=config.lr) for epoch in range(config.epochs): for x, y in loader: x, y = x.to(device), y.to(device).long() loss = F.cross_entropy(model(x), y) opt.zero_grad() loss.backward() opt.step() if os.environ.get("CHALK_TRAINING_RUN_ID"): log_metrics({"loss": loss.item()}, tags={"epoch": str(epoch)}) return model ``` sms_spam_v3 is a placeholder. Replace it with the name of a Dataset in your environment that has an integer label column with values 0 and 1, and numeric values in every other column. to_torch() reads every column other than label as a model input. dtype=pl.Float32 matches PyTorch's default weight type, and it converts the label to a float as well, which is why the loop converts y back to an integer with .long() for cross_entropy. nn.LazyLinear sets its input width from the first batch, so SpamNet does not need to know the number of feature columns in advance. ### Decorator parameters data is required, as is at least one of cpu, memory, or gpu. A run declaring no resources at all is rejected when it is created. | Parameter | Description | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data` | Name of an existing [Dataset](/docs/datasets) to train on. The run pins an immutable revision of it, so a rerun reads the same rows. | | `cpu`, `memory`, `gpu` | Resource requirements, each written as a string: `cpu="8"`, `memory="32Gi"`, `gpu="nvidia-a10g"`. See [Chalk Compute](/docs/compute/overview) for the values these accept. | | `image` | Container image for the run. Use Chalk's [`Image`](/docs/compute/images) builder, or a Docker image with your dependencies already installed. | | `env` | Environment variables injected into the container at runtime. | | `secrets` | A list of [`Secret`](/docs/compute/secrets) references, built with `Secret.from_chalk_env()` or `Secret.from_chalk_integration()`. | | `name` | Run name. Generated if omitted. | | `metadata` | Key-value metadata stored on the run, for tracking and filtering. | | `max_retries` | How many times to retry on failure or container crash. | ### Function signature ``` def train_fn(df, config: YourConfigClass): ... return model ``` - df is the resolved training data, as a Polars frame. run_training() passes a LazyFrame and local() passes an eager DataFrame, and the conversion methods differ between the two: to_torch(), to_pandas(), and to_arrow() exist only on the eager frame, while collect() exists only on the lazy one. Open with the line below so that the rest of the function receives an eager DataFrame either way:if hasattr(df, "collect"): df = df.collect() - config is a @dataclass you define, or a plain dict. It is serialized to a protobuf Struct and passed to the function. In a remote run, the function receives an instance of your dataclass only when its config parameter is annotated with it, as in config: SpamConfig. Without the annotation, it receives a dict. - The return value is the trained model. ### Running training ### Remotely ``` run = train_spam.run_training(config=SpamConfig(lr=1e-3, epochs=10)) run.wait() artifact = run.latest_checkpoint() ``` run_training() builds a deploy image containing your training function's source, provisions a managed container with the resources, environment variables, and secrets you declared, and resolves the data source inside it. It returns a TrainingRunHandle immediately, without waiting for the run to finish. ### Locally ``` model = train_spam.local(config=SpamConfig(lr=1e-3, epochs=2)) ``` local() runs the function in-process, with no container and no remote infrastructure, and returns the model directly. Use it while developing, then switch to run_training() once the code works. checkpoint() and log_metrics() raise an error when called outside a training run, and local() does not start one. Chalk sets the CHALK_TRAINING_RUN_ID environment variable in every training container, so the example above calls log_metrics() only when that variable is set. Guard checkpoint() the same way in a function you run with local(). It reads the latest revision of the named Dataset, where run_training() pins an immutable revision for the life of the run. ### Run lifecycle run_training() returns a TrainingRunHandle for observing and controlling the run. | Member | Description | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `run_id` | Property holding the run's id. | | `status()` | Returns the current status. | | `wait(poll_interval=5.0, timeout=None)` | Blocks until the run reaches a terminal state, polling every `poll_interval` seconds, and returns the handle. Raises `TrainingError` if the run failed or was canceled, or `TimeoutError` if `timeout` seconds elapse first. | | `latest_checkpoint()` | Returns the most recent checkpoint artifact for the run, or `None` if none has been uploaded. | | `cancel()` | Cancels the run. The platform terminates the container. | ### Checkpointing and resume Checkpoints let a run pick up where it left off after a container restart. ``` from safetensors.torch import load_file from chalk.ml.chalk_train import checkpoint, last_checkpoint_path ckpt = last_checkpoint_path() if ckpt is not None: model.load_state_dict(load_file(ckpt)) for epoch in range(config.epochs): # ... training loop ... if epoch % 5 == 0: checkpoint(model, metadata={"epoch": epoch}) ``` checkpoint() serializes the model, uploads it, and records it as a model artifact on the run. It takes the model plus four optional arguments: additional_files to store alongside the model, metadata recorded on the artifact, run_name, and path. last_checkpoint_path() returns the path to the most recent checkpoint file, or None if there is not one yet. checkpoint() saves a PyTorch model in the safetensors format, so load it with safetensors.torch.load_file(), and include safetensors in the training image. Checkpoints outlive the container that wrote them, so a run that restarts picks up from the last one rather than starting over. ### Naming checkpoints for experiment tracking Checkpoint storage is scoped to the training function's name, not to an individual run. Every run sharing a name writes into the same place, which is what lets you run several experiments against one model at the same time and compare them afterwards. Each checkpoint lands under its own name inside that storage. By default it is the run's id, which is accurate but tells you nothing later. Pass path to name it yourself: ``` checkpoint(model, path="experiment-one") ``` Run a sweep with path="experiment-one", path="experiment-two", and so on, and the checkpoints are labelled by what you were testing rather than by an opaque id. A latest entry alongside them records the most recent checkpoint, and that is what last_checkpoint_path() reads. ### Metrics Call log_metrics() from inside the training function with a dictionary of metric names and values, plus tags describing where in the run the measurement came from. ``` from chalk.ml.chalk_train import log_metrics for epoch in range(config.epochs): # ... training loop ... log_metrics({"accuracy": accuracy, "loss": loss}, tags={"split": "test", "epoch": str(epoch)}) ``` ### Supported names and tags The dashboard recognizes a fixed vocabulary. These metric names are supported: accuracy, loss, precision, recall, f1, mae, mse, rmse, r2, reward, reward_std, pass_rate, kl, entropy, grad_norm, learning_rate, response_length And these tags: | Tag | Description | | --------------- | --------------------------------------------------------------------------- | | `split` | Which subset of the data the measurement covers, such as `train` or `test`. | | `class` | The class the measurement covers, for per-class metrics. | | `threshold` | The decision threshold the measurement was taken at. | | `optimizer` | The optimizer in use, such as `adam` or `sgd`. | | `model version` | The model version the measurement belongs to. | | `epoch` | The training epoch. | | `step` | The training step. | Tagging epoch or step throughout a run lets the dashboard use either as the x-axis of its plots. Tagging split, class, threshold, optimizer, or model version lets you compare across those values with Group by. Metrics whose names or tags fall outside these lists are not recorded by the dashboard. Send those through log_custom_metric instead. ### Custom metrics chalk.log_custom_metric sends a metric to your own sink rather than the training dashboard. It emits through StatsD, so the environment needs StatsD export configured before any of these measurements leave the run. See Metrics Export. ``` import chalk chalk.log_custom_metric( "grad_norm", 3.14, tags={"split": "validation"}, ) ``` That is emitted as chalk.custom_metrics.grad_norm(3.14), with its tags attached. With no StatsD sink configured the call does nothing and reports nothing. It does not raise, warn, or fall back to the training dashboard, so verify your export is working before relying on a custom metric. This is the opposite of log_metrics, which warns on an unrecognized name and raises when none of the names it is given are valid. ### Rollouts For reinforcement learning, log_rollout() records one episode as a trace: the conversation between the model and its environment, plus the reward it earned. Each rollout appears on the run's Traces tab, where you can read what the model did on a task and why it scored what it did. This works for any model, including the one being trained. ``` from chalk.ml.chalk_train import log_rollout log_rollout( messages, # the episode, in order reward=1.0, step=7, task_id="qrdata/112", metrics={"correct": 1.0}, ) ``` messages uses the OpenAI chat format: each message has a role and content, and may carry tool_calls or tool_call_id. Each assistant message becomes a model turn, and the messages that follow it are the environment's responses. Pass rollout_id (32 hex characters) to reuse your framework's own rollout id as the trace id. Outside a Chalk training container, log_rollout warns once and records nothing. ### Models other than PyTorch @training works with any serializable model. This example trains an XGBoost classifier: ``` import os from dataclasses import dataclass import xgboost as xgb from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from chalk.ml.chalk_train import log_metrics from chalkcompute import Image, training @dataclass class ChurnConfig: num_rounds: int = 200 max_depth: int = 6 learning_rate: float = 0.1 @training( data="user_churn_v2", cpu="16", memory="64Gi", image=Image.debian_slim("3.12").pip_install([ "chalkcompute==2.11.10", "chalkpy[runtime,training]==2.157.22", "xgboost", "pandas", "scikit-learn", ]), ) def train_churn(df, config: ChurnConfig): if hasattr(df, "collect"): df = df.collect() pdf = df.to_pandas() X = pdf.drop(columns=["churned"]) y = pdf["churned"] X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2) dtrain = xgb.DMatrix(X_train, label=y_train) dval = xgb.DMatrix(X_val, label=y_val) model = xgb.train( { "max_depth": config.max_depth, "learning_rate": config.learning_rate, "objective": "binary:logistic", "eval_metric": "auc", }, dtrain, num_boost_round=config.num_rounds, evals=[(dval, "val")], ) if os.environ.get("CHALK_TRAINING_RUN_ID"): log_metrics( {"accuracy": accuracy_score(y_val, model.predict(dval) > 0.5)}, tags={"split": "test", "step": str(config.num_rounds)}, ) return model ``` As in the PyTorch example, user_churn_v2 is a placeholder. Replace it with a Dataset that has a churned column with values 0 and 1, and numeric values in every other column. ### Environment variables and secrets ``` from chalkcompute import Secret, training @training( env={"NCCL_DEBUG": "INFO", "TOKENIZERS_PARALLELISM": "false"}, secrets=[ Secret.from_chalk_env("hf_token"), Secret.from_chalk_env("wandb_api_key"), ], ... ) ``` Environment variables are injected directly. secrets takes Secret references rather than plain names: Secret.from_chalk_env() resolves a standalone secret from Chalk's secret store, and Secret.from_chalk_integration() resolves the credentials of a named integration. Both are exposed as environment variables in the container, and their values are never written to the run record. ### Registering a model Promote an artifact from a completed run into the Model Registry with ChalkClient.promote_model_artifact: ``` from chalk.client import ChalkClient client = ChalkClient() client.promote_model_artifact( name="RiskModel", run_id=run.run_id, aliases=["latest"], ) ``` name is the model namespace to promote into, and is the only required argument. Identify the artifact with model_artifact_id, run_id, or run_name. When a run produced several checkpoints, criterion selects among them; the latest artifact in the run is used if you omit it. aliases assigns version labels such as v1.0 or latest. ### See also - Model Registry: track and version trained models. - Model Deployments: serve a registered model. - Datasets: create the datasets a training run reads from. # Model Gateway source: https://docs.chalk.ai/docs/model-gateway ## Route LLM traffic to multiple providers through one OpenAI-compatible router. The Chalk Model Gateway routes LLM requests to OpenAI, Anthropic, and Google Gemini through one OpenAI-compatible API. Manage provider credentials, API keys, budgets, rate limits, and fallback policies in the Chalk dashboard. The same endpoint also serves the Anthropic Messages API, so Anthropic SDKs and Claude Code route through the Model Gateway without a translation layer. The Model Gateway runs in your Chalk environment. On a dedicated or self-hosted deployment, it uses your own cloud infrastructure. ### Getting started - Open Model Gateway → Routing → Connections in the Chalk dashboard and choose Add connection. Enter your provider credentials and save the connection. - Open Model Gateway → Playground, select a model, and send a prompt. Chalk sets up your gateway automatically when you add your first connection. The playground uses your Chalk session, so you can try it immediately. To use the gateway from your application, create an API key and follow Making a request below. ### Sending requests The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header. ### Endpoint The Model Gateway exposes the standard OpenAI-compatible paths under this base URL: | Path | Purpose | | --------------------- | ------------------------------------------------------- | | `/chat/completions` | Chat completions (streams via Server-Sent Events). | | `/responses` | OpenAI Responses creation (streaming or non-streaming). | | `/embeddings` | Text embeddings. | | `/moderations` | Text moderation. | | `/images/generations` | Image generation. | | `/models` | List the models available to your key. | The same base URL also serves the Anthropic Messages API. Anthropic SDKs append their own /v1 segment to the base URL they are given, so those paths read with the extra segment: | Path | Purpose | | --------------------------- | ------------------------------------------ | | `/v1/messages` | Messages (streams via Server-Sent Events). | | `/v1/messages/count_tokens` | Exact token count for a request. | | `/v1/models` | List the models available to your key. | The same base URL also serves the native Gemini Interactions API, preserving its /v1beta segment: | Path | Purpose | | ---------------------- | --------------------------------------------------------------------------------------------- | | `/v1beta/interactions` | Create a Gemini interaction with `POST`, with or without native Server-Sent Events streaming. | The full endpoint is https://api.chalk.ai/v1/router/v1beta/interactions. Use a configured Gemini connection prefix or a published route alias in model, and provide the native input field. Set stream: true for streaming. Agent/background jobs and interaction retrieval, deletion, and cancellation are not supported and return HTTP 501. The same base URL also serves the Decisions and System One APIs: | Path | Purpose | | ------------------ | ---------------------------------------- | | `/alpha/decisions` | Submit a Decisions request with `POST`. | | `/v1/systemone` | Submit a System One request with `POST`. | The full endpoints are https://api.chalk.ai/v1/router/alpha/decisions and https://api.chalk.ai/v1/router/v1/systemone. Both accept model, state, and questions in the JSON request body. Select a model on a connection that supports the corresponding API. The gateway preserves System One's explicit /v1 segment and adds /v1 when forwarding /alpha/decisions to the router. Use whichever supported format your client speaks: the Model Gateway resolves the requested model against your configured providers and enforces the key's restrictions across these APIs. See Use with Claude Code for an Anthropic client configuration. If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own API server host. You can find it in the API Server row of chalk config, or as apiServer.value in chalk config --format json. Requests are routed to the calling environment's Model Gateway, selected by the X-Chalk-Env-Id header described below. ### Text moderation Call client.moderations.create to classify a string or an array of strings: ``` result = client.moderations.create( model="openai/omni-moderation-latest", input="Text to check", ) print(result.results[0].flagged) ``` Use your connection's configured prefix in the model name. If you omit model, the gateway resolves omni-moderation-latest through the configured routes and OpenAI connection. The resolved connection and model must be permitted by your API key. The endpoint forwards the provider's verdict and returns provider errors when moderation fails. Moderation dispatches once to the resolved model. It does not use chat fallback, the judge, or prompt caching. Request-rate and concurrency limits apply, including limits scoped to the connection and model. Moderation does not report token usage, so token budgets and tokens-per-minute limits do not apply. Successful calls appear in usage tracking as one request with zero tokens. This endpoint accepts text input only. ### Authentication Every request carries an issued Model Gateway API key as a bearer token, plus the environment to route to: | Header | Value | | ---------------- | --------------------------------------- | | `Authorization` | `Bearer ` | | `X-Chalk-Env-Id` | The environment to route the request to | Model Gateway API keys are issued and revoked from the dashboard or the CLI. See API keys below. A key is shown only once when it is issued, so copy it immediately. Requests made with an API key are billed to your provider credentials. Treat a key like any other secret: scope it to the connections and models it needs, set a usage budget, and revoke any key you suspect is compromised. ### Making a request The Model Gateway speaks the OpenAI API, so any OpenAI-compatible client works with no code changes beyond the base URL, key, and environment header. ``` from openai import OpenAI client = OpenAI( base_url="https:///v1/router", api_key="", default_headers={"X-Chalk-Env-Id": ""}, ) resp = client.chat.completions.create( model="openai/gpt-5.6-luna", messages=[{"role": "user", "content": "Hello"}], ) print(resp.choices[0].message.content) ``` Use a connection-prefixed model ID, such as openai/gpt-5.6-luna or anthropic/claude-sonnet-5, or a published model route such as chalk/assistant. Run chalk router model list to see every model your configured providers expose. The same request with curl: ``` curl https:///v1/router/chat/completions \ -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \ -H "X-Chalk-Env-Id: " \ -H "Content-Type: application/json" \ -d '{"model": "openai/gpt-5.6-luna", "messages": [{"role": "user", "content": "Hello"}]}' ``` The model you request is resolved against the providers you have configured. A key's connection scopes, usage pool, rate limits, and usage budgets are all enforced on every request. An Anthropic client is configured the same way (the base URL, the key, and the environment header) and calls the Messages API: ``` from anthropic import Anthropic client = Anthropic( base_url="https:///v1/router", api_key="", default_headers={"X-Chalk-Env-Id": ""}, ) resp = client.messages.create( model="claude-sonnet-4-6", max_tokens=1024, messages=[{"role": "user", "content": "Hello"}], ) print(resp.content[0].text) ``` If a request does not work, two CLI commands report what the Model Gateway sees. chalk router status shows whether the Model Gateway is reachable and your harness configuration, and chalk router doctor diagnoses the Model Gateway, the model you asked for, and the harness together. ### Calling the Model Gateway from Chalk Because the Model Gateway is OpenAI-compatible, you can also call it from inside feature computation with chalk.functions. F.openai_complete issues a chat completion while a query runs. Route it through the Model Gateway rather than OpenAI in one of two ways: - Set api_server to the Model Gateway base URL, https://api.chalk.ai/v1/router. - Set the OPENAI_BASE_URL environment variable on the execution host and omit api_server. ``` import chalk.functions as F from chalk.features import _ from chalkdf import DataFrame df = DataFrame({"questions": ["Recommend some movies like High and Low by Akira Kurosawa"]}) ( df.with_columns( F.openai_complete( prompt=_.questions, model="openai/o4-mini", service_tier="flex", api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway ).alias("result") ) .run() .to_arrow() ) ``` F.openai_complete takes the following arguments: | Argument | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `prompt` | The prompt text to send to the model. | | `model` | The model to use. Defaults to `gpt-3.5-turbo` when omitted. | | `api_server` | Base URL of an OpenAI-compatible endpoint; `/chat/completions` is appended. Falls back to the `OPENAI_BASE_URL` env var, then to the default OpenAI endpoint. Point it at `https://api.chalk.ai/v1/router` to use the Model Gateway. | | `api_key` | API key for authentication. Falls back to the `OPENAI_API_KEY` env var, so the secret does not have to be threaded through feature data. | | `max_tokens` | Maximum number of tokens to generate. | | `temperature` | Sampling temperature between 0 and 2. | | `service_tier` | Optional OpenAI service tier: `"flex"` (cheaper, higher-latency), `"priority"`, or `"auto"`. `"flex"` is only supported on reasoning models (`o3`, `o4-mini`, `gpt-5`-class) and uses a longer request timeout; passing it with an unsupported model returns `null`. | It returns a struct with the completion text plus prompt_tokens, completion_tokens, total_tokens, model, finish_reason, and the upstream ratelimit_remaining_tokens / ratelimit_remaining_requests headers. ### Consuming rate limits from a resolver LLM calls are blocking and metered, so throttle them with the policy modifiers chained onto the expression. with_rate_limit caps how often the call may run across every expression that shares its key: ``` import chalk.functions as F from chalk.features import _ from chalkdf import DataFrame df = DataFrame({"questions": ["Recommend some movies like Buzzard by Joel Potrykus"]}) ( df.with_columns( F.openai_complete( prompt=_.questions, model="openai/o4-mini", service_tier="flex", api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway ) .with_rate_limit(rate=3, key="openai", per="minute") .alias("result") ) .run() .to_arrow() ) ``` The same modifier works on a feature defined in a @features class: chain .with_rate_limit(...) before selecting the .completion field. with_rate_limit takes: | Argument | Description | | ------------------ | ---------------------------------------------------------------------------------------------- | | `rate` | Number of calls allowed per window. | | `per` | Window length: `"second"` (default), `"minute"`, or `"hour"`. | | `key` | Bucket name; all expressions sharing a `key` draw from the same budget. | | `enforce_globally` | When `True`, enforce the limit across all workers rather than per-worker. Defaults to `False`. | These policy modifiers compose: chain with_concurrency, with_rate_limit, and with_retry on the same expression to bound in-flight calls, cap the call rate, and retry transient failures with backoff: ``` ( F.openai_complete( prompt=_.questions, model="openai/o4-mini", service_tier="flex", api_server="https://api.chalk.ai/v1/router", # route through the Model Gateway ) .with_concurrency(max_concurrent=4, key="my_api") .with_rate_limit(rate=100, key="my_api") .with_retry(max_retries=3, key="my_api") ) ``` Reusing one key ties the policies to the same logical resource, so every expression that calls "my_api" shares a single budget: here, at most 4 concurrent calls and 100 calls per second (per defaults to "second"), with each failed call retried up to 3 times. ### Playground Exercise the Model Gateway from the dashboard under Model Gateway → Playground, which authenticates with your Chalk session rather than a separate API key. It covers chat completions, structured Decisions requests, embeddings, and image generation. The Decisions tab provides Text/JSON state editors, Noul/Choice/Score question builders, example presets, answer summaries, token usage, cost when the provider reports it, and the raw response. Its model picker is populated from the Model Gateway's /models endpoint and also accepts a model ID entered manually. ### Coding agents A coding agent points at the Model Gateway the same way any other client does, so the provider credential, connection scopes, budgets and rate limits already configured apply to it. ### Claude Code Claude Code speaks the Anthropic Messages API, so the Model Gateway works as its LLM gateway: model requests route through Chalk, where the provider credential, connection scopes, usage budgets, and rate limits already live, and each developer holds a revocable Model Gateway API key instead of a provider key. The CLI configures this for you, using the credentials you are already logged in with: ``` chalk router claude on ``` Pass --model to pin a model. To start a single session without changing Claude Code's configuration at all, use chalk router claude launch instead: it writes the Model Gateway settings to a temporary file, hands them to Claude Code with --settings, and removes the file when the session ends, so ~/.claude/settings.json is never read or rewritten. chalk router claude off restores whatever the CLI changed, and chalk router claude status shows the current state. The rest of this section covers the same setup by hand, which is worth reading if you are scripting it or debugging a connection. Set the base URL and the key in your shell: ``` export ANTHROPIC_BASE_URL=https:///v1/router export ANTHROPIC_AUTH_TOKEN= export ANTHROPIC_CUSTOM_HEADERS="X-Chalk-Env-Id: " ``` Use ANTHROPIC_AUTH_TOKEN rather than ANTHROPIC_API_KEY: the Model Gateway reads the key from the Authorization header, and ANTHROPIC_API_KEY sends it in x-api-key. The Model Gateway accepts either, but ANTHROPIC_API_KEY also needs a one-time approval prompt in Claude Code before it takes effect. Confirm the Model Gateway answers before starting Claude Code, so a failure points at the Model Gateway rather than at your configuration: ``` curl -X POST "$ANTHROPIC_BASE_URL/v1/messages" \ -H "Authorization: Bearer $ANTHROPIC_AUTH_TOKEN" \ -H "X-Chalk-Env-Id: " \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model": "claude-sonnet-4-6", "max_tokens": 1, "messages": [{"role": "user", "content": "."}]}' ``` A body starting with {"id":"msg_ means the URL, key, and environment header all work. An unknown-model error also confirms them, since the Model Gateway authenticated the request before rejecting the model. To make the configuration apply everywhere Claude Code runs, including background agents, move the values into the env block of ~/.claude/settings.json: ``` { "env": { "ANTHROPIC_BASE_URL": "https:///v1/router", "ANTHROPIC_AUTH_TOKEN": "", "ANTHROPIC_CUSTOM_HEADERS": "X-Chalk-Env-Id: " } } ``` Run /status in Claude Code to confirm: the Status tab shows an Anthropic base URL line with your Chalk host and an Auth token line naming the variable you set. Two things to know before rolling this out: - The Model Gateway API key replaces a claude.ai login. While ANTHROPIC_AUTH_TOKEN is set, a saved claude.ai subscription is unused and its limits do not apply; usage bills against the provider credential the Model Gateway forwards. Features that need a claude.ai identity, such as Remote Control and voice dictation, are unavailable while it is set. - Serve the Model Gateway's model names. If a model route exposes a name that is not one of Claude Code's built-in models, set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 so the /model picker is populated from the Model Gateway's /v1/models response. ### Other harnesses The CLI covers four more harnesses. Each has on, off and status: | Harness | Command | Also supports | | -- | -- | -- | | Codex | chalk router codex | launch to start a session without changing its configuration, and resume to continue one whichever provider recorded it | | Cursor | chalk router cursor | | | OpenCode | chalk router opencode | | | VS Code | chalk router vscode | | Cursor is set up by hand. chalk router cursor on prints the settings to apply under Cursor Settings → Models → OpenAI, and off prints the steps to undo them; neither edits Cursor's configuration. For Codex, OpenCode and VS Code, on writes the base URL and key into that harness's own configuration and off restores whatever it changed. Run chalk router status to see every harness at once, including whether it is installed and whether its routing is currently on. ### Routing Routing decides which upstream model serves a request, and what happens when one is unavailable. Configure all of it under Model Gateway → Routing in the Chalk dashboard. ### Provider connections A connection holds the credential for one provider along with the settings that decide which of its models the Model Gateway offers. Create and edit connections under Model Gateway → Routing → Connections. You can create several connections for the same provider, each with its own credential, base URL, and model prefix. An API key is scoped to the connections it may use; see API keys. The Model Gateway supports these provider kinds: | Kind | Notes | | ------------------- | ----------------------------------------------------------------------- | | `openai` | | | `anthropic` | | | `gemini` | | | `vertex` | The stored credential is a GCP service account key rather than a token. | | `bedrock` | | | `vllm` | | | `voyage` | | | `cohere` | | | `mistral` | | | `qwen` | | | `xai` | | | `groq` | | | `fireworks` | | | `together` | | | `openrouter` | | | `typesafe` | | | `openai-compatible` | Any endpoint that speaks the OpenAI API. Set a base URL. | Each connection also carries: - Prefix: the segment the Model Gateway puts in front of a model ID, so /v1/models lists the connection's models as /. - Exposure: whether the connection's models appear in /v1/models. Dynamic lists every model the provider reports. Unlisted keeps those models reachable but hidden, so a caller has to know the ID already. Explicit serves nothing from the provider's catalog, so callers reach models only through a model route. Unspecified means no policy is stored, and the connection falls back to the default. - State: whether the credential is configured, readable, and healthy. Credentials apply at runtime, so changing one needs no redeploy. ### Model routes A model route gives callers a public model ID beginning with chalk\, such as chalk/assistant, and controls which connection and upstream model serve it. Manage routes under Model Gateway → Routing → Routes. Choose one of two routing behaviors when creating or editing a route: - Simple: use one provider connection and upstream model. - Custom: set a default connection and upstream model, then optionally add fallback chains and/or a judge policy in the route's decision grid. The custom route's default target serves requests when judging is off, in shadow mode, or cannot produce a usable classification. That target can have its own fallback chain. A custom route can also have fallback chains without a judge policy. Give each upstream model ID exactly as the provider expects it, without the connection's routing prefix. For example, select the OpenAI connection and upstream model gpt-4o. An upstream model ID may contain slashes of its own. All judge targets, outcome models, and fallbacks name concrete connections and upstream models; another route alias cannot be a target. Publish a route to make it callable. Each route reports a diagnostic alongside its status: ready, unconfigured, disabled, unreadable, or missing its connection. Editing one route's policy affects requests to that route; in-flight requests retain the configuration they started with. ### Fallback policies Configure fallback chains inside a custom route. Each chain identifies a selected connection and model, followed by an ordered list of alternatives. For example, a route could select gpt-4o on your OpenAI connection and fall back to gpt-5.6-luna on that connection, then claude-sonnet-4-6 on your Anthropic connection. Each judge outcome can also choose from the following fallback options: Use selected model's route chain, Stop after selected model, or Use outcome-specific chain. This lets two outcomes select the same model with different recovery behavior. The Model Gateway chooses one chain using this precedence: - A matching request-specific fallback entry, for Chat Completions. - The selected judge outcome's explicitly configured fallback list. - The route's fallback entry for the selected model. - No alternatives. An omitted outcome list inherits the route's chain. An explicit empty list disables fallback for that selection. Lists replace one another; they are not combined. When judging selects an outcome that uses the default target, that outcome's fallback setting still applies. The Model Gateway chooses the chain once and tries its alternatives in order. Reaching an alternative does not expand that model's chain or run the judge again. Exhausting an outcome or request chain does not resume the route's chain. Each list can contain at most 16 alternatives, with no repeated target or entry equal to the starting model. Each route or request fallback map can contain at most 128 entries. ### Request-specific fallbacks Chat Completions accepts a fallbacks object mapping selected models to lists of alternatives. This overrides fallback behavior for that request without changing the stored route. The judge policy itself remains configured on the route. Keys and alternatives use connection_id/upstream_model_id or connection_name/upstream_model_id. Connection names match exactly, including case; they are separate from routing prefixes. If both an ID and a name key identify the selected model, the ID-keyed entry wins. Connection IDs remain stable when you rename a connection. For a route whose judge can select gpt-4o on the connection named openai-prod, this request replaces that selection's alternatives with gpt-5.6-luna: ``` curl 'https:///v1/router/chat/completions' \ -H "Authorization: Bearer $CHALK_ROUTER_API_KEY" \ -H 'X-Chalk-Env-Id: ' \ -H 'Content-Type: application/json' \ --data-binary @- <<'JSON' { "model": "chalk/assistant", "messages": [{"role": "user", "content": "What is water in five words?"}], "fallbacks": { "openai-prod/gpt-4o": ["openai-prod/gpt-5.6-luna"] } } JSON ``` Replace the host, environment, route alias, and connection name with your configuration. If the judge selects a different model, this entry does not match, so the outcome or route configuration applies. Valid unmatched entries do not affect routing. The entire fallback map must pass validation, including entries that are not selected; an invalid entry returns HTTP 400. Setting "openai-prod/gpt-4o": [] instead explicitly disables fallback when that target is selected. Request overrides also work without a judge, using the initially resolved target as the lookup key. These request overrides are supported only on Chat Completions. ### Failures and limits Fallback runs for upstream connection failures and timeouts, HTTP 429, and HTTP 5xx responses. A candidate's connection or model rate limit can also cause the next candidate to be tried. Other upstream HTTP errors, unreadable responses, refusals, and answer-quality problems do not trigger fallback. Once the Model Gateway commits a streaming response, a subsequent error ends that stream without switching models. For background Responses requests, an uncertain dispatch failure stops retries because the upstream job may already be running. Access to a custom chalk/* route authorizes its judge-selected model and route-configured fallbacks under that route's public model ID. Each destination must still satisfy the key's provider and connection restrictions and the request's capability requirements. Request-supplied alternatives require direct-model access to each destination and cannot use route-only (Explicit) connections. Model-scoped rate limits use the requested route ID for route-owned destinations and the direct, connection-prefixed model ID for request-supplied alternatives. Missing, disabled, unauthorized, or incompatible candidates are skipped. Admission to the requested route must still succeed before the policy can run. Shared key or usage-pool quota exhaustion stops the whole request; fallback cannot bypass it. The Model Gateway reserves quota once for the generation plan using the most conservative eligible candidate estimate and checks each attempted connection's limits separately. The router stops starting additional fallback attempts after 300 seconds. This limit does not cancel an in-progress foreground request. Background Responses submissions are bounded by the deadline; the accepted background job can continue running. Other endpoint and provider timeouts still apply. Route fallbacks apply to Chat Completions, Messages, Responses creation, and Decisions/System One. Gemini Interactions supports route fallback only when you explicitly set store: false and send a fresh request without connection-bound history, such as previous_interaction_id, cached content, thought signatures, or tool-history items. Omitting store disables fallback, even for a new interaction. This restriction avoids retrying a potentially stored interaction on another connection after an uncertain failure. Operations such as Messages token counting and Responses retrieval or compaction do not walk fallback chains. ### Judge policy A route's judge classifies a request and selects a configured outcome. In the route editor, enable Judge policy, choose a backend and mode, then configure each outcome's model and fallback behavior in the Decision grid. Models are explicit choices; the Model Gateway does not infer a price or capability ordering, or restrict a choice to a downgrade. ### Classification and outcomes Choose one of two classification types: | Classification | Default outcomes | Meaning | | -------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Difficulty | `0`, `1` | `0` covers straightforward retrieval, rewriting, summaries, extraction, and small self-contained changes. `1` covers sustained reasoning, ambiguity, complex debugging, mathematics, and system design. | | Task | `coding`, `summarization`, `extraction`, `other` | Choose by the main deliverable: code or technical guidance, a condensed account of source material, specific information from a source, or another kind of request. | The editor supplies descriptions for these categories and starts new outcomes at the route's default target with inherited fallbacks. Assign a connection and model to each outcome. Remote judges use the descriptions to distinguish categories; customize them for your workload. Existing descriptions are preserved unless you edit them or choose Restore defaults. Restoring defaults resets categories and descriptions while preserving models and fallbacks for matching labels. Remote judges support custom task labels or a difficulty scale with consecutive integers starting at 0, ordered from easiest to hardest. Each judge supports up to 32 outcomes. Blank descriptions for standard task labels or the two-level difficulty scale use the default criteria. Supply your own criteria for custom labels or a larger difficulty scale. ### Judge backends | Backend | Configuration | | ---------------- | ---------------------------------------------------------------------------------------------------- | | Local heuristic | Uses text rules without a model call. Requires difficulty outcomes `0` and `1`. | | Completion model | Select a connection and upstream model that support completion-based classification, plus a timeout. | | Decisions model | Select a connection and upstream model that support Decisions-based classification, plus a timeout. | The editor checks remote judge compatibility before saving. A judge's backend determines how classification runs; it does not change the endpoint format used by callers. For Local heuristic, add or remove hard-task keywords, choose whether code blocks count as hard, and set the easy-input length cutoff. Keywords match user text as case-insensitive substrings. Code fences in any message role count as hard by default. The length rule counts UTF-8 text bytes across message roles; the default cutoff is 6,000 bytes, and 0 disables that rule. Any enabled hard signal selects 1; otherwise the result is 0. Outcome descriptions are retained but do not affect heuristic classification. Remote backend timeouts are in milliseconds, from 1 to 30,000. An omitted timeout or 0 uses the 1,000 ms default. If classification times out, is unavailable, or returns a malformed or unknown label, the route uses its default target and that target's applicable fallback chain. Remote judging consumes token budgets and is subject to the judge connection's limits, with model-scoped limits evaluated against the requested route ID. Shared token-budget exhaustion can reject a live request instead of proceeding to generation. If the remote judge’s concurrency limit is reached, classification is skipped and the request uses the default target and its applicable fallbacks. ### Modes and endpoint support | Mode | Serving behavior | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Off | Uses the default target and its applicable fallbacks without classifying. | | Shadow | Serves through the default target and its applicable fallbacks. After serving admission succeeds, classifies asynchronously and records the proposed outcome, model, and chain without sending generation requests to those proposed targets. | | Live | Classifies before the first generation attempt and uses the selected outcome and its applicable fallbacks. | Remote shadow classification still consumes judge tokens. To inspect a recorded decision in exported traces, look for chalk.router.judge.classification, chalk.router.judge.selected_model, chalk.router.judge.fallbacks, and chalk.router.judge.shadow. These follow your trace export and sampling settings. Judging applies to Chat Completions, Anthropic Messages, and self-contained OpenAI Responses creation, including streaming requests. Responses requests using previous_response_id, conversation, or a stored prompt use the default target because the judge does not fetch that external context. Their fallback eligibility remains subject to connection affinity. Other operations, including Decisions and Gemini Interactions, do not invoke a route judge. Successful remote classifications are cached for 60 seconds. Local heuristic classifications are computed on every request. The cache distinguishes the request endpoint, key, request content, and judge configuration and backend. Changing the judge configuration invalidates cached classifications. Cached labels are applied to the current request's routing configuration, so a change to the route fallback map can reuse a classification while applying the updated chain. Remote judges receive a bounded text view of the request; classification does not provide multimodal understanding of embedded images, audio, or video. ### Access Access decides who may call the Model Gateway and how much they may spend. Configure all of it under Model Gateway → Access in the Chalk dashboard. ### API keys Issue and revoke Model Gateway API keys from Model Gateway → Access → API keys in the Chalk dashboard, or from the CLI with chalk router api-key (create, list, and revoke). Scope each key at issue time so that a leaked or over-eager client cannot do more than you intend: | Setting | Effect | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Description** | A human-readable label for the key. | | **Connection scopes** | The [provider connections](#provider-connections) the key may use. Each scope covers all models on that connection, or the specific models you select. | | **Usage pool** | Associates the key with a [usage pool](#usage-pools), so pool-scoped rate limits apply. | | **Labels** | Custom key/value metadata. | Each key tracks its total token usage, and revoking a key takes effect immediately. ### Usage pools A usage pool groups API keys so you can manage and limit access by pool rather than key by key. Create a pool, then assign keys to it at issue time. Manage pools from the CLI with chalk router usage-pool (create, list, and delete). Scope a rate limit to a pool so every key in it shares one budget. Deleting a pool leaves its keys working and removes their pool association. ### Usage budgets A usage budget caps how many tokens a key or a pool may consume in a fixed window. Create budgets under Model Gateway → Access → Usage budgets. Each budget sets three things: - Scope: an individual API key, or a usage pool. - Token limit: the tokens allowed in one window. - Period: daily, weekly, or monthly. You can disable a budget rather than delete it. Each one reports the tokens counted so far in the current window and the time of the next reset. That counter decides admission rather than serving as a usage ledger, so read it as how close a scope is to being cut off, and use Access → Usage for historical accounting. ### Configuring rate limits Rate limit policies cap throughput and protect against runaway spend. Each policy has a limit type, an optional target, and a scope: - Limit type: tokens per minute, requests per minute, or concurrent requests. - Target (optional): narrow the policy to a specific provider or model. Leave it unset to apply across all traffic. - Scope: apply the limit per individual key (token) or shared across a usage pool. Enable or disable a policy without deleting it. Token-based limits — tokens per minute, and any usage budget — need a conservative token estimate before the Model Gateway forwards a request. A request carrying images, audio or video, encrypted content, or a reference to a stored conversation cannot be estimated. The Model Gateway rejects that request with HTTP 429 without contacting the provider. Send the conversation history inline rather than referencing a stored conversation to avoid this. ### Observability The Model Gateway records a trace for each request. Read traces for one key from the Traces tab on that key's detail page, and choose where the Model Gateway exports them under Model Gateway → Telemetry. ### Sampling policy A sampling policy decides which traces the Model Gateway keeps, and whether it records prompt and completion content. Edit it under Model Gateway → Routing → Sampling policy. The policy is an ordered list of rules. The first rule whose target matches a request supplies both the keep rate and the content setting for that request, and requests matching no rule fall to the policy defaults. A rule matches on one of: - The x-chalk-request-tag header the caller sends. - The calling key. - The calling key's usage pool. Keep rate and content logging are independent. A trace the policy keeps may carry no content, and a request a rule marked for content logging may still be sampled away. Outcome rules run once the response is known. Each reads a field from the response body and compares it to a value, and the first one that matches replaces the keep rate chosen at request time. Some behavior applies regardless of which rule matches: - A request that returns an error is always exported. - Conversation rate floor sets a keep rate, between 0 and 1, that a session-keyed trace gets whatever the rules decided. - A deployment-wide setting turns content logging off entirely, overriding every rule that would record it. ### See also - Authentication: service credentials and RBAC for the Chalk API. - MCP Gateway: govern the external MCP servers your agents reach. - LLM Toolchain: building AI features and agents on Chalk. # MCP Gateway source: https://docs.chalk.ai/docs/mcp-gateway ## Register, secure, and govern the external MCP servers your agents connect to. The Chalk MCP Gateway is a control plane for the external Model Context Protocol (MCP) servers your agents use. Instead of pointing each agent at a handful of MCP servers directly — each with its own URL, credentials, and trust assumptions — you register those servers once with the gateway. The gateway aggregates their tools, resources, and prompts behind a single connection, holds the credentials, enforces who may call what, and records every operation. The MCP Gateway is distinct from the MCP Server. The MCP Server exposes Chalk itself to agents (run queries, execute Chalk SQL, inspect your deployment). The MCP Gateway sits in front of other MCP servers and governs your agents' access to them. ### Deploying the gateway The gateway runs as a service in your Chalk environment, alongside the engine. To deploy it, open the Cloud Resource Configuration pane, add MCP Gateway to a resource group from the Add Config menu, where it is listed under Advanced, set its resource request, and choose Save and Apply Service. See Resource Configuration for how services and resource groups work. The gateway is unavailable in an environment until the service is added, so deploy it before registering backends. ### Registering MCP servers Register and manage backend servers from MCP gateway → MCP servers in the Chalk dashboard. Each server has: | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------ | | **Name** | A unique name for the backend. | | **Transport** | `HTTP` for a remote server (set a URL), or `stdio` for a local process (set a command and args). | | **Credential** | The credential used to connect (see [Credentials](#credentials)). | Once a server is registered and connected, the dashboard shows its connection status and lets you browse everything it exposes: its tools (with input schemas), resources (with URI, MIME type, and description), and prompts (with arguments). You can also call a tool directly from the UI to verify it works. ### Credentials Credentials are stored with the gateway, not with the agents, so secrets never have to be distributed to clients. The gateway supports several credential kinds to match how each backend authenticates: | Kind | Use for | | ------------------------- | ---------------------------------------------------------------------------------------- | | **Bearer token** | A server that accepts a static `Authorization: Bearer` token. | | **Headers** | A server that needs one or more custom HTTP headers. | | **Environment variables** | A `stdio` server that reads secrets from its environment. | | **OAuth (user)** | A server behind an OAuth 2.0 user flow (authorize/token URLs, client ID/secret, scopes). | | **OAuth (DCR)** | A server that supports OAuth Dynamic Client Registration (issuer + scopes). | For OAuth-backed servers, each user links their own account, so calls run with that user's identity rather than a shared credential. ### Connecting an agent Agents connect to the gateway over MCP's Streamable HTTP transport at your Chalk API host: ``` https:///v1/mcp ``` For Chalk's hosted metadata plane this is https://api.chalk.ai/v1/mcp. Every request carries two headers: | Header | Value | | ---------------- | ------------------------------------------------------------------------------- | | `Authorization` | `Bearer `, where the token is the output of [`chalk token get`](/cli/token_get) | | `X-Chalk-Env-Id` | The environment id that `chalk config` reports | Run chalk login once before chalk token get. The token identifies you, so the gateway applies your permissions and its authorization policies to every call the agent makes. A token lasts up to 8 hours; run chalk token get again when it expires. The MCP Servers page of the MCP Gateway section in the Chalk dashboard has an Add to… menu that generates this configuration for the current environment, for Claude Code, Codex, Cursor, Zed, and stdio-based clients. For Claude Code, the generated commands are: ``` export CHALK_USER_TOKEN="$(chalk token get)" claude mcp add --transport http chalk-gateway 'https:///v1/mcp' \ --header "Authorization: Bearer $CHALK_USER_TOKEN" \ --header 'X-Chalk-Env-Id: ' ``` Confirm the gateway answers before you configure a client by sending it an MCP initialize request: ``` curl -sS https:///v1/mcp \ -H "Authorization: Bearer $(chalk token get)" \ -H "X-Chalk-Env-Id: " \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' ``` A running gateway returns a JSON-RPC response whose result contains protocolVersion, capabilities, and serverInfo. A 401 means the token is missing or expired. A response reading MCP gateway is not available for this environment means no gateway is deployed in that environment. ### Authorization policies Policies decide which users and agents may call which tools, and with which arguments, independent of the backend itself. You write them in Rego, Open Policy Agent's policy language. A policy has a name, its Rego source, and an enabled flag, so you can keep a policy in place and turn it on or off without deleting it. Beyond per-tool policies, you can scope an agent to a fixed set of allowed backends, so a given agent only reaches the servers it is meant to use. ### Writing a policy The gateway evaluates data.chalk.policy.deny. Write your rules in a chalk.policy package, returning a message for each reason a request should be refused: ``` package chalk.policy # Only the platform owner may merge pull requests. Replace # with that user's Chalk user ID. deny contains msg if { input.request.method == "tools/call" input.gateway.upstream_tool == "merge_pull_request" input.auth.subject != "" msg := sprintf("user %v is not permitted to call merge_pull_request", [input.auth.subject]) } ``` Policies are deny-only. A request proceeds when no deny rule matches, so a policy lists what to refuse rather than what to permit, and an allow rule has no effect. Each message a rule produces becomes one of the decision's reasons, and those reasons appear in the audit log. ### What a policy can read Every evaluation receives one input document describing the request: | Path | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `input.auth.subject` | Who made the request: a Chalk user ID for a user token, the service token's ID for a service token, or the mapped agent name for a token from a trusted issuer. For Chalk users it is the user ID, not an email address. | | `input.auth.scopes` | The scopes that user holds. | | `input.gateway.backend` | The registered backend the request is bound for. | | `input.gateway.upstream_tool` | The tool name on that backend, for tool calls. | | `input.gateway.upstream_uri` | The resource URI on that backend, for resource reads. | | `input.gateway.upstream_prompt` | The prompt name on that backend, for prompt gets. | | `input.request.method` | `tools/call`, `resources/read`, or `prompts/get`. | | `input.request.params.arguments` | The call's arguments, for tool calls and prompt gets. | | `input.schema_version` | The version of this document's shape. Currently `1`. | To find the ID a rule should compare against, open the policy simulator and choose a team member or service token from its User list. The simulator shows the input.auth.subject value it sends. Policies cover resource reads and prompt gets as well as tool calls. Match on input.request.method when a rule should apply to only one of them. Match against the input.gateway fields instead of their equivalents under input.request.params. The gateway namespaces names and encodes URIs before passing them upstream, so a resource read arrives with input.gateway.upstream_uri set to bigquery://project/dataset/table while input.request.params.uri holds chalk-mcp://resource/v1//bigquery%3A%2F%2F.... Matching the encoded form works but is easy to get wrong, and a rule that fails to match allows the request. ### More examples Restrict a tool by its arguments: ``` package chalk.policy # The SQL tool may only run reads. deny contains msg if { input.gateway.upstream_tool == "execute_sql_query" not startswith(lower(trim_space(input.request.params.arguments.query)), "select") msg := "execute_sql_query is limited to SELECT statements" } ``` Require a scope for writes: ``` package chalk.policy deny contains msg if { input.gateway.upstream_tool in {"create_issue", "update_issue"} not "write" in input.auth.scopes msg := sprintf("%v requires the write scope", [input.gateway.upstream_tool]) } ``` Protect a set of resources: ``` package chalk.policy # Only these users may read the private tables. private_table_readers := {"", ""} deny contains msg if { input.request.method == "resources/read" startswith(input.gateway.upstream_uri, "bigquery://project/dataset/private_") not input.auth.subject in private_table_readers msg := sprintf("user %v cannot read private tables", [input.auth.subject]) } ``` ### Simulating before you enable Simulate a policy against a hypothetical request before turning it on. Supply a user, scopes, backend, tool, and arguments, and the gateway returns the decision it would reach along with its reasons. The backend, tool, prompt, and resource fields list what your registered MCP servers expose, and the simulator warns when a backend, tool, or prompt you enter is not among them. A policy that fails to deny something you expected usually has a field name or tool name wrong, since an unmatched rule fails silently. ### Audit log The gateway records every operation it handles: tool calls, resource reads, and prompt fetches. The recent audit feed shows, for each one: | Column | Description | | ----------------- | -------------------------------------------------------------------------------- | | **Time** | When the event was recorded. | | **Agent** | The agent, or user, that made the request. | | **Operation** | `tools/call`, `resources/read`, or `prompts/get`. | | **Target** | The namespaced tool, resource URI, or namespaced prompt the operation addressed. | | **Backend** | The registered server the request was bound for. | | **Policy** | What your policy recommended: `allow`, `deny`, `error`, or `NOT RUN`. | | **Outcome** | What the gateway did: `allow`, `deny`, or `error`. | | **Policy reason** | The messages your policy returned with a deny. Empty on an allow. | | **Arguments** | A preview of the call arguments, truncated when long. | | **Duration** | How long the call took, in milliseconds. | ### Policy and outcome Policy is what your policy recommended. Outcome is what the gateway did. A request can be allowed despite a deny recommendation, so the two are recorded separately. Three combinations have specific meanings: - Policy deny, Outcome allow: the policy ran in audit mode, which records decisions without enforcing them. - Policy error, Outcome allow: the policy failed to evaluate. The gateway allows the request rather than blocking on a rule it cannot run. - Policy NOT RUN: no policy evaluated the request, which is the case when none is enabled. ### See also - MCP Server: connect agents to your Chalk deployment over MCP. - Model Gateway: OpenAI-compatible LLM routing for your Chalk deployment. # LLM Toolchain source: https://docs.chalk.ai/docs/chalk-for-ai-engineers ## Integrate unstructured data and build context-aware prompts with Chalk. Chalk's LLM toolchain enables you to build enterprise-grade AI applications without stitching together LLMs, prompts, vector DBs, and retrieval logic: - Call chat completion APIs out-of-the-box - Cache expensive computations to avoid redundant processing and reduce latency - Retrieve real-time context into LLMs - Generate embeddings and vectors within pipelines - Define inline prompts - Score LLM outputs against a dataset with Evaluations Chalk provides a full-stack solution but can also integrate seamlessly with your current infrastructure. For instance, you can use an existing vector DB to store and retrieve your embeddings, while leveraging Chalk to evaluate new models and prompts. Chalk lets you integrate only the pieces you need: ``` import chalk.functions as F import chalk.prompts as P from chalk.features import DataFrame, Primary, Vector, embed, feature, features, has_many, _ from pydantic import BaseModel # Use structured output to easily incorporate unstructured data in our ML pipelines class AnalyzedReceiptStruct(BaseModel): expense_category: ExpenseCategoryEnum business_expense: bool loyalty_program: str return_policy: int @features class Transaction: id: int merchant_id: Merchant.id merchant: Merchant receipt: Receipt llm: P.PromptResponse = P.completion( model="gpt-5.1-2025-11-13", messages=[P.message( role="user", content=F.jinja( """Analyze the following receipt: Line items: {{Transaction.receipt.line_items}} Merchant: {{Transaction.merchant.name}} {{Transaction.merchant.description}}""") )], output_structure=AnalyzedReceiptStruct, ) # cache forever since transaction is finalized expense_category: str = feature( max_staleness="infinity", expression=F.json_value( _.llm.response, # from LLM "$.expense_category", ), ) @features class ProductRec: user_id: Primary[User.id] user: User # generate embeddings user_vector: Vector = embed( input=F.array_join(F.array_agg( _.user.products[ _.name, _.type == "liked" ]), delimiter=" || ", ), provider="vertexai", model="text-embedding-005", ) # do a vector search with the generated embedding similar_users: DataFrame[User] = has_many( lambda: ProductRec.user_vector.is_near( User.liked_products_vector ) ) ``` Chalk eliminates the complexity of orchestrating data and ETL pipelines by building a dependency graph (DAG) of your features, which are defined using Python. At inference time, Chalk dynamically builds query plans (subgraphs of your feature DAG) without manual configuration, based on the features you request. The (structured) outputs of chat completions are treated as just newly computed features and can feed into other feature computations until all of the features requested are computed. Chalk for AI engineers ### Call chat completion APIs out-of-the-box with Chalk's Completion function Use Chalk’s P.completion function to call LLMs from within your feature classes. Easily create structured LLM responses with full control over model selection, prompt construction, and output formatting. Combined with F.jinja templates, you can dynamically inject relevant feature values into your prompts for context-aware responses. ``` @features class Transaction: llm: P.PromptResponse = P.completion( model="gpt-5.1-2025-11-13", messages=[...], ) ``` P.completion accepts a range of parameters for fine-tuning your LLM interactions, including: - Model selection - Temperature control for output randomness - Token limits - Structured output formatting via Pydantic models - Various provider-specific settings You can also set timeout values, retry logic, and custom stop sequences to build the exact experience your application requires. Visit our API documentation for a complete reference of all available parameters. After executing a completion, Chalk returns a P.PromptResponse object that encapsulates the complete interaction, including: - response: the model's generated text - prompt: the original prompt for context - usage: token usage and cost metrics - runtime_stats: execution timing data to help you monitor and optimize your model interactions ``` class PromptResponse(BaseModel): response: Optional[str] = Field( description="Response from the model. Raw string if no output structure specified, json encoded string otherwise. `None` if the response was not received or incorrectly formatted." ) prompt: Prompt usage: Usage runtime_stats: RuntimeStats ``` After receiving a response from an LLM via Chalk's completion function, you can easily extract specific fields from the structured response using F.json_value. The extracted LLM-generated values become referenceable like ordinary features: ``` @features class Transaction: id: int llm: P.PromptResponse expense_category: str = F.json_value( _.llm.response, # from LLM "$.expense_category", ) ``` You can also compose multimodal prompts — combining image and text in a single call using P.completion(...). This unlocks image analysis workflows like receipt or PDF parsing with minimal code. ``` @features class Receipt: image_url: str image_response: P.MultimodalPromptResponse = P.completion( model="gpt-5.1", messages=[ P.message( "user", [ {"type": "input_text", "text": "describe this image"}, {"type": "input_image", "image_url": _.image_url}, ], ), ], ) ``` Chalk's LLM toolchain streamlines the entire experience of working with language models by providing: - Direct access to LLM providers without managing API keys, request formatting, or response parsing - A consistent API interface regardless of which LLM provider you're using - Automatic tracking of token usage, costs, and performance metrics - Built-in error handling with automatic retries without additional code - Easy extraction of fields from LLM responses for direct use as features - Seamless integration with Chalk's feature system for dynamic data incorporation through F.jinja ### Cache expensive computations to avoid redundant processing and reduce latency with Chalk's max_staleness setting Chalk makes it easy to cache features. This is done by setting the max_staleness of a feature, which lets you specify how old a cached feature value can be before it must be recomputed. This gives you full control in balancing freshness and performance requirements. You can specify cache duration in multiple ways: | Value | Description | | ---------------------- | ------------------------------------ | | `"30d"` `"1h"` `"15m"` | Natural language strings | | `timedelta(hours=1)` | Python timedelta objects | | `"infinity"` | Permanent caching for immutable data | These settings can be overridden at query time for additional flexibility. ``` @features class User: id: int # from API or 3rd-party client credit_score: int = feature( max_staleness="30d", ) @features class Transaction: id: int llm: P.PromptResponse # cache forever since transaction is finalized expense_category: str = feature( max_staleness="infinity", expression=F.json_value( _.llm.response, # from LLM "$.expense_category", ), ) ``` ### Retrieve real-time context to give LLMs with Chalk's Jinja function Chalk's F.jinja function lets you build prompts with live data. Feature values are accessed with the Jinja double curly brace syntax: {{ feature_name }}. ``` F.jinja( """Analyze the following receipt: Line items: {{Transaction.receipt.line_items}} Merchant: {{Transaction.merchant.name}} {{Transaction.merchant.description}}""" ) ``` Benefits of using Jinja for LLM prompting: - Improve performance by dynamically incorporating real-time data from Chalk features - Reduce token usage by only including relevant context in your prompts - Iterate on prompts faster with a structured template system that allows quick experimentation while maintaining consistency across team workflows ### Generate embeddings and vectors within pipelines with Chalk's embed function Convert structured or unstructured data into embeddings with one line. Chalk's embedding function automatically converts your features into vector embeddings for similarity search, etc. with built-in support for popular embedding models. Running in your VPC gives you quick and secure access to models like Bedrock and Gemini with minimal overhead and configuration, as Chalk abstracts away connecting to model providers. ``` @features class ProductRec: user_id: Primary[User.id] user: User bio_embeddings: Vector = embed( input=lambda: ProductRec.user.bio, provider="vertexai", model="text-embedding-005", ) ``` When using Chalk's built-in embedding functions, the vector dimensions don't need to be specified, as Chalk will automatically infer them from the embedding model. # Chalk Notebooks source: https://docs.chalk.ai/docs/chalk-notebooks ## Mix and match SQL, Python, and Markdown cells to explore, analyze and productionize features. Chalk Notebooks build on the notebook format you already know. You'll be on familiar ground if you've spent time in Jupyter or a similar environment: the interface is made up of cells. Chalk extends that with additional cell types, and your features, resolvers, and queries are all available from a notebook, so you can explore your data and create new features in one place. Those cells run on Chalk Compute, which runs sandboxed compute inside your own cloud, so notebook code executes on infrastructure you own and control rather than in a Chalk-hosted environment. Any data source you have already configured under Integrations is natively queryable from a notebook — no re-configuration and no new credentials. This page covers the notebooks Chalk hosts for you, which you can drive from the dashboard or from the chalk notebook CLI. You can also connect a notebook you run yourself, in Jupyter, Colab, Hex, or Deepnote: see External Notebooks for that setup, and the Jupyter notebook tutorial for a worked example that builds features and resolvers end to end. ### Example notebook Use SQL to select orders, Python to summarize the resulting DataFrame, and an online query to inspect Chalk features and their execution plan. This example brings the three cells together; select an operator in the plan to inspect its details. ### Cell types Pick a type when you add a cell. - Python: transforms data, runs Chalk queries, and displays results. - SQL: queries Chalk features, configured datasources, and notebook DataFrames. - Text and Markdown: adds prose as plain text or Markdown, with template expressions for live values from the notebook session. - Input: an interactive parameter bound to a Python variable. The available input types are Text, Number, Checkbox, Toggle, Date, Dropdown, Multiselect, and Table. - Section: a title cell, for grouping a notebook into named parts. - Single value: displays a single value computed anywhere in the notebook, rendered prominently. - Table: displays an existing DataFrame as a full-width, browsable table — a variable an earlier cell produced, a Chalk dataset, or a table in a configured SQL datasource. Display-only: it does not run and does not define a variable. ### Cell outputs Every cell that computes something leaves its result behind for the rest of the notebook to use. Output renders directly beneath the cell that produced it, and the value is bound to a name that later cells can read. ### Result variables and the variable explorer All cells share one Python session, so a name defined in one cell stays defined for the cells after it. Python cells bind names the ordinary way, by assignment. SQL and Input cells bind their output to a result variable instead. You can rename that variable, and it is worth naming it well early, because downstream cells refer to it by name. The variable explorer lists what the session currently holds, with a preview of each value. For examples and output details, see SQL cells, Python cells, and Text and Markdown cells. ### Chaining cells Because cell results can be referenced by name, work can be split across cells instead of piling into one. A SQL cell pulls a dataset, a Python cell reads that result by name and reshapes it, and a Single value cell surfaces one number from the reshaped frame. Each step stays independently runnable and independently readable. You never declare these connections. Chalk infers each cell's dependencies from what that cell reads and what it writes, and those inferred dependencies drive the notebook's run modes and its tracking of which cells have gone stale. ### Driving a notebook from the command line Everything on this page also has a CLI equivalent. chalk notebook create, cell add, run, get, and results cover creating a notebook, adding and running cells, and reading back their output from a terminal or a script instead of the dashboard, and chalk notebook dependencies lists the packages installed in a notebook's kernel. See the CLI reference for every command and flag, or run chalk notebook --help. When querying a branch from Python, set the branch on the client inside the cell. See Querying a branch. An AI coding assistant can reach notebooks two other ways. Chalk's MCP server exposes the same ground as tools, covering creating a notebook, editing its cells, running them, and reading their results. Separately, the chalk-notebooks agent skill covers the same workflow for assistants that read one. ### The notebook toolbar Every notebook's own controls sit in one bar at the top right, above the cells. The rest of this page walks that bar from left to right. ### App view App switches the notebook into app view, a dashboard canvas that presents results without the code behind them. It is a view toggle, not a publish step: the button flips to Notebook, which switches you back, and the view is recorded in the URL, so a refresh or a pasted link reopens it. Nothing reaches the canvas on its own. Every cell carries an Add to app dashboard button in its action rail, and clicking it places that cell on the app dashboard; the same button then offers Remove from app dashboard to take it back off. The Add cell dropdown in app view is the second path, and it also offers Add all cells — a notebook with no placed cells opens as an empty canvas. Placed cells render their results only — source code is never shown. Input cells stay interactive, so the controls that drive the notebook drive the app too. Section cells render as titles, and Single value cells render as metrics. You position widgets by dragging and resizing them, and the layout persists. ### Error count The warning badge shows how many cells have errored. Click it to jump to the cell with the error. ### Running cells and shortcuts How many cells a run covers is a setting. Pick a Run mode from the dropdown next to Run all, and every subsequent run obeys it until you change it. The mode is expressed in terms of the dependencies described above: "upstream" means the cells the selected cell reads from, and "downstream" means the cells that read from it. Auto: runs the selected cell and everything downstream of it, and refreshes any cells that had gone stale. Auto is the default, and a run in Auto never leaves a cell stale. In every other mode, a cell goes stale when something it depends on runs more recently than it did. Cell + upstream: runs the cells the selected cell depends on, then the selected cell. Cell only: runs the selected cell and nothing else. Cell + downstream: runs the selected cell, then the cells that depend on it. A handful of shortcuts are worth knowing. The Esc-prefixed ones are command-mode shortcuts: pressing Esc leaves the cell's editor while keeping the cell selected, so the keys that follow act on the cell as a whole rather than on its text. Finish a sequence within two seconds, or it resets. | Shortcut | Action | | ----------------------- | -------------------------------------------------------------------------- | | `Shift + Enter` | Run the selected cell using the active run mode, then select the next cell | | `Cmd + Enter` | Run the selected cell using the active run mode and stay on it | | `Cmd + Click` | Select multiple cells | | `Esc` then `A` | Insert a Python cell above the selected cell | | `Esc` then `B` | Insert a Python cell below the selected cell | | `Esc` then `D`, `D` | Delete the selected cell | | `Esc` then `Cmd + Up` | Move the selected cell up one position in the notebook | | `Esc` then `Cmd + Down` | Move the selected cell down one position in the notebook | | `Esc` then `F`, `F` | Open the selected cell full screen | | `Esc` then `F`, `S` | Format the selected SQL cell | Two whole-notebook actions sit in the same place. Run all runs the notebook end to end, and Run all without cached results does the same with caching bypassed, so every SQL cell executes against its datasource. ### Comments Comments come at two levels. The speech-bubble button in the toolbar opens the notebook's comments, and every cell carries its own comment button, which comments on that cell alone — so a notebook is something several people can work on together. ### Dependencies The dependencies button opens a Dependencies drawer on the right-hand side of the notebook — "Variable data-flow between cells, analyzed by the kernel." It draws the notebook top to bottom as an interactive graph. The drawer is resizable, and it is recorded in the URL, so a refresh or a pasted link reopens it. Most nodes are cells, drawn as a miniature preview of the source with the cell's name, how long its last run took, and a pill for each variable the cell outputs. A ring around a node marks it as selected, succeeded, or errored. Input cells render their live control inside the node. A SQL cell's datasource appears as its own node upstream of the cell, labeled with the datasource name. Edges are data flow, labeled with the actual variable names crossing them. Selection syncs in both directions: selecting a cell pans the graph, and clicking a node reveals that cell in the notebook. A minimap, zoom to fit, and zoom to selection handle navigation in a large notebook. The graph comes from static analysis of your Python source, run inside the kernel process. Cells are not executed to build it. Only module-level names are considered, so a name bound inside a function body or a class body does not become a dependency. Two limits are worth knowing. The drawer needs a running notebook runtime: without one you see the cells but no edges, and the empty state says so — "Add runnable cells, then start the notebook runtime so the kernel can analyze their variable dependencies." And the graph is built from the saved document, so an edit you have just typed is not reflected until autosave picks it up. ### Sharing a notebook The share button creates read-only links to the notebook. Each link carries a label, visible only to you, which is how you tell links apart later when you go to revoke one. Require a Chalk account is on by default. With it on, a viewer has to sign in and already have read access to the environment — the link points them at the notebook, it does not grant access on its own. With it off, anyone holding the URL can open the notebook without an account, so treat the URL itself as the credential. Each link has an expiry: 7, 30, or 90 days, or 1 year. The default is 30 days. The URL is shown exactly once, when you create the link. Chalk stores only a hash of it and will not display it again, so copy it then — if you lose it, revoke the link and create a new one. Active links lists every link the notebook has: its label, whether it requires sign-in or is public, when it expires, when it was last opened, and a Revoke button. Whoever opens a link sees the notebook, including its cells and the results of the most recent run, but cannot edit it, run it, read its secrets, or reach the app canvas. Interpolated Markdown renders for them too, since resolved values are saved with cell output; see Text and Markdown cells. Shared pages are set to noindex, so search engines do not index them. ### Notebook settings Runtime options that apply to the whole notebook can be found in the Notebook settings panel, which can be opened from the settings button in the toolbar. From there, you can: - Define the notebook's secrets. - Configure its compute resources and the behavior of its kernel. - Turn on Vim keybindings. - Turn on AI autocomplete. - Read the kernel's runtime information. ### History and versions The history button opens History & versions, one reverse-chronological timeline of what has happened to the notebook. Recent runs appear in it, paginated behind Load more runs, each labeled by what the run covered ("Ran all cells", "Ran cell + upstream"), along with its status, its duration, and who ran it. Open a run to see it cell by cell: per-cell status and duration, output and row counts, and the text of any error inline. Run history is retained for 30 days. Cell edits share the same timeline, one row per cell, and open into a syntax-highlighted view of that cell's source. # Python cells source: https://docs.chalk.ai/docs/notebook-python ## Run Python, query Chalk features, and install packages in a notebook. Use Python cells to transform data, run Chalk queries, and display results. Cells share one Python session, so variables you assign in one cell are available to later cells. You can also read the named results of SQL cells and values from Input cells. ### Python example This Python cell reads the orders DataFrame from the SQL example, selects completed orders, and prints their total revenue. The two completed orders total $214.50. Select Run to replay the sample result; this example does not connect to a Python kernel. ### Tables and DataFrames Tabular results render as browsable tables. You can work with pandas, polars, pyarrow, and Chalk's own DataFrame. Leave a value as the cell's last expression to display it. Assign it to a variable when later cells need to transform, join, or plot it without repeating the computation. ### Query results Run online and offline queries with the same ChalkClient methods you use outside a notebook. Leave a query call as the cell's last expression to display its response as a table of features and values, with tabs for query metadata and the saved execution plan. The Plan tab shows the saved execution plan. This example loads a recorded plan into the same interactive viewer used in notebooks. Select an operator to inspect it, or use the zoom controls to explore the graph. Offline queries return before their work finishes. offline_query hands back a Dataset as soon as the revision is created, and realizing that dataset waits for the computation to complete, reporting progress while it waits: ``` dataset = client.offline_query( input={"user.id": list(range(1000))}, output=["user.fraud_score"], recompute_features=True, ) # Waits for the revision to finish, then returns the rows. df = dataset.to_pandas() ``` ### Loading feature classes A notebook's kernel is a hosted Python environment, separate from your laptop and from your project's repository. The kernel does not have your project's package on its path, so from src.models import User raises ModuleNotFoundError. Refer to features by their string names instead: ``` from chalk.client import ChalkClient client = ChalkClient() client.query(input={"user.id": 1}, output=["user.fraud_score"]) ``` Or call load_features to bind the deployed feature classes into the session, and then use them the way you would in your project: ``` client.load_features() client.query(input={User.id: 1}, output=[User.fraud_score]) ``` load_features reads the feature classes from whatever deployment the client points at, so a client scoped to a branch loads that branch's features. To scope a client to a branch, see Querying a branch. ### Installing packages The kernel ships with the chalk client and a data-science baseline that includes pandas, polars, numpy, and Altair. It does not include every library you might reach for: matplotlib, plotly, seaborn, scikit-learn, and xgboost are absent, and importing one of them raises ModuleNotFoundError. Install what you need from a Python cell with !uv pip install. Both pip and uv are on the kernel's path. This cell replays a recorded SciPy installation when it comes into view; versions, timings, and downloads depend on your environment. To browse installed packages in the dashboard, select the settings icon in the notebook's upper-right toolbar, then scroll to Installed packages in Notebook settings. Start the kernel if it is stopped; the list loads once it is ready. Use the filter to find a package and check its version. This example shows SciPy and its NumPy dependency after the installation above. To list what a particular notebook's kernel already has, run chalk notebook dependencies against it. ### Querying a branch The --branch flag on chalk notebook cell add and chalk notebook run scopes that CLI invocation to a branch. It doesn't change what branch a Python cell's own code talks to. A cell that constructs its own ChalkClient() runs in the notebook's kernel, a separate process from the CLI, so the CLI's --branch flag never reaches it. If a cell queries a branch-only feature through a ChalkClient() with no branch set, the query doesn't raise. The cell's status still comes back SUCCESS, and the failure only shows up in the result, as a message like Query output referenced undefined feature 'user.risk_score'. Pass branch explicitly to the client constructed inside the cell to query that branch. # SQL cells source: https://docs.chalk.ai/docs/notebook-sql ## Query Chalk features, configured datasources, and notebook DataFrames with SQL. Use SQL cells to query Chalk features, tables in a configured Integration, or DataFrames in the notebook session. Give the result a variable name so later SQL, Python, and Markdown cells can use it. ### SQL example This Chalk SQL cell selects orders from the fictional orders feature namespace and binds its DataFrame result to orders. Use the two selectors above the query to choose a resource group and SQL dialect: - Resource group: choose the compute resources that execute a Chalk SQL query. The example selects default; other resource groups run queries on their independently configured resources. - SQL dialect: choose Chalk SQL to query Chalk features, as this example does with the orders namespace. Select a configured datasource, such as Analytics, to query its tables using that datasource's SQL dialect, or DataFrames to query DataFrame variables in the notebook session. Use the selectors, expand or collapse the source and output, or select Run to replay the sample result. This example does not connect to a warehouse. ### Result variables and tables A SQL cell binds its output to a result variable. Rename the variable to describe the data it holds; downstream cells refer to it by that name. The notebook's variable explorer lists the session's variables with a preview of each value. Results render as browsable tables. You can page through rows, aggregate columns, and download the result from the cell's output. ### Querying DataFrames Select DataFrames as the query target to query a DataFrame produced by an earlier cell. This lets you split a longer query into stages or query data prepared in Python. Chalk infers the dependency from the variables the cell reads, so its producers can run first under the notebook's run modes. # Text and Markdown cells source: https://docs.chalk.ai/docs/notebook-text ## Write notebook narrative as plain text or Markdown and interpolate values from other cells. Use Text cells to add explanations, notes, and summaries alongside your code and results. Write plain text when you want to show the source verbatim, or use Markdown for formatted headings, lists, links, and live values from the notebook session. ### Plain text Plain Text cells display exactly what you write. Double braces such as {{ revenue }} remain literal text. Use Markdown when you want an expression to resolve to its current value. ### Markdown Markdown cells render formatted prose. You can combine that prose with values computed by Python or SQL cells, as the order summary below demonstrates. ### Template interpolation in Markdown Every {{ expression }} site in the prose is evaluated against the same Python session the code cells use and rendered in place, so narrative text stays consistent with the data beside it. There is no separate templating language. What sits between the braces is ordinary Python: a variable, an attribute, an index, or a call. Result variables are in scope under their own names, so {{ my_sql_result["col"][0] }} works with no namespace prefix. Three forms cover the syntax: - {{ expression }} renders the value. - {{ expression:format_spec }} applies a Python format spec, exactly like an f-string: {{ ratio:.1% }} renders 12.3%, and {{ total:,.2f }} renders 1,234,567.89. Only the first colon at the top level of the site splits expression from format spec, so colons inside brackets or string literals (a slice like {{ xs[0:2] }}, for instance) are left alone. - \{{ escapes a literal {{, for prose that needs double braces verbatim. This Markdown cell uses completed and revenue from the Python example. The saved output substitutes the order count and formats revenue to two decimal places. Scalars render inline, DataFrames render as a bounded preview table, and an expression that raises renders an inline error marker at its own site while the rest of the document still renders. A site in a cell that has not run yet shows its source as inline code. Resolved values are saved with the cell's output, so a shared notebook renders its interpolated values without a live session — you only need one to refresh them. Two limits are worth knowing: only Markdown cells render resolved values, since plain Text cells show their source verbatim, and chart objects are not rendered at a template site, so display those from a Python cell. ### Dependencies and refresh Markdown cells with template sites take part as pure consumers. The names their expressions read become dependencies on the cells that produce those names, so Auto mode re-renders prose when its upstream cells re-run, Cell + upstream on a Markdown cell runs the producers it reads from before rendering, and editing an upstream cell marks the prose stale like any other dependent. The expressions never bind names, so nothing can depend on a Markdown cell in turn. See Running cells and shortcuts for the notebook's run modes. # External Notebooks source: https://docs.chalk.ai/docs/notebook-development ## Use Chalk from the external notebook of your choice. You can write and test new features from an external notebook: Jupyter, Colab, VS Code, or a hosted environment like Deepnote or Hex. To get started, you'll want to install and connect Chalk to your external notebook environment. Generally this involves two steps: - Installing the chalkpy[runtime] package, - Authenticating to Chalk (using the chalk CLI or by setting environment variables) We provide specific setup instructions for Deepnote, Hex and Vertex, but the process should be similar for other notebook environments. ### Installing the Chalk Python Dependency To install dependencies, run the !pip install "chalkpy[runtime]" bash command directly in a cell. Alternatively, you can add and install the chalkpy[runtime] dependency in your kernel's virtual environment. ### Connecting Your External Notebook to Chalk Authenticating takes a client ID and client secret pair. Authentication for External Notebooks covers generating that pair and wiring it into a local notebook, Hex, Deepnote, or Vertex. Once the credentials are available to your notebook, construct a client: ``` import os from chalk.client import ChalkClient client = ChalkClient( client_id=os.environ.get("CHALK_CLIENT_ID"), client_secret=os.environ.get("CHALK_CLIENT_SECRET"), ) ``` ### Verifying the Setup To verify that everything is properly set up, you can run the following code in a cell: ``` client.whoami() ``` If everything is properly configured, you should see your user_id, team_id and environment_id come back: ``` WhoAmIResponse(user='', environment_id=, team_id='') ``` ### Loading Features Now we can start building some new features! To load your Chalk features into your notebook, you'll want to run: ``` from chalk.client import ChalkClient client = ChalkClient() client.load_features() ``` This will print out your available features and load them into your notebook: ``` Director ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Feature ┃ Type ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ id │ str │ │ name │ str │ │ birth_year │ int │ │ active │ bool │ │ favorite_numbers │ list[int] │ │ death_year │ int | None │ │ known_for_titles │ list[str] │ │ known_for_genres │ list[str] │ │ movies │ DataFrame[Movie] │ │ num_movies │ int │ └────────────────────┴───────────────────────┘ Movie ┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Feature ┃ Type ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ id │ str │ │ director_id │ str │ │ release_number │ int │ │ type_ │ str │ │ title │ str │ │ original_title │ str │ │ release_date │ dt.datetime │ │ runtime_m │ float │ │ primary_genre │ str │ │ secondary_genre │ str │ └────────────────────────┴───────────────────┘ ``` We'll use the data schema above for the rest of this tutorial. Before we get started writing features, you'll want to import _ from Chalk. _ (underscore) allows you to build out features on your feature classes. ``` from chalk.features import _ ``` Additionally, if you want to create and work from a new branch (which is recommended), run the following command: ``` client.get_or_create_branch(branch_name="") ``` This command will: - create a new Chalk branch deployment that has the same features defined as your main Chalk deployment, - point your client to that new branch. If the branch already exists, your client will just be updated to point to that branch (no new branch deployment will be created). You can also point a client at a branch by passing branch= to the ChalkClient constructor. That form requires the branch to exist already, so prefer get_or_create_branch when you are not certain it does. ### Building New Features Let's say we want to write a new "exclaimed name" feature, which is a director's name with an exclamation mark tacked on. To do this, run the following code in a cell: ``` Director.name_exclaimed: str = _.name + "!" ``` In this example, the underscore acts as a reference to the Director feature class. We have now created a new feature in our notebook and can query for it in either an online or offline context. ### Querying for features To query for your new features, pass them as outputs to the ChalkClient query methods. For instance, to run an online query for your new Director.name_exclaimed feature, you can run the following: ``` new_features = client.query( input={ Director.id: 113, }, output={ Director.name, Director.name_exclaimed, } ) new_features ``` Running an offline query is equally straightforward: ``` chalk_dataset = client.offline_query( input={ Director.id: list(range(1000)), }, output={ Director.id, Director.name, Director.name_exclaimed, }, recompute_features=True, run_asynchronously=True, dataset_name="", ) df = chalk_dataset.to_pandas() ``` Running the offline query above creates a Pandas DataFrame with the columns Director.id, Director.name, and Director.name_exclaimed (alongside some metadata columns). This DataFrame will have one row for every Directors with an ID between 0 and 999. By providing a dataset_name to the offline_query, we've created a named dataset. Named datasets can easily be shared with other team members. Any historical dataset can be accessed using the ChalkClient. Named datasets can be accessed as follows: ``` client = ChalkClient() dataset = client.get_dataset(dataset_name="") ``` ### Building Out More Complex Features You can use expressions to build out more complex features. Generally these more complex features are composed of: has-one relationships, has-many relationships, and chalk functions. ### Using Has-one Relationships In the schema we've defined above, a Movie is in a has-one relationship with its Director (even if this is not necessarily true in practice). When building new features on the Movie class, features from joined classes can be directly accessed. For instance, let's say we want to define Movie.is_latest_release. We can accomplish this by defining a feature that reaches through the has_one join: ``` Movie.is_latest_release: bool = _.director.num_movies == _.release_number ``` The expression above defines a boolean feature which tests whether the release number for a movie (_.release_number) is equal to the number of movies a director has released (_.director.num_movies). ### Using Has-Many Relationships In the schema we've defined above, a Director is in a has-many relationship with their Movies. When building new features on the Director class, features from joined classes can be directly accessed. For instance, let's say we want to define Director.average_movie_runtime. We can accomplish this by defining an aggregation feature: ``` Director.average_movie_runtime: float = _.movies[_.runtime].mean() ``` In the above example, the two different _'s refer to two different feature class namespaces: - _.movies can be thought of as Director.movies, - _.runtime can be thought of as Movie.runtime. The expression above gets a directors movies (_.movies), specifies that the aggregation is happening on the runtime field of those movies (_.movies[_.runtime]) and then calculates the mean (.mean()). Filters can also be passed to these expressions. For instance, say we want to filter out any movie that has a primary or secondary genre of short from the average runtime. This can be done as follows: ``` Director.average_movie_runtime_filtered: float = _.movies[ _.runtime, # aggregation column _.primary_genre != "short", # filter _.secondary_genre != "short" # additional filter ].mean() ``` ### Using Chalk Functions Chalk exposes a number of functions that can be called and composed as part of expressions. The expressions docs list all of them. Below, we provide a few examples of how these functions can be used to create new features. Import chalk.functions once, and it's available to every cell in the session: ``` import chalk.functions as F ``` Identify whether a movie belongs to a Director's typical genres: ``` Movie.is_directors_usual_genre: bool = F.contains(_.director.genres, _.primary_genre) ``` Identify whether a movie was released on a US federal holiday: ``` Movie.released_on_holiday: bool = F.is_us_federal_holiday(_.release_date) ``` Get release_date of Director's last movie: ``` from datetime import datetime Director.latest_movie_release: datetime = F.max_by(_.movies[_.release_date], _.release_date) ``` ### Composing New Features The features you have defined in your notebook can also be composed into new features. For instance, you can define the Director.makes_movies_that_are_too_long feature: ``` Director.makes_movies_that_are_too_long: bool = _.average_movie_runtime > 150 ``` ### Translating this Code into Production Code Defining features as expressions has two advantages: - Features stay small and granular, so you can compose them into larger ones. - Everything written in this tutorial translates directly to your Chalk feature store, with no rewrite when you move it into your project. # Authentication for External Notebooks source: https://docs.chalk.ai/docs/notebook-setup ## Set up Chalk in the external notebook of your choice. In this section, we walk through authentication: how to connect to Chalk from an external notebook. To authenticate, we recommend generating a client ID and secret key pair. This can be done in the Chalk Dashboard. Go to Settings > Access Tokens, and generate a client ID/secret key pair by clicking on New Token and selecting all necessary permissions (typically, online query, offline query, and preview deployment permissions are sufficient). You can then use your access credentials to connect Chalk to any Jupyter notebook environment. In the following section, we show the process of authenticating to a local notebook. We then show the process for Hex, Deepnote, and Vertex. ### Local If you are working with a local notebook, Chalk will pick up your credentials generated from running chalk login in your terminal. All you need to do is initialize the client: ``` from chalk.client import ChalkClient client = ChalkClient() ``` ### Hex On Hex, you can store the client ID and secret key using secrets. Hex Secrets Select Python 3.11 as the image under Environment > Compute Profile. The Chalk client can then be initialized referencing the variables directly. ``` from chalk.client import ChalkClient client = ChalkClient( client_id=CHALK_CLIENT_ID, client_secret=CHALK_CLIENT_SECRET, branch='notebook' ) ``` ### Deepnote On Deepnote, you can set environment variables by going to the integrations tab and clicking on the "Environment Variables" button. Deepnote Environment Variables Set the CHALK_CLIENT_ID and CHALK_CLIENT_SECRET variables to the values you generated in the previous step. You can then initialize Chalk by running the following code in a cell: ``` import os from chalk.client import ChalkClient client = ChalkClient() ``` ### Vertex and Google Colab Enterprise On Google Colab Enterprise, you can set up secrets with the Google Secret Manager. Add two secrets named chalk-client-id and chalk-client-secret to Google Secret Manager that correspond to your Chalk client ID and client secret. Google Colab Secret Add three dependencies to your notebook environment: google-cloud-secret-manager, google-cloud-resource-manager, and chalkpy. You can now read your credentials securely from the Google Secret Manager and initialize your Chalk client! ``` # !pip install google-cloud-secret-manager # !pip install google-cloud-resource-manager # !pip install chalkpy import os from google.cloud import secretmanager as sm from google.cloud import resourcemanager_v3 as rm from chalk.client import ChalkClient # Create the Secret Manager client. PROJECT_NAME = os.environ['GOOGLE_CLOUD_PROJECT'] # These should match the names given to your secrets in Google Secret Manager client_id_name = "chalk-client-id" client_secret_name = "chalk-client-secret" # Get the project ID from google resources project_id = rm.ProjectsClient().get_project( request=rm.GetProjectRequest( name=f"projects/{PROJECT_NAME}" ) ).name # Get CHALK_CLIENT_ID and CHALK_CLIENT_SECRET using the secret manager secret_client = sm.SecretManagerServiceClient() CHALK_CLIENT_ID = secret_client.access_secret_version( name=f"{project_id}/secrets/{client_id_name}/versions/1" ).payload.data.decode("utf-8") CHALK_CLIENT_SECRET = secret_client.access_secret_version( name=f"{project_id}/secrets/{client_secret_name}/versions/1" ).payload.data.decode("utf-8") # Instantiate your Chalk Client chalk_client = ChalkClient( client_id=CHALK_CLIENT_ID, client_secret=CHALK_CLIENT_SECRET, branch="new-feature" ) ``` # Inline Syntax source: https://docs.chalk.ai/docs/notebook-inline ## Defining features inline in notebooks In notebooks, you can define features inline using the syntax shown below. Consider the following feature class definition for RocketEngine ``` from chalk.features import features @features class RocketEngine: id: int mass: float volume: float thrust: float ``` If we wanted to define a feature inline in a notebook cell, we could do so using the underscore syntax below: ``` RocketEngine.density: float = _.mass / _.volume ``` # Magics source: https://docs.chalk.ai/docs/notebook-magics ## Chalk cell magics for notebooks Chalk provides a set of IPython magics to improve the experience of defining resolvers in notebooks. ### Cell Magics ### %%sql_resolver You can define a SQL resolver in a notebook cell using the %%sql_resolver magic. The magic will parse the cell contents and upload your resolver to your current working branch. %%sql_resolver needs to be followed by a resolver name, e.g. root_authorization_resolver as shown below. Refer to the section on SQL Resolvers to learn more about how to define resolvers. ``` %%sql_resolver root_authorization_resolver -- resolves: Authorization -- source: snowflake SELECT id, amount_in_cents, card_id, merchant_id, created_at as authorized_at FROM authorizations ``` Chalk lines up the names of your target SQL columns with the names of your features. In this case, we have an Authorization feature class that contains a features called authorized_at. However, our Snowflake table has a column called created_at that we want to use to populate the authorized_at feature. So, we use the as keyword to rename the column in our resolver. # Sandbox source: https://docs.chalk.ai/docs/compute/sandbox ## What Chalk sandboxes are, where they run, and when to use one. ### Overview Chalk sandboxes are lightweight, isolated execution environments for running arbitrary code — agent workloads, model inference, data pipelines, or any container-based task. Each sandbox runs inside a gVisor-hardened container with its own filesystem, network namespace, and resource limits. Sandboxes are designed around three principles: - Strong isolation. Workloads from different tenants (and different workloads from the same tenant) cannot observe or interfere with each other. - Flexible deployment. The same sandbox abstraction runs on managed serverless infrastructure (compute_class=ComputeClass.HOST, the default) or on your own Kubernetes clusters (compute_class=ComputeClass.K8S). - First-class GPU support. GPU-accelerated workloads use the same Sandbox API as CPU workloads — see Resources and GPUs for the compute_class requirement. To create and run your first sandbox, see Quick Start. ### Deployment models ### Managed serverless Chalk operates a fleet of bare-metal nodes optimized for sandbox workloads. When you create a sandbox without additional configuration, it runs on this managed infrastructure: ``` from chalkcompute import Image, Sandbox sandbox = Sandbox(image=Image.debian_slim()).run() ``` Managed serverless handles provisioning, scaling, and node maintenance. Sandboxes are scheduled across availability zones and can cold-start in under two seconds for common base images. ### Features by compute class Sandboxes run on the host compute class unless you set compute_class=ComputeClass.K8S. Some features work on only one of them: | Feature | `host` | `k8s` | | ------------------------------------------------------------------------------ | ------ | ----- | | [Mounting volumes](/docs/sandboxes/configuring-sandboxes#mounting-volumes) | Yes | Yes | | [`exec`, `exec_stream`, and `exec_start`](/docs/sandboxes/running-commands) | Yes | No | | [Workload identity federation](/docs/workload-settings/workload-identity) | Yes | Yes | | [MCP Gateway](/docs/sandboxes/mcp-gateway) | Yes | Yes | | [`KernelPolicy.RESTRICTED`](/docs/sandboxes/isolation#kernel-hardening) | Yes | Yes | | [`KernelPolicy.OPEN`](/docs/sandboxes/isolation#kernel-hardening) | No | Yes | | [Network policies](/docs/sandboxes/networking#network-policy) | Yes | No | | [Resource detection](/docs/sandboxes/configuring-sandboxes#resource-detection) | Yes | No | | [Managed SSH](/docs/sandboxes/networking#managed-ssh) | Yes | No | | [Tunnels](/docs/tunnels) | Yes | No | ### When to use a Sandbox Use a Sandbox for a task you address directly as a single workload: running generated agent code, a one-off data backfill, or a per-session workspace that is torn down when the session ends. Sandboxes also run for hours or days, as Long-running sandboxes describes. For replicated, HTTP-fronted workloads with autoscaling, use a Scaling Group. For serverless, function-shaped invocations, use Functions. Run length alone does not decide between a sandbox and a scaling group, since both survive disconnects and can be reattached to by name. Choose by how the workload is reached: a sandbox is one instance you run commands in over exec, while a scaling group serves an HTTP endpoint backed by replicas that Chalk scales. A long-lived agent session that you drive through a web UI belongs in a scaling group, which is why Deep coding agents uses one. # Sandbox Quick Start source: https://docs.chalk.ai/docs/sandboxes/quick-start ## Build an image, run a script in a Chalk sandbox, and terminate the sandbox. Build an image, upload a local script, run it in a sandbox, and terminate the sandbox when the work is complete: ``` from chalkcompute import Image, Sandbox image = ( Image.debian_slim("3.12") .pip_install(["requests"]) .add_local_file("./script.py", "/app/script.py") .workdir("/app") ) sandbox = Sandbox( image=image, name="hello-sandbox", cpu="1", memory="2Gi", ).run() result = sandbox.exec("python", "/app/script.py") print(result.stdout_text) sandbox.terminate() ``` Sandbox(...).run() builds the Image, reuses a cached build when possible, creates the sandbox, and waits for it to become ready. Local files declared with add_local_file or add_local_dir are uploaded as part of the image setup. # Configuring Sandboxes source: https://docs.chalk.ai/docs/sandboxes/configuring-sandboxes ## Pass environment variables and secrets to a sandbox, mount volumes, and turn on resource detection. This page covers environment variables and secrets, volumes, and resource detection for sandboxes. Resources, workload identity, tags, and tracing apply to sandboxes, functions, and scaling groups, and each has its own page: - Resources and GPUs - Workload Identity - Tags - Tracing Policy ### Environment variables and secrets Pass plaintext configuration through env or resolve credentials through Chalk-managed Secret values: ``` from chalkcompute import Image, Sandbox, Secret sandbox = Sandbox( image=Image.debian_slim("3.12"), name="api-client", env={"LOG_LEVEL": "INFO"}, secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_integration("prod_postgres"), ], ).run() ``` Secrets are resolved at creation time and exposed to the sandbox as environment variables. See Secrets for the available Secret constructors. ### Mounting volumes Attach a Volume to share durable state between runs or expose a large dataset without baking it into the image: ``` from chalkcompute import Image, Sandbox, Volume vol = Volume("training-data") vol.put_file("inputs/example.txt", b"hello\n") sandbox = Sandbox( image=Image.debian_slim("3.12"), name="volume-reader", volumes=[("training-data", "/data")], ).run() result = sandbox.exec("cat", "/data/inputs/example.txt") print(result.stdout_text) # "hello\n" sandbox.terminate() ``` Mounted volumes persist across sandbox lifecycles and can be shared by other sandboxes that mount the same volume. ### Resource detection A sandbox can record resources it creates by inspecting its egress traffic. The GITHUB_PULL_REQUESTS detector inspects requests to api.github.com and records each pull request the sandbox opens, with the pull request URL as the resource ID. Detection is off by default, and works only on the host compute class. ``` import chalkcompute sandbox = chalkcompute.Sandbox( image=chalkcompute.Image.debian_slim(), resource_detectors=[chalkcompute.ResourceDetector.GITHUB_PULL_REQUESTS], ).run() ``` The SDK adds api.github.com to the sandbox's network policy. If the sandbox has no network_policy, this creates one that allows only that host. With the CLI, pass --resource-detection github-pull-requests to chalk sandbox create: ``` chalk sandbox create -i python:3.12 -n pr-bot \ --host api.github.com \ --resource-detection github-pull-requests ``` # Sandbox Lifecycle source: https://docs.chalk.ai/docs/sandboxes/lifecycle ## The stages a Chalk sandbox goes through, and how to inspect its status and terminate it. Sandboxes follow a straightforward lifecycle: - Image resolution. If the image is an Image spec (e.g. Image.debian_slim().pip_install([...])), it is built and cached. Pre-built OCI images are used directly. - Scheduling. The sandbox is placed on a node with sufficient resources. gVisor initializes the isolation boundary. - Running. The sandbox accepts exec calls. Volumes are mounted and accessible. - Termination. The sandbox is terminated explicitly or after its lifetime expires. Ephemeral filesystem state is discarded; volume data persists if synced. ``` from chalkcompute import Image, Sandbox sandbox = Sandbox( image=Image.debian_slim().pip_install(["numpy"]), cpu="2", memory="4Gi", lifetime="3600s", ).run() result = sandbox.exec("python", "-c", "import numpy; print(numpy.__version__)") print(result.stdout_text) sandbox.terminate() ``` lifetime is a protobuf duration string — a number of seconds followed by s (fractional seconds are allowed, e.g. "1.5s"), not a Go-style duration like "1h". Omitting it lets the sandbox run until it is terminated explicitly; see Long-running sandboxes for runs measured in hours or days. ### Inspecting status sandbox.info is None until the sandbox has been started or attached to. sandbox.refresh() both refreshes and returns the current SandboxInfo, so prefer its return value over sandbox.info when you've just refreshed: ``` info = sandbox.refresh() print(info.status) # "pending", "running", "succeeded", "failed", etc. print(sandbox.id) ``` ### Terminating ``` sandbox.terminate() # immediate termination + cleanup sandbox.terminate(grace_period_seconds=30) # allow processes to shut down first ``` Termination cleans up the sandbox and its ephemeral filesystem. Persistent volume data remains available according to the volume's sync and versioning semantics. # Running Commands in a Sandbox source: https://docs.chalk.ai/docs/sandboxes/running-commands ## Run processes inside a running Chalk sandbox with exec, exec_stream, and exec_start. Sandbox.exec(*command, timeout_secs=..., workdir=..., env=...) runs a process inside an already-running sandbox and returns its result after completion: ``` from chalkcompute import Image, Sandbox sandbox = Sandbox( image=Image.debian_slim("3.12").workdir("/app"), name="hello-sandbox", ).run() result = sandbox.exec("ls", "-la", "/app", timeout_secs=30) print(result.stdout_text) print(result.stderr_text) print(result.exit_code) sandbox.terminate() ``` The result captures stdout and stderr in full and includes the process exit code. Use sandbox.exec_stream(...) for streaming output or sandbox.exec_start(...) when you need interactive stdin and signal handling. exec, exec_stream, and exec_start require the host compute class. To run work in a k8s sandbox, pass the command as the sandbox's entrypoint when you create it. # Attaching and Sessions source: https://docs.chalk.ai/docs/sandboxes/attaching-and-sessions ## Reconnect to a running Chalk sandbox from another process, and run work that outlasts a single session. ### Attaching to an existing sandbox Use Sandbox.from_id or Sandbox.from_name to reconnect to a sandbox from another process, instead of creating a second one: ``` from chalkcompute import Sandbox sandbox = Sandbox.from_id("sandbox-id") # or, if it was created with a name=: sandbox = Sandbox.from_name("hello-sandbox") ``` Sandbox.list_all() returns SandboxInfo for every sandbox in the current environment, useful for finding an id or name to attach to. ### Running work across sessions These examples use the multi-day-training sandbox from Long-running sandboxes. Sandbox.exec blocks until the command finishes, so start work measured in hours or days as a detached process: ``` from chalkcompute import Sandbox sandbox = Sandbox.from_name("multi-day-training") sandbox.exec( "sh", "-c", "nohup python /app/train.py --checkpoint-dir /checkpoints > /checkpoints/train.log 2>&1 &", ) ``` Reattach by name from another process, or on another day, to check on it: ``` from chalkcompute import Sandbox sandbox = Sandbox.from_name("multi-day-training") print(sandbox.refresh().status) print(sandbox.exec("tail", "-n", "20", "/checkpoints/train.log").stdout_text) ``` Within a single exec call, auto_reattach (enabled by default) reconnects and replays buffered output after a transient transport disconnect, so brief network interruptions need no handling of their own. Terminate the sandbox once the work is done. A sandbox reserves the CPU, memory, and any GPU you requested for as long as it exists, and those resources are billed as provisioned rather than as used, so an idle sandbox costs the same as a busy one. RestartPolicy.ALWAYS keeps bringing it back until you terminate it: ``` sandbox.terminate() ``` # Tunnels source: https://docs.chalk.ai/docs/tunnels ## Reach a TCP service inside a Chalk sandbox from your machine, with ssh, scp, port forwarding, and remote IDEs. ### Overview A sandbox is a long-running container you create with chalk sandbox create or with the Sandbox class in chalkcompute. It accepts no inbound connections. To reach a server running in one, use chalk session proxy, which connects a TCP port inside the sandbox to its own standard input and output over an authenticated stream to the Chalk API. Set it as your ssh ProxyCommand to open a shell in a sandbox, copy files, forward ports, or attach a remote IDE. Reaching a sandbox takes both a Chalk token and an SSH key. Chalk authenticates the stream, and the SSH connection inside it is encrypted end to end between your machine and the sandbox's sshd. ### Requirements You need Chalk CLI v1.54.6 or later. To update, run chalk update. The sandbox must meet four requirements. Host-backed. Sandboxes run on the host compute class by default. Sandboxes on the k8s compute class, including GPU sandboxes, are not supported. An image with sshd installed. Install the SSH server when you build the image, and start it with --entrypoint when you create the sandbox. No files in /root/.ssh. Chalk rejects an image that has files in /root/.ssh, so keep the SSH server's authorized keys somewhere else, as the example below does. A shell and a connector. The image needs sh and one of socat, nc, or bash. If it has none of the three, add socat. By default, a host-backed sandbox has no network access in either direction, so install everything the tunnel depends on (the SSH server, the connector, and a shell) when you build the image. ### Building an image with sshd Install the SSH server and your public key when you build the image. This Debian example installs openssh-server, socat, and rsync, makes sure host keys exist, stores your public key in /etc/ssh/authorized_keys/root, and binds sshd to loopback. The same steps apply to other base images. ``` FROM debian:trixie-slim RUN apt-get update \ && apt-get install -y --no-install-recommends openssh-server socat rsync ca-certificates \ && rm -rf /var/lib/apt/lists/* # sshd needs a host key to present. -A generates any host key type that is missing # and leaves existing keys in place. RUN ssh-keygen -A # Chalk rejects images with files in /root/.ssh, so authorized keys live under # /etc/ssh. Only the tunnel can reach sshd, so it listens on loopback only. RUN mkdir -p /run/sshd /etc/ssh/authorized_keys /workspace \ && printf '%s\n' \ 'ListenAddress 127.0.0.1' \ 'PermitRootLogin prohibit-password' \ 'PasswordAuthentication no' \ 'AuthorizedKeysFile /etc/ssh/authorized_keys/%u' \ > /etc/ssh/sshd_config.d/chalk.conf COPY id_ed25519.pub /etc/ssh/authorized_keys/root RUN chmod 644 /etc/ssh/authorized_keys/root CMD ["/usr/sbin/sshd", "-D", "-e"] ``` Push the image to a registry your Chalk environment can pull from, or build the same Dockerfile through Chalk with Image.from_dockerfile (see Images). Then create a sandbox from it. Pass --entrypoint to start sshd, because a sandbox otherwise keeps itself running with its own process and the image's CMD does not run. Pass --lifetime too, because a sandbox's lifetime defaults to forever: ``` chalk sandbox create \ --image my-registry/dev-box:latest \ --name my-box \ --entrypoint "/usr/sbin/sshd -D -e" \ --lifetime 8h ``` Generate a key pair for development sandboxes only, and use its public key when you build the image. Every copy of the image carries that public key, so to revoke access you generate a new pair and rebuild the image. ### Configuring ssh Run chalk sandbox configure-ssh to add a block to ~/.ssh/config that routes hosts named chalk-sbx- through chalk session proxy. Pass --dir with the directory your Chalk credentials belong to: the directory where you ran chalk login, or your Chalk project. The CLI chooses credentials by directory, so without --dir the tunnel works only when you start ssh from that directory. To see the block without writing it, add --dry-run: ``` chalk sandbox configure-ssh --dir ~ --dry-run ``` ``` # >>> chalk sandbox configure-ssh >>> # Managed by 'chalk sandbox configure-ssh'. Edits between these markers are # overwritten on the next run; delete the markers too to stop managing it. # # 'ssh chalk-sbx-' tunnels to a Chalk sandbox over the Chalk API. # The sandbox must be host-backed ('--compute-class host') and running an sshd. Host chalk-sbx-* User root ProxyCommand sh -c 'exec "/Users/you/.chalk/bin/chalk-latest-1" session proxy --dir "/Users/you" "${0#chalk-sbx-}"' %n # Sandboxes are ephemeral and regenerate their host keys with them, so # recording those keys would only produce false mismatch warnings. The # connection is authenticated by Chalk before ssh ever sees it. UserKnownHostsFile /dev/null StrictHostKeyChecking no LogLevel ERROR # <<< chalk sandbox configure-ssh <<< ``` To write the block, run the command without --dry-run: ``` chalk sandbox configure-ssh --dir ~ ``` The block points at the specific chalk binary installed when you ran the command, so re-run the command after chalk update, after you move the chalk binary, or when you switch projects. It rewrites its block in place, between the markers. It also keeps the block at the top of the file: ssh uses the first value it finds for each option, so a Host * block above it would override its settings. Connect to a sandbox with the chalk-sbx- prefix followed by the sandbox's name or ID: ``` ssh chalk-sbx-my-box ssh chalk-sbx-54b4aa4c-6adf-46d0-a3ed-a92a24f19052 ``` The prefix keeps the Host pattern from matching real hosts, and the sh wrapper removes it before the name reaches chalk session proxy. If your key is not one ssh offers by default, such as a key you generated for development sandboxes, add an IdentityFile in a separate block after the markers. configure-ssh overwrites everything between them: ``` Host chalk-sbx-* IdentityFile ~/.ssh/chalk_sandbox_ed25519 ``` To change the host prefix, the login user, or the sshd port, use --prefix (default chalk-sbx-), --user (default root), and --port (default 22). To write to a file other than ~/.ssh/config, use --ssh-config: ``` chalk sandbox configure-ssh --prefix box- --user dev --port 2222 ``` ### Using the connection With ssh configured, you can use any tool that connects over ssh. To copy files: ``` scp ./train.py chalk-sbx-my-box:/workspace/ rsync -av ./src/ chalk-sbx-my-box:/workspace/src/ ``` To reach a server in the sandbox from your machine, such as a notebook, a profiler UI, or a debugger, forward its port: ``` ssh -N -L 8888:127.0.0.1:8888 chalk-sbx-my-box ``` Editors that read ~/.ssh/config, such as VS Code Remote-SSH and JetBrains remote clients, can connect to chalk-sbx-my-box too. In VS Code, run Remote-SSH: Connect to Host and type the host name. Do not use Remote-SSH: Add New SSH Host: it rewrites the configure-ssh block and breaks its quoting. If that happens, run chalk sandbox configure-ssh --dir ~ again. Interactive latency is the network round trip to your Chalk region, plus about a millisecond for the tunnel. ### Tunnelling other services chalk session proxy carries any TCP connection. Use --port to reach a database, a debugger, or an inference server in the sandbox the same way. To connect to an address other than 127.0.0.1 inside the sandbox, use --target-host. For example, to connect to Postgres running in the sandbox on port 5432: ``` chalk session proxy my-box --port 5432 ``` This connects the service to the command's standard input and output. To use a client such as psql, run a local listener that starts the proxy for each connection. With socat on your machine: ``` socat TCP-LISTEN:15432,reuseaddr,fork \ EXEC:'chalk session proxy my-box --port 5432' ``` ``` psql -h 127.0.0.1 -p 15432 -U postgres ``` Run these commands from the directory where you ran chalk login, or pass --dir to chalk session proxy, because the CLI chooses your credentials by directory. Each connection to the listener starts its own proxy and its own connector process in the sandbox, and both exit when the connection closes. If the sandbox runs an sshd, forward the ports with ssh instead. A single ssh process carries every forwarded port over one tunnel, and your machine needs nothing except ssh: ``` ssh -N -L 15432:127.0.0.1:5432 -L 8888:127.0.0.1:8888 chalk-sbx-my-box ``` ### How it works When you set a ProxyCommand, ssh runs that command and sends the SSH protocol over its standard input and output instead of opening a TCP connection. chalk session proxy provides that transport: - It resolves the sandbox and opens an authenticated, bidirectional stream to the Chalk API over an outbound HTTPS connection from your machine. - Chalk starts a connector process in the sandbox (socat, nc, or bash), attached to that stream. The connector opens a TCP connection to 127.0.0.1:22, or to the address and port set with --target-host and --port. - Bytes flow in both directions between ssh and the sandbox's sshd. Because ssh sees an ordinary byte stream, authentication, channels, port forwarding, and SFTP all work unchanged. chalk session proxy writes nothing to standard output except the proxied bytes, because standard output is the connection. When the local side closes, the proxy kills the connector in the sandbox, so a closed connection leaves no process running. A process started with chalk session exec keeps running after its stream ends. ### Troubleshooting The CLI reports chalk session proxy or chalk sandbox configure-ssh as an unknown command: your CLI is older than v1.54.6. Run chalk update. sandbox image … contains non-empty /root/.ssh; configure all SSH access through the managed SSH API: the image has files in /root/.ssh. Store authorized keys under /etc/ssh/authorized_keys and point AuthorizedKeysFile there, as the example image does. Managed SSH covers connections from a sandbox to other hosts, so it does not replace an sshd for connecting in. ssh fails with no valid token found and client credentials are incomplete: the proxy ran from a directory with no Chalk credentials. Run chalk sandbox configure-ssh --dir with the directory where you ran chalk login. container sessions require a host-backed container: the sandbox was created with --compute-class k8s. You cannot change the compute class of an existing sandbox, so create a new one without that flag. chalk session proxy: no socat, nc, or bash in the container: the image has no connector. Add socat to the image. Connection closed by UNKNOWN port 65535, after socat reports Connection refused and connector exited with code 1: the proxy reached the sandbox, but no server answered on the port. Check that sshd is running. The list of process names should include sshd: ``` chalk session exec my-box sh -- -c 'cat /proc/[0-9]*/comm' ``` If it does not, check the sandbox's entrypoint with chalk sandbox get --name my-box. An entrypoint of sleep infinity means the sandbox was created without --entrypoint. User not allowed because account is locked: sshd rejects key authentication for an account whose password field in /etc/shadow starts with !, when sshd runs without PAM. Alpine's sshd is built without PAM, and adduser -D creates accounts in this state. In the image, set the account's password field to *, which allows key login and keeps password login disabled: run usermod -p '*' on Debian or Ubuntu, or echo ':*' | chpasswd -e on Alpine. Do not use passwd -u: on Alpine it empties the password field, which leaves the account with no password. Authentication fails with the right key: run ssh -v to see which key ssh offered, and check that /etc/ssh/authorized_keys/ in the image contains that key. sshd ignores the file if a group or other users can write to it. ### Related - Sandbox: creating sandboxes, and the isolation model they run under. - Images: building the image that carries your sshd. - Volumes: persistent storage, for work that should outlive a sandbox. - CLI reference: chalk session proxy, chalk session exec, and chalk sandbox configure-ssh. # Sandbox Networking source: https://docs.chalk.ai/docs/sandboxes/networking ## Restrict a sandbox's outbound network access, give it managed SSH keys, and connect it to other workloads. ### Network policy A sandbox on the default host compute class allows outbound network access by default. To restrict access, pass a NetworkPolicy as the sandbox's network_policy (singular). The policy lists the destinations the sandbox can reach: ``` from chalkcompute import Image, NetworkPolicy, Sandbox policy = NetworkPolicy( allowed_hosts=[ "api.openai.com", "github.com", "pypi.org", ], ) sandbox = Sandbox( image=Image.debian_slim(), network_policy=policy, ).run() ``` Each entry in allowed_hosts is an exact hostname, a wildcard subdomain pattern, or *: - An exact hostname, such as github.com, matches only that host. - *.github.com matches every subdomain of github.com, such as api.github.com, but not github.com itself. To allow both, list both. - * matches every hostname. Matching ignores case. Requests to any other destination are blocked. A policy with an empty allowed_hosts allows no hostnames. NetworkPolicy accepts one policy per sandbox — network_policy takes a single NetworkPolicy, not a list. Network policies apply only to the host compute class. A sandbox on k8s has the network access of the cluster it runs on. ### CIDR-level routes Use allowed_routes and denied_routes for destinations that don't have a stable hostname — a VPC-peered range or an on-premise network, for example. denied_routes takes precedence over both allowed_routes and allowed_hosts: ``` from chalkcompute import NetworkPolicy policy = NetworkPolicy( allowed_routes=[ NetworkPolicy.Route("10.0.0.0/8", ports=[(5432, 5432)]), ], denied_routes=["169.254.169.254/32"], # block the cloud metadata endpoint ) ``` Route(route, *, ports=...) takes a destination CIDR and an optional list of (start_port, end_port) ranges; an empty ports allows every port on that route. ### Per-host request rules For per-host request matching or rewriting — for example, substituting a Secret into an auth header before forwarding to an internal service — pass allowed_hosts as a mapping of hostname to a list of NetworkPolicy.Rule instead of a plain list: ``` from chalkcompute import NetworkPolicy policy = NetworkPolicy( allowed_hosts={ "internal-api.example.com": [ NetworkPolicy.Rule( match=NetworkPolicy.Match( path=NetworkPolicy.Matcher(starts_with="/v1/"), ), transforms=[ NetworkPolicy.Transformer( headers={ "Authorization": "Bearer __chalk_secret_API_TOKEN", }, headers_secrets={ "__chalk_secret_API_TOKEN": "INTERNAL_API_TOKEN", }, ), ], forward_url="https://internal-api.example.com", ), ], }, ) ``` Each Rule can match on request path (exact, starts_with, or regex), HTTP method, query-string entries, and headers — a rule's transforms and forward_url only apply to requests matching every dimension set on match. The route matches the original request before any transforms run. Transformer.headers_secrets maps each literal placeholder token to the name of a Chalk Secret. For a matching rule, Chalk first applies the literal values in Transformer.headers, then replaces each placeholder occurrence in ordinary request-header values with that Secret's contents. Placeholders can also appear in headers supplied by the workload. Chalk does not substitute within pseudo-headers, authority or framing headers, hop-by-hop headers, or Chalk-owned routing headers. Placeholder matching is case-sensitive. When occurrences overlap, Chalk selects the leftmost match and then the longest placeholder that starts at that position. Replacement is non-recursive, so inserted Secret contents are not scanned for additional placeholders. An absent placeholder has no effect, and an empty Secret value removes the matched placeholder text. Empty placeholder tokens are rejected. The sandbox never receives the Secret contents. ### Managed SSH Managed SSH lets a sandbox authenticate to approved SSH destinations using a Chalk Secret, without exposing the private key to the sandbox itself. It requires the host compute class. First, store the private key: ``` chalk secret set GITHUB_DEPLOY_KEY < /path/to/privkey/id_ed25519 ``` Then configure the destination when creating the sandbox: ``` from chalkcompute import Image, SandboxClient, SshDest sandbox_client = SandboxClient.from_env() sandbox = sandbox_client.create( image=Image.debian_slim(), managed_ssh={ "github": SshDest( "github.com", "GITHUB_DEPLOY_KEY", username="git", ) }, ) ``` private_key_secret is the name of a Chalk Secret whose value is an SSH private key. The map key creates the SSH alias used to connect, so the example supports ssh github and git clone github:example/repo.git. The default port is 22, but can be overridden by setting an explicit port. By default, SSH accepts the server's host key on the first connection and remembers it for the rest of the sandbox's lifetime. Later connections fail if the server presents a different key. Set host_keys to override this, and to require specific host keys from the first connection. For example: ``` managed_ssh={ "a-server": SshDest( "cloud.example.com", "KEY_FOR_SANDBOX_USE", username="my_user", host_keys=[ "ssh-ed25519 AAAA...", "ecdsa-sha2-nistp256 AAAA...", ], port=2222, ) } ``` Managed SSH destinations are a separate security domain from Network Policy. A configured destination's resolved IP address and port remain reachable even if they overlap with denied_routes. Internally, Chalk loads the configured keys into an ssh-agent and exposes its socket through both SSH_AUTH_SOCK and IdentityAgent in /root/.ssh/config. Sandbox processes can ask the agent to sign with any configured key, but cannot read the private-key material. Sandbox images containing any existing /root/.ssh are rejected, whether or not Managed SSH is configured for that sandbox. ### Sandbox-to-sandbox and on-premise connectivity Each sandbox gets its own network namespace with no shared listening sockets (see Multi-tenancy). Chalk provides no direct connection between sandboxes and no tunnel into an on-premise network. To exchange data between workloads, share it through a mounted Volume or an external service that both can reach over the network. To reach a private network that your cloud network already connects to, such as a VPC-peered range, use CIDR-level routes. To connect to a sandbox from your own machine, for example with ssh or port forwarding, use Tunnels. # Host Pools source: https://docs.chalk.ai/docs/compute/host-pools ## Configure the hosts that back Chalk sandboxes, and size them so sandbox workloads can be scheduled. ### Overview A host pool is a group of hosts that Chalk sandboxes run on. Each host is sized with a fixed amount of CPU and memory, and Chalk packs sandbox containers onto the available hosts. Host pools back workloads that use the host compute class, which is the default for Sandboxes, along with interactive sessions and outbound managed SSH connections from a sandbox. Scaling Groups, Functions, model deployments, and notebook kernels run on Kubernetes directly and do not use host pools. ### Where host pools are configured Host pools can be configured at two scopes, both from the Chalk dashboard: - Cluster: the pool serves every environment in the cluster. This is the recommended scope, because one pool can be shared cluster-wide. To edit these pools, click Clusters in the team navigation, select the cluster to open its Settings tab, and find Compute Host Pools. - Environment: the pool serves a single environment, and cannot be shared with other environments. To edit these pools, open the environment's Settings, then Compute, and find Host Pools. For each sandbox, Chalk uses cluster pools when at least one matches its GPU requirement: a CPU-only sandbox matches a CPU-only pool, and a GPU sandbox matches a pool with the requested GPU type and count. Otherwise, Chalk uses the environment's pools. Once Chalk selects a scope, it does not fall back to the other scope when a matching pool has no available capacity. Sandbox isolation is enforced the same way regardless of scope, so workloads from different environments can share a cluster pool safely. Editing host pools requires the deploy.create permission, held by the Admin, Owner, and Team Manager roles. Every change is recorded in the audit log. Host pools are not available in serverless environments. ### The default host pool Until you configure a pool of your own, Chalk manages an implicit pool for you. It is created the first time a host-class workload runs in an environment, and it appears in the dashboard marked System managed with its fields locked: | Setting | Default | | --------------- | -------- | | Minimum hosts | 0 | | Maximum hosts | 2 | | CPU per host | 3 | | Memory per host | 12Gi | | Idle timeout | 1 minute | A sandbox requesting more than 3 CPU or more than 12Gi of memory does not fit on these hosts and fails to schedule. To run larger sandboxes, configure a pool. Creating your own host pool removes the system-managed default pool. Your pools then serve every host-class workload in scope, so size them to cover your smallest and largest sandboxes, not only the large one you added the pool for. Deleting your last pool restores the default. ### Configuring a host pool Both scopes use the same editor, which lists one row per pool. Click the add (+) button to create a pool, fill in its fields, and click Save changes. To delete a pool, click the remove icon on its row and save. The system-managed default pool appears with a System managed badge, and its fields cannot be edited. Each pool has the following fields: | Field | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | A unique name within the scope. Use 3 to 63 characters: lowercase letters, numbers, and hyphens, starting with a letter. | | **Minimum hosts** | Set to `0` for on-demand capacity, or to the same value as **Maximum hosts** for fixed capacity. Values in between are rejected. | | **Maximum hosts** | How many hosts the pool runs when it is active. Must be at least 1. | | **Idle timeout** | How long the pool sits empty before scaling back down, in minutes. Required for on-demand capacity, minimum 1 minute, and unavailable for fixed capacity. | | **CPU per host** | Cores per host, written as a core count such as `2`, `1.5`, or `500m`. Minimum `100m`. | | **Memory per host** | Memory per host, written as a byte quantity such as `512Mi`, `12Gi`, or `1T`. Minimum `512Mi`, maximum `1T`. | CPU and memory are set per host, not for the pool as a whole. A pool of 2 hosts at 8 CPU each can run two sandboxes of 8 CPU, not one sandbox of 16. Chalk selects host hardware that satisfies the CPU and memory you set, so there is no machine type to choose. On AWS and GKE, GPU host pools are available through the HostPoolService API. Set gpu to a supported :1 value, such as nvidia-l4:1. GKE host pools support T4, P4, V100, A100 (40 GB and 80 GB), L4, RTX PRO 6000, H100 (High and Mega), H200, and B200. Your GKE cluster needs GPU autoscaling limits, quota, and capacity for the selected type. GB200 and GB300 are not available through GPU host pools. The dashboard editor does not expose this field yet. GPU sandboxes use a pool with the matching GPU type; CPU-only sandboxes use CPU-only pools. ### On-demand and fixed capacity Setting Minimum hosts to 0 gives you on-demand capacity. The pool runs no hosts while it is empty, starts all of its hosts the first time a sandbox needs one, and returns to zero once it has been empty for the idle timeout. The idle timeout is measured from the moment the last sandbox leaves the pool, not from host CPU idleness. Setting Minimum hosts equal to Maximum hosts gives you fixed capacity. The hosts stay running, so sandboxes do not wait for a host to start. On-demand pools scale as a unit: they run either zero hosts or the full Maximum hosts count. ### Troubleshooting A sandbox that fails to be placed on a host reports one of three errors, depending on the cause. ### A sandbox fails with "no configured host pool can satisfy" ``` Unschedulable: no configured host pool can satisfy cpu=8000m, memory=8589934592 bytes ``` No pool in scope is large enough to hold this sandbox. Chalk compares the request against the CPU and memory of a single host in each pool, so a request larger than any one host fails even when the pool's hosts add up to more than the request. The values in the message are the sandbox's request: 8000m is 8 cores, and memory is in bytes, so 8589934592 is 8Gi. To resolve it: - Raise CPU per host or Memory per host on an existing pool so a host can hold the sandbox. - Add a pool sized for the larger workload. Chalk places each sandbox on the smallest pool that fits it, so adding a large pool alongside a small one does not waste the small one. - Reduce the sandbox's cpu or memory request. Size hosts somewhat larger than the biggest sandbox you intend to run. A host reserves part of its memory for its own overhead, so a sandbox requesting exactly the host's memory may still fail to be placed. ### A sandbox fails with "all registered host-pool capacity is allocated" ``` Unschedulable: all registered host-pool capacity is allocated ``` The pools are sized correctly, but their hosts are full. Raise Maximum hosts, or wait for running sandboxes to finish. ### A sandbox reports "no host with sufficient capacity" ``` Unschedulable for at least 600s: no host with sufficient capacity ``` Chalk waited for a host to become available and none did. This usually means the pool is still waiting on the cloud provider for a host. If it persists, check whether the pool's CPU per host and Memory per host are larger than the hardware available in your region. # Long-Running Sandboxes source: https://docs.chalk.ai/docs/sandboxes/long-running-sandboxes ## Run a Chalk sandbox for hours or days with a lifetime and a restart policy. Sandboxes can run for hours or days. Two independent settings govern how long one lasts: lifetime bounds its total run time, and restart_policy decides whether Chalk recreates it when its backing instance goes away. ### Setting a lifetime Omit lifetime to run until you terminate the sandbox yourself. Set it to bound the run: | Value | Effect | | ----------- | -------------------------------------------------- | | omitted | The sandbox runs until it is terminated explicitly | | `"86400s"` | The sandbox is terminated 24 hours after it starts | | `"604800s"` | The sandbox is terminated 7 days after it starts | There is no maximum value. lifetime="0s" raises ValueError rather than meaning "no limit"; omit the argument for that. When a lifetime expires, Chalk stops the sandbox immediately without draining in-flight work, and records the result as succeeded. Write anything you need to keep to a mounted volume before that point. ### Surviving node drains and preemption A sandbox running for days outlives the node it started on. By default, the backing instance terminating ends the sandbox. Set restart_policy to have Chalk recreate it instead: ``` from chalkcompute import Image, RestartPolicy, Sandbox, Volume Volume("training-checkpoints") # created if it does not already exist sandbox = Sandbox( image=Image.debian_slim("3.12").pip_install(["torch"]), name="multi-day-training", cpu="8", memory="32Gi", volumes=[("training-checkpoints", "/checkpoints")], restart_policy=RestartPolicy.ALWAYS, ).run() ``` Leaving restart_policy unset behaves the same as RestartPolicy.NEVER. A recreated sandbox starts from its image again with an empty ephemeral filesystem, so write progress to a mounted Volume and have the workload resume from it on startup. To start work that runs for hours and check on it later, see Running work across sessions. # Sandbox Isolation source: https://docs.chalk.ai/docs/sandboxes/isolation ## How Chalk isolates sandboxes with gVisor, separates tenants, and restricts kernel access. These controls also apply to higher-level abstractions like Scaling Groups, which share the same isolation primitives. ### Isolation with gVisor Every sandbox runs under gVisor, a container runtime that intercepts application system calls through a user-space kernel. Unlike traditional containers that share the host kernel directly, gVisor interposes a second layer of defense: ``` ┌────────────────────────┐ │ Application process │ ├────────────────────────┤ │ gVisor (Sentry) │ ← intercepts syscalls ├────────────────────────┤ │ Host kernel │ └────────────────────────┘ ``` This means a kernel exploit in one sandbox cannot compromise the host or other sandboxes. gVisor also restricts the set of available syscalls, reducing the attack surface exposed to untrusted code — particularly important for agent workloads that execute LLM-generated commands. Each sandbox additionally receives its own: - Filesystem namespace. The root filesystem is ephemeral and private. Persistent storage is available through Volumes. - Network namespace. Sandboxes have isolated network stacks. Outbound access is allowed by default; attach a network policy to restrict destinations. - PID namespace. Processes inside a sandbox cannot see or signal processes in other sandboxes. ### Multi-tenancy Chalk enforces tenant isolation at every layer of the stack: | Layer | Mechanism | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Runtime** | gVisor kernel-level syscall interception per sandbox | | **Network** | Separate network namespace per sandbox; no shared listening sockets | | **Storage** | Volumes are scoped to the owning environment; cross-tenant access is impossible | | **Scheduling** | Workloads from different tenants are placed on separate host nodes by default | | **Identity** | A sandbox created with `chalk_identity=True` receives its own [workload identity](/docs/workload-settings/workload-identity); no two sandboxes share credentials | For deployments with strict regulatory requirements, dedicated node pools can be configured so that a tenant's workloads never share physical hardware with any other tenant. ### Kernel hardening Sandboxes restrict the syscalls and host access available to the workload by default. Control this with security_policy: ``` from chalkcompute import ContainerSecurityPolicy, Image, KernelPolicy, Sandbox sandbox = Sandbox( image=Image.debian_slim(), security_policy=ContainerSecurityPolicy(kernel_policy=KernelPolicy.RESTRICTED), ).run() ``` kernel_policy is KernelPolicy.RESTRICTED (the default, both when security_policy is omitted and when set explicitly) or KernelPolicy.OPEN, which relaxes the sandboxing controls RESTRICTED enables. Use OPEN only for workloads that need syscalls or host access that RESTRICTED blocks. OPEN works only on the k8s compute class. # MCP Server Access source: https://docs.chalk.ai/docs/sandboxes/mcp-gateway ## Let sandboxes call your organization's MCP servers through the MCP Gateway without seeing the upstream credentials. The MCP Gateway lets sandboxes interact with Model Context Protocol servers exposed at your enterprise — without giving the sandbox direct access to the underlying credentials. A sandbox created with chalk_identity=True authenticates to the MCP Gateway with its workload identity, with no other credential. A sandbox created without it must send a Chalk access token in the Authorization header and the environment ID in the X-Chalk-Env-Id header. The gateway validates the identity, then proxies the request to the upstream MCP server using credentials managed by your organization. The sandbox never sees the real credential. ``` ┌─────────────┐ WIF token ┌─────────────┐ real credential ┌─────────────┐ │ Sandbox │ ──────────────────────▸ │ MCP Gateway │ ──────────────────────▸ │ MCP Server │ └─────────────┘ └─────────────┘ └─────────────┘ ``` This is particularly useful for agent workloads. A code-generation agent may need to call tool-use APIs, search indexes, or retrieval services. With the gateway: - The agent can leak its own temporary token without compromising the real upstream credential. - You retain centralized control over which MCP servers are reachable and by which workloads. - Audit logs capture every proxied call, tied to the originating workload identity. # Claude Code in a repository sandbox source: https://docs.chalk.ai/docs/sandboxes/coding-agent-sandbox ## Start Claude Code in a fresh repository checkout, detach, and reconnect to the same session with tmux. Give each coding session a fresh checkout with Claude Code, GitHub CLI, and your Git identity. Start it from a reusable Chalk function, then return to the same work through tmux. The provisioner below prepares a sandbox with 2 CPUs and 16 GiB of memory and clones your repository. The launch command opens Claude in the checkout. ### Define the provisioner Save this as coding_agent_sandbox.py. Replace your-org/your-repo, Your Name, and you@example.com with your repository and Git author identity. ``` # /// script # dependencies = ["chalkcompute==2.13.4"] # /// import base64 import os import uuid import chalkcompute @chalkcompute.function(secrets=[chalkcompute.Secret.from_chalk_env("GITHUB_TOKEN")], chalk_identity=True) def coding_agent_sandbox(memory_gib: int = 16) -> chalkcompute.Sandbox: """Create a repository checkout with Claude Code installed. Args: memory_gib: Sandbox memory in GiB. """ sandbox = ( chalkcompute.Sandbox( name="coding-agent-" + str(uuid.uuid4()), image=chalkcompute.Image.sandbox() .with_claude_code( "2.1.270", trusted_dirs=["/workspace/repo"], permission_mode="bypassPermissions", ) .with_gh() .with_tmux() .workdir("/workspace/repo"), cpu="2", memory=f"{memory_gib}Gi", network_policy=chalkcompute.NetworkPolicy( allowed_hosts={ "*": [], "github.com": [ chalkcompute.NetworkPolicy.Rule( transforms=[ chalkcompute.NetworkPolicy.Transformer( headers={ "Authorization": "Basic " + base64.b64encode( f"x-access-token:{os.environ['GITHUB_TOKEN']}".encode() ).decode() } ) ] ) ], } ), chalk_identity=True, env=chalkcompute.GitConfig(name="Your Name", email="you@example.com"), tags={ "provisioner": "coding_agent_sandbox", "repository": "your-org/your-repo", }, compute_class=chalkcompute.ComputeClass.HOST, resource_detectors=[chalkcompute.ResourceDetector.GITHUB_PULL_REQUESTS], ) .add_credentials( ["api.github.com", "uploads.github.com"], chalkcompute.Secret.from_chalk_env("GITHUB_TOKEN", alias="GH_TOKEN"), ) .add_credentials( "api.anthropic.com", chalkcompute.Secret.from_chalk_env( "CLAUDE_CODE_OAUTH_TOKEN" ), ) .run() ) try: sandbox.git.clone( url="https://github.com/your-org/your-repo.git", path="/workspace/repo", branch="main", ) except Exception: sandbox.terminate() raise return sandbox if __name__ == "__main__": coding_agent_sandbox.deploy() print("Deployed coding_agent_sandbox") ``` ### Set up once Log in to the Chalk environment where you want to run your sessions, save the credentials as Chalk secrets, and deploy the provisioner: ``` chalk login chalk secret set GITHUB_TOKEN claude setup-token chalk secret set CLAUDE_CODE_OAUTH_TOKEN uv run --script coding_agent_sandbox.py ``` The secret commands prompt for their values. Use a GitHub token with access to your repository, and paste the token from claude setup-token into the Claude secret prompt. The token-setup command requires a Claude subscription and a local Claude Code installation. ### Start Claude Launch the deployed function from your terminal: ``` chalk sandbox start coding_agent_sandbox --tmux -- claude ``` Each launch creates a fresh checkout. The provisioner installs the development tools and configures Git identity and Claude authentication before your session starts. To request more memory, pass the function's argument: ``` chalk sandbox start coding_agent_sandbox --arg memory_gib=32 --tmux -- claude ``` ### Detach and reconnect Press Ctrl-B, then D to detach. Your sandbox, checkout, and Claude session keep running. To return, select the same sandbox and tmux session: ``` chalk sandbox attach ``` Reattaching continues the existing session. It does not create another sandbox or clone your repository again. ### Finish the session Push or copy any changes you want to keep, then release the sandbox: ``` chalk sandbox terminate --id ``` Detaching leaves the sandbox running and billed for its provisioned resources. Terminating removes its ephemeral checkout. Mount a Chalk Volume for files you need across sandbox lifetimes. # SQLite on a Shared Volume source: https://docs.chalk.ai/docs/compute/sqlite-persistence ## Write a SQLite database in one sandbox and read it from others using a shared volume. ### What we're building Three sandboxes sharing a single SQLite database through a Chalk volume: - A writer sandbox creates the database, inserts rows, and terminates. - Two reader sandboxes start afterward, mount the same volume, and independently query the data. This proves that data written to a volume outlives the sandbox that created it — and that multiple sandboxes can read it concurrently. The pattern is useful for experiment tracking, lightweight coordination state, or any case where you want persistence without standing up a database server. ### Shared configuration All three sandboxes use the same slim Python image. SQLite ships with Python's standard library, so no extra packages are needed. ``` from chalkcompute import Image, Sandbox, Volume VOLUME_NAME = "sqlite-persistence" MOUNT_PATH = "/data" DB_PATH = f"{MOUNT_PATH}/app.db" image = Image.debian_slim("3.12") ``` The volume is the only piece of shared state. Each sandbox mounts it at /data and accesses the database at /data/app.db. ### Step 1 — Write the database The writer sandbox creates a users table and inserts three rows. After the script finishes, the sandbox is terminated — the data now lives only on the volume. ``` writer = Sandbox( image=image, name="sqlite-writer", volumes=[(VOLUME_NAME, MOUNT_PATH)], ).run() result = writer.exec("python3", "-c", """ import sqlite3, os db_path = "/data/app.db" os.makedirs(os.path.dirname(db_path), exist_ok=True) conn = sqlite3.connect(db_path) conn.execute( "CREATE TABLE IF NOT EXISTS users " "(id INTEGER PRIMARY KEY, name TEXT, email TEXT)" ) conn.execute("INSERT OR IGNORE INTO users VALUES (1, 'Alice', 'alice@example.com')") conn.execute("INSERT OR IGNORE INTO users VALUES (2, 'Bob', 'bob@example.com')") conn.execute("INSERT OR IGNORE INTO users VALUES (3, 'Charlie', 'charlie@example.com')") conn.commit() rows = conn.execute("SELECT * FROM users ORDER BY id").fetchall() conn.close() print(f"Wrote {len(rows)} rows to {db_path}") for r in rows: print(f" {r}") """) print(result.stdout_text) # Wrote 3 rows to /data/app.db # (1, 'Alice', 'alice@example.com') # (2, 'Bob', 'bob@example.com') # (3, 'Charlie', 'charlie@example.com') writer.terminate() ``` At this point the writer is gone. The volume still holds app.db. ### Step 2 — Read from independent sandboxes Start two new sandboxes that mount the same volume. Each opens the database in read-only mode and queries it independently — proving the data survived the writer's shutdown. ``` reader_a = Sandbox( image=image, name="sqlite-reader-a", volumes=[(VOLUME_NAME, MOUNT_PATH)], ).run() reader_b = Sandbox( image=image, name="sqlite-reader-b", volumes=[(VOLUME_NAME, MOUNT_PATH)], ).run() READ_SCRIPT = """ import sqlite3 conn = sqlite3.connect("file:/data/app.db?mode=ro", uri=True) rows = conn.execute("SELECT * FROM users ORDER BY id").fetchall() conn.close() for r in rows: print(r) print(f"Read {len(rows)} rows") """ for name, reader in [("reader-a", reader_a), ("reader-b", reader_b)]: result = reader.exec("python3", "-c", READ_SCRIPT) print(f"--- {name} ---") print(result.stdout_text) ``` ``` --- reader-a --- (1, 'Alice', 'alice@example.com') (2, 'Bob', 'bob@example.com') (3, 'Charlie', 'charlie@example.com') Read 3 rows --- reader-b --- (1, 'Alice', 'alice@example.com') (2, 'Bob', 'bob@example.com') (3, 'Charlie', 'charlie@example.com') Read 3 rows ``` Both readers see all three rows. The volume acts as durable shared storage that any number of sandboxes can mount. ### Cleanup Terminate the readers and — if you're done — delete the volume: ``` reader_a.terminate() reader_b.terminate() # Delete the volume when you no longer need the data. vol = Volume(name=VOLUME_NAME) vol.delete() ``` ### How it fits together ``` Writer Volume Reader A Reader B │ │ │ │ │ sqlite3.connect() │ │ │ │ CREATE TABLE users │ │ │ │ INSERT 3 rows │ │ │ │ conn.commit() │ │ │ │──────────────────────▸│ app.db │ │ │ │ │ │ │ terminate() │ │ │ ╳ │ │ │ │ │ │ │ volumes=[...] │ │ │◂────────────────────────│ │ │ volumes=[...] │ │◂────────────────────────────────────────│ │ │ │ │ SELECT * FROM users │ │ │◂────────────────────────│ │ │ 3 rows ───────────────▸│ │ │ │ │ │ SELECT * FROM users │ │◂────────────────────────────────────────│ │ 3 rows ───────────────────────────────▸│ │ │ │ ``` ### When to use this pattern Good fit: - Experiment metadata, run configs, or coordination state under ~1 GB - Data that one process writes and many read (write-once, read-many) - Situations where standing up Postgres or Redis is overkill Not a good fit: - Concurrent writers — SQLite uses file-level locking, so only one writer can hold the lock at a time. For multi-writer workloads, use Postgres. - Datasets larger than a few GB — SQLite performs well up to moderate sizes, but a dedicated database handles large-scale analytics better. - Low-latency reads under load — volume I/O goes through ChalkFS (FUSE), which adds a small overhead compared to local disk. If you need sub-millisecond reads, consider Redis or an in-memory cache. # Kafka Consumer source: https://docs.chalk.ai/docs/compute/kafka-consumer ## Subscribe to a Kafka topic from a sandbox and fan out processing to remote functions. ### What we're building A long-running sandbox that consumes messages from a Kafka topic and dispatches each one to a remote worker function for processing. The consumer handles backpressure, offset commits, and retries; the workers are stateless and scale independently. This pattern is useful when you need to react to a stream of events — new orders, log entries, user actions — and each event requires non-trivial compute like model inference, enrichment, or writing to an external system. ### The worker function Start with the leaf. Each Kafka message contains a JSON payload describing an event. The worker parses it, does some work, and returns a result. For this example, we'll classify support tickets and write the label back to a database. ``` # worker.py import json import os import chalkcompute from chalkcompute import Image, Secret worker_image = ( Image.base("python:3.12-slim") .pip_install(["openai", "psycopg2-binary"]) ) @chalkcompute.function( name="classify-ticket", image=worker_image, cpu="1", memory="2Gi", min_replicas=0, max_replicas=10, concurrency=8, secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("DATABASE_URL"), ], ) def classify_ticket(message_json: str) -> str: """Classify a support ticket and persist the label.""" import openai import psycopg2 message = json.loads(message_json) ticket_id = message["ticket_id"] subject = message["subject"] body = message["body"] # 1. Classify via LLM. client = openai.OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://router.chalk.ai/v1", ) response = client.chat.completions.create( model="gpt-4o-mini", messages=[ { "role": "system", "content": ( "Classify the support ticket into exactly one category: " "billing, bug, feature_request, account, other. " "Respond with only the category name." ), }, {"role": "user", "content": f"Subject: {subject}\n\n{body}"}, ], ) label = response.choices[0].message.content.strip().lower() # 2. Write the label back to Postgres. conn = psycopg2.connect(os.environ["DATABASE_URL"]) try: with conn.cursor() as cur: cur.execute( "UPDATE tickets SET category = %s, classified_at = NOW() WHERE id = %s", (label, ticket_id), ) conn.commit() finally: conn.close() return f"{ticket_id}:{label}" ``` Deploy the worker: ``` python worker.py ``` With max_replicas=10 and concurrency=8, up to 80 tickets can be classified in parallel. Chalk scales the workers up when the queue backs up and down to zero when it drains. ### The Kafka consumer The consumer is a long-running sandbox that reads from a topic and calls the worker function for each message. It runs a simple consume loop — no framework needed beyond confluent-kafka. ``` # consumer.py import json import os import logging import chalkcompute from confluent_kafka import Consumer, KafkaError logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") log = logging.getLogger(__name__) TOPIC = os.environ.get("KAFKA_TOPIC", "support-tickets") GROUP_ID = os.environ.get("KAFKA_GROUP_ID", "ticket-classifier") BATCH_SIZE = int(os.environ.get("BATCH_SIZE", "20")) def main() -> None: consumer = Consumer({ "bootstrap.servers": os.environ["KAFKA_BROKERS"], "group.id": GROUP_ID, "auto.offset.reset": "earliest", "enable.auto.commit": False, "security.protocol": os.environ.get("KAFKA_SECURITY_PROTOCOL", "SASL_SSL"), "sasl.mechanisms": "PLAIN", "sasl.username": os.environ["KAFKA_USERNAME"], "sasl.password": os.environ["KAFKA_PASSWORD"], }) consumer.subscribe([TOPIC]) # Get a reference to the worker function. classify = chalkcompute.RemoteFunction.from_name("classify-ticket") classify.wait_ready() log.info("Consuming from %s (group: %s)", TOPIC, GROUP_ID) pending: list[tuple[str, chalkcompute.FunctionCallHandle]] = [] try: while True: msg = consumer.poll(timeout=1.0) if msg is None: continue if msg.error(): if msg.error().code() == KafkaError._PARTITION_EOF: continue log.error("Consumer error: %s", msg.error()) continue # Dispatch to the worker. value = msg.value().decode("utf-8") handle = classify.defer(value) pending.append((value, handle)) # Flush in batches to balance throughput and latency. if len(pending) >= BATCH_SIZE: _flush(pending) consumer.commit() pending = [] except KeyboardInterrupt: log.info("Shutting down...") finally: if pending: _flush(pending) consumer.commit() consumer.close() def _flush(pending: list[tuple[str, chalkcompute.FunctionCallHandle]]) -> None: """Wait for all in-flight worker calls to complete.""" for value, handle in pending: try: result = handle.get(timeout=60) log.info("Classified: %s", result) except Exception: ticket = json.loads(value).get("ticket_id", "?") log.exception("Failed to classify ticket %s", ticket) log.info("Flushed batch of %d messages", len(pending)) if __name__ == "__main__": main() ``` The consumer commits offsets only after the entire batch has been processed. If a worker fails, the message will be redelivered on the next consumer restart — no data loss. ### Deploy the consumer The consumer runs as a plain sandbox (not a function). It needs Kafka credentials and chalkcompute to call the worker. ``` # deploy_consumer.py from chalkcompute import Image, Sandbox, Secret image = ( Image.base("python:3.12-slim") .pip_install(["confluent-kafka", "chalkcompute"]) .add_local_file("consumer.py", "/app/consumer.py") ) sandbox = Sandbox( image=image, name="ticket-consumer", cpu="1", memory="2Gi", secrets=[ Secret.from_chalk_env("KAFKA_BROKERS"), Secret.from_chalk_env("KAFKA_USERNAME"), Secret.from_chalk_env("KAFKA_PASSWORD"), ], env={ "KAFKA_TOPIC": "support-tickets", "KAFKA_GROUP_ID": "ticket-classifier", "BATCH_SIZE": "20", }, entrypoint=["python", "/app/consumer.py"], ).run() print(f"Consumer running: {sandbox.id}") ``` ``` python deploy_consumer.py # Consumer sandbox running: sandbox-id ``` ### How it fits together ``` Kafka Consumer Sandbox classify-ticket workers (support-tickets) (0–10 instances) │ │ │ │ msg: {ticket_id, │ │ │ subject, body} │ │ │────────────────────────▸│ │ │ msg │ │ │────────────────────────▸│ │ │ msg │ │ │────────────────────────▸│ │ │ ... │ │ │────────────────────────▸│ │ │ │ │ │ │ async_call(msg_json) x N │ │ │──────────────────────────▸│──▸ OpenAI (classify) │ │ │──▸ Postgres (persist) │ │ │ │ │ results (batch) │ │ │◂──────────────────────────│ │ │ │ │◂─────────────────────── │ │ │ commit offsets │ │ │ │ │ ``` The consumer polls messages, fans them out as async function calls, waits for the batch to complete, then commits. Workers scale independently — the consumer stays at one instance while workers expand to handle throughput spikes. ### Scaling the consumer For topics with many partitions, run multiple consumer instances in the same consumer group. Kafka will distribute partitions across them automatically: ``` from chalkcompute import ScalingGroup, Secret consumer_group = ScalingGroup( image=image, name="ticket-consumer", min_replicas=2, max_replicas=5, secrets=[ Secret.from_chalk_env("KAFKA_BROKERS"), Secret.from_chalk_env("KAFKA_USERNAME"), Secret.from_chalk_env("KAFKA_PASSWORD"), ], env={ "KAFKA_TOPIC": "support-tickets", "KAFKA_GROUP_ID": "ticket-classifier", "BATCH_SIZE": "20", }, entrypoint=["python", "/app/consumer.py"], ).deploy() ``` The scaling group keeps between two and five consumer replicas running. Set max_replicas no higher than your topic's partition count — extra consumers beyond that will sit idle. # Evaluations source: https://docs.chalk.ai/docs/compute/evaluations ## Evaluate any LLM workload against historical production data and compare outcomes based on criteria you define. ### Overview An evaluation is defined by three things: a completed dataset revision, one task, and one or more scorers. The task is usually a deployed @chalkcompute.function version. In the dashboard, Quick evaluation can also make it an LLM prompt, which it deploys as a function for you, or a Chalk SQL expression. A scorer is a deployed function version or a Chalk SQL expression. Creating an evaluation saves that definition in the active environment. Running it queues a separate execution that calls the task once per dataset row, passes each output to every scorer, and writes the scores to a result dataset. The definition is immutable, so two runs of the same evaluation stay comparable: the data, the task, and the scorers are fixed at creation time. To evaluate a new version of the task, create a new evaluation. ``` ┌──────────────────┐ define execute │ dataset revision │──┐ ┌──────────────┐ ┌─────────────────┐ │ task version │──┼──────▸│ Evaluation │──────▸│ EvaluationRun │ │ scorer versions │──┘ │ immutable │ │ repeatable │ └──────────────────┘ └──────────────┘ └────────┬────────┘ pinned inputs │ produces ▾ result dataset revision ``` Function tasks and scorers are ordinary Chalk Functions, so everything on that page applies to them: resource configuration, secrets, retries, rate limits, and tracing. Evaluations are offline. They score a pinned dataset revision, not live traffic. Before your first run, ask Chalk support to enable Evaluations and the async Function queue in your environment. A new environment also needs an active deployment. The Quick start walks through the remaining setup and shows how to check the result. # Evaluations Quick Start source: https://docs.chalk.ai/docs/evaluations/quick-start ## Upload a dataset, deploy a task, and run a first evaluation with a built-in scorer. This example uploads a small dataset, deploys a task function, and runs an evaluation that scores the task's output with one built-in scorer. ### Before you start - Select an environment with Evaluations, an authenticated Function queue, and dataset storage configured. Managed self-signup serverless environments enable Evaluations and Function queue authentication by default. - Install the Chalk CLI, run chalk login, and select the intended environment with chalk environment . Check it with chalk user. A Chalk Sandbox with managed Chalk identity enabled works instead. - You do not need to deploy feature definitions or resolvers. The task and scorer functions in this example are provisioned independently. - In the Python environment that will run the example, install:pip install chalkcompute pyarrow You can run this example from your own machine. A Chalk Sandbox is optional: it hosts the Python client while Chalk runs the task and scorer Functions remotely. ``` import chalkcompute import pyarrow.compute as pc @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) valid_json = chalkcompute.scorers.json_valid(name="valid-json") if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=[valid_json], ) run = evaluation.run(metadata={"git_sha": "abc123"}).wait() print(run.status) table = chalkcompute.DatasetClient().read(run.result_dataset) print(pc.mean(table["valid-json_value"]).as_py()) ``` Save it as a file, such as quick_start.py, and run it with python quick_start.py: ``` EvaluationRunStatus.SUCCEEDED 0.75 ``` Check that the result dataset has four rows, then inspect output_error and valid-json_error for row failures. A run can finish with SUCCEEDED while some rows have errors. Save the evaluation and run IDs if you need to replay the pinned definition or ask support to investigate. To replay it, call run() on the existing evaluation rather than uploading the same inputs again; see Running an evaluation. If a run fails with CHALK_FUNCTION_QUEUE_ENDPOINT is empty or unset, check that the environment has an authenticated Function queue configured. Chalk imports this file inside every function it deploys, here triage and the valid-json scorer, and the import runs the file's top-level code. That gives the file two parts: - Functions and scorers are defined at the top level, so each deployed function can find its own definition when it imports the file. - The code that uploads the dataset and creates and runs the evaluation goes under if __name__ == "__main__":, so it runs only when you run the file. Rows inline are one of the forms upload accepts, so there is no file to prepare first. A row needs whatever columns the task and scorers read: message for the task, and expected and expected_intent for the scorers on the Scorers page. triage reads the first number it finds, so the order number in the first row becomes a dollar amount. It checks for "refund" before "cancel", so the third row is classified as a refund. It returns text that is not JSON when nothing matches, as in the fourth. That last row is the one valid-json marks down, giving the 0.75 above. To create and run an evaluation from the Chalk dashboard without writing code, use Quick evaluation. To find out what the 0.75 is missing and improve it, continue with the tutorial. The other Evaluations pages continue with that example, through scorers and on to reading the results. Where a feature is clearer with something else, such as LLM judges, those sections use an example of their own. # Tutorial: Improving an Evaluation Score source: https://docs.chalk.ai/docs/evaluations/tutorial ## Evaluate a support ticket router. Read the rows behind its score, add a scorer that checks the routing, and fix the router. Suppose you are building a support ticket router. Each customer message comes in, and your code decides what kind of request it is (a refund, a cancellation, a billing question, or an order status check) so that it goes to the team that handles it. Before you ship a change to the router, you want to know two things: whether it sends messages to the right team, and whether the new version does that better than the old one. An evaluation answers both. The Quick start built a first version of that router, triage, and evaluated it on four messages, with a score of 0.75. This tutorial picks up from there. You will look at the rows behind the score, add a scorer that checks whether each message was routed correctly, fix the router, and compare the two versions. Along the way you will see that a score only covers what you grade for. You need the setup from the Quick start's Before you start section. The tutorial reuses the Quick start's task and dataset, so there is nothing else to prepare. ### Step 1: Look at the rows behind the score Save this file as improve_score.py. It is the Quick start with one addition: show prints each row's scores next to the task's output, and then the average of each score. ``` import chalkcompute import pyarrow.compute as pc @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) valid_json = chalkcompute.scorers.json_valid(name="valid-json") def show(run, *scorers): table = chalkcompute.DatasetClient().read(run.result_dataset) for row in table.to_pylist(): print([row[f"{name}_value"] for name in scorers], row["output"]) print("average", [pc.mean(table[f"{name}_value"]).as_py() for name in scorers]) if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=[valid_json], ) run = evaluation.run().wait() show(run, "valid-json") ``` Run it with python improve_score.py: ``` [1.0] {"intent": "refund", "amount": 1182.0} [1.0] {"intent": "status", "amount": 0.0} [1.0] {"intent": "refund", "amount": 12.0} [0.0] unclassified average [0.75] ``` Three of the four rows pass, and 0.75 looks like a reasonable start. Now compare each output with its message: - The first row is a refund for the right reason, but triage read the order number 1182 as the dollar amount. - The third row asks to close an account, which is a cancellation. triage checks for "refund" before "cancel", so it called it a refund. Both answers are wrong, and both scored 1. valid-json checks whether the output is well-formed JSON, and both outputs are. The scorer is doing exactly what it was written to do. It has no way to know the answers are wrong, because nothing in it compares the output with what the row expected. For the router, that means 0.75 says nothing about whether messages reach the right team. The cancellation in the third row would go to the refunds team. ### Step 2: Add a scorer that checks the intent What the router has to get right is the intent, since the intent decides which team gets the message. The dataset has an expected column holding the right answer for each row. Add a scorer that compares the intent field of the output with the intent field of expected. Put it below valid_json, at the top level of the file, so that it is defined when Chalk deploys it: ``` intent_match = chalkcompute.scorers.json_match( inputs=("output", "expected"), paths=["intent"], name="intent-match" ) ``` An evaluation's scorers are fixed when it is created, along with its dataset and task, so you cannot add one to the evaluation from Step 1. You can rescore its run instead. Add these lines at the end of the if __name__ == "__main__": block: ``` rescored = run.rescore(scorers=[valid_json, intent_match]).wait() show(rescored, "valid-json", "intent-match") ``` rescore creates a new evaluation named support-triage (rescored) with the same dataset revision and task and the scorers you pass. It copies each row's output from the original run and calls only the scorer that is new, so triage does not run again. Run the file again. After the Step 1 output, it prints: ``` [1.0, 1.0] {"intent": "refund", "amount": 1182.0} [1.0, 1.0] {"intent": "status", "amount": 0.0} [1.0, 0.0] {"intent": "refund", "amount": 12.0} [0.0, 0.0] unclassified average [0.75, 0.5] ``` The new score is lower, and nothing about the task changed. intent-match marks down the cancellation that was called a refund, and the row that was not JSON at all. The task was this good all along; the evaluation now measures more of it. Two of the four messages would reach the right team, and that is the number to improve. ### Step 3: Fix the task intent-match points at two problems in triage: it checks for "refund" before "cancel", and it returns plain text when no intent matches. Add a second version of the task below the first, with both fixed: ``` @chalkcompute.function def triage_v2(message: str) -> str: import json, re text = message.lower() if "cancel" in text or "close my account" in text: intent = "cancel" elif "refund" in text or "money back" in text: intent = "refund" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: intent = "other" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) ``` An evaluation also pins the version of its task, so evaluating triage_v2 takes a new evaluation. Keeping it as a separate function leaves the original in place, so both versions stay available to compare. Add these lines at the end of the if __name__ == "__main__": block: ``` fixed = chalkcompute.Evaluation.create( "support-triage-v2", dataset=dataset, task=triage_v2, scorers=[valid_json, intent_match], ) show(fixed.run().wait(), "valid-json", "intent-match") ``` Run the file again. After the output of the first two steps, it prints: ``` [1.0, 1.0] {"intent": "refund", "amount": 1182.0} [1.0, 1.0] {"intent": "status", "amount": 0.0} [1.0, 1.0] {"intent": "cancel", "amount": 12.0} [1.0, 0.0] {"intent": "other", "amount": 0.0} average [1.0, 0.75] ``` The cancellation is now classified correctly, and every output is JSON. Three of the four messages reach the right team, up from two. Both versions ran on the same dataset revision with the same scorers, so the scores compare directly, and the new version is better. ### What the scores still miss Two of the rows are still wrong. The fourth row, "Following up on my last email," still fails intent-match. The dataset labels it a status question, and triage_v2 answers other. The message could be about anything, so this may be a case where the label is the thing to fix. A failing row can point at the dataset as well as the task. The first row still reads the order number 1182 as the amount, and no score shows it, because neither scorer checks the amount. This is the same gap as in Step 1: an evaluation finds the mistakes you wrote a scorer for, and only those. ### Comparing the runs in the dashboard An evaluation created without a suite goes in the environment's Default suite, so all three are there in the Chalk dashboard. Open Evaluations, then Default, to see their runs side by side in the Runs table and the chart above it: Average score is the mean of a run's scorers, so it is 0.625 for the rescored run and 0.875 for support-triage-v2. To line up the two versions of the task on the same scorers, open support-triage-v2 and use Compare to add support-triage (rescored) beside it. See The Evaluations dashboard. ### The complete file ``` import chalkcompute import pyarrow.compute as pc @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) @chalkcompute.function def triage_v2(message: str) -> str: import json, re text = message.lower() if "cancel" in text or "close my account" in text: intent = "cancel" elif "refund" in text or "money back" in text: intent = "refund" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: intent = "other" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) valid_json = chalkcompute.scorers.json_valid(name="valid-json") intent_match = chalkcompute.scorers.json_match( inputs=("output", "expected"), paths=["intent"], name="intent-match" ) def show(run, *scorers): table = chalkcompute.DatasetClient().read(run.result_dataset) for row in table.to_pylist(): print([row[f"{name}_value"] for name in scorers], row["output"]) print("average", [pc.mean(table[f"{name}_value"]).as_py() for name in scorers]) if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=[valid_json], ) run = evaluation.run().wait() show(run, "valid-json") rescored = run.rescore(scorers=[valid_json, intent_match]).wait() show(rescored, "valid-json", "intent-match") fixed = chalkcompute.Evaluation.create( "support-triage-v2", dataset=dataset, task=triage_v2, scorers=[valid_json, intent_match], ) show(fixed.run().wait(), "valid-json", "intent-match") ``` ### Next steps - To grade qualities a fixed rule cannot check, such as whether an answer is helpful, use an LLM judge. - To group evaluations like these three so their runs appear together, file them in a suite. - To see exactly which calls a rescore copies and which it makes again, read Rescoring a run. # Evaluation Datasets source: https://docs.chalk.ai/docs/evaluations/datasets ## Upload, read, and refer to the dataset revisions that evaluations score. Upload a dataset with DatasetClient.upload(name, data). It registers a dataset revision and returns a DatasetRevisionRef you pass straight to Evaluation.create. Your data is ordinary tabular data, so this needs neither ChalkPy nor feature definitions. Pass any of the following as data: - a CSV or Parquet path, or a sequence of them, whose Arrow schemas match exactly, - a PyArrow table or record batch, - a mapping of column name to values, or a sequence of row mappings, - any dataframe exposing to_arrow or the dataframe interchange protocol. CSV files and in-memory values are converted to temporary Parquet files locally before upload. Uploading under a name that already exists creates a new revision of that dataset. ``` import chalkcompute client = chalkcompute.DatasetClient() # From files revision = client.upload("support-triage", ["jan.parquet", "feb.parquet"]) # From rows, with progress revision = client.upload( "support-triage", [{"message": "where is my order?", "expected": '{"intent": "status", "amount": 0.0}'}], on_progress=lambda done, total: print(f"{done}/{total}"), ) ``` upload also accepts part_size_bytes to set the multipart chunk size and timeout to bound each request, in seconds. Six column names are reserved by the evaluation runtime: output, trace, __chalk_evaluation_output, __chalk_evaluation_row_id, __chalk__session_id__ and __chalk__task_session_id__. A dataset carrying any of them is rejected by Evaluation.create with dataset column "" is reserved by the evaluation runtime. ### Reading a dataset DatasetClient.read returns a revision as a PyArrow table. Pass it the DatasetRevisionRef that upload returned, or a revision id: ``` import chalkcompute table = chalkcompute.DatasetClient().read(revision) ``` timeout bounds the request in seconds and defaults to 300. A revision with no output raises DatasetError, which is what you get for an evaluation run that is still going or that failed. This is how you read a run's scores as well; see Reading results for the columns a run writes. ### Referring to a dataset Pass Evaluation.create(dataset=...) the reference upload returned, or build a DatasetRevisionRef for a dataset that is already there: ``` import chalkcompute # What upload returns: the revision it just registered revision = chalkcompute.DatasetClient().upload("support-triage", "support_messages.csv") # An exact revision, by id chalkcompute.DatasetRevisionRef(dataset_id="...", revision_id="...") # The latest revision of a dataset, resolved when the evaluation is created chalkcompute.DatasetRevisionRef(dataset_name="support-triage") ``` A name resolves once, when you create the evaluation, and the resolved revision is then pinned. Later uploads under the same name leave existing evaluations alone. DatasetRevisionRef rejects a mix of the two forms: pass either dataset_name, or dataset_id together with revision_id. create also takes any object exposing dataset_id and revision_id, which pins that revision, or one exposing dataset_name, whose latest revision it resolves. A dataset handle from another Chalk client therefore passes straight through. # Evaluation Tasks source: https://docs.chalk.ai/docs/evaluations/tasks ## Write the task function an evaluation calls once per dataset row, or attach a function that is already deployed. Write the task as an ordinary @chalkcompute.function. Its parameters bind to your dataset's columns by name, so triage in the Quick start declares message: str and requires the dataset to have a message column. ``` import chalkcompute @chalkcompute.function def triage(message: str) -> str: ... ``` @chalkcompute.function starts deploying in the background as soon as the module runs, so consecutive definitions build concurrently. Evaluation.create waits on those handles before reading their immutable version IDs, so you do not need to call deploy() yourself for decorated functions. The remote Function worker imports the module containing the task or scorer. Keep their definitions importable at module scope, include dependencies used by the module in the Function image, and put dataset upload, suite creation, and evaluation runs inside main() under an if __name__ == "__main__": guard. The Quick start uses that structure. Avoid requiring a file that exists only on your local machine during a remote import. ### Attaching deployed functions by reference Attach functions that already exist rather than redefining them. In this example, dataset is the dataset uploaded in the Quick start: ``` import chalkcompute evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=chalkcompute.RemoteFunction.from_name("triage"), scorers=[chalkcompute.RemoteFunction.from_version_id("fn_intent_match_v2")], ) ``` RemoteFunction.from_name(name) resolves the currently selected version at lookup time, and creating the evaluation then pins that version. A decorated function registers under its Python name, so triage here is the def triage from the Quick start. RemoteFunction.from_version_id(id) attaches an exact, immutable version; from_id is a compatibility alias for it. Deploy an imperative RemoteFunction before you use it. Evaluation.create never deploys on your behalf: an undeployed handle raises EvaluationError before any request goes out, and a bare callable raises TypeError. # Evaluation Scorers source: https://docs.chalk.ai/docs/evaluations/scorers ## Score each evaluation row with built-in scorers, LLM judges, SQL expressions, or functions you write. A scorer reads one row and returns a number between 0 and 1. chalkcompute.scorers ships twelve of them, and you can write your own. Most evaluations mix several. The examples on this page continue the Quick start. To run them on their own, start a file with the code below, which deploys the Quick start's task and uploads its dataset, and add an example after it. As in the Quick start, definitions stay at the top level and the code that uploads data or runs an evaluation goes under if __name__ == "__main__":. ``` import chalkcompute @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) ``` ### Choosing a scorer The built-in scorers differ in how they arrive at a score: - Deterministic comparisons grade one column against another by a fixed rule, covering text, numbers, JSON, and regular expressions. They are cheap and repeatable, and they are the place to start. - Model-based scorers judge meaning rather than form. embedding_similarity catches an answer that is right but worded differently, and llm_judge grades qualities you can describe but not compute, such as tone or whether an answer actually helped. jev grades with a fixed set of labels and sends the rows it is unsure of to a second model. - SQL expressions score from the dataset's own columns through scorers.sql, the one built-in that deploys nothing. Writing your own function is the fourth option, and the only one outside chalkcompute.scorers. Reach for it for business rules, and anything that needs real computation. To add, change or remove an evaluation's scorers from the Chalk dashboard, use Edit scorers. ### Built-in scorers Every built-in is a function you call. Calling it deploys the scorer and returns a handle to pass to scorers=, so call it even when you change none of its options. Call it at the top level of the file, as with your own functions. scorers.sql is the exception, deploying nothing. llm_judge, jev and sql are configured differently from the rest, and have their own sections below. The options for the other nine follow the table. ``` import chalkcompute scorers = [ chalkcompute.scorers.json_valid(inputs=("output",), name="valid-json"), chalkcompute.scorers.json_match(inputs=("output", "expected"), name="full-match"), chalkcompute.scorers.json_match( inputs=("output", "expected"), paths=["intent"], name="intent-match" ), chalkcompute.scorers.contains(inputs=("output", "expected_intent"), name="mentions-intent"), ] if __name__ == "__main__": evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=scorers, ) ``` On the four rows from the Quick start, those four score valid-json 0.75, intent-match 0.5, mentions-intent 0.5 and full-match 0.25. full-match is lowest because it compares the whole document, so the wrong amount in the first row counts against it even though the intent is right. | Scorer | What it measures | | -- | -- | | exact_match | Two columns hold the same text | | contains | Expected substrings appear in a column | | regex_match | A column matches a regular expression | | levenshtein | Edit distance, normalized into a score | | string_similarity | A named similarity measure, such as Jaro-Winkler | | json_valid | A column parses as JSON | | json_match | Two columns agree as JSON documents | | numeric_close | Two columns agree as numbers | | embedding_similarity | Two columns mean the same thing | | llm_judge | A quality you describe in a pydantic model | | jev | A label a model picks, regraded by a second model when the first is unsure | | sql | A Chalk SQL expression you write yourself | Each of the nine keyword-configured scorers takes name, which labels the scorer, and inputs, which names the columns it reads. inputs defaults to the columns that scorer needs: ("output", "expected") for the seven that compare two columns, and ("output",) for regex_match and json_valid, which read one. Any remaining keywords go to @chalkcompute.function, so a built-in takes the same image, resource, and retry options as any other Chalk Function. Each one records metadata explaining its score. The options worth knowing per scorer: - exact_match strips each side by default. normalize_whitespace collapses runs of whitespace instead, and case_insensitive folds case. - contains reads one substring, or a JSON array of them, from the reference column. mode scores them together: "all" demands every one, "any" one of them, and "fraction" gives the share that appeared. - regex_match takes its pattern positionally, and full_match requires the pattern to cover the whole column. Named groups are recorded as metadata, so a pattern can also pull a value out of the row. - string_similarity takes its measure positionally: "jaro_winkler", "jaccard", "token_set", "token_sort", "partial", or "sequence". - numeric_close with a tolerance scores 1.0 or 0.0 on whether the two are within it, taken as a share of the larger value when relative. Without one, the score falls off with the gap between them. - json_match compares whole documents unless you pass paths, so object key order does not matter but list order does. With paths, written dotted as in "user.id", the score is the share of those paths whose values agree. - embedding_similarity embeds both columns with model, defaulting to text-embedding-3-small, and scores them by cosine similarity. It catches an answer that is right but worded differently, which the string scorers mark wrong. base_url and api_key name the endpoint and the Secret holding its key. ### LLM judges chalkcompute.scorers.llm_judge deploys a scorer described by a pydantic model (v1 or v2) instead of a function body. The model's score field becomes the score, and every other field is recorded as row metadata. The judge requests the reply with structured outputs and validates it against the model, so a malformed grade fails the row. With no api_key and no base_url, the judge calls the Model Gateway as your environment, so it needs no provider key. model names a chat model the gateway serves; run chalk router model list to list them. This example grades a browser agent instead of the Quick start's task, so it goes in a file of its own: ``` import chalkcompute from pydantic import BaseModel, Field class TraceQuality(BaseModel): score: float = Field(ge=0, le=1, description="Overall quality.") directness: int = Field(ge=0, le=2, description="Shortest reasonable path to the goal.") task_correctness: int = Field(ge=0, le=2, description="Was the task actually completed?") reason: str trace_quality = chalkcompute.scorers.llm_judge( TraceQuality, model="openai/gpt-5", instructions="You are grading a browser agent's login attempt.", ) ``` The scorer deploys under the model's class name in kebab case, so TraceQuality becomes trace-quality. Pass name= to choose another. The remaining keywords: - inputs names the dataset columns the judge reads, defaulting to ("output",). - prompt_fn replaces the default prompt, and parse_fn replaces structured outputs for an endpoint that does not support them. - completion_kwargs, for example {"temperature": 0, "max_tokens": 400}, are sent with every request. - base_url points at another OpenAI-compatible endpoint, and api_key takes a Secret holding its key, which the judge reads in the pod. With either one set, the judge calls that endpoint instead of the Model Gateway, and api_key on its own calls OpenAI. - image supplies a base image, and any other keyword goes to @chalkcompute.function. Every function deployed from a module imports that module, so give sibling functions in the same file an image that installs pydantic as well. A judge is generated at import time, which means it cannot be deployed in strip mode. ### Escalating judges chalkcompute.scorers.jev is a judge that grades each row with one of a fixed set of labels, and passes the rows it is unsure of to a second model. model picks a label for the row and gives a probability for every label. When its probability on the label it picked is below threshold, which defaults to 0.9, fallback_model grades the row again and its grade stands. ``` import chalkcompute intent_judge = chalkcompute.scorers.jev( labels={"correct": 1.0, "partial": 0.5, "wrong": 0.0}, rubric=( "correct: the intent in the output is what the customer asked for. " "partial: the intent is close, or the output names the right intent but is not valid JSON. " "wrong: anything else." ), model="openai/gpt-5-mini", fallback_model="openai/gpt-5", inputs=("message", "output"), name="intent-judge", ) if __name__ == "__main__": evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=[intent_judge], ) ``` labels maps each label to its score, between 0 and 1, and needs at least two. rubric says what each label means. Labels match regardless of case, and a reply that picks a label outside labels scores 0 and goes to fallback_model. Each row's metadata records the label in choice, the model's rationale, the normalized probabilities, confidence in the chosen label, whether the row was escalated, the judge_model whose grade stands, and primary_confidence, the first model's confidence. The probabilities are what the model reports about its own confidence. They are not calibrated, so the share of rows a given threshold escalates depends on the model. name defaults to jev and inputs to ("output",). instructions, base_url, api_key, completion_kwargs and image work as they do for llm_judge, including calling the Model Gateway when neither api_key nor base_url is set. ### SQL expression scorers scorers.sql scores a row with a Chalk SQL expression rather than a deployed function. Nothing is deployed, so there is no image to build and no cold start. ``` import chalkcompute answered = chalkcompute.scorers.sql( "answered", '''CASE WHEN "output" IS NOT NULL AND "output" <> '' THEN 1.0 ELSE 0.0 END''', ) ``` The expression is a select item over the evaluation's own columns: every dataset column by name, plus output, the task's result for that row. It must yield a number. name labels the result column and the row in the compare table. Quote column names, since output and other bare identifiers can collide with SQL keywords. ### Writing your own scorer Write a scorer function the same way as a task function. Its parameters bind to dataset columns by name, plus one argument the runner supplies: output, the task's return value for that row. ``` import chalkcompute @chalkcompute.function def amount_is_plausible(message: str, output: str) -> float: import json try: amount = json.loads(output)["amount"] except Exception: return 0.0 return 1.0 if amount == 0.0 or str(amount).rstrip("0").rstrip(".") in message else 0.0 ``` Here message comes from the dataset and output comes from the task. A scorer can read any column in the dataset, not only the one holding the expected answer, which is what lets this one check the extracted amount against the message it came from. ### Scorer results A scorer must return a floating point number between 0 and 1 inclusive, one EvaluationScorerResult, or a list[EvaluationScorerResult]. A list lets one scorer emit several scores from shared computation, such as a single model call you grade on three axes. An empty list emits no scores for that row. ``` import chalkcompute @chalkcompute.function def per_field(output: str, expected: str) -> list[chalkcompute.EvaluationScorerResult]: import json try: got, want = json.loads(output), json.loads(expected) except ValueError: return [] return [ chalkcompute.EvaluationScorerResult( score=1.0 if got.get(field) == value else 0.0, metadata={"field": field}, ) for field, value in want.items() ] ``` EvaluationScorerResult takes a score and optional metadata: - score must be a finite number between 0 and 1, inclusive. A boolean raises TypeError, and a score outside the range raises ValueError. This check runs when EvaluationScorerResult is constructed, so returning the dataclass is what enforces the range. A scorer returning a plain number, and a SQL expression scorer, are not checked. - metadata must be JSON-serializable, with no NaN or infinity. Chalk stores it per row alongside the score. The return annotation declares the Arrow schema for the function, and EvaluationScorerResult carries its own Arrow contract. Return the dataclass rather than a plain dictionary, which has no such contract and fails to serialize. # Creating and Running Evaluations source: https://docs.chalk.ai/docs/evaluations/running ## Create an evaluation definition, run it, wait for the result, and rescore a finished run. The examples on this page continue the Quick start. To run them on their own, start a file with the code below, which deploys the Quick start's task, uploads its dataset, and finds or creates a suite, and add the examples after it in order. As in the Quick start, definitions stay at the top level and the code that uploads data or creates and runs evaluations goes under if __name__ == "__main__":. ``` import chalkcompute @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) # Suite names are unique in an environment, so reuse "Release" if it exists. suite = next( (s for s in chalkcompute.EvaluationSuite.list_all() if s.name == "Release"), None ) or chalkcompute.EvaluationSuite.create("Release") ``` ### Creating an evaluation Pass the dataset, the task, and the scorers to Evaluation.create, along with any metadata you want stored on the definition: ``` import chalkcompute valid_json = chalkcompute.scorers.json_valid(name="valid-json") if __name__ == "__main__": evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=[valid_json], metadata={"owner": "support-eng"}, suite_id=suite.id, ) ``` name is required and must be non-empty, and at least one scorer is required. metadata must be a JSON-serializable mapping with string keys; Chalk stores it on the definition and returns it on every read. suite_id files the evaluation under a suite. The returned Evaluation exposes the pinned identifiers as dataset_id, dataset_revision_id, task_function_version_id, and scorers, a tuple of EvaluationScorer carrying an evaluation-local id and then whichever of function_version_id, builtin_id, or sql_name and sql_expression matches how that scorer was defined. Attach to an existing definition with chalkcompute.Evaluation.from_id("..."), and read the latest server state with evaluation.refresh(). ### Running an evaluation evaluation.run() is an asynchronous process that queues a run and returns a handle immediately: ``` if __name__ == "__main__": run = evaluation.run(metadata={"git_sha": "abc123"}) print(run.status) # queued, not yet finished final = run.wait() print(final.status, final.result_dataset) ``` wait() polls until the run reaches a terminal state and returns the final handle: | Parameter | Type | Default | Description | | ------------------ | --------------- | ------- | -------------------------------------------------------------------------------------------------------------------- | | `timeout` | `float \| None` | `600.0` | Seconds to wait; `None` waits indefinitely. Exceeding it raises `TimeoutError`. | | `poll_interval` | `float` | `2.0` | Seconds between status polls. | | `raise_on_failure` | `bool` | `True` | Raise `EvaluationRunFailedError` for a failed or canceled run. Pass `False` to inspect the terminal handle yourself. | EvaluationRunStatus covers PENDING, RUNNING, FINALIZING, SUCCEEDED, FAILED, CANCELED, and UNSPECIFIED. run.is_terminal is true for the three that end a run: SUCCEEDED, FAILED, and CANCELED. A successful run exposes run.result_dataset, a DatasetRevisionRef for the dataset revision holding the task output and each scorer's score and metadata per row. It stays None until the run produces one. run.error_message carries the failure detail, and a raised EvaluationRunFailedError exposes the terminal handle as error.run. A run also carries created_at, started_at, completed_at, created_by, the ID of whoever started it, and result_dataset_id and result_dataset_revision_id, the two IDs inside result_dataset. Read a definition's history with evaluation.runs(limit=...), newest first, or attach to one run with chalkcompute.EvaluationRun.from_id("...") and run.refresh(). Keep the run ID when starting a longer evaluation. If wait() times out, the run may still be active; reattach with EvaluationRun.from_id(run.id) and refresh or wait again. The run's pending time, execution time, and finalization time can differ. Inspect row errors in the result dataset before treating a terminal SUCCEEDED status as a pass. run() also takes execution_mode, which sets how Chalk carries a run to completion: - "workflow": a workflow on your environment's workflow orchestrator finishes the run, waits for its telemetry to arrive, and then computes its score and metric summaries. The run completes whether or not anything reads it. - "on_read": the run advances when something reads it, such as wait(), refresh(), or the dashboard. Without execution_mode, Chalk uses "workflow" when your environment has a workflow orchestrator and "on_read" when it does not. run.execution_mode holds the mode a run was given from the moment it is queued. A run that asks for "workflow" in an environment without an orchestrator is rejected. chalkcompute.EvaluationRunExecutionMode names the same two values. ### Identifying the run from a task or scorer Each call a run makes to its task or to a function scorer carries two headers: x-chalk-evaluation-id, the evaluation, and x-chalk-evaluation-run-id, the run the call serves. Read them inside the function with chalkcompute.get_call_context(): ``` import chalkcompute @chalkcompute.function def which_run(message: str) -> str: context = chalkcompute.get_call_context() return f'{context.get("x-chalk-evaluation-id")} {context.get("x-chalk-evaluation-run-id")}' ``` get_call_context() returns an empty mapping when the function is called outside Chalk, so both lookups return None there. During a rescore, x-chalk-evaluation-run-id is the rescored run. ### Rescoring a run The task is usually the slow, expensive half of an evaluation, and it stays the same while you iterate on how its outputs are graded. run.rescore() scores a finished run again and calls only what changed since that run. It reads the run's rows back out of its result dataset, copies the task outputs and scores that are unchanged, and calls the rest: ``` import chalkcompute intent_match = chalkcompute.scorers.json_match( inputs=("output", "expected"), paths=["intent"], name="intent-match" ) if __name__ == "__main__": run = chalkcompute.EvaluationRun.from_id("...") # Score the same outputs with new scorers. rescored = run.rescore(scorers=[valid_json, intent_match]).wait() ``` An evaluation's scorers are fixed when it is created, so rescore(scorers=...) creates a new evaluation. It copies the run's dataset revision, task and suite, replaces the scorers, and records the rescored run under it. That evaluation is named " (rescored)" unless you pass name=. Because the new evaluation is a normal one, later calls to evaluation.run() on it do run the task. To rescore into an evaluation that already exists, pass it instead: ``` if __name__ == "__main__": strict = chalkcompute.Evaluation.from_id("...") rescored = run.rescore(evaluation=strict) # or strict.rescore(run) ``` That evaluation must use the run's dataset revision, since the rescored run reads its rows out of the run's results. The server rejects any other dataset revision. The evaluation's task and scorers can differ from the run's, and the rescore compares them with the run's to decide what to call: - If the evaluation uses the run's task, each row's output is copied from the run. Otherwise the evaluation's task is called on the same rows. A run started by a post-training called the policy model in place of the task, so rescoring one always calls the task. - A scorer is copied when the run had the same scorer: the same function version, or a SQL scorer with the same expression. A copied function scorer is still called on the rows where its call failed in the run. - When the task is called again, a scorer that takes output or trace, and every SQL scorer, is called again too. - Every other scorer is called on every row. run.rescore() with no arguments rescores into the run's own evaluation, so it calls nothing except each function scorer on the rows where that scorer failed. Only a run that succeeded has outputs to rescore. The rescored run is an ordinary run of its evaluation: wait(), result_dataset and the dashboard all treat it like any other. Each row's new calls go to a session of their own, and a scorer that takes trace reads the spans of the task that produced the row's output: the original task's when it was copied, the new call's otherwise. # Reading and Comparing Evaluation Results source: https://docs.chalk.ai/docs/evaluations/results ## Read the scores a run writes to its result dataset, and compare evaluations and runs in the Evaluations dashboard. The examples on this page continue the Quick start and the built-in scorers example on the Scorers page. To run them on their own, start a file with the code below, which creates that evaluation, and add the examples after it in order. As in the Quick start, definitions stay at the top level and the code that uploads data or creates and runs evaluations goes under if __name__ == "__main__":. ``` import chalkcompute @chalkcompute.function def triage(message: str) -> str: import json, re text = message.lower() if "refund" in text or "money back" in text: intent = "refund" elif "cancel" in text or "close my account" in text: intent = "cancel" elif "invoice" in text or "billed" in text or "card" in text: intent = "billing" elif "where is" in text or "waiting" in text or "arrived" in text: intent = "status" else: return "unclassified" amount = re.search(r"\d+(?:\.\d+)?", message) return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0}) scorers = [ chalkcompute.scorers.json_valid(inputs=("output",), name="valid-json"), chalkcompute.scorers.json_match(inputs=("output", "expected"), name="full-match"), chalkcompute.scorers.json_match( inputs=("output", "expected"), paths=["intent"], name="intent-match" ), chalkcompute.scorers.contains(inputs=("output", "expected_intent"), name="mentions-intent"), ] if __name__ == "__main__": dataset = chalkcompute.DatasetClient().upload("support-triage", [ {"message": "I was charged twice for order 1182 ($49.99), please refund", "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"}, {"message": "Where is my order? It was due Tuesday.", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, {"message": "Please close my account and refund the last $12 charge", "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"}, {"message": "Following up on my last email", "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"}, ]) evaluation = chalkcompute.Evaluation.create( "support-triage", dataset=dataset, task=triage, scorers=scorers, ) ``` ### Reading results A run writes one row per dataset row into its result dataset. Read it with DatasetClient.read, which returns a PyArrow table: ``` import chalkcompute if __name__ == "__main__": run = evaluation.run().wait() table = chalkcompute.DatasetClient().read(run.result_dataset) ``` result_dataset is the DatasetRevisionRef for that run. See Reading a dataset for read's other arguments. The result carries the dataset's own columns, plus: | Column | Holds | | -- | -- | | output | the task's return value for that row | | output_error | the failure detail, when the task raised | | _value | that scorer's score | | _error | the failure detail, when that scorer raised | | _metadata | whatever the scorer recorded alongside its score | Check the task and scorer error columns for every row. A run-level SUCCEEDED status means a result dataset was produced; it does not mean that every task and scorer call succeeded. A SQL expression scorer contributes only _value, since nothing is deployed to record an error or metadata against it. The result also carries __chalk__-prefixed columns that the runtime uses to tie a row to its telemetry. These are internal and not part of the reference above. table is an ordinary PyArrow table, so summarize or filter it in place: ``` import pyarrow.compute as pc if __name__ == "__main__": print(pc.mean(table["intent-match_value"]).as_py()) # 0.5, on the Quick start rows print(table.filter(pc.equal(table["intent-match_value"], 0.0))) ``` table.to_pandas() works too, if you would rather go on in pandas. Index a score column with brackets rather than an attribute. A scorer's name becomes its column prefix, and a built-in names itself in kebab case, so intent-match_value is not a valid Python identifier. That is the per-row detail. For a run's average score, its per-scorer averages, and its cost and duration, use the dashboard, which computes those aggregates for you. ### The Evaluations dashboard The Chalk dashboard shows the same evaluations, runs and scores, and computes the aggregates for you. Open Evaluations in an environment's sidebar. It has two pages: Suites, for browsing evaluations and their runs, and Quick evaluation, for creating and running an evaluation without writing code. In an environment with no evaluations, Evaluations shows a Get started with evaluations checklist in place of the suites. Its steps are to create a deployment, create a dataset, define task and scorer functions, and create an evaluation, and each step is checked off once the environment has one. ### Suites and runs Suites lists the environment's suites. Open one to see its evaluations on the Definitions tab and their runs on the Runs tab. The Runs table gives each run's status, average score with the spread of its row scores, duration, LLM calls, tool calls, tokens, and estimated cost; Columns chooses which appear. Above the tabs, a chart plots the runs whose Plot box is checked in the Runs table. Performance compares their scores, and Token usage, Cost analysis, Execution time and Tool use compare the rest. Series picks the fields to plot, such as the average score and each scorer's score. Aggregate plots each field's average, and Percentiles plots its 5th, 25th, 50th, 75th and 95th percentiles across the run's rows. Right-click a run in the Runs table for two actions: - Set as baseline marks the run with a Baseline badge. An environment has at most one baseline run, so setting a new one clears the previous one. - Archive removes the run from the list. The Average score of a run or evaluation leaves out scorers whose values fall outside 0 to 1, such as a SQL expression scorer that returns a cost in dollars. The dashboard shows those scorers as plain numbers. ### Evaluation pages Click an evaluation to open its page. Run evaluation starts a new run, and the tabs show: - Runs: the evaluation's runs. - Scorers: each scorer's average score and the distribution of its scores in the evaluation's latest run. - Definition: the chalkcompute.Evaluation.create(...) call that recreates the evaluation on the same dataset revision, followed by the task function's source. By default the call refers to the task and function scorers by name, so it uses their current versions. Pinned versions switches it to the exact versions the evaluation used. Edit scorers adds, changes or removes scorers. Because an evaluation's scorers are fixed when it is created, saving creates a copy of the evaluation with the same dataset revision, task and suite. It can also rescore the latest succeeded run into the copy, which calls only the new and changed scorers. A scorer in the editor is an LLM judge, a Python function, or a Chalk SQL expression. Compare opens the evaluation in a table where you add other evaluations side by side. Each column is one evaluation, averaged over its succeeded runs. Alongside the scores, the table compares cost, tokens, duration, and LLM calls, and reads every value against the first column, where cheaper and faster count as improvements. A column whose dataset revision differs from the first column's is labeled "different dataset". ### Run pages Click a run to open it. A Details panel lists the run's evaluation, dataset revision, task version, timestamps, and result dataset, followed by a summary of its score, duration, and LLM and tool calls. The run page keeps the evaluation's Scorers and Definition tabs, and adds: - Runs: the run's result rows, as described in Reading results. Columns holding JSON objects, such as a structured task output or a scorer's metadata, are split into one column per field. - Summary: the run's average score, row count, wall time, total cost, tokens, LLM and tool calls, and median row time, then a timeline of when each row ran and a table of rows. Click a row to open its trace. - Sandboxes: any sandboxes the task started for the run, each linked to the result row it served. Wall time runs from when the run was created to when it completed, so it includes any wait before the run started. Duration, in the Details panel and the Runs table, covers only the time the run's work took. A run that is still in progress shows its rows as they arrive. The Rerun icon beside the run's title opens the evaluation's saved definition. Rerun evaluation starts another run of the same evaluation and opens its results. ### Quick evaluation Quick evaluation creates an evaluation and starts its first run in one step. Name it, pick a dataset, and choose a task: - LLM prompt: a prompt template filled from each row's columns, written as {{column}}, and sent to a model through the Model Gateway. The task is deployed as a function when the evaluation is created. - SQL expression: a Chalk SQL expression over the row's columns. Nothing is deployed or called. - Function: a function you have already deployed, called once per row. Its parameters must match dataset columns by name. Then pick scorers. The menu offers the deterministic built-ins, embedding similarity, and four LLM judges: Factuality, Closed QA, Answer relevance, and Custom rubric, which grades against labels you define. Custom scorers adds deployed functions and Chalk SQL expressions. Generated code shows the Python the page deploys. Create and run files the evaluation in the environment's Default suite and opens the run's results. # Evaluation Suites source: https://docs.chalk.ai/docs/evaluations/suites ## Group evaluations and their runs into named suites, and list what a suite holds. A suite is a named collection of evaluations and their runs, such as a release gate, a regression set, or one team's work. Suite names are unique in an environment, and EvaluationSuite.create raises EvaluationError for a name that is already taken. To reuse a suite across runs of the same script, look it up by name and create it only when it is missing: ``` import chalkcompute suites = chalkcompute.EvaluationSuite.list_all() suite = next((s for s in suites if s.name == "Release"), None) or chalkcompute.EvaluationSuite.create( "Release" ) for existing in suites: print(existing.id, existing.name) for evaluation in suite.evaluations(limit=20): print(evaluation.name) for run in suite.runs(limit=20): print(run.id, run.status) ``` suite.evaluations() and suite.runs() both yield newest first and page transparently. # Using EvaluationClient source: https://docs.chalk.ai/docs/evaluations/using-evaluationclient ## Hold one EvaluationClient across several calls, and handle the errors evaluation calls raise. The class methods on the other Evaluations pages, such as Evaluation.create and evaluation.run(), each construct a client for a single call. Hold an EvaluationClient when you make several calls, or when you want it to borrow credentials from a ConnectClient or a ChalkPy client you already have: ``` import chalkcompute client = chalkcompute.EvaluationClient(chalk_client=my_chalk_client) suite = client.create_suite("Release") evaluation = client.create("Nightly", dataset=dataset, task=answer, scorers=[exact_match]) run = client.run(evaluation.id, metadata={"git_sha": "abc123"}) for candidate in client.list(suite_id=suite.id, limit=50): print(candidate.name) for previous in client.list_runs(evaluation_id=evaluation.id, limit=10): print(previous.id, previous.status) ``` The client carries create_suite and list_suites for suites, create, get, and list for definitions, and run, get_run, list_runs, and rescore for runs. client.rescore(run_id, evaluation_id=...) is the call behind run.rescore(), and evaluation_id defaults to the run's own evaluation. list and list_runs are generators that page in batches of up to 100 and stop at limit. Pass only one of chalk_client or connect_client to the constructor. Every object a client returns keeps a reference to that client, which is what makes evaluation.run(), run.refresh(), and suite.evaluations() work. A handle you construct yourself has no client and raises EvaluationError on those methods, so fetch it through the client instead. ### Errors A failed request raises EvaluationError, which carries status_code, the HTTP equivalent of the underlying code. A missing evaluation or run raises EvaluationNotFoundError, a subclass of it: ``` import chalkcompute try: evaluation = chalkcompute.Evaluation.from_id("does-not-exist") except chalkcompute.EvaluationNotFoundError as e: print(e.status_code) # 404 ``` A run that fails or is canceled raises EvaluationRunFailedError out of wait(), with the terminal handle attached so you can read its status and error message. In this example, evaluation is an evaluation created as on the Creating and Running page: ``` try: run = evaluation.run().wait() except chalkcompute.EvaluationRunFailedError as e: print(e.run.status, e.run.error_message) ``` wait() raises TimeoutError when it runs out of time, as does a Trace read that never settles. Local validation raises ValueError or TypeError for an empty name, an empty scorer list, metadata that is not JSON-serializable, a score outside [0, 1], or a bare callable passed as a task or scorer. That validation runs before any request goes out, so a malformed call fails locally. # Functions source: https://docs.chalk.ai/docs/compute/functions ## Define and call remote functions that run on Chalk Compute. ### Overview A Chalk Function is a Python callable that you deploy onto a Scaling Group and register in the Chalk Catalog. Decorate a function with @chalkcompute.function and it becomes reachable from anywhere Chalk runs: local scripts and notebooks, other functions, and (because of the Catalog registration) the feature engine, including Chalk SQL, resolvers, the query server, and DataFrames via F.catalog_call(). Arguments and results are serialized automatically; you write ordinary Python and Chalk handles scheduling, scaling, and delivery. On top of what a bare Scaling Group provides, functions add async / streaming invocation, nested calls, automatic retries, concurrency caps, rate limits, and request-queue-driven autoscaling. Reach for a function when you want any of those features, or when you want the callable to be visible to the rest of Chalk. Functions support three return styles: - Single value: return a result and the caller receives it. - Async generator: yield results incrementally and the caller receives them as they're produced. See Streaming results for an end-to-end example. - Nested calls: a function can call other functions, composing pipelines without manual orchestration. See Nested function calls for a pipeline example. To define and deploy a function, see Defining a function. For jobs with multiple steps and separate task environments, see Workflows. ### How it works @chalkcompute.function is backed by Chalk Compute's async execution layer. At a high level, calling a function goes through three moving parts: ``` ┌──────────┐ ┌──────────────────┐ ┌───────────┐ │ Caller │──call──▸│ Function Queue │──run───▸│ Worker │ │ │◂results─│ │◂results─│ (Python) │ └──────────┘ └──────────────────┘ └───────────┘ ``` - The caller (local script, notebook, another function) submits an invocation and receives results over a single bidirectional stream. - The Function Queue durably accepts the work, assigns each call an ID, enforces configured policies (concurrency, rate limit, retry), and relays results back to the caller. - The worker is the Python process that actually runs your function code. Callers and workers are loosely coupled: the caller never opens a connection directly to a worker. Workers can scale up, scale down, or get rescheduled mid-flight and in-flight calls are not lost. This is what enables streaming async generators, nested calls, and policy-enforced dispatch. ### Policy enforcement The queue enforces the policies declared on each function (rate limit, concurrency, and retries) before dispatching work to the worker. | Policy | What it caps | | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`rate_limit`](/docs/functions/rate-and-concurrency-limits#rate-limiting) | Outbound requests per second / minute / hour, optionally shared across functions via a `key`. | | [`concurrency`](/docs/functions/rate-and-concurrency-limits#concurrency-limits) | In-flight invocations at once, optionally shared across functions via a `key`. Defaults to `1` per function. | | [`retries`](/docs/functions/retries) | Automatic retry of transient failures (`UNAVAILABLE`, `DEADLINE_EXCEEDED`, `RESOURCE_EXHAUSTED`, `ABORTED`) with exponential backoff and jitter. Other errors are terminal. | Concurrency is acquired before dispatching, so a burst of queued calls backpressures on the semaphore rather than spawning unbounded tasks. Rate-limit tokens are consumed per attempt: each retry takes a fresh token. ### Retry safety Retries are safe for failures that happen before the function has started streaming results: connection errors, DEADLINE_EXCEEDED on the initial call, etc. If a function has already yielded chunks when it fails, those chunks are visible to the caller and a retry would produce an ambiguous stream. Retries therefore apply cleanly only to pre-stream failures today; making mid-stream retries safe is a known follow-up. # Defining a Function source: https://docs.chalk.ai/docs/functions/defining-a-function ## Define a Chalk Function with the @chalkcompute.function decorator, call it, and ship the helpers it imports. To define a Chalk Function, decorate a Python callable with @chalkcompute.function: ``` import chalkcompute @chalkcompute.function def add(x: int, y: int) -> int: return x + y if __name__ == "__main__": result = add.remote(3, 5) print(result) # 8 ``` Deploy by running the script: the decorator registers the function with Chalk as soon as it runs, and .remote(...) waits for that deployment to be ready before invoking it. add(3, 5) runs the Python body locally instead; use .remote(...) to call the deployed version. To call an already-deployed function from a different script — without redeploying — attach to it by name with RemoteFunction.from_name: ``` import chalkcompute fn = chalkcompute.RemoteFunction.from_name("add") result = fn.remote(3, 5) print(result) # 8 ``` ### Function dependencies in external files A deployed function often calls helpers defined in other files. Whether a helper resolves remotely depends on where it lives relative to the decorated function (the function under @chalkcompute.function): - Helper in the decorated function's own package: the function's entire top-level package is shipped automatically, so imports of sibling modules resolve remotely with no extra configuration. - Helper anywhere else (a standalone module, or a different package): must be declared on the image with Image.add_local_python_source. The same applies to all file-external helpers when the decorated function is defined in a standalone script rather than a package. ### Same package: automatic If the decorated function's module runs as part of a package (imported from an entrypoint, or executed directly with python -m), Chalk ships the whole top-level package in the deploy image: ``` # mypkg/helpers.py def normalize(text: str) -> str: return text.strip().lower() ``` ``` # mypkg/pipeline.py import chalkcompute from mypkg.helpers import normalize @chalkcompute.function def clean(text: str) -> str: return normalize(text) ``` ``` # main.py (outside mypkg/) from mypkg.pipeline import clean print(clean.remote(" Hello ")) ``` ``` uv run python main.py ``` Running the entrypoint (or the function's own module with uv run python -m mypkg.pipeline) deploys clean with all of mypkg, so normalize is importable remotely exactly as it is locally. Running the file by path (uv run python mypkg/pipeline.py) ships only that one file because Python doesn't know that it belongs to a package. In this scenario, Chalk will raise an error at deploy time. Every run of an entrypoint that imports the decorated function deploys a new function version. To call an existing deployment without redeploying, attach to it by name with RemoteFunction.from_name instead. ``` # call_clean.py: call the deployed function without triggering a redeploy import chalkcompute fn = chalkcompute.RemoteFunction.from_name("clean") print(fn.remote(" Hello ")) ``` ``` uv run python call_clean.py ``` ### Everything else: add_local_python_source When the decorated function is defined in a standalone script, or imports helpers from outside its own top-level package, list those local modules or packages on the image: ``` # helpers.py (next to deploy.py) def normalize(text: str) -> str: return text.strip().lower() ``` ``` # deploy.py import chalkcompute from helpers import normalize @chalkcompute.function( image=chalkcompute.Image.debian_slim().add_local_python_source("helpers"), ) def clean(text: str) -> str: return normalize(text) ``` ``` uv run python deploy.py ``` add_local_python_source accepts any number of top-level module or package names. Each must be importable on your local Python path; packages are copied whole (excluding __pycache__). Dotted names, namespace packages spanning multiple directories, and compiled extensions are rejected with a ValueError. This ships your own source code only. Third-party dependencies still need to be installed in the image with pip_install or run_commands. # Configuring Functions source: https://docs.chalk.ai/docs/functions/configuring-functions ## Combine settings in the @chalkcompute.function decorator, inject secrets, mount volumes, and find the settings functions share with other workloads. This page covers the configuration specific to functions. These settings work the same way across workloads and have their own pages: - Resources and GPUs - Workload Identity - Tags - Autoscaling - Readiness and Shutdown - Authentication - Tracing Policy ### Decorator arguments The decorator accepts a wide set of arguments that control the compute environment a function runs in: image, CPU/memory/GPU, secrets, volumes, batching, retries, rate limits, concurrency, autoscaling, and tracing. A realistic decorator usually combines several of them: ``` import chalkcompute @chalkcompute.function( image=chalkcompute.Image.base("python:3.12-slim").run_commands( "pip install sentence-transformers", ), cpu="2", memory="4Gi", gpu="nvidia-l4", secrets=[chalkcompute.Secret.from_chalk_env("HF_TOKEN")], max_batching_size=25, max_buffer_duration=2000, retries=3, ) def embed(texts: list[str]) -> list[list[float]]: from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") return model.encode(texts).tolist() ``` Each argument is covered on its own page or in a section below: the shared settings listed above, Batching, Retries, Rate and Concurrency Limits, Secrets, and Volumes. For the full per-argument reference, types, and defaults, see @chalkcompute.function in the API reference. ### Secrets Inject secrets into a function using the secrets parameter. Secrets are resolved at deploy time and made available as environment variables inside the container: ``` import chalkcompute from chalkcompute import Secret @chalkcompute.function( secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_integration("prod_postgres"), ], ) def call_api(prompt: str) -> str: import os openai_api_key = os.environ["OPENAI_API_KEY"] database_url = os.environ["DATABASE_URL"] ... ``` See Secrets for the full Secret API. ### Volumes Attach persistent storage to a function with the volumes parameter. Each entry is a (name, mount_path) string tuple. Inside the function, read and write the volume as an ordinary filesystem path: ``` import chalkcompute @chalkcompute.function( volumes=[("model-cache", "/models")], gpu="nvidia-l4", ) def predict(text: str) -> list[float]: import os import json from sentence_transformers import SentenceTransformer cache_path = "/models/all-MiniLM-L6-v2" if os.path.exists(cache_path): model = SentenceTransformer(cache_path) else: model = SentenceTransformer("all-MiniLM-L6-v2") model.save(cache_path) return model.encode(text).tolist() ``` Volumes persist across invocations: data written by one call is available to subsequent calls. This is useful for caching large model weights, storing intermediate results, or sharing data between functions that mount the same volume. # Retries source: https://docs.chalk.ai/docs/functions/retries ## Retry a Chalk Function automatically on failure with a retry count or a RetryPolicy. Functions can automatically retry on failure. Pass a simple integer for max-attempts with default exponential backoff, or use RetryPolicy for full control: ``` import chalkcompute from chalkcompute import RetryPolicy # Simple: retry up to 3 times with exponential backoff @chalkcompute.function(retries=3) def flaky_api(query: str) -> str: ... # Exponential backoff with custom parameters @chalkcompute.function( retries=RetryPolicy.exponential( attempts=5, initial=0.5, max_wait=30.0, ), ) def resilient_call(payload: str) -> str: ... # Fixed delay, only retry on specific exceptions @chalkcompute.function( retries=RetryPolicy.linear( attempts=3, delay=2.0, retry_on=(TimeoutError, ConnectionError), ), ) def network_call(url: str) -> str: ... ``` ### RetryPolicy.exponential() Wait times grow geometrically: initial, initial * multiplier, initial * multiplier², ..., capped at max_wait: | Parameter | Type | Default | Description | | ------------ | ------------------ | -------------- | ---------------------------------- | | `attempts` | `int` | `3` | Max retry attempts | | `initial` | `float` | `1.0` | Base wait time in seconds | | `multiplier` | `float` | `2.0` | Backoff multiplier | | `max_wait` | `float` | `60.0` | Upper bound on wait time | | `jitter` | `bool` | `True` | Add random jitter | | `retry_on` | `tuple[type, ...]` | `(Exception,)` | Exception types that trigger retry | ### RetryPolicy.linear() Fixed delay between retries: | Parameter | Type | Default | Description | | ---------- | ------------------ | -------------- | ---------------------------------- | | `attempts` | `int` | `3` | Max retry attempts | | `delay` | `float` | `1.0` | Fixed wait in seconds | | `jitter` | `bool` | `False` | Add random jitter | | `retry_on` | `tuple[type, ...]` | `(Exception,)` | Exception types that trigger retry | ### Exception filtering By default, all Exception subclasses trigger a retry. Pass a tuple of exception classes to retry_on to restrict which failures are retried: ``` RetryPolicy.exponential( attempts=4, retry_on=(ConnectionError, TimeoutError, OSError), ) ``` Non-matching exceptions propagate immediately without consuming retry budget. # Rate and Concurrency Limits source: https://docs.chalk.ai/docs/functions/rate-and-concurrency-limits ## Cap how often a Chalk Function is called and how many of its calls run at once. Two settings limit how a Chalk Function is called: rate_limit caps requests over time, and concurrency caps how many calls run at once. ### Rate limiting Cap the outbound request rate of a function using rate_limit. Pass an integer for a simple "N per second" cap, or a RateLimitPolicy for full control: ``` import chalkcompute from chalkcompute import RateLimitPolicy # Simple: up to 100 requests per second @chalkcompute.function(rate_limit=100) def call_provider(query: str) -> str: ... # Full control: rate + time unit + shared-bucket key @chalkcompute.function( rate_limit=RateLimitPolicy(rate=1000, per="minute", key="provider-api"), ) def call_other_endpoint(query: str) -> str: ... ``` ### RateLimitPolicy | Parameter | Type | Default | Description | | --------- | ----- | ------------- | ---------------------------------------------- | | `rate` | `int` | *required* | Maximum requests per `per` unit | | `per` | `str` | `"second"` | Time unit: `"second"`, `"minute"`, or `"hour"` | | `key` | `str` | function name | Shared-bucket identifier (see below) | ### Sharing a bucket across functions Two functions declaring the same key share a single rate-limit bucket. This is useful when multiple functions call the same downstream service and the service imposes a limit across all callers: ``` shared = RateLimitPolicy(rate=1000, per="second", key="provider-api") @chalkcompute.function(rate_limit=shared) def lookup(query: str) -> str: ... @chalkcompute.function(rate_limit=shared) def enrich(record: dict) -> dict: ... # Calls to both functions count against the same 1000/s budget. ``` If key is omitted, each function gets its own bucket keyed by the function name, so rate limits are per-function by default. ### Concurrency limits Cap the in-flight count of a function using concurrency. This is distinct from rate_limit: rate limits gate requests-over-time, concurrency gates how many calls are executing simultaneously: ``` import chalkcompute from chalkcompute import ConcurrencyPolicy # Simple: at most 10 calls in flight at once @chalkcompute.function(concurrency=10) def gpu_infer(text: str) -> list[float]: ... # Shared gate across multiple functions hitting the same resource gpu_pool = ConcurrencyPolicy(max_concurrent=4, key="gpu-pool") @chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4") def embed(text: str) -> list[float]: ... @chalkcompute.function(concurrency=gpu_pool, gpu="nvidia-l4") def rerank(query: str, docs: list[str]) -> list[int]: ... # Only 4 calls across both functions run simultaneously. ``` ### ConcurrencyPolicy | Parameter | Type | Default | Description | | ---------------- | ----- | ------------- | ------------------------------------- | | `max_concurrent` | `int` | *required* | Maximum in-flight handler invocations | | `key` | `str` | function name | Shared-bucket identifier | When concurrency is not specified, the function queue consumer applies a default of max_concurrent=1 per function, so calls to a single function are serialized. Set concurrency=N explicitly to allow parallel execution. ### Rate limit vs concurrency Both can be set independently and they interact: | Policy | Controls | Example use case | | ------------- | ------------------------ | ------------------------------------ | | `rate_limit` | Requests per unit time | "1000 calls/sec across all replicas" | | `concurrency` | Parallel in-flight count | "4 GPU slots, don't exceed memory" | # Batching source: https://docs.chalk.ai/docs/functions/batching ## Have a Chalk Function receive incoming items in batches with max_buffer_duration and max_batching_size. When your handler is more efficient processing multiple items at once (e.g. GPU batch inference), enable batching with max_buffer_duration and/or max_batching_size. Chalk will accumulate incoming items and invoke the handler in chunks: ``` import chalkcompute @chalkcompute.function( max_buffer_duration=1000, # buffer up to 1000 ms max_batching_size=32, # or up to 32 items gpu="nvidia-l4", ) def batch_embed(texts: list[str]) -> list[list[float]]: from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") return model.encode(texts).tolist() ``` When batching is enabled, handler arguments are delivered as lists or a single pyarrow table. Defaults apply when either parameter is set: | Parameter | Default (when batching enabled) | | --------------------- | ------------------------------- | | `max_buffer_duration` | `1000` ms | | `max_batching_size` | `10` items | # Scheduling source: https://docs.chalk.ai/docs/functions/scheduling ## Run a Chalk Function on a recurring schedule with a crontab expression or a duration string. Some jobs run on a clock instead of in response to a request: refreshing a table of exchange rates, recomputing a daily rollup, warming a cache, syncing an external source. A scheduled function is a Function that Chalk invokes for you on a recurring schedule: declare a schedule in the decorator, and Chalk runs it at the cadence you specify, with the image, resources, secrets, and retry behavior you configured. There is no separate cron server to operate. ``` import chalkcompute @chalkcompute.function( schedule="0 * * * *", # every hour, on the hour secrets=[chalkcompute.Secret.from_chalk_env("EXCHANGE_API_KEY")], cpu="1", memory="2Gi", retries=3, ) def refresh_exchange_rates() -> None: import os import requests resp = requests.get( "https://api.example.com/v1/rates", params={"base": "USD"}, headers={"Authorization": f"Bearer {os.environ['EXCHANGE_API_KEY']}"}, ) rates = resp.json()["rates"] # write the latest rates to your store ... ``` schedule and retries govern how and when the platform invokes the function; everything else configures the Scaling Group that hosts it, exactly as it would for an on-demand function. Scheduled jobs often do heavier work, such as nightly batch inference or periodic re-scoring of a dataset, and can request a GPU the same way: ``` @chalkcompute.function( schedule="0 3 * * *", # daily at 03:00 UTC gpu="nvidia-l4", cpu="4", memory="16Gi", ) def nightly_batch_inference() -> None: ... ``` ### Schedule syntax schedule accepts either a crontab expression or a Chalk duration string. Use whichever reads more clearly for your cadence. All schedules are evaluated in UTC. Crontab: five space-separated fields (minute hour day-of-month month day-of-week). Standard syntax is supported: *, lists (1,15), ranges (1-5), and steps (*/10). | Expression | Runs | | ---------------- | ------------------------- | | `"*/10 * * * *"` | every 10 minutes | | `"0 * * * *"` | every hour, on the hour | | `"0 2 * * *"` | every day at 02:00 UTC | | `"0 0 * * 0"` | every Sunday at 00:00 UTC | Duration shorthand: a single interval, for simple cadences. Supported units are minutes (≤ 60), hours (≤ 24), and days (= 1), written as one value: | Shorthand | Runs | Equivalent crontab | | --------- | ---------------- | ------------------ | | `"10m"` | every 10 minutes | `*/10 * * * *` | | `"6h"` | every 6 hours | `0 */6 * * *` | | `"1d"` | every day | `0 0 * * *` | Shorthand intervals are aligned to clock boundaries, not to your deployment time: "6h" runs at 00:00, 06:00, 12:00, and 18:00 UTC. Anything longer or compound ("90m", "1h30m", "2d") must be written as a crontab expression. ``` @chalkcompute.function(schedule="0 2 * * *") # nightly at 02:00 UTC def nightly_rollup() -> None: ... @chalkcompute.function(schedule="15m") # every 15 minutes def warm_cache() -> None: ... ``` ### Rules for scheduled functions Because a scheduled function is invoked by the platform rather than a caller, two constraints apply and are enforced when the function is defined: - No inputs. A scheduled function must take no arguments. Have it read its own inputs from your sources. - Not a generator. A scheduled run has no caller to consume yielded values, so a scheduled function cannot yield. Return None (or a value) instead. Everything else about a Compute Function still applies: custom images, resource requests, secrets, and retries all work exactly as they do for on-demand functions. ### Observing runs Each invocation appears in the Chalk dashboard with its status, logs, and resource usage, just like a manually invoked function. Emit logs from the body to make each run observable: ``` from chalk import chalk_logger @chalkcompute.function(schedule="1h") def refresh_exchange_rates() -> None: chalk_logger.info("refresh started") rates = ... # fetch the latest rates and write them to your store chalk_logger.info("refreshed %s rates", len(rates)) ``` # Calling and Fan-Out source: https://docs.chalk.ai/docs/functions/calling-and-fan-out ## Call deployed Chalk Functions with defer() and get(), fan out with gather(), compose nested calls, and handle errors. This page covers deferred calls, fan-out, and nested calls to a deployed Chalk Function, and how its errors reach the caller. ### Deferred calls with defer() and get() .remote(...) blocks for the entire round trip. Use .defer(...) instead when you want to submit a call and collect its result later; it enqueues the call and returns a FunctionCallHandle immediately, without waiting for the function to run. This example calls the add function from Defining a function: ``` import chalkcompute add = chalkcompute.RemoteFunction.from_name("add") handle = add.defer(3, 5) # ...do other work while the call runs... result = handle.get() print(result) # 8 ``` handle.get(timeout=None) blocks until the result is ready and returns the same value .remote(...) would have returned. Each handle exposes the server-assigned call_id it was enqueued under: ``` handle = add.defer(3, 5) print(handle.call_id) ``` ### Fan-out and gather() Because .defer(...) returns immediately, you can enqueue many calls up front and collect their results afterward. Use chalkcompute.gather(...) to wait for a batch of handles concurrently and get their results back in order: ``` import chalkcompute handles = [classify.defer(text) for text in texts] results = chalkcompute.gather(handles) ``` gather(...) waits for the handles concurrently rather than one at a time, so it returns as soon as the slowest call finishes. Pass timeout to bound the whole wait: ``` results = chalkcompute.gather(handles, timeout=30.0) ``` Inside a running event loop, gather(...) returns an awaitable instead of blocking: await chalkcompute.gather(handles). Each deferred call is still subject to the function's own concurrency and rate limit policies, so fanning out this way doesn't bypass those caps. ### Timeouts Pass timeout (seconds) to bound how long get() waits: ``` try: result = handle.get(timeout=5.0) except TimeoutError: print("still running") ``` A timeout only stops waiting locally. It does not cancel the underlying call, which keeps running and can still be collected with another get(). ### Errors from deferred calls If the function raised an exception, .get() raises chalkcompute.FunctionError describing the failure. Transient infrastructure failures, such as a dropped connection between the queue and the worker, are retried automatically before get() ever sees them; only failures from the function body itself, or infra failures that exhaust retries, are raised to the caller. ### Nested function calls A function can call other functions with .remote(...). Each inner call is dispatched independently, so you can compose pipelines where each step runs on its own infrastructure. This example calls the transcribe async generator from Streaming results: ``` import chalkcompute @chalkcompute.function def translate(text: str, target_lang: str) -> str: from transformers import pipeline translator = pipeline("translation", model=f"Helsinki-NLP/opus-mt-en-{target_lang}") return translator(text)[0]["translation_text"] @chalkcompute.function async def transcribe_and_translate(audio_uri: str, target_lang: str): """Transcribe audio, then translate each segment.""" async for segment in transcribe.remote(audio_uri): translated = await translate.remote(segment["text"], target_lang) yield { **segment, "translated": translated, } ``` The caller sees a single stream of translated segments: ``` import asyncio async def main(): async for seg in transcribe_and_translate.remote( "https://example.com/meeting.wav", target_lang="es", ): print(f"[{seg['start']:.1f}s] {seg['translated']}") asyncio.run(main()) ``` ``` [0.0s] Bienvenidos a la revisión trimestral. [3.2s] Comencemos con las cifras de ingresos. [6.1s] El tercer trimestre llegó a 4,2 millones, un 18 por ciento más. ``` The outer function composes two inner calls: transcribe (async generator) and translate (single-value). Each runs on its own compute configuration. ### Error handling If a function raises an exception, the caller receives it as a RuntimeError with the original exception type and message: ``` import chalkcompute @chalkcompute.function def divide(a: float, b: float) -> float: return a / b try: divide.remote(1.0, 0.0) except RuntimeError as e: print(e) # ZeroDivisionError: division by zero ``` For async generators, an exception mid-stream terminates the iterator. Any segments already yielded are still available to the caller; only subsequent iteration raises: ``` import chalkcompute @chalkcompute.function async def fragile_gen(): yield "ok-1" yield "ok-2" raise ValueError("something went wrong") import asyncio async def main(): async for item in fragile_gen.remote(): print(item) asyncio.run(main()) # ok-1 # ok-2 # RuntimeError: ValueError: something went wrong ``` # Streaming Results source: https://docs.chalk.ai/docs/functions/streaming-results ## Stream results from a Chalk Function to the caller with an async generator. If your function is an async generator (uses yield), the caller receives each value as soon as the worker produces it, without waiting for the generator to finish. This makes functions a good fit for: - Streaming LLM token output. - Incremental ETL where downstream consumers start processing early. - Long-running jobs that emit progress events. ### Example: audio transcription Transcribe an audio file and yield each segment as it's decoded: ``` import chalkcompute @chalkcompute.function( image=chalkcompute.Image.base("python:3.12-slim").run_commands( "pip install faster-whisper httpx", ), cpu="4", memory="8Gi", ) async def transcribe(audio_uri: str): import httpx from faster_whisper import WhisperModel async with httpx.AsyncClient() as client: resp = await client.get(audio_uri) resp.raise_for_status() audio_path = "/tmp/audio" with open(audio_path, "wb") as f: f.write(resp.content) model = WhisperModel("base.en", compute_type="int8") segments, _info = model.transcribe(audio_path) for segment in segments: yield { "start": segment.start, "end": segment.end, "text": segment.text, } ``` The caller iterates over segments as they arrive: ``` import asyncio async def main(): async for segment in transcribe.remote("https://example.com/meeting-recording.wav"): print(f"[{segment['start']:.1f}s] {segment['text']}") asyncio.run(main()) ``` Output streams in real time: ``` [0.0s] Welcome to the quarterly review. [3.2s] Let's start with the revenue numbers. [6.1s] Q3 came in at 4.2 million, up 18 percent. ... ``` The caller receives the first segment as soon as it's decoded. It does not wait for the full file to finish processing. For a 30-minute recording, the first segment may arrive in seconds. # Calling from a DataFrame source: https://docs.chalk.ai/docs/functions/calling-from-a-dataframe ## Call a Chalk Function on every row of a dataset with F.catalog_call() in a static resolver. You can invoke a function across every row of a dataset using F.catalog_call() inside a static resolver's DataFrame pipeline. Chalk handles parallelism, batching, and delivery automatically. Unlike .remote() or .defer(), F.catalog_call() only takes effect once the resolver is applied to your environment with chalk apply. It's evaluated during chalk query (or an offline query), so there's no way to call it from a plain script. For example, define a function that extracts named entities from text: ``` import chalkcompute @chalkcompute.function( image=chalkcompute.Image.base("python:3.12-slim").run_commands( "pip install spacy && python -m spacy download en_core_web_sm", ), cpu="2", memory="4Gi", ) def extract_entities(memo: str) -> str: import json import spacy nlp = spacy.load("en_core_web_sm") doc = nlp(memo) return json.dumps([ {"text": ent.text, "label": ent.label_} for ent in doc.ents ]) ``` Then apply it across a parquet file of transaction memos with an @offline(static=True) resolver: ``` import chalkdf from chalk import offline from chalk.features import DataFrame, features, _ import chalk.functions as F @features class Transaction: id: int memo: str entities: str @offline(static=True) def ingest_transaction_entities() -> DataFrame[ Transaction.id, Transaction.memo, Transaction.entities, ]: return ( chalkdf.DataFrame.scan(["transactions.parquet"]) .with_columns({ "transaction.entities": F.catalog_call("function.extract_entities", _.memo), }) .select("transaction.id", "transaction.memo", "transaction.entities") ) ``` Each row's memo column is passed to extract_entities, and the result is written as the entities feature alongside it. .with_columns() keeps every existing column and adds the ones you name; use .project() instead when you want only the columns you specify. Note the qualified name passed to catalog_call: it's function.extract_entities, not a bare function name. model.* and function.* are the two valid namespaces. # Managing Functions source: https://docs.chalk.ai/docs/functions/managing-functions ## List, inspect, and delete deployed Chalk Functions with the chalk CLI and the chalkcompute SDK. This page covers listing, inspecting, and deleting deployed functions. The examples use the add function from Defining a function. ### Listing ``` $ chalk function list ``` Displays a table of the functions in your environment with their ID, name, current version, latest version, scaling group, status, and creation time. Pass --name to filter by function name, or --limit to cap the number of results: ``` $ chalk function list --name=add ``` ### Inspecting ``` $ chalk function get --name=add ``` Shows the function's ID, name, creation and update times, current version ID and number, scaling group, scaling-group revision, scaling-group status, and input and output schemas. Pass --id instead of --name to look a function up by its ID. Each deploy creates a new immutable version. To list a function's versions, or to show one of them: ``` $ chalk function versions list --name=add $ chalk function versions get --name=add --version=1 ``` To see calls submitted through the function queue, such as .defer(...) calls, use chalk function calls. count shows how many calls were submitted in the past hour, and recent lists the most recent calls with their status, enqueue time, and trace ID: ``` $ chalk function calls count --name=add $ chalk function calls recent --name=add --limit=50 ``` From the SDK, attach to a function by name and call refresh() to fetch the current version's details, or versions() to list its versions: ``` import chalkcompute fn = chalkcompute.RemoteFunction.from_name("add") info = fn.refresh() print(info.function_name) print(info.version) print(info.is_current) print(info.scaling_group_name) print(fn.web_url) for v in fn.versions(): print(v.version, v.id, v.created_at, v.is_current) ``` ### Deleting Delete a function and all of its versions through its handle: ``` import chalkcompute chalkcompute.RemoteFunction.from_name("add").delete() ``` From the CLI: ``` $ chalk function delete --name=add ``` Both tear down the Scaling Group behind each version, including that group's Kubernetes Deployment, Service, and HTTPRoute, along with the function's schedule and its Chalk Catalog entry. The underlying records are soft-deleted, so the function still appears in history. Calling delete() on a function that is already gone succeeds without raising. # Function Queue Setup source: https://docs.chalk.ai/docs/functions/function-queue-setup ## Enable dataplane Redis and add the function-queue service to set up the function queue on your Chalk cluster. This page sets up the function queue on your own Chalk cluster. The function queue is optional; skip this section if you don't need it. ### Enable dataplane Redis The function queue requires a dataplane Redis instance. In the team navigation click on Clusters, select the cluster, and open the Settings tab. Under Dataplane Redis, set the Redis mode to Managed. CPU and memory can be adjusted under Advanced if needed; the defaults are 1 CPU and 10Gi memory. Click Save. ### Add the function queue service In the environment sidebar, go to Infrastructure → Resource Configuration and add a function-queue service entry in the Default resource group. A reasonable starting configuration would be: | Resource | Value | | -------- | ----- | | CPU | 1 | | Memory | 1Gi | | Replicas | 1 | # Scaling Groups source: https://docs.chalk.ai/docs/compute/scaling-groups ## Deploy autoscaling HTTP services with routing. A scaling group is a replicated, HTTP-fronted service managed by Chalk. Each group is backed by a Kubernetes Deployment with rolling updates, automatic service discovery, and autoscaling between configurable min and max replica counts, driven by CPU or GPU utilization or by schedules you set. A group can also scale to zero replicas and start again when a request arrives. Use scaling groups for ML inference servers, internal APIs, agent backends, and any long-lived service that needs to be reachable from outside the cluster. For workloads that don't need replication or HTTP fronting, use a Sandbox instead. For serverless, function-shaped invocations, use Functions. Workflows provision scaling groups for Python tasks, with separate images and dependencies for each task. Workflow workers scale up while they have open runs and return to zero when idle. For registered models served through Chalk's Arrow inference contract, use the Model Deployments APIs. They expose typed deployment objects, readiness waiting, revision history, updates, and rollback. You can drive scaling groups two ways: the Python SDK (chalkcompute.ScalingGroup) or the chalk scaling-group CLI. Both surfaces target the same underlying service, so groups created one way are visible and manageable from the other. For every CLI flag, see the chalk scaling-group create reference. ### Quick start (SDK) Save a small Flask app as app.py: ``` from flask import Flask app = Flask(__name__) @app.get("/health") def health(): return {"ok": True} app.run(host="0.0.0.0", port=8080) ``` Then deploy it from a script in the same directory: ``` import chalkcompute img = ( chalkcompute.Image.debian_slim("3.12") .pip_install(["flask"]) .add_local_file("./app.py", "/app/app.py") .entrypoint(["python", "/app/app.py"]) ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, min_replicas=1, max_replicas=3, startup_probe=chalkcompute.StartupProbe.http("/health"), ).deploy() resp = sg.call("/health", method="GET") print(resp.status, resp.text()) sg.delete() ``` ScalingGroup(...).deploy() builds the image, uploads any local files declared via add_local_file / add_local_dir, creates the scaling group, and waits until at least one replica is ready and available. With startup_probe set, deploy() also waits until the group's web_url responds at /health. ### Quick start (CLI) ``` $ chalk scaling-group create \ --image=my-registry/my-inference-server:v1 \ --name=inference \ --max-replicas=1 \ --port=8080 ``` This creates a Kubernetes Deployment with a single replica running your image, a headless Service, and an HTTPRoute that exposes the service over HTTPS. You can tune replicas and resource limits at creation time: ``` $ chalk scaling-group create \ --image=my-registry/my-inference-server:v1 \ --name=inference \ --port=8080 \ --min-replicas=3 \ --max-replicas=3 \ --cpu=4 \ --memory=8Gi ``` ### Full example A GPU-accelerated inference service that loads model weights from a Chalk Volume, with custom resource limits, multiple replicas, autoscaling, and tags: ``` $ chalk scaling-group create \ --image=my-registry/llm-serving:v3 \ --name=llm-inference \ --port=8080 \ --min-replicas=2 \ --max-replicas=4 \ --gpu=nvidia-l4:1 \ --cpu=8 \ --memory=32Gi \ --volume=chalk://llama3-weights=/models/llama3 \ --tags="model=llama3,team=ml-platform" \ --entrypoint="python,-m,vllm.entrypoints.openai.api_server,--model,/models/llama3" ``` my-registry/llm-serving:v3 stands for your own serving image, which must have vLLM installed. chalk://llama3-weights stands for a Chalk Volume holding the model weights; create it with chalk volume create and upload the weights with chalk volume cp --recursive. After creation, the service is reachable at its assigned DNS name and can be referenced by resolvers or other services running in the same Chalk environment. ### When to use a Scaling Group vs other compute primitives See Choosing the right primitive on the Compute overview for the full decision table. # Configuring Scaling Groups source: https://docs.chalk.ai/docs/scaling-groups/configuring-scaling-groups ## Pass environment variables, secrets, and volumes to a scaling group, and find the settings it shares with other workloads. This page covers the configuration specific to scaling groups. These settings work the same way across workloads and have their own pages: - Resources and GPUs - Workload Identity - Tags - Autoscaling - Readiness and Shutdown - Authentication - Tracing Policy ### Environment, secrets, and volumes These use the same environment, secret, and volume configuration patterns as sandboxes. The examples below inject an environment variable, a secret, and a volume. Create the secret and the volume first: ``` $ chalk secret set EXAMPLE_API_TOKEN=replace-with-your-token $ chalk volume create chalk://example-api-data ``` Then pass them from the SDK. This example uses the app.py file from the Quick start, saved in the same directory: ``` import chalkcompute img = ( chalkcompute.Image.debian_slim("3.12") .pip_install(["flask"]) .add_local_file("./app.py", "/app/app.py") .entrypoint(["python", "/app/app.py"]) ) sg = chalkcompute.ScalingGroup( image=img, name="api", port=8080, env={"LOG_LEVEL": "INFO"}, secrets=[chalkcompute.Secret.from_chalk_env("EXAMPLE_API_TOKEN")], volumes=[("example-api-data", "/data")], min_replicas=1, max_replicas=3, ).deploy() ``` Or from the CLI. This example serves the volume's files with Python's built-in HTTP server: ``` $ chalk scaling-group create \ --image=python:3.12-slim \ --name=api \ --port=8080 \ --max-replicas=3 \ --env=LOG_LEVEL=INFO \ --secret=EXAMPLE_API_TOKEN \ --volume=chalk://example-api-data=/data \ --entrypoint="python,-m,http.server,8080,--directory,/data" ``` Repeat --env, --secret, and --volume to pass more than one. --env-file reads KEY=VALUE lines from a file. --secret also accepts integration:NAME[:KEY1,KEY2] to inject the secrets of a Chalk integration. Without =/mount/path, --volume mounts the volume at /mnt/. Secrets are resolved at deploy time and injected as environment variables in every replica. Volume mounts are shared across all replicas, so use them for read-mostly data — if multiple replicas write to the same volume path concurrently, the last writer wins on the next sync. # Entrypoint and Protocol source: https://docs.chalk.ai/docs/scaling-groups/entrypoint-and-protocol ## Override a scaling group's container entrypoint, and serve gRPC instead of HTTP. This page covers the command a scaling group's replicas run, and the protocol its route forwards. ### Entrypoint override To override the image's default entrypoint, pass entrypoint from the SDK or --entrypoint from the CLI. These examples run Python's built-in HTTP server on the public python:3.12-slim image: ``` import chalkcompute sg = chalkcompute.ScalingGroup( image="python:3.12-slim", name="custom-server", port=8000, entrypoint=["python", "-m", "http.server", "8000"], ).deploy() ``` ``` $ chalk scaling-group create \ --image=python:3.12-slim \ --name=custom-server \ --max-replicas=1 \ --port=8000 \ --entrypoint="python,-m,http.server,8000" ``` The CLI takes comma-separated arguments; the SDK takes a list. ### gRPC services To serve gRPC instead of HTTP, set protocol="grpc" from the SDK or pass --grpc from the CLI. The group's route then forwards gRPC traffic to your container's port. # DNS and Routing source: https://docs.chalk.ai/docs/scaling-groups/dns-and-routing ## Find a scaling group's DNS hostname and send requests to it from the SDK. Every scaling group is automatically assigned a TLS-terminated DNS hostname through the cluster's gateway. The pattern is: ``` -. ``` For example, a scaling group named inference in environment abc123 with gateway domain gw.chalk.ai would be reachable at: ``` https://abc123-inference.gw.chalk.ai ``` In the SDK examples on this page, sg is a deployed group, such as the one from the Quick start. From the SDK, read the hostname off the deployed group: ``` print(sg.web_url) # https://abc123-inference.gw.chalk.ai ``` From the CLI, the URL is shown as Web URL in the output of chalk scaling-group get and chalk scaling-group list. Traffic arriving at this hostname is routed to the port you specified with --port (or the port constructor argument). Because the hostname includes the scaling group name, you can reference it by a stable, human-readable URL rather than an opaque UUID. To require a Chalk access token on requests to this hostname, see Authentication. ### Invoking from the SDK The .call() helper sends a request to a path under web_url. The response it returns has status, headers, and content attributes, plus text() and json() methods. Use .call() for ad-hoc requests from the SDK, and send production traffic to web_url directly. This example sends JSON to /predict, which stands for a route your own service serves: ``` resp = sg.call( "/predict", method="POST", json={"text": "the cat sat on the mat"}, timeout=60.0, ) print(resp.json()) ``` # Revisions and Rollback source: https://docs.chalk.ai/docs/scaling-groups/revisions-and-rollback ## Roll a new revision onto a scaling group, list its revisions, and roll back to an earlier one. Each deploy of a scaling group records an immutable revision of the group's spec. The group keeps the same ID, name, and web_url across revisions, and serves one revision at a time. Redeploying with the same name from ScalingGroup(...).deploy() or chalk scaling-group create records a new revision and routes all traffic to it. That is how you roll a new image or spec onto a running group. To record a new revision from the CLI without serving it, add --no-traffic. The group keeps serving its current revision until you select the new one. To roll back, select an earlier revision. Selecting a revision routes all traffic to it and does not record a new revision. Revision history is also on the scaling group's Revisions tab in the dashboard. ### From the SDK This example uses the app.py file from the Quick start, saved in the same directory. It deploys two revisions of a group, lists them, and rolls back to the first: ``` import chalkcompute img = ( chalkcompute.Image.debian_slim("3.12") .pip_install(["flask"]) .add_local_file("./app.py", "/app/app.py") .entrypoint(["python", "/app/app.py"]) ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, env={"APP_VERSION": "1"}, ).deploy() # Deploying again under the same name records a second revision. sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, env={"APP_VERSION": "2"}, ).deploy() revisions = sg.revisions() for rev in revisions: print(rev.id, rev.status, rev.is_current) sg.rollback(revisions[1].id) print(sg.refresh().current_revision_id) ``` revisions() returns the group's revisions, newest first. The serving revision has is_current set to True, and the others report the Superseded status. rollback(revision_id) selects that revision and waits until the group is ready. Pass wait_ready=False to return without waiting, or timeout to change the wait from its default of 300 seconds. ### From the CLI List a group's revisions, inspect one, and route traffic to it: ``` $ chalk scaling-group revisions list --name=hello-api $ chalk scaling-group revisions get --name=hello-api --revision-id= $ chalk scaling-group update-traffic --name=hello-api --revision-id= ``` update-traffic routes all of the group's traffic to the revision you name. Use it to roll back, or to start serving a revision you recorded with --no-traffic. Each command also accepts --id in place of --name. # Managing Scaling Groups source: https://docs.chalk.ai/docs/scaling-groups/managing-scaling-groups ## List, inspect, and delete scaling groups from the CLI and the SDK. List, inspect, and delete the scaling groups in your environment from the CLI or the SDK. ### Listing ``` $ chalk scaling-group list ``` Displays a table of all scaling groups in your environment with their ID, name, selected revision, image, status, replica counts, URL, and creation time. ``` from chalk.client import ChalkClient client = ChalkClient() response = client.list_scaling_groups() for scaling_group in response.scalingGroups: print(f"Scaling group: {scaling_group.name}") print(f" Status: {scaling_group.status}") ``` ### Inspecting ``` $ chalk scaling-group get --name=inference ``` Shows the group's ID, name, selected revision, status, image, port, minimum and maximum replicas, web URL, and creation time. ``` import chalkcompute sg = chalkcompute.ScalingGroup.from_name("inference") # or sg = chalkcompute.ScalingGroup.from_id("scaling-group-id") info = sg.refresh() print(info.status) # 'Available', 'Pending', 'ScaledToZero', ... print(info.ready_replicas) print(info.available_replicas) print(sg.web_url) ``` A scaling group reports one of these statuses: | Status | Meaning | | -------------- | ------------------------------------------------------------------------------------ | | `Pending` | No replica is ready. | | `Available` | All desired replicas are ready. | | `Degraded` | Some replicas are ready, but not all desired replicas are, or a rollout has stalled. | | `ScaledToZero` | `min_replicas` is `0` and the autoscaler has scaled the group to zero replicas. | | `Unhealthy` | No replica is ready and a replica is failing, or a startup probe failed. | | `Error` | Chalk could not apply the group's spec. | | `Unknown` | Chalk has not reported a status for the group. | | `Deleted` | The group was deleted. | ### Deleting ``` $ chalk scaling-group delete --name=inference ``` ``` sg.delete() ``` Both surfaces remove the Kubernetes Deployment, Service, and HTTPRoute associated with the scaling group. The SDK additionally cleans up any temporary volumes created from add_local_file uploads. The database record is soft-deleted so the group still appears in history. # Workflows source: https://docs.chalk.ai/docs/compute/workflows ## Run Python workflows on dynamically provisioned scaling groups with separate images for each task. ### Overview A workflow is an async Python function that calls tasks and combines their results. Use workflows for jobs with multiple steps, such as preparing data, running inference, and writing results. The workflow orchestrator records progress and applies each task's retry and timeout settings. Deploy a workflow with chalkcompute.deploy_workflow_worker. Chalk packages the source file and its dependencies into an image and creates a scaling group to run it. Each task can declare its own image and Python dependencies. Chalk creates a separate scaling group for each distinct task image, so steps with different environments can run in the same workflow. Workers start at zero replicas. Submitting a run wakes the workflow's worker; calling a task with its own image wakes that task's worker. Each group stays up while it has an open workflow and returns to zero after its idle delay. ### Define and deploy a workflow Use an environment with a workflow orchestrator enabled, and authenticate with chalk login. Define tasks with @task and the orchestration function with @workflow. Tasks can be synchronous or asynchronous; workflow functions must be async def. Save this example as pipeline.py: ``` # /// script # dependencies = ["chalkcompute", "chalkpy[workflows]"] # /// import chalkcompute from chalk.workflows import task, workflow @task def score(rows: list[int]) -> list[float]: return [round(row * 0.1, 2) for row in rows] @task def summarize(batch_id: str, scores: list[float]) -> dict: return { "batch_id": batch_id, "rows": len(scores), "mean_score": sum(scores) / len(scores) if scores else None, } @workflow async def score_batch(batch_id: str, rows: list[int]) -> dict: scores = await score(rows) return await summarize(batch_id, scores) if __name__ == "__main__": worker = chalkcompute.deploy_workflow_worker(name="wf-score-batch") print("task queue:", worker.task_queue) ``` Run the script to deploy the worker: ``` uv run pipeline.py ``` The worker serves score_batch on the queue chalkcompute-workflows.wf-score-batch. Deploying the same name again updates the existing group. Deployment does not start a workflow run. Keep deployment and submission code inside if __name__ == "__main__":. Workers import the file to register its tasks and workflows, so code outside that block also runs on each worker. Workflow code must be deterministic. Put network requests, file access, randomness, and wall-clock reads inside tasks. Use JSON-serializable workflow inputs and return values. ### Submit a run Once the worker is deployed, submit runs by name from another script or service. The caller needs Chalk credentials and chalkpy[workflows]; it does not need to import pipeline.py. Save this as submit.py, using the task queue printed by the deployment: ``` # /// script # dependencies = ["chalkpy[workflows]"] # /// from chalk.client import ChalkClient run = ChalkClient().trigger_workflow( "score_batch", {"batch_id": "batch-001", "rows": list(range(8))}, task_queue="chalkcompute-workflows.wf-score-batch", wait=False, ) print("workflow id:", run.workflow_id) print("run id:", run.run_id) ``` ``` uv run submit.py ``` wait=False returns a handle once the run is accepted. The workflow continues after the submitting process exits. Pass wait=True to wait for completion and receive the workflow's return value instead of a handle: ``` result = ChalkClient().trigger_workflow( "score_batch", {"batch_id": "batch-001", "rows": list(range(8))}, task_queue="chalkcompute-workflows.wf-score-batch", wait=True, ) print(result) # {'batch_id': 'batch-001', 'rows': 8, 'mean_score': 0.35} ``` The first task may wait for a worker to start and pull its image when the group is at zero replicas. If you already have the worker returned by deploy_workflow_worker, use worker.trigger(...) to submit to its queue: ``` run = worker.trigger( score_batch, {"batch_id": "batch-002", "rows": [2, 4, 6]}, ) ``` worker.trigger also accepts a workflow name and defaults to wait=False. Pass wait=True to return the result. ### Task images and dependencies Set image on a task to give it a separate Python environment. For example, replace score in pipeline.py with a task that uses NumPy: ``` numpy_image = chalkcompute.Image.debian_slim("3.12").pip_install(["numpy"]) @task(image=numpy_image, retries=2, timeout="5m") def score(rows: list[int]) -> list[float]: import numpy as np return np.round(np.asarray(rows) * 0.1, 2).tolist() ``` Run uv run pipeline.py again to deploy the change. Chalk creates an additional scaling group for the NumPy image. score runs there, while score_batch and summarize run on the main worker. Tasks with equal image definitions share a group, including when the images are separate Python objects. Tasks without an image use the main worker's environment. Keep imports for task-specific packages inside the task body. Every worker imports the whole source file; importing NumPy at module scope would require it on the main worker and the deploying machine too. The default worker image installs the dependencies in the script's # /// script block, including chalkpy[workflows]. Each task image also receives those shared dependencies and the source file. Put packages shared by all workers in that block, and packages used only by a task on its image. See Images for installing system packages and adding local files. Inspect the groups through worker.scaling_group for the main worker and worker.task_workers for the mapping from task names to their groups. Groups remain deployed at zero replicas between runs; scaling down does not delete them. ### Worker configuration Pass worker settings to deploy_workflow_worker: ``` worker = chalkcompute.deploy_workflow_worker( source="pipeline.py", name="wf-score-batch", cpu="2", memory="4Gi", max_replicas=2, shutdown_delay_seconds=60, ) ``` source defaults to the running script. Without an explicit name, the worker name is derived from the file name, such as wf-pipeline. The main worker and task image workers use the same CPU, memory, replica limit, shutdown delay, env, and secrets settings. The helper sets min_replicas=0 and configures each group to scale on its workflow task queue. An open workflow keeps its group running, including while it waits for a task or retry. Pass image to customize the main worker's image. With a custom main image, install the script's dependencies and chalkpy[workflows] yourself; the helper adds the source file. ### Retries and parallel tasks Configure retries and timeouts on each @task. In the NumPy example, retries=2 permits two retries after the first failed attempt, and timeout="5m" limits each attempt to five minutes. The defaults are no retries and a ten-minute timeout per attempt. Use retry_delay, such as retry_delay="30s", to set the initial retry interval. A retried task can execute more than once. Make external writes safe to repeat, for example by using a batch ID as an idempotency key. Use .submit() inside a workflow to start independent tasks before waiting for their results: ``` import asyncio @workflow async def score_batches(batches: list[list[int]]) -> list[list[float]]: handles = [score.submit(rows) for rows in batches] return await asyncio.gather(*handles) ``` Each call uses the task's image, retry policy, and timeout. Add the workflow to pipeline.py and redeploy the worker before submitting it by name. ### Run locally Call a workflow directly to test it in the current process: ``` import asyncio from pipeline import score_batch result = asyncio.run(score_batch("local-batch", [2, 4, 6])) ``` Direct calls use the in-process orchestrator. They do not deploy scaling groups or use task images, so install each task's dependencies locally. To execute locally with durable coordination from the environment's orchestrator, use ChalkClient().run_workflow: ``` from chalk.client import ChalkClient result = ChalkClient().run_workflow( score_batch, {"batch_id": "local-batch", "rows": [2, 4, 6]}, ) ``` This call blocks until the workflow returns. Its tasks execute in the calling process and ignore image; keep that process running for the duration of the workflow. # Autoscaling source: https://docs.chalk.ai/docs/workload-settings/autoscaling ## Set replica ranges, scaling targets, scale to zero, and scheduled replica floors for scaling groups and functions. Autoscaling applies to scaling groups and functions. A function runs on a scaling group and passes these arguments to it. Functions can also scale on their own request queue depth. ### Parameters | Parameter | Applies to | Description | | ----------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `min_replicas` | Both | Lower bound for the autoscaler. Set to `0` to allow [scale to zero](#scale-to-zero). On a scaling group, defaults to `1`. | | `max_replicas` | Both | Upper bound for the autoscaler. On a scaling group, defaults to `1`. | | `target_cpu_utilization_percentage` | Both | Average CPU utilization the autoscaler targets, from 1 to 100. The autoscaler adds replicas when the average exceeds it and removes them when the average is below it. | | `target_gpu_utilization_percentage` | Both | Average GPU utilization the autoscaler targets, from 1 to 100. Requires `gpu` and `min_replicas` of at least `1`. | | `target_queue_depth` | Functions | Target pending-request queue depth per replica. When set, the autoscaler scales on the function's own queue depth. | | `scaling_interval_seconds` | Both | How often the autoscaler scrapes metrics and makes scaling decisions. Defaults to 60s on the server. | | `shutdown_delay_seconds` | Both | Grace period when removing replicas, to drain in-flight requests. See [Readiness and Shutdown](/docs/workload-settings/readiness-and-shutdown#shutdown-delay). | | `scale_from_zero_request_policy` | Both | How requests to a scaling group are handled while it scales up from zero. See [Scale to zero](#scale-to-zero). | | `cron_scaling_trigger` | Both | Replica floors for scheduled time windows. See [Scheduled replica floors](#scheduled-replica-floors). | A scaling group: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="inference", port=8080, min_replicas=2, max_replicas=20, target_cpu_utilization_percentage=70, shutdown_delay_seconds=30, ).deploy() ``` From the CLI, --min-replicas and --max-replicas set the replica range, and --target-cpu sets the CPU utilization target for autoscaling. --max-replicas is required. To run a fixed number of replicas, set both flags to the same value. A function: ``` import chalkcompute @chalkcompute.function( min_replicas=1, max_replicas=8, scaling_interval_seconds=30, target_cpu_utilization_percentage=70, shutdown_delay_seconds=120, gpu="nvidia-l4", ) def infer(text: str) -> str: ... ``` ### CPU-driven vs queue-driven scaling These two signals apply to functions: - CPU (target_cpu_utilization_percentage): good for compute-bound functions, such as model inference or data transformation, where CPU is the bottleneck. - Queue depth (target_queue_depth): good for I/O-bound or variable-latency functions, or when you want to scale on demand rather than resource saturation. Scales up when calls pile up regardless of CPU. Both can be set together; the autoscaler takes the higher signal. ### Scale to zero Set min_replicas=0 to let a scaling group scale down to zero replicas when it is not receiving traffic. A function also accepts min_replicas=0. The group reports the ScaledToZero status while it has no replicas. A request that arrives while the group has no replicas starts a scale-up. By default, Chalk responds to that request with HTTP 503, and your client has to retry it. To hold the request until a replica is ready to serve it, set scale_from_zero_request_policy to ScaleFromZeroRequestPolicy.HOLD: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="on-demand-api", port=8080, min_replicas=0, max_replicas=5, scale_from_zero_request_policy=chalkcompute.ScaleFromZeroRequestPolicy.HOLD, ).deploy() ``` From the CLI, pass --min-replicas=0 and --scale-from-zero-request-policy=hold. ### Scheduled replica floors To keep replicas running during set hours, pass a CronScalingTrigger. Each CronScalingWindow sets a replica floor from its start cron expression to its end cron expression, evaluated in the trigger's IANA timezone. During a window the group runs at least desired_replicas replicas. Outside its windows the group scales between min_replicas and max_replicas. This group runs at least three replicas from 8:00 to 18:00 New York time on weekdays, and can scale to zero the rest of the time: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="business-hours-api", port=8080, min_replicas=0, max_replicas=10, cron_scaling_trigger=chalkcompute.CronScalingTrigger( timezone="America/New_York", windows=[ chalkcompute.CronScalingWindow( start="0 8 * * 1-5", end="0 18 * * 1-5", desired_replicas=3, ), ], ), ).deploy() ``` A function takes the same cron_scaling_trigger argument. Scheduled replica floors are set from the SDK. The CLI has no flag for them. # Readiness and Shutdown source: https://docs.chalk.ai/docs/workload-settings/readiness-and-shutdown ## Control when a scaling group or function replica receives traffic, and how long it drains before shutdown. Readiness probes and the shutdown delay apply to scaling groups and functions. A function takes readiness_probe and shutdown_delay_seconds and passes them to the scaling group it runs on. Startup probes apply to scaling groups only. ### Health checks Two probes control when Chalk treats a replica or a group as ready: - Readiness probe (readiness_probe): checks each replica directly, on a repeating schedule. A replica receives traffic only while its readiness probe passes. - Startup probe (startup_probe): checks the group's web_url once per deploy. deploy() waits until the probe passes, and the group reports the Unhealthy status if it does not. ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, readiness_probe=chalkcompute.ReadinessProbe.http( "/", period_seconds=10, timeout_seconds=2, failure_threshold=3, ), startup_probe=chalkcompute.StartupProbe.http("/"), ).deploy() ``` ReadinessProbe.http(path) sends an HTTP GET to path on each replica. ReadinessProbe.grpc(service) calls the standard gRPC health check. Both take optional period_seconds, timeout_seconds, and failure_threshold. StartupProbe.http(path) sends an HTTP GET to path on the group's web_url and passes on a 2xx or 3xx response. Without a path, it requests /. StartupProbe.grpc(method) calls method through the group's route and passes when the call returns OK. The method must be fully qualified. Without a method, the probe passes once the route answers gRPC. A startup probe requires a port. An HTTP startup probe requires an HTTP group, and a gRPC startup probe requires protocol="grpc". From the CLI, --startup-probe adds a startup probe that requests / on the group's web_url. The CLI has no flag for readiness probes. ### Shutdown delay shutdown_delay_seconds sets the grace period, in seconds, before replicas are forcibly killed during scale-down. It lets in-flight requests finish cleanly. It defaults to 30 seconds on the server. ``` import chalkcompute @chalkcompute.function(shutdown_delay_seconds=120) def infer(text: str) -> str: return text.upper() ``` On a scaling group, pass the same argument to ScalingGroup. # Authentication source: https://docs.chalk.ai/docs/workload-settings/authentication ## Require a Chalk access token on every request to a scaling group or function. The authenticated setting applies to scaling groups and functions. A function takes the same argument, @chalkcompute.function(authenticated=True), and passes it to the scaling group it runs on. ### Requiring an access token A scaling group's web_url is public by default: anyone who has the URL can send requests to it. To require a Chalk access token on every request, set authenticated=True from the SDK or pass --authenticated from the CLI. A request without a valid token gets an HTTP 401 response. ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, authenticated=True, startup_probe=chalkcompute.StartupProbe.http("/"), ).deploy() resp = sg.call("/", method="GET") print(resp.status) ``` .call() sends your Chalk access token automatically. Any other client sends the token as a bearer token in the Authorization header. Get a token with chalk token get. In this example, WEB_URL holds the group's web_url: ``` $ curl -H "Authorization: Bearer $(chalk token get)" "$WEB_URL/" ``` Access tokens expire. To request a longer lifetime, pass --expires-in, for example chalk token get --expires-in 24h. The server can cap the lifetime at the maximum allowed for your credentials. Private routing, which keeps a group off the public internet, requires setup with Chalk. To use it, contact Chalk support. # Dynamic Chalk Queries source: https://docs.chalk.ai/docs/compute/dynamic-chalk-queries ## Let an agent write and run Chalk queries on demand, fetching exactly the features it reasons it needs. ### Overview The usual way to give an LLM access to your data is to gather it up front: run a fixed query, serialize the rows into the prompt, and hope you fetched the right columns. That is a guess made before the model has reasoned about anything. A better pattern, and the one Chalk Compute is built for, is to hand the agent a tool that issues Chalk queries and let it decide what to pull. Because the agent runs inside a Chalk Function — co-located with the feature engine, inside your own cloud — each tool call is a real online query against the feature store. The agent reads the fraud score, decides it needs the account age, fetches that too, and rules. Nothing is pre-staged, and no data leaves your VPC. This is what we mean by data-native agents: the feature store is the agent's context window, queried on demand. ``` # Inside the agent loop, a tool call becomes a live Chalk query: ctx = chalk_client.query( input={"user.id": user_id}, output=requested_features, # ← chosen by the model at inference time ) ``` ### The pattern Three pieces turn an LLM into a data-native agent: - A feature-fetch tool exposed to the model, whose parameters name the features it is allowed to request. - A handler that turns each tool call into a ChalkClient.query(...). - A @chalkcompute.function wrapping the whole loop so it runs next to the feature engine with credentials injected. Here is a refund-abuse investigator built exactly this way. The agent is given a get_chalk_features tool, and each time it calls that tool we run an online query for the features it asked for and feed the results back: ``` import json import chalkcompute @chalkcompute.function(secrets=[...], image=..., tracing=True) def investigate_refund(user_id: int, reason: str) -> str: from chalk.client import ChalkClient chalk_client = ChalkClient() messages = [...] # system prompt + the refund claim while True: response = openai_client.chat.completions.create( model="gpt-5.5", messages=messages, tools=[{ "type": "function", "function": { "name": "get_chalk_features", "parameters": { # the model names whichever features it wants "type": "object", "properties": { "user_id": {"type": "integer"}, "features": { "type": "array", "items": {"type": "string"}, }, }, }, }, }], ) msg = response.choices[0].message if not msg.tool_calls: return msg.content or "" # final verdict messages.append(msg.model_dump(exclude_none=True)) for tc in msg.tool_calls: args = json.loads(tc.function.arguments) # Each tool call becomes a live online query against the feature store. with chalkcompute.span("tool.get_chalk_features"): result = chalk_client.query( input={"user.id": args["user_id"]}, output=args["features"], # ← the query the agent composed ) rendered = "\n".join(f"{r.field}: {r.value}" for r in result.data) messages.append({"role": "tool", "tool_call_id": tc.id, "content": rendered}) ``` The model never sees a Chalk SDK or a feature definition — it names the features it wants as plain strings and emits a JSON array. The query is composed by the agent at inference time: output=args["features"] is whatever subset the model decided it needed on this turn. That is a Chalk query the agent wrote. The output argument accepts feature name strings like "user.is_fraud", not just User.is_fraud attributes. That is exactly what makes this work — the model emits strings, and they map straight onto a query. See Online Queries for the full client API. ### Best Practices ### Scope the agent's data surface with token permissions Notice the tool lets the model ask for any feature name — there is no hardcoded allowlist in the schema. That is deliberate. Pinning the permitted features into a tool schema is brittle: the list drifts the moment someone adds a feature, lives somewhere other than the feature definitions, and is only as trustworthy as the code that assembles the prompt. At best it is a hint to the model, never a security boundary. The boundary belongs in Chalk instead. The agent queries under a service token, and tokens carry tag-based feature permissions — an allowlist or blocklist of feature tags that Chalk enforces server-side at query time. Tag your features once, at definition: ``` from chalk.features import features, feature @features(tags=["agent-readable"]) class User: id: int total_spend: float is_fraud: bool # SSN can feed other features but never reach the agent: ssn: str = feature(tags=["pii"]) ``` Issue the agent a token that allowlists agent-readable and blocklists pii. Every query the agent composes is checked against that scope, and a request for an out-of-scope feature comes back Unauthorized — even if the model hallucinated the name. Chalk also keeps separate permissions for computed-only versus output features, so a feature can participate in deriving others without ever being returned to the agent. Don't keep sensitive features out of an agent's reach with a list in a prompt — the model can ignore it and the list rots. Give the agent the widest data surface its job needs and let tag permissions on its token decide what is off-limits. That boundary is enforced by the engine, not by the model's good behavior. ### Fetch incrementally, in small batches Prompt the agent to pull a few features, reason, then pull more — rather than requesting everything at once. Cheaper queries, smaller prompts, and a tool-call trace that reads like an investigation: checked fraud score → looked up account age → ruled. It also lets the agent stop early when it has seen enough. ### Run the loop inside a Chalk Function Wrap the agent in @chalkcompute.function and inject Chalk and model credentials with Secret. The function gives you autoscaling, retries, and concurrency limits for free, and keeps the ChalkClient co-located with the engine. Call it from anywhere with RemoteFunction.from_name(...): ``` from chalkcompute import RemoteFunction investigate = RemoteFunction.from_name("investigate_refund") verdict = investigate.remote(user_id=1, reason="item not received") ``` ### Trace every query Wrap each tool call in chalkcompute.span(...) and set tracing=True on the function. You get a span per query showing which features the agent fetched and when — the audit trail for why an agent decided what it did. Indispensable for debugging agent behavior and reviewing high-stakes decisions. ### Control freshness with staleness If an agent can tolerate slightly stale values, pass staleness to accept cached features and skip recomputation — faster and cheaper. Reserve fresh computation for the signals where it matters. ``` result = chalk_client.query( input={"user.id": user_id}, output=["user.total_spend"], staleness={"user.total_spend": "1h"}, # cached value up to 1h old is fine ) ``` ### Gate side-effecting tools Read-only feature fetches are safe to let the agent fire at will. Tools that act — banning a user, issuing a refund, writing to a table — deserve more care: require human approval, restrict them to specific verdicts, or log them prominently. Keep the agent's ability to read data wide and its ability to change things narrow. ### Replay decisions with point-in-time correctness To backtest an agent or reproduce a past decision, run the same feature reads as an offline query with input_times. Chalk computes each feature as of that timestamp, so the agent sees the data it would have seen then — no leakage from the future. The agent logic is unchanged; only the query mode differs. ### How it fits together ``` ┌─────────────┐ tool call: get_chalk_features(["user.is_fraud"]) │ LLM │ ───────────────────────────────────────────────┐ │ (the agent)│ ▼ └─────────────┘ ┌──────────────────────────┐ ▲ │ @chalkcompute.function │ │ features as messages │ ChalkClient().query(...)│ └───────────────────────────────────────── │ → Chalk feature engine │ └──────────────────────────┘ (same cluster, your cloud) ``` The model decides what to fetch; the function executes it as a real Chalk query against the engine next door. The agent's reasoning and your feature store operate as one system — which is the whole point of running agents on Chalk Compute. # Batched Calls for Embeddings source: https://docs.chalk.ai/docs/compute/batched-embeddings ## Batch and parallelize embedding calls across a DataFrame with F.catalog_call(). ### Overview Embedding models typically accept batches of inputs and return batches of vectors. When you need to embed a large dataset, sending one row at a time wastes round-trips and leaves GPU capacity idle. Calling a deployed function with F.catalog_call() inside a static resolver's DataFrame pipeline lets Chalk parallelize the calls across rows automatically. Combined with the function's own max_batching_size and max_buffer_duration, rows are grouped into batches before they ever reach your handler. There's no chunk size or concurrency limit to configure by hand. ### Define the embedding function The remote function receives its input as a columnar batch. Declare a list[str] or pyarrow.Table parameter so it can accept multiple rows at once: Your embedding function must accept batch input: either a list[str], a list[list[float]], or a pyarrow.Table. A function that accepts a single str will only process one row per call, defeating the purpose of batching. ``` import chalkcompute @chalkcompute.function( name="embed", image=chalkcompute.Image.base("python:3.12-slim").run_commands( "pip install sentence-transformers", ), cpu="2", memory="4Gi", gpu="nvidia-l4", max_batching_size=25, max_buffer_duration=2000, retries=3, ) def embed(text: list[str]) -> list[list[float]]: from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") return model.encode(text).tolist() ``` When max_batching_size or max_buffer_duration is set, the function queue automatically buffers incoming items and delivers them to the handler as a batch. The handler is invoked when either the buffer reaches max_batching_size items or max_buffer_duration milliseconds have elapsed, whichever comes first. Deploy the function: ``` python embed.py ``` ### Call embed() from a DataFrame Once embed is deployed, call it across every row of a DataFrame with F.catalog_call() inside a static resolver: Unlike .remote() or .defer(), F.catalog_call() only takes effect once the resolver is applied to your environment with chalk apply. It's evaluated during chalk query (or an offline query), so there's no way to call it from a plain script. ``` from chalk import online from chalk import functions as F from chalk.features import DataFrame, Vector, features, _ @features class Document: id: int text: str embedding: Vector[384] @online(static=True) def compute_document_embeddings( df: DataFrame[Document.id, Document.text], ) -> DataFrame[Document.id, Document.embedding]: return ( df.with_columns({ "document.embedding": F.catalog_call("function.embed", _.text), }) .select(str(Document.id), "document.embedding") ) ``` Each row's text column is passed to embed, and the result is written as the embedding feature alongside it. .with_columns() associates each computed embedding with its row by the DataFrame's own row identity, not by call-completion order, so results line up correctly with their inputs even though embed's internal batching and Chalk's parallelization can complete rows out of order. Note the qualified name passed to catalog_call: it's function.embed, not a bare function name; model.* and function.* are the two valid namespaces. ### Ingesting embeddings from a file To compute embeddings for documents that live in a file rather than an existing DataFrame, scan the file directly inside an @offline(static=True) resolver: ``` import chalkdf from chalk import offline from chalk import functions as F from chalk.features import DataFrame, Vector, features, _ @features class Document: id: int text: str embedding: Vector[384] @offline(static=True) def ingest_document_embeddings() -> DataFrame[ Document.id, Document.text, Document.embedding, ]: return ( chalkdf.DataFrame.scan(["documents.parquet"]) .with_columns({ "document.embedding": F.catalog_call("function.embed", _.text), }) .select("document.id", "document.text", "document.embedding") ) ``` This resolver reads documents.parquet, computes an embedding for every row via the same embed function, and ingests the result as Document feature data once applied and queried. There's no separate step to manually chunk or batch the file's rows. # RAG Pipeline source: https://docs.chalk.ai/docs/compute/rag-pipeline ## Build a retrieval-augmented generation function that embeds a query and searches OpenSearch. ### What we're building A single remote function that implements the retrieval half of RAG: - Embed the user's question using an embedding model via Chalk's OpenAI-compatible Model Gateway. - Run a k-NN vector search against an OpenSearch index to find relevant documents. - Return the top matches with their text, ready to be stuffed into an LLM prompt. This is the pattern you'd use inside a resolver or as a building block for a larger agent — the function handles the "R" in RAG so the caller only deals with the final generation step. ### Define the sandbox-backed function The function needs the OpenAI SDK (for embeddings) and the OpenSearch client. We also inject secrets for both services so credentials stay out of the code. ``` # rag.py import chalkcompute from chalkcompute import Image, Secret rag_image = ( Image.base("python:3.12-slim") .pip_install(["openai", "opensearch-py"]) ) ``` ### The retrieve function The function takes a plain-text query, embeds it, and searches OpenSearch in one shot. No volume needed here — the document corpus already lives in the index. ``` @chalkcompute.function( name="retrieve", image=rag_image, cpu="1", memory="2Gi", secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("OPENSEARCH_HOST"), Secret.from_chalk_env("OPENSEARCH_USERNAME"), Secret.from_chalk_env("OPENSEARCH_PASSWORD"), ], ) def retrieve(query: str, top_k: int = 5) -> list[dict]: """Embed a query and return the top-k matching documents from OpenSearch.""" import os import openai from opensearchpy import OpenSearch # 1. Embed the query through Chalk's router. ai = openai.OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://router.chalk.ai/v1", ) response = ai.embeddings.create( model="text-embedding-3-small", input=query, ) query_vector = response.data[0].embedding # 2. Connect to OpenSearch. client = OpenSearch( hosts=[os.environ["OPENSEARCH_HOST"]], http_auth=( os.environ["OPENSEARCH_USERNAME"], os.environ["OPENSEARCH_PASSWORD"], ), use_ssl=True, verify_certs=True, ) # 3. Run a k-NN search against the document index. results = client.search( index="documents", body={ "size": top_k, "query": { "knn": { "embedding": { "vector": query_vector, "k": top_k, } } }, "_source": ["text", "title", "url"], }, ) # 4. Return the hits in a simple format. return [ { "title": hit["_source"].get("title", ""), "text": hit["_source"].get("text", ""), "url": hit["_source"].get("url", ""), "score": hit["_score"], } for hit in results["hits"]["hits"] ] ``` The function is stateless — every call creates a fresh OpenSearch connection. For high-throughput use cases, you can cache the client in a module-level variable since each sandbox instance is long-lived. ### Deploy it ``` python rag.py ``` ### Calling the function From any Python process with chalkcompute installed: ``` import chalkcompute retrieve = chalkcompute.RemoteFunction.from_name("retrieve") docs = retrieve.remote("How do I set up streaming resolvers?", top_k=3) for doc in docs: print(f"[{doc['score']:.3f}] {doc['title']}") print(f" {doc['text'][:120]}...") print() ``` ``` [0.891] Streaming Resolvers Streaming resolvers let you subscribe to a data source and compute features as events arrive, rather than polling on a ... [0.847] Resolver Overview Resolvers are Python functions that compute feature values. They can query databases, call APIs, or run arbitrary logic... [0.823] Kafka Integration Connect Chalk to a Kafka topic to power streaming resolvers. Events are delivered to your resolver function as they arr... ``` ### Request lifecycle ``` Caller retrieve() Chalk Router OpenSearch │ │ │ │ │ retrieve("How do │ │ │ │ I set up streaming │ │ │ │ resolvers?") │ │ │ │──────────────────────▸│ │ │ │ │ │ │ │ │ embeddings.create() │ │ │ │──────────────────────▸│ │ │ │ │ │ │ │ [1536-dim vector] │ │ │ │◂──────────────────────│ │ │ │ │ │ │ │ k-NN search(vector, k=5) │ │ │──────────────────────────────────────────────▸│ │ │ │ │ │ │ [{title, text, url, score}, ...] │ │ │◂──────────────────────────────────────────────│ │ │ │ │ │ [{title, text, │ │ │ │ url, score}, ...] │ │ │ │◂──────────────────────│ │ │ │ │ │ │ ``` When plugged into generation, the caller adds one more hop — passing the retrieved documents into an LLM to produce the final answer: ``` ask() retrieve() Chalk Router OpenSearch │ │ │ │ │ retrieve(question) │ │ │ │──────────────────────▸│ (as above) │ │ │ [{docs}] │ │ │ │◂──────────────────────│ │ │ │ │ │ │ │ chat.completions.create( │ │ │ model="gpt-4o", │ │ │ messages=[system: {docs}, user: question]) │ │ │──────────────────────────────────────────────▸│ │ │ │ │ │ "Streaming resolvers let you..." │ │ │◂──────────────────────────────────────────────│ │ │ │ ``` ### Plugging into generation The retrieve function returns plain dicts, so wiring it into a generation step is straightforward — call retrieve, format the results into a prompt, and pass it to your LLM: ``` import chalkcompute import openai retrieve = chalkcompute.RemoteFunction.from_name("retrieve") def ask(question: str) -> str: # Retrieval docs = retrieve.remote(question, top_k=5) context = "\n\n---\n\n".join( f"## {d['title']}\n{d['text']}" for d in docs ) # Generation client = openai.OpenAI( base_url="https://router.chalk.ai/v1", ) response = client.chat.completions.create( model="gpt-4o", messages=[ { "role": "system", "content": ( "Answer the user's question using only the context below.\n\n" f"{context}" ), }, {"role": "user", "content": question}, ], ) return response.choices[0].message.content print(ask("How do I set up streaming resolvers?")) ``` The retrieval runs on Chalk Compute (close to your OpenSearch cluster), while generation can run anywhere — your laptop, a notebook, or another remote function. # Fan-Out Inference source: https://docs.chalk.ai/docs/compute/fan-out-inference ## Orchestrate parallel embedding jobs across worker functions with shared volumes. ### What we're building A two-function system for batch-embedding documents at scale: - orchestrate: reads a manifest of file paths from a volume, fans out one embed call per document, and collects the results. - embed: reads a single file from the shared volume, calls an embedding model through Chalk's OpenAI-compatible Model Gateway, writes the embedding back to the volume, and upserts it into Turbopuffer for vector search. Both functions are deployed by running their Python scripts and communicate through a shared Volume. The orchestrator doesn't need a GPU or heavy dependencies — it just coordinates. The workers carry the inference libraries. ### The embed worker Start with the leaf function. Each invocation receives a file path on a shared volume, embeds its contents, and pushes the result to two places: back to the volume (so the orchestrator can read it) and to Turbopuffer for indexing. ``` # embed_worker.py import json import os import chalkcompute from chalkcompute import Image, Secret, Volume worker_image = ( Image.base("python:3.12-slim") .pip_install(["openai", "turbopuffer"]) ) ``` The worker function's configuration pins its resource limits and mounts the shared volume at /data. Secrets are injected as environment variables — the function code reads them with os.environ as usual, but they never appear in the image or source. Now the function itself: ``` @chalkcompute.function( name="embed", image=worker_image, cpu="2", memory="4Gi", secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("TURBOPUFFER_API_KEY"), ], volumes=[("embed-data", "/data")], ) def embed(file_path: str) -> str: """Embed a single document and push to Turbopuffer. Args: file_path: Path relative to the volume root, e.g. "docs/readme.md". Returns: The path where the embedding was written on the volume. """ import openai import turbopuffer as tpuf # 1. Read the document from the shared volume. full_path = os.path.join("/data", file_path) with open(full_path) as f: text = f.read() # 2. Call the embedding model through Chalk's OpenAI-compatible router. # The router handles rate limiting, retries, and cost tracking. client = openai.OpenAI( api_key=os.environ["OPENAI_API_KEY"], base_url="https://router.chalk.ai/v1", ) response = client.embeddings.create( model="text-embedding-3-small", input=text, ) embedding = response.data[0].embedding # 3. Write the embedding to the volume so the orchestrator can access it. output_path = file_path.rsplit(".", 1)[0] + ".embedding.json" full_output = os.path.join("/data", output_path) os.makedirs(os.path.dirname(full_output), exist_ok=True) with open(full_output, "w") as f: json.dump({"file": file_path, "embedding": embedding}, f) # 4. Upsert into Turbopuffer for vector search. tpuf.api_key = os.environ["TURBOPUFFER_API_KEY"] ns = tpuf.Namespace("documents") ns.upsert( ids=[file_path], vectors=[embedding], attributes={"text": [text[:1000]]}, # store a preview ) return output_path ``` Each worker instance is stateless. The volume provides the input; the function writes its output back to the same volume and to Turbopuffer. Ten workers processing ten files all write to different paths — no coordination needed. ### The orchestrator The orchestrator reads a manifest file from the volume, fans out embed calls in parallel, and waits for all of them to finish. ``` # orchestrator.py import json import os import chalkcompute from chalkcompute import Image, Volume orchestrator_image = ( Image.base("python:3.12-slim") .pip_install(["chalkcompute"]) ) ``` The orchestrator is deliberately lightweight — it doesn't need the OpenAI or Turbopuffer SDKs. It only needs chalkcompute to call the remote embed function. ``` @chalkcompute.function( name="orchestrate", image=orchestrator_image, cpu="1", memory="2Gi", volumes=[("embed-data", "/data")], ) def orchestrate(manifest_path: str) -> dict: """Fan out embedding jobs for every file listed in a manifest. The manifest is a JSON file on the shared volume containing a list of relative file paths, e.g. ["docs/readme.md", "docs/guide.md"]. """ # 1. Read the manifest from the shared volume. full_path = os.path.join("/data", manifest_path) with open(full_path) as f: file_paths = json.load(f) # 2. Get a reference to the embed worker by name. embed_fn = chalkcompute.RemoteFunction.from_name("embed") # 3. Fan out: call embed() for every file in parallel. handles = [embed_fn.defer(fp) for fp in file_paths] results = chalkcompute.gather(handles) # 4. Summarize. return { "total": len(file_paths), "completed": len([r for r in results if r is not None]), "output_paths": list(results), } ``` The key line is chalkcompute.RemoteFunction.from_name("embed") — this returns a handle to the remote embed function by name, without importing it. The orchestrator and worker can live in separate files, separate sandboxes, even separate repos. ### Deploying both functions Put both files in the same directory and deploy them together: ``` python embed_worker.py python orchestrator.py ``` ### Preparing the volume Before running the orchestrator, upload your documents and a manifest: ``` from chalkcompute import Volume vol = Volume(name="embed-data") # Upload documents. vol.put_file("docs/readme.md", open("./corpus/readme.md", "rb").read()) vol.put_file("docs/guide.md", open("./corpus/guide.md", "rb").read()) vol.put_file("docs/api-ref.md", open("./corpus/api-ref.md", "rb").read()) # Upload a manifest listing every file to embed. import json manifest = ["docs/readme.md", "docs/guide.md", "docs/api-ref.md"] vol.put_file("manifest.json", json.dumps(manifest).encode()) ``` ### Running it Call the orchestrator — it reads the manifest and fans out to workers: ``` import chalkcompute orchestrate = chalkcompute.RemoteFunction.from_name("orchestrate") result = orchestrate.remote("manifest.json") print(result) # { # "total": 3, # "completed": 3, # "output_paths": [ # "docs/readme.embedding.json", # "docs/guide.embedding.json", # "docs/api-ref.embedding.json", # ], # } ``` Each embedding is now in two places: on the volume (for downstream pipelines) and in Turbopuffer (for vector search). The orchestrator finished as soon as all three workers returned. ### How it fits together ``` ┌─────────────────┐ │ orchestrate │ manifest.json ────────────▸│ (1 CPU, 2Gi) │ on Volume │ │ └───────┬─────────┘ │ ┌────────────┼────────────┐ ▼ ▼ ▼ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ embed │ │ embed │ │ embed │ │ worker 0 │ │ worker 1 │ │ worker 2 │ │ (2 CPU,4Gi)│ │ (2 CPU,4Gi)│ │ (2 CPU,4Gi)│ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ │ │ │ ┌─────────┼──────────────┼──────────────┼─────────┐ │ ▼ ▼ ▼ │ │ Shared Volume │ │ /data/docs/*.embedding.json │ └─────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────┐ │ Turbopuffer │ │ "documents" namespace │ └─────────────────────────────────────────┘ ``` ### Scaling up For larger corpora, adjust the number of concurrent workers by setting max_replicas on the worker function: ``` @chalkcompute.function( name="embed", image=worker_image, cpu="2", memory="4Gi", secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("TURBOPUFFER_API_KEY"), ], volumes=[("embed-data", "/data")], min_replicas=0, max_replicas=20, concurrency=4, ) ``` With this configuration, Chalk scales from zero to 20 worker replicas as the orchestrator's fan-out grows. Each replica handles up to 4 concurrent embed calls. A corpus of 1,000 documents would be processed by up to 80 parallel embedding requests across 20 workers. # LangChain Agents source: https://docs.chalk.ai/docs/compute/langchain-agents ## Deploy LangChain agents as Functions with persistent memory. ### Overview LangChain agents that call tools, browse the web, or execute code benefit from running in isolated Chalk Compute workloads with dedicated compute and network access. This tutorial deploys a function-shaped LangChain ReAct agent that uses tool-calling to answer research questions, with an optional volume-backed function for persistent state. ### Write the agent Create agent.py — a self-contained LangChain agent implementation used by the Function: ``` # agent.py from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_react_agent from langchain_core.prompts import PromptTemplate from langchain_community.tools import DuckDuckGoSearchRun llm = ChatOpenAI(model="gpt-4o", temperature=0) tools = [DuckDuckGoSearchRun()] prompt = PromptTemplate.from_template( "Answer the following question using the tools available to you.\n\n" "Tools: {tools}\nTool names: {tool_names}\n\n" "Question: {input}\n{agent_scratchpad}" ) agent = create_react_agent(llm, tools, prompt) executor = AgentExecutor(agent=agent, tools=tools, verbose=True) ``` ### Define the function Create answer.py: ``` # answer.py import chalkcompute from chalkcompute import Image, Secret agent_image = ( Image.base("python:3.12-slim") .pip_install([ "langchain", "langchain-openai", "langchain-community", "duckduckgo-search", ]) .add_local_file("agent.py", "/app/agent.py") .workdir("/app") ) @chalkcompute.function( name="langchain-answer", image=agent_image, secrets=[Secret.from_chalk_env("OPENAI_API_KEY")], min_replicas=0, max_replicas=1, ) def answer(question: str) -> str: from agent import executor return executor.invoke({"input": question})["output"] ``` ### Deploy it ``` # deploy_langchain.py from answer import answer print(answer.remote("What is Chalk?")) ``` ``` python deploy_langchain.py # Chalk is a feature platform for building and serving machine learning systems. ``` Once the function is ready, invoke it remotely: ``` from answer import answer print(answer.remote("What is the capital of France?")) # The capital of France is Paris. ``` ### Add persistent memory with a volume LangChain agents can persist conversation history or vector store data across restarts using a Volume. Create agent_memory.py: ``` import uuid import chalkcompute from chalkcompute import Image, Secret memory_image = ( Image.base("python:3.12-slim") .pip_install([ "langchain-openai", "chromadb", ]) ) @chalkcompute.function( name="langchain-memory-answer", image=memory_image, secrets=[Secret.from_chalk_env("OPENAI_API_KEY")], volumes=[("agent-memory", "/app/memory")], min_replicas=0, max_replicas=1, ) def answer(question: str, session_id: str) -> str: import chromadb from langchain_openai import ChatOpenAI client = chromadb.PersistentClient(path="/app/memory") collection = client.get_or_create_collection("conversation-history") history = collection.get( where={"session_id": session_id}, include=["documents"], )["documents"] prompt = "Answer the question using this conversation history:\n" prompt += "\n".join(history) prompt += f"\nUser: {question}" response = ChatOpenAI(model="gpt-4o").invoke(prompt) collection.add( ids=[str(uuid.uuid4())], documents=[f"User: {question}\nAssistant: {response.content}"], metadatas=[{"session_id": session_id}], ) return response.content ``` Deploy and invoke the function with an explicit session ID: ``` from agent_memory import answer print(answer.remote("What did we discuss?", session_id="session-123")) ``` The volume at /app/memory survives function instance restarts. The explicit session_id keeps separate conversations from sharing history. ### Example: transaction risk scoring with Chalk features This example builds a LangChain agent tool that receives a financial transaction, enriches it with features from Chalk, runs a PyTorch risk model, and escalates high-risk transactions to a Kinesis review queue. ### Define the feature namespace Assume your Chalk project defines features like these: ``` from chalk.features import features, Features @features class Transaction: id: str merchant_id: str amount: float merchant_category: str merchant_risk_tier: int customer_avg_spend_30d: float customer_transaction_count_7d: int country_code: str ``` ### Write the agent tool Create risk_tool.py — a LangChain tool that queries Chalk, scores the transaction, and posts flagged results to Kinesis: ``` # risk_tool.py import json import boto3 import torch import torch.nn as nn from chalk.client import ChalkClient from langchain_core.tools import tool RISK_THRESHOLD = 0.85 KINESIS_STREAM = "transaction-review-queue" chalk = ChalkClient() kinesis = boto3.client("kinesis", region_name="us-east-1") class RiskModel(nn.Module): def __init__(self) -> None: super().__init__() self.net = nn.Sequential( nn.Linear(4, 32), nn.ReLU(), nn.Linear(32, 1), nn.Sigmoid(), ) def forward(self, x: torch.Tensor) -> torch.Tensor: return self.net(x) # Load pre-trained weights model = RiskModel() model.load_state_dict(torch.load("/app/models/risk_model.pt", weights_only=True)) model.eval() @tool def score_transaction(transaction_id: str) -> str: """Score a financial transaction for fraud risk. Retrieves enriched features from Chalk, runs a risk model, and escalates to a review queue if the score exceeds the threshold. """ # 1. Query Chalk for enriched transaction features result = chalk.query( input={"transaction.id": transaction_id}, output=[ "transaction.amount", "transaction.merchant_risk_tier", "transaction.customer_avg_spend_30d", "transaction.customer_transaction_count_7d", "transaction.merchant_category", "transaction.country_code", ], ) amount = result.get_feature_value("transaction.amount") merchant_risk_tier = result.get_feature_value("transaction.merchant_risk_tier") avg_spend = result.get_feature_value("transaction.customer_avg_spend_30d") txn_count = result.get_feature_value("transaction.customer_transaction_count_7d") merchant_category = result.get_feature_value("transaction.merchant_category") country = result.get_feature_value("transaction.country_code") # 2. Run the risk model features = torch.tensor([[ amount / max(avg_spend, 1.0), # spend ratio float(merchant_risk_tier), float(txn_count), amount, ]]) with torch.no_grad(): risk_score = model(features).item() # 3. Escalate if above threshold if risk_score > RISK_THRESHOLD: kinesis.put_record( StreamName=KINESIS_STREAM, Data=json.dumps({ "transaction_id": transaction_id, "risk_score": round(risk_score, 4), "amount": amount, "merchant_category": merchant_category, "country": country, "reason": "automated_risk_score_exceeded", }), PartitionKey=transaction_id, ) return ( f"Transaction {transaction_id}: risk score {risk_score:.2%} " f"EXCEEDS threshold. Escalated to review queue." ) return ( f"Transaction {transaction_id}: risk score {risk_score:.2%}. " f"Below threshold — no action required." ) ``` ### Wire it into a function ``` # fraud_review.py import chalkcompute from chalkcompute import Image, Secret review_image = ( Image.debian_slim("3.12") .pip_install([ "langchain", "langchain-openai", "chalkpy", "torch", "boto3", ]) .add_local_file("risk_tool.py", "/app/risk_tool.py") .workdir("/app") ) @chalkcompute.function( name="fraud-review", image=review_image, secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("CHALK_CLIENT_ID"), Secret.from_chalk_env("CHALK_CLIENT_SECRET"), ], volumes=[("risk-models", "/app/models")], min_replicas=0, max_replicas=1, ) def review(transaction_id: str) -> str: from langchain_openai import ChatOpenAI from langchain.agents import AgentExecutor, create_react_agent from langchain_core.prompts import PromptTemplate from risk_tool import score_transaction llm = ChatOpenAI(model="gpt-4o", temperature=0) prompt = PromptTemplate.from_template( "You are a fraud analyst assistant. Use the score_transaction tool " "to evaluate transactions when asked.\n\n" "Tools: {tools}\nTool names: {tool_names}\n\n" "Question: {input}\n{agent_scratchpad}" ) agent = create_react_agent(llm, [score_transaction], prompt) executor = AgentExecutor(agent=agent, tools=[score_transaction], verbose=True) result = executor.invoke({ "input": f"Score transaction {transaction_id} for fraud risk." }) return result["output"] ``` ### Deploy the function ``` from chalkcompute import Volume vol = Volume("risk-models") vol.put_file("risk_model.pt", open("risk_model.pt", "rb").read()) from fraud_review import review print(review.remote("txn_8a3f2c")) ``` Query the agent: ``` from fraud_review import review print(review.remote("txn_8a3f2c")) # Transaction txn_8a3f2c: risk score 92.31% EXCEEDS threshold. Escalated to review queue. ``` # OCR from Text Messages source: https://docs.chalk.ai/docs/compute/ocr-pipeline ## Process images received via SMS by running OCR in a remote function. ### What we're building A webhook-driven pipeline that receives inbound text messages (via Twilio), downloads the attached image, runs OCR to extract text, and replies with the result. The heavy lifting — downloading, OCR, and replying — happens in a remote function that scales independently from the webhook receiver. This is a common pattern for SMS-based workflows: receipt scanning, document intake, ID verification, or any case where users send photos and expect a text reply. ### The OCR worker The worker function receives a message payload, downloads the image from the MMS URL, runs OCR with Tesseract, and sends the extracted text back via Twilio. ### Define the OCR function Tesseract needs system packages, so we bake them into the image: ``` # ocr_worker.py import json import os import chalkcompute from chalkcompute import Image, Secret, Volume ocr_image = ( Image.base("python:3.12-slim") .run_commands( "apt-get update -qq && apt-get install -y -qq tesseract-ocr libtesseract-dev", ) .pip_install(["pytesseract", "Pillow", "httpx", "twilio"]) ) ``` The volume at /archive stores every processed image and its extracted text — useful for auditing or reprocessing later. ### Function implementation ``` @chalkcompute.function( name="ocr-image", image=ocr_image, cpu="2", memory="4Gi", min_replicas=0, max_replicas=8, concurrency=4, secrets=[ Secret.from_chalk_env("TWILIO_ACCOUNT_SID"), Secret.from_chalk_env("TWILIO_AUTH_TOKEN"), Secret.from_chalk_env("TWILIO_PHONE_NUMBER"), ], volumes=[("ocr-archive", "/archive")], ) def ocr_image(message_json: str) -> str: """Download an MMS image, run OCR, archive the result, and reply via SMS.""" import io import httpx import pytesseract from PIL import Image as PILImage from twilio.rest import Client from datetime import datetime, timezone message = json.loads(message_json) image_url = message["image_url"] from_number = message["from_number"] message_sid = message["message_sid"] # 1. Download the image from Twilio's media URL. # Twilio media URLs require basic auth with your account credentials. resp = httpx.get( image_url, auth=(os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"]), follow_redirects=True, timeout=30, ) resp.raise_for_status() img = PILImage.open(io.BytesIO(resp.content)) # 2. Run OCR. extracted_text = pytesseract.image_to_string(img).strip() if not extracted_text: extracted_text = "(no text detected in image)" # 3. Archive the image and result to the volume. timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S") archive_dir = f"/archive/{from_number}/{timestamp}_{message_sid}" os.makedirs(archive_dir, exist_ok=True) img.save(os.path.join(archive_dir, "image.png")) with open(os.path.join(archive_dir, "text.txt"), "w") as f: f.write(extracted_text) # 4. Reply via SMS with the extracted text. twilio = Client( os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"], ) # Truncate to SMS-friendly length. reply_body = extracted_text[:1500] if len(extracted_text) > 1500: reply_body += "\n\n(truncated)" twilio.messages.create( body=f"Extracted text:\n\n{reply_body}", from_=os.environ["TWILIO_PHONE_NUMBER"], to=from_number, ) return f"{message_sid}:ok:{len(extracted_text)} chars" ``` Deploy the worker: ``` python ocr_worker.py ``` ### The webhook receiver Twilio sends an HTTP POST to your webhook URL whenever a text message arrives. The receiver parses the Twilio payload, checks for an image attachment, and dispatches the OCR function. ``` # webhook.py import json import logging from fastapi import FastAPI, Form from fastapi.responses import Response import chalkcompute logging.basicConfig(level=logging.INFO) log = logging.getLogger(__name__) app = FastAPI() ocr_fn = chalkcompute.RemoteFunction.from_name("ocr-image") @app.on_event("startup") def on_startup() -> None: ocr_fn.wait_ready() log.info("OCR worker ready") @app.post("/sms") async def handle_sms( From: str = Form(...), Body: str = Form(""), NumMedia: str = Form("0"), MessageSid: str = Form(""), MediaUrl0: str = Form(None), MediaContentType0: str = Form(None), ) -> Response: """Handle an inbound Twilio SMS/MMS webhook.""" if int(NumMedia) == 0 or MediaUrl0 is None: # No image attached — return a TwiML response. return Response( content=( '' "Send me a photo and I'll extract the text from it." ), media_type="application/xml", ) if not MediaContentType0.startswith("image/"): return Response( content=( '' "I can only process images, not other file types." ), media_type="application/xml", ) # Dispatch to the OCR worker — don't block the webhook response. payload = json.dumps({ "image_url": MediaUrl0, "from_number": From, "message_sid": MessageSid, "body": Body, }) ocr_fn.defer(payload) log.info("Dispatched OCR for message %s from %s", MessageSid, From) # Return empty TwiML — the worker will send the reply directly. return Response( content='', media_type="application/xml", ) @app.get("/health") def health() -> dict: return {"status": "ok"} ``` The webhook returns immediately with empty TwiML. The OCR worker handles the reply asynchronously via the Twilio API, so users don't see a timeout even if OCR takes several seconds. ### Deploy the webhook ``` # deploy_webhook.py from chalkcompute import Image, ScalingGroup, Secret image = ( Image.base("python:3.12-slim") .pip_install(["fastapi", "uvicorn", "chalkcompute", "python-multipart"]) .add_local_file("webhook.py", "/app/webhook.py") ) scaling_group = ScalingGroup( image=image, name="sms-ocr-webhook", port=8000, min_replicas=0, max_replicas=1, cpu="0.5", memory="1Gi", secrets=[ Secret.from_chalk_env("TWILIO_AUTH_TOKEN"), ], entrypoint=[ "uvicorn", "webhook:app", "--host", "0.0.0.0", "--port", "8000", "--app-dir", "/app", ], ).deploy() print(f"Webhook URL: {scaling_group.web_url}/sms") print("Set this as your Twilio phone number's webhook URL.") ``` ``` python deploy_webhook.py ``` Copy the /sms URL into your Twilio phone number's messaging webhook configuration. ### How it fits together ``` User's phone Twilio Webhook ScalingGroup OCR Workers │ │ │ │ │ MMS (photo) │ │ │ │──────────────────▸│ │ │ │ │ POST /sms │ │ │ │ (From, MediaUrl0) │ │ │ │──────────────────────▸│ │ │ │ │ │ │ │ empty TwiML │ │ │ │◂──────────────────────│ │ │ │ │ │ │ │ │ defer(msg) │ │ │ │───────────────────▸│ │ │ │ │ │ │ │ │── download image │ │ │ │── tesseract OCR │ │ │ │── archive to Volume │ │ │ │ │ │ twilio.messages.create(reply) │ │ │◂───────────────────────────────────────────│ │ │ │ │ │ SMS reply with │ │ │ │ extracted text │ │ │ │◂──────────────────│ │ │ │ │ │ │ ``` ### Browsing the archive Every processed image is saved to the volume, organized by phone number and timestamp. You can browse or reprocess them from any script: ``` from chalkcompute import Volume vol = Volume(name="ocr-archive") # List recent OCR results for a number. for entry in vol.listdir("+15551234567/"): print(entry.path) # Read extracted text from a specific run. text = vol.read_file("+15551234567/20260410_143022_SM1234abc/text.txt") print(text.decode()) ``` # Deep Coding Agents source: https://docs.chalk.ai/docs/compute/deep-coding-agents ## Run coding agents like OpenCode in persistent Chalk scaling groups. ### Overview Deep coding agents — long-running AI sessions that clone a repo, explore the codebase, and write code — need more compute than a laptop and a stable environment that survives disconnects. Chalk Compute lets you deploy a scaling group, install your agent, and connect from a browser or terminal. This tutorial walks through deploying OpenCode in a Chalk scaling group. The same pattern works for any agent that runs as a server process (Aider, Continue, etc.). ### Define the image Build an image with curl, git, and opencode baked in. Chalk caches the built image, so subsequent launches skip the install step entirely. ``` from chalkcompute import Image, ScalingGroup opencode_image = ( Image.base("python:3.12-slim") .run_commands( "apt-get update -qq && apt-get install -y -qq curl git", "curl -fsSL https://opencode.ai/install | bash", ) ) ``` ### Write the deploy script Create a file called deploy_opencode.py: ``` import time from chalkcompute import Image, ScalingGroup opencode_image = ( Image.base("python:3.12-slim") .run_commands( "apt-get update -qq && apt-get install -y -qq curl git", "curl -fsSL https://opencode.ai/install | bash", ) ) OPENCODE_PORT = 4096 scaling_group = ScalingGroup( image=opencode_image, name="opencode-server", env={ "OPENAI_API_KEY": "sk-...", # your LLM provider key "GH_TOKEN": "ghp_...", # for private repos }, port=OPENCODE_PORT, min_replicas=0, max_replicas=1, entrypoint=[ "bash", "-c", "git clone --depth 1 https://github.com/your-org/your-repo.git /root/code" " && /root/.opencode/bin/opencode serve" " --hostname=0.0.0.0" f" --port={OPENCODE_PORT}", ], ).deploy() print(f"Web UI: {scaling_group.web_url}") print("Press Ctrl-C to stop.") try: time.sleep(43200) # 12 hours except KeyboardInterrupt: pass finally: scaling_group.delete() ``` ### Deploy it ``` python deploy_opencode.py ``` The script builds the image (first run only), starts the scaling group, and prints a URL you can open in your browser. The coding agent is now running with full cloud compute behind it. ### Manage a running scaling group You can reconnect to a deployed scaling group by name: ``` from chalkcompute import ScalingGroup # Reconnect to the deployed scaling group scaling_group = ScalingGroup.from_name("opencode-server") print(scaling_group.web_url) # Delete when done scaling_group.delete() ``` ### Pass files with volumes Use a Volume to share configuration or model files with the scaling group without baking them into the image: ``` from chalkcompute import Image, ScalingGroup, Volume vol = Volume(name="agent-config") vol.put_file("opencode.json", '{"model": "claude-sonnet-4-20250514", "provider": "anthropic"}') scaling_group = ScalingGroup( image=Image.base("python:3.12-slim").run_commands( "apt-get update -qq && apt-get install -y -qq curl git", "curl -fsSL https://opencode.ai/install | bash", ), name="opencode-with-config", port=4096, min_replicas=0, max_replicas=1, volumes=[("agent-config", "/root/.config/opencode")], entrypoint=["bash", "-c", "/root/.opencode/bin/opencode serve --hostname=0.0.0.0 --port=4096"], ).deploy() ``` The volume is mounted at /root/.config/opencode so the agent picks up your configuration on startup. # Webhook Agent Orchestrator source: https://docs.chalk.ai/docs/compute/webhook-agent-orchestrator ## Build a scaling group that listens for webhooks, spawns coding agents, and tracks state in SQLite. ### What we're building A single long-running scaling group that: - Exposes an HTTP endpoint for incoming webhooks (GitHub, Linear, Slack, etc.). - On each webhook, creates an agent session via the Chalk Controller API. - Tracks every agent's lifecycle — created, running, completed, failed — in a SQLite database persisted on a Volume. This pattern is useful when you want a lightweight orchestration layer that turns external events into autonomous coding sessions without running a full queue system. ### Secrets This orchestrator needs four secrets. The scaling group won't start if any of them are missing. See Secrets for the full Secret API. ``` from chalkcompute import ScalingGroup, Image, Secret, Volume scaling_group = ScalingGroup( image=image, secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("GITHUB_WEBHOOK_SECRET"), Secret.from_chalk_env("CONTROLLER_URL"), Secret.from_chalk_env("CONTROLLER_API_KEY"), ], # ... ) ``` ### Project layout ``` webhook-orchestrator/ ├── orchestrator.py # FastAPI app: webhook handler + agent tracker └── deploy.py # Chalk scaling group definition ``` We'll walk through each file, then deploy the whole thing with a single command. ### The orchestrator The orchestrator is a small FastAPI app. It does three things: initializes a SQLite database, accepts webhook POST requests, and calls the Chalk Controller to spawn agent sessions. ### Database setup We store agent state in SQLite on a mounted volume. This means the data survives scaling group restarts — if the orchestrator crashes at 3 AM, it comes back up and knows exactly which agents are still running. ``` # orchestrator.py import sqlite3 import os import httpx from datetime import datetime, timezone from contextlib import contextmanager from fastapi import FastAPI, Request, HTTPException DB_PATH = os.environ.get("DB_PATH", "/data/agents.db") CONTROLLER_URL = os.environ["CONTROLLER_URL"] CONTROLLER_API_KEY = os.environ.get("CONTROLLER_API_KEY", "") app = FastAPI() def init_db() -> None: with _db() as conn: conn.execute(""" CREATE TABLE IF NOT EXISTS agents ( id TEXT PRIMARY KEY, webhook_event TEXT NOT NULL, repo TEXT NOT NULL, prompt TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'pending', session_id TEXT, task_id TEXT, created_at TEXT NOT NULL, updated_at TEXT NOT NULL ) """) @contextmanager def _db(): conn = sqlite3.connect(DB_PATH) conn.row_factory = sqlite3.Row try: yield conn conn.commit() finally: conn.close() ``` Nothing exotic here — a single agents table that tracks the mapping from webhook event to controller session. ### Webhook handler When a webhook arrives, we parse it, insert a row, and kick off an agent session in the background. The endpoint returns 200 immediately so the webhook source doesn't time out. ``` import uuid import asyncio @app.on_event("startup") def on_startup() -> None: init_db() @app.post("/webhook") async def handle_webhook(request: Request) -> dict: payload = await request.json() # Extract the relevant fields — adapt this to your webhook source. repo = payload.get("repository", {}).get("full_name", "unknown/unknown") event_type = request.headers.get("X-GitHub-Event", "unknown") prompt = _build_prompt(event_type, payload) agent_id = str(uuid.uuid4()) now = datetime.now(timezone.utc).isoformat() with _db() as conn: conn.execute( "INSERT INTO agents (id, webhook_event, repo, prompt, status, created_at, updated_at)" " VALUES (?, ?, ?, ?, 'pending', ?, ?)", (agent_id, event_type, repo, prompt, now, now), ) # Fire-and-forget: spawn the agent without blocking the response. asyncio.create_task(_spawn_agent(agent_id, repo, prompt)) return {"agent_id": agent_id, "status": "pending"} ``` ### Prompt construction Different webhook events need different instructions. A new issue gets an investigative prompt; a PR review comment gets a targeted fix prompt. ``` def _build_prompt(event_type: str, payload: dict) -> str: if event_type == "issues" and payload.get("action") == "opened": issue = payload["issue"] return ( f"Investigate GitHub issue #{issue['number']}: {issue['title']}\n\n" f"{issue.get('body', 'No description provided.')}\n\n" "Diagnose the root cause and open a PR with a fix." ) if event_type == "issue_comment": comment = payload["comment"] issue = payload["issue"] return ( f"Respond to comment on #{issue['number']}: {issue['title']}\n\n" f"Comment by @{comment['user']['login']}:\n{comment['body']}\n\n" "Address the feedback and push an updated commit." ) return f"Handle {event_type} event for this repository." ``` ### Spawning the agent This is where the orchestrator calls the Chalk Controller. It creates a session (which provisions a workspace), then submits a task with the generated prompt. As each step completes, it updates the SQLite row. ``` async def _spawn_agent(agent_id: str, repo: str, prompt: str) -> None: try: async with httpx.AsyncClient( base_url=CONTROLLER_URL, headers={"Authorization": f"Bearer {CONTROLLER_API_KEY}"}, timeout=120, ) as client: # Step 1: Create a session (provisions a workspace with the repo). resp = await client.post("/api/v1/sessions", json={ "description": f"Webhook agent for {repo}", "repo": repo, }) resp.raise_for_status() session = resp.json() session_id = session["id"] _update_agent(agent_id, status="session_created", session_id=session_id) # Step 2: Submit the task. resp = await client.post(f"/api/v1/sessions/{session_id}/tasks", json={ "prompt": prompt, "model": "claude-sonnet-4-20250514", }) resp.raise_for_status() task = resp.json() _update_agent(agent_id, status="running", task_id=task["id"]) except Exception as e: _update_agent(agent_id, status="failed") raise def _update_agent(agent_id: str, status: str, session_id: str | None = None, task_id: str | None = None) -> None: now = datetime.now(timezone.utc).isoformat() with _db() as conn: conn.execute( "UPDATE agents SET status = ?, session_id = COALESCE(?, session_id)," " task_id = COALESCE(?, task_id), updated_at = ? WHERE id = ?", (status, session_id, task_id, now, agent_id), ) ``` ### Status endpoint A simple GET endpoint lets you check on all agents — useful for dashboards or debugging. ``` @app.get("/agents") def list_agents(status: str | None = None) -> list[dict]: with _db() as conn: if status: rows = conn.execute( "SELECT * FROM agents WHERE status = ? ORDER BY created_at DESC", (status,) ).fetchall() else: rows = conn.execute( "SELECT * FROM agents ORDER BY created_at DESC" ).fetchall() return [dict(row) for row in rows] @app.get("/health") def health() -> dict: return {"status": "ok"} ``` ### The deploy script Now we wire everything together with a ScalingGroup that mounts a Volume for the database, injects secrets, and starts the FastAPI server. ``` # deploy.py from chalkcompute import ScalingGroup, Image, Secret, Volume vol = Volume(name="orchestrator-data") image = ( Image.base("python:3.12-slim") .pip_install(["fastapi", "uvicorn", "httpx"]) .add_local_file("orchestrator.py", "/app/orchestrator.py") ) scaling_group = ScalingGroup( image=image, name="webhook-orchestrator", port=8000, min_replicas=0, max_replicas=1, secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_env("GITHUB_WEBHOOK_SECRET"), Secret.from_chalk_env("CONTROLLER_URL"), Secret.from_chalk_env("CONTROLLER_API_KEY"), ], volumes=[("orchestrator-data", "/data")], entrypoint=[ "uvicorn", "orchestrator:app", "--host", "0.0.0.0", "--port", "8000", "--app-dir", "/app", ], ).deploy() print(f"Orchestrator URL: {scaling_group.web_url}") print(f"Point your GitHub webhook to: {scaling_group.web_url}/webhook") ``` ### Deploy it ``` python deploy.py # Orchestrator URL: https://d81f42e9-6ab7-4c3d-9e15-2f0a8b73c4d1.compute.chalk.ai # Point your GitHub webhook to: https://d81f42e9-6ab7-4c3d-9e15-2f0a8b73c4d1.compute.chalk.ai/webhook ``` Copy the scaling group URL with /webhook appended into your GitHub repository's webhook settings (Settings → Webhooks → Add webhook). Select the events you want to trigger agents for — typically Issues and Issue comments. ### How it fits together ``` GitHub/Linear/Slack Chalk Compute ─────────────────── ───────────── webhook POST ──────────────▸ ┌───────────────────┐ │ Orchestrator │ │ (FastAPI) │ │ │ │ SQLite on Volume │──▸ /data/agents.db └────────┬──────────┘ │ │ POST /api/v1/sessions │ POST /api/v1/sessions/:id/tasks ▼ ┌──────────────────┐ │ Chalk Controller │ │ │ │ Provisions a │ │ workspace, │ │ runs the agent │ └──────────────────┘ ``` Each webhook creates one row in SQLite and one agent session in the controller. The volume ensures the database survives restarts, and secrets keep credentials out of your code. # Model Inference with vLLM source: https://docs.chalk.ai/docs/compute/model-inference ## Serve open-weight models from Hugging Face with vLLM on a Chalk GPU, behind an authenticated, OpenAI-compatible endpoint. ### Overview This guide serves an open-weight model from Hugging Face on a GPU, using vLLM's OpenAI-compatible server. You deploy the upstream vllm/vllm-openai image as a Scaling Group and set its entrypoint to vLLM's API server. Chalk supplies the GPU, the container, and the URL. vLLM already ships the HTTP server, so you do not build an image or write serving code. The example serves Qwen3 4B Instruct, which anyone can download without a Hugging Face account. Every flag after api_server in the entrypoint is vLLM's, so you can serve another model by changing --model. To serve a model behind Chalk features and queries, see Model Deployments. ### Before you begin You need a GPU in your cluster. The example requests one NVIDIA T4. The GPU types available to you depend on the node pools in your cluster. See Resources and GPUs. ### Deploy the model Create deploy_vllm.py: ``` import chalkcompute sg = chalkcompute.ScalingGroup( image="vllm/vllm-openai:latest", name="qwen-vllm", gpu="nvidia-tesla-t4:1", cpu="4", memory="16Gi", port=8000, entrypoint=[ "python3", "-m", "vllm.entrypoints.openai.api_server", "--model", "Qwen/Qwen3-4B-Instruct-2507", "--port", "8000", "--max-model-len", "4096", "--dtype", "half", "--gpu-memory-utilization", "0.90", ], readiness_probe=chalkcompute.ReadinessProbe.http("/health"), authenticated=True, min_replicas=1, max_replicas=2, ) sg.deploy(ready_timeout=1800.0) print(sg.web_url) ``` Run it: ``` $ python deploy_vllm.py ``` deploy() returns when a replica is ready and prints the group's URL. The readiness probe reports a replica as ready once vLLM's /health endpoint responds, which happens after the model weights are loaded onto the GPU. The first deploy can take 20 minutes or more, while a GPU node starts and vLLM downloads and loads the weights, so ready_timeout is raised from its 300-second default to 30 minutes. | Parameter | Purpose | | ---------------------------------- | --------------------------------------------------------------------------------------------------- | | `gpu="nvidia-tesla-t4:1"` | Request one T4 GPU for each replica. | | `readiness_probe=...` | Send traffic to a replica only after vLLM has loaded the model. | | `authenticated=True` | Require a Chalk access token on every request. See [Securing the endpoint](#securing-the-endpoint). | | `min_replicas=1`, `max_replicas=2` | Keep one replica running, and add a second under load. See [Scaling](#scaling). | ### Requesting a GPU gpu takes a ":" string. The type selects the node pool, and the count sets the number of GPUs each replica gets. "nvidia-tesla-t4:1" gives each replica one T4, which serves a 4B parameter model at 16-bit precision. The T4 does not support bfloat16, so the example sets --dtype half. On an L4 or a newer GPU, you can set --dtype bfloat16 instead. Without gpu, the group requests no GPU, its replicas run on CPU nodes, and the vLLM image does not start. See Resources and GPUs for the GPU types and for clusters where you pass a count instead of a type. ### Serving a different model Change --model in the entrypoint: ``` entrypoint=[ "python3", "-m", "vllm.entrypoints.openai.api_server", "--model", "Qwen/Qwen3-8B", "--port", "8000", "--max-model-len", "8192", ], ``` Chalk passes the list through as the container's entrypoint, so every flag vLLM's server accepts works here. Choose the GPU, --max-model-len, and --gpu-memory-utilization to fit the model. A model too large for the GPU fails at startup with an out-of-memory error while loading weights. ### Serving a gated model Some models, including the Gemma and Llama families, are gated: Hugging Face serves their weights only to accounts that have accepted the model's license. To serve one: - Accept the license on the model's Hugging Face page, and create an access token for the same account. - In the Chalk dashboard, go to Integrations → Secrets & Variables and add a secret named HF_TOKEN with the token as its value. - Pass the secret to the group, which makes it available to vLLM: secrets=[chalkcompute.Secret.from_chalk_env("HF_TOKEN")], To use a token from your shell instead of the dashboard, write Secret.from_local_env("HF_TOKEN"), which stores the value as a Chalk secret when you deploy. If vLLM cannot download a gated model, it exits with an authorization error from huggingface.co and the group never becomes ready. The replica logs show the error. ### Query the endpoint vLLM exposes an OpenAI-compatible API under /v1. Because the group sets authenticated=True, every request needs a Chalk access token. Put the group's URL and a token in your shell: ``` $ export WEB_URL= $ export CHALK_TOKEN=$(chalk token get) ``` The OpenAI client sends its api_key as a bearer token, so pass the Chalk token there: ``` import os from openai import OpenAI client = OpenAI( base_url=f"{os.environ['WEB_URL']}/v1", api_key=os.environ["CHALK_TOKEN"], ) response = client.chat.completions.create( model="Qwen/Qwen3-4B-Instruct-2507", messages=[{"role": "user", "content": "Explain feature stores in two sentences."}], ) print(response.choices[0].message.content) ``` The model value must match the --model string in the entrypoint, because vLLM serves the model under that name. With curl: ``` $ curl "$WEB_URL/v1/chat/completions" \ -H "Authorization: Bearer $CHALK_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-4B-Instruct-2507", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ### Scaling min_replicas and max_replicas set the range the autoscaler works in. Each replica runs on its own GPU. min_replicas=1 keeps one replica, and its GPU, running while the group is idle. To have the group scale to zero when traffic stops, set min_replicas=0. While the group starts a replica, requests fail unless you set a scale-from-zero request policy. See Autoscaling for scale to zero, the request policy, and GPU utilization targets. ### Securing the endpoint A scaling group's URL is public by default: the gateway in front of it is the cluster's internet-facing load balancer, so anyone with the URL can reach your GPU. Protect the group on its first deploy, with a Chalk access token or a vLLM API key. Private routing, which keeps a group off the public internet, requires setup with Chalk. To use it, contact Chalk support. ### Require a Chalk access token The example sets authenticated=True. The gateway in front of the group then requires a Chalk access token on every request, on every path, before the request reaches vLLM. See Authentication for how tokens work. ### Set a vLLM API key vLLM's server reads VLLM_API_KEY from its environment and requires it as a bearer token. To use a key instead of a Chalk access token, remove authenticated=True and add the key to secrets: ``` secrets=[chalkcompute.Secret.from_chalk_env("VLLM_API_KEY")], ``` Callers then send Authorization: Bearer $VLLM_API_KEY, and the OpenAI client takes the key as its api_key. The key protects only these path prefixes: /v1, /v2, /inference, and /cohere. Other paths do not require it. That includes POST /invocations, which runs the same inference as /v1/chat/completions. vLLM's security guidance says: "Do not rely exclusively on --api-key for securing access to vLLM." A request carries one Authorization header, which holds either a Chalk access token or a vLLM key. Set authenticated=True or VLLM_API_KEY, not both. # Images source: https://docs.chalk.ai/docs/compute/images ## Build and cache container images for compute workloads. ### Overview Every sandbox runs from an OCI container image. You can reference a pre-built image by tag, or build one declaratively using the Image API. Images are content-addressed — the build service hashes the full specification (base image, installed packages, file contents) and skips any build whose hash already exists in the cache. Rebuilds after a minor dependency change typically complete in seconds. ### Building images The Image class exposes a fluent, method-chaining API. Each method returns a new Image instance, so intermediate images can be shared and extended without mutation. ``` from chalkcompute import Image img = ( Image.debian_slim("3.12") .pip_install(["requests", "pandas>=2.0"]) .run_commands("apt-get update && apt-get install -y git curl") .workdir("/app") .env({"PROJECT_ROOT": "/app"}) ) ``` ### Base images | Constructor | Description | | ----------------------------------- | -------------------------------------------------------------------------------- | | `Image.debian_slim(python_version)` | `python:-slim-bookworm` — lightweight default for Python workloads | | `Image.base(image)` | Arbitrary OCI image reference (e.g. `"node:22-slim"`, `"ghcr.io/org/image:tag"`) | | `Image.from_dockerfile(path)` | Parse an existing Dockerfile and continue chaining on top | ### Installing dependencies Python packages — use pip_install with a list of requirement specifiers, or point at an existing requirements file: ``` img = Image.debian_slim().pip_install(["torch==2.3.0", "transformers"]) # Or from a requirements file img = Image.debian_slim().pip_install_from_requirements("requirements.txt") ``` System packages — use run_commands for apt, apk, or any other shell command that should execute during the build: ``` img = Image.debian_slim().run_commands( "apt-get update && apt-get install -y ffmpeg libsndfile1", ) ``` Raw Dockerfile instructions — for anything not covered by the builder methods: ``` img = Image.debian_slim().dockerfile_commands([ "EXPOSE 8080", "RUN useradd -m worker", "USER worker", ]) ``` ### Adding local files Local files can be included in the image via copy or volume mount: ``` # Volume mount (default) — file is uploaded at sandbox start, not baked into the image. # Faster iteration when the file changes frequently. img = Image.debian_slim().add_local_file("app.py", "/app/app.py") # Copy — inlines the file contents into the image spec. # Better for small, stable files that should be part of the cached image. img = Image.debian_slim().add_local_file( "app.py", "/app/app.py", strategy="copy" ) # Entire directories work the same way img = Image.debian_slim().add_local_dir("./src", "/app/src") ``` Local Python source — when the files you're adding are Python code that a deployed function needs to import, prefer add_local_python_source. It takes module or package names instead of paths. ``` img = Image.debian_slim().add_local_python_source("helpers", "mypkg") ``` Use add_local_file/add_local_dir for everything that isn't importable Python source, like configuration or data files. See Function dependencies in external files for how this fits nicely into function deploys. ### Content-addressed caching Image builds are content-addressed. The build service computes a deterministic hash from the full image specification — base image digest, build steps, package lists, inlined file contents, and environment variables. If that hash already exists in the image registry, the build is skipped entirely and the existing image is returned. This means: - Identical specifications across different sandboxes or users share the same cached image. - Adding a single pip package to an existing image only rebuilds the final layer, not the entire image. - Re-running a sandbox definition with no changes incurs zero build time. ### GPU images For GPU-accelerated workloads, start from a base image that includes CUDA and the driver toolkit rather than installing them during the build. NVIDIA publishes pre-built images with CUDA, cuDNN, and popular ML frameworks: ``` # PyTorch with CUDA — starts from NVIDIA's pre-built image img = ( Image.base("nvcr.io/nvidia/pytorch:24.01-py3") .pip_install(["transformers", "accelerate"]) ) # TensorFlow img = Image.base("nvcr.io/nvidia/tensorflow:24.01-tf2-py3") ``` Installing CUDA from scratch during an image build is slow and produces large layers that defeat caching. Pre-distributed images also ensure driver compatibility with the host GPU hardware. ### Image pull performance Built images are stored in a regional OCI registry close to the compute fleet. To minimize cold-start latency, Chalk maintains an on-node image cache on each host: - Warm start. If the image (by content hash) is already cached on the node, the sandbox starts immediately — no pull required. - Cold start. The first time a node sees a particular image, it pulls from the registry. For large images (multi-GB GPU images, for example) this pull may take 30–60 seconds. Subsequent sandboxes on the same node using the same image start instantly. Common base images (debian_slim, popular NVIDIA containers) are pre-warmed across the fleet. If startup latency is critical for a custom image, you can trigger a pre-warm explicitly: ``` chalk compute image warm --image ``` This distributes the image to nodes ahead of sandbox creation. ### Setting up a container registry To build images on your own EKS or GKE cluster, link a container registry and bind it to the cluster. Chalk uses the registry to build and store images for sandboxes, scaling groups and functions. ### Registry permissions Grant the following permissions on the container registry you'd like to use: AWS ECR: grant to the Chalk management role: - Account-level: ecr:GetAuthorizationToken - Repository-level: ecr:DescribeImages, ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer (pull), ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:PutImage (push) Google GAR: grant roles/artifactregistry.writer on the repository to the workload service account that runs docker image building. ### 1. Create a container registry In the team navigation (click on the Chalk logo in the top left of the page), go to Registries and create a new container registry (Link Registry). Select the registry type, provide the repository path, and link a cloud account. | Type | Example path | | ---------- | ------------------------------------------------------------- | | AWS ECR | `123456789012.dkr.ecr.us-east-1.amazonaws.com/custom/compute` | | Google GAR | `us-docker.pkg.dev/my-project/custom-images/compute` | | Azure ACR | `myregistry.azurecr.io/custom/compute` | ### 2. Bind the registry to the cluster In the team navigation click on Clusters, and select the cluster. This will open the Settings tab. Under Container Registry Binding, select the registry created in step 1 and click Save Binding. This sets the default registry used by sandboxes and functions on the cluster. The cluster also needs a volume bucket for volumes. The function queue is optional. # Resources and GPUs source: https://docs.chalk.ai/docs/workload-settings/resources-and-gpus ## Set CPU, memory, and GPU requests for sandboxes, scaling groups, and functions. CPU, memory, and GPU settings apply to sandboxes, scaling groups, and functions. All three take the same cpu, memory, and gpu parameters. ### CPU and memory The cpu and memory arguments (SDK) and the --cpu and --memory flags (scaling group CLI) set the resource requests and limits. Both the Kubernetes request and limit are set to the same value, guaranteeing your workload the resources it asks for, so it is not subject to noisy-neighbor throttling. CPU values follow Kubernetes conventions: 1 means one full core, 500m means half a core. Memory values use standard suffixes: Mi (mebibytes) and Gi (gibibytes). CPU must be 250m, 500m, or a power-of-two number of cores (1, 2, 4, 8, 16, 32, ...). Memory must be at least 2Gi per core: 512Mi for 250m, 1Gi for 500m, 2Gi for 1, 4Gi for 2, and so on. If you set CPU and omit memory, the workload gets that minimum. A sandbox: ``` from chalkcompute import Image, Sandbox sandbox = Sandbox( image=Image.debian_slim(), cpu="4", memory="16Gi", ).run() ``` A function: ``` import chalkcompute @chalkcompute.function(cpu="1", memory="2Gi") def normalize_name(name: str) -> str: return " ".join(part.capitalize() for part in name.split()) print(normalize_name.remote("ada lovelace")) # Ada Lovelace ``` A scaling group, from the CLI: ``` $ chalk scaling-group create \ --image=my-registry/my-app:v1 \ --name=my-service \ --max-replicas=1 \ --port=8080 \ --cpu=2 \ --memory=4Gi ``` If you omit CPU and memory, a sandbox gets 250m CPU and 512Mi memory. Each scaling group replica gets the same defaults, set as both the request and the limit. A sandbox's request must fit on a single host in a configured host pool. A sandbox that asks for more CPU or memory than any host provides fails to schedule with Unschedulable: no configured host pool can satisfy .... To run it, configure a host pool whose hosts are large enough for the request. ### GPUs The gpu argument (SDK) and the --gpu flag (scaling group CLI) take a type:count value, or just type or count: | Value | GPU type | GPUs | | ------------- | ------------- | ---- | | `nvidia-l4:2` | `nvidia-l4` | 2 | | `nvidia-l4` | `nvidia-l4` | 1 | | `2` | Any available | 2 | To build an image with GPU libraries, see GPU images. ### Scaling groups and functions A function runs on a scaling group, so the same rules apply to both. ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="gpu-inference", port=8080, cpu="8", memory="32Gi", gpu="nvidia-l4:1", min_replicas=1, max_replicas=4, ).deploy() ``` ``` $ chalk scaling-group create \ --image=my-registry/my-gpu-model:v1 \ --name=gpu-inference \ --max-replicas=1 \ --port=8080 \ --gpu=nvidia-tesla-t4:1 \ --cpu=4 \ --memory=16Gi ``` The type portion selects the GPU node the replicas are scheduled on, and the count sets the number of GPUs each replica gets (the nvidia.com/gpu request and limit). Chalk adds the nvidia.com/gpu toleration automatically, so replicas can schedule onto GPU-tainted nodes. Write type as a GKE accelerator name on both GCP and AWS: - GCP: the value is used as the cloud.google.com/gke-accelerator node selector, so any accelerator in your cluster's node pools works. - AWS: Chalk maps the value to Karpenter's karpenter.k8s.aws/instance-gpu-name label. The types in the table below are supported. | GPU type | GPU | | ----------------------- | ------------------- | | `nvidia-tesla-t4` | NVIDIA T4 | | `nvidia-a10g` | NVIDIA A10G | | `nvidia-l4` | NVIDIA L4 | | `nvidia-l40s` | NVIDIA L40S | | `nvidia-tesla-v100` | NVIDIA V100 | | `nvidia-tesla-a100` | NVIDIA A100 | | `nvidia-a100-80gb` | NVIDIA A100 80GB | | `nvidia-h100-80gb` | NVIDIA H100 80GB | | `nvidia-h100-mega-80gb` | NVIDIA H100 Mega | | `nvidia-h200-141gb` | NVIDIA H200 | | `nvidia-rtx-pro-6000` | NVIDIA RTX PRO 6000 | | `nvidia-b200` | NVIDIA B200 | Which types you can schedule depends on the GPU node pools configured in your cluster. To run on any available GPU node without selecting a type, pass only the count: ``` $ chalk scaling-group create \ --image=my-registry/my-gpu-model:v1 \ --name=gpu-inference \ --max-replicas=1 \ --port=8080 \ --gpu=1 ``` To request multiple GPUs, increase the count: ``` $ chalk scaling-group create \ --image=my-registry/my-training-server:v1 \ --name=multi-gpu \ --max-replicas=1 \ --port=8080 \ --gpu=nvidia-tesla-a100:4 ``` To scale on GPU utilization, see Autoscaling. ### Sandboxes A sandbox requests a GPU type at creation time. This example runs on the k8s compute class: ``` from chalkcompute import ComputeClass, Image, Sandbox sandbox = Sandbox( image="nvcr.io/nvidia/pytorch:24.01-py3", gpu="nvidia-l4", cpu="8", memory="64Gi", compute_class=ComputeClass.K8S, ).run() ``` The Chalk scheduler matches the request to a node with the appropriate GPU hardware and configures the NVIDIA device plugin and driver mounts automatically. A k8s sandbox does not support exec, so pass its workload as entrypoint when you create it. To run commands in a GPU sandbox with exec, use the host compute class. On AWS and GKE, you can also run a GPU sandbox on the default host compute class when a host pool with the same GPU type is configured. Omit compute_class or set it to ComputeClass.HOST. Each GPU host pool host provides one GPU. GKE GPU host pools support the accelerator types listed in the host pool guide. On other clouds, use ComputeClass.K8S for GPU sandboxes. # Secrets source: https://docs.chalk.ai/docs/compute/secrets ## Inject secrets into sandboxes, functions, and scaling groups without exposing them in source or images. ### Overview Secrets let you inject credentials (API keys, database URLs, tokens) into a workload without putting them in source code or baking them into an image. Pass a list of Secret references to the secrets parameter of Sandbox, ScalingGroup, or @chalkcompute.function; each one is resolved at deploy time and exposed to the workload as an environment variable. ``` import chalkcompute from chalkcompute import Secret @chalkcompute.function( secrets=[ Secret.from_chalk_env("OPENAI_API_KEY"), Secret.from_chalk_integration("prod_postgres"), ], ) def call_api(prompt: str) -> str: import os openai_api_key = os.environ["OPENAI_API_KEY"] database_url = os.environ["DATABASE_URL"] ... ``` The same secrets=[...] parameter and the same Secret constructors work identically across all three primitives. ### The Secret constructors | Constructor | Description | | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `Secret.from_chalk_env(name, *, alias=None, prefix=None)` | Reference a standalone secret by name from Chalk's managed secret store, e.g. `Secret.from_chalk_env("OPENAI_API_TOKEN")`. | | `Secret.from_chalk_integration(name, *, keys=None, aliases=None, prefix=None)` | Inject every secret associated with a named Chalk integration (e.g. `"prod_postgres"`) as env vars. | | `Secret.from_local_env(local_env_var, *, env_var_name=None)` | Read a local environment variable at deploy time and upsert it as a Chalk secret, handy for dev loops. | | `Secret.from_local_env_file(path)` | Upsert every `KEY=VALUE` line of a local `.env` file as a Chalk secret. | Secret.from_env and Secret.from_integration still work as deprecated aliases for from_chalk_env/from_chalk_integration, but new code should use the from_chalk_* names. ### Aliasing and prefixing By default, a secret is injected under an environment variable matching its Chalk secret name. Use alias to rename it, or prefix to namespace every key an integration exposes: ``` Secret.from_chalk_env("OPENAI_API_TOKEN", alias="API_KEY") # injects OPENAI_API_TOKEN's value as API_KEY Secret.from_chalk_integration("prod_postgres", prefix="PG_") # injects prod_postgres's keys as PG_HOST, PG_USER, PG_PASSWORD, etc. Secret.from_chalk_integration("prod_postgres", keys=["host", "password"]) # injects only the listed keys, instead of every key the integration exposes ``` ### Local development Secret.from_local_env and Secret.from_local_env_file resolve a value from your own machine at deploy time, upsert it as a Chalk secret, and inject it like any other secret, which is useful for iterating locally without first creating the secret through chalk secret set or the dashboard: ``` from chalkcompute import Secret Secret.from_local_env("OPENAI_API_KEY") Secret.from_local_env("OPENAI_API_KEY", env_var_name="API_KEY") # inject under a different name Secret.from_local_env_file(".env") ``` These are lazy references — resolved once, when the workload actually deploys — not evaluated at import time. # Volumes source: https://docs.chalk.ai/docs/compute/volumes ## Persistent, versioned file storage for compute workloads. ### Overview Volumes provide persistent file storage that can be shared across sandboxes. Under the hood, volumes are backed by a Rust-based FUSE driver that mounts directly into the container's filesystem. This is file-level storage, not block storage — workloads interact with volumes through normal filesystem operations, and the driver handles tiered caching and replication to object storage transparently. Volumes can be managed through the Python SDK, the CLI, or the web dashboard. ### Creating and using volumes ### CLI ``` # Create a volume chalk volume create chalk://my-data # Upload a file chalk volume cp ./local-model.bin chalk://my-data/models/latest.bin # Upload a directory chalk volume cp ./models/ chalk://my-data/models --recursive # List contents chalk volume ls chalk://my-data # Download a file chalk volume cp chalk://my-data/models/latest.bin ./downloaded-model.bin ``` ### Chalk Volume URIs A Chalk Volume URI starts with chalk://VOLUME, where VOLUME is the volume name. Everything after the next / is a path inside that volume. ``` chalk://my-data/models/latest.bin └─────┘ volume name ``` | Chalk Volume URI | Meaning | | ----------------------------------- | ------------------------------ | | `chalk://my-data` | Volume named `my-data` | | `chalk://my-data/models/latest.bin` | File inside `my-data` | | `chalk://my-data:experiment` | Fork `experiment` of `my-data` | | `chalk://my-data@3` | Version `3` of `my-data` | ### Python SDK ``` from chalkcompute import Volume vol = Volume("my-data") # Upload files vol.put_file("models/latest.bin", model_bytes) # Batch upload for efficiency with vol.batch_upload() as batch: batch.put("data/train.csv", train_csv) batch.put("data/eval.csv", eval_csv) # Read files data = vol.read_file("models/latest.bin") # List files for f in vol.listdir("data/"): print(f"{f.path} {f.size} bytes") ``` ### Mounting into sandboxes Create a sandbox with a volumes parameter to mount that volume before starting it. The mounted volume is available as an ordinary directory inside the sandbox: ``` from chalkcompute import Image, Sandbox, Volume vol = Volume("my-data") vol.put_file("inputs/message.txt", b"hello from the volume\n") sandbox = Sandbox( image=Image.debian_slim("3.12"), name="volume-reader", cpu="1", memory="2Gi", volumes=[("my-data", "/data")], ).run() result = sandbox.exec( "python3", "-c", "print(open('/data/inputs/message.txt').read())", ) print(result.stdout_text) sandbox.terminate() ``` ### Syncing a mounted volume To persist modified files, run sync from the mount to commit modifications: ``` sync /data ``` This creates a new immutable version of the mounted volume. This can be run after writing files from a shell session or long-running process. The mount also contains a control file at .chalk-volume/commit. Run it to commit the files instead of sync. The .chalk-volume directory is not saved in the volume. ``` /data/.chalk-volume/commit ``` To reload the mount with the latest committed version, run the reload control file: ``` /data/.chalk-volume/reload ``` ### Web browser Volumes are also browsable from the Chalk dashboard. You can navigate the file tree, preview file contents, and upload or download files directly from the browser. ### Architecture ### FUSE driver The volume mount is implemented as a Rust-based FUSE (Filesystem in Userspace) driver that runs inside the sandbox. The driver exposes the volume as a standard POSIX directory, so workloads don't need special libraries or APIs to read and write files — any tool that works with the filesystem works with a volume. The FUSE driver handles: - Tiered caching. Frequently accessed files are cached on the sandbox's local disk. Cold data is fetched from object storage on demand. - Transparent replication. All persisted data is durably stored in object storage (S3 or GCS, depending on your environment). Local cache is ephemeral and rebuilt automatically. ### Copy-on-write semantics Volumes use batch copy-on-write. When a sandbox writes to a mounted volume, the writes are buffered locally on the sandbox's filesystem. These local writes are not visible to other sandboxes until a commit is made to persist writes via a sync call and sandboxes reload their current state. ``` # Writes inside the sandbox are local until committed sandbox.exec("cp", "output.parquet", "/volumes/my-data/output.parquet") ``` This design avoids partial-write visibility — other consumers of the volume see a consistent snapshot, not a stream of in-progress file mutations. If the sandbox terminates before sync is called, uncommitted writes are discarded. ### Versioning and fork semantics Every commit creates an immutable snapshot of the volume's state. Past versions are retained and can be retrieved by version ID: ``` # List available versions versions = vol.versions() # Open a previous version (read-only) old = vol.at_version(versions[-2].id) data = old.read_file("models/latest.bin") ``` Because prior versions are immutable and cheaply addressable, volumes support fork semantics. You can spawn multiple sandboxes from the same volume version, let them diverge independently, and sync their results into separate version lineages — without copying the underlying data. This is particularly useful for coding agents that need to explore multiple solution paths in parallel: ``` from chalkcompute import Image, Sandbox base_version = vol.latest_version() # Fork two sandboxes from the same starting state sandbox_a = Sandbox( image=Image.debian_slim(), volumes=[vol.at_version(base_version.id).fork("sandbox_a")], ).run() sandbox_b = Sandbox( image=Image.debian_slim(), volumes=[vol.at_version(base_version.id).fork("sandbox_b")], ).run() # Each sandbox writes independently — no interference sandbox_a.exec("python", "approach_a.py") sandbox_b.exec("python", "approach_b.py") sandbox_a.terminate() sandbox_b.terminate() ``` Each sandbox's writes are isolated until explicitly synced, and the original version remains available regardless of what the forks produce. ### Setting up a volume bucket To use volumes on your own EKS or GKE cluster, link an object storage bucket and bind it to the cluster. Chalk volumes are backed by that bucket. ### Bucket permissions The following permissions are required on the bucket you'd like to use: GCP: The service account used by background persistence (configured via "Service Account Name") needs roles/storage.objectAdmin on the bucket, plus iam.serviceAccounts.signBlob on the service account. AWS: Background persistence obtains AWS credentials through IRSA. The IAM role needs s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, and s3:GetBucketCors on the bucket. ### CORS policy The Chalk dashboard uploads and downloads volume files directly from the bucket using signed URLs, so the bucket needs a CORS rule that allows: - Origin: the URL you open the dashboard at, for example https://chalk.ai - Methods: GET and PUT - Headers: Content-Type Without it, dashboard uploads and downloads are proxied via the metadata server instead of going directly to the bucket, and the cluster's volume health check reports a CORS error. To add the rule, follow Set up and view CORS configurations for GCS or Using cross-origin resource sharing for S3. ### 1. Create a cloud storage reference Create the bucket first if you haven't already. In the team navigation (click on the Chalk logo in the top left of the page), go to Cloud storage and click Link Storage. Fill in: | Field | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Storage Type | The bucket's provider: **Google Cloud Storage**, **AWS S3**, or **Azure Blob Storage**. | | URI | The bucket URI, optionally with a prefix (for example `gs://my-bucket/volumes`, `s3://my-bucket/volumes`, or `https://myaccount.blob.core.windows.net/my-container`). | | Cloud Account | The cloud account whose credentials grant access to the bucket. | ### 2. Bind the storage reference to the cluster In the team navigation click on Clusters and select the cluster to open the Settings tab. Under Volume Bucket Binding, select the cloud storage reference created in step 1 and click Save Binding. This sets the bucket the cluster uses for volumes; the cluster manager picks it up automatically. To change or remove the binding later, select a different reference and save, or click Delete Binding. # Workload Identity source: https://docs.chalk.ai/docs/workload-settings/workload-identity ## Give a sandbox, scaling group, or function a Chalk identity so code inside it can call Chalk APIs without a client ID or secret. A Chalk workload identity applies to sandboxes, scaling groups, and functions. Each one receives an identity only when you set chalk_identity=True from the SDK, or pass --chalk-identity from the scaling group CLI. ### Enabling a workload identity With chalk_identity=True, Chalk mounts a Chalk token into the container and sets CHALK_WEB_IDENTITY_TOKEN_FILE, CHALK_API_SERVER, and CHALK_ENVIRONMENT. ChalkClient and chalkcompute read these variables, so code in the container can call Chalk APIs without a client ID or secret. A sandbox: ``` from chalkcompute import Image, Sandbox sandbox = Sandbox(image=Image.debian_slim(), chalk_identity=True).run() result = sandbox.exec("sh", "-c", "echo $CHALK_WEB_IDENTITY_TOKEN_FILE") print(result.stdout_text) # /var/run/secrets/chalk/identity-token sandbox.terminate() ``` A scaling group: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, chalk_identity=True, ).deploy() ``` A function takes the same argument, @chalkcompute.function(chalk_identity=True), and passes it to the scaling group it runs on. ### OIDC federation Chalk workload identities are OIDC-compliant. If your organization runs services that accept federated tokens (e.g. an internal model registry or a secrets manager), you can configure them to trust the Chalk identity provider directly. This lets a workload authenticate to your infrastructure without static credentials: - Register the Chalk OIDC issuer as a trusted identity provider in your service. - Map the workload's sub claim to an appropriate role or policy. - The workload can then exchange its identity token for access to your service at runtime. No secrets need to be injected into the workload's environment. # Tags source: https://docs.chalk.ai/docs/workload-settings/tags ## Attach key-value tags to sandboxes, scaling groups, and functions for organization and filtering. Tags apply to sandboxes, scaling groups, and functions. A tag is an arbitrary key-value pair that you attach for organization and filtering. ### Setting tags From the SDK, pass a dictionary to tags. A sandbox: ``` from chalkcompute import Image, Sandbox sandbox = Sandbox( image=Image.debian_slim(), name="tagged-sandbox", tags={"team": "ml", "version": "2.1"}, ).run() ``` A scaling group: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="my-service", port=8080, tags={"team": "ml", "version": "2.1"}, ).deploy() ``` A function takes the same argument, @chalkcompute.function(tags={"team": "ml"}), and passes the tags to the scaling group it runs on. From the CLI, pass comma-separated key=value pairs to --tags on chalk sandbox create or chalk scaling-group create: ``` $ chalk scaling-group create \ --image=my-registry/my-app:v1 \ --name=my-service \ --max-replicas=1 \ --port=8080 \ --tags="team=ml,version=2.1" ``` ### Viewing tags A sandbox's tags appear under spec.tags in the output of chalk sandbox get --json: ``` $ chalk sandbox get --name tagged-sandbox --json ``` A scaling group's tags are applied as Kubernetes labels with the prefix chalk.ai/tag-, making them visible in standard Kubernetes tooling. # Chalk SQL source: https://docs.chalk.ai/docs/chalksql/what-is-chalk-sql ## A SQL dialect for federated queries across everything in a Chalk environment. Chalk SQL is a dialect of SQL developed within Chalk built to create federated queries across a Chalk environment. Chalk SQL is a highly efficient, easy-to-write, and expressive language that enables: - Querying across multiple SQL sources - Introspecting the online and offline store, datasets from offline queries, and many other entities across the Chalk ecosystem - Applying functions from the Chalk Function registry in ad-hoc contexts. - Integrating with existing online and offline query paths as a new form of input. Chalk SQL is distinct from .chalk.sql files. A .chalk.sql file defines a SQL file resolver, which resolves features with a SQL query. A SQL file resolver's query is written in Chalk SQL only when the file declares source: chalksql, as described in Chalk SQL resolvers. ``` WITH total_spending AS ( SELECT u.user_id AS user_id, SUM(t.amount) AS user_spending, any_value(t.random_transaction_data) AS random_transaction_data, FROM "my_postgres.public.users" u -- Query against a data source named "my_postgres" LEFT JOIN "my_bigquery.my_dataset.transactions" t -- Query against a data source named "my_bigquery" ON u.user_id = t.user_id GROUP BY u.user_id ) SELECT u.user_id AS user_id, u.embedding AS user_embedding, array_normalize(u.embedding, 2) AS l2_normalized_embedding, json_extract(ts.random_transaction_data, '$.timestamp') AS txn_timestamp, json_extract(ts.random_transaction_data, '$.metadata.country') AS country, ts.user_spending AS user_spending, FROM "my_postgres.public.users" u LEFT JOIN total_spending ts ON u.user_id = ts.user_id ``` ### The Basics Chalk SQL comprises three main components to both plan and run queries: - A custom-built transpiler from a dialect of SQL to Chalk's internal planning system. - Chalk's existing infrastructure to execute plans, which allows Chalk SQL to benefit from Native SQL drivers, static compilation of expressions, and integration with Velox, the open-source low-latency execution engine. - The Chalk Catalog, which is a culmination of various objects within a Chalk deployment. This includes features, resolvers, datasets, the online and offline store, and registered SQL sources. A Chalk Catalog exists per deployment, meaning that as an environment changes over time through feature engineering, the Chalk Catalog will also update to reflect the newest changes. This means there's no additional work to sync the catalog to pick up changes: if an additional resolver is added to a deployment, the Chalk Catalog will automatically know about it, and Chalk SQL can make queries against it. ### Referencing the Chalk Catalog The Chalk Catalog is organized much like a traditional relational database. Primarily, objects are classified under the three-level namespacing of CATALOG.SCHEMA.TABLE. Chalk Catalog tables are primarily views, meaning that they are not materialized in memory like a conventional database table. These views exist over the various components of the Chalk Catalog. When referencing these tables in your queries to Chalk SQL, you must fully qualify their name. For example, if you wish to query the keys table in the online_store schema within the chalk catalog, you should make queries like the following. ``` SELECT * FROM "chalk.online_store.keys"; ``` You can learn more about available catalogs, schemas, and tables via the Database Explorer, or by making queries to Chalk's INFORMATION_SCHEMA. ``` -- Example query to find all schemas available in the "chalk" catalog SELECT * FROM "chalk.information_schema.schemas" WHERE catalog_name = 'chalk'; ``` ### Running Chalk SQL Chalk SQL is automatically enabled and available for use in any environment that has at least one gRPC query server running. Both new and existing environments can start running Chalk SQL queries with no additional setup. Chalk SQL is primarily run in three different contexts: - Ad-hoc Chalk SQL queries stand alone and are useful for introspection or workflows that aren't directly adjacent to the main query path. These can be run via the CLI, via the Python SDK's GRPC Client, or via the SQL Explorer, Chalk's built-in SQL console on the dashboard. - Chalk SQL can be used as a replacement to the inputs field in online and offline query. Learn more about using Chalk SQL as inputs to traditional Chalk queries here. - A .chalk.sql file resolver can declare source: chalksql, in which case its query is written in Chalk SQL and compiled into the query plan rather than sent to an integration. See Chalk SQL resolvers. ### See also - Supported SQL statements - Chalk SQL function reference - Example workflows - Chalk Catalog Components - Chalk SQL as query inputs - Chalk SQL resolvers # Example Chalk SQL Workflows source: https://docs.chalk.ai/docs/chalksql/example-workflows ## Common ways to run Chalk SQL queries, from introspecting datasets to joining across sources. This page summarizes some workflows that are commonly used in Chalk SQL. ### Introspecting datasets In Chalk SQL, you can query named datasets within the chalk.datasets schema. Note that only datasets created within the last 60 days will be available to Chalk SQL. ``` from chalk.client import ChalkClient client = ChalkClient() client.offline_query( input={ "user.id": [i for i in range(1, 10)] }, output=["user.name", "user.sum_amt_txns", "user.cnt_txns"], recompute_features=True, dataset_name="my_users_dataset" ).get_data_as_polars() ``` ``` SELECT "user.id" AS id, "user.sum_amt_txns" / "user.cnt_txns" AS avg_txn_amt, "__ts__" AS created_at FROM "chalk.datasets.my_users_dataset" WHERE "user.id" >= 5 AND "user.id" <= 8; ``` ### Federating SQL over multiple data sources If you define named data sources in your Chalk environment, you can reference schemas and tables within them in Chalk SQL queries. This means Chalk SQL can be used to create queries that join across these data sources. ``` # In your Chalk project from chalk.sql import PostgreSQLSource, BigQuerySource pg = PostgreSQLSource(name="my_postgres") bq = BigQuerySource(name="my_bigquery") ``` ``` WITH txn_sum AS ( SELECT user_id, SUM(amount) AS total FROM "my_bigquery.my_dataset.transactions" -- Table "transactions" in dataset "my_dataset" GROUP BY user_id ) SELECT u.id AS user_id, u.name AS user_name, t.total AS user_spending FROM "my_postgres.my_schema.users" u -- Table "users" in schema "my_schema" JOIN txn_sum t ON u.id = t.user_id; ``` ### Views into online and offline query results When making queries to Chalk with persistence turned on, these values will be ingested into the offline store, where they can be re-used for analytical queries. Chalk SQL provides many useful views into the offline store. The query log, which is accessible via the chalk.query_log.data table, consists of information about each query. It can be hard to determine the table within the offline store that corresponds to the results of the query. Chalk SQL provides a table-valued function query_values_from_operation_ids() that performs this resolution in-house. ``` from chalk.client import ChalkClient client = ChalkClient() result = client.query_bulk( input={ "user.id": [i for i in range(1, 100)] }, output=["user.name", "user.favorite_color"], ) # In the UI or in the query meta information, an operation ID will be set. ``` ``` SELECT * FROM query_values_from_operation_ids('c012345678901234567890123') WHERE "user.id" = 1 ``` Query values are recorded for online and cron queries while the deployment has CHALK_PLANNER_PERSIST_VALUES_OFFLINE_STORE=true; offline query runs do not record them. For an offline query, use get_dataset_givens('') to view the inputs it ran with. If an operation ID has no recorded values, the function returns an error explaining which IDs are missing and how to enable persistence. You can also directly query the offline store via the offline_store catalog. ``` SELECT * FROM "offline_store.schema_name.feat_0123456789"; ``` # The SQL Explorer source: https://docs.chalk.ai/docs/chalksql/sql-explorer ## Chalk's built-in console for Chalk SQL The SQL Explorer is built into Chalk's web dashboard for exploratory and experimental Chalk SQL queries. Open it from SQL > SQL explorer. ### Database explorer Select the Databases tab in the sidebar to open the Database Explorer, an overview of the catalogs, schemas, and tables that Chalk SQL can query. Your deployment determines this catalog of items, so it changes as you make deployments to your environment. ### Worksheets Create Chalk SQL worksheets in the SQL Explorer from the Worksheets tab. Select Add new and choose a worksheet type (SQL query, Online query, or Offline query), or select Save to save changes to an existing worksheet. Worksheets are listed under Private or Shared, are tied to users, and persist across sessions, so you can revisit queries within the explorer. ### Datasources Each worksheet's toolbar includes a datasource dropdown that shows Chalk SQL by default. The default selection runs the worksheet on Chalk's query engine, which can read from and join across every catalog listed in the Database Explorer. Selecting any other datasource sends the worksheet's SQL to that datasource verbatim. Chalk does not translate the query, so write it in the selected datasource's own SQL dialect: Snowflake SQL for a Snowflake source, PostgreSQL for a Postgres source, and so on. Because the query executes inside the selected datasource, reference tables by the names the datasource itself uses, without a catalog prefix from the Database Explorer. To query across catalogs again, set the dropdown back to Chalk SQL. ### Resource groups The resource group dropdown, marked with a server icon, selects which of your environment's resource groups executes the query. Every environment has a default group. Additional resource groups run on separate compute, which keeps exploratory or analytical queries from contending with latency-sensitive workloads in another group. The selected group applies to both synchronous and asynchronous runs of the worksheet. ### Synchronous vs. asynchronous queries Run a worksheet in synchronous or asynchronous mode with the Run and Run async options, which are tailored for transactional and analytical workloads, respectively. Run waits for results from the query server: synchronous queries hit the gRPC query servers directly. Run async runs on a separate machine and persists the results: asynchronous queries create an isolated environment, which takes some time to provision. Large synchronous Chalk SQL queries can cause gRPC query servers to run out of memory, which can affect any production traffic running on that engine. Clients can set memory limits for queries to mitigate this. # SQL as Input source: https://docs.chalk.ai/docs/chalksql/sql-as-input ## Compute the inputs to a bulk online or offline query dynamically with Chalk SQL. If you need to compute the inputs to a query dynamically, you can use Chalk SQL. This can improve performance and reduce cost by avoiding extra round trips across the network. - This feature works in:ChalkGRPCClient.online_query_bulk()ChalkClient.offline_query() - You can pass input_sql= as a SQL string, instead of input=.It uses the Chalk SQL dialect.It has access to the usual Chalk SQL catalog, which includes Data Sources and zero-argument Resolvers. You can see what's available in the SQL Data Explorer. - Each column that the SQL query returns specifies an input feature.The SQL can return a __ts__ column to specify a query time. ### In online queries For comparison, let's start with a simple query that does not use SQL: ``` grpc_client = ChalkGRPCClient(...) grpc_client.online_query_bulk( input={ User.id: [1, 2], User.email: ["alice@example.com", "bob@example.com"], }, output=[ User.id, User.email, User.is_fraud, ], ) ``` This query says we know the User.id and email, and we want to know is_fraud. The known values (inputs) are passed as literal data: 1 or "alice@example.com". Because we're using the _bulk method, we can pass many input examples in one query. Conceptually the input is a table, where each column is a feature, and each row is a different example: ``` ┌─────────┬───────────────────┐ │ user.id │ user.email │ │ int32 │ varchar │ ├─────────┼───────────────────┤ │ 1 │ alice@example.com │ │ 2 │ bob@example.com │ └─────────┴───────────────────┘ ``` In any SQL implementation (not only Chalk SQL), you can build the same table with VALUES: ``` select * from (values (1, 'alice@example.com'), (2, 'bob@example.com')) as t("user.id", "user.email"); ``` Note the as t(...) alias, which renames the columns to match our features' fully-qualified names (FQNs). You can plug this SQL query into the Chalk query by replacing the input argument with input_sql: ``` grpc_client = ChalkGRPCClient(...) grpc_client.online_query_bulk( input_sql=""" select * from (values (1, 'alice@example.com'), (2, 'bob@example.com')) as t("user.id", "user.email"); """, output=[ User.id, User.email, User.is_fraud, ], ) ``` This returns the same result as the original query with hardcoded inputs. But now we can replace the SQL with something more dynamic, such as a data source. For example, to get the IDs of all users whose last name starts with "S": ``` select id as "user.id", email_address as "user.email" from "my_postgres.public.users" where last_name like 'S%' ``` (Note the as aliases, which rename each column to exactly match the feature FQN.) The overall Chalk query, using this SQL as its input, would look like this: ``` grpc_client = ChalkGRPCClient(...) grpc_client.online_query_bulk( input_sql=""" select id as "user.id", email_address as "user.email" from "my_postgres.public.users" where last_name like 'S%' """, output=[ User.id, User.email, User.is_fraud, ], ) ``` ### In offline queries This is the same as the previous example: the only difference is that it uses ChalkClient.offline_query instead of ChalkGRPCClient.online_query_bulk. ``` client = ChalkClient(...) client.offline_query( input_sql=""" select id as "user.id", email_address as "user.email" from "my_postgres.public.users" where last_name like 'S%' """, output=[ User.id, User.email, User.is_fraud, ], recompute_features=True, ) ``` ### Setting a query time Normally, online and offline queries let you specify a query time by passing an input_times or now argument. This time value is not a feature value, but is part of the overall input to the query. For example, this query specifies values for one input feature, and also a query time: ``` grpc_client = ChalkGRPCClient(...) grpc_client.online_query_bulk( input={ Thermometer.id: [2, 3], }, now=[datetime(2025, 1, 2, 12, 0, 0), datetime(2025, 1, 2, 12, 0, 0)], output=[ Thermometer.id, Thermometer.measured_at, Thermometer.degrees_fahrenheit, ], ) ``` Conceptually, the overall input to the query is this table: ``` ┌────────────────┬─────────────────────┐ │ thermometer.id │ __ts__ │ │ int32 │ timestamp │ ├────────────────┼─────────────────────┤ │ 2 │ 2025-01-02 00:00:00 │ │ 3 │ 2025-01-02 00:00:00 │ └────────────────┴─────────────────────┘ ``` So to get the same behavior with SQL as the input, you would write a SQL query that outputs two columns: thermometer.id and __ts__: ``` grpc_client = ChalkGRPCClient(...) grpc_client.online_query_bulk( input_sql=f""" select * from (values (2, '2025-1-2 12:00:00'::datetime), (3, '2025-1-2 12:00:00'::datetime) ) as t("thermometer.id", "__ts__") """, output=[ Thermometer.id, Thermometer.measured_at, Thermometer.degrees_fahrenheit, ], ) ``` And the syntax for offline query follows the same pattern: ``` client = ChalkClient(...) client.offline_query( input_sql=f""" select * from (values (2, '2025-1-2 12:00:00'::datetime), (3, '2025-1-2 12:00:00'::datetime) ) as t("thermometer.id", "__ts__") """, output=[ Thermometer.id, Thermometer.measured_at, Thermometer.degrees_fahrenheit, ], recompute_features=True, ) ``` ### Using versioned features Versioned features work as usual by appending a "@N" suffix to the fully-qualified name: ``` grpc_client.online_query_bulk( input_sql=""" select id as "user.id", email_address as "user.email@2" from "my_postgres.public.users" where last_name like 'S%' """, output=[ User.id, User.email @ 2, User.is_fraud, ], ) ``` You may also find it convenient to use str() to access the fully-qualified name instead of hardcoding it: ``` input_sql=f""" select id as "{str(User.id)}", email_address as "{str(User.email @ 2)}", from ... """, ``` # Chalk Catalog Components source: https://docs.chalk.ai/docs/chalksql/chalk-catalog-components ## How features, resolvers, datasets, and other Chalk concepts are exposed to Chalk SQL. The Chalk Catalog is an umbrella term encompassing all things Chalk. This page describes how various concepts within Chalk are translated into the Catalog and made available to Chalk SQL. Like traditional relational databases, the Catalog is organized under a three-tiered namespace system of catalogs, schemas, and tables. ### The chalk catalog Most concepts in Chalk will be exposed as tables under the chalk database catalog. ### chalk.datasets schema The chalk.datasets schema contains a view for each named dataset created within the last 60 days, and the view's name is the dataset's name. Note that if two offline queries reference the same named dataset, only the latest dataset revision is available in this schema. To reference dataset revisions that are not the most recent or were created without a name, consider using the get_dataset_revision() table-valued function. This schema is available in all Chalk deployments. ### chalk.historical_values schema The chalk.historical_values schema contains a view for each feature, as well as a view for each feature namespace within the deployment. - For each feature, a view with the same name as the feature's fully-qualified name is available in the Catalog (i.e. chalk.historical_values.user.name). This view contains information about all observations of the feature, with the following schema:observation_id, string, non-nullable: The unique identifier for the observation of the feature value.pkey, (type of feature namespace's pkey), non-nullable: The value of the primary feature associated with this observation.value, (type of feature namespace's value): The observed value of the feature.value_json, string, DEPRECATED: The observed value of the feature, as JSON, for backwards compatibility.observed_at, timestamp, non-nullable: The time at which the feature value was observed.inserted_at, timestamp, non-nullable: The time at which the feature value was inserted into the offline store.resolver_fqn, string, nullable: The resolver that computed this feature.deployment_id, string, nullable: The deployment ID where the observation was made.operation_kind, int8, nullable: The kind of operation that produced the observation.operation_id, string, nullable: The operation ID of the query that produced the observation. - For each feature namespace, a view with the same name as the namespace is available in the Catalog (i.e. chalk.historical_values.user). This view contains the freshest values of each feature for each primary key at the current point in time, and has the following schema:__pkey__: (type of feature namespace's pkey), non-nullable: The value of the primary feature associated with this row, (type of feature), for each feature in the namespace: The freshest observed value of the feature for the primary key..__observed_at__, timestamp, for each feature in the namespace: The time at which the freshest value was observed. This schema is available in Chalk deployments with an offline store configured. ### chalk.query_log schema The chalk.query_log schema consists of exactly one table, data, which is the query log table. This table provides a list of query metadata for each operation ID, including which tables in the offline store correspond to values queried in that operation ID. The data table has the following schema: - operation_id, string, non-nullable: The unique identifier for the operation. - environment_id, string, non-nullable: The environment ID where the operation was run. - deployment_id, string: The deployment ID where the operation was run. - operation_kind, string: The kind of operation that was run (inference, streaming, etc.). - query_timestamp, timestamp: The value of now() that was used by the query. - execution_started_at, timestamp: The time at which the operation started executing. Null if there were errors during planning. - execution_finished_at, timestamp: The time at which the operation finished executing. Null if there were errors during execution. - query_status, string: The status of the query (success, error, running, etc.). - query_name, string: The query name for named queries, null otherwise. - query_name_version, string: The version of the query for named queries, null otherwise. - branch_name, string: The branch the query was run on, null if the query was not run on a branch. - correlation_id, string: The correlation ID for the query, null if the query was not run with a correlation ID. - value_tables: array of strings: The list of tables in the offline store that correspond to result values from this query. This schema is available in Chalk deployments with an offline store configured. ### chalk.telemetry schema The chalk.telemetry schema provides three environment-scoped tables: - spans has one row per OpenTelemetry span. It includes query and resolver spans as well as OpenInference AI spans such as AGENT, LLM, and TOOL spans. - traces has one summary row per trace. - sessions has one summary row per agent session — the multi-turn conversation a set of traces belongs to. chalk.telemetry.spans includes these columns: - Timing and identity: start_time, end_time, duration_ns, duration_ms, trace_id, span_id, parent_span_id, is_root_span, and trace_state. - OpenTelemetry metadata: span_name, span_kind, service_name, environment_id, scope_name, scope_version, status_code, and status_message. - Complete JSON payloads: resource_attributes_json, span_attributes_json, events_json, and links_json. - AI input and output: openinference_span_kind, input_value, input_mime_type, output_value, and output_mime_type. - Agent and tool metadata: turn_iterations, tool_call_count, tool_name, tool_call_id, and session_id. - LLM metadata: llm_provider, llm_model_name, llm_system, llm_finish_reason, and llm_cost. - LLM tokens: llm_token_count_prompt, llm_token_count_cache_read, llm_token_count_cache_write, llm_token_count_completion, llm_token_count_reasoning, and llm_token_count_total. chalk.telemetry.traces includes start_time, end_time, duration_ns, duration_ms, trace_id, span_count, root_span_id, root_parent_span_id, root_span_name, root_span_attributes_json, root_resource_attributes_json, and environment_id. chalk.telemetry.sessions includes session_id, start_time, end_time, duration_ns, duration_ms, trace_count, span_count, first_trace_id, first_root_span_name, and environment_id. A session is idle between turns, so its duration is the wall-clock span it covers rather than the sum of its traces' durations. Traces with no session ID — anything that is not an agent conversation — are not counted as sessions. Join sessions to spans on session_id to reach the individual spans of a conversation. For example, this query returns recent LLM and tool-call spans: ``` SELECT start_time, trace_id, span_name, openinference_span_kind, llm_model_name, llm_token_count_total, tool_name FROM "chalk.telemetry.spans" WHERE start_time >= now()::timestamp - INTERVAL 1 DAY AND openinference_span_kind IN ('LLM', 'TOOL') ORDER BY start_time DESC LIMIT 100 ``` To find the largest recent traces: ``` SELECT start_time, trace_id, root_span_name, span_count, duration_ms FROM "chalk.telemetry.traces" WHERE start_time >= now()::timestamp - INTERVAL 1 DAY ORDER BY span_count DESC LIMIT 100 ``` To find the longest-running agent conversations: ``` SELECT session_id, start_time, first_root_span_name, trace_count, span_count, duration_ms FROM "chalk.telemetry.sessions" WHERE start_time >= now()::timestamp - INTERVAL 1 DAY ORDER BY duration_ms DESC LIMIT 100 ``` This schema is available in Chalk deployments with managed telemetry configured. Results are always restricted to the active environment. ### chalk.resolvers schema The chalk.resolvers schema contains a view for each dataframe-returning resolver that takes in no inputs. When querying this view, a resolver run will be triggered and have the resolver's outputs as the view's contents. This schema is available in all Chalk deployments. ### chalk.online_store schema The chalk.online_store schema contains a fixed view keys, which contains different information depending on which online store is configured for your environment. - For Redis online stores, information about the keys' type (Scalar, HasMany, TimeSeries, etc.), memory usage, TTL, and feature FQN are available in this view. - For DynamoDB online stores, information about the keys' namespace and value are available in this view. Additionally, for each feature namespace, a view with the same name (i.e. chalk.online_store.user) is available, consisting of the following schema: - __id__ - . for each feature - ._ts for each feature This schema is available in Chalk deployments with a Redis or DynamoDB online store configured. ### chalk.online_delta_writes schema The chalk.online_delta_writes schema shows the digest state that online delta writes (CHALK_ONLINE_DELTA_WRITES) keep for scheduled queries and scheduled aggregate backfills. Each run records a digest of every row it writes to the online store, and the next run skips rows whose digest is unchanged. Every schedule that writes to the online store appears here, with or without the setting enabled; runs made with it disabled record no digests. The schedules view has one row per schedule. A scheduled aggregate backfill has one row per resolver it backfills, told apart by backfill_child. Its columns describe the schedule's two most recent runs: - schedule_key, query_name, kind (scheduled_query or scheduled_aggregate_backfill), backfill_child - current_run_id, previous_run_id - retained_runs: runs whose digests are still in storage - current_committed_attempts, current_uncommitted_attempts: shards or retries of the current run that finished successfully, and those that recorded digests without finishing - current_digest_files, current_digest_rows: the committed digests the next run compares against - current_oldest_stored_at, current_latest_stored_at: the range of times the rows behind those digests were last written to the online store - previous_committed_attempts, previous_uncommitted_attempts, previous_digest_files, previous_digest_rows The runs view has one row per retained run, with the schedule columns above plus run_id, is_current, is_previous, committed_attempts, uncommitted_attempts and digest_files. ``` SELECT query_name, current_run_id, current_digest_rows, current_latest_stored_at FROM chalk.online_delta_writes.schedules ``` ### chalk.information_schema schema The chalk.information_schema schema describes the catalog itself. Alongside the conventional schemas (aliased as schemata), tables, functions and catalogs views, it exposes the environment's models and external functions — the targets you can invoke from F.catalog_call(...) — as models and compute_functions. ``` -- Which models can this environment call, and what do they return? SELECT model_name, qualified_name, output_type FROM "chalk.information_schema.models"; -- The same for deployed external functions. SELECT function_name, qualified_name, output_type FROM "chalk.information_schema.compute_functions"; ``` Both views share a schema: - model_name / function_name — the registered name. - qualified_name — the name to pass to F.catalog_call, i.e. model. or function.. - endpoint_uri — the gateway endpoint the call is routed to. - method_name — the method invoked on the remote service. - output_type — the SQL type the handler returns, or NULL when the deployment did not register one. These views list what the running deployment was told about. An environment whose catalog snapshot predates per-entry kind information shows entries in neither view until its next deploy. ### The offline_store catalog The offline store is available as a catalog. Similar to data sources as catalogs, this catalog works as a direct interface to querying the underlying database that powers it. ### Data Sources as Catalogs Each data source that you register with Chalk will be automatically registered in the Chalk Catalog as a database catalog. The catalog shares the same name as the data source and provides a more direct interface into querying that data source. Data sources that are not named will not be registered in the Chalk Catalog. As an example, if you define a Postgres source and a BigQuery source in your code: ``` from chalk.sql import PostgreSQLSource from chalk.sql import BigQuerySource pg = PostgreSQLSource(name='pg_source') bq = BigQuerySource(name='bq_source') ``` You can write a federated query in Chalk SQL to join a transactions table in pg with a users table in bq: ``` SELECT bq_users.user_id AS user_id, bq_users.full_name AS full_name, pg_transactions.transaction_id AS transaction_id, pg_transactions.amount AS transaction_amount FROM "bq_source.my_dataset.users" AS bq_users FULL OUTER JOIN "pg_source.public.transactions" AS pg_transactions ON bq_users.user_id = pg_transactions.user_id /* pg_source and bq_source are the name of the datasources and therefore also the name of their respective catalogs */ ``` The following data source types will be registered to the Chalk Catalog, and will be able to be queried in Chalk SQL: - PostgreSQL - Spanner - Athena - MySQL - ClickHouse - BigQuery - Redshift - Snowflake - Databricks # Getting Started source: https://docs.chalk.ai/docs/chalkdf/getting-started ## Install and run your first chalkdf commands This guide will walk you through installing chalkdf and performing some basic operations. If you are new to chalkdf or DataFrame libraries, this is a great place to start. If you are already familiar with DataFrame libraries you can skip ahead to the full Reference Library or the installation guide. ### Installation Given Python 3.11-3.13 and Linux/macOS 15+, you can install chalkdf via pip: ``` uv pip install "chalkdf[chalkpy]" ``` ### Reading Data chalkdf supports reading data from several formats and sources, including JSON, CSV, and Parquet files, Arrow objects and tables, as well as AWS Glue Iceberg tables, and SQL sources defined via chalkpy. Below we can initialize a chalkdf.DataFrame from a Python dictionary. ``` from chalkdf import DataFrame df = DataFrame( { "name": ["Alice", "Bob", "Charlie"], "age": [25, 30, 35], "city": ["New York", "Los Angeles", "Chicago"], } ) df.run() ``` ``` DataFrame(materialized 3 rows x 3 columns) ┌─────────┬───────┬─────────────┐ │ name ┆ age ┆ city │ │ ─────── ┆ ───── ┆ ─────────── │ │ string ┆ int64 ┆ string │ ╞═════════╪═══════╪═════════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ Los Angeles │ │ Charlie ┆ 35 ┆ Chicago │ └─────────┴───────┴─────────────┘ ``` You can also scan files directly into a DataFrame. Below, we scan a CSV file: ``` from chalkdf import DataFrame import pyarrow as pa df = DataFrame.scan( ['people.csv'], name="people", schema=pa.schema([ ('name', pa.string()), ('birthday', pa.date32()), ('occupation', pa.string()) ]) ) df.run() ``` ``` DataFrame(materialized 3 rows x 3 columns) ┌───────────────┬─────────────┬───────────────────┐ │ name ┆ birthday ┆ occupation │ │ ───────────── ┆ ─────────── ┆ ───────────────── │ │ string ┆ date32[day] ┆ string │ ╞═══════════════╪═════════════╪═══════════════════╡ │ Alice Chen ┆ 1990-03-15 ┆ Software Engineer │ │ Bob Smith ┆ 1985-07-22 ┆ Teacher │ │ Carol Johnson ┆ 1992-11-08 ┆ Data Analyst │ └───────────────┴─────────────┴───────────────────┘ ``` ### DataFrame Expressions DataFrame expressions allow you to transform and explore your data through a variety of operations. For example, given a DataFrame of values with IDs, you could group the values by id and compute the aggregate sum. We use the underscore notation _ to reference the current DataFrame, to easily access columns without any additional wrappers. You can import _ from the chalk module. ``` >>> from chalk import _ >>> df.run() DataFrame(materialized 5 rows x 2 columns) ┌───────┬───────┐ │ id ┆ value │ │ ───── ┆ ───── │ │ int64 ┆ int64 │ ╞═══════╪═══════╡ │ 1 ┆ 10 │ │ 1 ┆ 20 │ │ 2 ┆ 1 │ │ 2 ┆ 2 │ │ 2 ┆ 3 │ └───────┴───────┘ ``` ``` >>> from chalk.features import _ >>> grouped = df.agg(["id"], _.value.sum().alias("value_sum_by_id")) >>> grouped.run() DataFrame(materialized 2 rows x 2 columns) ┌───────┬─────────────────┐ │ id ┆ value_sum_by_id │ │ ───── ┆ ─────────────── │ │ int64 ┆ int64 │ ╞═══════╪═════════════════╡ │ 1 ┆ 30 │ │ 2 ┆ 6 │ └───────┴─────────────────┘ ``` Below are some common operations you can use to construct DataFrame expressions. ### select DataFrame.select allows you to extract specific columns from a DataFrame. ``` >>> interactions = df.select("user_id", "item_id", "interaction_type", "timestamp") >>> interactions.run() DataFrame(materialized 5 rows x 4 columns) ┌──────────┬───────────┬──────────────────┬─────────────────────┐ │ user_id ┆ item_id ┆ interaction_type ┆ timestamp │ │ ──────── ┆ ───────── ┆ ──────────────── ┆ ─────────────────── │ │ string ┆ string ┆ string ┆ timestamp[us] │ ╞══════════╪═══════════╪══════════════════╪═════════════════════╡ │ user_001 ┆ item_4521 ┆ click ┆ 2024-11-08 14:23:15 │ │ user_002 ┆ item_8832 ┆ purchase ┆ 2024-11-08 15:10:42 │ │ user_003 ┆ item_1203 ┆ view ┆ 2024-11-08 16:05:30 │ │ user_004 ┆ item_4521 ┆ add_to_cart ┆ 2024-11-08 17:45:12 │ │ user_005 ┆ item_9944 ┆ click ┆ 2024-11-08 18:20:05 │ └──────────┴───────────┴──────────────────┴─────────────────────┘ ``` ### with_columns To add new columns or modify existing columns in your DataFrame while retaining all original columns, you can use DataFrame.with_columns. ``` >>> from chalk import _ >>> processed_interactions = df.with_columns({"user_id": _.user_id, "long_session": _.session_duration_sec > 180}) >>> processed_interactions.select("user_id", "session_duration_sec", "long_session").run() DataFrame(materialized 5 rows x 3 columns) ┌──────────┬──────────────────────┬──────────────┐ │ user_id ┆ session_duration_sec ┆ long_session │ │ ──────── ┆ ──────────────────── ┆ ──────────── │ │ string ┆ int64 ┆ bool │ ╞══════════╪══════════════════════╪══════════════╡ │ user_001 ┆ 145 ┆ False │ │ user_002 ┆ 320 ┆ True │ │ user_003 ┆ 78 ┆ False │ │ user_004 ┆ 210 ┆ True │ │ user_005 ┆ 167 ┆ False │ └──────────┴──────────────────────┴──────────────┘ ``` ### project To define more complex transformations without retaining all original columns, you can use DataFrame.project to take in a DataFrame as input and project to new columns based on provided expressions. In these expressions, you can use the underscore notation _ to reference columns within your source DataFrame, and utilize the chalk.functions library (typically imported as F) for a variety of operations. ``` from chalk import functions as F, _ from chalkdf import DataFrame import pyarrow as pa tbl = pa.table( { "txns_last_hour": [[1, 2, 3, 4, 5], [100], [200, 201]], "max_txns_allowed": [3, 5, 4], } ) df = DataFrame.from_arrow(tbl) out = df.project( { "velocity_score": _.txns_last_hour .cardinality() .cast(float) .least(_.max_txns_allowed + 0.5) .ceil() .cast(int), "velocity_score_2": F.cast( F.ceil( F.least( F.cast(F.cardinality(_.txns_last_hour), float), _.max_txns_allowed + 0.5 ) ), int, ), } ) out.run() ``` ``` DataFrame(materialized 3 rows x 2 columns) ┌────────────────┬──────────────────┐ │ velocity_score ┆ velocity_score_2 │ │ ────────────── ┆ ──────────────── │ │ int64 ┆ int64 │ ╞════════════════╪══════════════════╡ │ 4 ┆ 4 │ │ 1 ┆ 1 │ │ 2 ┆ 2 │ └────────────────┴──────────────────┘ ``` ### filter You can filter rows in a DataFrame using DataFrame.filter, which takes in a boolean expression. ``` >>> from chalk import _ >>> df.run() DataFrame(materialized 5 rows x 7 columns) ┌──────────┬───────────┬──────────────────┬─────────────────────┬────────┬─────────────┬──────────────────────┐ │ user_id ┆ item_id ┆ interaction_type ┆ timestamp ┆ score ┆ category ┆ session_duration_sec │ │ ──────── ┆ ───────── ┆ ──────────────── ┆ ─────────────────── ┆ ────── ┆ ─────────── ┆ ──────────────────── │ │ string ┆ string ┆ string ┆ timestamp[us] ┆ double ┆ string ┆ int64 │ ╞══════════╪═══════════╪══════════════════╪═════════════════════╪════════╪═════════════╪══════════════════════╡ │ user_001 ┆ item_4521 ┆ click ┆ 2024-11-08 14:23:15 ┆ 0.85 ┆ electronics ┆ 145 │ │ user_002 ┆ item_8832 ┆ purchase ┆ 2024-11-08 15:10:42 ┆ 0.92 ┆ fashion ┆ 320 │ │ user_003 ┆ item_1203 ┆ view ┆ 2024-11-08 16:05:30 ┆ 0.67 ┆ home ┆ 78 │ │ user_004 ┆ item_4521 ┆ add_to_cart ┆ 2024-11-08 17:45:12 ┆ 0.78 ┆ electronics ┆ 210 │ │ user_005 ┆ item_9944 ┆ click ┆ 2024-11-08 18:20:05 ┆ 0.81 ┆ sports ┆ 167 │ └──────────┴───────────┴──────────────────┴─────────────────────┴────────┴─────────────┴──────────────────────┘ ``` ``` >>> df.filter(_.score > 0.8).run() DataFrame(materialized 3 rows x 7 columns) ┌──────────┬───────────┬──────────────────┬─────────────────────┬────────┬─────────────┬──────────────────────┐ │ user_id ┆ item_id ┆ interaction_type ┆ timestamp ┆ score ┆ category ┆ session_duration_sec │ │ ──────── ┆ ───────── ┆ ──────────────── ┆ ─────────────────── ┆ ────── ┆ ─────────── ┆ ──────────────────── │ │ string ┆ string ┆ string ┆ timestamp[us] ┆ double ┆ string ┆ int64 │ ╞══════════╪═══════════╪══════════════════╪═════════════════════╪════════╪═════════════╪══════════════════════╡ │ user_001 ┆ item_4521 ┆ click ┆ 2024-11-08 14:23:15 ┆ 0.85 ┆ electronics ┆ 145 │ │ user_002 ┆ item_8832 ┆ purchase ┆ 2024-11-08 15:10:42 ┆ 0.92 ┆ fashion ┆ 320 │ │ user_005 ┆ item_9944 ┆ click ┆ 2024-11-08 18:20:05 ┆ 0.81 ┆ sports ┆ 167 │ └──────────┴───────────┴──────────────────┴─────────────────────┴────────┴─────────────┴──────────────────────┘ ``` ### agg To compute aggregations over groups of data, you can use DataFrame.agg. ``` >>> processed_interactions.agg( ... ["long_session"], ... processed_interactions.column("score").mean().alias("avg_score") ... ).run() DataFrame(materialized 2 rows x 2 columns) ┌──────────────┬───────────┐ │ long_session ┆ avg_score │ │ ──────────── ┆ ───────── │ │ bool ┆ double │ ╞══════════════╪═══════════╡ │ False ┆ 0.776667 │ │ True ┆ 0.85 │ └──────────────┴───────────┘ ``` ### join You can combine data from multiple DataFrames using DataFrame.join. Below is an example of joining two DataFrames using an inner join on the user_id column. You can also specify right and left joins. ``` >>> txns_df.join( ... users_df, ... on=["user_id"], ... how="inner" ... ).select( ... "transaction_id", ... "user_id", ... "name", ... "amount", ... "tier", ... "status" ... ).run() DataFrame(materialized 5 rows x 6 columns) ┌────────────────┬──────────┬─────────┬────────┬─────────┬───────────┐ │ transaction_id ┆ user_id ┆ name ┆ amount ┆ tier ┆ status │ │ ────────────── ┆ ──────── ┆ ─────── ┆ ────── ┆ ─────── ┆ ───────── │ │ string ┆ string ┆ string ┆ double ┆ string ┆ string │ ╞════════════════╪══════════╪═════════╪════════╪═════════╪═══════════╡ │ txn_101 ┆ user_001 ┆ Alice ┆ 49.99 ┆ premium ┆ completed │ │ txn_102 ┆ user_002 ┆ Bob ┆ 19.99 ┆ basic ┆ completed │ │ txn_103 ┆ user_001 ┆ Alice ┆ 89.5 ┆ premium ┆ pending │ │ txn_104 ┆ user_003 ┆ Charlie ┆ 120 ┆ premium ┆ completed │ │ txn_105 ┆ user_001 ┆ Alice ┆ 15.75 ┆ premium ┆ completed │ └────────────────┴──────────┴─────────┴────────┴─────────┴───────────┘ ``` # Installation source: https://docs.chalk.ai/docs/chalkdf/installation ## Install chalkdf to access DataFrame utilities. chalkdf is a DataFrame library that provides utilities for building fast, portable data pipelines on top of Apache Arrow -- powered by Chalk's libchalk execution engine. ### Installation To install chalkdf, check the following requirements: - Python 3.11-3.13 - Linux/macOS 15+ Install from PyPI: ``` pip install "chalkdf[chalkpy]" ``` Alternatively, download platform-specific wheels directly from PyPI. # Chalk Assistant source: https://docs.chalk.ai/docs/chalk-assistant ## An in-dashboard AI assistant that runs on your AI provider. Chalk Assistant is an agentic assistant built into the Chalk dashboard. It runs on an AI provider you connect: Chalk supplies the harness and the tools, and you supply the model. It acts on your environment through the same tool set as the Chalk MCP Server, letting you drive tasks from the UI that you can also drive from your own coding agents. ### Enabling Chalk Assistant in your environment Chalk Assistant is disabled by default. An Owner or Team Manager must enable it for an environment and configure the AI provider and model to use. Chalk Assistant supports OpenAI, Anthropic, and any OpenAI-compatible endpoint, which takes a base URL you supply. Every conversation in that environment runs on the provider and model configured there. Notes: - Your model provider can see any information about your environment carried in the conversation and in tool results. - Messages, tool arguments, and tool results are stored in your environment's data plane. - An Owner or Team Manager can view every conversation in the environment. - Every tool call runs the same permission check as the MCP Server against your account, so the assistant can change anything you can change (e.g., notebooks, monitors, Chalk Functions, sandboxes, and deployment source). - Chalk never shows the API key again after you save it. - Rotating the key takes effect immediately. ### Using Chalk Assistant Chats with the assistant open in windows along a rail at the bottom of the dashboard. You can start a chat two ways: - Click Assistant on the bottom rail. - Press ⌘K (Ctrl+K on Windows and Linux), type a question, and press Shift+Enter to open it in the Ask composer. Press Enter to send it to the assistant. You can attach objects from your environment instead of pasting their names. Click + Context to search and attach any of the following: - features, resolvers, and named queries from the active deployment, - datasets, registry models, Chalk Functions, and scaling groups, - SQL catalog tables. The assistant's tools cover queries and Chalk SQL, features and resolvers, sandboxes and notebooks, deployments, infrastructure, and observability. See the MCP Server page for more information. Clicking Assistant opens a chat window you have not used yet, or a new window if every open window already has a conversation in it. You can have up to 20 chat windows open, including minimized ones. To open a conversation as a full page, click the expand button on its window. Click the history button on the rail to see your recent chats, then View all conversations for the full list. The list shows your own conversations. ### What you can do with Chalk Assistant Chalk Assistant is designed to help you with the work that surrounds a model: - agent-driven machine learning investigations, - data exploration and analysis, - online and offline query development, - reading what a feature or resolver actually computes, - debugging a slow or failing query, - environment administration. The examples below show how it can help. ### Explore features and data Ask about the shape of your environment before you write code against it. - "Which features exist on the user namespace, and which of them are computed by SQL resolvers?" - "Show me the definition of user.fraud_score and every resolver that contributes to it." - "Run an online query for user.id=1234 and return the fraud features with their timestamps." - "Use Chalk SQL to count transactions per merchant over the last 30 days." ### Build and iterate in Chalk Notebooks The assistant can create a Chalk Notebook, add cells, install dependencies, run cells, and read the results back. Changes it makes apply immediately, so review them in the notebook afterward. - "Create a notebook that loads the last 90 days of transactions and plots daily volume." - "Add a cell that trains an XGBoost classifier on this dataframe and prints the AUC." - "This cell fails with a KeyError. Read the error and fix the cell." ### Debug queries and deployments The assistant reads the same query errors, plans, logs, and traces you would open by hand. - "The last online query for user.fraud_score returned an error. Find it and explain the cause." - "Get the query plan and performance summary for the slowest resolver in this environment." - "Search the logs for OOMKilled in the last hour and tell me which workload restarted." ### Operate the environment Ask it to inspect infrastructure and set up monitoring, then check its work before you rely on it. - "List the deployments in this environment and tell me which one is active." - "Search the deployed source for every resolver that reads from Snowflake." - "Create a monitor that alerts when online query p99 latency exceeds 200ms." ### Call your own functions Chalk Functions you have deployed are callable from a conversation, so a function already running in your environment becomes a step the assistant can take inside a longer task. - "Call the score_applicant function with this payload and summarize what it returned." ### See also - MCP Server for connecting an external agent to the same tools. - Development with LLMs for writing Chalk code with an AI assistant outside the dashboard. # MCP Server source: https://docs.chalk.ai/docs/mcp-server ## Connect AI agents to your Chalk deployment over the Model Context Protocol. Chalk hosts a Model Context Protocol (MCP) server so that AI agents (Claude, Cursor, your own LangChain or agent code, or any MCP-compatible client) can interact with your Chalk deployment directly. Through the MCP server an agent can run online queries, execute Chalk SQL, read feature and resolver definitions, search logs and traces, and inspect query errors, all scoped to the permissions of the credentials it presents. The MCP server is hosted alongside the Chalk API and requires no installation or self-hosting. The MCP Server is distinct from the MCP Gateway. The MCP Server exposes Chalk itself to agents (run queries, execute Chalk SQL, inspect your deployment). The MCP Gateway sits in front of other MCP servers and governs your agents' access to them. ### Connecting a client Most MCP clients accept a remote server URL plus a set of headers. Point your client at https://api.chalk.ai/v1/mcp/sse using its streamable-HTTP or remote-server configuration. The snippets below read your credentials from chalk config, so they work as written once you are logged in. See Authentication for where those credentials come from, and Endpoint if you are on a dedicated or self-hosted deployment. ### Claude Code The CLI registers the server for you, using the credentials you are already logged in with: ``` chalk mcp configure-claude ``` This adds Chalk in Claude Code's user scope, so it is available in every project. Pass --claude-scope local to limit it to the current directory, or --claude-scope project to write it to .mcp.json for your teammates. Start Claude Code and run /mcp to confirm the connection. Under the hood this runs claude mcp add, which you can also call yourself: ``` claude mcp add --transport http chalk https://api.chalk.ai/v1/mcp/sse \ --header "X-Chalk-Client-Id: $(chalk config --format json | jq -r .clientId.value)" \ --header "X-Chalk-Client-Secret: $(chalk config --format json | jq -r .clientSecret.value)" ``` ### Codex The CLI writes the Codex configuration for you: ``` chalk mcp configure-codex ``` This adds an [mcp_servers.chalk] block to ~/.codex/config.toml, leaving the rest of the file untouched. Codex has no per-project scope, so the server is available everywhere. Pass --env-headers to reference environment variables instead of writing your credentials into the file, and confirm the result with codex mcp list. Codex's own codex mcp add command covers stdio servers only, so you can also add Chalk by editing that file yourself: ``` [mcp_servers.chalk] url = "https://api.chalk.ai/v1/mcp/sse" env_http_headers = { "X-Chalk-Client-Id" = "CHALK_CLIENT_ID", "X-Chalk-Client-Secret" = "CHALK_CLIENT_SECRET" } ``` env_http_headers maps each header to the name of an environment variable Codex reads when it connects, which keeps the credentials out of the file. Export them first with eval "$(chalk config --format shell)". To write the values into the file instead, use http_headers, which takes the header values themselves. ### JSON configuration This is the format used by Cursor, Windsurf, and many other clients (mcp.json / mcpServers): ``` { "mcpServers": { "chalk": { "url": "https://api.chalk.ai/v1/mcp/sse", "headers": { "X-Chalk-Client-Id": "token-392c737aa1e467e42e85ae3e8417a003", "X-Chalk-Client-Secret": "ts-6307f46a00a68436b0f955b82b7fb30075d" } } } } ``` ### Clients that only support stdio For a client that cannot connect to a remote server directly, bridge to it with mcp-remote: ``` { "mcpServers": { "chalk": { "command": "npx", "args": [ "mcp-remote", "https://api.chalk.ai/v1/mcp/sse", "--header", "X-Chalk-Client-Id: ${CHALK_CLIENT_ID}", "--header", "X-Chalk-Client-Secret: ${CHALK_CLIENT_SECRET}" ], "env": { "CHALK_CLIENT_ID": "token-392c737aa1e467e42e85ae3e8417a003", "CHALK_CLIENT_SECRET": "ts-6307f46a00a68436b0f955b82b7fb30075d" } } } } ``` ### Verifying the connection with curl You can confirm your credentials are accepted by issuing an initialize request directly: ``` curl -sS https://api.chalk.ai/v1/mcp/sse \ -H "X-Chalk-Client-Id: $(chalk config --format json | jq -r .clientId.value)" \ -H "X-Chalk-Client-Secret: $(chalk config --format json | jq -r .clientSecret.value)" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}' ``` A 401 with a WWW-Authenticate header means the credentials were missing or rejected. ### Endpoint The server speaks the Streamable HTTP transport (MCP specification 2025-03-26). A single URL handles both the JSON-RPC POST requests and the Server-Sent Events (SSE) streams that the transport uses, so most clients only need this one URL. Configure your client for a streamable-HTTP (or "HTTP") remote server, not a legacy SSE server. The /sse suffix in the path is historical: this endpoint speaks the modern Streamable HTTP transport, not the deprecated HTTP+SSE transport. If you are on a dedicated or self-hosted Chalk deployment, replace api.chalk.ai with your own API server host. You can find it in the apiServer field of chalk config --format json. ### Authentication The MCP server authenticates with the same service credentials used everywhere else in Chalk. ### Getting your credentials Generate or look up a service token in the Settings → Service Tokens tab of the Chalk dashboard, then retrieve the values from the CLI with chalk config: ``` chalk config ``` ``` Name Value Description client_id token-392c737aa1e467e42e85ae3e8417a003 default token client_secret ts-6307f46a00a68436b0f955b82b7fb30075d default token environment btfxgaqqxbt7z ... api_server https://api.chalk.ai ... ``` To pull the values out programmatically, use a machine-readable format: ``` # JSON object with clientId / clientSecret / apiServer fields chalk config --format json # Shell export statements: CHALK_CLIENT_ID / CHALK_CLIENT_SECRET / ... chalk config --format shell ``` ### Presenting your credentials Provide your client_id and client_secret as HTTP headers on every request: | Header | Value | | ----------------------- | ------------------------------------ | | `X-Chalk-Client-Id` | Your service token's `client_id` | | `X-Chalk-Client-Secret` | Your service token's `client_secret` | The server exchanges these for a short-lived access token internally, so you do not need to run the OAuth client-credentials flow yourself. They can query your Chalk deployment and, depending on the permissions they carry, modify it. Use a scoped service token instead of personal credentials. ### Bearer tokens and OAuth The server also accepts a standard Authorization: Bearer header. If a request arrives without credentials, the server responds with 401 and a WWW-Authenticate challenge pointing at Chalk's OAuth discovery endpoints (/.well-known/oauth-protected-resource), so MCP clients that implement the MCP authorization spec (including Dynamic Client Registration) can negotiate a token interactively. For headless and server-to-server use, the X-Chalk-Client-Id / X-Chalk-Client-Secret headers are the simplest option. Interactive registration currently works for clients whose callback is a hosted web address, such as the claude.ai web app. A client that runs on your own machine, including Claude Code and other terminal or desktop clients, asks to be sent back to a local address on a random port, and Chalk does not yet accept those, so registration fails before a browser opens. Use the header-based setup in Connecting a client for those clients. As an alternative for Claude Code specifically, adding Chalk as a claude.ai custom connector, under Settings then Connectors, uses an allowlisted callback and syncs down into the CLI. ### Selecting an environment By default, requests run against the environment associated with your service token. To target a different environment, either: - send the X-Chalk-Env-Id: header, or - pass an environment_id argument to an individual tool call. Environment overrides are only honored for user (personal) credentials. Use the list_environments tool to discover the environment IDs available to your team. ### Available tools The server currently advertises the following 66 tools. The tools/list response is the runtime source of truth for their input schemas and behavior annotations. Every tool is authorized individually: it checks the permissions on the credentials that called it and returns a permission error if they fall short, so what a client can do is set by its token rather than by the server. run_online_query requires query.online. The observability and Kubernetes inspection tools require monitoring.read. Sandboxed Python execution and the deployment filesystem tools require deploy.redeploy, which is also the permission for running a notebook cell, so a token that can run notebook cells can also execute code against a deployment. Reading a stream resolver's status requires migrate.read, and pausing or resuming one requires deploy.create. The deployment file tools provision or reuse a source sandbox, including tools that only read or search files. The server therefore marks those tools as destructive and non-idempotent. ### General and discovery | Tool | Description | | -------------------------- | ----------------------------------------------------------------------------- | | `echo` | Return a supplied message. This is primarily useful for testing a connection. | | `whoami` | Return the authenticated identity and selected Chalk environment. | | `next_steps` | Propose a follow-up action for the user without executing it. | | `list_environments` | List environments available to your team, including their IDs. | | `list_environment_secrets` | List environment secret names and metadata without returning secret values. | | `search_docs` | Search the Chalk documentation, or read one exact documentation page. | | `search_docs_functions` | Search exact Chalk function signatures and API reference entries. | ### Queries and SQL | Tool | Description | | ---------------------------- | ------------------------------------------------------------------------------------ | | `run_online_query` | Run an online query for one or more features. | | `execute_sql_query` | Execute a [Chalk SQL](/docs/chalksql/what-is-chalk-sql) query, with preview support. | | `search_offline_queries` | Search offline queries with filters, or fetch one complete query by operation ID. | | `search_sql_catalog` | Browse catalogs and schemas, or search tables and columns. | | `lint_sql_query` | Plan Chalk SQL against the catalog without executing it. | | `propose_sql_worksheet_edit` | Return a complete SQL worksheet edit for review in the Chalk dashboard. | ### Feature graph, deployments, and functions | Tool | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------ | | `search_graph` | Search deployed features and resolvers, or fetch one exact definition. | | `get_stream_resolver_status` | Report whether a stream resolver is running or paused. | | `set_stream_resolver_status` | Pause or resume a stream resolver. | | `list_branches` | List the deployment's branches. | | `get_deployment` | Return metadata for a deployment. | | `list_functions` | List external functions in the environment. | | `inspect_function` | Inspect an external function's metadata, schemas, configuration, readiness, and source when available. | | `invoke_function` | Invoke an external function with JSON or Arrow IPC input. | | `deploy_chalk_function` | Create an external function or deploy a new revision from Python source. | | `deployment_read` | Read one file from a deployment source sandbox. | | `search_deployment_files` | Search deployment source files by regular expression or glob. | | `write_deployment_file` | Write a complete deployment file or replace exact text in one. | | `deployment_bash` | Run a shell command in a deployment source sandbox. | ### Notebooks These tools create, edit, and run Chalk Notebooks and their cells. Several of them propose a change instead of making it. A proposal renders in the notebook as a pending card at the position the agent chose, showing the new or rewritten source as a diff with Confirm and Undo controls, and the notebook is unchanged until someone confirms. Confirming inserts the cell but does not run it. That review step only exists in the Chalk dashboard, where a notebook is open to display it. A client connecting over this endpoint has no reviewer, so a proposal it returns is never shown to anyone and nothing is written. add_notebook_cell and delete_notebook_cell take persist=true, which applies the change directly and requires deploy.redeploy. edit_notebook_cell has no such argument: to change an existing cell from a client like this, use edit_and_run_notebook_cell, which persists the new source but also executes the cell. That path covers python and sql cells only. | Tool | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `create_notebook` | Create a notebook. | | `list_notebooks` | List the notebooks in the environment. | | `read_notebook` | Read a notebook's ordered cells and optionally their most recent execution results. SQL and query results include the column schema, a row count, and a capped sample rather than the full dataframe, and the sampled rows are referenced rather than inlined in the response. | | `add_notebook_cell` | Propose adding a python, sql, text, markdown, or input cell. Returns a proposal for the user to confirm rather than persisting; pass `persist=true` to apply it directly. | | `edit_notebook_cell` | Propose a rewrite of a cell's source. Returns a before/after diff for the user to confirm rather than persisting. | | `move_notebook_cell` | Move a cell to a new position. Persists immediately. | | `delete_notebook_cell` | Propose deleting a cell. Returns a proposal for the user to confirm rather than persisting; pass `persist=true` to apply it directly. | | `run_notebook_cells` | Execute cells and wait for them to finish, provisioning the kernel if needed. Omit the cell ids to run every runnable cell in document order. | | `add_and_run_notebook_cell` | Add a python or sql cell and execute it in one call, returning the run output. Persists immediately, so it requires deploy permission. | | `edit_and_run_notebook_cell` | Replace a python or sql cell's source and execute it in one call. Persists immediately, so it requires deploy permission. | | `install_python_dependencies` | Install python packages into a notebook's kernel by adding and running a `!uv pip install` cell. The cell persists immediately, so this requires deploy permission. Packages last for the kernel's lifetime, and rerunning the notebook reinstalls them. | | `attach_secrets_to_notebook` | Attach existing environment secrets to a notebook by reference, without values passing through the conversation. | ### Sandboxes These tools run commands or Python in a general-purpose sandbox. | Tool | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `start_sandbox` | Start a general-purpose sandbox and return its id, for reuse across `exec_in_sandbox` calls. Defaults to the `python:3.12-slim` image. | | `exec_in_sandbox` | Run a command in an existing sandbox. | | `eval_python` | Execute Python in a fresh ephemeral sandbox and return its output. Bare expressions produce nothing, so print what you want back. For repeated work, `start_sandbox` and `exec_in_sandbox` reuse one container. | ### Observability and Kubernetes | Tool | Description | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_query_errors` | List recorded query errors, with filtering and pagination. | | `get_query_plan_json` | Return a `chalk://query-plans/...` resource URI for a query plan. Plans can run to hundreds of kilobytes, so the tool hands back the URI and the client reads the resource. | | `get_perf_summary_json` | Return a `chalk://performance-summaries/...` resource URI for a query's performance summary, read the same way as a query plan. | | `search_logs` | Search application logs for the environment. | | `inspect_log_facets` | List log facets, or inspect the observed values and counts for one facet. | | `search_traces` | Search distributed traces and their spans. | | `search_kube_events` | Search Kubernetes cluster events (scheduling, OOM kills, crashes). | | `list_kube_events_in_namespace` | List recent Kubernetes events in one namespace. | | `get_kube_pod` | Inspect a pod's specification, status, and recorded events. | | `inspect_kube_deployments` | List Kubernetes deployments or inspect one deployment in full. | | `inspect_kube_capacity` | List or inspect Karpenter NodePools and NodeClasses. | | `get_unschedulable_workloads` | Diagnose workloads that Kubernetes cannot schedule. | ### Datasets, dashboards, monitors, and volumes | Tool | Description | | --------------------------------- | ------------------------------------------------------------------------ | | `list_datasets` | List datasets and their latest revision metadata. | | `inspect_dataset` | Fetch one dataset's metadata and optionally its revision history. | | `get_dashboard_authoring_context` | Return the dashboard JSON contract and metric authoring catalog. | | `find_dashboards` | Search dashboards, or fetch one complete dashboard definition. | | `validate_dashboard` | Validate a dashboard definition without saving it. | | `create_dashboard` | Create a dashboard from a validated definition. | | `get_chart` | Fetch one saved metric chart. | | `get_monitor_authoring_context` | Return the monitor JSON contract, metric catalog, and alert channels. | | `find_monitors` | List monitors, or fetch one complete monitor definition. | | `validate_monitor` | Validate a monitor definition without saving it. | | `create_monitor` | Create a monitor from a validated definition. | | `list_volumes` | List Chalk volumes, optionally filtering by name prefix. | | `create_volume` | Create a versioned Chalk volume. | | `create_volume_from_github_repo` | Copy a GitHub repository into a Chalk volume. | | `create_github_pr_from_volume` | Open a GitHub pull request that makes a repository match a Chalk volume. | ### Resources and prompts Alongside tools, the server exposes MCP resources, which a client reads by URI rather than calling. Two tools hand back a resource URI instead of the payload itself, because a query plan or performance summary can run to hundreds of kilobytes. Two resources are available directly: | URI | Contents | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `chalk://sql/catalogs` | The Chalk SQL catalogs for the environment, including the built-in `chalk` catalog, `offline_store`, and any connected data sources. Use these names when writing Chalk SQL. | | `chalk://server/info` | Basic information about the Chalk server. | Others are addressed by filling in a template: | URI template | Contents | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `chalk://environments/{environment_id}` | An environment's name, project, cloud provider, and region. | | `chalk://sql/schemas/{catalog}` | The schemas within a Chalk SQL catalog. | | `chalk://query-plans/{query_plan_id}` | A query plan, as JSON. Call `get_query_plan_json` first to obtain a valid URI. | | `chalk://performance-summaries/{operation_id}` | A query's performance summary, as JSON. Call `get_perf_summary_json` first to obtain a valid URI. | The server also advertises resource subscriptions, so a client that supports them can be notified when a resource changes. One prompt is available, help, which takes a topic argument and returns guidance on Chalk operations. ### See also - MCP Gateway: register and govern the external MCP servers your agents reach. - Model Gateway: OpenAI-compatible LLM routing for your Chalk deployment. - Authentication: service credentials and RBAC for the Chalk API. - Development with LLMs: prompts for writing Chalk code with AI assistants. - Chalk SQL: the SQL dialect available through execute_sql_query. # Observability Overview source: https://docs.chalk.ai/docs/observability ## Monitor the execution of your feature pipelines & alert you when problems arise. It’s inevitable — production data drifts from historical baselines, pipelines break, and partners change data formats. Chalk monitors the execution of your feature pipelines and alerts you when problems arise. ### What to read next Resolvers support many workflows and common orchestration patterns: - Dashboards - Build custom dashboards over metrics, logs, and Kubernetes events. - Metrics Monitor - Define critical metrics for monitoring and alerting. - Metrics Export - Export Chalk metrics to other monitoring systems. - Alert Configuration - Configure which alerts get sent where and to whom. - Logs Export - Export Chalk logs to other monitoring systems. - Datadog Integration - Ingest Chalk metrics into Datadog. - PagerDuty - Integrate PagerDuty alerts into your feature pipelines. - Slack - Consume Chalk alerts in your Slack channels. - incident.io - Integrate Chalk alerts with your incident.io workflow. # Dashboards source: https://docs.chalk.ai/docs/dashboards ## Combine metrics, logs, and events from your feature pipelines into a single custom view. Use dashboards to visualize metrics, logs, and events from your feature pipelines in a single view. A dashboard is a grid of widgets: timeseries charts, tables, and single-number statistics over your data; status widgets for monitors, incidents, deployments, and connection health; and text notes and section titles that give the page structure. Dashboards are shared across your environment: every teammate sees the same dashboards, and every edit saves automatically. ### Getting started This walkthrough builds a first dashboard from scratch. ### Create a dashboard In the left navigation, open Monitoring > Dashboards. The Dashboards page lists every dashboard in the environment, along with its owner, when it was last updated, and how often it is viewed. You can search by name, sort the columns, and star the dashboards you use most. Click New dashboard to create an empty dashboard named "Untitled dashboard". Click the title to rename it, and click below the title to add a description. ### Add your first widget Click Add widget in the top right of the screen, or click the dashed placeholder tile on the empty grid. The menu offers seven kinds of widgets: - Data widget: a chart, table, or statistic over your metrics, logs, access logs, or Kube events - Monitor widget: the current status of an existing monitor, including any active incident - Latest deployment: a summary of the environment's most recent deployment - Incident widget: a list of recent incidents, limited to ongoing or resolved incidents if you choose - Connection health: the health summaries from the Connections page for the connections you choose - Text: a free-form markdown block - Section title: a full-width heading with a divider line Choose Data widget to open the widget editor: ### Choose a data source and visualization At the top of the editor, pick a Data source (Metrics, Logs, Access Logs, or Kube Events) and a Visualization (Timeseries, Table, Tree map, Pie chart, or Statistic). Options that do not apply to the selected source are disabled; see Visualizations below for examples of each. The editor shows a live preview of the widget as you configure it. ### Configure the query The editor below the pickers depends on the data source. For Metrics, you build one or more series: pick a metric (for example Feature Latency or Query Count), a window function (mean, sum, a percentile, and so on), optional group-by dimensions, and optional filters. You can also add formulas that combine series arithmetically. For Logs, Access Logs, and Kube Events, you can write a search query using the same search syntax as the explorer pages, then describe what to show with an aggregation row. Optionally name the widget (if you leave the name blank, Chalk derives one from the configuration), then click Add widget. The widget lands in the first free slot on the grid. ### Arrange the layout Drag a widget by its top to move it, and drag its bottom corners to resize. Every layout change saves automatically. ### Visualizations A data widget renders as one of five visualizations. Timeseries and Statistic work with every data source; Table, Tree map, and Pie chart are for Logs, Access Logs, and Kube Events. ### Timeseries Values over time, drawn as lines or stacked bars, with one series per group. For example, this widget overlays the p95, p50, and mean of access log duration, filtered to a related set of HTTP paths, to profile their latencies: ### Table An aggregate table shows one row per group with a column per measure. For example, this widget counts access logs by HTTP path: With the List view, the table instead shows the matching records themselves, as a live tail on the dashboard. For example, this widget lists recent log lines matching a specific error message: ### Tree map and pie chart These views show how a count divides among groups. Group by one or more dimensions, and each group becomes a rectangle in the tree map or a slice of the pie. They use the Count measure only, because other measures do not add up to a whole. The pie chart can also hide its legend or its labels. This pie chart groups access-log requests by scaling group and sizes each wedge by request count: ### Statistic A single number summarizing the whole time range. A statistic reduces to one value, so it takes a single measure or series and no grouping. You can add a unit label (for example queries or ms) and turn on Compare to previous period to show the change versus the preceding window of equal length. For example, this statistic widget counts Evicted Kube events: ### Putting it all together ### Text and section titles Two widget types carry no data: - Text widgets render GitHub-flavored markdown, including headers, lists, links, code, and tables. Use them for runbook links, context on what a section means, or instructions for on-call engineers. The editor shows a live preview, and long content scrolls within the widget. - Section title widgets render a heading followed by a divider line that spans the row. Leave the title empty to get a plain divider. As an example: ### Working with widgets Every widget has a menu with Edit, Duplicate, and Delete. Duplicate creates a copy of the widget, including its configuration and size, in the first free slot. To build a row of similar charts, configure one widget, duplicate it, and edit the copies. Metric widgets additionally offer Open in metrics explorer to investigate the same configuration outside the dashboard. Hovering a chart reveals its toolbar: switch the chart type, zoom into a time range by dragging, view fullscreen, and annotate. Hovering any chart also highlights the same moment in time across every chart on the dashboard. ### Dashboard controls The controls in the top right header apply to every widget on the dashboard at once. Their settings live in the URL, so copying a link shares exactly what you are looking at. ### Time range and live data The time selector takes a rolling window, a calendar window, or a range you type. Rolling presets run from Past minute to Past 365 days. Calendar presets are Month to date, Year to date, Current month, and Current year, aligned to UTC midnight. To set any other window, type it into the time range field, for example last 7 days or jun 1 to jun 5. With a rolling window, Live mode (on by default) keeps the dashboard updating; turning it off reveals a manual refresh button. Expand doubles the window, Condense halves it, and Reset returns to the default. Zooming into any chart narrows the whole dashboard to that range. ### Annotations The annotations menu overlays operational context on every chart at once: incident markers, incident ranges, deployments, Kubernetes resource patches, and custom annotations you have drawn on charts. For example, a deployment marker that lines up with a latency spike shows what changed at that moment. ### TV mode Open TV mode from the dashboard's actions menu to get a chrome-free, read-only rendering of the dashboard that refreshes every five minutes. Exit with the menu in the corner. ### Duplicating, exporting, and importing Dashboards serialize to a portable JSON format, which powers several features in the dashboard's actions menu: - Duplicate creates (Copy) with the same widgets and layout, and opens it. - Export as JSON and Copy JSON download the dashboard as a .dashboard.json file or copy the JSON to the clipboard. - Import, on the Dashboards page, accepts pasted JSON or an uploaded file. Invalid documents are rejected with specific, per-field errors so you can fix and retry. Exports are portable across environments: a dashboard exported from one environment can be imported into another. ### Dashboards and alerting Alerts come from monitors. To get notified when a metric crosses a threshold, create a monitor, which has its own metric configuration and trigger independent of any dashboard. Monitors are configured under Alerting > Monitors, and alert routing is covered in Alert Configuration. # Monitors source: https://docs.chalk.ai/docs/overview/monitors ## Continuously evaluate metrics, platform health, or log queries, and open incidents when thresholds are met. ### Overview Monitors continuously evaluate a monitored entity (a metric formula, a log query, or a platform healthcheck) against a configured condition. When the condition is met, the monitor opens an incident and routes notifications to the configured channels. When the monitored value recovers, the incident is closed. Each monitor keeps a timeline of its events (incidents opened, triggering values, incidents closed) so you can see exactly when a problem started and when it resolved. Monitors live in the dashboard under Alerting > Monitors. From there you can see every monitor in your environment with its status (OK, Failing, or Muted), type, frequency, description, alert channels, creator, and creation time. You can search by name, filter by type, creator, or alert channel, and create a new monitor with the New monitor button. Every monitor, regardless of type, is configured with: - Name and description: user-provided metadata about the purpose of this monitor, shown in the monitors list and in alerts - Monitored entity: the metric chart, log query, or environment healthcheck being evaluated, depending on the monitor type - Alert channels: where notifications are sent when the monitor fires or recovers, which can be a Slack channel, a PagerDuty service, or an incident.io service - Schedule: how often Chalk evaluates the monitor, every minute by default, set under Advanced Settings when you create the monitor or with Edit schedule on its detail page Metric chart and log monitors also have a threshold: a comparison (above, below, greater than or equal, less than or equal, equal, or not equal) against a target value. Environment health monitors have no threshold. ### Monitor types On the New monitor page, choose the type in the Monitor type list: Metric, Logs, Access Logs, Kube Events, or Environment health. ### Metric chart monitors Metric chart monitors evaluate one or more metric series, optionally combined into a single value with a formula. Each series is defined the same way as a series on a dashboard metrics chart: pick a metric kind, window, grouping function, and filter conditions. Multiple series can be aggregated together with a formula that references the series by letter (a, b, ...). The resulting value is compared against the threshold on every evaluation. Use metric chart monitors to notify on things like request counts, latencies, error ratios, or feature drift metrics. Series and formulas are validated when the monitor is created or updated, so a misconfigured query fails at save time instead of at the first evaluation. The editor previews the series over a time range you choose, and shades the alerting region red once you set a threshold. ### Log monitors Log monitors run a query against your environment's logs, access logs, or Kube events, depending on the type you choose, on a schedule. You configure: - Query string: a literal or wildcard expression match against log contents, written in the same search syntax as the explorer pages - Lookback: the period over which matching entries are counted, chosen with the time selector above the preview The number of matching entries in each lookback window is compared against the threshold. Use log monitors to open incidents on error messages, exception signatures, or any log pattern that should be rare (or that should always be present). Log monitors require Chalk-managed (ClickHouse-backed) log storage in your environment. The editor previews the matching entries and a chart of how often they occur. ### Environment health monitors Use environment health monitors to get paged when a piece of your Chalk infrastructure degrades, without needing to construct a metric query for it. Not all systems are metric queryable, so health monitors provide additional functionality on top of metrics. Environment health monitors watch one of Chalk's built-in environment healthchecks, the same checks that power the Settings > Connections page, covering components such as the online store, offline store, query engines, streaming and branch servers, and supporting infrastructure. Pick the healthcheck to monitor. The monitor opens an incident while that check reports unhealthy, and closes it once the check reports healthy again. ### Incidents When a monitor's condition is met, an incident is opened for the monitor. Additional evaluations while the incident remains open are attached to the same incident rather than opening new ones. When the monitored value returns within the threshold, the incident is closed. A monitor's detail page has two tabs. Configuration holds the monitor's settings, and Events shows the full timeline, newest first, including every failing evaluation, incident opening, and incident closure. Click Edit schedule at the top of the page to change how often the monitor runs, then click Save changes on the Configuration tab to apply it. Every incident also appears on the Alerting > Incidents page, and clicking an incident's title in the timeline opens it. Mute a monitor to stop it from sending alerts while it is failing. Click the bell icon next to the monitor's status, then choose how long to mute it. A muted monitor shows Muted in the monitors list. Click the bell icon again to unmute it. # Metrics Monitor source: https://docs.chalk.ai/docs/metricmonitor ## Define and monitor metrics Chalk's Monitors consolidate all of your alerting in one place: in addition to metric-based alerts like the ones described on this page, Monitors can alert on platform health and log contents, and track incidents from first alert through resolution. Because Monitors cover everything chart alert triggers do and more, alert triggers on the metric charts dashboard are deprecated — prefer creating a Monitor, under Alerting > Monitors for new alerting. Chalk's online dashboard provides a simple way to view metrics about performance of your queries and resolvers. By default, the resolver and query detail pages show request count and latency for each 30 minute interval over the last 2 days, but you may wish to customize this to make it more useful. ### Resolver Details In addition to the default charts, the resolver details page tells you the input and output features of the resolver, plus the implementation code of the resolver. Metric Dashboard ### Query Details The query details page shows you much of the same information as the resolver details page, including the input and output features. You can also see which resolvers executed on the right hand sidebar. Query Dashboard ### Customize Your Metrics Dashboard Customize your monitoring view with additional charts, combine metrics into complex formulas, and configure alerting thresholds. Clicking "Add Metric" brings up the metric editor which you can use to add a new chart to your page. Edit or remove a chart using the ellipsis menu. Create a metric ### Add Series By default, when you click add metric the editor is pre-populated with a new series called Request Count. Use the drop down fields to change the metric kind, window, grouping function, and conditions. Add additional series to the chart by clicking "Add Series" below. ### Add Formula Aggregate multiple series together into one chart using a formula. Chalk supports aggregating by sum, total ratio, ratio, product, and abs. ### Add Alert Trigger Alerts can be triggered on any series or formula. To set up an alert, you should first configure at least one of a PagerDuty integration or a Slack integration. Then you can set up Alert Configurations to define which channels to use to alert different users or teams. Then, in your metrics chart, you can select the series, the threshold logic, and the severity, and route the alert appropriately with the alert owner dropdown. You can select from users in the Chalk environment or free type an owner name corresponding to one of your alert configurations. # Charts with Code source: https://docs.chalk.ai/docs/charts ## Define charts and alerts with code. You can also create charts using Python, just as you would your features and resolvers. ### Charts and Series The heart of the Chart paradigm is the Series, a single collection of data points over time. They must be instantiated with a metric and name, and can accumulate filters and group-bys to support advanced calculations. Some Series metrics require a window function parameter to decide how to calculate values. Window functions often include aggregates such as count and mean and percentiles such as 95% and 99%. Methods and options will auto-complete in your text editor based on the type of series. Series objects must be attached to a Chart, which can have multiple series and up to one trigger. Let's say you have a resolver get_fraud_profile and you want to make sure it goes fast. The following chart will track the p95 latency of your resolver, and attach it to its resolver page on the dashboard. ``` from chalk.monitoring import Chart, Series Chart( name="my_chart_1", window_period="30m" ).with_series( Series.resolver_latency_metric( name="p95 latency for `get_fraud_profile`", window_function="95%" ).where(resolver=get_fraud_profile) ).with_resolver_link(get_fraud_profile) ``` This is all that's necessary to ship a chart to your online dashboard upon chalk apply! Using this builder pattern, we're able to generate charts quickly and by reusing intermediate instances if necessary since every instance is deep-copied from the previous instance. By default, only non-copied charts will be registered to the server. Consider the following example: ``` from chalk.monitoring import Chart, Series base = Chart( name="my_chart_1", window_period="30m", ).with_series( Series.resolver_latency_metric(window_function="99%") ) modified = base.with_series( Series.resolver_latency_metric( window_function="99%", ).where(resolver_type='online') ) ``` Since modified has not been copied, it will be the only one of the three to be shipped to the server. However, base has been copied, and will not create a chart. To force-register charts to the server, use .keep(), which will be applied to the chart itself and all its descendants. In the above example, we also added a filter to our Series object. Here, we restrict the series to only calculate latencies for online resolvers. The filters enabled differ depending on the series metric, and the appropriate options should be available through auto-complete in your text editor. ### Triggers Triggers can also be added to charts. The following error trigger will alert myemail@chalk.ai when the resolver get_fraud_profile takes more than 200ms to execute. ``` from chalk.monitoring import Chart, Series Chart(name="my_chart_1", window_period="30m").with_trigger( Series.resolver_latency_metric( window_function="95%" ).where(resolver=get_fraud_profile) > 0.2, trigger_name=f"High latency alert", severity="error", channel_name="myemail@chalk.ai", description=""" *Debugging* When this alert is triggered, we're parsing null values from a lot of our FICO reports. It's likely that Experian is having an outage. Check the . """ ).with_resolver_link(get_fraud_profile) ``` ### Advanced Chart Building We can also easily create charts in bulk. Let's say we have a feature class Transaction with four scalar features. ``` from chalk.monitoring import Chart, Series for feature in Transaction: Chart( name=f"{feature} request count chart", window_period="5m" ).with_trigger( Series.feature_request_count_metric().where(feature=feature) < 10, trigger_name=f"Low volume requests for {feature}", severity="error", channel_name="myemail@chalk.ai", ).with_feature_link(feature) ``` Upon chalk apply, four charts will be registered to the server filtering on feature requests for each of the four features. These charts will be available on the relevant feature pages on the dashboard. # Feature Drift source: https://docs.chalk.ai/docs/featuredrift ## Detect and setup alerting for drift in feature values Chalk provides a simple way to monitor feature drift by running the Kolmogorov-Smirnov test for feature values over a given time period. ### Kolmogorov–Smirnov test The K-S test is a statistical test that can be used to determine if two samples are drawn from the same distribution. Chalk runs the test on samples from the online store and data from a given dataset to determine if the feature values are drawn from the same distribution. Note that the K-S test can not be used if the feature values in the dataset are all null, if they are not numeric or if there is only one unique value. ### Charts and Alerts To setup a chart and alert for the Kolmogorov-Smirnov test, start by creating a named dataset for the features you want to monitor. Then under Alerting > Metric Charts, create a new chart and click on Add Formula, choosing KS TEST as the function. Next select the dataset you created and the feature you want to monitor. The y-axis of the chart displays the difference between the Kolmogorov-Smirnov test statistic and the critical value at significance level 0.05. When the value is greater than 0, the feature values are considered to be drawn from different distributions. KS Test Chart To create an Alert, click on Add Alert Trigger and configure the alert to trigger when the KS Test value is greater than 0. # Metrics Export source: https://docs.chalk.ai/docs/metricexport ## Export Chalk metrics to other monitoring systems. Chalk's online dashboard provides a simple way to view metrics about performance of your feature pipelines. However, you may wish to export these metrics from Chalk into other observability tools so that you can view your Chalk-related data alongside data from other systems you maintain. ### Exporting metrics Chalk tracks various time series metrics that measure the latency and throughput of resolvers and streaming pipelines. Chalk stores these metrics in your cluster. You can use any OpenMetrics-compatible collector to collect metrics about the execution of your feature pipelines from Chalk. Examples include: - Prometheus - OpenMetrics (Datadog) - NewRelic - Stackdriver Prometheus ### Available metrics The table below summarizes the metrics that are available for export. The headers in the table are the exported metric name followed by the OpenMetrics metric type (gauge, histogram, summary, or counter). Provides information about the time it takes to compute a resolver. The name of the resolver, for example, my.company.get_user Whether this latency represents the median, 75th percentile, 95th percentile, or 99th percentile of the latency The type of the resolver - online, offline, or stream. Provides information about the time it takes to execute an online query. The name of the query, for example, eligibility_query_v2. Queries without names are labeled "Unnamed" Whether this latency represents the median, 75th percentile, 95th percentile, or 99th percentile of the latency Provides information about the time it takes to execute a cron run. The name of the resolver executed by the cron run, for example, my.company.get_user Whether this latency represents the median, 75th percentile, 95th percentile, or 99th percentile of the latency Provides information about the number of times a feature was computed. The name of the feature, for example, user.age The status of the computed feature (success or failure) The context in which the feature was generated Provides information about the number of times a resolver was computed. This metric informs the number of times that resolvers are being called and the context in which they are called, for example in a cron run as part of a scheduled job or in inference as part of a query plan. The name of the resolver, for example, my.company.get_user The status of the resolver run (success or failure) The context in which the resolver ran The type of the resolver - online, offline, or stream. Provides information about the number of times a cron run was executed. This metric is useful for monitoring the status of resolver runs that are scheduled or triggered via API to load data into the online and/or offline store. The name of the resolver executed by the cron run, for example, my.company.get_user The status of the cron run (success or failure) Counts scheduled query executions since midnight UTC, grouped by query name and status. The counter resets at midnight UTC. The /v2/metrics/export endpoint exports this counter as scheduled_query_count_total. To alert on failures for each query, filter on query_status="failure" and group by query_name. The name of the scheduled query. The execution status, such as success or failure. The ID of the environment. The name of the environment. The fraction of scheduled query executions that succeeded since midnight UTC, from zero to one. Each query with executions in this period has a separate series. Queries with no executions in this period have no ratio series. The name of the scheduled query. The ID of the environment. The name of the environment. Provides information about the number of features computed by cron written to online / offline store. This metric is useful for monitoring resolver runs that are scheduled or triggered via API to load data into the online and/or offline store. The name of the resolver executed by the cron run, for example, my.company.get_user Whether the features were written to online or offline store. Provides statistical information about the value of features. The name of the feature, for example, user.age Whether this value represents the median, 75th percentile, 95th percentile, or 99th percentile of the feature value Provides information about the number of times an online query was executed. The name of the query, for example, eligibility_query_v2. Queries without names are labeled "Unnamed" The status of the query (success or failure) The active deployment version. This gauge will always have a value of 1 for active deployments. A gauge of this kind is sometimes called an Info metric. The ID of the deployment. The response counts by HTTP response code. The ID of the environment. The human-readable name of the environment (for example, prod or staging). Provides information about how far streaming message processors are behind, measured in messages. The name of the stream resolver. Whether this lag represents the median, 75th percentile, 95th percentile, or 99th percentile of the offset lag. The ID of the environment. The human-readable name of the environment (for example, prod or staging). The maximum container CPU utilization percentage observed over the recent lookback window, grouped by Chalk service. The ID of the environment. The human-readable name of the environment (for example, prod or staging). The Chalk service reporting the metric. http is the HTTP query server, grpc is the gRPC query server, and branch is the branch server. The maximum container memory usage in bytes observed over the recent lookback window, grouped by Chalk service. The ID of the environment. The human-readable name of the environment (for example, prod or staging). The Chalk service reporting the metric. http is the HTTP query server, grpc is the gRPC query server, and branch is the branch server. The maximum number of distinct replicas observed over the recent lookback window, grouped by Chalk service. The ID of the environment. The human-readable name of the environment (for example, prod or staging). The Chalk service reporting the metric. http is the HTTP query server, grpc is the gRPC query server, and branch is the branch server. The current max_ingested_timestamp in UNIX epoch time for resolvers. The ID of the resolver. # Export logs and traces to Datadog source: https://docs.chalk.ai/docs/datadog-telemetry-export ## Configure the Vector aggregator to export logs, metrics, and traces to Datadog. Use the Vector aggregator to send Chalk logs, metrics, and traces to your Datadog account. This Vector aggregator export is available when your organization pays for Metrics Export and the feature is enabled for your team. You also need an existing telemetry deployment and must select the Vector runtime. If Exporters is missing from the dashboard, contact Chalk to check access to the feature. ### Store your Datadog API key Create a secret in AWS Secrets Manager whose value is only the raw Datadog API key string. Do not wrap the value in JSON (such as {"api_key":"..."}) or add quotation marks around it. Use the AWS account and region of the cloud cluster that runs your telemetry deployment, and allow that cluster's cloud credentials to read the secret. Copy the secret's full ARN, for example: ``` arn:aws:secretsmanager:us-east-1:123456789012:secret:datadog-api-key-AbCdEf ``` Chalk reads the secret when it deploys the telemetry configuration and places the key in a Kubernetes secret for the customer Vector aggregator. ### Configure export in the dashboard - Open Settings → Shared resources → Telemetry for the environment. This tab is available when the environment already has a telemetry deployment. - Use the guided form and select Vector under Telemetry Runtime. Open Exporters → Export to Datadog. - Set API Key Lookup Type to ARN and paste the ARN into API Key Secret ARN. Set API Host to your Datadog site, such as datadoghq.eu, if you do not use the default datadoghq.com site. - Turn on Export Logs to Datadog, Export Metrics to Datadog, and Export Traces to Datadog for the signals you want to send. Turn off a signal's switch to stop exporting that signal to Datadog. - Save the form and review the configuration changes in the confirmation dialog. Confirm to deploy the update. You can change the switches, ARN, API host, and VRL in this form without editing the telemetry JSON. The Telemetry switch controls collection for the entire environment. Leave it enabled when you want exports to continue. ### Remap events with VRL Use Vector Remap Language (VRL) to change events before export. Under Exporters → Shared VRL, the Logs Remap VRL, Metrics Remap VRL, and Traces Remap VRL fields apply to every export destination for the corresponding signal. Under Export to Datadog → VRL, the same three fields apply only to your Datadog export, after the shared VRL for that signal. For example, enter this in Export to Datadog → VRL → Metrics Remap VRL to add a tag to metrics sent to your Datadog account: ``` .tags.customer = "acme" ``` Save and confirm the telemetry update after changing VRL. Leave a remap field empty when you do not need to change that signal. # Alert Configuration source: https://docs.chalk.ai/docs/alertconfig ## Configure Alerts Connect your Alert Thresholds to external alerting tools like PagerDuty, Slack, and incident.io. ### Configure your alerts Chalk enables you to set up alert configurations to route alerts to different channels based on the owner of the alert. In the dashboard, visit Alerting > Alert Channels to define an owner (the dropdown will list all users in the environment, but the owner can be any string, so you can use a team name or other identifier). For each owner, you can select one or more channels to send alerts to. The channels can be Slack channels, PagerDuty services, and incident.io integrations, depending on which integrations you have set up in your Chalk environment. All alerts that belong to an owner (as configured on the alert trigger) will be sent to the channels associated with that owner. Edit or remove the alert configuration, or send a test message, using the buttons on the right hand side. Alert Configuration # Log Export source: https://docs.chalk.ai/docs/logexport ## Export Chalk logs to other monitoring systems. Chalk's online dashboard provides a simple way to view logs from resolver execution. However, you may wish to export these logs into other observability tools so that you can view your Chalk-related data alongside data from other systems you maintain. ### Exporting logs Chalk can export your logs to a sink of your choosing. You can configure sinks for log export from your settings page. Exported logs will contain the following fields: ISO8601 timestamp. The log line format version. Currently constant "1". The log message. Name of the logger, if relevant. The log level as a string: "INFO", "WARN", "ERROR", "DEBUG". Stack trace of the associated exception, if relevant. A trace ID in OpenCensus format. Used for correlating logs corresponding to a single query. Chalk Project ID. Chalk Deployment ID. Can be used to understand log patterns as code evolves. Your commit SHA, if available for the deployment. "online" or "offline". The name of the resolver that produced this log line, if available. Environment name. Environment ID. An array of the tags that were provided for the associated query. The ID of the agent that made the request (a service token or Chalk user). ### Custom logs Chalk provides an implementation of the standard Python Logger interface that supports emitting logs to the web UI and your metrics export sink. You can use this logger like this: ``` from chalk.clogging import chalk_logger @online def your_resolver(...): chalk_logger.info("Hello world!") ``` # Search Syntax source: https://docs.chalk.ai/docs/log-search-syntax ## Search Chalk logs and similar pages (access logs, kube events, traces, and spans) with text terms, fields, Boolean operators, wildcards, and grouped values. A query is composed of terms and operators. Terms can search the message or a specific field. There are two types of terms: - A single term is one unquoted word, such as timeout or chalkruntime. - A phrase is text surrounded by double quotes, such as "hello world". Use Boolean operators to combine terms. Operators are case sensitive and must be uppercase. | Operator | Description | Example | | -------- | ------------------------------------------------------------------------------------- | ------------------------------------- | | `AND` | Intersection: both terms must match. This is also the default between adjacent terms. | `timeout AND service:chalkruntime` | | `OR` | Union: either term can match. | `service:chalkruntime OR service:api` | | `NOT` | Exclusion: the following term or group must not match. | `service:api NOT severity:ERROR` | | `-` | Short exclusion syntax. | `service:api -severity:ERROR` | ### Text Search A term without a field is matched against the record's message as a substring, so timeout also finds timeouts and connection_timeout_ms. Quotes are required to search a multi-word phrase. | Search syntax | Description | | ------------------- | ------------------------------------------------------------------------- | | `timeout` | Searches message text for `timeout`. | | `"request failed"` | Searches message text for the phrase `request failed`. | | `timeout error` | Searches message text for both `timeout` and `error`. | | `timeout OR error` | Searches message text for either `timeout` or `error`. | | `timeout and error` | Searches for all three words. Lowercase `and` is a term, not an operator. | | `"AND"` | Searches for the literal word `AND`. Unquoted, it parses as the operator. | ### Field Search To search a specific log field, use field:value. ``` service:chalkruntime severity:ERROR resolver:my_resolver ``` Unlike a message term, a field value must match exactly unless it contains a wildcard. The available fields depend on which explorer you are in; the Logs fields are: | Field | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `service` | Service name. Also accepts `service_name` and `ServiceName`. | | `component` | Kubernetes or Chalk component. | | `severity` | `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `FATAL`. Case-insensitive, and `warn` and `information` are folded into their canonical spellings. | | `resolver` | Resolver name or FQN. Also accepts `resolver_fqn`. | | `query_name` | Named query. | | `operation_id` | Operation ID. | | `correlation_id` | Correlation ID. | | `pod_name` | Kubernetes pod name. | | `app` | Kubernetes app label. | | `resource_group` | Resource group. | | `deployment_id` | Deployment ID. Also accepts `deployment`. | | `scaling_group` | Scaling group label. | | `container_name` | **Filter only.** Container label. | | `http_path` | Request path, for services behind the proxy. | | `response_code_class` | `2xx`, `4xx`, `5xx` — the response code bucketed by its first digit. | | `user_logger` | `true` for logs your resolvers emitted, `false` for Chalk's own. | | `trace_id` | Trace ID. | | `workflow_execution_id` | Workflow execution ID. | Fields that contain punctuation can be quoted: ``` "k8s.pod.name":engine-grpc ``` ### Wildcards * in a field value matches any run of characters: | Search syntax | Matches | | --------------------------- | ---------------------------------- | | `pod_name:engine-*` | Values beginning with `engine-`. | | `pod_name:*abcdef` | Values ending with `abcdef`. | | `http_path:*VolumeService*` | Values containing `VolumeService`. | ### Grouped Field Values Put parentheses after a field name to apply that field to every unqualified value in the group. ``` service:(chalkruntime OR engine-grpc-mcyfr4j8u134) ``` The query above is equivalent to: ``` service:chalkruntime OR service:engine-grpc-mcyfr4j8u134 ``` Groups can be nested and can include NOT: ``` service:((chalkruntime OR "engine grpc") AND NOT (proxy OR api)) ``` If a grouped value has its own field, Chalk keeps that explicit field: ``` service:(chalkruntime OR component:api) ``` ### Grouping And Precedence Use parentheses to control evaluation order: ``` (service:chalkruntime OR service:api) AND severity:ERROR ``` Without parentheses, the precedence order is: - NOT and - - AND and adjacent terms - OR For example: ``` "timeout" OR service:api AND NOT severity:ERROR ``` is evaluated as: ``` "timeout" OR (service:api AND NOT severity:ERROR) ``` ### Special Characters And Spaces Use double quotes when a value contains spaces, colons, parentheses, or other punctuation: ``` message:"key:value" resource_name:"hello world" service:"engine grpc" ``` Inside a quoted value, escape a double quote with a backslash: ``` message:"failed with \"permission denied\"" ``` ### Examples | Search query | Description | | -------------------------------------------------------- | -------------------------------------------------------------------- | | `service:chalkruntime severity:ERROR` | Error logs from `chalkruntime`. | | `service:(chalkruntime OR api) -severity:INFO` | Logs from either service, excluding routine ones. | | `resolver:my_resolver "timed out"` | Resolver logs whose message contains `timed out`. | | `(component:engine OR component:api) AND severity:ERROR` | Error logs from either component. | | `service:(NOT (proxy OR api)) timeout` | Logs containing `timeout` from services other than `proxy` or `api`. | | `pod_name:engine-* severity:ERROR` | Error logs from any pod whose name starts with `engine-`. | ### Fields By Surface Every explorer (Logs, Access Logs, Kube Events, Spans, and Traces) accepts the syntax above. What differs between them is the useful attributes to query by. Most fields can also be a group by dimension in Search Aggregations. Where a field cannot, its description says Filter only. The numeric measures — duration and count — are the exception: they are what the aggregation row measures rather than groups by, so they appear there as fields for Mean, Percentile and the rest. ### Access Logs | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------- | | `authority` | Target host, from the `:authority` header. | | `http_path` | Request path. Also accepts `path`. | | `method` | HTTP method, such as `GET` or `POST`. | | `status_code` | HTTP response code. Also accepts `response_code`. | | `grpc_status` | Status name, not the numeric code: `OK`, `NOT_FOUND`, `UNAVAILABLE`, and so on. | | `response_flags` | How Envoy terminated the request: `-` when nothing went wrong, else a code such as `UH`, `UF` or `UT`. | | `upstream_host` | IP `address:port` the request was proxied to, so this identifies the receiving pod. | | `downstream_remote_address` | IP `address:port` the request arrived from. The port is ephemeral, so match on the address with `*`. | | `scaling_group` | Scaling group label. Also accepts `scaling_group_name`. | | `container` | Container label. Also accepts `container_name`. | | `pod_name` | Kubernetes pod name. | | `duration` | Request duration, in milliseconds. The numeric measure the aggregation row offers for this surface. | | `trace_id` | **Filter only.** Trace ID, to pivot from a request to its spans. | | `min_duration` | **Filter only.** Lower bound on duration. A bare number is milliseconds; `us`, `ms`, `s` and `m` also work. | | `max_duration` | **Filter only.** Upper bound on duration, same units as `min_duration`. | ### Spans and Traces Both views filter the same spans over the fields below. Only the Spans view offers the aggregation row, so on Traces every field here is filter-only. | Field | Description | | --------------- | ----------------------------------------------------------------------------------------------------------- | | `operation` | Span name, such as a resolver or RPC name. | | `service` | Service that emitted the span, which for a client span is the caller rather than the callee. | | `status_code` | Span status: `Ok`, `Error`, or `Unset`. | | `kind` | Span kind, such as `Server`, `Client`, or `Internal`. | | `trace` | Trace ID. Groupable, so grouping by it counts how many spans each trace has. | | `duration` | Span duration, in nanoseconds. The numeric measure the aggregation row offers for this surface. | | `span` | **Filter only.** Span ID, to pull up one span and its children. | | `min_duration` | **Filter only.** Lower bound on duration. A bare number is microseconds; `us`, `ms`, `s` and `m` also work. | | `max_duration` | **Filter only.** Upper bound on duration, same units as `min_duration`. | | `attr` | **Filter only.** Span attribute, written `attr:key=value`. | | `resource_attr` | **Filter only.** Resource attribute, written `resource_attr:key=value`. | ### Kube Events | Field | Description | | ------------------ | ----------------------------------------------------------------------------------------------------- | | `severity` | `INFO`, `WARNING` or `ERROR` — Chalk's level, derived from the event rather than reported by it. | | `reason` | Kubernetes event reason, such as `Unhealthy` or `FailedScheduling`. | | `name` | Name of the object the event concerns, such as a pod or node name. | | `kind` | Kind of that object, such as `Pod` or `Node`. | | `namespace` | Namespace the object lives in. | | `cluster_name` | Cluster the event came from, for an environment spanning more than one. | | `source_component` | Component that reported the event, such as `kubelet`. | | `event_type` | Kubernetes' own type, such as `Warning`, `DiskPressure` or `Disruption`. | | `count` | Occurrences Kubernetes coalesced into one event. The numeric measure the aggregation row offers here. | | `pod_name` | **Filter only.** Pod name — `name` narrowed to objects of kind `Pod`. | | `message` | **Filter only.** Substring match against the event message. | ### Aggregations Filtering queries can be combined with aggregations, to summarize data rather than list matching records. For more information, see Search Aggregations. # Search Aggregations source: https://docs.chalk.ai/docs/search-aggregations ## Summarize search results from logs, access logs, kube events, and spans with counts, percentiles, and other measures. A search on its own returns a list of matching records. The aggregation row beneath the search bar allows you to calculate summary statistics instead. It reads as a sentence: Everything past top lives in the ··· advanced options, visible only once you have grouped by something. The same row appears on the Logs, Access Logs, Kube Events, and Traces (Spans view) explorers, and on dashboard data widgets. It is independent of the search bar above it, which filters records using the syntax in Search Syntax. The leading control is the view: List, the default, shows the matching records themselves. Anything else is a measure, which replaces the list with a table of groups and draws one chart series per group. ### Measures | Measure | Operates on | Description | | ---------------- | -------------- | ----------------------------------- | | **Count** | Records | Number of matching records. | | **Count unique** | String fields | Distinct values of a field. | | **Percentile** | Numeric fields | p50, p95, and so on. | | **Mean** | Numeric fields | Arithmetic mean. | | **Max** | Numeric fields | Largest value. | | **Min** | Numeric fields | Smallest value. | | **Sum** | Numeric fields | Total. Most useful for Kube Events. | Every surface offers Count and Count unique; the numeric measures are only available on some: - Access Logs and Spans offer the full set over duration. - Kube Events offers Sum over count, the aggregated occurrence counter Kubernetes keeps. This is more useful than the Count measure for most Kube Event searches. Other numerical measures are deliberately absent, as most statistics over an already aggregated count are not useful. - Logs have no numeric field, so Count and Count unique are the whole list. ### Grouping and Limiting group by adds up to three dimensions, restricted to a fixed set of groupable fields (see Fields By Surface) for performance reasons. With no dimension the result is a single number over everything matched. top caps how many groups come back, i.e. rows in the table and series in the chart. Groups are sorted server-side by the measure, so top 5 means the five groups with the largest value for the measure, or the smallest value if you sort ascending. (none) collects records whose grouped field is empty; set the relevant input to hide to exclude them. (other) folds everything past the top cut into one row, ranked between the real groups. Folding needs a measure that can be easily recombined from parts: currently Count, Sum, Min and Max generate this row by default, while Mean, Count unique, and percentiles always omit it. The header always tells you the total number of groups found in the time window. ### Comparing several measures You can search for up to four measures in one query using the + button next to the first measure. The common case is finding percentile skew for duration, or comparing mean and median. You might also want to compare both duration and frequency between different groups. The table gains a column per measure. The chart draws a series per group and measure, so three measures over five groups is fifteen series; top still counts groups, not series. ### Which measure ranks the result With more than one measure, you can choose which one to order the groups by. sorted by chooses which groups come back at all, not merely their order, and displays the selected measure on the far-right of the table with green value bars. In the example above, operations counted 15 and 12 outrank one counted 3.78K, because the ranking is by p95 rather than by volume. ### Mixed units Counts and duration statistics cannot share a chart axis meaningfully, so if you select both, a selector above the chart groups measures into families and picks which family to draw. Multiple percentiles are displayed together. Only the chart on the explorer pages is affected: the table still shows every measure, and dashboard charts have no such restrictions. ### Sharing and dashboards The aggregation is part of the URL, so a link reproduces the chart and table you were looking at. The same aggregations are available in dashboard data widgets - as a timeseries, a table, or a single statistic. A statistic reduces to one number, so it takes one measure and no grouping. # Tracing source: https://docs.chalk.ai/docs/tracing ## Use tracing to debug and optimize query performance. Chalk provides traces for online queries so you can identify performance bottlenecks and optimize low-latency queries. A trace breaks down every function call, from feature computation to data retrieval, and shows where time is spent. Chalk's tracing is built on OpenTelemetry standards. Traces are stored in a ClickHouse database deployed within your Kubernetes cluster and viewable via the query pages in the Chalk dashboard. Chalk, by default, sets a TTL of 7 days for trace data, but will adjust this automatically as storage limits are approached. ### Enabling traces Tracing is typically enabled by default in Chalk environments. In order to fetch a trace for an online query, you can either add a flag at query time or set configuration variables for environment-wide behavior. ### Requirements To use tracing features, add the following Python dependency: - chalkpy[tracing] This links required peer dependencies for OpenTelemetry tracing: - opentelemetry-api - opentelemetry-sdk Tracing is fully supported in the Python client starting with chalkpy 2.95.9. ### Query-time traces To fetch a trace for a specific online query, use the --trace flag in the Chalk CLI: ``` chalk query --in customer.id=123 --out customer.num_recent_transactions --trace ``` You can also specify a trace flag in your ChalkClient call: ``` from chalk.client import ChalkClient ChalkClient().query( input={"customer.id": 123}, output=["customer.avg_recent_transactions_amt"], trace=True ) ``` After you run a traced query, open the Trace tab of the Online Query page. The trace opens as a flame graph of the query's spans. Click a span to see its attributes in the drawer below the graph, or switch to the Waterfall, Span list, Resolvers, Details, Logs, or Metrics view. ### Environment configuration Following OpenTelemetry standards, you can also configure tracing via configuration variables. Tracing across your environment can be configured using the configuration variables OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, which you can set in your dashboard under Integrations > Config Variables. The default values of OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.01 will be used, unless an explicit value is set. The supported sampling strategies are detailed below: | Configuration variables | Description | | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_TRACES_SAMPLER=always_on` | Trace 100% of online queries; no query-time overrides | | `OTEL_TRACES_SAMPLER=always_off` | Disable tracing for all online queries; no query-time overrides | | `OTEL_TRACES_SAMPLER=traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.005` | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.005 for 0.5%); no query-time overrides | | `OTEL_TRACES_SAMPLER=parentbased_traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.01` | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.01 for 1%) if there is no parent span; accepts query-time overrides | | `OTEL_TRACES_SAMPLER=parentbased_always_on` | Trace 100% of online queries if there is no parent span; accepts query-time overrides | | `OTEL_TRACES_SAMPLER=parentbased_always_off` | Disable tracing for all online queries if there is no parent span; accepts query-time overrides | When configuring tracing via configuration variables, you can either enforce tracing settings for all queries ("always_on" or "always_off") or enable query-time overrides ("parentbased_*" samplers). For example, if you set OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.01, then 1% of online queries will be traced by default, but you can still enable tracing for specific queries using the --trace flag or the trace=True argument in ChalkClient. However, if you set OTEL_TRACES_SAMPLER=always_on, then all online queries will be traced, and the query-time flags will be ignored. ### Troubleshooting Traces not showing up, a "ClickHouse server is not currently reachable" error, or a trace waterfall that does not render? See the Telemetry FAQ for fixes to common telemetry pipeline issues. # Tracing Policy source: https://docs.chalk.ai/docs/workload-settings/tracing ## Set how many traces Chalk keeps for a function, scaling group, or sandbox with the tracing setting. The tracing setting applies to functions, scaling groups, and sandboxes. It sets how many traces Chalk keeps for the workload. Its default and behavior differ by component. ### Functions Tracing helps you understand where time is spent when a function runs. Use it to debug slow calls, follow nested function calls, and see which parts of your application are doing work. After a traced call runs, view it from the function's Scaling Group page under Remote Call Traces. ### Enabling tracing Set tracing=True on a function: ``` import chalkcompute @chalkcompute.function( cpu="1", memory="2Gi", tracing=True, ) def score_transaction(transaction_id: str) -> float: return 0.42 ``` Nested calls appear in the same trace when tracing is enabled on each function: ``` import chalkcompute @chalkcompute.function(tracing=True) def fetch_profile(user_id: str) -> dict: return {"user_id": user_id, "segment": "active"} @chalkcompute.function(tracing=True) def score_user(user_id: str) -> float: with chalkcompute.span("score_user.fetch_profile"): profile = fetch_profile.remote(user_id) return 0.9 if profile["segment"] == "active" else 0.1 with chalkcompute.span("demo.score_user"): score = score_user.remote("user_123") ``` In the trace you can see the outer function, the call to fetch_profile, and the inner function's execution together. Add manual spans like score_user.fetch_profile to label important parts of your own code. ### Manual spans Use chalkcompute.span() to label meaningful application work: ``` with chalkcompute.span("checkout.score_transaction"): score = score_transaction.remote("txn_123") ``` For a whole caller function, use @chalkcompute.traced: ``` @chalkcompute.traced("batch.score_transactions") def run_batch(transaction_ids: list[str]) -> list[float]: return [score_transaction.remote(txn_id) for txn_id in transaction_ids] ``` Manual spans are useful around model calls, retrieval steps, preprocessing, and other code you want to separate in the trace. To see a remote function's own execution, enable tracing=True or a TracingPolicy on that function. ### Tracing policies Use TracingPolicy when you only want to collect a percentage of calls: ``` @chalkcompute.function( tracing=chalkcompute.TracingPolicy(sample_rate=0.25), ) def recommend(user_id: str) -> list[str]: ... ``` sample_rate must be between 0.0 and 1.0. | Configuration | Behavior | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `tracing=True` | Collect traces for every call. | | `tracing=TracingPolicy(sample_rate=0.25)` | Collect traces for about 25% of calls. | | `tracing=TracingPolicy(sample_rate=0.0)` | Trace this function only when its caller is already traced. | | `tracing=False` | Turn tracing off for the function. | | Omitted | Use the default behavior: only continue a caller's trace if tracing support is already installed in the image. | Once a call is traced, nested function calls stay in the same trace. tracing=True and TracingPolicy(...) automatically install tracing support for functions deployed with @chalkcompute.function or RemoteFunction. If you omit tracing, Chalk does not add those packages for you. The default can still continue an existing trace if your image already includes tracing support, for example because it installs chalkcompute or chalk-remote-call-python[tracing]. ### Troubleshooting If you do not see remote function spans: - Redeploy the function after changing tracing. - Enable tracing on both outer and inner functions to see both functions' execution in a nested trace. - Wrap local tests in chalkcompute.span() if you want the call to be part of an explicit trace. - Make sure you are looking at the called function's Scaling Group page and a time range that includes the call. ### Scaling groups Chalk collects and stores the traces your app sends with OpenTelemetry. Chalk sets the standard OpenTelemetry environment variables in each replica, including OTEL_EXPORTER_OTLP_ENDPOINT, which OpenTelemetry SDKs read to find where to send traces. An app produces traces only if it is instrumented with an OpenTelemetry SDK. By default, Chalk keeps about 1% of traces. Set tracing to change how many traces it keeps: ``` import chalkcompute img = chalkcompute.Image.debian_slim("3.12").entrypoint( ["python", "-m", "http.server", "8080"] ) sg = chalkcompute.ScalingGroup( image=img, name="hello-api", port=8080, tracing=chalkcompute.TracingPolicy(sample_rate=0.25), ).deploy() ``` | Configuration | Behavior | | ----------------------------------------- | ------------------------- | | Omitted | Keep about 1% of traces. | | `tracing=True` | Keep every trace. | | `tracing=TracingPolicy(sample_rate=0.25)` | Keep about 25% of traces. | | `tracing=False` | Keep no traces. | sample_rate must be between 0.0 and 1.0. Unless tracing is False, a request that arrives as part of a trace Chalk is already keeping stays in that trace, whatever the sample rate. tracing sets the OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG environment variables. Set the sample rate with tracing or with those variables in env; a group cannot use both. ### Sandboxes tracing sets trace sampling for processes inside the sandbox. tracing=True traces every request, a TracingPolicy sets the root sample rate, and tracing=False turns tracing off. # Telemetry FAQ source: https://docs.chalk.ai/docs/telemetry-faq ## Troubleshoot missing logs and traces, and other telemetry pipeline issues. Telemetry is the observability pipeline that powers the logs and traces you see in the Chalk dashboard: the structured-logs view, the query-logs view, and the Trace tab on the Online Query page. It runs entirely inside your Kubernetes cluster, and almost all of it is configurable under Settings > Shared Resources > Telemetry. This page answers the most common questions about why telemetry might not behave as expected, and how to resolve them yourself. Telemetry covers logs and traces, not metrics. Metrics flow through a separate pipeline (the Metrics Database / TimescaleDB) configured under Settings > Shared Resources > Metrics Database. If a metric or chart is missing, see Metrics Monitor and Metrics Export instead. ### How telemetry works Logs and traces flow through four stages, all running in your cluster in the chalk-telemetry namespace: Collector (OpenTelemetry or Vector) → Aggregator → ClickHouse → Dashboard - The collector receives logs and traces emitted by the engine and resolvers. These pods are named chalk-collector-*. - The aggregator batches and forwards them. This pod is named chalk-telemetry-aggregator-*. - ClickHouse is the database that stores the logs and traces. This is the clickhouse-0 pod. Traces have a default TTL of 7 days, which Chalk shortens automatically as storage fills. - The dashboard reads from ClickHouse to render the logs views and the Trace tab. Because the whole pipeline is just a set of pods, most telemetry problems are infrastructure problems (a pod that can't schedule, or one that's out of memory) rather than bugs in your features or resolvers. The two places you'll work from are: - Settings > Shared Resources > Telemetry: view and edit the deployment (CPU/memory for ClickHouse, the collector, and the aggregator; ClickHouse storage size; collector runtime; and more). The guided form and a raw JSON editor are both available here. - Infrastructure > Kubernetes: inspect pod health. On the All Pods tab, set the namespace filter to chalk-telemetry to see only the telemetry pods and their status, restarts, and age. Click a pod to view its logs. After any change on the Telemetry settings page, Save & Apply and then watch the change through to resolution. Confirm the affected pods actually reschedule and become healthy. Reconfiguring an infrastructure component and walking away is the most common way a small change turns into an outage. ### Why are my logs or traces not showing up in the dashboard? Logs and traces share the same pipeline, so the first checks are the same: - Did the telemetry settings change recently? A recent edit to the ClickHouse or aggregator resources on the Settings > Shared Resources > Telemetry page is the most common trigger. - Are the telemetry pods healthy? Go to Infrastructure > Kubernetes, filter the namespace to chalk-telemetry, and confirm clickhouse-0, the chalk-collector-* pods, and the chalk-telemetry-aggregator-* pod are all Running. If any are Pending, OOMKilled, or in CrashLoopBackOff, see the next question. - For traces specifically, also confirm tracing is actually enabled and sampled. See below. Note the 7-day trace TTL: traces older than roughly a week are expected to age out, and the window can be shorter if ClickHouse storage filled up. A missing old trace is usually expected behavior, not a failure. ### What does "ClickHouse server is not currently reachable" mean? You may see: This is the most frequent telemetry error. It appears when the logs / traces views try to query ClickHouse and can't reach it. It only affects your ability to view telemetry in the dashboard. Online and offline query serving and feature computation are completely unaffected, and your application keeps working normally. Causes, in order of likelihood: - Transient unavailability. ClickHouse was briefly down for maintenance, or its pod was just rescheduled onto a new node. Wait a few minutes and refresh. If it clears, no action is needed. - The ClickHouse (or aggregator) pod can't schedule. Check the clickhouse-0 pod in the chalk-telemetry namespace (Infrastructure > Kubernetes, namespace filter chalk-telemetry). If it's Pending with FailedScheduling events, its resource request is too large for the node it targets. The classic mistake is setting a memory request equal to the node's total memory (for example, a 32Gi request on a 32Gi m6a.2xlarge). Daemonsets and system overhead consume part of every node, so a request that equals node capacity can never be placed and the pod stays Pending forever. Pinning a fixed instance type and a high request makes this especially easy to trip. - OOM / CrashLoopBackOff. The aggregator can run out of memory under heavy query volume, and ClickHouse can restart under memory pressure. Repeated OOMKilled status means memory needs to go up, but only to a value that still fits the node. Self-serve fix: go to Settings > Shared Resources > Telemetry and adjust the memory in the ClickHouse and/or Aggregator sections. - If a pod can't schedule: lower the request so it fits the node (leave headroom below the instance's total memory, for example request 30Gi on a 32Gi machine rather than 32Gi), or raise the instance type. Never set the request equal to the machine size. - If a pod is OOMing: raise the memory request for the resource that ran out of memory (the aggregator is often the right knob), staying under node capacity. Then Save & Apply, confirm clickhouse-0 reaches Running, and reload the dashboard. If it still can't reach ClickHouse after about 10 minutes, contact Chalk support. Leave Serve over HTTP disabled in the ClickHouse settings unless the Chalk team explicitly asks you to enable it, since it can break ClickHouse connectivity. ### I added --trace but no trace appears Work through these in order: - Check the flag syntax. It is --trace, with no space. Writing chalk query … -- trace (a space after --) is parsed as "end of flags" plus a stray positional argument, so tracing is silently not enabled and you get no error. - Check sampling. Tracing across your environment is governed by OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG (Settings > Integrations > Config Variables). The defaults are parentbased_traceidratio at 0.01 (1%). If the sampler is set to always_off, query-time flags are ignored and nothing is traced. With a parentbased_* sampler you can still force a single query with --trace (CLI) or trace=True (ChalkClient). See Tracing for the full table of samplers. - Check the client. Tracing requires the chalkpy[tracing] extra and chalkpy >= 2.95.9. - View it in the right place. A traced query's waterfall appears in the Trace tab of the Online Query page. ### How do I use traces to find a slow query? Once tracing works, traces are the primary tool for locating latency. Enable tracing on a representative query, open the Trace tab, and look for the longest spans. The waterfall shows where time is spent and what called what. For example, it distinguishes fast Redis reads from slow warehouse or Postgres round trips. From there: - Target the offending data source (add indices, increase its CPU/memory, or add read replicas). - Accelerate any unaccelerated resolvers that dominate the trace. See Static Resolver Optimization. - Consider querying through the gRPC client (ChalkGRPCClient), which is generally faster than HTTP. For the broader set of tools for debugging queries (Query Plan Visualizer, Resolver Replay, logging), see Debugging Queries. ### When should I contact Chalk support? Reach out to Chalk support if: - The "not reachable" error persists beyond about 10 minutes after confirming the telemetry pods are healthy. - Pods keep OOMing or restarting even after correctly sizing memory within node limits. - ClickHouse won't schedule and it's unclear which combination of request and instance type will fit. - You're considering Serve over HTTP or other advanced fields. Confirm with the Chalk team first. ### Additional resources - Tracing: how to enable tracing, sampling configuration, and viewing the Trace tab. - Observability Overview: the full observability section, including metrics and alerting. - Log Export: export logs to external monitoring systems and the log field schema. - Debugging Queries: Query Plan Visualizer, Resolver Replay, and logging. - Kubernetes Resources Overview: how Chalk uses Kubernetes resources. # Introspection source: https://docs.chalk.ai/docs/introspection ## View metadata about your deployed Chalk features and resolvers. Occasionally, you may need to inspect a running Chalk environment to review metadata about your configured Chalk pipelines. You can use the provided REST API to query for this metadata as part of your admin dashboards, monitoring tooling, or team knowledge management. Note: if you use a user-scoped token (i.e. one that’s derived from chalk login), you will need to specify the environment in which to trigger your resolver via the X-Chalk-Env-Id header. ### View an environment's live feature and resolver graph The auth token that should be used for the request, such as Authorization: Bearer $(chalk token) The environment ID for which this request should be executed, if the provided Authorization token was not created with an implicit environment ### Response Pipelines object ### Example ``` curl -XGET -H "Authorization: Bearer $(chalk token)" \ -H "X-Chalk-Env-Id: " \ https://api.chalk.ai/v1/environment/graph # Returns { "deployment_id": "", "features": […], "resolvers": […] } ``` ### View a deployment's feature and resolver graph The machine-generated ID for your deployment, usually a 25-character string that begins with c and looks something like cl789789aoa002chalkexample. The deployment can be associated with a preview or production environment. The auth token that should be used for the request, such as Authorization: Bearer $(chalk token) The environment ID for which this request should be executed, if the provided Authorization token was not created with an implicit environment ### Response Pipelines object ### Example ``` curl -XGET -H "Authorization: Bearer $(chalk token)" \ -H "X-Chalk-Env-Id: " \ https://api.chalk.ai/v1/deployments//graph # Returns { "deployment_id": "", "features": […], "resolvers": […] } ``` ### Pipelines The deployment ID that is being described by this graph Array of FeatureInfo objects, one for each Feature in your deployment. Most of the metadata in each FeatureInfo is defined in your Feature() declaration The fully-qualified name of this feature, like user.id The name of this feature, like "User first name" Additional metadata about this feature's value type, including its underlying raw type, whether it's a primary key, and feature version metadata. The user tags associated with this feature An ISO-8601 datetime string describing when this feature was first introduced to your deployment An ISO-8601 datetime string describing when this feature was last modified by a deployment Whether this feature has Offline-to-Online Chalk ETL integration Array of ResolverInfo objects, one for each Resolver in your deployment. The fully-qualified name of this resolver, like example.resolvers.get_user Whether this resolver is configured to run as an online or offline resolver, or a stream or sink resolver The user tags associated with this resolver The declared feature input FQNs to this resolver, as a string array The declared feature output FQNs of this resolver, as a string array If the resolver is configured to run with a schedule, this field will describe that schedule The last time this resolver was modified by a deployment, including its declaration or its implementation, as an ISO-8601 datetime string # Online and Offline Stores source: https://docs.chalk.ai/docs/choosing-online-offline-stores ## Selecting an online and offline store for your environment When setting up a new environment, Chalk provides several options for choosing a storage provider for your online and offline store. The DBMS (Database Management System) that you choose will depend on your specific requirements, as well as your cloud deployment provider. ### Online Store Options The Chalk online store is used for low-latency serving of real-time feature values. Although Chalk's query engine is optimized for efficient computation of feature values by compiling Python and SQL resolvers into Rust and C++, caching computed values in the online store can further improve performance. ### GCP Online Store Options For customers on GCP (Google Cloud Platform), we offer Memorystore for Valkey as the online store: | Store | Description | Performance | Scaling | | ---------------------- | ---------------------------------------- | ------------ | -------- | | Memorystore for Valkey | Valkey (in-memory key-value NoSQL store) | microseconds | vertical | We generally recommend Memorystore for Valkey, as it is optimized for low-latency and high throughput. ### AWS There are two options for customers on AWS (Amazon Web Services): | Store | Description | Performance | Scaling | | ---------------------- | ------------------------ | ------------ | ----------- | | ElastiCache for Valkey | Redis (in-memory store) | microseconds | vertical | | DynamoDB | Key-Value NoSQL database | milliseconds | auto-scales | We generally recommend DynamoDB for its performance optimizations and ElastiCache for Valkey for its low-latency and greater storage capacity for large rows. However, some customers may choose to use RDS depending on their existing data platform. ### Azure For customers on Azure, we offer Azure Cache for Redis for the online store. | Store | Description | Performance | Scaling | | --------------------- | --------------------------------- | ------------ | ---------- | | Azure Cache for Redis | Redis (key-value in-memory store) | microseconds | horizontal | We generally recommend the Azure Cache for Redis to customers on Azure for a low-latency and scalable database with storage based pricing. ### Offline Store Options The Chalk offline store is used for storing historical feature values. The offline store also serves data for offline queries, which can be used for analytics, batch processing, and other use cases. We offer four options for offline stores: | Store | Storage | | --------------------- | --------------------------- | | Google BigQuery | Columnar (Bigtable-based) | | Amazon Redshift | Columnar (Parquet) | | Snowflake | Columnar (Micro-partitions) | | Databricks Delta Lake | Columnar (Parquet) | | Iceberg | Columnar (Parquet) | For customers with GCP cloud deployments, we generally recommend Google BigQuery for its performance with analytical queries. However, generally, customers will choose the data store that they already use within their data platform. # Online Store Capacity source: https://docs.chalk.ai/docs/online-store-capacity ## Understand how your online store fills up, how eviction affects Chalk, and how to add headroom. ### Overview The online store holds the latest computed feature values so queries can be served without recomputing them. It is a fixed-size cache: as you cache more features, for longer, across more entities, it fills. This page explains what happens as the store fills, which parts of that behavior are specific to Chalk, and what you can do about it. The store runs in your own cloud account, so monitoring and alerting on it are yours to configure — the next section covers what to track. This page applies to online stores backed by Valkey or Redis, on any provider — GCP Memorystore, AWS ElastiCache, Azure Managed Redis, Azure Cache for Redis, or self-hosted. The behavior is the same across all of them. If your online store is backed by DynamoDB, see the DynamoDB section below — none of the eviction guidance here applies to it. ### Watching utilization Two signals matter: - Memory utilization: how full the store is, as a percentage of its limit. - Evicted keys: how much data is being dropped to make room for new writes. Your provider publishes both, and each one has its own monitoring and alerting stack. See the provider reference below for the metric names and monitoring guides for your platform. We recommend setting an alert on utilization rather than checking it by hand. Provider guidance generally points at acting once sustained usage reaches roughly 75–80%, which leaves time to add capacity or trim what you cache before eviction becomes significant. Watch the peak across nodes, not the average. In a clustered store, data is distributed across shards and they do not fill evenly. A single shard can be at its limit while the average across the cluster still looks comfortable — and as described in the next section, one shard filling is enough to matter. To see what is consuming the space rather than how much is used, use Online Store Survey. The chalk.online_store.keys table reports key counts, memory usage, and remaining TTL broken down by feature namespace, which is the fastest way to find the features driving your footprint. ### How eviction affects Chalk When a Valkey or Redis store reaches its memory limit, the maxmemory-policy setting decides what gets removed to make room. The policies fall into two families: - volatile-*: only keys that carry a TTL are eligible for eviction. - allkeys-*: any key is eligible, whether or not it has a TTL. Alongside your cached feature values, Chalk stores mapping metadata in the same instance. That metadata is what lets the engine interpret everything else in the store, and it is written without a TTL. Under a volatile-* policy it is never an eviction candidate, so it stays in place. Under an allkeys-* policy it is eligible like any other key, and Chalk depends on it being present to read the store. This matters more than the share of memory it occupies. In a clustered store, the mapping keys are deliberately co-located so they resolve in a single round trip, which means they all live on one shard. One shard reaching its limit under an allkeys-* policy is enough to affect the mapping for the whole store — which is why the peak-across-nodes reading is the one to watch. Chalk requires volatile-lru. For most customers this needs no action, because it is already the default everywhere: - GCP Memorystore: default volatile-lru - AWS ElastiCache: default volatile-lru - Azure Managed Redis: default volatile-lru - Azure Cache for Redis: default volatile-lru Please keep the eviction policy set to volatile-lru. If you would like to explore a different policy, talk to us first and we will work through it with you. ### Features cached indefinitely The same rule applies to your own data. Under volatile-lru, only keys with a TTL can be evicted — so a feature cached with max_staleness="infinity" is never an eviction candidate. It stays resident whether or not anything is still reading it, and frees space only when it is overwritten or deleted. ``` @features class User: id: int # never expires, never evicted — occupies memory until overwritten fraud_score: float = feature(max_staleness="infinity") ``` That is often exactly what you want. But if you have not looked recently, it is worth reviewing your pipelines for features written without a TTL that could reasonably carry one — those are the cheapest capacity you have. Online Store Survey will show you which namespaces are consuming the most memory and how much of their data carries a TTL. See Caching indefinitely for the "infinity" setting, Feature Caching for how max_staleness works generally, and Garbage Collection for how expired values are reclaimed. ### When the store is near full Once the store is at its limit, every new write requires evicting something else. Two consequences follow. Evicted features become cache misses. The next query for that feature recomputes it and reads from the underlying data source instead of serving from memory. Expect query latency to rise and load on your upstream sources to increase as the eviction rate climbs. Eviction is driven by writes, not by time. If the working set no longer fits, data is replaced before the query that needed it arrives — features get computed, stored, and evicted without ever being read, so the work of populating the cache stops paying for itself. There is also a floor on how much can be reclaimed. Because volatile-lru only evicts keys that carry a TTL, once those can no longer free space the store stops accepting new writes rather than evicting further. That boundary is what keeps the mapping metadata in place. ### Adding capacity Valkey and Redis stores scale two ways: - Horizontally, by adding shards, which increases total memory and spreads writes across more primaries. - Vertically, by moving to a larger node type. Your provider documents both paths, and what to expect while the operation runs — see the provider reference. Whenever modifying the cluster, consider the load your cluster is under and whether it will be significantly impacted by the modification operation. ### Reducing what's cached Lowering the resident set is the other lever, and it can be the cheaper one: - Survey first. Online Store Survey tells you which feature namespaces dominate, so you change the ones that matter. - Shorten max_staleness on high-volume features that do not need long retention. - Add a TTL to features currently cached indefinitely, where a bounded lifetime is acceptable. - Remove features from the online store that do not need to be served online at all. These take effect as existing keys expire or are overwritten — a feature already written without a TTL keeps its current lifetime until the next write. ### DynamoDB If your online store is backed by DynamoDB, none of the eviction guidance above applies. DynamoDB is not memory-bound and has no maxmemory-policy; it does not drop data under memory pressure, and cached values are removed only by TTL expiry or explicit deletion. Capacity there is a throughput and storage question rather than an eviction one. See DynamoDB deployment for capacity sizing, and Feature Caching for how expired values are cleaned up. ### Provider reference | Provider | Utilization | Eviction policy | Scaling | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | GCP Memorystore | [Monitor instances](https://cloud.google.com/memorystore/docs/valkey/monitor-instances) · [Metrics](https://cloud.google.com/memorystore/docs/valkey/supported-monitoring-metrics) | [Supported configurations](https://cloud.google.com/memorystore/docs/valkey/supported-instance-configurations) | [Scaling capacity](https://cloud.google.com/memorystore/docs/valkey/about-scaling-instance-capacity) | | AWS ElastiCache | [Which metrics to monitor](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.WhichShouldIMonitor.html) | [Engine parameters](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ParameterGroups.Engine.html) | [Scaling ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Scaling.html) | | Azure Managed Redis | [Memory management](https://learn.microsoft.com/azure/redis/best-practices-memory-management) | [Memory management](https://learn.microsoft.com/azure/redis/best-practices-memory-management) | [Architecture and tiers](https://learn.microsoft.com/azure/redis/architecture) | | Azure Cache for Redis | [Memory management](https://learn.microsoft.com/azure/azure-cache-for-redis/cache-best-practices-memory-management) | [Configure a cache](https://learn.microsoft.com/azure/azure-cache-for-redis/cache-configure) | — | A note on reading these numbers: providers reserve part of an instance's advertised memory for replication, fragmentation, and background saves, so the usable capacity is lower than the headline size. AWS documents this as reserved memory, and the percentage metrics on each platform already account for it — which is why a utilization percentage is a more reliable signal than raw bytes used. Microsoft is retiring Azure Cache for Redis in favor of Azure Managed Redis. If your online store runs on Azure Cache for Redis, see the retirement FAQ for timelines and migration guidance. ### See also - Online Store Survey — find which feature namespaces are consuming memory. - Feature Caching — how max_staleness controls what is cached and for how long. - Online Store Setup — connecting and configuring an online store. - Choosing an Online and Offline Store — store options per cloud. # Online Store Survey source: https://docs.chalk.ai/docs/online-store-survey ## Use SQL Explorer to analyze online store key distribution and memory utilization Chalk SQL provides a virtual table that lets you survey the contents of your Redis or Valkey online store. This is useful for capacity planning, identifying deprecated features that haven't been cleaned up, and understanding which feature namespaces consume the most memory. ### Prerequisites To run online store survey queries, you need: - A Redis or Valkey online store configured and operational - Access to the SQL Explorer in the Chalk dashboard - Queries can run in either synchronous or asynchronous mode — the online store schema is available in both execution paths For background on these tools, see: - SQL Explorer — the query interface - Online and Offline Stores — store configuration options - Chalk Catalog Components — full schema reference ### The chalk.online_store.keys table The chalk.online_store.keys view exposes metadata about every key in your Redis or Valkey online store. The table has the following schema: - key (binary) — The raw key stored in the online store. - key_type (string) — The type of key: Scalar, HasMany, TimeSeries, etc. - cache_prefix (string) — The configured cache prefix for this key. - fqn (string) — The fully qualified feature name (e.g. user.fraud_score). - memory_usage (uint64) — Bytes consumed in the store by this key. - ttl_seconds (uint64) — Time-to-live remaining in seconds. ### Safety warning Querying chalk.online_store.keys without a LIMIT performs a full Redis/Valkey SCAN, which can significantly impact production performance. Always use a LIMIT clause in the inner subquery to cap the number of keys scanned. Because keys are proportionally distributed across the scan space, a limited sample provides a representative picture of your online store's contents. Redis/Valkey are key-value stores and as such do not support indexing on different "columns" out-of-the-box. A "select-from-where" query with a filter will still scan the online store, yielding only rows that match the filter. This means that a filter that doesn't match any keys might perform a full scan -- e.g. SELECT * FROM "chalk.online_store.keys" WHERE fqn='nonexistent_feature'. Support for filtering by primary key (i.e. performing a direct Redis GET instead of a SCAN for a certain id) is not currently supported for this SQL interface but is coming soon! ### Survey queries In the SQL Explorer settings sidebar, enable Persist Results to save query output to cloud storage. This makes results available in the dashboard for later review. Asynchronous queries always persist results automatically, but for synchronous queries you must enable this toggle explicitly. This is recommended for larger survey queries where you want to revisit or share results. ### Sampling keys Start with a small sample to see what's in your online store: ``` SELECT * FROM chalk.online_store.keys LIMIT 100; ``` ### Key distribution and memory by feature namespace Aggregate over a sample to see how keys and memory are distributed across feature namespaces: ``` SELECT key_type, fqn, count(*), sum(memory_usage) FROM (SELECT * FROM chalk.online_store.keys LIMIT 20000) GROUP BY key_type, fqn; ``` ### Ordered by memory usage Find the largest consumers by sorting on total memory: ``` SELECT key_type, fqn, count(*), sum(memory_usage) FROM (SELECT * FROM chalk.online_store.keys LIMIT 100000) GROUP BY key_type, fqn ORDER BY sum(memory_usage) DESC; ``` ### Checking for a specific feature namespace Verify whether a particular namespace still has keys in the store — useful after removing a feature class: ``` SELECT * FROM ( SELECT * FROM chalk.online_store.keys LIMIT 100 ) WHERE fqn = 'my_feature_namespace'; ``` ### Interpreting results - Representative sampling: Because keys are proportionally distributed across the Redis/Valkey scan space, a limited sample gives an accurate picture of the overall distribution. Derive percentages by dividing the count for each fqn by the total count in your sample. - Memory usage: The memory_usage column reports bytes per key. Sum these values per namespace to estimate how much of the store each feature class occupies. - Key types: The key_type column indicates the kind of feature data stored: Scalar for single values, HasMany for relationship data, TimeSeries for time-indexed features, etc. - TTL: The ttl_seconds column shows how long until a key expires. Keys with no expiration will show a large or zero value depending on your store configuration. ### Common use cases ### Capacity planning Use the memory-ordered query to identify which namespaces dominate your online store. This informs scaling decisions and helps you decide whether to tune max_staleness to reduce retention for features that don't need long cache lifetimes. ### Verifying deprecated feature cleanup After removing a feature class from your Chalk project, keys may still exist in the online store. Chalk runs a nightly cleanup job that removes keys for features no longer in active deployments. Use the namespace-specific query above to confirm that the cleanup has completed. ### Reducing online store size If your online store is growing large, consider: - Lowering max_staleness on features that don't need long retention — see Feature Caching - Removing feature classes that are no longer used and waiting for the nightly cleanup - Reviewing HasMany and TimeSeries key types, which tend to consume more memory per entity ### Related resources - Feature Caching — Configure max_staleness for online store retention - SQL Explorer — The SQL Explorer interface - Chalk Catalog Components — Full catalog and schema reference - Online and Offline Stores — Store configuration options # Offline Store source: https://docs.chalk.ai/docs/offline-stores ## Historical feature storage and retrieval. The Chalk offline store persists all computed feature values — enabling monitoring and dataset generation for model training. It is backed by a data warehouse such as BigQuery, Snowflake, or Iceberg. While the online store — backed by low-latency stores like Redis or DynamoDB — caches the latest value of each feature for fast serving, the offline store retains a deep historical record of feature values, timestamped and keyed by entity. The offline store is typically implemented using BigQuery, Snowflake, Iceberg, or other data warehouses. ### How data flows into the offline store Feature values can be written to the offline store from several sources: - Online queries: freshly computed features from online queries are automatically persisted to the offline store. This means your offline store passively accumulates a historical record of every feature value served in production. - Offline queries - offline queries can write to the offline store if specified. - Scheduled resolver runs: scheduled or triggered resolver runs execute in batch across your upstream data sources and write all computed values to the offline store. - Backfills: you can explicitly ingest historical feature data from external sources using batch backfilling. ### How features are stored: Fully Qualified Names Each feature in Chalk has a Fully Qualified Name (FQN) that uniquely identifies it. An FQN is composed of: ``` . ``` The namespace comes from the @features class name (stripped of a trailing "Features" suffix if present), and the feature name comes from the Python attribute name — unless overridden via feature(name=...) or @features(name=...). For example: ``` @features class User: id: int fraud_score: float ``` | Python attribute | FQN | | ------------------------- | -------------------------- | | `UserExample.id` | `user_example.id` | | `UserExample.fraud_score` | `user_example.fraud_score` | Historical feature value tables in the offline store are named after a hash of the feature's namespace, FQN, and internal version. More information on the historical feature value table schema can be found in Chalk catalog components. ### Internal versioning and type safety In addition to the FQN, Chalk tracks an internal version for every feature when naming its offline store table. This internal version is managed automatically and is separate from the explicit user-defined versions you can set with feature(version=...). The internal version increments whenever you change the type of a feature. For example, changing fraud_score from float to int will cause Chalk to write new values into a new table, rather than mixing them with the existing one. This guarantees that the offline store never serves values of incompatible types for the same feature. In practice: - Type changes are safe: old data is preserved under the previous internal version and is no longer served, while new values accumulate in the new table. - No manual migration is needed: Chalk handles the versioning automatically on deployment. - History resets on type change: if you need to preserve historical values after a type change, backfill the new table explicitly. ### Querying the offline store There are a few ways to query the offline store: Offline queries — use ChalkClient.offline_query() from the Python client to retrieve historical feature values, build training datasets, and run batch inference. See Offline Queries for the full API reference, including point-in-time lookups, spine SQL input, timebounds, and large query execution. SQL Explorer — run SQL directly against historical feature value tables from the Chalk dashboard. Navigate to the SQL Explorer tab to query the offline store without writing any code. See Chalk SQL for more detail. ### Offline store options Chalk supports several offline store backends. In general, choose the store you already use in your data platform — Chalk handles routing and FQN-based isolation regardless of backend. | Store | Format | Cloud | Setup guide | | --------------------- | --------------------------- | ---------- | ------------------------------------------------------------------------------------------------- | | Google BigQuery | Columnar (Bigtable-based) | GCP | [BigQuery](/docs/bigquery) | | Amazon Redshift | Columnar (Parquet) | AWS | — | | Snowflake | Columnar (Micro-partitions) | GCP or AWS | [Snowflake (AWS)](/docs/snowflake-deployment) · [Snowflake (GCP)](/docs/snowflake-gcp-deployment) | | Databricks Delta Lake | Columnar (Parquet) | Any | [Databricks](/docs/databricks_offline_store) | | Iceberg | Columnar (Parquet) | Any | [Iceberg](/docs/iceberg-deployment) | For GCP deployments, BigQuery is generally recommended for its performance with analytical queries. For AWS deployments, Redshift or Snowflake are common choices depending on your existing data platform. # Snowflake Offline Store Overview source: https://docs.chalk.ai/docs/snowflake-offline-store ## Understanding Snowflake offline store architecture and multi-environment deployments. Chalk supports Snowflake as an offline store for persisting feature values, enabling historical data access for training set generation and batch inference. This guide explains the architecture, component hierarchy, and how to plan your Snowflake offline store setup — especially for deployments with multiple environments. ### Architecture Overview When using Snowflake as your offline store, components are organized at two levels: cluster-level (shared) and environment-level (per environment). Understanding this hierarchy is essential for planning your setup. A Chalk cluster is a Kubernetes cluster that can host one or more Chalk environments (for example dev, staging, and production) — see Data Plane Architecture for how clusters and environments relate. Most customers run one cluster per cloud region, so a deployment with separate US and EU installs typically has two clusters, one per region. Snowflake Offline Store Architecture Chalk uses fully-qualified names (FQNs) to route data and ensure isolation between environments without contamination. Each environment can use its own storage integration, or share a single cluster-level integration. The storage bucket is the one exception to "cluster-level." Snowflake and its storage bucket are third-party resources that don't live inside a Chalk cluster at all, so the real sharing boundary is the cloud account backing the bucket (the AWS account or GCP project), not the Chalk cluster. For almost every deployment these line up one-to-one (one cloud account per cluster), which is why thinking of the bucket as "cluster-level" gives the right answer in practice. But if your clusters live in separate cloud accounts (the common case for regionally-separated deployments), each account needs its own bucket. See Multi-Account Deployments below for a worked example. ### Time Travel Requirement Chalk's wide-table materialization reads every skinny source table at one shared warehouse timestamp using Snowflake Time Travel. Keep the effective DATA_RETENTION_TIME_IN_DAYS at 1 or greater for the Chalk offline-store schema and its tables. Snowflake defaults this parameter to one day, and tables inherit it from their schema, database, or account unless explicitly overridden. An effective value of 0, whether inherited or set directly on a table, causes wide fills to fail because Snowflake cannot serve the required snapshot. Snowflake change tracking is not required. Audit the effective retention of the tables in an offline-store schema with: ``` SELECT TABLE_NAME, RETENTION_TIME FROM .INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = ''; ``` ### Component Sharing Rules ### Must Be Unique Per Environment | Component | Notes | | ---------- | ------------------------------------------------------------------------------------------- | | **Schema** | Each environment must have its own schema (e.g., `OFFLINE_STORE_DEV`, `OFFLINE_STORE_PROD`) | | Role | Recommended: separate roles for security isolation | | User | Recommended: separate users with separate credentials | ### Must Be Shared (Per Cloud Account) | Component | Notes | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | **Storage Bucket** | One per cloud account — AWS account or GCP project (not per cluster, see [Multi-Account Deployments](#multi-account-deployments)) | ### Your Choice: Share or Separate | Component | Shared | Separated | When to Separate | | ------------------- | ------------------------- | ------------------------------- | --------------------------------------------------------------------- | | Storage Integration | One per cluster | One per environment | Stricter isolation, or separate cloud credentials/IAM per environment | | Database | `CHALK` for all envs | `CHALK_DEV`, `CHALK_PROD` | Regulatory requirements, separate billing | | Warehouse | `CHALK_WAREHOUSE` for all | `CHALK_WH_DEV`, `CHALK_WH_PROD` | Independent scaling, separate cost tracking | | Snowflake Account | One account for cluster | Separate accounts | Cannot have multiple per cluster | The IAM policy/role is tied to whichever storage integration it backs — a shared integration uses one IAM role, while per-environment integrations each have their own. ### Key Takeaways - Schema is the critical isolation boundary: each environment must have its own unique schema - The storage bucket follows the cloud account, not the cluster: one bucket per AWS account or GCP project; clusters sharing a cloud account can share a bucket, but clusters in separate accounts each need their own - Storage integration is flexible: share one across the cluster, or give each environment its own for stricter isolation - Database and Warehouse are flexible: share them for simplicity, or separate them for independent scaling/billing - Chalk handles routing: data is isolated by environment using fully-qualified paths regardless of whether the storage integration is shared ### Multi-Account Deployments Many customers run their US, EU, or other regional deployments as separate clusters in separate cloud accounts, for example one AWS account per region. In that case: - Each cloud account needs its own storage bucket, storage integration, and IAM role or service-account binding. - Within one cloud account, every cluster and environment there can still share a single bucket and storage integration, exactly as described above. - The Snowflake account, database, warehouse, and schema aren't tied to cloud accounts at all: a single Snowflake account can serve environments across multiple cloud accounts and clusters, as long as each cloud account's storage integration is configured to read from and write to that Snowflake account. For example, a customer with a US cluster (two environments) and an EU cluster (one environment), each in its own AWS account, needs two S3 buckets (one per AWS account) but can still use one shared Snowflake account, database, and warehouse across all three environments if they choose to. ### Required Snowflake Permissions Use a dedicated role for Chalk offline-store access. For multi-environment deployments, we recommend a separate role for each offline-store schema. Chalk does not require ownership of the database or schema. The role needs the following minimum privileges on its warehouse and schema: | Object | Privileges | Why Chalk needs them | | -------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | Warehouse | `USAGE` | Execute offline-store queries and writes | | Database | `USAGE` | Access the database containing the offline-store schema | | Schema | `USAGE`, `CREATE TABLE`, `CREATE FUNCTION` | Access the schema, create feature and temporary work tables, and install the `CMIN` and `CMAX` functions used by offline queries | | Storage integration | `USAGE` | Load or unload data through an external cloud location when bulk operations are configured | | Named external stage | `USAGE` | Access the stage, if the deployment uses a named stage instead of a cloud location directly | For example, grant the baseline privileges as a Snowflake administrator: ``` GRANT USAGE ON WAREHOUSE TO ROLE ; GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT CREATE TABLE, CREATE FUNCTION ON SCHEMA . TO ROLE ; -- Required when bulk operations use a storage integration. GRANT USAGE ON INTEGRATION TO ROLE ; -- Required only when the deployment uses a named external stage. GRANT USAGE ON STAGE .. TO ROLE ; ``` CREATE FUNCTION is required because Chalk's offline-store migrations create CMIN and CMAX SQL functions that implement null-safe timestamp comparisons. Chalk creates tables lazily as features and queries evolve. The offline-store role automatically owns the tables and functions it creates, which lets Chalk select, insert, delete, alter, and drop those objects without granting ownership of the database or schema. Creating a storage integration is an administrative setup operation. The Chalk role needs USAGE on the configured integration, not the account-level CREATE INTEGRATION privilege. Similarly, grant CREATE STAGE only if Chalk itself must create a named stage; it is not part of the baseline offline-store permissions. ### Dedicated User Account For a user account created exclusively for Chalk, grant the offline-store role to that user and make it the default role: ``` GRANT ROLE TO USER ; ALTER USER SET DEFAULT_ROLE = ; ``` Configure the Chalk offline-store connection to use explicitly, even when it is the user's default role. ### Shared User Account If an existing user account is also used for a Snowflake data source or another workload, do not add offline-store write privileges to its existing role. Grant the isolated offline-store role to the shared user: ``` GRANT ROLE TO USER ; ``` Set the offline-store connection's role field to . Other connections using the same user can continue to select their existing, narrower roles. ### Existing Offline Stores The role that creates a Snowflake table or function owns it. If an existing Chalk offline store was initialized under another role, granting ALL PRIVILEGES is not sufficient: Snowflake excludes OWNERSHIP from that grant, and Chalk must be able to alter and drop its objects during schema and feature evolution. For a schema dedicated entirely to one Chalk environment, a Snowflake administrator can transfer the existing objects to the new offline-store role: ``` GRANT OWNERSHIP ON ALL TABLES IN SCHEMA . TO ROLE COPY CURRENT GRANTS; GRANT OWNERSHIP ON ALL FUNCTIONS IN SCHEMA . TO ROLE COPY CURRENT GRANTS; ``` Do not run these bulk ownership transfers on a schema containing non-Chalk objects. Transfer only the individual Chalk objects, or provision a dedicated schema instead. For details about Snowflake privileges and ownership transfers, see Snowflake's access control privileges and GRANT OWNERSHIP documentation. ### Multi-Environment Deployment Options When setting up multiple Chalk environments (e.g., dev, staging, production), you have two approaches: ### Option 1: Full Separation (Recommended for Strict Isolation) Create completely separate resources for each environment: | Resource | Dev | Stage | Prod | | --------- | ---------------- | ------------------ | ----------------- | | Database | `CHALK_DEV` | `CHALK_STAGE` | `CHALK_PROD` | | Schema | `OFFLINE_STORE` | `OFFLINE_STORE` | `OFFLINE_STORE` | | Role | `CHALK_ROLE_DEV` | `CHALK_ROLE_STAGE` | `CHALK_ROLE_PROD` | | User | `CHALK_USER_DEV` | `CHALK_USER_STAGE` | `CHALK_USER_PROD` | | Warehouse | `CHALK_WH_DEV` | `CHALK_WH_STAGE` | `CHALK_WH_PROD` | Benefits: - Maximum isolation between environments - Independent resource management and billing - Easier audit trails ### Option 2: Minimal Separation (Simpler Setup) Share what you can, separate only what you must: | Resource | Dev | Stage | Prod | | --------- | -------------------------- | -------------------------- | -------------------------- | | Database | `CHALK` (shared) | `CHALK` (shared) | `CHALK` (shared) | | Schema | `OFFLINE_STORE_DEV` | `OFFLINE_STORE_STAGE` | `OFFLINE_STORE_PROD` | | Role | `CHALK_ROLE_DEV` | `CHALK_ROLE_STAGE` | `CHALK_ROLE_PROD` | | User | `CHALK_USER_DEV` | `CHALK_USER_STAGE` | `CHALK_USER_PROD` | | Warehouse | `CHALK_WAREHOUSE` (shared) | `CHALK_WAREHOUSE` (shared) | `CHALK_WAREHOUSE` (shared) | The suffixes _DEV, _STAGE, _PROD are examples. Use suffixes that fit your use case — other common patterns include _UAT, _SANDBOX, _QA, _TEAM1, or environment-specific identifiers. Benefits: - Fewer resources to manage - Simpler credential management - Lower administrative overhead ### Recommendation Use Option 2 (Minimal Separation) for most deployments. The schema-level isolation provides sufficient separation for feature data, and Chalk's FQN-based routing ensures data integrity. Use Option 1 (Full Separation) when: - Regulatory requirements mandate complete resource isolation - Different environments need different warehouse sizes or configurations - You need separate billing or resource quotas per environment ### Setup Checklist Before starting your Snowflake offline store setup, gather the following: ### From Your Cloud Environment - [ ] AWS Account ID (where Chalk is deployed) or GCP Project ID - [ ] S3 bucket name or GCS bucket name (typically chalk-{organization}-data-bucket) ### From Snowflake - [ ] Snowflake account name (SELECT CURRENT_ACCOUNT_NAME();) - [ ] Snowflake organization name (SELECT CURRENT_ORGANIZATION_NAME();) - [ ] DATA_RETENTION_TIME_IN_DAYS of at least 1 for the offline-store schema and its tables ### Permissions Required - [ ] AWS: Permissions to create IAM policies, roles, and update trust relationships - [ ] GCP: Permissions to create IAM custom roles and assign roles to service accounts - [ ] Snowflake administrator: One-time permissions to create the selected database, warehouse, schema, storage integration, role, and user. Do not grant these administrative permissions to the runtime Chalk role. ### Cloud-Specific Setup Guides Choose the guide that matches your cloud provider: - Snowflake Setup for AWS: S3 storage integration with IAM roles - Snowflake Setup for GCP: GCS storage integration with service accounts # Private EKS API Server Connectivity source: https://docs.chalk.ai/docs/private-eks-api-server ## How Chalk reaches a fully private EKS API server from the metadata plane. Some Chalk customers require the data plane EKS API server to be fully private — it is created with endpointPublicAccess=false so the Kubernetes control plane has no public DNS record or public IP. This page describes how the Chalk metadata plane continues to manage such a cluster, for two deployment topologies: - Chalk-hosted GCP metadata plane reaching the private EKS API server through a Chalk-managed AWS proxy account. The path is GCP metadata plane ↔ Wireguard ↔ Chalk AWS proxy account ↔ VPCE ↔ customer data plane VPC, where an L7 proxy stack in the customer VPC terminates the last hop to the EKS API. - Customer-hosted AWS metadata plane reaching the private EKS API server directly over VPC peering, when the metadata plane and data plane live in separate AWS accounts. No proxy stack is required. The Metadata Plane & Data Plane Communication page describes why EKS API access is required (deployments, scaling, rolling updates, pod health). ### The endpointPublicAccess=false problem When endpointPublicAccess is disabled on the EKS cluster, the API server is only reachable from inside the data plane VPC or from VPCs that have an explicit network path to it. The Chalk metadata plane is by construction outside that VPC, so the two topologies below each describe how to establish that path. ### Topology 1: Chalk-hosted GCP metadata plane In the standard Chalk-hosted deployment, Chalk runs the metadata plane in GCP while the customer's data plane lives in AWS. The full path has three segments: - Chalk GCP metadata plane ↔ Chalk AWS proxy account over a Wireguard UDP tunnel. The Chalk API server in GKE sends Kubernetes API traffic through a Wireguard client; it egresses GCP via Cloud NAT (so the tunnel endpoints are one of Chalk's published static egress IPs) and terminates at a Wireguard peer in the Chalk AWS proxy account. - Chalk AWS proxy account ↔ customer data plane VPC over AWS PrivateLink. The proxy account holds a VPCE interface endpoint that consumes a PrivateLink endpoint service the customer exposes from the data plane VPC. Traffic never crosses the public internet on this hop. - Inside the customer data plane VPC the PrivateLink service fronts an L4 NLB, the NLB targets an EC2 ASG running an L7 proxy, and the proxy forwards to the private EKS API server. The proxy EC2 instance lives in the customer's data plane VPC — not in the Chalk AWS proxy account — because it needs in-VPC DNS resolution for the EKS API hostname and in-VPC reachability to the EKS control-plane ENIs. Chalk GCP metadata plane reaches the Chalk AWS proxy account over WireGuard, then the proxy account reaches the customer data plane VPC through a VPCE PrivateLink endpoint ### Why the L7 proxy is needed EKS occasionally rotates the control-plane ENIs behind the private API server hostname. Any network construct that pinned a specific target IP would break on each rotation. The L7 proxy re-resolves https://..eks.amazonaws.com on every connection, so the ENI rotation is invisible to the PrivateLink endpoint service, the VPCE in the Chalk AWS proxy account, and everything upstream on the Wireguard path. ### The proxy stack, zoomed in Private EKS API server proxy stack in the customer VPC | Component | Location | Purpose | | ---------------------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------- | | **PrivateLink endpoint service** | Customer data plane VPC | The VPCE-consumable front door. Exposes the internal NLB to accepted consumer accounts (the Chalk AWS proxy account). | | **L4 NLB** | Customer data plane VPC | Stable virtual IP in front of the proxy ASG. Internal; only reachable through the PrivateLink service. | | **EC2 Auto Scaling Group running an L7 proxy** | Customer data plane VPC | Resolves the EKS API hostname and forwards the TLS session to the current set of private ENIs. | | **EKS API server (private IP only)** | Customer data plane VPC | Terminates Kubernetes API calls. No public endpoint. | | **VPCE interface endpoint** | Chalk AWS proxy account | Consumes the customer's PrivateLink service. The proxy account's side of the cross-account bridge. | | **Wireguard peer** | Chalk AWS proxy account | Terminates the UDP tunnel from the GCP metadata plane. | ### What the customer has to configure - PrivateLink endpoint service fronting the internal NLB in the data plane VPC, with the Chalk AWS proxy account's AWS account ID on its allowlist. - Internal L4 NLB targeting the proxy ASG. - EC2 ASG running the L7 proxy (Chalk provides the image/configuration). - EKS cluster security group allowing 443 from the proxy ASG's security group. - Private DNS resolution for the EKS API hostname inside the data plane VPC. The proxy uses the default VPC resolver; no additional setup is needed beyond the standard private-cluster configuration. The customer does not configure a VPC peering connection, an internet-facing NLB, or an IP allowlist against Chalk's static NAT IPs — those elements are internal to Chalk's proxy account's connection to GCP. ### Topology 2: Customer-hosted AWS metadata plane with VPC peering When the customer runs their own metadata plane on AWS (the Customer Cloud / Air-Gapped deployment model), traffic stays entirely inside AWS. If the metadata plane and data plane live in the same AWS account, the metadata plane VPC can simply be added as a public-access endpoint consumer on the EKS cluster and route to the API server's private ENI IPs directly. This section covers the case where the metadata plane and data plane live in separate AWS accounts — for example, a customer who isolates control and data workloads into different AWS Organizations OUs. A VPC peering connection (or a Transit Gateway attachment) bridges the two VPCs, and cross-VPC DNS resolution does the rest: once the EKS private hosted zone is associated with the metadata plane VPC, the Chalk API server resolves the EKS API hostname to the private ENI IPs and routes to them over the peering link. Customer-hosted AWS metadata plane to private EKS via VPC peering No L7 proxy or internal NLB is needed here. The ENI rotation that necessitates the proxy in Topology 1 is handled by the customer's own DNS client — it resolves on each connection and picks up the current ENI set. ### Traffic flow - The customer's Chalk API server in the metadata plane VPC looks up the EKS API hostname. Because the EKS-managed private hosted zone is associated with the metadata plane VPC, resolution returns the current private ENI IPs of the EKS control plane. - The metadata plane VPC's route table sends the packet across the VPC peering connection (or Transit Gateway) into the data plane VPC. - The request terminates on the EKS API server's private ENI, authenticated with AWS IAM. ### Prerequisites - Non-overlapping CIDR ranges between the metadata plane VPC and data plane VPC. VPC peering cannot translate addresses. - Route table entries on both VPCs pointing the peer's CIDR at the peering connection. - EKS cluster security group allowing 443 from the metadata plane VPC CIDR. - Cross-account private hosted zone association: the EKS control plane creates its private hosted zone in the data plane account; associate it with the metadata plane VPC (via aws route53 associate-vpc-with-hosted-zone) so DNS resolution works across the peering. No traffic traverses the public internet, and no Chalk-owned IPs or accounts are involved. ### When peering is — and isn't — required | Metadata plane location | Data plane location | Bridge | | ----------------------- | -------------------- | -------------------------------------------------------------------------------- | | Same AWS account | Same AWS account | None — in-account routing to the EKS private ENIs | | Separate AWS account | Separate AWS account | **VPC Peering or Transit Gateway** (with cross-account hosted zone association) | | GCP (Chalk-hosted) | AWS (customer) | Chalk AWS proxy account: Wireguard from GCP, VPCE into customer VPC (Topology 1) | ### Trade-offs vs. public EKS with IP allowlisting The public EKS with IP whitelisting pattern is still Chalk's default recommendation because it eliminates the proxy stack in Topology 1 and the cross-account DNS configuration in Topology 2. The private-API pattern described here is appropriate when: - Regulatory policy prohibits any public endpoint on the Kubernetes control plane, or - A central cloud governance policy denies endpointPublicAccess=true on EKS clusters. The trade-offs to accept with the private-API pattern: - In Topology 1, additional infrastructure in the path: the L7 proxy ASG, the internal NLB, the PrivateLink endpoint service, and the Wireguard tunnel all have to stay healthy. A proxy outage breaks Chalk's ability to deploy, scale, or health-check the cluster. - In Topology 2, VPC peering and cross-account hosted zone state to maintain: route tables, security groups, and the hosted zone association must stay in sync as the customer data plane VPC evolves. For most deployments the simpler public-EKS-with-allowlist pattern is sufficient. The topologies on this page exist for the cases where it is not. # Resource Configuration source: https://docs.chalk.ai/docs/resource-configuration ## Configure your Chalk Kubernetes cluster ### Overview This page covers resources for Chalk's own platform services (the Query Server, Job Queue Consumer, Streaming Server, and similar) via the dashboard. To size a Sandbox, Scaling Group, or Function that you deploy yourself through Chalk Compute, set the cpu, memory, and gpu string parameters in code. See the Compute overview instead of this page. Go to Infrastructure > Resources to open the Cloud Resource Configuration page, where you can configure the resources allocated to each of your Chalk services and most other system-related settings. ### Resource groups and services Your Chalk environment consists of multiple Services, each of which serves a different purpose and represents a different kind of feature, compute, or background workload. For instance, the Query Server (also referred to as the engine) is responsible both for serving online inference queries and also for serving dataplane data requests, the Job Queue Consumer is responsible for pulling jobs like offline queries (backfills, dataset ingestions, and so on) from a queue and running them, and the Streaming Server handles execution and coordination of Stream Resolvers. Chalk also provides a higher-level isolation concept called a Resource Group. Each resource group provides its own independently-configurable set of Chalk Services, and is backed by logically-independent Kubernetes deployments. This provides isolation between different kinds of workloads so that they do not interfere with each other. For instance, you may want to run low-latency high-volume IO-bound inference queries against the Query Server in one resource group, and slower CPU-bound analytics queries against the Query Server in a separate resource group. You may also want to configure several different resource groups of different sizes so that you can target long-running offline workloads against machines of different sizes. Every Chalk environment has a Default resource group. You can add new resource groups and configure each one independently in the Cloud Resource Configuration pane. For online inference queries, each resource group exposes its own URL for the query client to use, and for offline workloads like Offline Query, a resource group can be specified in the resource_request parameter of the ChalkClient.offline_query() method and other related methods. ### Configuring service resources Each Chalk Service can be independently configured to consume different amounts of compute resources, depending on your needs. The primary way to configure resource consumption for a Service is to select a Chalk Machine Type from the dropdown selector on the Cloud Resource Configuration page when viewing the individual Service configuration pane. This guarantees that each replica of the Service will have full access to the listed CPU and memory amounts, and will run isolated on its own cloud instance, ensuring that separate services cannot interfere with each other. Different machine types are available for online, offline, and other kinds of workloads, since these workloads tend to have different performance requirements. In addition to configuring the resources allocated to each replica of a Service, you can configure the Service's scaling properties. Chalk autoscales on CPU utilization by default. The Query Server and gRPC Query Server can also scale on a schedule, and the Streaming Server can scale on stream backlog. You configure the min and max instance counts and the target cpu % for each Service independently in each Resource Group, so different workloads scale independently. More information about Chalk Machine Types can be found on the detailed docs page. ### Autoscaling Chalk supports autoscaling of resources in your environment in order to optimize resource usage. For the branch server, set the Auto Shutdown Period to the duration of inactivity after which Chalk automatically shuts down the branch server pod. The default is 1 hour. See The branch server. For the Query Server and gRPC Query Server, which serve your production traffic, you can configure autoscaling based on a CPU utilization target or set time-based pre-scaling. For stream resolvers, you can also set an autoscaling policy based on the stream queue message backlog. Chalk defines scaling policies with KEDA. See stream source autoscaling for supported sources, requirements, and consumer group configuration. ### CPU-based autoscaling To enable CPU-based autoscaling, set Target CPU % in the Cloud Resource Configuration. Chalk then monitors the CPU utilization of the service and scales the number of instances up or down based on the target CPU percentage. CPU-based autoscaling requires Min Instances, Max Instances, and Target CPU % to all be set for the service. ### Scheduled autoscaling Time-based scaling is available for the Query Server and gRPC Query Server. To autoscale on a schedule, define scaling triggers for a service on the Cloud Resource Configuration page: - Under Scaling, set Min Instances and Max Instances for the service. Time-based scaling is available only when the minimum is lower than the maximum. - Open Advanced, then open Time-based Scaling. - Select a timezone, then select Add Scaling Trigger. - For each trigger, enter the number of instances and the start and end times. Write the start and end times in standard cron syntax. The service scales to that number of instances between the start and end times. A trigger's instance count should fall between the service's minimum and maximum. The form shows a warning when it does not. Select Preview Schedule to chart the scheduled instance count for the previous and the next day. The underlying implementation of time-based scaling is the KEDA Cron scaler. ### Handling predictable traffic bursts Scheduled capacity and CPU-based autoscaling work best together: the schedule handles known demand, while CPU autoscaling provides additional capacity for unplanned or longer-lived demand. They do not replace capacity planning. If traffic can arrive as a large synchronized fan-out, consider spreading it over a short interval with batching or jitter where your application permits. This reduces the amount of capacity that must be available at the exact start of the burst. CPU-based autoscaling suits sustained or unexpected changes in demand, but it reacts after demand rises. When traffic rises abruptly, a new pod must still be created, scheduled, and pass readiness checks before it can serve requests. If the cluster needs to provision a node as well, this can take longer. Requests that arrive at the start of a burst may therefore see increased latency before autoscaling adds capacity. Allow several minutes of lead time for a scheduled increase when additional nodes might be needed. The actual time depends on signal evaluation, pod scheduling, node provisioning, image startup, and readiness checks. Exact times vary by environment, so use the p95 time from a scaling signal to a ready replica to choose the schedule's lead time. When your production traffic has a recurring peak (for example, a daily batch, campaign, or scheduled workflow), use scheduled autoscaling to make capacity available before the peak begins: - Measure the burst. Review request rate at a short interval, end-to-end latency, client-visible errors or timeouts, CPU utilization, and ready replica count. A five-minute average can hide a burst that lasts only seconds or minutes. - Choose a scheduled baseline. Set the desired replica count to the capacity required at the start of the peak, rather than the normal idle baseline. Establish this count with a representative load test or prior peak data. - Start early enough for readiness. Schedule the increase with enough lead time for pod scheduling, image startup, and readiness checks. Include extra time if a new node may need to be provisioned. - Keep overflow headroom. Set Max Instances above the scheduled baseline so CPU-based autoscaling can still respond to demand beyond the expected peak. The maximum should be tested against the resource requests, nodepool capacity, and any downstream dependencies. - Scale down after the peak. End the schedule once traffic has returned to its normal level to avoid paying for unused capacity. ### Programmatic management You can read and edit the same resource-group settings that Infrastructure > Resources exposes from the CLI. This is useful for codifying environment configuration in source control, scripting per-environment changes (image bumps, replica counts, autoscaler overrides), and applying targeted edits without leaving the terminal. The full configuration lives on the environment record under spec_config_json. A typical shape looks like: ``` { "id": "tm-abc123", "name": "production", "spec_config_json": { "services": { "engine-grpc": { "min_instances": 2, "max_instances": 10, "target_cpu_utilization_percentage": 50, "chalk_machine_type": "small" }, "branch": { "chalk_machine_type": "large" } }, "resource_groups": [ { "resource_group_name": "high-throughput", "services": { "engine-grpc": { "min_instances": 5, "chalk_machine_type": "large-highmem", "engine_image_uri_override": "gcr.io/chalk/engine:sha-abc123" } } } ] } } ``` The fields directly under spec_config_json (alongside resource_groups) make up the default resource group. Each entry in resource_groups is a named override group keyed by resource_group_name. Three CLI commands manage this configuration: | Command | What it does | | --------------------------------- | ------------------------------------------------------------------------- | | `chalk environment config --json` | Print the active environment, including `spec_config_json`, as JSON. | | `chalk environment update` | Replace whole top-level fields of an environment from a YAML file. | | `chalk environment patch` | Apply a targeted JSON-merge or RFC 6902 patch to a single resource group. | environment is aliased to env, so chalk env config, chalk env update, and chalk env patch all work. ### Inspecting the current configuration Use chalk environment config --json to dump the active environment. With --json the command emits the full environment proto as JSON; without it you get a YAML rendering. ``` $ chalk environment config --json > env.json # Inspect just the resource groups $ chalk environment config --json | jq '.spec_config_json.resource_groups' # Find a specific group $ chalk environment config --json \ | jq '.spec_config_json.resource_groups[] | select(.resource_group_name=="high-throughput")' # Grab the environment id (needed by `chalk environment update`) $ chalk environment config --json | jq -r '.id' ``` Your CLI auth context determines the active environment. Switch with chalk environment before running the command, or use --env-id to scope individual operations. ### Bulk updates with chalk environment update chalk environment update overwrites whole top-level fields on the environment from a YAML config file. Use this when you want to replace the resource-group configuration wholesale (for example, deploying a checked-in environment.yaml from a repo). ``` $ chalk environment update \ --env-id env_abc123 \ --config-file environment.yaml ``` The YAML file may contain any subset of the following keys; only keys that are present get sent in the update mask, so omitted keys are left untouched on the server: | YAML key | Effect | | -------------------------- | --------------------------------------------------------------------------------------- | | `specs_config` | Replaces the whole `spec_config_json` blob (resource groups, services, autoscalers, …). | | `additional_env_vars` | Replaces the environment-level env-var map (engine configuration variables) | | `private_pip_repositories` | Sets the private PyPI URL string. | | `online_store_secret` | Sets the secret name for the online store. | | `feature_store_secret` | Sets the secret name for the feature/offline store. | A minimal environment.yaml that edits only resource groups looks like: ``` specs_config: services: engine-grpc: min_instances: 2 max_instances: 10 target_cpu_utilization_percentage: 50 chalk_machine_type: small resource_groups: - resource_group_name: high-throughput services: engine-grpc: min_instances: 5 max_instances: 20 engine_image_uri_override: gcr.io/chalk/engine:sha-abc123 ``` specs_config is a wholesale replacement: anything you omit under it is removed. The typical workflow is therefore "fetch → edit → apply": ``` $ chalk environment config --json | jq '.spec_config_json' > spec.json # edit spec.json (or convert to YAML and edit) $ yq -P spec.json > environment.yaml # wrap under `specs_config:` first $ chalk environment update --env-id $(chalk environment config --json | jq -r .id) \ --config-file environment.yaml ``` For per-field edits without round-tripping the whole config, prefer chalk environment patch. ### Targeted patches with chalk environment patch chalk environment patch applies a kubectl-style patch to a single resource group within the active environment's spec_config_json. The rest of the spec config is preserved as-is, which makes it safe to script narrow changes (image bumps, replica overrides, autoscaler tweaks) without touching unrelated services. The command operates on the environment selected in your CLI auth context, and it has no --env-id flag. Switch with chalk environment first if needed. There are two input modes. ### Sugar form: --service + --set Use this form to change a single field on a single service inside a resource group: ``` $ chalk environment patch \ --resource-group high-throughput \ --service engine-grpc \ --set engine_image_uri_override=gcr.io/chalk/engine:sha-abc123 ``` Each --set key=value entry nests under services. of the resource group. You can pass --set multiple times. Values parse as JSON when they look like JSON (true, 42, "quoted", [1,2,3]); otherwise they are treated as strings. Dotted keys build nested objects, so: ``` $ chalk environment patch \ --resource-group high-throughput \ --service engine-grpc \ --set min_instances=5 \ --set max_instances=20 \ --set resources.requests.cpu=500m \ --set resources.requests.memory=1Gi ``` is equivalent to a merge patch of: ``` { "services": { "engine-grpc": { "min_instances": 5, "max_instances": 20, "resources": { "requests": { "cpu": "500m", "memory": "1Gi" } } } } } ``` ### Raw form: --patch / --patch-file For anything the sugar form cannot express (multiple services in one shot, deletes, array operations), supply a raw patch body. The --patch-type flag selects the format: - merge (default): An RFC 7396 JSON merge patch. Setting a field to null deletes it. - json: An RFC 6902 JSON patch (an array of {op, path, value} operations). ``` # Inline merge patch $ chalk environment patch --resource-group high-throughput \ --patch-type merge \ --patch '{"services":{"engine-grpc":{"engine_image_uri_override":"gcr.io/chalk/engine:sha-abc123"}}}' # Delete a field by setting it to null in a merge patch $ chalk environment patch --resource-group high-throughput \ --patch '{"services":{"engine-grpc":{"engine_image_uri_override":null}}}' # RFC 6902 patch from a file $ chalk environment patch --resource-group high-throughput \ --patch-type json --patch-file replace-image.json # Read the patch from stdin $ jq -n '{services:{"engine-grpc":{min_instances:5}}}' \ | chalk environment patch --resource-group high-throughput --patch-file - ``` Sugar (--service/--set) and raw (--patch/--patch-file) modes are mutually exclusive. ### Patching the default resource group The fields directly under spec_config_json (the ones that live alongside resource_groups) are treated as the default resource group. Pass --resource-group default to patch them: ``` $ chalk environment patch --resource-group default \ --service branch \ --set request.cpu=2 \ --set request.memory=6Gi ``` Patches that target default cannot touch the resource_groups key. Patch each named group by name instead. ### Previewing and confirming changes Every patch prints a JSON diff of the resource group before and after the change, then prompts for confirmation before applying. Use --dry-run to see the diff and exit without applying, and --yes (or -y) to skip the prompt in scripts: ``` # Preview only $ chalk environment patch --resource-group high-throughput \ --service engine-grpc --set min_instances=5 --dry-run # Apply non-interactively (CI) $ chalk environment patch --resource-group high-throughput \ --service engine-grpc --set min_instances=5 --yes ``` The patch fails immediately if the named resource group does not exist, if the patch body is invalid JSON, if the merge would change resource_group_name, or if a default patch would mutate resource_groups. ### Shared resources In Settings > Shared resources, environment admins can view and manage shared resources across different services in your environment, including the Metrics Database configuration, Gateway configuration, and Background Persistence configurations. You can adjust these configurations here if, for example, you need to scale up your Metrics Database to handle larger volumes of data, or if you drastically increase the volume of data being written to the online or offline stores. To autoscale Background Persistence workers as the volume of data persisted online and offline varies, set the Horizontal Pod Autoscaling (HPA) Settings in the Background Persistence Configuration tab. Autoscaling requires you to set: - Max replicas - Min replicas - Target average value (the approximate target undelivered message count in queue per replica) - Pub/Sub Subscription ID ### Engine configuration variables In Integrations > Config variables, you can view and edit global configuration variables for data plane services like the query server or offline query consumer for your environment. Contact the Chalk team with any questions about which configuration variables to set for your use case. Setting configuration variables directly is an advanced operation for enabling experimental or advanced features. In most cases, use the Resources pane in Infrastructure > Resources instead. Configuration variables are directly injected into the query server or offline query jobs (the Chalk execution engine runtime) for the purpose of configuration. To make values available as environment variables in your resolver runtime, set them in Integrations > Secrets & variables instead. ### (Legacy) Custom resources configuration The legacy resources configuration system allows flexible configuration of your Chalk workloads, but it can also cause resource provisioning issues that are hard to debug. It remains available as an advanced form of configuration. Use it only for special cases that Chalk Machine Types do not cover. The primary controls that are available for configuring individual Chalk Services in this system are effectively passthroughs for the underlying Kubernetes API: requests and limits. Each of these allows specifying amounts of an available resource: cpu, memory, and ephemeral storage. Since the underlying system is the Kubernetes Resource API, specifying a request reserves the specified amount of that resource; any node that the service schedules on must have at least that much of that resource available; the available amount is calculated by adding up all the other resource requests (reservations) on that machine along with Kubernetes overhead and subtracting that from the available amount of the resource on the machine. For example, if there is an 8-CPU machine (node) in the cluster, and there are 3 pods running on the node requesting 2 CPU each, the node will have (8 - 2*3 - overhead)=~1.5cpu available. A new pod with a 1-CPU request may be able to schedule on the node, but a new pod with a 2-CPU request will not, because of the overhead. Because this overhead is always present, a request for 2 CPU or 4Gi of memory can never schedule on a node of that size. It typically schedules on a node twice the size (4 CPU or 8Gi of memory). Requests reserve capacity so that too many pods do not schedule on the same node. They do not require the pod running your workload to use only the requested amount, and this applies to all pods running on a node. Setting limits instructs Kubernetes to terminate a pod that uses too much memory, or to throttle a pod that uses too much CPU. With Chalk Machine Types, Chalk sets requests and limits automatically, so you do not need to fine-tune resource requests to make sure your workload can schedule. ### (Legacy) Isolating your workloads The legacy resources configuration system contains a few different techniques for isolating your workloads, depending on your cloud provider: - AWS/Azure (Karpenter): Your services can explicitly select an Instance Type and/or a Nodepool. - GCP: Your services can explicitly select a GCP Nodepool or a GCP Machine Family. - All clouds: Your services can isolate from each other with the Isolate from other Chalk Services checkbox. AWS and Azure clusters use Karpenter, a flexible node provisioning system that creates nodes on-demand. Karpenter provides the Nodepool as an available tool for isolation and configuration; Nodepools can be configured with an arbitrary and flexible set of constraints that determine what kinds of nodes they can produce. This flexibility can lead to internally unsatisfiable configuration that prevents workloads from scheduling. In clusters using Karpenter, you can explicitly select a Nodepool via the Nodepool selector and you can independently select an Instance Type with the Instance Type selector. Using the Instance Type selector ensures that you will schedule on exactly that type of instance. In GCP, each Nodepool can produce only one Instance Type, so you cannot independently select Instance Type and Nodepool for your Services. If your cluster is set up with NAP then you can also select a GCP Machine Family, although this gives you less control. In all clusters, you can use the Isolate from other Chalk Services checkbox to set an anti-affinity that prevents your selected Service from running on the same node alongside other services; this is recommended for production configurations in order to avoid cross-service interference. This system is flexible, and it is possible to set various kinds of unsatisfiable configuration, which makes problems hard to debug. For instance, you can configure a Karpenter Nodepool that only makes m6a.2xlarge instances, then explicitly request that nodepool, explicitly request an instance type of r8i.4xlarge, and independently configure a CPU request of 32. These configurations are all mutually incompatible: the Nodepool cannot make instances of the required type, and the 32-CPU request cannot fit on either of the specified instance types (it cannot fit on the larger machine because of overhead). When you use a Chalk Machine Type, all of these settings are chosen automatically, including requests, limits, the Instance Type, the Nodepool, and the isolation setting. For this reason, we recommend selecting a Chalk Machine Type unless you need custom configuration. More information is available on the detailed docs page. ### (Legacy) nodepools Configuring and using custom nodepools in AWS, GCP, and Azure environments is a legacy feature that will eventually be deprecated. Selecting a Chalk Machine Type instead of using custom resource configuration ensures that your workloads always run isolated from each other on an appropriately sized cloud instance. In Infrastructure > Nodepools, you can view and configure the nodepools used to run your Chalk environment. A NodePool defines constraints on the Kubernetes nodes that can be scheduled in your Kubernetes cluster. You can select the machine families to use in your deployment, as different machine families are optimized for different workloads. You can also set a CPU limit on your NodePool. Use this page to provision the resources and machine families that best suit your workload and usage requirements. Configuring a nodepool includes two checkboxes, "Isolate this nodepool" and "Restrict to Chalk workloads only". When "Isolate this nodepool" is set, workloads cannot be scheduled in the nodepool unless they are specifically configured to do so under Infrastructure > Resources, which can be useful to minimize interference or reserve certain instance types for specific workloads. This gives the nodepool the Isolated label. When "Restrict to Chalk workloads only" is set, unrelated non-Chalk workloads in the cluster cannot schedule on this nodepool, which can be useful for environments deployed into pre-existing clusters that also run unrelated workloads. This prevents those non-Chalk workloads from inadvertently consuming Chalk credits. This gives the nodepool the Restricted label. Nodepools that you create through the Chalk dashboard automatically have the Chalk label. This label indicates that the nodepool makes Chalk-managed nodes, whose uptime contributes to credit usage. The dashboard also displays nodepools without the label. Chalk workloads can only schedule on nodes from nodepools with the Chalk label. More details about these labels can be found in the billing documentation. After you configure your Nodepools, go to Infrastructure > Resources to select the Nodepool to use for each service in each resource group in your environment, as well as the Pod Disruption Budget. # Chalk Machine Types source: https://docs.chalk.ai/docs/chalk-machine-types ## Configure your Chalk Kubernetes cluster ### Overview Chalk Machine Types provide a simple and robust interface for configuring cloud resource consumption for workloads running in your Chalk environment. They are a replacement for the legacy configuration system that required more direct configuration of the underlying Kubernetes API and provide improvements in performance, simplicity, and durability of your workloads, as well as enhanced usage reporting. ### What is a Chalk Machine Type? Chalk Machine Types are a simple way of categorizing cloud instances of different sizes, based on the amount of CPU and memory they have available. For instance, small corresponds to a cloud instance with 4 vCPU and 16GiB of memory. Each Chalk Machine Type corresponds to a single concrete instance type in the appropriate cloud provider backend, along with an ordered list of fallback instance types that will automatically be used if the primary instance type is not available due to a cloud provider stockout or other availability issue (like regional availability restrictions). For an online workload, small in an AWS region may map to m8a.xlarge, then fall back to m8i.xlarge, m7a.xlarge, and m7i.xlarge in that order when an instance type is unavailable. This list of fallbacks is configurable per-cluster and per-environment. When you pick a Chalk Machine Type, your workload is guaranteed to run isolated from other Chalk workloads on its own cloud instance, with appropriate requests and limits automatically configured to ensure that it will schedule and that it can make full use of the resources on the machine. ### Usage Labeling for Chalk Machine Types Workloads configured with a Chalk Machine Type will run on Karpenter nodepools created and configured by Chalk in AWS and Azure, and will run on equivalent ComputeClasses in GCP. These nodepools and ComputeClasses will provide appropriate labeling for the Chalk credits consumed by these workloads, depending on the type of workload: online, offline, compute, infrastructure, etc. These nodepools and ComputeClasses are not directly configurable and are automatically created and configured by Chalk. ### Disk Spilling for Offline Workloads For offline workloads, using a Chalk Machine Type will automatically enable disk spilling via local SSDs. Offline workloads will be automatically configured to take advantage of this feature, which significantly reduces memory usage for large workloads, reducing their cost and improving their efficiency. No manual configuration will be required to set up disk spilling when using a Chalk Machine Type. ### GPU Support for Online Workloads Some inference and training workloads require access to a GPU on the selected instance type. GPU instance support for Chalk Machine Types is not yet enabled but will be available soon. ### Chalk Machine Types and Resource Groups Chalk Machine Types integrate nicely with the high-level categorization and isolation provided by Resource Groups. Using a Chalk Machine Type already provides inter-service isolation, and moving workloads with different logical isolation requirements (often useful for separating different business use-cases) to different resource groups allows both for isolation between use-cases and also allows you to select different machine sizes according to the compute needs of each different use-case. For offline workloads like the Job Queue Consumer, it can be convenient to make several different Resource Groups, each configured to run the consumer against a different Chalk Machine Type, and potentially each with different scaling limits. Then, workloads like offline queries can be directed at the appropriate Resource Group depending on the resources they need, and resource consumption can be controlled via the different scaling configurations. ### Customizing Chalk Machine Types Chalk Machine Types are mapped to a primary instance type along with an ordered list of fallback instance types, depending on regional availability. The default list of instance types can be customized in the organization-level settings page for viewing and configuring attached clusters. You can change the order in which different instance types will be chosen as fallbacks, and you can change which instance types are available at all for a given Chalk Machine Type. This can be useful, for example, if you want to ensure that your environment always tries to use the fastest instance types available for maximum performance, or less expensive instance types for cost savings. Customizing Chalk Machine Type definitions requires organization-level (team-level) permissions. Browse the default instance and fallback catalog to see the built-in lists for each cloud provider, workload, machine size, disk spilling setting, and GPU setting. Regional availability and your cluster or environment overrides can change the list used by your workloads. ### Migrating to Chalk Machine Types Migrating to Chalk Machine Types from a custom configuration is a fairly simple process in most cases. Select the machine type that corresponds most closely to your currently-configured CPU and memory requests, and click Save and apply. Note that the minimum size of machine type that is suitable for production workloads is small-highcpu with 4 CPU and 8Gi of memory; the micro machine type is not recommended for production workloads. There are a few edge cases and caveats that you may run into: ### Account for Kubernetes overhead Because Kubernetes nodes have inherent overhead, pod requests never perfectly match up one-to-one with the node size that a pod is eventually scheduled on. For instance, a request of 4 CPU and 16Gi cannot schedule on a machine that only has 4 CPU and 16Gi available, and will always schedule on a larger machine, often a machine that is twice as large (8 CPU and 32Gi). When migrating a service from custom config to a Chalk Machine Type, you can choose a machine type that corresponds either to the existing request, or to the size of the machine that the workload was actually scheduled on. We recommend picking the size that corresponds to the existing request, but you may find that the workload's performance implicitly depended on the extra resources from the actual scheduled node. If the workload's performance degrades because of the smaller node size, we recommend picking the larger size. ### Translate custom instance type configuration Each Chalk Machine Type corresponds to a "primary" concrete instance type in the underlying cloud provider, and then falls back to an ordered series of fallback instance types if the primary is out of stock in your region. If you configured your workloads to use specific instance types from the underlying cloud provider (EC2 Instance Type in AWS, Nodepool / Instance Type in GCP), then you may find that switching to a machine type causes the system to choose a different underlying instance type than your custom config, which may mean a different CPU generation or CPU manufacturer. Typically, the system will pick the fastest and most modern available processor type, so you are likely to see a performance improvement rather than a degradation, but if you have a specific instance type configured it may be helpful to check the default instance and fallback catalog to see what underlying instance types your workload will select when migrated. If you want to change which concrete instance types correspond to your selected machine type in order to balance performance and cost considerations or guarantee a specific instance type, then it's possible to override the defaults in the organization-level cluster settings page. ### Remove disk spilling configuration (offline only) In the legacy custom resource configuration system, special settings were required in order to enable disk spilling for offline workloads like the job queue consumer or async offline query. With Chalk Machine Types, disk spilling is always automatically enabled for offline workloads, and those custom values may conflict and cause spilling to fail or cause the workload to fail to boot. Remove these custom values from your offline workload configuration before migrating to a machine type: - Engine Configuration Variables CHALK_VELOX_SPILL_DIRECTORY and CHALK_VELOX_QUERY_SPILLING_MODE - Custom Kubernetes Volume Mounts that point to a spill directory ### List of Available Chalk Machine Types Machine types larger than xlarge are only available for offline workloads. "High-CPU" machine types are only available for online workloads. "High-memory" and "high-cpu" indicate a larger and smaller ratio of memory to vCPU respectively on the instances compared to the "standard" machine type sizes, approximately: - High-CPU: 2GiB per vCPU - Standard: 4GiB per vCPU - High-memory: 8GiB per vCPU This is the full list of machine type sizes: Standard | AWS Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `small` | 4 | 16Gi | | `medium` | 8 | 32Gi | | `large` | 16 | 64Gi | | `xlarge` | 32 | 128Gi | | `2xlarge` | 48 | 192Gi | | `3xlarge` | 64 | 256Gi | | `4xlarge` | 96 | 384Gi | | `5xlarge` | 128 | 512Gi | High-CPU | AWS Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `small-highcpu` | 4 | 8Gi | | `medium-highcpu` | 8 | 16Gi | | `large-highcpu` | 16 | 32Gi | | `xlarge-highcpu` | 32 | 64Gi | High-memory | AWS Machine type | vCPU | Memory | | ----------------- | ---- | ------ | | `small-highmem` | 4 | 32Gi | | `medium-highmem` | 8 | 64Gi | | `large-highmem` | 16 | 128Gi | | `xlarge-highmem` | 32 | 256Gi | | `2xlarge-highmem` | 48 | 384Gi | | `3xlarge-highmem` | 64 | 512Gi | | `4xlarge-highmem` | 96 | 768Gi | | `5xlarge-highmem` | 128 | 1024Gi | Micro | AWS Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `micro` | 2 | 4Gi | Standard | Azure Machine type | vCPU | Memory | | ------------------ | ---- | ------ | | `small` | 4 | 16Gi | | `medium` | 8 | 32Gi | | `large` | 16 | 64Gi | | `xlarge` | 32 | 128Gi | | `2xlarge` | 48 | 192Gi | | `3xlarge` | 64 | 256Gi | | `4xlarge` | 96 | 384Gi | | `5xlarge` | 128 | 512Gi | High-CPU | Azure Machine type | vCPU | Memory | | ------------------ | ---- | ------ | | `small-highcpu` | 4 | 8Gi | | `medium-highcpu` | 8 | 16Gi | | `large-highcpu` | 16 | 32Gi | | `xlarge-highcpu` | 32 | 64Gi | High-memory | Azure Machine type | vCPU | Memory | | ------------------ | ---- | ------ | | `small-highmem` | 4 | 32Gi | | `medium-highmem` | 8 | 64Gi | | `large-highmem` | 16 | 128Gi | | `xlarge-highmem` | 32 | 256Gi | | `2xlarge-highmem` | 48 | 384Gi | | `3xlarge-highmem` | 64 | 512Gi | | `4xlarge-highmem` | 96 | 768Gi | | `5xlarge-highmem` | 128 | 1024Gi | Micro | Azure Machine type | vCPU | Memory | | ------------------ | ---- | ------ | | `micro` | 2 | 4Gi | Standard | GCP Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `small` | 4 | 15Gi | | `medium` | 8 | 31Gi | | `large` | 16 | 62Gi | | `xlarge` | 32 | 124Gi | | `2xlarge` | 48 | 186Gi | | `3xlarge` | 64 | 248Gi | | `4xlarge` | 96 | 372Gi | | `5xlarge` | 192 | 744Gi | High-CPU | GCP Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `small-highcpu` | 4 | 7Gi | | `medium-highcpu` | 8 | 15Gi | | `large-highcpu` | 16 | 30Gi | | `xlarge-highcpu` | 32 | 60Gi | High-memory | GCP Machine type | vCPU | Memory | | ----------------- | ---- | ------ | | `small-highmem` | 4 | 31Gi | | `medium-highmem` | 8 | 63Gi | | `large-highmem` | 16 | 126Gi | | `xlarge-highmem` | 32 | 252Gi | | `2xlarge-highmem` | 48 | 378Gi | | `3xlarge-highmem` | 64 | 504Gi | | `4xlarge-highmem` | 96 | 756Gi | | `5xlarge-highmem` | 192 | 1512Gi | Micro | GCP Machine type | vCPU | Memory | | ---------------- | ---- | ------ | | `micro` | 2 | 4Gi | The "micro" machine type is not suitable for production workloads and is only recommended for scenarios where a minimal deployment is required for testing. # How `chalk apply` Works source: https://docs.chalk.ai/docs/chalk-apply ## How `chalk apply` packages a Chalk project, builds container images, and rolls them out into the Data Plane. chalk apply is the command that turns a local Chalk project (Python source plus the schemas of features and resolvers it defines) into a running deployment in the Data Plane. This page describes the end-to-end pipeline that runs when you invoke it: what the CLI sends, what the Metadata Plane builds, where the artifacts live, and how the rollout reaches your Kubernetes cluster. ### High-level Flow chalk apply sequence diagram At a high level, chalk apply performs five stages: - Schema + source upload. The client sends the schemas of features and resolvers, along with the project's Python source code, to the Metadata Plane. - Image build. The Metadata Plane uses Argo Workflows to package Chalk's platform software together with the customer's source code into OCI ("Docker") container images and accompanying infrastructure configuration. - Source archival. The uploaded source archive is stored in object storage (S3, GCS, or Azure Blob Storage) for audit and reproducibility. - Image storage. The built Docker images are pushed to a container registry in the customer's cloud account: Amazon ECR, Google Artifact Registry (GAR), or Azure Container Registry, depending on the cloud provider. - Rollout. Once the build is ready, the Metadata Plane executes a rollout into the Data Plane Kubernetes cluster (EKS, GKE, or AKS) and manages traffic weighting between the previous deployment and the new one. ### Stage 1: Schema and source upload When you run chalk apply, the CLI first introspects the local project: - Graph extraction. chalkpy runs locally to enumerate every feature class and resolver and produce a serialized representation of the feature graph. - Validation. The graph is sent to the Metadata Plane's GraphQL API for remote validation (type checks, dependency cycles, namespace conflicts). If --force is not passed, the CLI also fetches a diff against the currently deployed graph and prompts the user to confirm. - Deployment record. The CLI calls BuilderService.CreateDeployment with git metadata, Python requirements, and project settings. The Metadata Plane responds with a deployment_id. - Source archive. The CLI tars and uploads the project directory to the Metadata Plane via BuilderService.UploadSource, keyed by deployment_id. Only schemas, source code, and project metadata transit to the Metadata Plane at this stage. No feature values are involved. ### Stage 2: Image build via Argo The Metadata Plane delegates the actual image build to Argo Workflows running in the Metadata Plane cluster. Each deployment kicks off a workflow that: - Pulls Chalk's platform base images (engine, query server, stream worker, batch worker, background persistence) from Chalk's release registry. - Layers the customer's Python source and dependencies on top of the base images. - Generates the infrastructure configuration (Kubernetes manifests, Helm values, KEDA scalers, Envoy routes) that the Data Plane needs to run the new deployment. - Runs lint, type-check, and bytecode-compile steps to surface errors before rollout. Build progress is observable in real time via BuilderService.GetDeploymentSteps, which the CLI polls when --await is set (the default). ### Stage 3: Source archival The uploaded source archive is persisted to object storage in the customer's cloud account, in the source bucket provisioned during installation: - AWS: Amazon S3 - GCP: Google Cloud Storage - Azure: Azure Blob Storage These archives are retained for audit and reproducibility. Any past deployment can be re-built or inspected from its source archive. ### Stage 4: Image storage Built images are pushed to a container registry in the customer's cloud account, so that image bytes never leave the customer's cloud boundary: - AWS: Amazon Elastic Container Registry (ECR) - GCP: Google Artifact Registry (GAR) - Azure: Azure Container Registry (ACR) The Data Plane cluster's nodes pull images directly from this registry using IRSA (AWS), Workload Identity (GCP), or Managed Identity (Azure). The Metadata Plane never proxies image pulls. ### Stage 5: Rollout into the Data Plane Once the build is ready, the Metadata Plane uses its EKS/GKE/AKS API access to roll the new deployment into the Data Plane cluster: - Manifests applied. The Metadata Plane applies the generated Kubernetes manifests (Deployments, Services, HTTPRoutes, KEDA ScaledObjects) into the environment's namespace in the Data Plane cluster. - Pod scheduling. Karpenter (or the equivalent autoscaler) provisions nodes and schedules pods that pull the newly-built images from the customer's registry. - Readiness gating. The Metadata Plane polls DeployService.GetDeployment until the new pods report ready. Terminal statuses are SUCCESS, FAILURE, INTERNAL_ERROR, BOOT_ERRORS, TIMEOUT, and CANCELLED. - Traffic weighting. Once the new pods are ready, the Metadata Plane shifts traffic weights at the Envoy Gateway. By default this is an immediate cutover; with Blue-Green Deployment, traffic is shifted gradually between the old (blue) and new (green) tag groups, and old pods are kept running until the new deployment is verified. For details on the Metadata Plane → Data Plane connectivity that enables this rollout, see Metadata Plane & Data Plane Communication. ### Branch deploys chalk apply --branch follows a slimmed-down version of the same pipeline, optimized for fast iteration: - The CLI calls BuilderService.StartBranch to ensure a branch server is running, then uploads the source archive directly to DeployService.DeployBranch. - The branch server hot-reloads the new code in place rather than building a fresh OCI image, so feedback is on the order of seconds rather than minutes. - All branches in an environment run on one branch server, in the same Data Plane cluster and namespace as the environment's other services. Branch deployments do not affect production traffic. See Branches for more. ### What never leaves the customer's cloud Even though chalk apply is orchestrated by the Metadata Plane, the heavy artifacts stay inside the customer's cloud account: - Source archives live in the customer's object storage bucket. - Container images live in the customer's container registry. - Pods run in the customer's Kubernetes cluster, on the customer's nodes. The Metadata Plane retains only deployment metadata (deployment ID, git SHA, status, build logs) and the feature-graph schema needed to plan and validate queries. # Building Images Without Deploying source: https://docs.chalk.ai/docs/chalk-deploy-build ## Build a Chalk engine image and push it to a registry you own, without creating a deployment — and mirror it into a separate cloud partition such as AWS GovCloud. chalk deploy build builds the engine image for your project and pushes it to container registries that you name. Unlike chalk apply, it does not deploy anything. Concretely, it does not: - create a deployment record, - promote the image to your environment, - change what your environment is currently serving, - move the floating :latest tag on your environment's engine repository. The result is an image in a registry you control, and nothing else. Your active deployment keeps serving exactly what it was serving before. ### When to use it Use chalk deploy build when the image is an artifact you want to handle yourself rather than a change you want live: - Delivery into a separate cloud partition or an air-gapped network. Chalk builds in the partition your environment lives in. Getting the image somewhere Chalk's build identity cannot reach — AWS GovCloud is the common case — means Chalk hands you the image and you move it the rest of the way. See Mirroring into AWS GovCloud below. - Promotion through your own pipeline. Build once, scan or sign the image, run it through your own approval gates, then deploy the exact bits you approved. - Vulnerability scanning before rollout. Get the image into a registry your scanner watches without exposing it to traffic first. If you just want to deploy, use chalk apply. ### Usage ``` chalk deploy build --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 ``` Repeat --destination to push the same image to several registries in one build: ``` chalk deploy build \ --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25 \ --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidate ``` The build pushes to every destination in a single pass, so a second destination costs almost nothing. On success the command prints the build's identifier and the references it pushed to: ``` Build ID: chalk-build-argo-8f3c21d4 → 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:2026-08-25 → 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:release-candidate ``` The command runs from your project directory, exactly like chalk apply — it archives the same source tree, honoring .chalkignore. ### Destination rules Every destination must be a fully-qualified reference with an explicit tag: ``` /: ``` | Rule | Accepted | Rejected | | ------------------ | -------------------------------------------------------------- | ---------------------------------------- | | Registry host | `123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1` | `myorg/engine:v1` (Docker Hub shorthand) | | Explicit tag | `.../engine:v1` | `.../engine` | | Tag, not digest | `.../engine:v1` | `.../engine@sha256:…` | | Supported registry | Amazon ECR, Google Artifact Registry, GCR | Azure Container Registry, other hosts | An untagged repository is rejected rather than defaulted to :latest, because silently writing latest in a registry you own can overwrite an image your other systems already pull. ACR destinations are rejected. Chalk's builder authenticates to exactly one ACR today, so there is nowhere to register a second registry. Rather than build an image that silently never arrives, the build fails up front. ### Authentication Chalk pushes with the build's own cloud identity. It never accepts, stores, or transmits registry credentials. That means you grant Chalk's build identity push access to the destination before running the command. What that identity is depends on your cloud: - Google Cloud: the build's service account. One access token authenticates every Artifact Registry or GCR host that account can push to, so any such destination works once IAM allows it. - AWS: the environment's ECR authorization token, issued for the environment's management role. Grant access by adding that role to the destination repository's policy. This model implies two limits on AWS: Cross-account works. A destination in a different AWS account is fine, as long as that repository's policy grants your environment's management role. Cross-region does not. An ECR authorization token is only valid in the region that issued it, so the destination must be in the same region as your environment's own registry. A destination elsewhere is rejected before the build starts: ``` image destination "123456789012.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1" is in region us-gov-west-1 but this environment's registry is in us-east-1; an ECR authorization token is only valid in the region that issued it ``` Destinations must also be on the same cloud as your environment — a GCP environment cannot push to ECR, and vice versa — because the build pod holds exactly one cloud identity. ### Knowing when the build finished Because there is no deployment record, this build does not appear in chalk deployment list, the deployments page, or chalk deployment build-steps. Those surfaces are all keyed on a deployment. Today, the way to observe completion is to watch for the image to appear at its destination: ``` crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 ``` or equivalently aws ecr describe-images. The GovCloud example below builds this wait into the mirroring workflow. Automating on top of this? Poll the destination registry rather than any Chalk deployment API. The build ID that chalk deploy build prints identifies the build workflow, not a deployment, so deployment-scoped endpoints will not find it. ### What the image contains The image is the same engine image chalk apply produces: Chalk's platform software and the resolved Python virtualenv for your requirements. Your project source is not baked in. The engine downloads it at boot, from the source bundle the deploy uploaded to your environment's bucket. That is why one image serves every deployment whose requirements are unchanged — and it is what makes chalk apply --from-image possible. Runtime configuration is not baked in. When Chalk deploys an image it supplies environment variables pointing at your online store, offline store, and graph artifacts. An image you run outside Chalk's control plane needs equivalent configuration supplied by whatever runs it. ### Mirroring into AWS GovCloud AWS GovCloud is a separate AWS partition (aws-us-gov). This is a stronger boundary than a region: - Credentials issued in the commercial partition have no identity in GovCloud at all. - sts:AssumeRole does not cross partitions, so you cannot assume a GovCloud role from a commercial account. So when your Chalk environment runs in the commercial partition, Chalk cannot push into GovCloud directly, and --destination will reject a GovCloud reference on the region rule above. The split is: - Chalk builds and pushes to a commercial-partition ECR repository that you own. - You copy that image into GovCloud, using credentials issued in the GovCloud partition. Step 2 is your code, running in your account, holding your GovCloud credentials. Chalk is not involved and never sees them. What follows is a working example you can adapt. ### GovCloud credentials Because role assumption cannot cross partitions, the copy needs a credential issued in GovCloud. In practice that is an IAM user in your GovCloud account with permission to push to the destination repository, stored as a Kubernetes secret: ``` kubectl create secret generic govcloud-ecr \ --from-literal=access_key_id="$GOVCLOUD_ACCESS_KEY_ID" \ --from-literal=secret_access_key="$GOVCLOUD_SECRET_ACCESS_KEY" ``` Partition boundaries rule out the short-lived, federated credentials you would normally prefer. Scope the IAM user to ecr:PutImage and the related upload actions on the single destination repository, rotate it on a schedule, and keep it out of the commercial account's blast radius. The commercial side needs no secret — give the workflow's service account an IRSA role with read access to the source repository. ### The mirroring workflow This WorkflowTemplate logs into both registries, copies the image, and verifies that the digest survived the copy. All three containers share one pod so they can share a docker config through an emptyDir. ``` apiVersion: argoproj.io/v1alpha1 kind: WorkflowTemplate metadata: name: mirror-chalk-engine-to-govcloud spec: entrypoint: mirror # IRSA role with ECR read on the commercial source repository. serviceAccountName: chalk-image-mirror arguments: parameters: - name: source-image value: 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 - name: destination-image value: 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1 - name: source-region value: us-east-1 - name: destination-region value: us-gov-west-1 templates: - name: mirror volumes: - name: docker-config emptyDir: {} containerSet: volumeMounts: - name: docker-config mountPath: /docker containers: - name: login image: amazon/aws-cli:2 env: - name: GOVCLOUD_ACCESS_KEY_ID valueFrom: secretKeyRef: name: govcloud-ecr key: access_key_id - name: GOVCLOUD_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: govcloud-ecr key: secret_access_key command: [sh, -c] args: - | set -eu SRC='{{workflow.parameters.source-image}}' DST='{{workflow.parameters.destination-image}}' SRC_HOST="${SRC%%/*}" DST_HOST="${DST%%/*}" # Commercial partition: the pod's own IRSA role. SRC_PW=$(aws ecr get-login-password \ --region '{{workflow.parameters.source-region}}') # GovCloud is a different partition, so this must use the static # credential. `env -u` drops the IRSA variables so the SDK cannot # prefer web identity over the keys we are handing it. DST_PW=$(env -u AWS_ROLE_ARN -u AWS_WEB_IDENTITY_TOKEN_FILE \ AWS_ACCESS_KEY_ID="$GOVCLOUD_ACCESS_KEY_ID" \ AWS_SECRET_ACCESS_KEY="$GOVCLOUD_SECRET_ACCESS_KEY" \ aws ecr get-login-password \ --region '{{workflow.parameters.destination-region}}') SRC_AUTH=$(printf 'AWS:%s' "$SRC_PW" | base64 -w0) DST_AUTH=$(printf 'AWS:%s' "$DST_PW" | base64 -w0) umask 077 cat > /docker/config.json <&2 exit 1 fi echo "digests match" ``` Run it with: ``` argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \ -p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \ -p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1 ``` ### Why crane copy crane copy transfers the manifest and its layers as-is, so the image keeps its digest across partitions. That is what makes the verify step meaningful: a matching digest is evidence that the bits authorized in GovCloud are byte-identical to the ones Chalk built, which is usually what an authorization boundary review wants to see. A docker pull and docker push through an intermediate host can rewrite the manifest and change the digest. skopeo copy --all is an equivalent choice if it fits your tooling better. ### End to end ``` # 1. Chalk builds and pushes to your commercial repository. chalk deploy build \ --destination 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 # 2. Wait for the image to land (there is no deployment to poll). until crane digest 123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \ >/dev/null 2>&1; do sleep 30; done # 3. Mirror it into GovCloud. argo submit --from workflowtemplate/mirror-chalk-engine-to-govcloud \ -p source-image=123456789012.dkr.ecr.us-east-1.amazonaws.com/chalk/engine:v1 \ -p destination-image=210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine:v1 ``` Tag each build with something immutable — a git SHA or a release version — rather than reusing one tag. The digest check in step 3 only proves the copy was faithful; an immutable tag is what lets you say which build was authorized. ### Deploying a pre-built image Once an image exists in a registry your environment's cluster can pull from, deploy it with chalk apply --from-image: ``` chalk apply --from-image \ 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256: ``` This creates a normal deployment — a deployment record, a graph, a rollout — but builds nothing. The reference you give becomes the deployment's engine image verbatim, and the Python source uploaded by that same chalk apply is what the engine downloads and serves. Pin the image by its digest instead of a tag. A digest is the same identifier the mirroring step verified, so pinning one is how you say that the deployment runs exactly the image that was authorized. Nothing checks that the image matches the source you are deploying. The image carries the virtualenv for the requirements it was built from; if your source imports something that virtualenv does not have, the failure appears when the engine boots, not when you run chalk apply. Deploy from source whose requirements.txt produced the image. Because no image is built, anything that describes how to build one is rejected rather than ignored: a base image override or a build profile cannot be combined with --from-image. ### Carrying artifacts forward An image build also produces artifacts beside the image — the source bundle it was built from, the virtualenv tarball, and the graph export an indexing job wrote. Three optional flags serve those instead of producing them again: | Flag | What it replaces | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `--with-source-bundle` | The source bundle the engine downloads at boot, in place of the archive `chalk apply` just uploaded | | `--with-export` | The deployment's `export.pb`, in place of the one an indexing job would have written | | `--with-venv` | Seeds your environment's virtualenv cache, so a later indexing or branch build reuses it instead of resolving the requirements again | Each takes an object that is already in your environment's source bundle bucket — a bare object key, or a full s3://, gs:// or az:// URI naming that same bucket. Chalk copies within that one bucket; it does not read from anywhere else, and an object in another bucket is rejected. Copy the artifacts across the partition boundary yourself, the same way you copied the image. --with-venv must keep the filename the cache uses, shallow_copy_.tar.zst, because that name is the cache key. --with-export is optional in a different sense: without it the engine builds its graph from the source bundle at boot, which is slower but correct. ``` chalk apply \ --from-image 210987654321.dkr.ecr.us-gov-west-1.amazonaws.com/chalk/engine@sha256: \ --with-source-bundle s3://my-chalk-bundles/authorized/build-v1.tar.gz \ --with-export s3://my-chalk-bundles/authorized/build-v1-export.pb \ --with-venv s3://my-chalk-bundles/authorized/shallow_copy_.tar.zst ``` ### Related - How chalk apply Works — the deploying counterpart, and what the image build stage does in detail. - Artifact Repository — private Python indexes the build resolves packages from. # GitHub App Setup source: https://docs.chalk.ai/docs/github-app-setup ## Configure a GitHub App for a self-hosted Chalk metadata plane. Chalk's GitHub integration lets dashboard users install a GitHub App into their GitHub organization, connect Chalk projects to GitHub repositories, and import repository contents into volumes. The integration is powered by a GitHub App whose credentials are held by the metadata plane (the Chalk API server). In Chalk Cloud, Chalk operates this app for you. When you self-host the metadata plane, you must register your own GitHub App and configure your API server with its credentials. This page walks metadata-plane administrators through that setup. ### How the integration works The API server authenticates to GitHub as a GitHub App: - It signs a short-lived JWT with the app's private key. - It exchanges that JWT for installation access tokens, scoped to the GitHub organizations (or user accounts) that installed the app. - Installation tokens are used to list repositories and branches, validate project↔repository connections, and download repository archives for volume imports. No user-level OAuth is involved, and GitHub credentials are never sent to the browser: the dashboard talks only to the API server, which proxies all GitHub API access. ### Step 1: Register a GitHub App Create a new GitHub App at https://github.com/organizations//settings/apps/new (or Settings → Developer settings → GitHub Apps → New GitHub App for a personal account). Configure it as follows: | Setting | Value | | ------------------------------------ | ------------------------------------------------------------------------------------------------ | | **GitHub App name** | Anything you like, e.g. `Acme Chalk`. The URL slug GitHub derives from it is used below. | | **Homepage URL** | Your dashboard URL, e.g. `https://chalk.acme.com` | | **Setup URL** | `https:///github-redirect` | | **Redirect on update** | Enabled | | **Webhook** | Disabled (the integration does not consume webhooks) | | **Where can this App be installed?** | "Only on this account" is sufficient if all of your GitHub repositories live in one organization | The Setup URL is required: after a user installs the app, GitHub redirects their browser there with an installation_id, and that page records the installation with the API server. If it is missing or points at the wrong host, installs will succeed on GitHub but never appear in Chalk. ### Repository permissions Grant the following repository permissions. All are read-only: | Permission | Access | Used for | | ----------------- | --------- | ----------------------------------------------------------------------- | | **Metadata** | Read-only | Listing the repositories an installation can access | | **Contents** | Read-only | Listing branches and downloading repository archives for volume imports | | **Pull requests** | Read-only | Listing pull requests | No account permissions or event subscriptions are needed. ### Generate a private key After creating the app, scroll to Private keys and click Generate a private key. GitHub downloads a .pem file. Keep the key in the format GitHub provides: an RSA (PKCS#1) PEM beginning with -----BEGIN RSA PRIVATE KEY-----. Do not convert it to PKCS#8 (-----BEGIN PRIVATE KEY-----); the API server only accepts the PKCS#1 format. Note down from the app's settings page: - App ID (a number, shown under "About") - Client ID (e.g. Iv23...) - The app slug — the final path segment of the app's public page, https://github.com/apps/ ### Step 2: Configure the API server Provide the app's identity to the API server through environment variables: | Environment variable | Required | Description | | -------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------- | | `CHALK_GITHUB_APP_ID` | Yes | The numeric App ID | | `CHALK_GITHUB_APP_SLUG` | Yes | The app's URL slug; the dashboard's "Install GitHub App" button links to `https://github.com/apps//installations/new` | | `CHALK_GITHUB_APP_PRIVATE_KEY` | Yes | The full PEM contents of the private key, including the `BEGIN`/`END` lines | | `CHALK_GITHUB_APP_CLIENT_ID` | No | The app's client ID | | `CHALK_GITHUB_APP_CLIENT_SECRET` | No | Reserved for future use; not currently required | All three required variables must be set together: if only some are present, the API server logs an error at startup and disables the integration (teams with their own explicit configuration are unaffected). If the API server runs on Kubernetes, the typical pattern is to add the variables to the secret that already backs the API server's environment: ``` kubectl -n patch secret \ --type merge \ -p "$(jq -n --rawfile key chalk-app.private-key.pem '{stringData: { CHALK_GITHUB_APP_ID: "123456", CHALK_GITHUB_APP_SLUG: "acme-chalk", CHALK_GITHUB_APP_CLIENT_ID: "Iv23xxxxxxxxxxxxxxxx", CHALK_GITHUB_APP_PRIVATE_KEY: $key }}')" ``` Then restart the API server pods so they pick up the new environment. On a successful boot, the API server logs: ``` githubapp: default GitHub App enabled from environment ``` This environment-provided app acts as the default for every team on the metadata plane. A team can override it with its own GitHub App through the GitHubAppService/UpsertGitHubAppConfig RPC; an explicit per-team configuration always takes precedence over the environment default. ### Step 3: Verify - In the dashboard, open Settings → Team → GitHub. The app card should show your app's slug and App ID instead of a "not configured" notice. - Click Install GitHub App. You should land on https://github.com/apps//installations/new; pick an organization and confirm. - GitHub redirects back to /github-redirect on your dashboard, which records the installation and links back to the settings page. The new installation appears in the Installations list. - Connect a project to a repository under Connected projects, or use Import from GitHub on a volume page to clone a repository into a volume. ### Troubleshooting | Symptom | Likely cause | | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `failed_precondition: GitHub App not configured for this team` | The `CHALK_GITHUB_APP_*` variables are not visible to the API server process (not set, set on the wrong deployment, or the pods were not restarted after the change), and the team has no explicit configuration. | | `githubapp: ignoring invalid default GitHub App env configuration` in API server logs | The variables are partially set, or `CHALK_GITHUB_APP_ID` is not a number. Set all of `CHALK_GITHUB_APP_ID`, `CHALK_GITHUB_APP_SLUG`, and `CHALK_GITHUB_APP_PRIVATE_KEY`. | | Install completes on GitHub but never appears in the dashboard | The app's **Setup URL** does not point at `https:///github-redirect`, or the installing user's browser could not reach the dashboard afterwards. Use **Sync from GitHub** on the settings page to reconcile. | | Errors mentioning the private key when listing repositories or completing an installation | The key was converted to PKCS#8. Re-download the key from GitHub (it must begin with `-----BEGIN RSA PRIVATE KEY-----`), or rotate it and configure the new one. | | Repository missing from the picker | The installation only grants access to selected repositories. Update the repository selection for the installation on GitHub, then **Sync from GitHub**. | ### Key rotation To rotate the app's private key: - Generate a new private key on the GitHub App settings page. - Update CHALK_GITHUB_APP_PRIVATE_KEY wherever the API server's environment is managed. - Restart the API server pods. - Delete the old key on GitHub once the restart is complete. Installation tokens are short-lived and re-minted on demand, so no other state needs to change. # Blue-Green Deployments source: https://docs.chalk.ai/docs/blue-green-deployment ## Set up blue-green deployments for reliability. Chalk offers a variety of tools and strategies for testing your features, resolvers, and queries in the process of deploying them to production. Testing through writing unit tests and integration tests can ensure that your code is working as expected. Chalk also offers blue-green deployments for a release strategy that can further minimize downtime and risk when deploying changes. If your team is interested in using blue-green deployments in your production environment, please reach out to the Chalk team. ### Blue-Green Deployments with Chalk Chalk offers blue-green deployments as a way to minimize downtime when deploying new code by running two deployments, one "blue" and one "green", in parallel. The blue deployment is the current production deployment, while the green deployment is the new version of the code that is being released. Using deployment tags, Chalk can route a specified percentage of traffic to the blue or green deployment, allowing for a gradual rollout of the new code, as well as an easy path to rollback if any issues arise. ### Releasing using Blue-Green Deployments Once you have confirmed with the Chalk team that blue-green deployments are enabled in your Chalk environment, there are a few ways that you can start to use blue-green deployments. ### Full rollouts using Blue-Green Deployments The simplest way to start to use blue-green deployments is to use the --blue-green tag when deploying your code. When you run chalk apply --blue-green, that will automatically tag your latest deployment with either blue if the latest deployment was tagged green, or green if the latest deployment was tagged blue. Then, when you're ready to migrate traffic to your new deployment, you can run chalk traffic promote, which will migrate traffic from the deployment currently in use to the other tagged deployment. The following example demonstrates how to use these blue-green deployments. Suppose you have made changes that you now want to release. You can then run ``` $ chalk apply --deployment-tag blue # or `green` depending on the current deployment ``` This will create a new deployment tagged with blue or green. Then, you can choose to migrate traffic to the new deployment by running: ``` $ chalk traffic promote ``` If the previous deployment was tagged blue and the latest deployment was tagged green, this command migrates 100% of the traffic from the blue deployment to the green deployment. You can then monitor the green deployment traffic. If you encounter any errors with your new deployment, then you can once again run chalk traffic promote and it will migrate traffic from the green deployment back to the blue deployment. This allows you to easily rollback during releases. ### Gradual rollouts using Blue-Green Deployments If you would like to gradually roll out your new deployment with staged increments of traffic, you can use the --deployment-tag flag to specify a new deployment towards which to direct traffic. To get started with blue-green deployments, you can tag the current deployment with blue. To do so you can run the following Chalk CLI commands: ``` $ chalk apply --deployment-tag blue $ chalk traffic set --tags blue=100 ``` This will tag the current deployment with blue and direct 100% of traffic to the blue deployment. Then, when you are ready to begin rolling out the next deployment, you can deploy the new version of code with the tag green. ``` $ chalk apply --deployment-tag green $ chalk traffic set --tags blue=90,green=10 ``` This will direct 90% of traffic to the blue deployment and 10% of traffic to the green deployment. You can verify the traffic distribution by running chalk traffic get. ``` $ chalk traffic get tags: - deployment_id: cm9n46thy000bwe6y784y4j0l tag: blue weight: 90 - deployment_id: cm9n3vx7c0004we6yefnsw8kg tag: green weight: 10 ``` Then, you can gradually increase traffic to the green deployment until it is at 100%, and for the next time you release, you can tag the latest deployment with blue and migrate traffic from the green deployment to the even newer blue deployment. This workflow enables you to have fine-tuned control over the traffic distribution between your deployments. Furthermore, Chalk will automatically scale down unused deployments to minimize unnecessary resource usage. If you run the command chalk traffic set and do not specify some amount of traffic for a tagged deployment, then Chalk will recognize that the deployment is no longer in use and will scale it down. ### Managing your Blue-Green Deployments In order to create a new tagged deployment in Chalk, you can either specify a --deployment-tag when running chalk apply, or you can use the --blue-green flag to automatically target the inactive tag for the new deployment. In addition to creating new tagged deployments, you can also use the Chalk CLI command chalk traffic to manage your blue-green deployments. To view all available commands, you can run chalk traffic --help. To view the current traffic distribution between your blue and green deployments, you can run: ``` $ chalk traffic get tags: - deployment_id: cm9n46thy000bwe6y784y4j0l tag: blue weight: 50 mirror_weight: 10 - deployment_id: cm9n3vx7c0004we6yefnsw8kg tag: green weight: 50 mirror_weight: 0 ``` To mirror a percentage of traffic to a tagged deployment, you can use the chalk traffic mirror command: ``` $ chalk traffic mirror --mirror blue=15,green=5 tags: - deployment_id: cm9n46thy000bwe6y784y4j0l tag: blue mirror_weight: 15 - deployment_id: cm9n3vx7c0004we6yefnsw8kg tag: green mirror_weight: 5 ``` As shown above, you can use chalk traffic promote and chalk traffic set to configure the distribution of traffic between your blue and green deployments. Once you roll out your new deployment, by default we would still keep pods up for your inactive deployment to enable a fast rollback. However, if you would like to spin down pods for any inactive deployment, you can also use chalk traffic suspend to essentially set the traffic weight to nil, which will scale down the pods for that deployment. ``` $ chalk traffic suspend --deployment-tag blue ``` ### Should I use Blue-Green Deployments? Blue-green deployments are a powerful tool for minimizing downtime through a gradual rollout deployment. However, running two deployments in parallel also has resource implications and may not be necessary for all teams. We would recommend blue-green deployments for teams that have a high volume of traffic in their production environments where the benefit of minimizing downtime risk is worth the cost of additional resources required. If you are interested in using blue-green deployments, please reach out to the Chalk team and we can help you determine what configurations are best suited for your release strategy. ### Mirror Traffic Chalk also offers a mirror traffic feature that allows you to mirror traffic from your production deployment to a new deployment. This is useful for testing new features in a production-like environment without affecting the production traffic. To use mirror traffic, you can run the following command: ``` # Mirror 5% of traffic to the green deployment. $ chalk traffic set --tags blue=100,green=0 --mirror green=5 ``` Note: mirror traffic does not block responses for existing production traffic. It simply mirrors the traffic to the new deployment without waiting to receive a response from the new deployment. Metrics, logs, and traces from the mirrored traffic will be available in the new deployment. Queries will not impact the online store, but will impact the offline store if the new deployment is configured to write to the offline store. This permits statistical analysis of the new deployment using tooling like the query log or the offline store. ### Light-weight Cookbook You can make use of only mirroring with minimal reference to blue-green by: - Deploying your new code with the --deployment-tag flag to tag the deployment. - Mirroring traffic to the new deployment using the --weight flag to specify the percentage of traffic to mirror. - Monitoring the new deployment for any issues. Example commands: ``` # Make the new deployment -- it won't receive any traffic yet $ chalk apply --deployment-tag green # Mirror 5% of traffic to the green deployment $ chalk traffic set --tags blue=100,green=0 --mirror green=5 # Monitor the new deployment for any issues. Once you're satisfied, you can promote it to production with: $ chalk traffic promote ``` A key point is that the mirrored deployment does need to have 0% weight specified, which indicates to Chalk that the deployment should have pods running to handle the mirrored traffic, but not to handle any production traffic. Failing to specify the --weight flag will result in Chalk not running any pods for the off-color deployment. # Multi-Region Failover source: https://docs.chalk.ai/docs/multi-region-failover ## Configure multi-region failover for high availability on AWS. Chalk supports multi-region failover on AWS so that your feature serving infrastructure remains available even if an entire region goes down. This guide covers how to set up active-passive failover using multiple EKS clusters, resource groups, and Route 53 failover routing. ### Architecture overview Multi-region failover works by deploying Chalk into EKS clusters in two or more AWS regions. Each region runs an independent set of Chalk services behind its own ALB. Route 53 failover routing directs traffic to the healthy region based on health checks against each ALB. The key components are: - Multiple EKS clusters: one per region, each running a full Chalk data plane. - Resource groups: each region's Chalk deployment is assigned to a distinct resource group with a unique hostname. - Route 53 failover routing: a single DNS name resolves to the healthy region's ALB, with automatic failover when health checks fail. ### Setting up multi-region failover ### 1. Create EKS clusters in each region Work with the Chalk team to provision EKS clusters in your target regions (for example, us-east-1 and us-west-2). Each cluster runs a complete Chalk data plane with its own query servers, online store, and background persistence workers. ### 2. Configure resource groups Each regional deployment is assigned to its own resource group. Resource groups provide workload isolation and give each region a unique hostname for its query servers. For example, you might have: | Region | Resource group | Hostname | | ----------- | -------------- | ----------------------------- | | `us-east-1` | `prod-east` | `prod-east.query.example.com` | | `us-west-2` | `prod-west` | `prod-west.query.example.com` | Each resource group's query server exposes a /healthz endpoint that the ALB uses for health checking. When the query server is healthy, /healthz returns a 200 response. ### 3. Configure Route 53 failover routing Create a Route 53 hosted zone with failover routing records that point to the ALB in each region: - Primary record: points to the ALB in your primary region (e.g. us-east-1), with a health check against the /healthz endpoint. - Secondary record: points to the ALB in your secondary region (e.g. us-west-2), with its own health check. Both records share the same DNS name (e.g. chalk-failover.example.com). Route 53 resolves this name to the primary ALB when it is healthy, and automatically switches to the secondary ALB when the primary's health check fails. Route 53 health checks poll the /healthz endpoint on each ALB. The default health check interval is 30 seconds. You can configure the failure threshold (number of consecutive failures before failover) and request interval to tune failover speed vs. sensitivity. ### 4. Point clients at the failover hostname Configure your application to send queries through the failover DNS name using the query_server parameter on ChalkClient: ``` from chalk.client import ChalkClient client = ChalkClient( query_server="https://chalk-failover.example.com", ) result = client.query( input={"user.id": 123}, output=["user.name", "user.risk_score"], ) ``` The client does not need any awareness of which region is active. DNS resolution handles routing transparently. ### How failover works Under normal operation, Route 53 resolves the failover hostname to the primary region's ALB. All client traffic flows to the primary EKS cluster. When the primary region becomes unhealthy: - The ALB health check against /healthz begins failing. - Route 53 detects consecutive health check failures (typically within 1-2 minutes). - Route 53 updates DNS to resolve the failover hostname to the secondary region's ALB. - Client traffic is routed to the secondary EKS cluster. When the primary region recovers: - The /healthz endpoint starts returning 200 again. - Route 53 detects the recovery and switches DNS back to the primary region. Failover is DNS-based, so clients must respect DNS TTLs. Set a low TTL (e.g. 60 seconds) on the failover record to minimize the window during which clients may still resolve to the unhealthy region. ### Data replication Each region's Chalk deployment operates independently, but the underlying data stores must be replicated across regions so the passive region can serve traffic with current feature values. ### Online store -- DynamoDB Global Tables Chalk uses DynamoDB Global Tables to replicate online feature values across regions. Global Tables provide automatic, asynchronous replication so that the passive region's online store stays up to date with writes from the active region. No application-level changes are required -- both regions read and write to the same logical table. ### Online store -- ElastiCache Valkey Global Datastore If you use ElastiCache Valkey as your online store instead of DynamoDB, cross-region replication is handled by ElastiCache Global Datastore. Global Datastore creates a fully managed, active-passive replication topology across regions: - Primary cluster: the ElastiCache Valkey cluster in your primary region accepts all writes. Chalk's persistence workers in the active region write computed feature values here. - Secondary cluster: a read-only replica cluster in the secondary region receives writes asynchronously from the primary. Replication is continuous and typically completes within single-digit milliseconds under normal network conditions. When a region failover occurs, the secondary cluster must be promoted to become the new primary before it can accept writes. ElastiCache supports managed promotion through the AWS console or API, and the process typically completes within minutes. ### How async replication works ElastiCache Global Datastore uses the Valkey replication stream to replicate data cross-region: - Writes land on the primary cluster's primary node as normal Valkey commands. - The replication stream is forwarded over an AWS-managed, encrypted cross-region link to the secondary cluster. - The secondary cluster applies the replication stream to its own dataset, keeping it eventually consistent with the primary. Because replication is asynchronous, there is a small window (typically under 1 ms for in-region replication and low single-digit milliseconds cross-region) where the secondary may be behind the primary. In a failover scenario, any writes that had not yet been replicated to the secondary at the moment of failure may be lost. Monitor cross-region replication health with the ReplicationLag CloudWatch metric on your Global Datastore. Under normal conditions this stays below 1 ms for in-region replicas, and low single-digit milliseconds for cross-region replicas. ### Failover behavior During a region failover: - Route 53 detects the primary region's health check failure and switches DNS to the secondary region. - You (or an automated runbook) promote the secondary ElastiCache cluster to primary using the Global Datastore failover API. - Chalk's query servers in the secondary region begin reading from the now-promoted cluster. Persistence workers in the secondary region begin writing new feature values to it. Unlike DynamoDB Global Tables, ElastiCache Global Datastore requires an explicit promotion step during failover. Until the secondary cluster is promoted, it remains read-only. Plan for this in your failover runbook or automation. ### Configuration Work with the Chalk team to provision ElastiCache Global Datastore across your target regions. Each region's Chalk deployment should be configured with a connection URI pointing to the local ElastiCache cluster endpoint, so reads are always served from the nearest region. See the online store setup guide for connection URI format details. ### Persistence coordination -- MSK Replicator Chalk uses Amazon MSK (Managed Streaming for Apache Kafka) to coordinate background persistence operations such as writing computed features to the online and offline stores. In a multi-region deployment, MSK must be configured for cross-region replication using MSK Replicator. MSK Replicator continuously replicates topics from the active region's MSK cluster to the passive region's MSK cluster. This ensures that persistence operations initiated by the active region -- such as online store writes and offline store ingestion -- are also applied in the passive region. When failover occurs, the passive region's persistence workers pick up from the replicated topic offsets. ### Metadata plane -- RDS Aurora The Chalk metadata plane uses Amazon Aurora as its sole state store. Because Aurora is the only stateful component, multi-region metadata plane availability maps directly to Aurora's cross-region capabilities: - Active-passive: use an Aurora Global Database with a read replica in the secondary region. Aurora handles asynchronous replication and supports managed failover to promote the secondary cluster when the primary region is unavailable. - Active-active: use Aurora Global Database write forwarding or an Aurora multi-master configuration so that both regions can accept writes. The choice between active-passive and active-active for the metadata plane should match the routing strategy you choose for the data plane (see Active-active vs. active-passive below). ### Offline store Offline stores are typically regional. Training data queries and dataset generation should target a specific region rather than the failover hostname. ### Deployments chalk apply should target both regions to keep feature definitions and resolver code in sync. You can automate this in CI/CD by deploying to each resource group. ### Recovery point objective (RPO) RPO -- the maximum acceptable amount of data loss measured in time -- depends on the failure scenario: | Failure scenario | RPO | Notes | | ------------------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Single AZ failure | **0** | EKS, DynamoDB, and ElastiCache span multiple AZs within a region. No data is lost. | | Full region failure (recoverable) | **Replication lag** | Determined by your online store's replication lag (DynamoDB Global Tables or ElastiCache Global Datastore) and MSK Replicator lag, typically seconds. Once the primary region recovers, any in-flight writes that had not yet replicated are recovered from the primary. | | Full region failure (permanent loss) | **Replication lag** | If the primary region is permanently lost, any writes that had not yet replicated to the passive region are lost. RPO equals the replication lag at the time of failure. | You can monitor replication lag via CloudWatch: ReplicationLatency for DynamoDB Global Tables, ReplicationLag for ElastiCache Global Datastore, and ReplicationLatency for MSK Replicator. Under normal conditions all are single-digit seconds or less. ### Active-active vs. active-passive The architecture described above is active-passive: only one region serves live traffic at a time. Chalk also supports active-active configurations using Route 53 weighted or latency-based routing, where both regions serve traffic simultaneously. Contact the Chalk team to discuss which approach is best for your availability and latency requirements. # Artifact Repository source: https://docs.chalk.ai/docs/artifact-repositories ## Configure private Python package indexes (AWS CodeArtifact, Google Artifact Registry, preauthenticated) for Chalk deployments. Chalk can resolve Python packages from one or more private indexes alongside PyPI. You configure them in Infrastructure > Artifact repositories. Each index is rendered into the builder's uv.toml as an [[index]] block. PyPI is always emitted first; the global prefer-private option controls whether PyPI is marked as the fallback (default = true) so private indexes are searched first. Three index kinds are supported: - AWS CodeArtifact: An AWS CodeArtifact repository. Chalk authenticates using a Chalk IAM principal that you grant read access to. - Google Artifact Registry (GAR): A GAR Python repository. Chalk authenticates with the management service account configured for your environment. - Preauthenticated: Any pip-compatible index URL whose full authenticated URL is stored in your team's secret store. Chalk reads the URL from the secret at build time and uses it as-is. ### AWS CodeArtifact Chalk supports downloading Python packages from private AWS CodeArtifact repositories. To use this feature, give a Chalk IAM principal access to your repository so that Chalk can authenticate with the CodeArtifact pip registry. ### Grant access to Chalk Chalk authenticates to your CodeArtifact repository using an IAM principal. You can view the IAM principal in your team's settings page. The principal looks like this: ``` arn:aws:iam::***********:role/chalk-********* ``` After you have the IAM principal ARN, grant the following permissions to the principal: ``` codeartifact:DescribePackageVersion codeartifact:DescribeRepository codeartifact:GetPackageVersionReadme codeartifact:GetRepositoryEndpoint codeartifact:ListPackages codeartifact:ListPackageVersions codeartifact:ListPackageVersionAssets codeartifact:ListPackageVersionDependencies codeartifact:ReadFromRepository ``` These permissions are derived from AWS's instructions. They grant Chalk read access to the repository, but not write access. Here is an example IAM policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "CodeArtifactPermissions", "Effect": "Allow", "Action": [ "codeartifact:DescribePackageVersion", "codeartifact:DescribeRepository", "codeartifact:GetPackageVersionReadme", "codeartifact:GetRepositoryEndpoint", "codeartifact:ListPackages", "codeartifact:ListPackageVersions", "codeartifact:ListPackageVersionAssets", "codeartifact:ListPackageVersionDependencies", "codeartifact:ReadFromRepository" ], "Principal": { "AWS": "arn:aws:iam::***********:role/chalk-*********" }, "Resource": "*" } ] } ``` After you create this policy, attach it to your repository with the following example command: ``` aws codeartifact put-repository-permissions-policy --domain my_domain --domain-owner 111122223333 \ --repository my_repo --policy-document file:///PATH/TO/policy.json ``` Or go to Infrastructure > Artifact repositories and fill in the necessary fields: ### Google Artifact Registry Chalk supports downloading Python packages from private Google Artifact Registry (GAR) Python repositories. Chalk authenticates using the management service account configured for your GCP environment, so no per-repo credentials need to be supplied. ### Grant access to Chalk Grant the management service account read access to the Artifact Registry repository. The roles/artifactregistry.reader role is sufficient. For example, run: ``` gcloud artifacts repositories add-iam-policy-binding my_repo \ --location=us-central1 \ --member="serviceAccount:chalk-management@my-project.iam.gserviceaccount.com" \ --role="roles/artifactregistry.reader" ``` ### Configure the index Go to Infrastructure > Artifact repositories and add a Google Artifact Registry index, filling in the project ID, region, and repository name. Chalk uses these to construct the index URL (https://-python.pkg.dev///simple/) and mints a fresh OAuth access token at build time. ### Preauthenticated Use a Preauthenticated index when you have a fully formed authenticated index URL that you can hand to pip/uv as-is, such as a JFrog Artifactory or self-hosted devpi URL containing a username and access token. ### Store the index URL as a secret Store the complete URL (including any auth) as a secret in your team's secret store. The URL should look like one of: ``` https://:@/api/pypi//simple/ https:///api/pypi//simple/ # if no auth is needed ``` ### Configure the index Go to Infrastructure > Artifact repositories, add a Preauthenticated index, and set Index URL Secret to the locator or name of the secret you just created. Chalk reads the secret at build time and uses the value verbatim as the index URL. ### Configure your Python environment After you configure any of the indexes above, include your private packages in your pyproject.toml like any other dependency. Chalk's builder will resolve them through the configured indexes during deployment. ### Naming and other options Each index gets an auto-derived name (the first label of the index URL's hostname; e.g. https://pypi.org/simple → pypi). You can override this with the Name (override) field. If two indexes derive the same name, Chalk auto-suffixes with -2, -3, etc. The Options tab exposes two global toggles that affect the rendered uv.toml: - Prefer private indexes: When enabled, the rendered [[index]] block for PyPI is marked default = true, so uv treats PyPI as the fallback and searches your private indexes first. - Unsafe first match: When enabled, the rendered uv.toml sets index-strategy = "unsafe-first-match". # Debugging Persistence source: https://docs.chalk.ai/docs/debug-persistence ## Learn how to debug and troubleshoot feature persistence issues in Chalk Chalk uses background persistence writers to asynchronously persist query results to the online store, offline store, and metrics store. When these writers encounter issues, features may appear stale, offline queries may return incomplete data, or online store reads may return unexpected results. This guide walks through how to identify and resolve common persistence problems. ### How persistence works Understanding the data flow is essential for debugging. When a query executes on the engine, results flow through a message bus before reaching their final storage destinations: - The query server (engine) computes feature values and publishes them to the result bus (a Pub/Sub topic on GCP or an MSK/Kafka topic on AWS). - Background persistence writers consume messages from the result bus and write to their respective stores:The online writer persists values to the online store (ElastiCache, DynamoDB, Redis, etc.) for low-latency reads.The offline writer reads from the result bus, creates per-feature table schemas in the offline store if they don't already exist, transforms query results into parquet-encoded messages, and publishes them to the offline store streaming insert topic. It does not insert data into the database directly.The offline store streaming insert writer reads from the streaming insert topic and performs the actual database writes to per-feature observation tables and the query_log table in the offline store. Note that other components (such as the query engine) may also publish messages directly to the streaming insert topic, so this writer handles more than just the output of the offline writer.The offline store bulk insert writer handles writes that originate as parquet files in cloud storage (S3 or GCS) rather than as messages on a bus. When components like the offline query engine run with store_offline=True, they write result parquet files to a cloud storage bucket and publish a notification to a bulk upload topic. The bulk insert writer picks up these notifications and loads the files into the offline store using COPY INTO operations. This writer is required for non-BigQuery offline store backends (Snowflake, Redshift, Databricks); for BigQuery, dedicated BigQuery streaming writers handle this role instead. It requires a configured upload bucket (BQ_UPLOAD_BUCKET) with the correct cloud prefix (s3:// for AWS, gs:// for GCP).The metrics writer transforms results into metrics format and publishes them to a metrics bus, where the metrics bus writer persists them to the metrics store for observability. - Each writer runs as an independent Kubernetes deployment, autoscaled by KEDA based on queue depth. The offline store write path is a two-stage pipeline: the offline writer creates table schemas, transforms result bus messages, and republishes them to the streaming insert topic, then the streaming insert writer writes the data to the database. Both must be deployed for offline persistence to function. The bulk insert writer is independent of this pipeline — it handles loading parquet files from cloud storage into the offline store, which is how offline query results (e.g. from store_offline=True) are persisted. Which offline writers you need depends on your offline store backend: for BigQuery, use dedicated BigQuery streaming writers; for all other backends (Snowflake, Redshift, Databricks), use the bulk insert writer and streaming insert writer. A failure at any stage of this pipeline can cause data to stop flowing to one or more stores. ### Triage flowchart When you suspect a persistence issue, follow this decision tree: ### 1. Identify the symptom Start by determining which store is affected: - Online store returning stale or null values: the online writer may be down or behind. - Offline queries returning incomplete data: either the offline writer or the streaming insert writer may be down or behind. - Missing metrics in the dashboard: the metrics writer may be affected. - All stores affected: the result bus itself may be clogged, or the query server may not be publishing. ### 2. Check Kubernetes pod status Navigate to Infrastructure > Kube Events in the Chalk dashboard and filter by the background-persistence namespace. If the background-persistence namespace does not exist, the writers may not exist and we need to create them in the dashboard by navigating to Settings > Shared Resources > Background Persistence. See Creating Background Persistence Writers for how to configure the online, offline, and streaming insert writers. Look for pods in an unhealthy state: - OOMKilled: the writer is running out of memory. See Raising memory requests below. - CrashLoopBackOff: the writer is repeatedly crashing. Check logs for the root cause. - ContainerCreating (stuck for more than a few minutes) — there may be a node scheduling issue or resource shortage. ### 3. Check the result bus If writers appear healthy but data is not flowing, the result bus itself may be the bottleneck. Navigate to Infrastructure > Kubernetes and expand the drop down for Background persistence writers. Click on a result-bus-offline-writer or result-bus-online-writer pod to access the logs that can help indicate why data may not be ingesting. and filter by component:"background-persistence" to look for consumer lag or error messages on the pods with the names rust-result-bus-online-writer and the result-bus-offline-writer. Signs of a clogged result bus include: - Consumer lag increasing over time (messages are being produced faster than consumed). - Writers logging errors when attempting to deserialize or process messages. - Writers processing messages but failing to write to the downstream store (e.g. connection errors to Redis or BigQuery). ### Common issues and solutions ### OOMKilled writers This is the most common persistence issue. When a writer's memory usage exceeds its Kubernetes memory limit, the pod is terminated with an OOMKilled status. While KEDA will restart the pod, repeated OOM kills cause the writer to fall behind, creating a backlog on the result bus. Symptoms: pods cycling between Running and OOMKilled, increasing consumer lag on the result bus, stale or missing feature values. Solution: increase the memory request and limit for the affected writer. See Raising memory requests. ### Result bus backlog If writers were down for a period (due to OOM kills, crashes, or scaling issues), a backlog of unprocessed messages may accumulate on the result bus. Once writers are healthy again, they will work through the backlog, but this can take time depending on the volume. Symptoms: writers are healthy and running, but data is still delayed. Consumer lag is high but decreasing. Solution: wait for the backlog to drain. If you need to accelerate processing, you can temporarily increase the replica count for the affected writer in the background persistence configuration. ### Writer crash loops If a writer is in CrashLoopBackOff, it is repeatedly failing to start. Symptoms: pod restarts with increasing backoff intervals. Solution: check the pod logs in Infrastructure > Kubernetes and expanding the Background Persistence drop down. ### Features not persisting to offline store If online queries return correct values but offline data is missing or stale: - Verify that both the offline writer and the offline store streaming insert writer are deployed and healthy. A common issue is having the offline writer configured but missing the streaming insert writer (or vice versa). Check Settings > Team > Shared Resources > Background Persistence to confirm both are present. If the environment has no offline writers configured at all, you need to create them first. See Creating Background Persistence Writers for how to set them up. - Check the Health Metrics dropdown on the Background Persistence page — if the queues associated with offline writers are growing, the writers may need more replicas or resources. - Check that skip_offline is not set to True on the relevant stream resolvers. - Ensure that max_staleness is configured appropriately. - If offline query results (from store_offline=True) are not appearing in the offline store, the bulk insert writer may be misconfigured or missing. This writer is required for non-BigQuery backends. Verify that BQ_UPLOAD_BUCKET is set to a valid cloud storage path with the correct prefix (s3:// for AWS, gs:// for GCP), and that the associated storage integration has the correct permissions. ### Writer stuck in ContainerCreating If a writer pod is stuck in ContainerCreating for an extended period, it is typically unable to mount a required volume or secret. Symptoms: pod remains in ContainerCreating state for more than a few minutes. Kube Events may show FailedMount errors. Solution: check the pod logs in Infrastructure > Kubernetes and expanding the Background Persistence drop down for a possible cause. ### Features not persisting to online store If online queries unexpectedly get cache misses: - Verify that the online writer is deployed and healthy. If the environment has no online writer configured at all, you need to create one first. See Creating Background Persistence Writers for how to set it up. - Confirm the features queried are configured with a max_staleness. With zero (the default) staleness, an online read recomputes from resolvers instead of returning a stored value, so writes can appear to be missing even when they succeeded. - Confirm result persistence is not globally disabled for the environment — see Result persistence disabled by an environment variable. ### Result persistence disabled by an environment variable If query results are not being written to either the online or offline store even though the writers are deployed and healthy, check whether the CHALK_SKIP_RESULT_PERSISTENCE environment variable is set on the environment. This is a diagnostic flag that disables persistence of computed result values to the online store, offline store, and result bus. It is intended for local development and testing. It is an engine config variable, so in the dashboard it can be set in either of two places: - Integrations > Config variables: environment-wide engine config variables. Look for CHALK_SKIP_RESULT_PERSISTENCE set to 1 here. - Infrastructure > Resources: expand a service's Advanced section and check Engine Config Variable Overrides, which override config variables for that specific service or resource group. It can also be set on a single query as env_overrides={"CHALK_SKIP_RESULT_PERSISTENCE": "1"} passed to offline_query(...). ### Raising memory requests Background persistence writers run as Kubernetes deployments with configurable CPU and memory requests and limits. When a writer is OOMKilled, you need to increase its memory allocation. ### Via the Chalk dashboard UI - Navigate to Settings > Shared Resources > Background Persistence. - Select the writer to increase the memory on. - Increase the memory value in both the request and limit fields. - Click Save and Apply. ### Additional resources - Background Persistence Installation — configuration reference for persistence writers. - Kubernetes Resources Overview — details on Chalk Kubernetes components including background persistence scaling. - Feature Caching — configure max_staleness to control how stale features are served. - Common Errors — broader troubleshooting guide. # Metadata Plane & Data Plane Communication source: https://docs.chalk.ai/docs/metadata-plane-data-plane-communication ## What data flows between the Chalk Metadata Plane and Data Plane, and what the security implications are. Chalk's service architecture is divided into two planes: the Metadata Plane and the Data Plane. Understanding what flows between them is critical for security-conscious deployments, especially when configuring data residency controls or auditing what can transit outside your cloud boundary. Key principle: The Metadata Plane orchestrates the Data Plane but never stores customer feature values. All production query traffic flows directly from your API clients to the Data Plane. ### Summary of Data Flows | Flow | Direction | Contains Customer Data? | Required? | Can Be Disabled? | | ---------------------------------------- | --------------- | ------------------------ | --------- | ---------------- | | Logs | Data → Metadata | No (metadata only) | No | Yes | | Metrics | Data → Metadata | No | No | Yes | | Query Execution | Metadata → Data | **Yes** (feature values) | No | **Yes** | | EKS API Access | Metadata → Data | No | **Yes** | No | | Container Images (ECR/Artifact Registry) | Metadata → Data | No | **Yes** | No | ### Data Flows In Detail ### Logs Direction: Data Plane → Metadata Plane Logs are collected by an OpenTelemetry Collector running in the Data Plane EKS/GKE cluster and optionally forwarded to the Metadata Plane for centralized dashboarding. What's included: - Application logs from query servers, stream workers, and batch workers - Resolver execution logs (which features were computed and when) - Error logs and exception traces - Query audit logs What's NOT included: Actual feature values or customer PII. Logs contain metadata about computations (e.g. resolver name, latency, error type), not the data itself. Disabling: Logs can be kept exclusively within the Data Plane by configuring the OpenTelemetry Collector to export to your own observability tooling (e.g. Dynatrace, Datadog) instead of forwarding to the Metadata Plane. ### Metrics Direction: Data Plane → Metadata Plane Performance and operational metrics are emitted by the Data Plane and optionally forwarded for centralized monitoring. Metrics collected: - Query latency (P50, P95, P99) - Query success and error rates - Ingestion delay (time from event to feature availability in the online store) - Kafka consumer lag - Cache hit rates - Resource utilization (CPU, memory, disk) Metrics do not contain customer data — only aggregated operational statistics. Disabling: Like logs, metrics can be routed exclusively to your own monitoring systems via OpenTelemetry exporter configuration. ### Query Execution Direction: Metadata Plane → Data Plane This flow is what enables the Chalk web UI to execute queries interactively against your live data plane: the UI sends a request to the Metadata Plane, which forwards it as an API client to your Data Plane, and returns the results. What's included: Query inputs, feature outputs, and execution plan metadata. This flow can transmit customer feature values (including PII). Disabling: This flow requires a VPC Endpoint (VPCE / PrivateLink) between the Metadata Plane and Data Plane. Removing that connection disables it entirely, ensuring customer data never transits to the Metadata Plane. ### What you lose by disabling this flow Disabling query execution connectivity is not a simple on/off toggle — it removes a significant portion of Chalk's product capabilities: - Planning and backfill engines: aggregate backfill planning, historical feature computation scheduling, and incremental update strategies require the Metadata Plane to be able to query the Data Plane - Web UI query testing: interactive query debugging and feature value inspection - Real-time monitoring dashboards: feature freshness views and query plan visualization - Data quality tooling: automated health checks, feature assertions, and resolver performance analysis ### What is never affected Regardless of how you configure Metadata-to-Data-Plane connectivity, production online query traffic is not affected. When using Named Queries, your API clients talk directly to the Data Plane and never route through the Metadata Plane. OAuth token exchange still occurs via the Metadata Plane, but no feature values transit it. ### EKS API Access Direction: Metadata Plane → Data Plane The Metadata Plane needs access to your Data Plane's Kubernetes API server to manage the lifecycle of your Chalk deployment. What it's used for: - Deploying updated container images (built by Argo Image Builder) - Scaling Kubernetes workloads (query servers, stream workers, batch workers) - Executing rolling updates with zero downtime - Monitoring pod health and readiness states What's NOT included: No customer feature data is exposed via the Kubernetes API — only infrastructure metadata (pod status, deployment state, etc.). Disabling: This flow is required. Without it, Chalk cannot deploy code changes, scale resources, or perform health monitoring. ### EKS API access patterns There are two main options for how the Metadata Plane connects to the Data Plane Kubernetes API: Option A: Public EKS API with IP whitelisting (recommended) The EKS API server endpoint is publicly accessible, but access is restricted to the Chalk Metadata Plane's IP ranges via whitelist. All traffic is encrypted in transit (TLS), and AWS IAM authentication is required for all API calls. Benefits: simpler setup, zero-downtime deployments guaranteed, no dependency on VPC Endpoint infrastructure. Option B: Fully private EKS API The EKS API server is only accessible from within the VPC. If the Metadata Plane and Data Plane are in separate VPCs, this requires VPC peering or Transit Gateway. See Private EKS API Server Connectivity for the full topology and configuration walkthrough. Drawbacks: operationally complex, and zero-downtime deployments cannot be guaranteed because EKS API endpoints don't have stable IP addresses — a change in endpoint IP can break connectivity until network rules are manually updated (estimated recovery: 15+ minutes). This option is typically only warranted when regulatory requirements mandate a fully private control plane. ### Container Images Direction: Metadata Plane → Data Plane When you deploy a new version of your Chalk project, the Metadata Plane's Argo Image Builder builds Docker container images and pushes them to your ECR (AWS) or Artifact Registry (GCP) repository. The Data Plane then pulls these images when deploying. What's included: Docker container images for Chalk services. No customer feature data. Disabling: This flow is required for deployments. ### Direct Communication with Data Plane Direction: Customer API Client → Data Plane (with auth token exchange via Metadata Plane) Production query traffic from customer-owned API clients does not flow through the Metadata Plane. Instead, customer-owned API clients: - Exchange an auth token with the Metadata Plane. The client authenticates against the Metadata Plane (typically via OAuth client credentials) and receives an access token scoped to the target environment. Tokens are intended to be cached by the client for a long time — typically only re-fetched as they near expiry — so the Metadata Plane is in the request path only on the rare token refresh, never on the hot path of queries or ingestion. - Speak directly to the Data Plane's load balancers. Using that token, the client issues query and ingestion requests directly to the Data Plane. The load balancer fronting the Data Plane may be either private (VPC-internal, accessed via PrivateLink, VPC peering, or on-prem connectivity) or public (internet-facing with TLS) — the choice is up to the customer based on their network and compliance posture. Public load balancers can additionally restrict access via IPv4 allowlists (e.g. AWS security groups or WAF IP set rules) to limit which client networks can reach the Data Plane. This means feature values flow only between the customer's API clients and their own Data Plane; the Metadata Plane sees the auth handshake but not the query payloads or results. ### Online query traffic Online queries are synchronous: the client posts inputs to the Data Plane and receives the computed feature values in the response. The auth token is exchanged once with the Metadata Plane (and cached), then reused across many queries against the Data Plane load balancer. Online query sequence diagram ### Offline query traffic Offline (batch) queries are asynchronous. The client submits a job to the Data Plane, which enqueues it onto a job queue; an offline worker pod consumes the queue, runs the query against the offline store, and writes the result dataset. The client polls the Data Plane for status using the same cached token, and finally fetches the materialized dataset. Offline query sequence diagram ### Direct ingestion (upload_features) Producers can push feature values into the online store without going through a resolver by calling the Data Plane's upload_features endpoint. As with queries, the client first exchanges an auth token with the Metadata Plane (cached for a long time), then sends rows directly to the Data Plane load balancer. Background persistence asynchronously flushes the same rows to the offline store. Direct ingestion sequence diagram ### Configuration Options Chalk uses a shared gateway with a single load balancer for Chalk ingress. Depending on your security posture, there are multiple options for configuring connectivity to the Data Plane from the Metadata Plane. ### Public connectivity (recommended) For Public connectivity, the Data Plane Load balancer is internet facing, which allows both the Metadata Plane and data scientist to both easily talk to your Data Plane deployment. Chalk supports both IP whitelisting and TLS for stateful firewalls with encryption in transit. This is the simplest configuration and is recommended for most deployments. ### Private connectivity (optional) Establish a VPC Endpoint between the Metadata Plane and Data Plane, and rely on Chalk's RBAC system for access control. This gives you full product functionality: planning engines, web UI query testing, real-time dashboards, and data quality tooling. This is done by creating a VPC Endpoint Service in the Data Plane attached to the primary gateway load balancer. Subsequently, Chalk will provision a VPC Endpoint in the Chalk Metadata Plane to facilitate fully private connectivity between the two Planes. # Performance Profiling source: https://docs.chalk.ai/docs/profiling ## Collect CPU profiles and system traces from engine nodes and upload them to cloud storage for performance analysis. Chalk supports two profiling tools to help diagnose performance bottlenecks in your deployment: - Perf profiling: samples CPU activity using Linux perf and uploads gzip-compressed perf script output to your bucket. - Perfetto tracing: captures system-wide traces using Perfetto and uploads them in Perfetto's native proto format. Both are configured as observability daemons in your background persistence configuration. Once enabled, a daemon runs on each Kubernetes node and periodically uploads results to your bucket. ### Choosing which tool to use Perfetto offers a superset of the features that perf offers, but at the cost of some performance. With Perfetto, you can collect not only CPU profile information about the nodes that Chalk runs on, but also correlate those with traces emitted by the Chalk Engine. It is recommended to start with Perf profiling only, and move over to using Perfetto if Perf does not provide enough information. For Perfetto to work, in addition to configuring the profiling in the background persistence (as is outlined in this guide), you also need to: - Deploy a version of the engine with Perfetto traces compiled in (any O2 image will do) - Enable trace outputs from your engine by setting the environment variable LIBCHALK_ENABLE_PERFETTO_TRACING=true ### Prerequisites - A running Chalk deployment with background persistence configured. To install, please follow this guide - An S3 or GCS bucket for storing profiles and traces - Write access from the background persistence service account to that bucket ### Cloud storage permissions GCP: The service account used by background persistence (configured via service_account_name in common_persistence_specs) needs write access to the target GCS bucket. You also need google_cloud_project set in common_persistence_specs. AWS: Background persistence pods obtain AWS credentials through IAM Roles for Service Accounts (IRSA). The default IAM role is provisioned with broad S3 access within your account, so any same-account bucket will work without additional configuration. To write to a bucket in a different AWS account, add a bucket policy on the destination bucket granting access to the background persistence IAM role ARN. ### Perf profiling The perf collector samples CPU activity for Chalk-related processes on each node and periodically uploads gzip-compressed perf script output to your bucket. ### Enabling perf profiling There are two ways to enable perf profiling — through the Chalk dashboard if you manage your background persistence deployments there, or using the Chalk CLI. ### Using the Chalk dashboard - Navigate to Settings > Shared Resources > Background Persistence. - Click Edit JSON. - Add an observability_daemons array at the top level of the JSON, alongside the existing common_persistence_specs and writers: ``` "observability_daemons": [ { "keep_running_when_suspended": true, "perf_collector": { "perf_polling_frequency_hz": 99, "call_graph": true, "max_dumps_retained": 3, "dump_duration_seconds": 120, "bucket_subdirectory": "perf-data", "export_to": "s3://your-bucket-name" } } ] ``` - Replace s3://your-bucket-name with your bucket URI. Use gs:// for GCS. The subdirectory name can be anything you like. - Save and apply. ### Using the Chalk CLI With the Chalk CLI, you can export your current configuration, edit it, and re-apply to your background persistence deployment. ``` chalk infra describe persistence --json > persistence.json ``` Open persistence.json and add an observability_daemons array at the top level, alongside the existing common_persistence_specs and writers: ``` "observability_daemons": [ { "keep_running_when_suspended": true, "perf_collector": { "perf_polling_frequency_hz": 99, "call_graph": true, "max_dumps_retained": 3, "dump_duration_seconds": 120, "bucket_subdirectory": "perf-data", "export_to": "s3://your-bucket-name" } } ] ``` Replace s3://your-bucket-name with your bucket URI. Use gs:// for GCS. Then apply: ``` chalk infra apply persistence -f persistence.json ``` The CLI will show a diff of the changes and prompt for confirmation. ### Configuration reference Sampling frequency in Hz. 99 is a standard default that avoids aliasing with timer interrupts. Capture full call stacks with each sample. Required for flame graph generation. Defaults to true. Maximum number of profile files to keep on disk per node. Older files are uploaded and deleted. How often, in seconds, perf record rotates its output file and the cleanup loop runs. Defaults to 60. Bucket URI for uploads. Use s3://bucket-name for AWS, or gs://bucket-name for GCS. Path prefix within the bucket. Files are organized as . ### Where to find profiles Profiles appear in your bucket organized by node name: ``` BUCKET_SUBDIRECTORY/NODE_NAME/TIMESTAMP-perf-data.gz ``` For example, with bucket_subdirectory set to "perf-data" and a node named ip-10-0-1-42.ec2.internal: ``` s3://your-bucket-name/perf-data/ip-10-0-1-42.ec2.internal/20260220T153012-perf-data.gz ``` Each file contains gzip-compressed perf script output, filtered to Chalk-related processes. ### Tuning overhead Perf profiling adds CPU and I/O overhead to your nodes. You can adjust the following settings to find the right balance: - Lower the sampling frequency: Reduce perf_polling_frequency_hz (e.g., from 99 to 49). - Increase dump duration: A larger dump_duration_seconds produces fewer, larger files and reduces I/O frequency. ### Perfetto tracing The Perfetto daemon captures system-wide traces on each node. Traces can be triggered on a fixed time interval or on-demand via an HTTP endpoint that the Chalk CLI can call. Trace files are in Perfetto's native proto format and can be opened directly in the Perfetto UI. ### Enabling Perfetto tracing There are two ways to enable Perfetto tracing — through the Chalk dashboard if you manage your background persistence deployments there, or using the Chalk CLI. ### Using the Chalk dashboard - Navigate to Settings > Shared Resources > Background Persistence. - Click Edit JSON. - Add an observability_daemons array at the top level of the JSON, alongside the existing common_persistence_specs and writers: ``` "observability_daemons": [ { "keep_running_when_suspended": true, "perfetto_daemon": { "trigger": "PERFETTO_TRIGGER_TIME_INTERVAL", "interval": 60000, "max_retained_runs": 3, "bucket_subdirectory": "perfetto-traces", "export_to": "s3://your-bucket-name", "trigger_name": "chalk_traces", "config_text": "buffers: {\n size_kb: 102400\n fill_policy: RING_BUFFER\n}\n\ndata_sources: {\n config {\n name: \"linux.perf\"\n perf_event_config {\n all_cpus: true\n sampling_frequency: 100\n }\n }\n}\n\ntrigger_config {\n trigger_mode: CLONE_SNAPSHOT\n triggers {\n name: \"chalk_traces\"\n stop_delay_ms: 1000\n }\n}\n" } } ] ``` - Replace s3://your-bucket-name with your bucket URI. Use gs:// for GCS. Replace the config_text value with a valid Perfetto text proto config. - Save and apply. ### Using the Chalk CLI With the Chalk CLI, you can export your current configuration, edit it, and re-apply to your background persistence deployment. ``` chalk infra describe persistence --json > persistence.json ``` Open persistence.json and add an observability_daemons array at the top level, alongside the existing common_persistence_specs and writers: ``` "observability_daemons": [ { "keep_running_when_suspended": true, "perfetto_daemon": { "trigger": "PERFETTO_TRIGGER_TIME_INTERVAL", "interval": 60000, "max_retained_runs": 3, "bucket_subdirectory": "perfetto-traces", "export_to": "s3://your-bucket-name", "trigger_name": "chalk_traces", "config_text": "buffers: {\n size_kb: 102400\n fill_policy: RING_BUFFER\n}\n\ndata_sources: {\n config {\n name: \"linux.perf\"\n perf_event_config {\n all_cpus: true\n sampling_frequency: 100\n }\n }\n}\n\ntrigger_config {\n trigger_mode: CLONE_SNAPSHOT\n triggers {\n name: \"chalk_traces\"\n stop_delay_ms: 1000\n }\n}\n" } } ] ``` Replace s3://your-bucket-name with your bucket URI. Use gs:// for GCS. Then apply: ``` chalk infra apply persistence -f persistence.json ``` The CLI will show a diff of the changes and prompt for confirmation. ### Generating Perfetto text proto config The config_text field must contain a valid Perfetto text proto config. Regardless of which trigger mode you use, the config must include a trigger_config block with trigger_mode: CLONE_SNAPSHOT and a trigger whose name exactly matches the trigger_name field in your daemon config. This is how Perfetto knows when to snapshot the ring buffer and emit a trace. Write your config in a .pbtxt file. For example, a config that samples CPU at 99 Hz and snapshots on the trigger "chalk_traces" would look like: ``` buffers: { size_kb: 102400 fill_policy: RING_BUFFER } data_sources: { config { name: "linux.perf" perf_event_config { all_cpus: true sampling_frequency: 99 } } } trigger_config { trigger_mode: CLONE_SNAPSHOT triggers { name: "chalk_traces" stop_delay_ms: 1000 } } ``` Because config_text is embedded as a JSON string, newlines and quotes must be escaped. Use jq to produce the correctly escaped value from your .pbtxt file: ``` jq -Rs '.' < perfetto.pbtxt ``` This prints the file contents as a quoted, escaped JSON string. Copy the output (including the surrounding quotes) and use it as the config_text value in your persistence config. ### Trigger modes The Perfetto daemon supports two ways to initiate a trace capture: - PERFETTO_TRIGGER_TIME_INTERVAL: Traces are collected automatically on a fixed interval (controlled by the interval field). Use this for continuous background profiling. - PERFETTO_TRIGGER_HTTP: An HTTP endpoint is exposed on port 3565. The cluster manager can call this endpoint to trigger a trace on demand. Use this when you want to capture a trace at a specific moment, such as during a known slow request. At most one HTTP-triggered Perfetto daemon may be configured per environment. When PERFETTO_TRIGGER_HTTP is used, the cluster manager is automatically configured with the CHALK_PERFETTO_DAEMON_PORT and CHALK_PERFETTO_DAEMON_NAMESPACE engine configuration variables. You can then trigger a snapshot with: ``` chalk profiling perfetto-snapshot ``` Regardless of the trigger mode you use for the daemon, the underlying Perfetto config needs to use trigger_mode: CLONE_SNAPSHOT for the system to work properly. ### On-demand tracing (HTTP trigger) To enable on-demand tracing via chalk profiling perfetto-snapshot, use PERFETTO_TRIGGER_HTTP as the trigger mode and set a trigger_name: ``` "observability_daemons": [ { "keep_running_when_suspended": true, "perfetto_daemon": { "trigger": "PERFETTO_TRIGGER_HTTP", "trigger_name": "chalk_snapshot", "max_retained_runs": 5, "bucket_subdirectory": "perfetto-traces", "export_to": "gs://your-bucket-name", "config_text": "buffers: {\n size_kb: 102400\n fill_policy: RING_BUFFER\n}\n\ndata_sources: {\n config {\n name: \"linux.perf\"\n perf_event_config {\n all_cpus: true\n sampling_frequency: 100\n }\n }\n}\n\ntrigger_config {\n trigger_mode: CLONE_SNAPSHOT\n triggers {\n name: \"chalk_snapshot\"\n stop_delay_ms: 1000\n }\n}\n" } } ] ``` Once deployed, trigger a trace capture with: ``` chalk profiling perfetto-snapshot ``` ### Configuration reference Perfetto tracing configuration in text proto format (.pbtxt). This is required. See the Perfetto config documentation for available data sources and options. How traces are initiated. Use PERFETTO_TRIGGER_TIME_INTERVAL for automatic periodic collection, or PERFETTO_TRIGGER_HTTP for on-demand collection via chalk profiling perfetto-snapshot. Interval between traces in milliseconds. This is also how frequently the system will be scanned for new trace files to upload. Perfetto trigger name. Required. Must exactly match the trigger name in the trigger_config block of config_text. Maximum number of trace files to keep on disk per node. Older files are uploaded and deleted. Recommended to set this to 0 to start uploading immediately. Bucket URI for uploads. Use s3://bucket-name for AWS, or gs://bucket-name for GCS. Path prefix within the bucket. Files are organized as . ### Where to find traces Traces appear in your bucket organized by node name: ``` BUCKET_SUBDIRECTORY/NODE_NAME/TIMESTAMP-perfetto-trace.pb ``` For example, with bucket_subdirectory set to "perfetto-traces" and a node named ip-10-0-1-42.ec2.internal: ``` s3://{your-bucket-name}/perfetto-traces/ip-10-0-1-42.ec2.internal/20260220T153012-perfetto-trace.pb ``` ### Common configuration The following fields apply to both the perf_collector and perfetto_daemon daemon objects. Keep the daemon running when background persistence is suspended. Kubernetes resource requests (cpu, memory). Defaults to 25m CPU and 64Mi memory. Kubernetes resource limits (cpu, memory). Custom container image. Omit to use the default. ### Sending data to Chalk Once data has been collected, download it from your bucket, compress it into an archive, and send it to your Chalk support contact. ### Perf profiles For AWS: ``` aws s3 sync s3://your-bucket-name/perf-data/ ./perf-data/ tar czf perf-profiles.tar.gz perf-data/ ``` For GCS: ``` gcloud storage rsync -r gs://your-bucket-name/perf-data/ ./perf-data/ tar czf perf-profiles.tar.gz perf-data/ ``` ### Perfetto traces For AWS: ``` aws s3 sync s3://your-bucket-name/perfetto-traces/ ./perfetto-traces/ tar czf perfetto-traces.tar.gz perfetto-traces/ ``` For GCS: ``` gcloud storage rsync -r gs://your-bucket-name/perfetto-traces/ ./perfetto-traces/ tar czf perfetto-traces.tar.gz perfetto-traces/ ``` ### Disabling profiling When profiling is no longer needed, remove the observability_daemons entry from your background persistence configuration and re-apply. The profiling daemonset will be removed automatically. In the dashboard, navigate to Settings > Shared Resources > Background Persistence, click Edit JSON, delete the observability_daemons block, and save. With the CLI: ``` chalk infra describe persistence --json > persistence.json # Remove the observability_daemons array from persistence.json chalk infra apply persistence -f persistence.json ``` # Tutorial: Fraud Detection Pipeline source: https://docs.chalk.ai/docs/fraud-tutorial ## Build a feature pipeline for fraud detection. ### Introduction Chalk helps you build out feature pipelines for training and serving machine learning models. The building blocks of Chalk are features and resolvers. Features specify what you want your data to look like and resolvers tell Chalk where to find your "data". In Chalk, features are defined as Python classes with annotated attributes. Note how the User class is annotated with the @features decorator that's imported from chalk.features. ``` from chalk.features import features import datetime as dt @features class User: id: int birthday: dt.datetime name: str email: str # computed features username: str age: int is_adult: bool ``` Features are the abstract underlying attributes of your data. In the example above id, birthday, name, email, username, age, and is_adult are all features. Features are grouped together (namespaced) in what are called feature classes. For instance, the User class is a feature class (also sometimes referred to as a feature set). Specific instances of a feature class are feature class instances. Note, you define your features before defining how they will be computed. Some of these features might come directly from a database table: for instance, the id, birthday, and name fields that we've defined above. Some might be calculated in real time based on the current timestamp (like the age feature) or a different upstream feature (like the is_adult feature, which will depend on the age feature). You could deploy the code above to Chalk and it would be considered valid. However, if you tried to query for a field (for example, if you asked chalk to return the name of a user with id 1) then you'd get an error. In particular, you would get the following error: ``` $ chalk query --in user.id=1 --out user.name Results No scalar features Errors Resolver Not Found user.age Failed to find any valid resolver for feature: 'user.name' and no `default` value or `max_staleness` was specified for 'user.name', so the feature cannot be defaulted or resolved from the online store. ``` We'll talk more about deploying code to Chalk later, but, if you'd like to, you're welcome to skip ahead to get a better sense of deploying code to Chalk and running queries. This should not be surprising. Chalk has no connection to your underlying data. So it doesn't know how to get the name of the user with an id of 1 (or any id for that matter). In particular, the error message tells us that a "Resolver" was not found. A resolver is a function that takes in features and outputs features. Broadly, there are two types of resolvers: root and non-root. Root resolvers don't take any features (or take only a primary key) as their input and are used to fetch data from external sources. In the User example we sketched out above, you could think of a root resolver as a SQL query that reads from a users table in an external database. While there are exceptions, this is what the majority of root resolvers look like. For example, to read from the users table in your PostgreSQL database, you might write a get_users.chalk.sql file, with the following contents: ``` -- source: postgres -- resolves: User select id, birthday, email, name from users ``` By writing this file, you've defined a SQL resolver, named get_users, which knows how to resolve birthday, name, or email for any given id in the users table. At this point, you might have noticed that the SQL query above doesn't explicitly know how to get the birthday or name of a given id, it just returns all users. Chalk gets around this by pushing filters into your SQL queries. Let's say you've defined the SQL resolver above and deployed it to Chalk. You rerun the query that failed above: ``` $ chalk query --in user.id=1 --out user.name ``` Chalk will look at the resolvers you've defined and ask "given a User.id, can I calculate a User.name". It will see the SQL resolver defined above and realize that it can be used to get User.name from a User.id. It will push the id filter into the SQL query, running. ``` select name from users where id=1 ``` In addition to the filter a projection will be pushed into your SQL query: only the name column is requested. When running a query, Chalk processes the minimum amount of data required to calculate a desired output features. Getting this to work fully, will involve setting up a database connector to Chalk and deploying code, both of which we will cover later. For now, it is important to note that the process of defining resolvers is incredibly modular. Now lets look into defining a non-root resolver, which computes a User's email username. In Chalk, non-root resolvers take one of two forms: - Python functions with feature annotations, - inline expressions. We will look at both variants, but functionally they are similar. Both take input features and generate output features. ``` from chalk import online from chalk.features import features import datetime as dt @features class User: id: int birthday: dt.datetime name: str email: str # computed features username: str age: int is_adult: bool @online def get_email_username(email: User.email) -> User.username: return email.split('@')[0] ``` As mentioned above, this could also be written as an inline expression, like so: ``` from chalk import online from chalk.features import features import datetime as dt @features class User: id: int birthday: dt.datetime name: str email: str # computed features username: str = F.split_part(_.email, delimiter="@", index=0) age: int is_adult: bool ``` The advantage of inline expressions is that they are statically compiled into the execution engine, which often provides performance advantages. Chalk also tries to statically compile Python resolvers by parsing the AST of the function! By writing the get_email_username Python function, we've defined another resolver. If we deploy this new code to Chalk, we can now query for the username of a user with a given id. ``` $ chalk query --in user.id=1 --out user.username ``` When running the above query, Chalk will identify that it knows how to determine an email from a given id using the get_user resolver. It will also determine that it knows how to calculate a username for an email using the get_email_username resolver. Chalk will execute both these resolvers and return the result. The focus on data instead of pipelines may be unfamiliar at first. Traditional orchestration platforms like Airflow or Dagster explicitly compose functions which produce data into a DAG of tasks. With Chalk, the DAG of resolvers is defined implicitly by the features they produce. This architecture makes it easy to build out feature pipelines that are reusable and composable. Chalk handles tracking your features for temporal consistency, running your resolvers in parallel, and horizontally scaling your feature pipelines. Alright, enough with the intro. Let's get started building! ### Fraud Detection Pipeline This tutorial walks through the process of building a feature pipeline for fraud detection. We'll cover the full feature development lifecycle: - Data Modeling - Creating feature classes for the data we want to compute. - SQL Resolvers - Mapping data from SQL sources to feature classes. - Python Resolvers - Defining resolvers in Python that compute derived features and call external APIs. - Inference - Integrating Chalk into production decisioning systems. - Backtesting - Experimenting with new features In this tutorial, we'll assume the following existing data architecture: - a PostgreSQL database with three tables: users, accounts, and transactions. - a Snowflake analytics database with the same three tables: users, accounts, and transactions, which is periodically updated in response to changes in the PostgreSQL database. Before you get started, make sure you have the Chalk CLI installed. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. ### Define features We'll start by modeling out the Users feature class. We'll start simple with three scalar features: User.id, User.name, and User.email. First, we'll create a new file called feature_sets.py where we'll define a User class decorated with @features. ``` from chalk.features import features @features class User: id: int # The name the user provided to us at signup. # :owner: identity@chalk.ai # :tags: pii name: str # :tags: pii email: str ``` Note that at this point we haven't defined how to compute these features. We are only thinking about the data that we would like to have. ### Primary keys There are a few things to note here. First, all our feature classes need to have a unique id field. By default, this is the field named id. However, if you want to use a different feature name as the primary key, you can specify it by annotating the primary key feature with the Primary type. ``` from chalk.features import features, Primary @features class User: - id: int + user_id: Primary[int] name: str email: str ``` ### Tags, Descriptions, and Owners In our features above, we've added some comments and annotations. These are optional, but can be useful for documentation and for setting alerting policies. For example, you may wish to send Pagerduty alerts to different teams based on the owner of the related feature. All of the comments and tags from the code also show up in the Chalk dashboard, and are indexed for search. For example, we've added a pii tag to the name and email fields. This means that these fields will be treated as personally identifiable information and will be subject to additional restrictions. ### Has-One Relationships Next up, we'll define a feature class related to our User. We'll call this class Account and it will represent a user's bank account. ``` from chalk.features import features @features class Account: id: int # The name of the owner of the account. title: str # The id of the user that owns this account. user_id: int # The balance of the account, in dollars. balance: float ``` This should look much like what we did for the User class. However, we may want to link these two classes together. We can do this by adding a user field to the Account class. ``` @features class Account: id: int - user_id: int + user_id: User.id balance: float + # The user that owns this account. + user: User ``` This denotes that each account has one user, and that the Account.user_id and the User.id are equal and of type int, as described by User.id. Once we've defined the relationship on one side of the join, we can define the inverse relationship on the other side without needing to specify the foreign key again. ``` @features class User: id: int name: str email: str + # The account that this user owns. + account: "Account" ``` Note that the account annotation in the User feature class is in quotes. Because the Account class is defined later in the file, Chalk will recognize this as a valid feature reference and process it correctly. ### Has-Many Relationships The final feature entity that we'll define in this tutorial is for transactions. Each account has many transactions and each transaction is linked to a single account. We'll define the Transaction class and link it to our Account class as follows: ``` + from enum import Enum - from chalk.features import features + from chalk.features import features, DataFrame, FeatureTime + class TransactionStatus(str, Enum): + PENDING = "pending" + CLEARED = "cleared" + FAILED = "failed" + + @features + class Transaction: + id: int + + # The id of the account that this transaction belongs to, set to a join. + # We refer to features and feature classes defined further down in the file + # using quotation marks, so Chalk will recognize that it is a valid + # feature reference to be processed later. + account_id: "Account.id" + + # The amount of the transaction, in dollars. + amount: float + + # The status of the transaction, defined as an enum above. + status: TransactionStatus + + # When the transaction occurred + created_at: FeatureTime + + # Because we define the join condition between + # `Transaction` and `Account` below, we don't + # need to repeat it here. + account: "Account" @features class User: id: int name: str email: str # The account that this user owns. account: "Account" @features class Account: id: int user_id: User.id balance: float user: User + transactions: DataFrame[Transaction] ``` This is the first time we're seeing the DataFrame type. A Chalk DataFrame models tabular data in much the same way that Pandas does. However, there are some key differences that allow the Chalk DataFrame to increase type safety and performance. Like pandas, Chalk's DataFrame is a two-dimensional data structure with rows and columns. You can perform operations like filtering, grouping, and aggregating on a DataFrame. However, there are two main differences. - Lazy implementation - Chalk's DataFrame is lazy and can be backed by multiple data sources, where a pandas.DataFrame executes eagerly in memory. - Usable as a type - Chalk's DataFrame[...] can be used to represent a type of data with pre-defined filters. You might also notice that we used an Enum feature. Chalk supports many feature types, including Enums, lists/sets, and dataclasses. We also added a created_at field to the Transaction class of type FeatureTime. FeatureTime is a special annotation that can only be assigned to a single feature in a feature class. This feature specifies the logical time of a feature instance — it is used for point in time correctness and incremental updating. We'll cover both later in this tutorial. ### Configuring SQL sources The primary source of data for most companies are SQL databases (note, by SQL database we don't mean relational databases: we mean any database that can be queried with SQL — in practice this means pretty much any of them). Chalk can automatically ingest data from SQL databases and map results into feature classes. In our example application, we have two databases: PostgreSQL and Snowflake. Our PostgreSQL database is the primary database used in our codebase, and our Snowflake database is used for analytics, with tables populated from DBT views and batch jobs. To configure your SQL sources in Chalk, we'll: 1). Add your data sources to the Chalk dashboard 2). Define the data sources in your Chalk code. ### Adding Data Sources in the Chalk Dashboard To add your data sources to the Chalk dashboard, go to the Data Sources tab of the dashboard and click on the Add a data source button. You can then select the type of data source you want to add. Select PostgreSQL and fill in the required information. Repeat the same process with a Snowflake data source. Keep track of the names that you've given to each of your data sources: these will be used in the next step. We recommend naming your PostgreSQL data source postgres and your Snowflake data source snowflake. Note, the data source connection information you add in the dashboard will be stored securely using the secrets manager of your cloud provider. ### Define Your Data Sources in Your Chalk Code To define your data sources, you'll need to create a datasources.py file that contains a SnowflakeSource and a PostgreSQLSource: ``` from chalk.sql import SnowflakeSource, PostgreSQLSource # if you named your postgres or snowflake resolvers something other than `postgres` # and `snowflake`, you'll have to update the code accordingly. snowflake = SnowflakeSource(name="postgres") postgres = PostgreSQLSource(name="snowflake") ``` Now that we've defined our data sources, we can use them to create root SQL resolvers. ### Online data Chalk's preferred way to ingest data from SQL databases is to use SQL file resolvers. With SQL file resolvers, you can write your root resolvers in SQL, and use whatever database tooling you're familiar with to test, lint, and debug your code. To create a SQL resolver, add a file to your project directory with the extension .chalk.sql. You can now write a SQL query in this file. You will also need to add metadata to the top of the file to tell Chalk how to ingest the data. For example, say that we want to resolve the name and email features of our User feature class from a PostgreSQL table. To do this, we can write the following SQL file resolver: ``` -- get users from postgres. -- resolves: User -- source: postgres select id, full_name as name, email from users ``` The resolves key (--resolves: User) tells Chalk which feature class the columns in the select statement should be mapped into. Then, the target names of the query are compared against the names of the features on the specified feature class. If the names match after stripping underscores and lower-casing, the select target is mapped to the feature. In the example above, we aliased the full_name column to name, so it will be mapped to the name attribute on the User feature class. Chalk validates your SQL file resolvers when you run chalk apply. If you want to validate your code without deploying, you can run the chalk lint command. The source key (-- source: postgres) tells Chalk which integration to use to connect to the database. Other comments in the SQL file resolver are indexed by Chalk and can be searched in the Chalk dashboard. In the example resolver above, the comment get users from postgres becomes the description of the resolver in the dashboard. Let's also add our get_accounts and get_transactions SQL file resolvers: ``` -- get accounts from postgres. -- resolves: Account -- source: postgres select id, user_id, balance, from accounts ``` ``` -- get transactions from postgres. -- resolves: Transaction -- source: postgres select id, account_id, amount, status, created_at from transactions ``` ### Deploying! Now that we've written a few resolvers, we can deploy our feature pipeline and query our data in realtime. To deploy you'll want to run chalk apply. You will be shown the changes that you've made since your last deployment and prompted for confirmation ``` chalk apply ✓ Found resolvers ✓ Successfully validated features and resolvers! ✓ Checked against live resolvers Added Features Name ─────────────────────────── + transaction.id + transaction.account_id + transaction.amount + transaction.status + transaction.created_at + transaction.account + user.id + user.name + user.email + user.account + account.id + account.user_id + account.balance + account.user + account.transactions Would you like to deploy? [y/n] ``` If you accept, chalk will build and deploy your new code. Once that's done, you can start querying your data in realtime! ### Querying Now that we've deployed our feature pipeline, we can query our data in realtime. One of the easiest ways to do this is from the Chalk CLI. ``` $ chalk query --in user.id=1 --out user.name --out user.email user.name "John Doe" email "john@doe.com" ``` This query will fetch the name and email attributes from the User feature class for the user with id=1, hitting the PostgreSQL database directly. ### Push-down filters Note that in SQL file resolver that we wrote, we didn't include a where clause. However, Chalk automatically pushes down filters to the database when querying features. The SQL query that executes against our PostgreSQL database is: ``` select id, full_name as name, email from users + where id = 1; ``` Chalk can also push down non-primary key filters to SQL databases. For example, to fetch all transactions for a user, Chalk will modify the get_transaction resolver to include a where clause for the user's account_id: ``` select id, account_id, amount, status, date from txns + where account_id = 38; ``` You can see this in action, by querying for a user's transactions: ``` $ chalk query --in user.id=1 --out user.account.transactions Results No scalar features user.account.transactions id account_id amount status created_at ───────────────────────────────────────────────────────────────────────────────────── 197524 38 12.00 "cleared" "2023-12-02T21:05:54.057868+00:00" 198604 38 27.51 "cleared" "2023-04-29T22:27:12.058023+00:00" 210326 38 93.27 "cleared" "2023-02-17T21:49:27.058144+00:00" 228363 38 1.05 "cleared" "2023-04-20T21:46:33.058240+00:00" 230225 38 23.91 "cleared" "2023-06-09T03:07:44.058314+00:00" 235551 38 12.20 "failed" "2022-12-26T15:45:07.058416+00:00" ``` ### Derived Features We've noticed that some fraudsters try to link stolen accounts to our platform and attempt to transfer money through our system. To detect this behavior, we want to compute a similarity score between the user's name and the account's title. We'll start by adding this new feature, account_name_match, to our User feature class. ``` @features class User: id: int name: str email: str account: "Account" + # The similarity between the user's name and the account's title. + account_name_match: float ``` Next, we'll define a resolver that computes this feature. We'll use Jaccard similarity to compute the similarity score. ``` from src.feature_sets import User from chalk import online @online def account_name_match( title: User.account.title, name: User.name, ) -> User.account_name_match: """Docstrings show up in the Chalk dashboard""" intersection = set(title) & set(name) union = set(title) | set(name) return len(intersection) / len(union) ``` The @online decorator tells Chalk that this resolver should be called in realtime when the User.account_name_match feature is requested. Our feature dependencies are declared in the function signature as User.account.title and User.name. Chalk will automatically retrieve User.account_id and User.name with our get_user.chalk.sql resolver. Then, using this account_id, Chalk will retrieve Account.title from the get_account.chalk.sql resolver. Lets deploy our code. ``` $ chalk apply --branch tutorial ✓ Found resolvers ✓ Deployed branch ``` Note, this time we are using a branch (by specifying the --branch flag). In testing, new features, we recommend deploying your feature pipeline to a branch, which allows you to test your changes without affecting your production feature pipelines. An additional benefit of branches is that they are incredibly lightweight — they should deploy pretty much instantly. ### Testing Resolvers are callable functions, so we can also test them like any other Python function. Let's test our new resolver by writing a unit test: This is one of the only times when you'll explicitly call your resolver functions. We covered this idea already in the intro, but since its one of the central concepts behind Chalk it is worth reiterating. Most of the time, Chalk will determine which resolvers it needs to run to compute the features you've requested in a query. This is just like how a SQL database decides how to compute the output features you've requested. ``` from src.resolvers import account_name_match def test_names_match(): """Resolvers can be unit tested exactly as you would expect. Here, the `account_name_match` resolver should return 1.0 because the `title` and `name` are identical. """ assert 1 == account_name_match( title="John Coltrane", name="John Coltrane", ) def test_names_completely_different(): """The `account_name_match` resolver should return 0 because the `title` and `name` don't share any characters. """ assert 0 == account_name_match( title="John Coltrane", name="Zyx", ) ``` If you want to learn more, we also provide additional docs on testing resolvers. ### Reverse ETL Let's say we have a rather complex feature that we can't serve in realtime because it increases the latency of a query too much for a particular production use case. For instance, lets say we want to add a new feature to our users that indicates the average time between their transactions. Maybe we believe that we can use this to detect whether a new transaction is fraudulent. To add this feature, we'd add it to the User feature class: ``` @features class User: id: int name: str email: str account: "Account" # The similarity between the user's name and the account's title. account_name_match: float # The average time between each of a user's transactions, from the last # 30 days, in ms. + average_time_between_transactions_30d: float ``` We would then write a resolver to calculate this feature: ``` from chalk import online from chalk.features import after @online def get_average_time_between_transactions( transactions: User.account.transactions[ Transaction.created_at, after(days_ago=30) ] ) -> User.average_time_between_transactions: """Computes the average time between transactions for a user""" # conversion to polars is cheap since both Chalk and Polars DataFrames # Use Arrow as their memory representation. df = transactions.to_polars() # Get the date column, convert to milliseconds timestamps, and sort date_col = df.collect().get_column( str(Transaction.created_at) ).dt.timestamp("ms").sort() return (date_col - date_col.shift(1)).mean() ``` For good measure, lets also add this to our tests and make sure that it is running as expected: ``` import datetime as dt from chalk.features import DataFrame from src.resolvers import get_average_time_between_transactions def test_get_average_time_between_transactions(): """To unit test Dataframe taking resolvers, you can construct and pass in a Chalk DataFrame: """ now = dt.now(tzinfo=dt.timezone.utc), transactions = DataFrame({ Transaction.created_at: [ # will be filtered out since it did not occur in last 30d dt.datetime(1990, 1, 1, tzinfo=dt.timezone.utc), now, now - dt.timedelta(seconds=1), # diff = 1000 now - dt.timedelta(seconds=2), # diff = 1000 now - dt.timedelta(seconds=3), # diff = 1000 now - dt.timedelta(seconds=4), # diff = 1000 now - dt.timedelta(seconds=6), # diff = 2000 ] # ----------- }) # 6000 / 5 = 1200 assert 1200 == get_average_time_between_transactions( transactions ) ``` Lets deploy this new feature to a branch and run a couple test queries: ``` chalk apply --branch new-fraud-feature ``` ``` chalk query \ --in user.id=1 \ --out user.average_time_between_transactions_30d \ --branch new-fraud-feature ``` After some testing, we realize a couple of things: 1). This feature is too slow to compute in real time (for our use case), 2). This feature doesn't change very often. In practice, even executing the above resolver in realtime should be really fast. When building new features, do some testing. You may not even need to reverse-ETL features into the online store to achieve your target latency. Reverse-ETL-ing features into the online store adds state and complexity to your feature pipeline. We recommend only doing this if you are unable to achieve your latency targets. We decide to reverse-ETL this computed feature from our snowflake data store into our Chalk online store. At a high level, this means that periodically (maybe once a day), we'll: - Look for all users with new transactions in our snowflake data source, - Recompute the average_time_between_transactions_30d for these users, - Load the new average_time_between_transactions_30d values into the Chalk online store. Of note, we don't want to use our PostgreSQL database for this, we want to use Snowflake, which is optimized for bulk data loads unlike Postgres. As a result, we'll need to define some offline resolvers. So far we've only written online resolvers. You can think of offline resolvers as overrides for online resolvers that are run in bulk data request scenarios. At a high level offline resolvers let you pull your feature data from data stores that are optimized for bulk data loading — this is exactly what we want for our reverse-ETL process. To set up a reverse-ETL process, we'll need to do three things: - set a staleness policy on your target feature, - add offline resolvers (technically optional, but strongly recommended), - create a scheduled query. ### Setting a Max Staleness on Our Feature We can have Chalk reverse-ETL our offline data into our online store by setting the max_staleness on a feature and using scheduled queries. ``` from chalk.features import feature @features class User: id: int name: str email: str account: "Account" # The similarity between the user's name and the account's title. account_name_match: float # The average time between each of a user's transactions, from the last # 30 days, in ms. - average_time_between_transactions_30d: float + average_time_between_transactions_30d: float = feature(max_staleness="infinity") ``` The max_staleness keyword argument tells Chalk how stale a feature value can get before it should be refreshed. In this case, we're telling Chalk that we'll tolerate arbitrarily old feature values. However, we could also specify a max_staleness of 1h or 1d to tell Chalk not to serve feature values that are older than 1 hour or 1 day. To take advantage of our max_staleness, we need to get computed feature into the online store. Feature are only written to the online store if they (or their feature class) have been given a max_staleness value. Features with a max_staleness are written to the online store in two ways: - Passively, when a feature value is computed in response to an online query, - Actively, in response to an ETL or scheduled query. The first time average_time_between_transactions_30d is queried for any given user it will be computed by the get_average_time_between_transactions resolver. All subsequent times that the feature is requested it will be read from the online store (until the feature is evicted from the cache). ### Adding Offline Resolvers By setting up offline SQL resolvers, we are telling Chalk that there are times when we don't want to pull data from our Postgres data source and instead want to get our data from a bulk data store. There are a couple reasons why we might want to do this: 1). We don't want to overwhelm our Postgres data source with requests that don't require real time data, 2). We want to query for large chunks of data (which is the purpose of a database like snowflake). In the case of reverse-ETLing data, we have decided that we are tolerating some degree of staleness in our average_time_between_transactions_30d feature in exchange for being able to serve the feature faster. As a result, it makes sense to read the new data from snowflake. To connect our feature classes to Snowflake, we'll write three more SQL resolvers, this time labeling them with type: offline and source: snowflake. ``` -- get users from snowflake. -- resolves: User -- source: snowflake -- type: offline select id, full_name as name, email from users ``` ``` -- get accounts from snowflake. -- resolves: Account -- source: snowflake -- type: offline select id, user_id, balance, from accounts ``` ``` -- get transactions from snowflake. -- resolves: Transaction -- source: snowflake -- type: offline select id, account_id, amount, status, created_at from transactions ``` ### Creating a Scheduled Query With our offline SQL resolvers defined, we can now write a ScheduledQuery which will periodically run and write updated average_time_between_transactions_30d features to the online store. ``` from chalk import ScheduledQuery ScheduledQuery( name="user-transaction-reverse-etl-features", schedule="0 * * * *", output=[User.average_time_between_transactions_30d], store_online=True, store_offline=True, incremental_resolvers="get_transactions_offline", ) ``` Our schedule "0 * * * *", means that we will run this query and update the online store every hour. This means our features will be one hour fresh (disregarding the snowflake data source lag). Note, this allows really precise and granular trade-offs between staleness, latency, and compute. In addition, by specifying an incremental resolver, we are telling Chalk to remember the high ingest time from the previous run and only pull data that is fresher than that high water mark. Again Chalk is processing the minimum amount of information it requires to calculate the average_time_between_transactions_30d. ### API Calls Any Python function can be used as a resolver. This means that we can call APIs to compute features. Let's add a feature that computes the user's FICO score from our credit scoring vendor, Experian. As we did earlier, we'll first add the features that we want to compute: ``` + from chalk.features import feature @features class User: id: int name: str email: str account_name_match: float + # The fraud score, as provided by a third-party vendor. + fico_score: int = feature(min=300, max=850, strict=True) + + # Tags from our credit scoring vendor. + credit_score_tags: list[str] ``` We are adding strict validation to our fico_score feature to ensure that we only store and utilize valid FICO scores. Now, we can write a resolver to fetch the user's FICO score from Experian. ``` from src.feature_sets import User from src.mocks import experian from chalk.features import online, Features @online def get_fraud_score( name: User.name, email: User.email, ) -> Features[User.fico_score, User.credit_score_tags]: response = experian.get_credit_score(name, email) # We don't need to provide all the features for # the `User` class, only the ones that we want to update. return User( fico_score=response['fico_score'], credit_score_tags=response['tags'], ) ``` Here, we are returning two features of the user, User.fico_score and User.credit_score_tags. We use the Features type to indicate which feature we expect to return. Also note that we are initializing the User class with only the features that we want to update. This partial initialization is the primary difference between Python's @dataclass and Chalk's @features. ### Setting up a Materialized Windowed Aggregation When defining aggregations, it is often useful to compute a windowed aggregation to define the same aggregation over different time windows. When defining a windowed aggregation over high-cardinality datasets like transactions, it can be more performant to materialize these aggregations in the online store by defining a materialized windowed aggregation. In a materialized windowed aggregation, you can define time buckets, and Chalk will pre-aggregate the minimum data required over all the data in each time bucket necessary to compute the aggregation for all time windows in the feature definition. For example, say we want to compute the count of transactions associated with an Account over the past 7 days, 30 days, and 90 days. Then we can define the materialized windowed aggregation below: ``` from chalk.features import feature @features class Account: id: int user_id: User.id balance: float user: User transactions: DataFrame[Transaction] + + num_transactions: Windowed[int] = windowed( "7d", "30d", "60d", materialization={ "bucket_durations": {"1d": ["7d"], "3d": ["30d", "60d"]}, }, expression=_.transactions[ _.created_at > _.chalk_window, _.created_at < _.chalk_now ].count(), ) ``` This defines a materialized windowed aggregation over the Account.transactions DataFrame, which counts the number of transactions associated over the past 7, 30, and 60 days. The materialization argument tells Chalk to pre-aggregate the data in buckets in the online store, such that the 7-day aggregation is pre-aggregated in 1-day buckets, and the 30-day and 60-day aggregations are pre-aggregated in 3-day buckets. Because this aggregation .count() only requires the count of transactions that fall within each bucket, we would store the count of transactions for each bucket, and at query time sum the counts for the buckets that fall within the requested time window. This configuration allows you to define more performant aggregations over high-cardinality datasets, such as transactions, where recomputing the aggregation could be computationally expensive by materializing the aggregations in buckets. Because the materialized windowed aggregation relies on the materialized buckets in the online store, and branch deployments do not persist data to the online store, we would need to run a full deploy with chalk apply to use this new feature. After running a chalk apply, we can run chalk aggregate backfill --feature account.num_transactions to populate the buckets, and then we can query the feature as normal. In production, you could also set up a schedule to periodically backfill the buckets, or stream the data into the online store, to keep the buckets up to date. ### Deploying Finally, we'll want to deploy our new resolvers. As we did earlier, we can check our work by using a branch deployment: ``` $ chalk apply --branch tutorial ✓ Found resolvers ✓ Deployed branch ``` We can then query our new features: ``` $ chalk query --branch tutorial \ --in user.id=1 \ --out user.name_match_score ``` ### CLI Query Now that we've written some features and resolvers and deployed them to Chalk, we're ready to integrate Chalk into our production decisioning systems. As a sanity check, it can be helpful to use the Chalk CLI to query a well-known input and ensure that we get the expected output. We can use the chalk query command, passing in the id of a user, and the names of the features we want to resolve: ``` $ chalk query --in user.id=1 \ --out user.name \ --out user.email \ --out user.account.balance Results user.name "John Doe" email "john@doe.com" user.account.balance 2032.91 ``` ### API Client Query Once we're satisfied that our features and resolvers are working as expected, we can use a client library to query Chalk from our application. In this first example, we'll use the ChalkClient in the chalkpy package to query Chalk from our application: ``` from src.feature_sets import User from chalk.client import ChalkClient # Create a new Chalk client. By default, this will # pick up the login credentials generated after running # `chalk login`. client = ChalkClient() client.query( input=User(id=1234), output=[ User.id, User.name, User.fico_score, User.account.balance, ], ) ``` We use the same feature definitions for querying our data as we used for defining our features and resolvers. Chalk has API client libraries in several languages, including Python, Go, Java, Typescript, and Elixir. ### Code Generation (Optional) All API clients can operate on the string names of features. However, in a production system, you may have many hundreds or thousands of features, and want to avoid hard-coding the names of each feature in your code. To help with this, Chalk can codegen a library of strongly-typed feature names for you. For example, say the service that calls into Chalk is written in Go. We can generate a Go library of feature names with the following command: ``` $ chalk codegen go --out ./clients/go/client.go --package=client ✓ Found resolvers ✓ Wrote features to file './clients/go/client.go' ✓ Please do not change the generated code. ``` This generates a file clients/go/client.go that looks like this: ``` package client /************************************** Code generated by Chalk. DO NOT EDIT. > chalk codegen go --out ./clients/go/client.go --package client **************************************/ import ( "github.com/chalk-ai/chalk-go" "time" ) var InitFeaturesErr error type Account struct { Id *int64 Title *string UserId *int64 Balance *float64 User *User UpdatedAt *time.Time } type User struct { Id *int64 Name *string Email *string Account *Account AccountNameMatch *float64 FicoScore *int64 CreditScoreTags *[]any } var Features struct { Account *Account User *User } func init() { InitFeaturesErr = chalk.InitFeatures(&Features) } ``` We can then use this library to query Chalk: ``` import ( "github.com/chalk-ai/chalk-go" ) // Create a new Chalk client. client := chalk.NewClient() // Create an empty struct to hold the results. user := User{} // Query Chalk, and add the results to the struct. _, err = client.OnlineQuery( chalk.OnlineQueryParams{}. WithInput(Features.User.Id, 1234). WithOutputs( Features.User.Id, Features.User.LastName, Features.User.FicoScore, Features.User.Account.Balance, ), &user, ) // Now, you can access the properties of the // user for which there was a matching `output`. fmt.Println(user.Account.Balance) ``` If your calling service is written in Python, but you don't want to take a dependency on the repository that contains your Chalk features, you can generate your Python features into a separate repository: ``` $ chalk codegen python --out ./clients/python/client.py ``` You can see the generated code in clients/python/client.py. If you are generating Python into a subdirectory of your Chalk project, be sure to add an entry to your .chalkignore containing the directory of your generated code (in the above example, clients/). Otherwise, Chalk will find duplicate definitions of your features. # Tutorial: Feature Versions source: https://docs.chalk.ai/docs/feature-version-tutorial ## Learn how to create and interact with versioned features ### Introduction With Feature Versioning on the Chalk platform, you can manage multiple versions of a feature. In this guide, you'll add versioning to a feature in an existing Chalk project. The following example shows you how to add versions to a feature, manage resolvers for different versions, query versioned features, and generally interact with versioned features. ### Creating your first versioned feature - Create a versioned_feature.py file in your Chalk project. - Copy the following Python contents into the versioned_feature.py file. ``` from chalk.features import feature, features, online @features class Animal: id: str sound: str = feature(version=2) @online def resolver_1(id: Animal.id) -> Animal.sound@1: return "Hello" @online def resolver_2(id: Animal.id) -> Animal.sound@2: match id: case "Cow": return "Moo" case "Dog": return "Woof" case _: return "Hello" ``` - Deploy your new feature with chalk apply --branch test. - Now try a basic query for animal.sound. When no explicit version is provided, and no default_version is set, version 1 will be requested, so this function will return "Hello". ``` $ chalk query --in animal.id=Cow --out animal.sound --branch test ``` - To query a specific version, use the @ syntax ``` $ chalk query --in animal.id=Cow --out animal.sound@2 --branch test ``` ### Changing default versions When you create a versioned feature, the default version is 1. You can control this default version and change the behavior of querying and resolvers. - Replace the contents of versioned_feature.py with the following Python code. Note that we didn't specify @2 in resolver_2. We can do that because 2 is the default version of Animal.sound. ``` from chalk.features import feature, features, online @features class Animal: id: str sound: str = feature(version=2, default_version=2) @online def resolver_1(id: Animal.id) -> Animal.sound@1: return "Hello" @online def resolver_2(id: Animal.id) -> Animal.sound: match id: case "Cow": return "Moo" case "Dog": return "Woof" case _: return "Hello" ``` - chalk apply --branch test to deploy. - Now try querying without a version, and resolver_2 will get executed. ``` $ chalk query --in animal.id=Cow --out animal.sound --branch test ``` ### Using versioned features in resolvers Versioned features can also be used as the inputs for resolvers, which lets you do some clever things! Let's check it out. - Replace the contents of versioned_features.py with this code. ``` from chalk.features import feature, features, online @features class Animal: id: str sound: str = feature(version=2) @online def resolver_1(id: Animal.id) -> Animal.sound@1: return "Hello" @online def resolver_2(id: Animal.id, sound: Animal.sound@1) -> Animal.sound@2: sounds = {"Cow" : "Moo", "Dog": "Woof"} return f"The {id} says {sounds.get(id, 'Hello')}, but its trying to say {sound}" ``` - Now query for an Cow.sound@2 and notice how Chalk automatically generates the dependent Cow.sound@1 feature! ``` $ chalk query --in animal.id=Cow --out animal.sound@2 --branch test ``` # Tutorial: Jupyter Notebook source: https://docs.chalk.ai/docs/notebook-tutorial ## Work through an example using Chalk in a Jupyter notebook Chalk enables data science and machine learning teams to build and deploy feature pipelines for machine learning. For data science workflows, Chalk can be used in entirely in a notebook to iteratively build features and generate training data. In this tutorial, we will use Chalk in a Jupyter notebook to explore a dataset of credit card authorizations and build out some features and resolvers. This tutorial assumes a notebook you run yourself, connected to Chalk. For those setup steps, see External Notebooks and Authentication for External Notebooks. To work in a notebook Chalk hosts instead, see Chalk Notebooks. ### Table of Contents - Configure Data Sources - Defining Features - Defining Resolvers - Computed Features - Troubleshooting - Summary ### Configure data sources The dataset we will be using for this tutorial is stored in tables in our Snowflake warehouse. Chalk has built-in support for a number of SQL-like sources and can ingest data using SQL strings. ### Adding a Snowflake source Chalk provides a native integration with Snowflake as a SQL Source. Before we can start ingesting data, we will initialize a SnowflakeSource. If you're working on an existing project, it's likely that this step has already been done for you. ``` from chalk.sql import SnowflakeSource snowflake = SnowflakeSource() ``` If you have multiple sources, you can initialize a SnowflakeSource by passing in the name of your Snowflake integration which can be defined in the Chalk Dashboard. ``` from chalk.sql import SnowflakeSource snowflake = SnowflakeSource(name="snowflake-integration") ``` ### Defining Features Once we have our data source setup, we can start defining features. We will start by defining the features we will ingest from our data source. In further sections, we will define derived features that we will use to train a fraud classification model. Our dataset consists of the following tables: - cards - merchants - authorizations - cardholders Chalk lets you define your features in Python by decorating classes with the @feature decorator. Chalk lets you define features directly in Python. To create a new FeatureSet, apply the @features decorator to a Python class with typed attributes. In our notebook, we will now define the following feature classes: ``` from datetime import datetime from chalk.features import features, has_many, DataFrame, FeatureTime @features class Merchant: id: int name: str category: str country_code: str @features class Authorization: id: int amount_in_cents: int card_id: int merchant_id: int country_code: str status: str authorized_at: FeatureTime # Relationships card: "Card" @features class Card: id: int cardholder_id: int issued_at: datetime # Relationships authorizations: DataFrame[Authorization] = has_many( lambda: Authorization.card_id == Card.id ) @features class CardHolder: id: int name: str address: str created_at: datetime # Relationships cards: DataFrame[Card] = has_many(lambda: CardHolder.id == Card.cardholder_id) ``` Here we have defined four feature classes: Merchant, Authorization, Card, and CardHolder. In the features sets, we have defined the root features which are the features that are directly fetched from the datasource as well as derived features. In the next section, we will look at how to resolve these features. ### Primary Keys Feature classes in Chalk need to have a unique id field. By default, Chalk will use the id field as the primary key for the feature class. However, if you want to use a different field as the primary key, you can specify it using the Primary argument as shown below. ``` from chalk.features import Primary @features class Merchant: - id: int + merchant_id: Primary[int] name: str category: str country_code: str ``` ### Namespacing Features are namespaced by their containing FeatureSet and by the name of the variable. For example, as defined above Authorization would be the containing FeatureSet and its corresponding features would be named as follows: | Feature Name | Type | | ----------------------------- | ----------- | | authorization.id | Integer | | authorization.amount_in_cents | Integer | | authorization.card_id | Integer | | authorization.merchant_id | Integer | | authorization.country_code | String | | authorization.status | String | | authorization.authorized_at | FeatureTime | ### Relationships We can also define relationships between features using the has_one or has_many functions, where the first argument specifies a function returning how to join the tables. In the feature definitions above, we have defined a one-to-many relationship between Card and Authorization using the has_many function. ``` @features class Authorization: id: int amount_in_cents: int card_id: int merchant_id: int country_code: str status: str authorized_at: FeatureTime # Relationships + # We defined the join condition between `Card` and `Authorization` below, + # we don't need to repeat it here + card: "Card" @features class Card: id: int cardholder_id: int issued_at: datetime # Relationships + # The has-many relationship between `Card` and `Authorization` + # specifying the join condition + authorizations: DataFrame[Authorization] = has_many( + lambda: Authorization.card_id == Card.id + ) ``` ### Feature Types Chalk supports a number of different feature types including scalars, collections, dataclasses, Pydantic models and custom types. For a complete list of features, refer to Feature Types. ### Using feature time By default, our features are timestamped with the execution time of their resolvers. Since we want to be able to run point-in-time correct backfills, we will need to use the FeatureTime type to override the default behavior and explicitly use the authorized_at field. To learn more about how to use FeatureTime, refer to our time documentation. ``` from chalk.features import FeatureTime @features class Authorization: # Root features id: int amount_in_cents: int card_id: int - authorized_at: datetime + authorized_at: FeatureTime # Relationships card: "Card" ``` ### Defining Resolvers Next we will define the resolvers for the features we have defined above. A resolver is a function that defines how features are fetched or derived. To ingest data from Snowflake for the features we defined above, we will define resolvers using SQL strings. Specifically we will use the query_string function on our Snowflake source defined above. It is important to make sure the names of the features we are resolving match the names of the features we defined above. For example, in the resolver definition below, we alias created_at to authorized_at for Authorization ### SQL Resolvers We will use the %%sql_resolver magic to define SQL resolvers in our notebook. ``` %%sql_resolver get_merchant_features -- resolves: Merchant -- source: snowflake SELECT id, name, category, country_code FROM merchant ``` ``` %%sql_resolver get_cardholder_features -- resolves: CardHolder -- source: snowflake SELECT id, name, address, created_at FROM cardholder ``` ``` %%sql_resolver get_authorization_features -- resolves: Authorization -- source: snowflake SELECT id, amount_in_cents, card_id, merchant_id, status, country_code, created_at as authorized_at FROM authorization ``` ``` %%sql_resolver get_card_features -- resolves: Card -- source: snowflake SELECT id, cardholder_id, issued_at FROM card ``` ### Python resolvers Alternatively, you can define resolvers using Python functions using the @offline decorator A note on namespaces Resolvers can take in multiple features as input, however, all feature dependencies in a single resolver must be from the same namespace. Requiring features from the same root namespace ``` @offline def fn( authorization_amount: Authorization.amount_in_cents, card_id: Authorization.card_id, ) -> Authorization.some_feature: return ... ``` Here, we incorrectly request features from the root namespaces of Authorization and Card: Requiring features from different root namespaces ``` @online def fn( authorization_amount: Authorization.amount_in_cents, card_id: Card.id ) -> Authorization.some_feature: return ... ``` ### Computed Features The ChalkClient provides the offline_query method to compute features from the offline store. Initialize a client first, if you haven't already — see Authentication for External Notebooks for the options: ``` from chalk.client import ChalkClient client = ChalkClient() ``` To validate that we are able to resolve features from our offline store, we can run an offline query to resolve the features defined on Merchant. If inputs are given, the query will return rows corresponding to those inputs, otherwise it will return a random sample according to the max_samples parameter. Offline queries return a Dataset instance. Convert it to a Chalk DataFrame with get_data_as_dataframe, or to a Pandas DataFrame with get_data_as_pandas. ``` dataset = client.offline_query( input={ Merchant.id: [1, 2, 3] }, output=[ Merchant.id, Merchant.name, Merchant.category ], recompute_features=True, ).get_data_as_dataframe() ``` We get back the following DataFrame, validating that our resolvers are working as expected ``` ┌───────────────────────┬───────────────────────────┬─────────────────┐ │ merchant.category ┆ merchant.name ┆ merchant.id │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ i64 │ ╞═══════════════════════╪═══════════════════════════╪═════════════════╡ │ Gas Station ┆ Tucker, Hull and Gallegos ┆ 1 │ │ E-commerce ┆ Silva-Odonnell ┆ 2 │ │ Grocery ┆ Taylor-Davis ┆ 3 │ └───────────────────────┴───────────────────────────┴─────────────────┘ ``` Note that we specified the recompute_features parameter to True to ensure that the features are recomputed by the resolvers. When set to False, output features are sampled from the offline store. ### Feature Definitions Let's expand on the feature classes we have defined and add the following computed features: ``` @features class Authorization: id: int amount_in_cents: int card_id: int merchant_id: int country_code: str status: str authorized_at: FeatureTime + # The authorization amount (in cents) of the previous transaction + previous_auth_amount_in_cents: int # Relationships card: "Card" @features class Card: id: int cardholder_id: int issued_at: datetime + # The total number of transactions + count_transactions_total: int + + # The total number of transactions in the last 7 days + count_transactions_7d: int + + # The number of days since the card was created + days_since_card_created: int + + # Days since first transaction + days_since_first_transaction: int + + # Days since last transaction + days_since_last_transaction: int # Relationships authorizations: DataFrame[Authorization] = has_many( lambda: Authorization.card_id == Card.id ) ``` Next, we will define resolvers for these features. ``` %%sql_resolver get_prev_auth_amounts -- resolves: Authorization -- source: snowflake WITH ordered AS ( SELECT id, card_id, amount_in_cents, LAG(amount_in_cents, 1) OVER (PARTITION BY card_id ORDER BY created_at) AS previous_auth_amount_in_cents FROM AUTHORIZATION ) SELECT id, previous_auth_amount_in_cents FROM ordered WHERE previous_auth_amount_in_cents IS NOT NULL ``` ``` from chalk import offline from chalk.features import after @offline def get_count_all_txns( txns: Card.authorizations[Authorization.id], ) -> Card.count_transactions_total: return txns.count() @offline def get_count_7d_txns( txns: Card.authorizations[Authorization.id, after(days_ago=7)] ) -> Card.count_transactions_7d: return txns.count() ``` ### Projections & Filters Chalk has support for time windows using the before and after functions. In the resolvers above, we use the after operator to filter the transactions by the created_at field. Additionally, since we don't need to resolve all the features on Authorization to compute the counts, we can specify the features we need for this resolver using a projection. Projections allow us to scope down a DataFrame to only include the features we need. In this instance, we are using projections to only fetch the id field from Authorization. In the resolvers above, we have combined filtering with a projection on the authorizations DataFrame. Refer to the section on composing projections and filters for more details. ``` from datetime import datetime from chalk import Features, Now, offline @offline def get_days_since_card_created( card_id: Card.id, issued_at: Card.issued_at, now: Now, ) -> Card.days_since_card_created: return (now - issued_at).days @offline def get_days_since_first_last_txn( txns: Card.authorizations[Authorization.created_at], now: Now, ) -> Features[ Card.days_since_first_transaction, Card.days_since_last_transaction, ]: # Sort transactions by created_at sorted_txns = txns.sort(by=Authorization.created_at, descending=False) # Get first and last transaction dates first_txn_date = sorted_txns.first(col=Authorization.created_at) last_txn_date = sorted_txns.last(col=Authorization.created_at) return Card( days_since_first_transaction=(now - first_txn_date).days, days_since_last_transaction=(now - last_txn_date).days, ) ``` ### Time-dependent resolvers In the resolvers above, we made use of the Chalk feature Now which allows us to express time-dependency in our resolvers. This is useful for performing backfills which compute values that depend on values that are semantically similar to datetime.now(). In online queries, Now represents datetime.now(). In offline queries, we can use the input_times parameter to specify the times Now should resolve to allowing us to run backfills for many different historical points in time. ``` from datetime import datetime, timedelta # Example of running an offline query for multiple historical points in time client.offline_query( input={Card.id: [1, 1, 1]}, output=[Card.days_since_card_created], input_times=[ datetime.now(), datetime.now() - timedelta(days=10), datetime.now() - timedelta(days=50), ], ) # Output: # ┌─────────┬──────────────────────────────┐ # │ card.id ┆ card.days_since_card_created │ # │ --- ┆ --- │ # ╞═════════╪══════════════════════════════╡ # │ 1 ┆ 90 │ # │ 1 ┆ 80 │ # │ 1 ┆ 40 │ # └─────────┴──────────────────────────────┘ ``` ### Troubleshooting Some queries that involve multiple operations might need additional tracking. Users can supply store_plan_stages=True to store intermediate outputs at all operations of the query. This will dramatically slow things down, so use wisely! These results are visible in the dashboard under the "Queries" page as shown below. ### Query Plan The Query Plan shows the operations that were executed to compute the query as well as the intermediate results at each stage. The numbers on the edges represent the number of rows of data that were passed from one stage to the next. Chalk Query Plan ### Intermediate Results You can examine the intermediate results at each stage of the query plan by clicking on a specific stage and download the results as a parquet file. Chalk Query Intermediate Results ### Summary In this tutorial, we learned how to use Chalk in a notebook to define features, resolvers and run offline queries. To dive deeper into Chalk, check out our documentation on the topics listed below - Chalk Notebooks - External Notebooks - DataFrame - Offline Queries - SQL Integrations - Backfills - Temporal Consistency # Tutorial: AWS SageMaker with Chalk source: https://docs.chalk.ai/docs/sagemaker-tutorial ## Build a Chalk feature pipeline for training and serving models with AWS SageMaker. ### Introduction Chalk enables you to build feature pipelines for machine learning models. AWS SageMaker is a popular service for building, training, and deploying models. Chalk and SageMaker fit together naturally to create a streamlined machine learning architecture that uses the best aspects of each system. ### Why use Chalk? There are several advantages to using Chalk in your ML architecture alongside SageMaker. - Use the same Python codebase for notebook iteration, training, and serving. Chalk enables you to maintain a single Python codebase that integrates with many data sources, including Postgres, Databricks, Snowflake, Kafka, and others. As you tweak your feature definitions, you can deploy your updates to separate branches to try your new features without needing to wait for a long build step. - Generate features when they're needed. Chalk computes features in response to queries. Other systems such as SageMaker Feature Store require you to precompute features from batch data. We believe the precomputation style is limiting because this style prevents users from relying on real-time data. For example, precomputation prevents users from writing features reliant on recency (such as the number of failed logins in the past minute) or third-party APIs (which may be costly to exhaustively pre-fetch). ### What you'll learn In this tutorial, we'll take a deep dive into how you can use Chalk to serve features combined with SageMaker for model training and serving. You will: - Set up your feature pipeline with Chalk:Connect your data sources (in this tutorial, DynamoDB)Write feature definitions in PythonWrite resolvers defining how to generate those features from your data source - Set up your SageMaker pipeline:Run Chalk offline queries and prepare datasets for training and evaluationTrain your modelEvaluate your model - Deploy your model with SageMaker We'll go through each of these steps using fraud detection as our example use case. The full code for this tutorial can be found in GitHub. ### Set up Chalk In this section, we'll set up your Chalk feature pipeline. ### Connect your data source Chalk integrates with AWS, GCP, various SQL databases, Databricks, and more. For this tutorial, we'll pull transaction data from AWS DynamoDB, but you can add as many data sources as you need. To connect DynamoDB, enter your AWS access key into the Chalk dashboard. After adding AWS, define your DynamoDB data source in Python and give it a name: ``` from chalk.sql import DynamoDBSource DynamoDBSource(name="our_dynamo") ``` ### Define features Features are structured data you can use as inputs to your model. With Chalk, you define your features in one place as Pydantic-style Python classes: ``` from datetime import date from chalk import DataFrame, FeatureTime from chalk.features import features, feature @features class Transaction: id: int amount: float confirmed_fraud: bool # Use strings to reference Customer because Customer is defined beneath this class customer_id: "Customer.id" customer: "Customer" transacted_at: FeatureTime @features class Customer: id: int name: str email: str dob: date income: int # :tags: pii ssn: str # Set max_staleness on fico because fico relies on a third-party API call. # We will read this value from a cache (as long as it was computed # sometime in the last day). fico: int = feature(max_staleness="1d") # The transactions, linked by the Customer.id type on the Transaction.customer_id field transactions: DataFrame[Transaction] ``` In this code, Transaction is the feature class and each of its properties is a feature. ### Define resolvers Resolvers are Chalk's way of defining how to retrieve data and convert them into features. Most of our features will be loaded from DynamoDB using SQL file resolvers. Here's how we retrieve basic Transaction and Customer data: ``` -- type: online -- resolves: transaction -- source: our_dynamo SELECT id, amount, confirmed_fraud, customer_id, created_at AS transacted_at FROM txns; ``` ``` -- type: online -- resolves: customer -- source: our_dynamo SELECT id, name, email, dob, income, ssn FROM customers; ``` Each column in the result must match the name of a feature. Use AS to rename columns as needed. For features requiring computation, we use Python resolvers. Here's a resolver for retrieving a customer's FICO score from an API: ``` from chalk import online @online def get_fico(name: Customer.name, ssn: Customer.ssn) -> Customer.fico: # Replace with your preferred provider. fico_score = FICOApi().get_score(name, ssn) return Customer(fico=fico_score) ``` This resolver depends on Customer.name and Customer.ssn and returns Customer.fico. When you deploy, Chalk uses your resolver inputs and outputs to create a dependency graph to confirm your features can be generated safely. The get_fico resolver will be invoked in response to Chalk queries. Resolvers do not run exhaustively over your customer dataset, which helps you reduce costs related to unnecessary computation and API usage. Our resolver documentation goes into greater detail about how you can schedule resolvers and set up batch offline jobs. We also set max_staleness to 1 day on the fico feature, so this feature will be read from the online store's cache if the value has already been computed in the past day. With a combination of Python and SQL resolvers, we have now told Chalk how to retrieve each of our features. ### Deploy Chalk At this point, you can deploy Chalk to a local branch and start writing queries. To deploy to a non-production branch, pass --branch to chalk apply: ``` $ chalk apply --branch sagemaker_tutorial ✓ Found resolvers ✓ Deployed branch ``` You can run ad hoc queries on this branch from the CLI: ``` $ chalk query --branch sagemaker_tutorial \ --in transaction.id=1 \ --out transaction.confirmed_fraud ``` Once you're satisfied with your feature pipeline, you can move on to setting up SageMaker. ### Set up SageMaker There are many ways to set up SageMaker. In this tutorial, we'll use SageMaker steps to compose our model training pipeline. Steps allow us to modularize the code and we find them easier to work with than some alternatives, such as running the pipeline directly from one super long Jupyter notebook. In this section, we'll write steps for creating the dataset, training a candidate model, and evaluating the model. The steps use the sagemaker.workflow module, which SageMaker Python SDK v3 removed. Install a 2.x release of the SDK: ``` $ pip install "sagemaker<3" ``` ### Create the dataset In this step, we'll use Chalk to generate a random sample of features from our dataset, split the dataset for training and testing using scikit-learn, write the results to S3, and return the S3 paths to be used in the next step. Writing the split datasets to S3 provides a clear history of the data you used to train and evaluate your model. Additionally, it allows you re-run later steps in your pipeline without re-executing the dataset generation step. In the previous section, we set up our features and resolvers without going into great detail about how the code would be executed. Here, we'll finally show how the code gets executed through a Chalk offline query: ``` from sagemaker.workflow.function_step import step from src.feature_sets import Transaction @step(name="create_dataset") def create_dataset( test_size: float, run_bucket: str, model_name: str ) -> tuple[str, str, str, str]: from chalk.client import ChalkClient from sklearn.model_selection import train_test_split + client = ChalkClient(branch="sagemaker_tutorial") + + chalk_dataset = ChalkClient().offline_query( + recompute_features=True, + output=[ + Transaction.amount, + Transaction.customer.fico, + # ... and more features + ], + # By giving this dataset a name, we will be able to view it in the Chalk + # dashboard, download it locally, or share it with collaborators. + dataset_name=model_name, + ) + + # Converting a chalk_dataset to a pandas (or polars) dataframe is + # efficient because both are backed by Apache Arrow. + dataset = chalk_dataset.to_pandas() # Create training data by removing the target column. X = dataset.drop(columns=[Transaction.confirmed_fraud]) # Pull target from dataset. y = dataset[Transaction.confirmed_fraud] # Split data into train and test set. We will use k-fold cross-validation # for hyperparameter tuning X_train, X_test, y_train, y_test = train_test_split( X, y,test_size=test_size ) xtrain_path = f"{run_bucket}/input/X_train.parquet" xtest_path = f"{run_bucket}/input/X_test.parquet" ytrain_path = f"{run_bucket}/input/y_train.parquet" ytest_path = f"{run_bucket}/input/y_test.parquet" dataset.to_parquet(f"{run_bucket}/raw_data/data.parquet") X_train.to_parquet(xtrain_path) y_train.to_parquet(ytrain_path) X_test.to_parquet(xtest_path) y_test.to_parquet(ytest_path) return xtrain_path, xtest_path, ytrain_path, ytest_path ``` The full code for this step can be found in steps/dataset.py. ### Train and evaluate Our last two steps will retrieve our features from S3 and train and evaluate our model. These steps don't depend on Chalk or any other data sources, so you can rapidly iterate here without being blocked on feature engineering if your training data does not require changes. In steps/training.py, we load the training data from S3 and train a model using scikit-learn: ``` from sagemaker.workflow.function_step import step PARAM_GRID = { 'xgb__n_estimators': [20, 50, 100, 200], 'xgb__learning_rate': [0.01, 0.1, 0.2], 'xgb__max_depth': [3, 5, 7, 9], } @step( name="model-training", instance_type="ml.m5.xlarge", keep_alive_period_in_seconds=300, ) def train( xtrain_path: str, ytrain_path: str, num_rounds: int ): from sklearn.pipeline import Pipeline import pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.impute import SimpleImputer from sklearn.ensemble import GradientBoostingClassifier from sklearn.model_selection import RandomizedSearchCV # Read data files from S3. X_train = pd.read_parquet(xtrain_path) y_train = pd.read_parquet(ytrain_path) pipeline = Pipeline( steps=[ ("impute", (SimpleImputer())), ("scaler", StandardScaler()), ("xgb", GradientBoostingClassifier()), ] ) # Note: Fraud detection commonly experiences the imbalanced learning problem. # In other words, fraud may be over or underrepresented in the dataset, leading # to unreliable results in production. When training a model for production, # the imbalance should be addressed, for example, by upsampling, downsampling, # or model choice. rsc = RandomizedSearchCV( pipeline, param_distributions=PARAM_GRID, n_iter=num_rounds, cv=3, scoring="f1", n_jobs=-1, ) rsc.fit(X_train, y_train) return rsc.best_estimator_ ``` In steps/evaluate.py, we load our newly trained model and evaluate it against our test dataset: ``` from sagemaker.workflow.function_step import step @step( name="model-evaluation", instance_type='ml.t3.medium', keep_alive_period_in_seconds=300, ) def evaluate(model, xtest_path: str, ytest_path: str, run_bucket: str) -> str: import pandas as pd from sklearn.metrics import ( accuracy_score, f1_score, precision_score, recall_score, ) import s3fs import json X_test = pd.read_parquet(xtest_path) y_test = pd.read_parquet(ytest_path) predictions = model.predict(X_test) results = { "accuracy": accuracy_score(y_test, predictions), "f1": f1_score(y_test, predictions), "precision": precision_score(y_test, predictions), "recall": recall_score(y_test, predictions), } # Upload evaluation report to s3 s3_fs = s3fs.S3FileSystem() eval_src_s3 = f"{run_bucket}/evaluation/evaluation.json" with s3_fs.open(eval_src_s3, "wb") as file: file.write(json.dumps(results)) return eval_src_s3 ``` ### Assemble the steps into a SageMaker pipeline Finally, let's assemble our three steps into a SageMaker pipeline in chalk_sagemaker_pipeline.py: ``` from sagemaker.workflow.pipeline import Pipeline from steps.dataset import create_dataset from steps.training import train from steps.evaluate import evaluate from sagemaker.workflow.parameters import ( ParameterInteger, ParameterString, ParameterFloat, ) from uuid import uuid4 BUCKET_PREFIX = "s3://chalk-sagemaker-models/" if __name__ == "__main__": # Create Run Parameters model_package_group = "chalk-sagemaker-xgb" run_bucket = f"s3://chalk-sagemaker-models/{model_package_group}/{uuid4()}/" # Required F1 Threshold for model registration f1_threshold = ParameterFloat(name="F1Threshold", default_value=0.8) # Size of test split test_size = ParameterFloat(name="TestSize", default_value=0.2) # Number of estimators to evaluate num_rounds = ParameterInteger(name="NumRounds", default_value=50) run_bucket = ParameterString(name="RunBucket", default_value=run_bucket) model_package_group = ParameterString(name="ModelPackageGroup", default_value="chalk-sagemaker-xgb") + delayed_data = create_dataset(test_size=test_size, run_bucket=run_bucket) + delayed_model = train(xtrain_path=delayed_data[0], ytrain_path=delayed_data[2], num_rounds=num_rounds) + delayed_evaluation = evaluate(model=delayed_model, xtest_path=delayed_data[1], ytest_path=delayed_data[3], run_bucket=run_bucket) + + pipeline = Pipeline( + name="ChalkaiSagemakerPipeline", + steps=[delayed_evaluation], + parameters=[ + f1_threshold, + test_size, + run_bucket, + model_package_group, + num_rounds, + ] + ) ``` ### Deploy your model with SageMaker There are a number of different ways to deploy the model that you trained on SageMaker. We recommend deploying your model to an endpoint, which is documented here. ### Efficient SageMaker inference with Chalk underscores Once you've defined your SageMaker endpoint, you can run predictions against SageMaker from Chalk. This can be accomplished by encoding your input features and then using the F.sagemaker_predict underscore. function. You can specify your target SageMaker endpoint in the underscore function's parameters. ``` import chalk.functions as F @features class Transaction: id: int amount: float confirmed_fraud: bool # Use strings to reference Customer because Customer is defined beneath this class customer_id: "Customer.id" customer: "Customer" transacted_at: FeatureTime encoded_features: str transaction_fraud_prediction_raw: bytes = F.sagemaker_predict( F.string_to_bytes(_.encoded_features), endpoint="chalk-sagemaker-xgb-endpoint", ) @online def get_encoded_features(amount: Transaction.amount, customer_fico: Transaction.customer.fico, ...) -> Transaction.encoded_features: return f"{amount},{customer_fico},..." ``` ### Conclusion In this tutorial, you got a look at how Chalk accelerates your feature engineering workflows by making it easy to connect your data sources and unifying your feature code into a single Python codebase. Chalk fits naturally into a SageMaker step for dataset generation and hands feature data off to the rest of your SageMaker pipeline for model training, evaluation, and serving. # End-to-End Model Tutorial source: https://docs.chalk.ai/docs/model-tutorial ## Build a complete ML pipeline from features to inference This tutorial walks through a complete machine learning workflow in Chalk: defining features, connecting to Snowflake data sources, creating point-in-time correct training datasets, training a model, registering it, and deploying it for inference. We'll build an event engagement prediction model that predicts whether a user is likely to engage with a notification. ### Step 1: Define Your Features First, define feature classes that represent the entities in your domain. Feature classes use the @features decorator and define typed attributes for each feature. ``` from datetime import datetime from chalk import windowed, Windowed from chalk.features import features, DataFrame, _ import chalk.functions as F @features class User: id: int # Account features email: str signup_date: datetime account_type: str # one of: "free", "basic", "premium" # Windowed activity features - automatically creates versions for each window total_logins: Windowed[int] = windowed( "1d", "7d", "30d", expression=_.events[ _.type == "login", _.ts > _.chalk_window, _.ts <= _.chalk_now ].count(), default=0 ) first_login_unix_seconds: Windowed[int | None] = windowed( "1d", "7d", "30d", "all", expression=_.events[ _.ts_unix_seconds, _.type == "login", _.ts > _.chalk_window, _.ts <= _.chalk_now ].min(), default=None ) # Non-windowed activity feature days_since_last_login: int | None = ( (F.unix_seconds(_.chalk_now) - _.first_login_unix_seconds["all"]) / 86400 ) # Windowed billing features monthly_spend: float # Computed features account_age_days: int engagement_score: float # Has-many relationship to events events: "DataFrame[Event]" # Timestamp for feature time ts: datetime @features class Event: id: int user_id: User.id user: User type: str ts: datetime ts_unix_seconds: int = F.unix_seconds(_.ts) event_clicked: "EventClicked | None" event_was_clicked: bool = F.is_not_null(_.event_clicked.ts) @features class EventClicked: id: Event.id ts: datetime ``` The windowed() function creates multiple versions of a feature, one for each time window. You can reference specific windows using bracket notation: - User.total_logins["1d"] - logins in the last day - User.total_logins["7d"] - logins in the last 7 days - User.total_logins["30d"] - logins in the last 30 days ### Step 2: Configure Snowflake Data Sources Connect Chalk to your Snowflake data warehouse. First, configure the integration in the Chalk dashboard, then reference it in your code. ### Dashboard Configuration In the Chalk dashboard, navigate to Integrations > Data Sources and add a Snowflake integration with your credentials (account, warehouse, database, schema, role). ### Define Sources in Code ``` from chalk.sql import SnowflakeSource # Reference the integration configured in the dashboard snowflake = SnowflakeSource(name="sf") ``` ### Create SQL File Resolvers Create SQL files that map your Snowflake tables to Chalk features. ``` -- type: offline -- resolves: User -- source: sf SELECT id, email, signup_date, account_type, monthly_spend FROM users ``` ``` -- type: offline -- resolves: Event -- source: sf SELECT id, user_id, ts, type FROM events ``` ``` -- type: offline -- resolves: EventClicked -- source: sf SELECT id, ts FROM events_clicked ``` ### Create Python Resolvers Create python resolvers to compute some simple derived features: ``` from datetime import datetime from chalk import online, Now from src.features import User @online def compute_account_age( signup_date: User.signup_date, now: Now ) -> User.account_age_days: return (now - signup_date).days @online def compute_engagement_score( logins: User.total_logins["30d"], # Reference 30-day window days_inactive: User.days_since_last_login, ) -> User.engagement_score: # Simple engagement formula using 30-day windowed features activity_score = min(logins / 30, 1.0) * 0.6 recency_score = max(0, 1 - (days_inactive / 30)) * 0.4 score = (activity_score + recency_score) * 100 return score ``` ### Step 3: Create a Point-in-Time Correct Training Dataset Use offline_query to create a training dataset with point-in-time correctness. This ensures that feature values reflect what was known at each historical point, preventing data leakage. ``` from datetime import datetime, timedelta from chalk.client import ChalkClient from src.features import Event client = ChalkClient() # Create the dataset with point-in-time correctness # Use bracket notation to select specific windows for windowed features dataset = client.offline_query( input_sql=""" SELECT id as "event.id", ts as "__ts__" FROM "sf.prod.events" WHERE type='recommendation'; """, output=[ Event.id, Event.user.id, Event.user.account_type, Event.user.total_logins["1d"], # 1-day window Event.user.total_logins["7d"], # 7-day window Event.user.total_logins["30d"], # 30-day window Event.user.days_since_last_login, Event.user.monthly_spend, Event.user.account_age_days, Event.user.engagement_score, Event.event_was_clicked ], dataset_name="event_recommendation_dataset", run_asynchronously=True, # For large datasets ) # Retrieve the data as a pandas DataFrame df = dataset.get_data_as_pandas() print(f"Dataset shape: {df.shape}") print(df.head()) ``` ### Step 4: Train Your Model With your point-in-time correct dataset, train a model and save it as a pickle. The pickle will be uploaded to a volume and loaded by the handler at container startup. ``` import joblib from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import precision_score, recall_score, roc_auc_score from sklearn.model_selection import train_test_split # Column names from the offline_query dataset in Step 3 feature_columns = [ 'event.user.total_logins__30d__', 'event.user.days_since_last_login', 'event.user.monthly_spend', 'event.user.account_age_days', 'event.user.engagement_score', ] # Use the dataset from Step 3 X = df[feature_columns] y = df["event.event_was_clicked"] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y, ) model = GradientBoostingClassifier( n_estimators=100, max_depth=5, learning_rate=0.1, random_state=42, ) model.fit(X_train, y_train) y_pred_proba = model.predict_proba(X_test)[:, 1] y_pred = model.predict(X_test) metrics = { "auc_roc": float(roc_auc_score(y_test, y_pred_proba)), "precision": float(precision_score(y_test, y_pred)), "recall": float(recall_score(y_test, y_pred)), } print(f"Model metrics: {metrics}") joblib.dump(model, "model.pkl") ``` Next, create a handler that loads the pickle on startup and runs inference. The handler receives input features as PyArrow arrays and returns a PyArrow array of predictions. ``` import joblib import numpy as np import pyarrow as pa input_features = [ "login_count_30d", "days_since_last_login", "monthly_spend", "account_age_days", "engagement_score", ] model = None def on_startup(): global model model = joblib.load("/app/artifacts/model.pkl") def handler(event: dict[str, pa.Array], context: dict) -> pa.Array: X = np.column_stack([event[name].to_numpy() for name in input_features]) probabilities = model.predict_proba(X)[:, 1] return pa.array(probabilities, type=pa.float64()) ``` The handler's on_startup function runs once when the container starts, loading the pickle so the model is ready before any inference requests arrive. ### Step 5: Register the Model Build a container image with the handler, register the model version, and upload the trained pickle to a volume. ``` import os import pyarrow as pa import chalkcompute from chalk.client import ChalkClient from chalk.client.model_image import chalk_handler_volume_name, upload_model_to_volume input_features = [ "login_count_30d", "days_since_last_login", "monthly_spend", "account_age_days", "engagement_score", ] DIR = os.path.dirname(__file__) client = ChalkClient() image = ( chalkcompute.Image.debian_slim("3.11") .pip_install([ "chalk-remote-call-python", "pyarrow", "scikit-learn>=1.4", "numpy>=1.26", "joblib", ]) .add_local_file(os.path.join(DIR, "src/handler.py"), "/app/handler.py", strategy="copy") .env({"PYTHONPATH": "/app"}) .workdir("/app") .entrypoint([ "chalk-remote-call", "--handler", "handler.handler", "--on-startup", "handler.on_startup", "--port", "8080", ]) ) result = client.register_model_version( name="interaction-model", input_schema={name: pa.float64() for name in input_features}, output_schema={"interaction_probability": pa.float64()}, model_image=image, ) print(f"Registered model: {result.model_name} v{result.model_version}") volume_name = chalk_handler_volume_name("interaction-model", result.model_version) upload_model_to_volume( volume_name=volume_name, model_filename="model.pkl", model_file_path=os.path.join(DIR, "model.pkl"), chalk_client=client, ) print(f"Uploaded model pickle to volume: {volume_name}") ``` ### Step 6: Deploy the Model for Inference Create a named deployment for the registered model version. The call waits for readiness and returns a ModelDeployment you can inspect or invoke. ``` from chalk.client import ChalkClient from chalk.scalinggroup import AutoScalingSpec, ScalingGroupResourceRequest client = ChalkClient() # Use the version returned from registration, or look it up model_version = 1 deployment = client.create_model_deployment( name="interaction-model-sg", model_name="interaction-model", model_version=model_version, handler="handler.handler", scaling=AutoScalingSpec( min_replicas=1, max_replicas=2, target_cpu_utilization_percentage=70, ), resources=ScalingGroupResourceRequest(cpu="1000m", memory="1Gi"), ) print(f"Deployment {deployment.name} is {deployment.status}: {deployment.web_url}") ``` Creation waits up to 300 seconds by default. For update, rollback, and deletion examples, see Managing deployments. ### Step 7: Wire Up Predictions with catalog_call Add an interaction_probability feature to your Event class that calls your deployed model using F.catalog_call. The first argument is model., and the remaining arguments are the input features in the same order as your input_schema. ``` from chalk.features import features, _ import chalk.functions as F @features class Event: # ... existing features ... interaction_probability: float = F.catalog_call( "model.interaction-model-sg", _.user.total_logins["30d"], _.user.days_since_last_login, _.user.monthly_spend, _.user.account_age_days, _.user.engagement_score, ) ``` If your model's inference is long-running (for example, LLM generation), use F.catalog_call_async instead — it takes the same arguments and produces the same result, but the engine enqueues the call and polls for the result rather than holding a blocking connection while the model works. Deploy to make the feature available: ``` chalk apply ``` Now you can query predictions through the Chalk API: To get good performance, you'll want to add an online source for your features (like a postgres or a stream). ``` from chalk.client import ChalkClient from src.features import Event client = ChalkClient() result = client.query( input={Event.id: 12345}, output=[Event.interaction_probability], ) print(f"Interaction probability: {result.get_feature_value(Event.interaction_probability)}") ``` ### Next Steps - Monitor your model: View model performance metrics and feature distributions in the Chalk dashboard - A/B test models: Use feature versions to run multiple model versions simultaneously - Retrain on schedule: Set up periodic retraining using scheduled queries to refresh your training data - Add more features: Expand your feature set with windowed aggregations and has-many relationships For more details on specific topics: - Model Registry - Managing model versions and aliases - Datasets - Working with offline query results - Temporal Consistency - Understanding point-in-time correctness - Snowflake Integration - Configuring Snowflake data sources # ChalkDF Static Resolvers Tutorial source: https://docs.chalk.ai/docs/chalkdf/static-resolvers ## Build vectorized batch feature computations using chalkdf and static resolvers This tutorial shows you how to use chalkdf with Chalk's static=True resolver pattern to compute batch features across multiple entities in a single, vectorized pass. ### When to Use Static Resolvers A static DF resolver, defined with static=True, receives a batch of entities as a DataFrame and returns a DataFrame of results. Static resolvers are the right choice when: - Your feature logic naturally operates over a collection — like aggregating a has-many relationship which may have more complex steps than default aggregations - Your computation requires cross-row logic such as self-joins or ranking ### Step 1: Define Your Features Define feature classes for User and Transaction. User has a has-many relationship to Transaction, and two output features that the static resolver will populate. ``` from chalk.features import features, DataFrame @features class Transaction: id: int user_id: "User.id" amount: float @features class User: id: int email: str # Has-many relationship to transactions transactions: DataFrame[Transaction] # Output features computed by the static resolver transaction_count: int total_spend: float ``` ### Step 2: Write the Static Resolver The input type annotation declares every feature the resolver needs — including the has-many relationship projected down to the specific columns it uses. Note that the primary key of the feature class must be in the both the input and output type annotations. ``` from chalk import online, DataFrame from chalk.features import _ import chalk.functions as F from src.features import User, Transaction @online(static=True) def compute_transaction_stats( df: DataFrame[ User.id, User.transactions[ Transaction.id, Transaction.user_id, Transaction.amount, ], ], ) -> DataFrame[User.id, User.transaction_count, User.total_spend]: # Explode the has-many relationship into one row per transaction df_exploded = df.explode(str(User.transactions)) # Lift the nested amount field into a top-level column txns = df_exploded.with_columns({ "amount": _.transactions.amount, }) # Aggregate per user stats = txns.agg( [str(User.id)], _.count().alias("txn_count"), _.amount.sum().alias("spend_total"), ) # Join back to the original df so users with no transactions still appear return ( df.join(stats, on=[str(User.id)], how="left") .with_columns({ "user.transaction_count": F.coalesce(_.txn_count, 0), "user.total_spend": F.coalesce(_.spend_total, 0.0), }) .select(str(User.id), "user.transaction_count", "user.total_spend") ) ``` A few things worth noting: - df.explode(): Flattens the has-many list into individual rows — one per transaction. - with_columns: Lifts a field out of the nested struct into a plain top-level column so it can be used in expressions. - agg: Groups by user and computes the count and sum in one pass. - Join back to df: Ensures every user appears in the output. F.coalesce fills in 0 for users with no transactions. ### Step 3: Unit Test the Resolver chalkdf ships a Testing class for asserting equality between DataFrames. Because static resolvers take and return plain DataFrame objects, you can test them directly — no network calls or running Chalk environment required. ``` from chalkdf import DataFrame, Testing from src.resolvers import compute_transaction_stats def test_compute_transaction_stats(): transactions_data = [ {"transaction.id": 1, "transaction.user_id": 1, "transaction.amount": 25.00}, {"transaction.id": 2, "transaction.user_id": 1, "transaction.amount": 50.00}, {"transaction.id": 3, "transaction.user_id": 1, "transaction.amount": 75.00}, ] input_df = DataFrame({ "user.id": [1], "user.transactions": [transactions_data], }) result_df = compute_transaction_stats(input_df) expected = DataFrame({ "user.id": [1], "user.transaction_count": [3], "user.total_spend": [150.0], }) Testing.assert_frame_equal(result_df, expected, check_row_order=False) ``` A few details on constructing the input: - Column names use the full feature path as a string: "user.id", "user.transactions". - The has-many column is a list-of-dicts, where each dict also uses feature-path keys: "transaction.id", "transaction.amount", etc. - check_row_order=False makes the assertion order-independent. Install chalkdf locally with pip install "chalkdf[chalkpy]" to run tests without deploying. Run the test with: ``` pytest tests/test_resolvers.py -v ``` ### Ingesting Features from S3 Parquet Files Beyond computing features on-demand, you can use static resolvers to bulk-ingest historical feature data from parquet files in S3. This is the right pattern when you have existing data — warehouse exports, data lake snapshots, third-party feeds — that you want to make available for training set generation without recomputing it on every query. ### S3 Ingestion Resolver Use ChalkDF.scan() to lazily read one or more parquet files, then .select() to rename raw columns to Chalk feature paths. Reusing the same Transaction feature class from above: ``` from chalk import offline, DataFrame from chalkdf import DataFrame as ChalkDF from chalk.features import _ from src.features import Transaction S3_PATH = "s3://my-bucket/data/transactions/*.parquet" @offline(static=True) def ingest_transactions() -> DataFrame[ Transaction.id, Transaction.user_id, Transaction.amount, ]: return ( ChalkDF.scan([S3_PATH]) .select( _.txn_id.alias("transaction.id"), _.uid.alias("transaction.user_id"), _.amt.alias("transaction.amount"), ) ) ``` A few things to note: - ChalkDF.scan(): Accepts a list of S3 URIs — glob patterns like *.parquet are supported. Files are read lazily; only the columns referenced in .select() are fetched. - .alias(): Maps each raw column name to its Chalk feature path (e.g. "transaction.amount"). The alias must match the dotted feature path exactly. - Additional transforms: You can chain any ChalkDF operations between .scan() and the final return — filters, type casts, derived columns — before handing the returning the resulting features. ### Next Steps - Explore chalkdf operations: Learn about filter, join, agg, project, and more in the chalkdf getting started guide - Build declarative time-window features: See windowed aggregations for an alternative approach when you don't need cross-row logic - Understand has-many relationships: Read the has-many guide for details on modeling one-to-many relationships between feature classes - Add expressions: Combine static resolvers with Chalk expressions for lightweight derived features that build on your computed values # Chalk SQL Statements source: https://docs.chalk.ai/docs/chalksql/statements/overview ## The SQL statements supported by Chalk SQL. Chalk SQL supports a focused set of SQL statements built around reading and introspecting data across your Chalk environment. Each page in this section documents one statement in detail, including its clauses, restrictions, and examples. | Statement | Description | | --------------------------------------------------------- | ---------------------------------------------------------------------------- | | [`SELECT`](/docs/chalksql/statements/select) | Query data from data sources, the offline store, and the Chalk catalog | | [`DESCRIBE` / `SHOW`](/docs/chalksql/statements/describe) | Inspect the schema of a table or query, or list available tables | | [`EXPLAIN`](/docs/chalksql/statements/explain) | Show the physical plan for a statement, optionally with execution statistics | | [`COPY ... TO`](/docs/chalksql/statements/copy) | Write the result of a query to Parquet, CSV, or Iceberg files | The Chalk SQL function reference documents the scalar functions available in Chalk SQL expressions. For the complete inventory, including aggregate and table-valued functions, query "chalk.information_schema.functions", whose is_aggregate and is_table_valued columns say which kind each name is. ### General rules - One statement per query. Each Chalk SQL request executes exactly one statement. Submitting multiple ;-separated statements returns an error. - Read-only. Chalk SQL is a read-only interface over the Chalk catalog and your configured data sources. The one statement that produces output outside of the query response is COPY ... TO, which exports query results to files without modifying any source. - Supported syntax. Each statement page documents the syntax and restrictions that Chalk SQL supports. For expression syntax, such as literals, struct and list access, CASE, and casts, see Expressions. For named functions, see the Chalk SQL function reference. - Fully qualify catalog tables. Tables in the Chalk catalog and in configured data sources are referenced by a quoted, fully-qualified "catalog.schema.table" name, for example SELECT * FROM "chalk.online_store.keys". See Referencing the Chalk Catalog. # SELECT Statement source: https://docs.chalk.ai/docs/chalksql/statements/select ## Retrieve rows from tables across your Chalk environment. The SELECT statement retrieves rows from tables in the Chalk catalog, from configured data sources, and from table-valued functions. It is the primary statement in Chalk SQL: every query reads data through a SELECT (including the query embedded in COPY ... TO and EXPLAIN). ### Examples ``` -- Select all columns from a data source table SELECT * FROM "my_postgres.public.users"; -- Aggregate transactions per merchant SELECT merchant_id, count(*) AS txn_count, sum(amount) AS total FROM "my_bigquery.my_dataset.transactions" GROUP BY merchant_id ORDER BY total DESC LIMIT 10; -- Join tables that live in different data sources SELECT u.user_id, sum(t.amount) AS user_spending FROM "my_postgres.public.users" u LEFT JOIN "my_bigquery.my_dataset.transactions" t ON u.user_id = t.user_id GROUP BY u.user_id; -- Use a CTE and a window function to pick each user's latest transaction WITH ranked AS ( SELECT user_id, amount, row_number() OVER (PARTITION BY user_id ORDER BY created_at DESC) AS rn FROM "my_bigquery.my_dataset.transactions" ) SELECT user_id, amount FROM ranked WHERE rn = 1; -- Query historical feature values from the offline store SELECT pkey, value, observed_at FROM "chalk.historical_values.user.account_balance" WHERE observed_at > TIMESTAMP '2026-08-25 00:00:00'; ``` ### Syntax Clauses appear in the following order. Every clause other than the SELECT list is optional. ``` [ WITH [ RECURSIVE ] cte_name [ (column_alias, ...) ] AS ( select ) [, ...] ] SELECT [ DISTINCT | DISTINCT ON (expression, ...) ] expression [ AS alias ] [, ...] [ FROM source [, ...] ] [ [ join_type ] JOIN source [ ON condition | USING (column, ...) ] ] [ WHERE condition ] [ GROUP BY expression [, ...] ] [ HAVING condition ] [ QUALIFY condition ] [ ORDER BY expression [ ASC | DESC ] [ NULLS FIRST | NULLS LAST ] [, ...] ] [ LIMIT count [ OFFSET count ] ] ``` Two or more SELECT statements can also be combined with set operations (UNION, INTERSECT, EXCEPT). ### WITH clause Common table expressions (CTEs) name a subquery so it can be referenced like a table in the rest of the statement. CTEs may declare column aliases and may reference earlier CTEs in the same WITH list. ``` WITH spend(user_id, total) AS ( SELECT user_id, sum(amount) FROM "my_bigquery.my_dataset.transactions" GROUP BY user_id ) SELECT u.user_id, s.total FROM "my_postgres.public.users" u JOIN spend s ON u.user_id = s.user_id; ``` MATERIALIZED CTEs are not currently supported. ### Recursive CTEs WITH RECURSIVE defines a CTE as an anchor query combined with a recursive term that references the CTE itself. The anchor runs once. The recursive term then runs repeatedly, each time over the rows the previous round added, until a round adds no rows. The CTE's result is every row produced along the way. ``` -- Integers 1 through 5 WITH RECURSIVE counter(n) AS ( SELECT 1 UNION ALL SELECT n + 1 FROM counter WHERE n < 5 ) SELECT n FROM counter; ``` With UNION ALL every row is kept. With UNION, a row that the CTE has already produced is dropped, so recursion over cyclic data still terminates. The CTE's columns take their names and types from the anchor; the recursive term's rows are cast to those types. Restrictions: - The recursive term may reference its own CTE but not an enclosing recursive CTE. - USING KEY is not supported. - Recursion that has not stopped after 10,000 rounds fails the query. ### SELECT list The SELECT list contains any supported expression or function, optionally renamed with AS. An alias defined earlier in the SELECT list can be referenced by later expressions in the same list (a "lateral" column alias): ``` SELECT amount * 100 AS cents, cents / 3 AS thirds FROM "my_bigquery.my_dataset.transactions"; ``` DISTINCT deduplicates the result rows, and DISTINCT ON (expressions) keeps one row per distinct value of the given expressions: ``` -- One row per user SELECT DISTINCT ON (user_id) user_id, amount FROM "my_bigquery.my_dataset.transactions" ORDER BY user_id, created_at DESC; ``` The star expression * selects all columns of the source, and t.* selects the columns of one source in a join. Both accept DuckDB's modifiers: ``` -- Cast one column; it keeps its name and position SELECT * REPLACE (CAST(amount AS BIGINT) AS amount) FROM "my_bigquery.my_dataset.transactions"; -- Drop columns SELECT * EXCLUDE (country, year) FROM "my_bigquery.my_dataset.transactions"; -- Rename a column in place SELECT * RENAME (created_at AS ts) FROM "my_bigquery.my_dataset.transactions"; ``` COLUMNS(...) selects columns by a regular expression, a list of names, or a lambda over the column name, and applies the surrounding expression to each one. Every result column keeps the name of the column it came from: ``` -- Cast every column whose name matches the regex SELECT COLUMNS('amount|year')::BIGINT FROM "my_bigquery.my_dataset.transactions"; SELECT max(COLUMNS(['amount', 'created_at'])) FROM "my_bigquery.my_dataset.transactions"; SELECT COLUMNS(c -> c LIKE '%_id') FROM "my_bigquery.my_dataset.transactions"; ``` * LIKE 'pattern' and * SIMILAR TO 'regex' filter columns by name. A COLUMNS lambda supports LIKE/ILIKE, SIMILAR TO, regexp_matches, contains, starts_with, ends_with, lower/upper, =, <>, IN, IS [NOT] NULL, AND, OR, and NOT. BigQuery's * EXCEPT (...) is not supported; use EXCLUDE. Restrictions: - SELECT * cannot be combined with GROUP BY or aggregate functions; list the grouped columns explicitly instead. - A bare * cannot appear inside another expression (e.g. count(*) is supported, but * + 1 is not); use COLUMNS(*) + 1 instead. ### FROM clause The FROM clause names the sources rows are read from. A source can be: - A catalog or data source table, referenced with a quoted fully-qualified name: "catalog.schema.table". This includes data source tables (e.g. "my_postgres.public.users") and Chalk catalog views such as "chalk.online_store.keys" or "chalk.historical_values.user". See Chalk Catalog Components. - A table-valued function, such as read_parquet(...), query_values_from_operation_ids(...), or get_dataset_givens(...). For the full list, query "chalk.information_schema.functions" where is_table_valued. - A subquery, wrapped in parentheses and optionally aliased. - A VALUES list, for inline literal rows:SELECT * FROM (VALUES (1, 'ada'), (2, 'grace')) AS t(id, name); - A file path or glob, which reads files directly:SELECT * FROM 's3://my-bucket/events/*.parquet'; A SELECT without a FROM clause evaluates its expressions over a single row (SELECT 1 + 1). ### TABLESAMPLE File-backed scans (direct file paths and read_parquet(...)) support percentage-based sampling with the SYSTEM and BERNOULLI methods: ``` SELECT * FROM 's3://my-bucket/events/*.parquet' TABLESAMPLE SYSTEM (10 PERCENT); ``` Not currently supported: sampling catalog or data source tables, RESERVOIR sampling, absolute row-count samples, sampling subqueries or joins, and the USING SAMPLE clause. ### JOIN clause Chalk SQL supports joining tables, including tables that live in different data sources, with the following join types: - [INNER] JOIN - LEFT [OUTER] JOIN - RIGHT [OUTER] JOIN - FULL [OUTER] JOIN - CROSS JOIN - ASOF JOIN and ASOF LEFT JOIN, for matching each row to the nearest preceding row by an inequality condition (commonly a timestamp) Join conditions can be written with ON or USING (columns). Inner joins also accept non-equality ON conditions. ``` -- Match each transaction to the feature value observed at or before it SELECT t.user_id, t.amount, f.value AS balance_at_txn FROM "my_bigquery.my_dataset.transactions" t ASOF LEFT JOIN "chalk.historical_values.user.account_balance" f ON t.user_id = f.pkey AND t.created_at >= f.observed_at; ``` Not currently supported: NATURAL joins, POSITIONAL joins, and SEMI/ANTI joins. ### WHERE clause Filters rows with any supported boolean expression, including subqueries with IN and EXISTS. Where possible, Chalk SQL pushes filters down into the underlying data source so that less data is scanned; pushdown is best-effort and does not change results. ``` SELECT * FROM "my_postgres.public.users" WHERE signup_date > DATE '2026-01-01' AND country IN ('US', 'CA'); ``` ### GROUP BY clause Groups rows by one or more expressions for use with aggregate functions, which are listed in "chalk.information_schema.functions" where is_aggregate. HAVING filters the grouped rows and requires a GROUP BY clause or an aggregate in the SELECT list. ``` SELECT date_trunc('day', created_at) AS day, count(*) AS txns FROM "my_bigquery.my_dataset.transactions" GROUP BY date_trunc('day', created_at) HAVING count(*) > 100; ``` GROUPING SETS, ROLLUP, and CUBE are supported for aggregating across several groupings in one query: ``` -- Per-country counts plus a grand-total row (country IS NULL) SELECT country, count(*) FROM "my_bigquery.my_dataset.transactions" GROUP BY ROLLUP (country); ``` Not currently supported: GROUP BY ALL. ### Window functions and QUALIFY Window functions compute a value over a window of rows related to the current row, using OVER (PARTITION BY ... ORDER BY ... [frame]). ROWS frames with literal bounds are supported (e.g. ROWS BETWEEN 1 PRECEDING AND CURRENT ROW). The QUALIFY clause filters on window function results, the way HAVING filters on aggregate results. A query must contain at least one window function to use QUALIFY: ``` SELECT user_id, amount, row_number() OVER (PARTITION BY user_id ORDER BY created_at DESC) AS rn FROM "my_bigquery.my_dataset.transactions" QUALIFY rn <= 3; ``` Not currently supported in window functions: the FILTER clause, frame EXCLUDE options, non-literal frame bounds, and RANGE frames with offset bounds (the default frame implied by ORDER BY works). ### ORDER BY clause Sorts the result by one or more expressions. Each sort key accepts ASC (default) or DESC, and NULLS FIRST / NULLS LAST. Sort keys can be arbitrary expressions (including expressions not in the SELECT list), positional references to SELECT list items (ORDER BY 2), or ORDER BY ALL to sort by every selected column left to right. ### LIMIT and OFFSET clauses LIMIT restricts the number of returned rows and OFFSET skips rows before returning. Both must be literal integers. ``` SELECT * FROM "my_postgres.public.users" ORDER BY user_id LIMIT 100 OFFSET 200; ``` ### Set operations Two or more SELECT statements can be combined with: - UNION (deduplicates) and UNION ALL (keeps duplicates) - INTERSECT - EXCEPT An ORDER BY following a set operation may only reference output column names or positions. To apply a LIMIT (or sort by a computed expression), wrap the set operation in a subquery: ``` SELECT * FROM ( SELECT user_id FROM "my_postgres.public.users" UNION SELECT user_id FROM "my_postgres.public.archived_users" ) ORDER BY user_id LIMIT 50; ``` ### Subqueries Subqueries are supported in the FROM clause, as scalar expressions, and in IN / EXISTS predicates, including correlated forms: ``` SELECT * FROM "my_postgres.public.users" u WHERE EXISTS ( SELECT 1 FROM "my_bigquery.my_dataset.transactions" t WHERE t.user_id = u.user_id AND t.amount > 10000 ); ``` Some constructs are not currently supported inside subqueries, including GROUP BY within an EXISTS subquery, DISTINCT in correlated scalar subqueries, and ARRAY(SELECT ...). # DESCRIBE and SHOW Statements source: https://docs.chalk.ai/docs/chalksql/statements/describe ## Inspect the schema of a table or query, or list available tables. The DESCRIBE statement returns the schema of a table or of an arbitrary query. SHOW is a synonym for DESCRIBE
, and SHOW TABLES lists the tables available in the Chalk catalog. ### Examples ``` -- Describe a data source table DESCRIBE "my_postgres.public.users"; -- Describe the output schema of a query without running it over all data DESCRIBE SELECT user_id, sum(amount) AS total FROM "my_bigquery.my_dataset.transactions" GROUP BY user_id; -- List all available tables SHOW TABLES; ``` ### DESCRIBE ``` DESCRIBE "catalog.schema.table"; DESCRIBE select_statement; SHOW "catalog.schema.table"; ``` DESCRIBE returns one row per column with the following schema: | Column | Type | Description | | ------------- | ------ | ----------------------------------------------- | | `column_name` | string | The name of the column | | `column_type` | string | The data type of the column | | `null` | string | `YES` if the column is nullable, `NO` otherwise | | `key` | string | Always `NULL` | | `default` | string | Always `NULL` | | `extra` | string | Always `NULL` | DESCRIBE is compiled as a table expression, so it can also be used in the FROM clause of a SELECT: ``` SELECT column_name FROM (DESCRIBE "my_postgres.public.users") WHERE column_type = 'string'; ``` ### SHOW TABLES ``` SHOW TABLES; ``` Lists the tables visible to Chalk SQL. It is equivalent to querying the tables view of the information_schema schema: ``` SELECT * FROM "chalk.information_schema.tables"; ``` To explore available catalogs and schemas interactively, you can also use the Database Explorer in the dashboard, or query chalk.information_schema.schemas. ### Limitations - SHOW DATABASES is not supported. - SUMMARIZE is not supported. # EXPLAIN Statement source: https://docs.chalk.ai/docs/chalksql/statements/explain ## Show the physical plan for a statement, optionally with execution statistics. The EXPLAIN statement compiles a statement through Chalk SQL's planner and optimizer and returns the resulting physical plan as text, without executing the query. EXPLAIN ANALYZE additionally executes the query and annotates the plan with runtime statistics. ### Examples ``` -- Show the optimized physical plan without running the query EXPLAIN SELECT u.user_id, sum(t.amount) FROM "my_postgres.public.users" u JOIN "my_bigquery.my_dataset.transactions" t ON u.user_id = t.user_id GROUP BY u.user_id; -- Run the query and include execution statistics in the plan EXPLAIN ANALYZE SELECT count(*) FROM "my_bigquery.my_dataset.transactions"; ``` ### Syntax ``` EXPLAIN statement; EXPLAIN ANALYZE statement; ``` The result is a single row with a single string column named explain containing the plan. ### Behavior - EXPLAIN binds, optimizes, and compiles the statement without reading any data, so it surfaces planning-time errors such as missing tables or columns and unsupported constructs. - EXPLAIN ANALYZE executes the statement to completion and returns the plan annotated with per-operator execution statistics. The query's normal output rows are discarded; only the annotated plan is returned. Because the query really runs, EXPLAIN ANALYZE costs as much as the query itself. EXPLAIN is useful for checking which parts of a query are pushed down into an underlying data source versus executed by Chalk's engine, such as whether a WHERE filter was pushed into the data source scan. # COPY TO Statement source: https://docs.chalk.ai/docs/chalksql/statements/copy ## Write the result of a query to Parquet, CSV, or Iceberg files. The COPY ... TO statement writes the result of a query (or the contents of a table) to files. It is the only Chalk SQL statement that produces output outside of the query response itself; all other statements are read-only. ### Examples ``` -- Export a query result to a single Parquet file COPY ( SELECT user_id, sum(amount) AS total FROM "my_bigquery.my_dataset.transactions" GROUP BY user_id ) TO 's3://my-bucket/exports/user_totals.parquet'; -- Export a table to a directory of CSV files COPY "my_postgres.public.users" TO 's3://my-bucket/exports/users/' (FORMAT 'csv'); -- Export selected columns only COPY "my_postgres.public.users" (user_id, email) TO 'gs://my-bucket/users.parquet'; -- Hive-partitioned Parquet output COPY (SELECT * FROM "my_bigquery.my_dataset.transactions") TO 's3://my-bucket/exports/transactions/' (FORMAT 'parquet', PARTITION_BY (country, year)); -- Write an Iceberg table with a partition spec COPY (SELECT * FROM "my_bigquery.my_dataset.transactions") TO 's3://my-bucket/warehouse/transactions' (FORMAT 'iceberg', PARTITION_BY (bucket(16, user_id), day(created_at))); ``` ### Syntax ``` COPY ( select_statement ) TO 'uri' [ ( option [, ...] ) ]; COPY "catalog.schema.table" [ (column [, ...]) ] TO 'uri' [ ( option [, ...] ) ]; ``` The destination uri determines where files are written; the storage backend is selected from the URI scheme (for example a local path, s3://, or gs://). A URI ending in .parquet or .csv targets a single file, while a directory-style URI (trailing /) writes one or more files with generated names beneath it. ### Options | Option | Description | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FORMAT` | `'parquet'` (default), `'csv'`, or `'iceberg'`. When omitted, the format is inferred from the destination file extension, falling back to Parquet. | | `PARTITION_BY` | Partitions the output. For Parquet and CSV, accepts column names and produces Hive-style partitioned directories. For Iceberg, additionally accepts the partition transforms `bucket(N, col)`, `truncate(W, col)`, `year(col)`, `month(col)`, `day(col)`, and `hour(col)`. | | `WRITE_BLOOM_FILTER` | Parquet only: write bloom filters for the output files. | | `BLOOM_FILTER_FALSE_POSITIVE_RATIO` | Parquet only: bloom filter false-positive ratio in `(0, 1)`; defaults to `0.01`. Setting this implies `WRITE_BLOOM_FILTER true`. | ### Iceberg output With FORMAT 'iceberg', the destination URI names the Iceberg table root (a directory, not a file). Chalk SQL writes the Parquet data files and commits the Iceberg metadata (manifests and a new snapshot) beneath that root, creating the table on first write. PARTITION_BY entries define the table's partition spec; a bare column name is an identity transform. ### Limitations - COPY ... FROM (loading data) is not supported — Chalk SQL is a read-only interface over your data sources. - FORMAT 'delta' is not supported. - Bloom filter options require FORMAT 'parquet' and cannot be combined with PARTITION_BY. # Chalk SQL Expressions source: https://docs.chalk.ai/docs/chalksql/expressions ## Syntax for literals, struct, map, and list access, CASE, casts, and lambdas. An expression computes a value. Expressions appear in the SELECT list, in WHERE, GROUP BY, HAVING, and ORDER BY, and as function arguments. This page documents the expression syntax that is not a named function. Named functions are documented in the Chalk SQL function reference. The star expression, COLUMNS(...), and subqueries are documented on the SELECT page. Every example on this page builds its input from literal values, so it runs as written against any Chalk environment. ### Literals A literal is a value written directly in the query. ### Lists, structs, and maps A list literal is a comma-separated list of values in square brackets. A struct literal is a set of 'key': value pairs in curly braces. A map literal is the keyword MAP followed by key: value pairs in curly braces. ``` SELECT [10, 20, 30] AS amounts, {'city': 'Oakland', 'zip': '94612'} AS address, MAP {'small': 1, 'large': 3} AS sizes; ``` Struct keys are field names and are always strings. Map keys can be any type, and every key in one map has the same type. ### Dates, timestamps, and intervals A typed literal is a type name followed by a string. INTERVAL takes either a number and a unit or a quoted string. ``` SELECT DATE '2026-10-14' AS day, TIMESTAMP '2026-10-14 08:00:00' AS local_time, TIMESTAMPTZ '2026-10-14 08:00:00+00' AS utc_time, TIMESTAMP '2026-10-14 08:00:00' + INTERVAL 1 DAY AS next_day, TIMESTAMP '2026-10-14 08:00:00' + INTERVAL '2 hours' AS two_hours_later; ``` See Timestamp types for the difference between TIMESTAMP and TIMESTAMPTZ. ### Struct field access Read a struct field with a dot followed by the field name. Quote a field name that contains spaces or other special characters with double quotes. Brackets with a string literal read the same field. ``` SELECT address.city, -- 'Oakland', in a column named "city" address."zip code", -- '94612', in a column named "zip code" address['city'] AS city_2 -- 'Oakland' FROM (SELECT {'city': 'Oakland', 'zip code': '94612'} AS address); ``` Dots chain through nested structs, and a table alias can come first: ``` SELECT t.customer.address.city -- 'Oakland' FROM (SELECT {'address': {'city': 'Oakland'}} AS customer) AS t; ``` Chalk SQL first reads a dotted name as written, so t.customer is the column customer of the table t. When no table or column matches the full name, Chalk SQL takes the longest leading part of the name that is a column and reads the rest as struct fields. Field access works anywhere an expression is allowed, including WHERE and correlated subqueries. ### Map lookup Read a map value with the key in square brackets. A key that is not in the map returns NULL. ``` SELECT sizes['large'] AS large, -- 3 sizes['medium'] AS medium -- NULL FROM (SELECT MAP {'small': 1, 'large': 3} AS sizes); ``` The key must have the map's key type. Looking up an integer key in a map with string keys is a type error. ### List indexing and slicing List indexes start at 1. A negative index counts from the end of the list, so -1 is the last element. An index past either end of the list returns NULL. Index 0 is an error. list[start:end] returns the elements from start to end, including both ends. Leave out start to slice from the first element, or end to slice to the last. ``` SELECT amounts[1] AS first, -- 10 amounts[-1] AS last, -- 30 amounts[5] AS missing, -- NULL amounts[2:3] AS middle, -- [20, 30] amounts[:2] AS first_two, -- [10, 20] amounts[-2:] AS last_two -- [20, 30] FROM (SELECT [10, 20, 30] AS amounts); ``` Brackets chain, so [[1, 2], [3, 4]][2][1] returns 3. ### CASE A searched CASE returns the result of the first WHEN condition that is true. A simple CASE compares one value against each WHEN value. When no branch matches and there is no ELSE, the result is NULL. ``` SELECT amount, CASE WHEN amount >= 100 THEN 'large' WHEN amount >= 10 THEN 'medium' ELSE 'small' END AS size, CASE status WHEN 1 THEN 'open' WHEN 2 THEN 'closed' END AS status_name FROM (VALUES (250, 1), (40, 2), (5, 3)) AS t(amount, status); ``` The third row has status 3, which matches no branch, so its status_name is NULL. ### Casts CAST(value AS type) and value::type convert a value to another type. When a conversion fails, CAST returns an error and TRY_CAST(value AS type) returns NULL. ``` SELECT CAST('42' AS BIGINT) AS a, -- 42 '42'::BIGINT AS b, -- 42 TRY_CAST('abc' AS BIGINT) AS c; -- NULL ``` ### Timestamp types Chalk SQL has two kinds of timestamp type: | Type | Meaning | | ------------------------------------------------------ | -------------------------------------------------- | | `TIMESTAMP` | A date and time with no time zone, in microseconds | | `TIMESTAMP_S`, `TIMESTAMP_MS`, `TIMESTAMP_NS` | The same, in seconds, milliseconds, or nanoseconds | | `TIMESTAMPTZ`, also written `TIMESTAMP WITH TIME ZONE` | An instant in time, stored in UTC | Conversions between the two kinds use UTC: - Casting a TIMESTAMP to TIMESTAMPTZ reads its date and time as UTC. - Casting a TIMESTAMPTZ to TIMESTAMP returns its date and time in UTC. - Casting a string that includes a UTC offset to TIMESTAMP returns the date and time in UTC. ``` SELECT CAST(TIMESTAMP '2026-10-14 08:00:00' AS TIMESTAMPTZ) AS a, -- 2026-10-14 08:00:00+00 CAST(TIMESTAMPTZ '2026-10-14 08:00:00+02' AS TIMESTAMP) AS b, -- 2026-10-14 06:00:00 CAST('2026-10-14 05:00:00+02' AS TIMESTAMP) AS c; -- 2026-10-14 03:00:00 ``` A function or comparison that combines a TIMESTAMP with a TIMESTAMPTZ reads the TIMESTAMP as UTC. A function called with only TIMESTAMP arguments, such as date_trunc, returns a TIMESTAMP. To convert between UTC and another time zone, use AT TIME ZONE, as in ts AT TIME ZONE 'America/New_York'. ### Lambdas A lambda is an inline function passed as an argument to a function that applies it to each element of a list. Chalk SQL accepts three forms: - x -> x * 10 for one parameter - (a, b) -> a + b for two parameters - lambda x: x * 10, the Python-style form ``` SELECT list_transform([1, 2, 3], x -> x * 10) AS tens, -- [10, 20, 30] list_filter([1, 2, 3, 4], lambda x: x % 2 = 0) AS evens, -- [2, 4] zip_with([1, 2], [10, 20], (a, b) -> a + b) AS sums, -- [11, 22] array_reduce([1, 2, 3], 0, (acc, x) -> acc + x, acc -> acc) AS total; -- 6 ``` Lambdas are accepted only by the following functions: | Function | Lambda parameters | Result | | ----------------------------------------------------------------------------- | ---------------------------------- | --------------------------------------------------- | | `list_transform`, `array_transform`, `transform`, `list_apply`, `array_apply` | The element | A list of the lambda's results | | `list_filter`, `array_filter` | The element | A list of the elements for which the lambda is true | | `list_sort`, `array_sort` | The element | The list, sorted by the lambda's result | | `find_first` | The element | The first element for which the lambda is true | | `find_first_index` | The element | The index of that element | | `map_contains` | The map key | Whether any key satisfies the lambda | | `zip_with` | One element from each of two lists | A list of the lambda's results | | `array_reduce` | Two lambdas, described below | The final value | array_reduce(list, initial_state, (state, x) -> new_state, state -> result) takes two lambdas. The first combines the running state with each element, starting from initial_state. The second turns the final state into the result. A lambda parameter that holds a struct supports field access, for example list_transform(people, p -> p.name). Outside these function calls, -> is the JSON extraction operator. COLUMNS(...) also accepts a lambda over column names. # TypeScript Client source: https://docs.chalk.ai/docs/client-typescript ## Query features with Chalk's TypeScript client library The Chalk TypeScript client library provides a convenient way to query features from your TypeScript and JavaScript applications. - chalk-ts GitHub Repository - NPM Package ### Installation Install the Chalk TypeScript client using npm or yarn: ``` npm install @chalk-ai/client ``` or ``` yarn add @chalk-ai/client ``` ### Basic Usage Here's a quick example of how to use the Chalk TypeScript client: ``` import { ChalkClient } from '@chalk-ai/client'; // Initialize the client const client = new ChalkClient({ clientId: 'your-client-id', clientSecret: 'your-client-secret', environmentId: 'your-environment-id' }); // Query features const result = await client.query({ inputs: { 'user.id': 'user123' }, outputs: [ 'user.credit_score', 'user.account_age_days' ] }); console.log(result); ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Python Client source: https://docs.chalk.ai/docs/client-python ## Query features with Chalk's Python client library The Chalk Python client library (chalkpy) provides a convenient way to query features from your Python applications. - PyPI Package - Python SDK Reference - Complete API documentation ### Installation Install chalkpy using pip. Chalkpy currently supports Python 3.11 to 3.13. ``` pip install chalkpy ``` ### Basic Usage Here's a quick example of how to use the Chalk Python client: ``` from chalk.client import ChalkClient # Initialize the client client = ChalkClient( client_id='your-client-id', client_secret='your-client-secret', environment='your-environment-id' ) # Query features result = client.query( input={ 'user.id': 'user123' }, output=[ 'user.credit_score', 'user.account_age_days' ] ) print(result) ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API - Python SDK Reference - Full API documentation # Go Client source: https://docs.chalk.ai/docs/client-go ## Query features with Chalk's Go client library The Chalk Go client library provides a convenient way to query features from your Go applications. - chalk-go GitHub Repository - pkg.go.dev Documentation ### Installation Install the Chalk Go client using go get: ``` go get github.com/chalk-ai/chalk-go ``` ### Basic Usage Here's a quick example of how to use the Chalk Go client: ``` package main import ( "context" "fmt" "log" "github.com/chalk-ai/chalk-go" ) func main() { // Initialize the client client, err := chalk.NewClient( chalk.WithClientID("your-client-id"), chalk.WithClientSecret("your-client-secret"), chalk.WithEnvironment("your-environment-id"), ) if err != nil { log.Fatal(err) } // Query features result, err := client.Query(context.Background(), chalk.QueryParams{ Inputs: map[string]interface{}{ "user.id": "user123", }, Outputs: []string{ "user.credit_score", "user.account_age_days", }, }) if err != nil { log.Fatal(err) } fmt.Println(result) } ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Rust Client source: https://docs.chalk.ai/docs/client-rust ## Query features with Chalk's Rust client library The Chalk Rust client library provides a convenient way to query features from your Rust applications. - chalk-rust GitHub Repository - crates.io Package ### Installation Add the Chalk Rust client to your Cargo.toml: ``` [dependencies] chalk-client = "*" ``` Or install using cargo: ``` cargo add chalk-client ``` ### Basic Usage Here's a quick example of how to use the Chalk Rust client: ``` use chalk_client::ChalkClient; use chalk_client::types::QueryOptions; use std::collections::HashMap; #[tokio::main] async fn main() -> Result<(), Box> { // Initialize the client let client = ChalkClient::new() .client_id("your-client-id") .client_secret("your-client-secret") .environment("your-environment-id") .build() .await?; // Query features for a single entity let inputs = HashMap::from([("user.id".to_string(), serde_json::json!(1))]); let outputs = vec![ "user.credit_score".to_string(), "user.account_age_days".to_string(), ]; let response = client.query(inputs, outputs, QueryOptions::default()).await?; for feature in &response.data { println!("{}: {:?}", feature.field, feature.value); } Ok(()) } ``` For lower-latency, higher-throughput workloads, Chalk also offers a gRPC client (ChalkGrpcClient) that speaks Protocol Buffers over HTTP/2. Here's a bulk online query, where inputs and results are exchanged as Arrow IPC ("feather") data: ``` use chalk_client::ChalkGrpcClient; use chalk_client::gen::chalk::common::v1::online_query_bulk_request::Inputs; use chalk_client::gen::chalk::common::v1::output_expr::Expr; use chalk_client::gen::chalk::common::v1::{OnlineQueryBulkRequest, OutputExpr}; use arrow::array::Int64Array; use arrow::datatypes::{DataType, Field, Schema}; use arrow::ipc::reader::FileReader; use arrow::ipc::writer::FileWriter; use arrow::record_batch::RecordBatch; use std::io::Cursor; use std::sync::Arc; #[tokio::main] async fn main() -> Result<(), Box> { // gRPC enforces a 4 MiB limit when decoding responses by default. Bulk // queries can exceed that, so raise it to fit your largest result set. let client = ChalkGrpcClient::new() .client_id("your-client-id") .client_secret("your-client-secret") .environment("your-environment-id") .max_decoding_message_size(256 * 1024 * 1024) // 256 MiB .build() .await?; // Inputs are an Arrow record batch (one row per entity), serialized to // Arrow IPC ("feather") bytes. let schema = Arc::new(Schema::new(vec![Field::new("user.id", DataType::Int64, false)])); let batch = RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(vec![1, 2, 3]))])?; let mut feather = Vec::new(); { let mut writer = FileWriter::try_new(&mut feather, &batch.schema())?; writer.write(&batch)?; writer.finish()?; } let response = client .query_bulk_proto(OnlineQueryBulkRequest { inputs: Some(Inputs::InputsFeather(feather)), outputs: vec![ OutputExpr { expr: Some(Expr::FeatureFqn("user.name".to_string())) }, OutputExpr { expr: Some(Expr::FeatureFqn("user.age".to_string())) }, ], ..Default::default() }) .await?; // Results come back as Arrow IPC bytes in `scalars_data`. if !response.scalars_data.is_empty() { let reader = FileReader::try_new(Cursor::new(&response.scalars_data), None)?; for batch in reader { let batch = batch?; println!("{} rows x {} cols", batch.num_rows(), batch.num_columns()); } } for err in &response.errors { eprintln!("error: {}", err.message); } Ok(()) } ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Java Client source: https://docs.chalk.ai/docs/client-java ## Query features with Chalk's Java client library The Chalk Java client library provides a convenient way to query features from your Java applications. - chalk-java GitHub Repository ### Installation Add the Chalk Java client to your project using Maven or Gradle: ### Maven ``` ai.chalk chalk-java latest ``` ### Gradle ``` dependencies { implementation 'ai.chalk:chalk-java:latest' } ``` ### Basic Usage Here's a quick example of how to use the Chalk Java client: ``` import ai.chalk.ChalkClient; import ai.chalk.models.QueryParams; import ai.chalk.models.QueryResult; import java.util.HashMap; import java.util.List; import java.util.Map; public class Main { public static void main(String[] args) { // Initialize the client ChalkClient client = ChalkClient.builder() .clientId("your-client-id") .clientSecret("your-client-secret") .environmentId("your-environment-id") .build(); // Query features Map inputs = new HashMap<>(); inputs.put("user.id", "user123"); QueryParams params = QueryParams.builder() .inputs(inputs) .outputs(List.of( "user.credit_score", "user.account_age_days" )) .build(); QueryResult result = client.query(params); System.out.println(result); } } ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # C# Client source: https://docs.chalk.ai/docs/client-csharp ## Query features with Chalk's C# client library The Chalk C# client library provides a convenient way to query features from your C# and .NET applications. - chalk-csharp GitHub Repository - NuGet Package ### Installation Install the Chalk C# client using the .NET CLI: ``` dotnet add package Chalk.Client ``` Or using the Package Manager Console: ``` Install-Package Chalk.Client ``` ### Basic Usage Here's a quick example of how to use the Chalk C# client: ``` using Chalk; using Chalk.Models; using System; using System.Collections.Generic; using System.Threading.Tasks; class Program { static async Task Main(string[] args) { // Initialize the client IChalkClient client = ChalkClient.Builder() .WithClientId("your-client-id") .WithClientSecret("your-client-secret") .WithEnvironmentId("your-environment-id") .Build(); // Query features OnlineQueryResult result = await client.OnlineQueryAsync(new OnlineQueryParams { Inputs = new Dictionary> { { "user.id", new List { "user123" } } }, Outputs = new List { "user.credit_score", "user.account_age_days" } }); Console.WriteLine(result); } } ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Elixir Client source: https://docs.chalk.ai/docs/client-elixir ## Query features with Chalk's Elixir client library The Chalk Elixir client library provides a convenient way to query features from your Elixir applications. - chalk-elixir Github Repository - Hex Package ### Installation Add chalk_elixir to your list of dependencies in mix.exs: ``` def deps do [ {:chalk_elixir, "~> 0.1.0"} ] end ``` Then run: ``` mix deps.get ``` ### Basic Usage Here's a quick example of how to use the Chalk Elixir client: ``` # Configure the client config :chalk_elixir, client_id: "your-client-id", client_secret: "your-client-secret", environment_id: "your-environment-id" # Query features defmodule MyApp do alias ChalkElixir.Client def query_features do {:ok, result} = Client.query( %{ inputs: %{ "user.id" => "user123" }, outputs: [ "user.credit_score", "user.account_age_days" ] } ) IO.inspect(result) end end ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Ruby Client source: https://docs.chalk.ai/docs/client-ruby ## Query features with Chalk's Ruby client library The Chalk Ruby client library provides a convenient way to query features from your Ruby applications. - chalk-ruby GitHub Repository - RubyGems Package ### Installation Add chalk_ruby to your Gemfile: ``` gem 'chalk_ruby' ``` Then run: ``` bundle install ``` Or install it directly: ``` gem install chalk_ruby ``` ### Basic Usage Here's a quick example of how to use the Chalk Ruby client: ``` require 'chalk_ruby' # Initialize the client client = ChalkRuby::Client.new( client_id: 'your-client-id', client_secret: 'your-client-secret', environment_id: 'your-environment-id' ) # Query features result = client.query( inputs: { 'user.id' => 'user123' }, outputs: [ 'user.credit_score', 'user.account_age_days' ] ) puts result ``` ### Next Steps - Query Basics - Learn how to query features - Authentication - Set up authentication for your client - Online Query API - Explore the online query API # Changelog source: https://docs.chalk.ai/docs/changelog ## Updates to Chalk! ### October 8, 2026 ### Model Training You can now train models where your features live. Add the @training decorator to a function and declare its data, resources and container image. Training data comes from a Chalk Dataset, which is pinned to an immutable revision when the run starts, so retries and audits always use the same inputs. Chalk Volumes are mounted onto the training container, so your files are available to the training code without any setup. Runs execute in managed containers with GPU support. Checkpoints are saved to managed storage that survives container restarts, so a restarted run resumes from its last checkpoint. Metrics you log feed the loss curves and charts on the run detail page. When a run finishes, the trained model can be registered in the Model Registry and hosted for Model Serving, linked back to the config and data that produced it. ### October 7, 2026 ### Direct Fetch for Snowflake Online Queries Online queries against native Snowflake sources now fetch results directly instead of unloading them to Parquet first, which lowers latency for inference workloads. Batch operations still use the unload path. ### Unknown Named Query Versions Are Rejected Breaking: An online query that sets a query_name_version that does not match a deployed named query now fails, and the error lists the deployed versions. Queries that set only query_name are unaffected. The behavior is controlled by CHALK_REJECT_UNKNOWN_NAMED_QUERY_VERSION, which defaults to true. ### October 5, 2026 ### Redesigned Offline Query Form The offline query explorer has a new card-based form that groups inputs and options into sections. An Advanced panel shows how many options you have set, and a toggle switches back to the previous layout. ### Bloom Filter Pruning for Parquet Scans Hive and Iceberg scans now skip row groups whose Parquet bloom filters rule out the values a query filters on, which speeds up point lookups and IN queries. ### October 1, 2026 ### Model Deployments in the Dashboard The dashboard has a new Model deployments page for the models you deploy as services, with lifecycle controls, revision history and metrics for each deployment. See Model Deployments. ### Redesigned Trace Views The trace waterfall and flame graph views have been redesigned. The timeline supports scroll-to-zoom, and the span drawer can filter a span's attributes and export the span as JSON. ### September 30, 2026 ### GPUs in Sandboxes Sandboxes can now request GPUs on AWS and GKE host pools. Pass a GPU type such as gpu="nvidia-l4" when you create the sandbox, and Chalk schedules it on a node with that GPU. See Sandboxes for the compute class requirement. ### Run Mode in SQL Worksheets The Run button in SQL worksheets has a dropdown to choose synchronous or asynchronous execution. See Synchronous vs. asynchronous queries. ### September 28, 2026 ### Star Modifiers in Chalk SQL SELECT * in Chalk SQL now accepts DuckDB's EXCLUDE, REPLACE and RENAME modifiers, and COLUMNS(...) applies an expression to every column that matches a pattern or list. See SELECT. ### Approvals for Assistant Write Tools Chalk Assistant now pauses before a tool call that would change your environment, and shows the pending operation in the conversation for you to approve. ### Key-Pair Authentication for Snowflake New Snowflake integrations default to private key authentication. Password authentication still works and now shows a deprecation warning. ### September 25, 2026 ### Redesigned Online Query Explorer The online query explorer has type-aware value editors, searchable selects and collapsible advanced options. You can prefill a query from a named query and embed the query form in a notebook cell. ### Test Data Source Monitors Before Saving When you create a data source monitor, Test now runs the connection test inline. Connection cards show each data source's health check monitor status. ### September 24, 2026 ### AI Decision Functions F.prompt_jev asks Jev's structured-decision API typed questions about a piece of text, such as a support message or a transaction description, and returns the answers with token usage. Each question asks for a yes/no probability (noul), one choice from a set of criteria (choice), or a position on an ordered scale (score), and all of them are answered in one request. F.decision_complete is an alias, and both are also available as Chalk SQL functions. See prompt_jev. ### September 12, 2026 ### Chalk Assistant You can now work with an AI assistant inside the Chalk dashboard. Chalk Assistant runs on an AI provider you connect: OpenAI, Anthropic or any OpenAI-compatible endpoint. An Owner or Team Manager turns it on for an environment and picks the provider and model that every conversation in that environment uses. To start a chat, click Assistant on the rail at the bottom of the dashboard. Chats open in windows along the rail, so you can keep working while the assistant runs. The assistant uses the same tools as the Chalk MCP Server. It can run online queries and Chalk SQL, read feature and resolver definitions, create and run notebook cells, search logs and traces, and call the Chalk Functions you have deployed. You can attach features, resolvers, datasets, registry models and SQL tables to a conversation instead of pasting their names. Every tool call goes through the same permission check as the MCP Server, using your account, so the assistant can change only what you can change. Messages, tool arguments and tool results are stored in your environment's data plane, and an Owner or Team Manager can view every conversation in the environment. See Chalk Assistant. ### September 11, 2026 ### Materialized Aggregations Without an Online Store You can now deploy materialized aggregations in an environment that has no online store. ### August 27, 2026 ### OpenAI Embeddings in Expressions F.openai_embed calls OpenAI's embeddings API during feature computation and returns the embedding vector with its token usage and model. See openai_embed. ### August 17, 2026 ### AWS RDS IAM Authentication for PostgreSQL PostgreSQL data sources on Amazon RDS can authenticate with IAM database authentication instead of a stored password. See Authenticating with AWS RDS IAM. ### August 10, 2026 ### Lint Chalk SQL from Your Agent The Chalk MCP Server has a new lint_sql_query tool. It plans a Chalk SQL query against your catalog and returns the planner's errors without running it, so an agent can check SQL it has drafted. See MCP Server. ### August 5, 2026 ### Iceberg REST Catalogs Chalk can now scan and write Iceberg tables in REST catalogs such as Apache Polaris and Snowflake Open Catalog, in addition to AWS Glue. See Integration with AWS Glue and Iceberg REST Catalogs. ### August 1, 2026 ### Reworked Deployment Source Viewer The deployment source viewer supports multiple tabs, and when you compare source code between deployments you can switch the deployment on either side. ### July 30, 2026 ### Promote Datasets to Iceberg Dataset.promote_to_iceberg() writes a dataset revision to an Iceberg table that warehouses such as Snowflake and BigQuery can query directly. See Promote a Dataset to Iceberg. ### July 22, 2026 ### Better Offline Query Reruns The offline query explorer can rerun an offline query using Parquet inputs that are already in bucket storage. Point the query at the existing bucket URIs, and you do not need to upload the inputs again. ### July 7, 2026 ### Late-Arriving Data in Scheduled Backfills Set backfill_lookback_duration on a materialized aggregation to start each scheduled backfill that much earlier than the previous run's upper bound, so events that arrive late are included. See Materialized Aggregations. ### July 4, 2026 ### Charts in the SQL Explorer SQL Explorer results can be charted, with a wizard to configure the axes. ### July 2, 2026 ### Faceted Search for Traces Trace search turns your query into filters and suggests facet values as you type. ### Audit Log Diffs You can compare an audit log entry's request JSON against an earlier entry. ### July 1, 2026 ### Skewness and Kurtosis Aggregations skewness and kurtosis are now available as aggregation functions for describing the shape of a feature's distribution. See Aggregations. ### June 16, 2026 ### Webhooks for Offline Query Events Webhook subscriptions now cover offline queries. Chalk can send an event when a full query or one of its shards completes, fails or is canceled. See Webhooks. ### June 11, 2026 ### Model Gateway The Chalk Model Gateway routes LLM traffic to OpenAI, Anthropic and Google Gemini behind one OpenAI-compatible API. It runs in your Chalk environment, and you configure its providers in the dashboard. See Model Gateway. ### June 9, 2026 ### Versioned Features Serve the Default Version A query that names a versioned feature by string, such as "my_feature.score", now returns the feature's default_version instead of version 1. Queries that reference the feature through its class already returned the default version and are unchanged. ### June 4, 2026 ### SQL Inputs for Scheduled Queries ScheduledQuery now accepts input_sql, so a scheduled offline query can take its inputs from a SQL query. See ScheduledQuery and SQL as Input. ### May 29, 2026 ### Read-Only Access to Offline Query Results The new query.offline_read permission lets a role view offline query results without starting offline queries. Tokens that already have query.offline also get this permission. ### May 26, 2026 ### COPY TO in Chalk SQL The COPY ... TO statement writes the result of a Chalk SQL query, or the contents of a table, to files. See COPY. ### May 20, 2026 ### Window Functions in Chalk SQL Chalk SQL now supports window functions. See Window functions and QUALIFY. ### April 16, 2026 ### Filter Stream Messages by Header make_stream_resolver accepts message_header_filters, a list of (header_name, header_value) pairs. The streaming server drops any message whose headers do not match before deserializing it, so several resolvers can read different message types from one topic. See Filtering Multiplexed Topics with Message Headers. ### March 27, 2026 ### Parse Expressions for Stream Resolvers We now show you the compiled C++ expression actively running for stream resolvers in the resolver page in the feature catalog under “Definition”, along with the original Python function. ### Online Query Reruns Online queries can now be rerun using the button on the top right of the online query page. This will open the online data explorer and pre-fill the configuration and inputs from a specific query run. This functionality previously only existed for offline queries. ### Offline Store Table Names We now expose offline store table names, which are hashes of feature fqns (fully-qualified names) . Previously, if you wanted to build your own observability stack and integrate with Chalk’s offline store feature value table names, it was pretty difficult. These names were previously only exposed in the dashboard on the feature catalog page. Now, there are commands through Chalk clients and the Chalk CLI to expose this. ### March 20, 2026 ### Team Scoped Access Tokens Team-scoped access tokens authenticate to Chalk programmatically across various environments for infra-level actions that require broad team-level access. ### Dark Mode in Dashboard Dark Mode is now publicly available, because your features deserve to look good at 2 AM too. The toggle is available under your profile settings in the bottom left. Dashboard dark mode interface with the profile settings toggle highlighted ### Resource Group Support for Streaming Resolvers Resource group support is now available for streaming resolvers - allowing you to decide which set of compute/configs to run a resolver on. You can find Resource Groups under Infra > Resource Configurations on the left navbar of the web UI. ### March 13, 2026 ### Access Logs We now display all access logs under our “Logging” page. You can filter by Status Code, request path, and other properties. This page also overlays Kubernetes events on the access logs chart, which can help debug connectivity and uptime issues. Happy debugging! ### Run Predictions against Vertex AI We've added a new chalk.functions.vertex_predict function that allows you to run predictions against Vertex AI endpoints. Chalk will vectorize and run multi-row predictions in parallel on isolated pods, preventing resource contention. Read more here. ### March 6, 2026 ### Chart Annotations You can now annotate your charts - vertical lines for high signal activities, shaded boxes for times, and text annotations for communication. Chart annotations interface showing vertical lines and shaded time ranges ### March 2, 2026 ### Chalk Rust and C# Clients We have officially launched our open source Rust and C# Chalk clients. You can now query for features natively in these languages. ### Externally Managed Integration Secrets By default, we require you to create secrets through the Chalk UI for their integrations. Now, you can link to secrets in your cloud secret manager to configure data source integrations, making it easy to manage secret rotation outside of Chalk. Externally managed integration secrets configuration view ### Autospilling for Large Offline Queries When an offline query processes more data than can fit in memory, the execution engine automatically spills intermediate results to local disk rather than failing with an out-of-memory error. This currently defaults to 60% of available memory and should prevent a lot of OOMing. ### February 26, 2026 ### Materialized Aggregations - weighted approximate Top K Chalk now supports a weighted version of approximate Top K as a materialized aggregation. Instead of ranking values purely by frequency, you can rank them by weight, such as transaction amount. This enables questions like: - Which category accounted for the most revenue last month - Which segment contributed the highest weighted score The aggregation maintains constant memory usage and supports streaming updates. ### PyTorch dataset integration Offline query results can now be converted directly into PyTorch Dataset objects via the Python client. This creates a direct path from feature computation to model training, eliminating custom data conversion steps in ML pipelines. ### Trigger Perfetto profiling on demand Perfetto profiling can now be triggered via HTTP instead of running continuously. You can deploy the profiling daemon and use chalk profiling perfetto-snapshot to capture traces only when needed. This reduces storage overhead while improving debugging workflows for performance issues. ### February 23, 2026 ### Search source code from anywhere You can now search your deployment source code directly from the global command menu. Press Command+K or pull up the search bar, select Source Code, and jump straight to matching resolvers or SQL definitions. This significantly improves debugging workflows that require inspecting feature definitions or resolver logic. Global search for cod example ### Offline queries support numWorkers The numWorkers parameter is now supported for offline queries when the Job Queue Manager is enabled. This allows teams to rate-limit shard execution at the query level instead of modifying cluster-wide configuration. It improves predictability and resource control for large backfills and scheduled jobs. Read more: https://docs.chalk.ai/docs/job-queue ### Combined Aggs We recently shipped an optimization to the Chalk query planner that combines aggregations in the query plan by default, which should dramatically improve performance especially for large offline queries computing multiple aggregations over high-cardinality datasets. We have previously exposed this optimization through the environment variable CHALK_PLANNER_COMBINE_AGGREGATION_OPTIMIZATION as well as the planner option enable_combine_aggregation_optimization, but this is now defaulted to true in all platform versions v3.31.5 and onwards. ### January 26, 2026 ### Faster Query Planning Query planning is now even faster due to optimization work on the planner. Planning time improved by an average of 58%, reducing time-to-results and preventing online query timeouts for larger feature graphs. ### ClickHouse for logs and tracing Logs and tracing now run on ClickHouse, improving search performance. Facets—indexed attributes like service, environment, status, region, and user_id—help narrow searches and surface the most common values for easier production debugging. Log Explorer Page with New Facets ### Enhanced query plan visualization The query plan visualizer now supports node-by-node navigation through query DAGs. View associations between plan nodes and see how data flows through each query. Async batch UDFs and conditional nodes display in the data flow, making query plans easier to read and understand. Log Explorer Page with New Facets ### Store plan stages for scheduled queries Scheduled queries now include a "Store plan stages" feature. This allows for improved query debugging. ### December 22, 2025 ### Node monitoring improvements The Nodes tab on the Kubernetes page in the dashboard now supports filtering and sorting by node type, node pool, and CPU utilization. ### Expanded materialized aggregations Although previously supported as windowed aggregations, approx_percentile aggregations are now supported as materialized aggregations as well. Teams can compute windowed percentile statistics efficiently, improving latency for percentile-based monitoring and modeling. ### Native SQL driver for MS SQL Chalk now provides a native drive for Microsoft SQL Server, enabling users to write SQL resolvers against their MS SQL data sources, and Chalk compiles and executes the queries using Rust and C++ for performance optimization. This expands Chalk's coverage across data ecosystems, allowing teams that store features or raw data in MS SQL to integrate directly without custom pipelines or data exports. ### December 12, 2025 ### Preview messages when setting up stream sources You can now preview sample messages while configuring a new stream source. This makes it easier to verify that data is flowing correctly and that message formats (like JSON) are being parsed as expected before saving the connection. Example of stream message preview interface ### Updated navigation pane for faster workflows We refreshed the left-hand navigation pane to make Chalk easier to move through. Related concepts are now grouped more clearly, visual noise has been reduced, and high-frequency actions are easier to access. The result is a more predictable, friction-free workflow, especially in larger projects. Optional new navigation pane layout example ### November 25, 2025 ### Tracing for query performance diagnosis Tracing now provides full visibility into how queries execute inside Chalk. Each resolver and model call is instrumented and timed, allowing teams to pinpoint slow spans, identify inefficient SQL, and understand query behavior end to end. Trace data is surfaced directly in the dashboard, so engineers can drill into individual spans to view execution time, rows returned, and linked data source details. Traces can be enabled per query using CLI, ChalkClient(), or configured to sample automatically in production for continuous monitoring. Example Query Trace ### October 27, 2025 ### Stream consumption analytics Stream incidents now record failure events at one-minute granularity, up from five minutes previously. This provides finer visibility into where and when stream consumption fails, improving incident analysis and recovery time. Read more about Chalk’s metrics monitoring. Stream Monitoring Incident Graph ### Method chaining for Chalk DataFrame The Chalk DataFrame library now supports method chaining, allowing expressions like ``` tbl = pa.table( { "txns_last_hour": [[1, 2, 3, 4, 5], [100], [200, 201]], "max_txns_allowed": [3, 5, 4], } ) df = DataFrame.from_arrow(tbl) out = df.project( { "velocity_score": _.txns_last_hour .cardinality() .cast(float) .least(_.max_txns_allowed + 0.5) .ceil() .cast(int), "velocity_score_2": F.cast( F.ceil( F.least( F.cast(F.cardinality(_.txns_last_hour), float), _.max_txns_allowed + 0.5 ) ), int, ), } ) ``` The underscore interface is now the recommended API — it’s cleaner, type-safe, and validated by the Chalk function registry. Discover more about Chalk DataFrame aggregations and filters. ### Cron query tracking Scheduled (cron) queries now include detailed job-level status tracking, mirroring the system used for backfills. Teams can now see when jobs start, skip, or fail — and identify the exact cause directly in the UI. Learn more about scheduling queries for automated runs. Stream Monitoring Incident Graph ### Vector aggregates Chalk now supports materialized vector aggregates such as sum and mean. These precomputed embeddings accelerate workloads involving vector math and high-dimensional similarity features. ### Velox integration in Query Plan Viewer The query plan viewer now links Velox physical execution data to individual plan nodes. This lets teams analyze operator-level performance, data flow, and throughput directly from plan output — helping diagnose slow or memory-heavy offline queries. ### October 23, 2025 ### Environment Visibility Pods now display their Kubernetes service account, namespace, and role ARN. This makes deployment and debugging more transparent for engineers from within Chalk. Check the pod view in your Chalk environment to confirm service account mappings. Learn more about environment visibility in our docs. Kubernetes Service Accounts ### October 14, 2025 ### New Chalk expressions for retrieving the top and bottom N records Added new (materialize-able) aggregations (min_by_n/max_by_n) for computing the smallest/largest n elements based on any specified ordering column. Easily compute features like the 5 most recent transactions or the top 3 highest value purchases; combine them with windowed aggregations to compute these at various intervals – e.g., past day, week, month, etc. ``` @features class Transaction: id: int user_id: "User.id" timestamp: datetime amount: float @features class User: id: int transactions: DataFrame[Transaction] recent_transaction_amounts: list[int] | None = F.max_by_n(_.transactions[_.amount], _.timestamp, 3) ``` ### October 1, 2025 ### Chalk Dashboard Improvements - Custom role creation and assignment functionality, enabling specialized roles like "Machine Learning Scientist" and "Platform Engineer" - You can now search and filter environment variables in your Chalk dashboard - Added a connections pane to the overview section of Chalk dashboard that shows the health status of each underlying service in your Chalk deployment Connections pane ### September 26, 2025 ### Derive features at query time with Chalk's Dynamic Expressions Dynamic expressions enable you to compute new features on-the-fly during online queries without needing to define them in advance. Leverage Chalk's extensive function library directly in an online query to build adaptive systems that respond to changing requirements without code changes or deployments; this enables: - Rapid experimentation, A/B testing, and hotfixes to feature logic - Customization and conditional logic based on user context, business rules, etc. - Seasonal or time-sensitive feature adjustments (e.g., holiday promotions) ### Added new filtering options to Chalk's log explorer The log explorer now supports escaped quotes and also negation--making it easier to filter out unwanted results. Clicking on a log operator toggles it to negation to filter out logs e.g. from a particular service. ### Improvements to Chalk's diff viewer Enhanced Chalk's diff viewer to show the entire file contents for added/removed files instead of just file names; helps track feature and resolver changes between deployments and branches. ### New Chalk expression for computing the top most common values for a field Added a new windowed aggregation (approx_top_k), for computing an approximation of the most common values in a field. Takes in a required k parameter, such as approx_top_k(k=5) for the five approximately-most common values. ``` @feature class User: id: int transactions: DataFrame[Transaction] most_common_transaction_category: Windowed[list[str]] = windowed( "1h", max_staleness="15m", expression=_.transactions[ _.category, _.at > _.chalk_window, ].approx_top_k(k=5) ) ``` ### Chalk developer experience improvements - Re-running an offline query from the Chalk Dashboard auto-fills the input parameters with the previous values for easier experimentation - Added support for aggregations over Vector types (sum, mean) - Pod metrics e.g. request count, etc. are now easily exportable from the Chalk Dashboard (as a CSV) Re-running Offline Queries ### July 30, 2025 ### Chalk diff viewer New chalk diff command for visualizing code differences across deployments and branches. The diff viewer highlights additions, deletions, and modifications in a color-coded format, making it easy to spot changes at a glance. Compare any two branches chalk diff --branch= --other-branch= Compare a branch with main (default) chalk diff --branch= Compare two specific deployments chalk diff --deployment-id= --other-deployment-id= Compare a deployment with main branch chalk diff --deployment-id= ### Chalk's Online Query Explorer now supports viewing has-one and has-many relationships Chalk's Online Query Explorer now supports querying has-one and has-many relationship joins directly in the dashboard. When exploring a feature like User, you can now run online queries that include data from related features such as Profile (has-one) and Transfers (has-many) without leaving the dashboard interface. Output explorer with has-one and has-many features ### Chalk dashboard improvements - Aggregation backfills, which can be viewed from Chalk's Offline Query page now also include a table of the features that have been persisted with the backfill into your online store - Errors encountered while deploying a branch now bubble up to the branch deployment page in the Chalk Dashboard - Deactivated users are now visible in UI for more efficient management of inactive accounts ### June 23, 2025 ### Process images with Chalk's builtin chat completion API Chalk's chat completion API now supports multimodal inputs, enabling you to process images alongside text in your LLM workflows. This enhancement allows you to: - Pass images directly to vision-capable models like GPT-4o - Combine text and image inputs in a single prompt - Extract structured information from visual content ``` @features class Receipt: image_url: str image_response: P.MultimodalPromptResponse = P.completion( model="gpt-5.1", messages=[ P.message( "system", [ {"type": "input_text", "text": "describe this image"}, {"type": "input_image", "image_url": _.image_url}, ], ), ], ) ``` ### Workspace audit logs and monitoring The audit log page provides comprehensive tracking of workspace changes, helping you maintain security compliance, troubleshoot issues, and understand your team's activity. Navigate to the audit logs page from the Chalk dashboard by going to Settings → Audit Logs. The audit log captures: - Timestamps - Users or service accounts that made changes - API endpoints accessed - Descriptions of the operation - Success/failure status - IP addresses - Trace IDs for debugging ### Enhanced metadata export for online queries Added a "Full Export" option that includes both query metadata and the actual input/output data (query_values.parquet) to the online queries page. Each export includes query execution details, configurations, query plans, data values, and GraphQL information. ### Added an offline query input explorer Chalk's offline query explorer makes it easy to inspect, debug, and validate your offline query results through interactive SQL queries and data visualization. We've introduced an Inputs Explorer that makes it easier to see the inputs of the offline queries that you're running. Having visibility into both inputs and outputs provides a complete picture of your offline query lifecycle, making it easier to troubleshoot issues and optimize query performance. Inputs explorer for offline queries ### June 6, 2025 ### New documentation on optimizing and verifying static Python resolvers We've added a new page to our docs that serves as a primer on how Chalk optimizes static Python resolvers. You can verify if your resolvers are being accelerated by checking the query plan page—optimized resolvers are highlighted in yellow and marked as "accelerated". Python resolvers that cannot be optimized will show an error explaining why. We've extended support for accelerating Python expressions that - cast from strings to integers - contain enums that are inherited from other enums - use zip and enumerate built-ins - returns a list of another feature class ``` @online def get_search_results_documents( query_text: SearchQuery.text, ) -> SearchQuery.documents: # requests gets accelerated docs = requests.get(f"{BASE_URL}/vector_search/{query_text}" headers=HEADERS, ) # returning a list of another feature class is also accelerated return [ Document( query_id=query_text, content=doc.content, title=doc.title, similarity_score=doc.similarity_score, metadata=doc.metadata, rank=rank, ) # enumerate built-in is accelerated for rank, doc in enumerate(docs) ] ``` Accelerated Python function ### Easily view all of your charts from the Settings page of Chalk Dashboard We have added a new "All Charts" tab under the "Metric Charts" section of your Settings that displays all charts available in Chalk. The new tab shows: - A complete list of all the charts in your environment - Chart types and categories - The entities or features that each chart is attached to (for example, a link to view the chart) - Any associated alerts All charts page ### May 30, 2025 ### Export user permissions as a CSV You can export information about all users across all environments as a CSV file for easy auditing and compliance reporting. This export, available from the User Permissions section in your Chalk Dashboard, generates a CSV containing each user's name, email address, unique Chalk ID, and their assigned roles for every project-environment combination in your workspace. ### Chalk resource configuration and dashboard improvements - Service isolation, which ensures your workloads run on dedicated infrastructure with 1:1 pod-to-node isolation, is now easily toggle-able using a checkbox in the configuration options. - The instance type selector is now a searchable dropdown that displays machine specifications and validates your resource requests in real-time. If you request more CPUs than the selected instance can provide, you'll see an immediate warning preventing deployment failures and reducing troubleshooting time. - Added a dropdown for selecting which node pools services should run on. When selected, services will be directed to the specified node pools with automatic toleration handling. This prevents services from being assigned to incompatible node types and ensures pods are scheduled correctly on tainted nodes. Resource configuration ### May 23, 2025 ### Embed Chalk expressions in versioned features Feature versions support Chalk expressions, allowing you to define different computation logic for each version of a feature. This makes it easy to experiment with new feature definitions—you can create multiple versions with different expressions and test them side-by-side to compare their accuracy, performance, and business impact. ``` @features class Merchant: id: str fulfilled_orders: int total_orders: int customer_rating: float max_rating: float trust_score: float = feature( default_version=2, versions={ 1: feature( description="Trust score based on orders", expression=_.fulfilled_orders / _.total_orders, ), 2: feature( description="Trust score based on CSAT", expression=(_.customer_rating / _.max_rating), ), } ) ``` ### gRPC support for TypeScript SDK Our gRPC clients are faster than their HTTP counterparts, offering reduced latency and improved query response times. We've published a migration guide for transitioning from the HTTP client to the new gRPC implementation in TypeScript. The function signatures are identical, making for a seamless transition with minimal code changes. Currently, the gRPC client supports query(), queryBulk(), and multiQuery() operations, covering the most common query use cases. ``` import { ChalkGRPCClient } from "@chalk-ai/client" import { FeaturesType } from "local/generated_types"; interface FeaturesType { "user.id": string; "user.fraud_score": number; } const client = new ChalkGRPCClient(); const result = await client.query({ inputs: { "user.id": "1", }, outputs: ["user.fraud_score"], }); console.log(result.data["user.fraud_score"].value); ``` ### May 16, 2025 ### We now support ClickHouse as a Chalk data source Chalk can connect to your ClickHouse database and other data sources with standard SQL syntax. Here's an example of a SQL resolver for pulling transaction data: ``` -- type: online -- resolves: Transaction -- source: clickhouse select transaction_id, transaction_timestamp, transaction_amount, user_id, product_category, transaction_channel, payment_method_type, transaction_status, store_location_region from transactions ``` ### Refreshed our "What is Chalk?" overview and documentation page We've refreshed our introductory documentation with more comprehensive examples and new sections that showcase how Chalk serves different roles (AI Engineers, Data Scientists, and Data Engineers). ### Case study with MoneyLion—a leading fintech for banking, investing, and credit building With Chalk, MoneyLion replaced fragmented Java microservices and manual rewrites with a unified Python-first feature platform. MoneyLion now iterates in hours instead of weeks, replacing their "slow and disjointed" prototype-to-production cycle with a central feature catalog that makes starting "every use case from scratch" a thing of the past. ### May 2, 2025 ### Retrieve historical feature values with SQL Feature values can now be retrieved from historical storage with SQL enabling direct access to previously computed values with the option to fall back to on-demand re-computation when needed. ### Manage traffic spikes with per-pod rate limiting Rate limiting now supports per-pod controls through rate-per-second and concurrency parameters, enabling customers to maintain performance targets even during high traffic periods. The configurable limits help prevent resource exhaustion and ensure consistent application behavior under varying loads. ### Quarterly product update Spring 2025 We've published our quarterly product update, which summarizes improvements and updates from this past Spring, including expanded Python-to-C++ compilation, selective feature persistence controls, and enhanced observability tools. ### April Events Roundup We've published a recap of the various AI and data conferences our team attended in April 2025. The blog post covers our participation at NexGen Banking Summit, VeloxCon, Agents & GenAI Infrastructure Summit, OptimizedAI Conference, and Data Council, including key talks and insights we shared about building infra for real-time ML at scale. ### April 21, 2025 ### Presented at VeloxCon 25 (April 15-16 at Meta HQ) Nathan Fenner presented how Chalk leverages Velox as a common compute engine for both online and offline queries: - Improvements to Velox's expression analysis for low-latency applications - Custom enhancements for avoiding redundant computations and a specialized online hash join - Our symbolic Python interpreter that automatically converts Python feature transformations into efficient Velox expressions Check out the talk (18 mins) on our YouTube page! ### Faster HTTP requests using Chalk Expressions Chalk functions now support direct HTTP requests via the chalk.functions.http module, enabling you to call external APIs with high performance and without the overhead of Python. The HttpResponse type provides access to response data including status codes, bodies, headers, and the final URL after redirects. ``` import chalk.functions as F from chalk import Primary, _ from chalk.features import features from chalk.functions.http import HttpResponse @features class ExternalAPI: id: int api_endpoint: str # GET request returning string response api_response: HttpResponse[str] = F.http_get(_.api_endpoint) status_code: int = _.api_response.status_code response_body: str = _.api_response.body headers: dict[str, str] = _.api_response.headers final_url: str = _.api_response.final_url ``` ### Chalk accelerated Python resolvers are colored as yellow Symbolic Python resolvers accelerated by Chalk are now highlighted in yellow in the query plan viewer (standard Python resolvers remain purple), making it easier to differentiate resolvers that are accelerated or candidates for symbolic execution. ### New offline query explorer added to the Chalk dashboard The offline query explorer includes a new grid/table view for entering multiple input features in a single query, with support for adding, editing, and deleting rows and the option to recompute outputs as needed. The updated interface enhances visibility into these query results through interactive preview panels, comprehensive statistics displays, and detailed metadata exploration. Offline query explorer ### April 4, 2025 ### Chalk developer experience improvements and engineering blog post We've expanded the coverage for the types of Python resolvers that Chalk can statically compile into C++, speeding up computation. Check out our new engineering blog post for a deep dive into how our Symbolic Python Interpreter works. ### Chalk dashboard improvements - Improved how we render stack bar charts within the Chalk dashboard - Can now filter out and explicitly view Scheduled Queries from the Offline Queries page - New "Resolved by" column in the features page clearly displays the source resolver for each feature improving traceability - The Offline Queries page now includes a "Request" button in the details section that displays the parameters used when the query was processed - Added a "Performance" tab when viewing an Online Query that enables inspecting each resolver referenced e.g. SQL query, code, input and output params, and why it may have not been statically accelerated Resolved by column Online Query Performance ### March 26, 2025 ### Case study with Apartment List—a thriving apartment rental marketplace transforming search with real-time personalization With Chalk, Apartment List delivers instant apartment recommendations with real-time price and location flexing, dynamically adapting to user behavior while making low-latency calls to third-party services for the most current pricing information. ### Chalk developer experience improvements Added store_offline and store_online overrides for features with max staleness, making it easy to exclude features from being persisted, particularly helpful for intermediate features that do not need to be saved e.g. very long text. ``` @features(max_staleness="infinity") class Document: id: str file_type: str corpus_text: str = feature(store_offline=False, store_online=False) extracted_feature_from_corpus_text: str ``` ### Chalk dashboard improvements Chalk's Data Explorer now displays the source of each feature e.g. "Live Resolver" Sources in Chalk Data Explorer ### March 14, 2025 ### Breaking changes to Chalk Client's Go SDK (1.2.0) Released v1.2.0 of the Chalk Go Client which introduces some breaking changes, most notably: - The gRPC implementation of Client is removed and replaced with a consolidated NewGRPCClient constructor--configuration remains mostly unchanged. - OnlineQueryBulk removed to prevent inefficient data conversions, migrate to OnlineQueryBulk`.Use GRPCOnlineQueryBulkResult.UnmarshalInto for structured responses. - Most GRPCClient methods now return user-friendly wrappers around proto responses, exposing raw responses via RawResponse. Errors are now lifted properly. ### Chalk developer experience improvements - Singletons can be used to filter DataFrames and has-many relationships. - New Chalk functions: max_by_n and min_by_n Chalk functions: retrieve the top n rows with the maximum and minimum values from a specified column in a DataFrame or has-many relationship, equivalent to sort_by(sort_col, DESC or ASC).head(n)[result_col] - New Chalk functions: array_median, array_average, array_sum, array_stddev - Expanded Python resolver acceleration to support strings manipulation e.g. slices, substrings, and reverses ### Chalk dashboard improvements - Chalk deployments running on AWS now support manually creating and editing node pools via the dashboard, making it easier to organize and manage different groups of worker nodes in a cluster. This helps distribute workloads more efficiently and scale specific groups as needed. - The query plan viewer now displays the static expressions generated to accelerate Python resolvers Static expressions ### February 27, 2025 ### Case study with Verisoul--a leading provider of real-time fake account detection With Chalk, Verisoul iterates on fraud signals, deploys new updates in hours, and increases detection accuracy. ### Easily import feature classes with Chalk Client when experimenting locally We have implemented two new functions for Chalk Client to aid in local experimentation: client.load_features() and client.get_or_create_branch(). - client.load_features() imports your environment's feature classes into the global namespace, making it easier to reference features when running queries with Chalk Client in a Jupyter notebook. - client.get_or_create_branch() enables programmatically creating branches or fetching branches for local experimentation. We've also updated our guide for developing with Jupyter Notebooks. ### Breaking changes to Chalk Client's Go SDK Released v1.0.0 of the Chalk Go Client which introduces some breaking changes, most notably: - Removed custom error structs which helps us standardize how Client errors are bubbled up - Added context.Context to all Client methods enabling us to specify timeouts and custom loggers ### Chalk dashboard improvements - The deployment metrics chart can now be filtered by named (custom) tags - The Chalk dashboard has a new errors page for viewing exceptions and issues with Chalk queries Errors page ### February 20, 2025 ### Persist Datasets to your AWS Glue Catalog ML teams often use offline queries to build training sets by pulling historical data and loading multiple features at specific points in time. We call the results returned from these offline queries Datasets. Chalk supports exporting Datasets back into your AWS Glue catalog, letting other teams discover them and enabling downstream analytical workflows. ``` from chalk.integrations import GlueCatalog dataset = client.offline_query(...) catalog = GlueCatalog( name="aws_glue_catalog", aws_region="us-west-2", catalog_id="123", aws_role_arn="arn:aws:iam::123456789012:role/YourCatalogueAccessRole", ) dataset.write_to(destination="database.table_name", catalog=catalog) ``` ### Chalk dashboard improvements - The Resolvers page in the Chalk dashboard now has a Request Count column; quickly sort and see which resolvers are being called the most - The pages for Named Queries and Resolvers now also include errors, use the "Logs and Errors" tab to switch to this filtered viewClick and drag your mouse within the time series graph to zoom in into a time interval and filter down the errors table Logs and Errors ### February 13, 2025 ### Expressive filtering: DataFrames & Has-Manys now support Chalk Expressions! Has-many (DataFrame) features can now be filtered using any Chalk Expression, unlocking new design patterns and improving DevX! ``` high_cash_flow_merchants: int = _.df[_.id].where(_.total_deposit_amount + F.abs(_.total_withdrawn_amount) > 10**7).count() average_revenue_from_amazon_retailers: float = _.df[_.revenue].where(F.starts_with(_.merchant_code, "amazon-")).mean() publicly_traded_not_tech: int = _.df.where((_.industry != "tech") & (_.is_public == True)).count() holiday_shifts: int = _.df.where(F.is_us_federal_holiday(_.timestamp)).count() expenses_past_month_with_three_day_lag: float = _.expenses[_.amount].where(_.timestamp - timedelta(days=3) > _.chalk_now - timedelta(days=31)).sum() ``` ### Has-many joins using composite keys (Chalk Expressions) A composite key is a combination of two or more attributes that together uniquely identify an entity. Link feature classes with composite keys created by a Chalk Expression or through referencing multiple features in a has-one or has-many join. Here's a has-one relationship between User and Profile classes using both user_id and email as composite join keys. ``` @features class User: id: str = _.alias + "-" + _.org + _.domain org_domain: str = _.org + _.domain org: str domain: str alias: str # join with composite key posts: DataFrame[Posts] = has_many(lambda: User.id == Post.email) # multi-feature join org_profile: Profile = has_one(lambda: (User.alias == Profile.email) & (User.org == Profile.org)) @features class Workspace: id: str # join with child-class's composite key users: DataFrame[Users] = has_many(lambda: Workspace.id == User.org_domain) ``` ### Chalk runtime improvements - Chalk attempts to convert each Python resolver into a chain of static expressions that can be accelerated using the function's AST. We've expanded the parser with preliminary support for accelerating Python for loops. - Improved how we incorporate traffic from Kafka Streams into our (KEDA) autoscaler ### Chalk dashboard improvements - Reduced the chart load times of large offline queries and queries run through the Data Explorer page of the Chalk dashboard - Added support for configuring Chalk query contexts and planner options in the Data Explorer page Data Explorer ### February 06, 2025 ### Chalk dashboard improvements - Added a configuration page for modifying Chalk's connection to your artifact registry e.g. AWS CodeArtifact and Google Artifact Registry - The deployments overview page now includes hyperlinks to the source code associated with each deployment - Added a unified page for viewing scheduled resolvers, queries, and pending backfills - The detailed performance metrics now displays metrics from a query's physical plan e.g. peak memory, input bytes, output rows Scheduled queries ### January 30, 2025 ### Support for embedding models from Vertex AI (GCP) Chalk can automatically create embeddings for vector feature types through Open AI. We've expanded our native embed function to support Vertex, Google Cloud's AI platform. ### Extended support for Now--a Chalk primitive for referencing the current time Now is a Chalk built-in that enables us to incorporate the current time as a filter. Extending the support for Now unlocks new design patterns; build resolvers and queries that take in - multiple ids at a single point in time - a single id at various points in time - multiple ids at various points in time Easily sample historical state like all the transactions for 3 users from a particular date or include the current time when calling out to a microservice. ``` sample_some_transactions = offline_query( input={Transaction.user_id: [103, 150, 170]}, input_times=[datetime(2024, 1, 1, tzinfo=timezone.utc)] * 3, output=[Transaction], recompute_features=True, ) ``` ### Chalk branches can be deployed without requiring a local Chalkpy installation Chalk checks your features for errors locally before deploying them to a remote branch. Normally, if chalkpy (Chalk’s Python SDK) isn’t installed locally, running chalk apply would fail. However, we can now lint branches remotely and show your validation errors in the CLI as usual. ### Breaking change in the 1.0.0 major release of Chalk's Java client This release introduces performance improvements to OnlineQueryResult.unmarshal, but with a few breaking changes: - Unrequested features are now left as null instead of being initialized as empty objects, similarly so for unrequested dataclass features (subclasses of StructFeaturesClass). - OnlineQueryResult.unmarshal now consumes scalarsTable, resulting in getScalarsTable() returning an empty table post unmarshaling. Please create a copy of your results if they need to be retained with .copy(). ### Chalk dashboard improvements - Drag to zoom: Click and drag within a chart or online query to zoom into the selected timeframe; this helps drill down into your charts. - Faster chart loads: We've improved the load times of charts in the Chalk dashboard! - Offline queries that run with explain=True now have a performance summary tab that helps trace query performance. Identify operations with their memory consumption, input size, output size, and whether the operation was blocked before executing. - The Chalk dashboard now supports directly exporting (as Parquet) any view that uses our output explorer: feature previews, datasets, offline queries, and online queries that get persisted to the offline store. Profiling an offline query ### January 22, 2025 ### Chalk's C++ SQL Driver for Spanner now supports inputs (parameterized queries) Chalk’s C++ driver now supports Spanner queries that explicitly accept input parameters. These resolvers now run faster, with some query latencies dropping from 24ms to 10ms. ### Enhanced visibility into your Chalk deployment The Chalk query plan viewer now highlights nodes for Chalk expressions and SQL resolvers using Chalk's C++ driver in orange. Differentiate between various node types at a glance to debug Chalk queries in less time. The Kubernetes resource page in the Chalk dashboard has been enhanced to include data from the Kubernetes events API, increasing visibility into nodes, pods, and more objects to come. Expanding the types of events and their granularity provides better insights into jobs. These include events like node removal, pod scheduling, and deployment failures. The query plan viewer in the Chalk dashboard now supports breadcrumbs to hyperlink to a feature’s parent in the query plan viewer, enabling quicker traversal through a feature’s lineage. Colored nodes in Chalk's query plan viewer ### January 15, 2025 ### Improved the granularity at which Chalk services can be rolled back We improved our CI/CD to support releasing new Chalk images in parallel. We also extended both the quantity and types of builds (across various Chalk microservices) that we cache, improving the speed and flexibility of Chalk rollbacks! ### Kubernetes node UI enhanced for readability We improved the Kubernetes Node UI to display additional data about the nodes in your cluster and polished the UI for compactness and readability. ### Chalk dashboard bubbles up feature value metrics The feature section of the Chalk dashboard will now show salient metrics for your top feature values. The number of observations, when features were observed, the most observed features, and more. Metrics can be sliced and grouped by filters like deployment ID, resolver, and operation type, which helps in debugging (at a glance insight into) deployments. ### Query plan viewer now supports node lookups The Chalk dashboard now also supports query plan node look ups via resolver name, node type, and the input/output features associated with the node. This makes it easier to trace the behavior of a resolver in complex query plans. Feature metrics view ### January 6, 2025 ### Optimized resource usage with KEDA We've implemented KEDA (Kubernetes-based Event Driven Autoscaler) and are rolling it out to all customer clusters over the next few weeks. With KEDA, we've expanded the types of resources that can be dynamically scaled up and down during periods of reduced or no workload. We look forward to potential improvements both in the elasticity and efficiency of your deployments. ### View and query logs using the CLI Inspect your log stream the same way you would from the Chalk dashboard but with the command line: chalk logs [--query="the same string you can put in the web ui"] Read more about how to view logs using the Chalk CLI in our docs. ### Visualize and trace your feature's data lineage with a Graph View The Data Lineage page in the Chalk dashboard lets you track your features, specifically how they are derived, when they are referenced (Named Queries), etc. In addition to the default list view, we have implemented a graph view, making it easier than ever to trace how your features are created and connected. Graph view for a data lineage ### December 23, 2024 ### Search and filter logs by query name, log message, deployment, and more! You can now search logs in the Logs Explorer in the dashboard using a key-value pair query format: key1:value1 key2:value2 .... This allows you to combine multiple filters at once. The Logs Explorer displays the available keywords for filtering, shown below. Updated log explorer ### Search by file name in the source code viewer The Source Code Viewer on the Deployments page of the dashboard has been extended! In addition to searching for keywords in your source code, you can now also search for your files by name. Source code viewer with file search ### Include query context in Chalk Client queries The Chalk Client now offers a query_context field that can be accessed by Python resolvers, allowing you to pass additional information in your queries. The query_context is available for online queries, bulk online queries, and offline queries for both ChalkClient and ChalkGRPCClient, and can be tested using ChalkClient.check. ### Expressions support more string, encoding, and date functions The chalk.functions library now supports more string, encoding, and date functions that can be used in Chalk expressions. You can now use jaro_winkler_distance to compute the Jaro-Winkler distance between two strings, partial_ratio to compute the fuzzywuzzy partial ratio between two strings, and length to compute the length of a string. We've also added struct_pack to construct a struct from a mapping, year to extract the year from a date, and last_day_of_month to return the last date in the month given a date. To read more about all of the chalk.functions available for writing expressions, check out our API docs. ### December 16, 2024 ### Source code viewer in dashboard supports text search The source code viewer in the Deployments page of the dashboard now supports fuzzy text search across the source code for the linked deployment. Source code viewer with text search ### Dataset revision metadata get and set DatasetRevision objects now have get_metadata and set_metadata methods to get and set metadata as a dictionary. This can be useful for storing information such as whether a dataset has been ingested to the online or offline store, or for tagging and labeling datasets. Read more about all of the DatasetRevision methods in our API documentation. ### Expressions support more array functions The chalk.functions library now supports more array functions that can be used in expressions. array_max and array_min return the maximum and minimum values in an array, respectively. array_distinct returns a list of the distinct elements in an array, and array_sort sorts the elements of an array by ascending or descending order. Read more about all of the chalk.functions available for writing expressions in our API docs. ### December 9, 2024 ### Expressions support struct field access and more functions You can now reference dataclass struct fields with expressions. We've also added more functions to chalk.functions that can be used in expressions, including is_us_federal_holiday for determining if a date is a federal holiday in the United States, array_sort for sorting the elements of an array by ascending or descending order, and element_at for retrieving the element at a specified index in an array. To read more about all of the chalk.functions in your toolbox for writing expressions, check out our API docs. ### New overview page and dashboard updates for viewing stack traces and recent feature values We've updated the overview page in the dashboard to show an overview of query activity and cluster health for your environment. You can now view graphs and metrics for online and offline queries, resolver runs, query latency, online store activity, deployments, incidents, errors, and connections. Please reach out to the Chalk team if you have any questions about any of these metrics Overview page in dashboard In addition, you can now view the latest stack trace for a pod in your cluster under Settings > Kubernetes, which can be helpful for debugging offline queries or scheduled runs. Stack trace for pods in the cluster Customers with a gRPC engine can also use the new Values Preview tab on the Features page to view recently observed feature values as loaded from the offline store. Values preview for features in dashboard ### Postgres Native SQL driver improvements The Postgres Native SQL driver now supports parameters, as well as returning relationships and scalar aggregations, expanding the kinds of SQL resolvers querying Postgres sources that we can support with performant C++ execution. ### Read our updated documentation on windowed aggregations We've updated our documentation on windowed aggregations to include more information on how to fuse real-time aggregations with materialized pre-aggregations for windowed features. ### December 2, 2024 ### We now support Poetry for managing Python dependencies You can now use Poetry to manage Python dependencies in your Chalk project. Simply set the requirements field in your chalk.yaml file to pyproject.toml to use Poetry for dependency management. Read more about how to use configuration files here. ``` project: no-fun-project-names environments: default: runtime: 'python310' requirements: pyproject.toml ``` ### Expressions now support more datetime, JSON, string, and math functions We've added more functions to chalk.functions that can be used in expressions. You can now use json_extract_array to extract arrays of strings, bools, numbers, or nulls from a JSON feature, safe_divide to divide two numbers safely, returning None if the denominator is zero, to_iso8601 to convert a datetime feature to an ISO 8601 string, from_unix_seconds and from_unix_milliseconds to convert a Unix timestamp in seconds or milliseconds to a datetime feature, and finally regexp_replace to replace substrings in a string using a regular expression. To read more about all of the chalk.functions in your toolbox for writing expressions, check out our API docs. ### Warning banners in dashboard for deployments We've added warning banners in the dashboard to alert you to potential issues in your deployments, such as when pods fail to start up cleanly. These banners will help with monitoring and observability for deployment health. Warning banners in dashboard ### November 25, 2024 ### Offline query now accepts parquet inputs and relative input times Offline queries can now accept parquet files as input by passing in a ”s3://“ or ”gs://“ URL as the offline_query(input=“…”) parameter. In addition, you can now define upper and lower bounds for offline queries with a timedelta that is set relative to the min/max input times per shard. For example, you can now set offline_query(lower_bound=timedelta(days=-30) to set the lower bound to be 30 days before the earliest input_time or offline_query(upper_bound=timedelta(days=30) to set the upper bound to be 30 days after the latest input_time. To read more about offline query parameters, see our API docs. ### Backfill materialized window aggregations over computed features You can now define a materialized aggregate backfill over features that are returned by multiple resolvers rather than just one resolver. Read more about the chalk aggregate backfill command in our CLI docs. ### Expressions now support more array and encoding functions We've added more functions to chalk.functions that can be used in expressions. You can now use array_join to concatenate array elements into a string, cardinality to count the number of elements in an array, as well as max and min to find the maximum and minimum values, respectively, in an array. You can also use max_by and min_by to find the row with the maximum and minimum values in a given column in a DataFrame. For mathematical operations, you can now round numbers and use from_base for base conversion. For encoding, you can now compute the sha1, sha256, and sha512 hashes of strings. Finally, you can use format_datetime to format datetime features using a format string and strpos to find the first position of a substring in a string. To read more about all of the chalk.functions in your toolbox for writing expressions, check out our API docs. ### Optionally cache nulls or default feature values in the online store Where you could previously choose whether to cache null feature values in the online store, you can now use the cache_defaults parameter to optionally cache default feature values in the online store. Customers with DynamoDB or Redis online stores can also choose to evict_defaults from the online store, evicting the entry that would have been a default value from the online store if it exists. Read more about how to use cache_defaults and cache_nulls in our API docs. ### Define features as map types You can now define features as map types and reference these map features in expressions. ``` from chalk.features import features @features class User: id: str user_preferences: dict[str, bool] = {"Advertising cookies": True, "Functional cookies": True} ``` ### Retrieve map types from DynamoDB data sources as either dicts or strings You can now retrieve Map document types from DynamoDB data sources as either dicts or strings, using the new map feature type. ### Dashboard updates for viewing Kubernetes logs We’ve exposed much more granularity for Kubernetes logs in the dashboard. You can now filter logs by pod name, component, resource group, and deployment. These filters are set in the page query params, which means that you can send a link to pre-filtered log views. The links that are newly exposed in the Kubernetes pod view now directs to a pre-filtered logs viewer page. Finally, you can now view pod conditions in the Kubernetes pod view, which can include useful information for debugging failed or pending pods. Detailed Kubernetes logs in dashboard ### Improved stacktrace rendering in Chalk CLI We've drastically improved the rendering of stacktraces in the Chalk CLI with improved syntax highlighting and clearer formatting to make debugging easier. ### November 18, 2024 ### View Kubernetes resources created by each deployment in the dashboard You can now view the Kubernetes pods created by each deployment in the dashboard along with additional details like the pod states and the resources requested by each pod. Pod Resources in Deployment Page ### Resolve list features with expressions We've added the array_agg function to chalk.functions to help you resolve list features with underscore expressions. For example, you can now express the following features to aggregate all categories of videos watched for a user. ``` import chalk.functions as F from chalk import DataFrame from chalk.features import _, features @features class VideoInteraction: id: str video_url: str video_category: str user_id: "User.id" @features class User: id: str videos_watched: DataFrame[VideoInteraction] all_watched_video_categories: list[str] = F.array_agg(_.videos[_.category]) ``` To see all of the chalk.functions that you can use in expressions, see our API documentation. ### View usage information through the Chalk CLI Users can now use the chalk usage commands to view usage information for their projects and environments. If you have any questions, please reach out to the Chalk team. ### November 11, 2024 ### Idempotency in triggered resolver runs We now provide an idempotency key parameter for triggering resolver runs so that you can ensure that only one job will be kicked off per idempotency key provided. ### ChalkClient.check() function for easy integration testing The ChalkClient now has a check function that enables you to run a query and check whether the query outputs match your expected outputs. This function should be used with pytest for integration testing. To read more about different methods and best practices for integration testing, see our integration test docs. ### Expressions support more mathematical and logical functions This week, we've added mathematical functions floor, ceil, and abs to chalk.functions, along with the logical functions when, then, otherwise, and is_null. We've also added the haversine function for computing the Haversine distance between two points on Earth given their latitude and longitude. These points can be used in expressions to define features with code that can be statically compiled in C++ for faster execution. See the full list of functions you can use in underscore expressions in our API docs. ### Dashboard improvements for providing more insights into resolver performance and execution In the dashboard, users can now view the P50, P75, P95, and P99 latencies for resolvers in the table under the Resolver tab of the menu. You can also customize which columns are displayed in the table by clicking the gear icon in the top left corner of the table. In addition, we've added a SQL Explorer for examining resolver output for queries that are run with the store_plan_stages=True parameter. ### chalk healthcheck in CLI You can now use the chalk healthcheck command in the CLI to view information on the health of Chalk's API server and its services. The healthcheck provides information for the API server based on the active environment and project. To read more about the healthcheck command, see the CLI documentation. ### November 4, 2024 ### Offline Query Specifications for shards and workers When running an asynchronous offline query, you can now specify the inputs num_shards and num_workers as parameters to allow for more granular control over the parallelization of your query execution. To see all of the offline query options, check out the offline query documentation. In addition, offline query progress reporting now specifies progress by shard, giving developers more insight into where their offline query is in the execution progress. ### ChalkClient can now use the default Git branch You can now default to using the name of your current Git branch when developing using the ChalkClient. For example, if you have checked out a branch named my-very-own-branch you can now set ChalkClient(branch=True) and all of your client calls will be directed at my-very-own-branch. To read more about how to use ChalkClient, see our API documentation. ### Expressions support functions for URL parsing, regular expressions, and more We've added more functions to chalk.functions that can be used in expressions. You can now use regexp_like, regexp_extract, split_part, and regexp_extract_all to do regular expression matching and use url_extract_host, url_extract_path, and url_extract_prtocol to parse URL's. In addition, we've added helpful logical functions like if_then_else, map_dict, and cast to broaden the span of features that you can define using expressions. To read more about all of our functions, check out our API documentation. ### Deployment build logs for AWS environments We now provide more detailed build logs for deployments in AWS environments in the dashboard! ### October 28, 2024 ### Run predictions against SageMaker from Chalk, and do so much more in expressions We've added a new function chalk.functions.sagemaker_predict that allows you to run predictions against a SageMaker endpoint to resolve features. Read more about how to define a SageMaker endpoint, encode your input data, and run predictions in our SageMaker tutorial. In addition to being able to make SageMaker calls, expressions now support a variety of new functions. With these functions imported from chalk.functions, you can perform encoding, decoding, math, datetime manipulation, string manipulation, and more! For example, say you have a Transaction feature, where you make a SageMaker call to enrich the transaction data and provide a label for the transaction, and you parse this label for other features. You can now define all of these features related to transaction enrichment using expressions and Chalk functions in the feature definition: ``` from datetime import date import chalk.functions as F from chalk.features import _, features, Primary @features class Transaction: id: str amount: float date: date day: int = F.day_of_year(_.date) month: int = F.month_of_year(_.date) sagemaker_input_data: bytes = F.string_to_bytes(_.id, encoding="utf-8") transaction_enrichment_label: bytes = F.sagemaker_predict( _.sagemaker_input_data, endpoint="transaction-enrichment-model_2.0.2024_10_28", target_model="model_v2.tar.gz", target_variant="production_variant_3" ) transaction_enrichment_label_str: str = F.bytes_to_string(_.transaction_enrichment_label, encoding="utf-8") is_rent: bool = F.like(_.transaction_enrichment_label_str, "%rent%") is_purchase: bool = F.like(_.transaction_enrichment_label_str, "%purchase%") ``` ### Nested materialized windowed aggregation references! You can now reference other windowed aggregations in your windowed aggregation expressions. To read more about how to define your windowed aggregations, see our example here. ### Updated usage dashboard to view CPU and storage requests grouped by pod and namespace We've updated the Usage Dashboard with a new view under the Pod Resources tab that allows you to view CPU and storage requests by pod as grouped by cluster, environment, namespace, and service! If you have any questions about the usage dashboard, please reach out to the Chalk team. ### Dropping support for Python 3.8 From chalkpy==2.55.0, Chalk is dropping support for Python 3.8, which has reached end-of-life. If you are still using Python 3.8, please upgrade to Python 3.9 or higher. ### October 21, 2024 ### Pub/Sub streaming source We've enabled support for using Pub/Sub as a streaming source. Read more about how to use Pub/Sub as a streaming source here. ### Online/Offline Storage for Offline Queries You can automatically load offline query outputs to the online and offline store using the boolean parameters store_online and store_offline. Below is an example of how to use these parameters. ``` from chalk.client import ChalkClient client = ChalkClient() ds = client.offline_query( input={"user.id": [1, 2, 3, 4, 5]}, output=["user.num_interactions_l7d", "user.num_interactions_l30d", "user.num_interactions_l90d"], store_online=True, store_offline=True ) ``` ### SQL explorer for query outputs in the dashboard Customers running gRPC servers can now run SQL queries on the dataset outputs of online and offline queries in the dashboard. To enable this feature for your deployment, please reach out to the team. SQL Explorer for Query Outputs ### Color updates in the dashboard We've updated our color scheme in the dashboard to more clearly differentiate between successes and failures in metrics graphs! Red for failures and green for successes ### October 14, 2024 ### SQL explorer in dashboard for datasets Customers can now run SQL queries on dataset outputs in the dashboard. To use this feature, navigate to the Datasets page in the menu, select a dataset, and click on the Output Explorer tab. SQL Explorer for Datasets ### Optionally evict nulls from your DynamoDB online store Last week we enabled the option to decide whether to persist null values for features in Redis lightning online stores, and this week we have enabled this feature in DynamoDB online stores. By default, null values are persisted in the online store for features defined as Optional, but you can set cache_nulls=False in the feature method to evict null values. Read more about how to use the cache_nulls parameter here. ### Set environment variables and more in the Advanced section of the Cloud Resource Configurations page in the dashboard You can set cloud resource configurations for your environment by navigating to Settings > Resources in the dashboard. In addition to specifying resource configurations for resource groups like instance counts and CPU, you can now also set environment variables and other settings like Kubernetes Node Selectors. The Kubernetes Node Selector enables you to specify the machine family you would like to use for your deployment. For example, this would map to EC2 Instance Types for AWS deployments or Compute Engine Machine Families for GCP deployments. If you have any questions about how to use any of these settings in the configuration page, please reach out to the team. Cloud Resource Configurations Advanced Settings ### October 7, 2024 ### Expressions support datetime subtraction and total_seconds Expressions now support datetime subtraction and the use of a new library function chalk.functions.total_seconds. This allows you to compute the number of seconds in a time duration and define more complex time interval calculations using performant underscore expressions. For example, to define a feature that computes the difference between two date features in days and weeks, we can use chalk.functions.total_seconds and underscore date expressions together. ``` from chalk.functions as F from chalk.features import _, features, Primary from datetime import date @features class User: id: str created_at: date last_activity: date days_since_last_activity: float = F.total_seconds(date.today() - _.last_activity) / (60 * 60 * 24) num_weeks_active: float = F.total_seconds(_.last_activity - _.created_at) / (60 * 60 * 24 * 7) ``` ### Optionally evict nulls from your Redis lightning online store You can now select whether to persist null values for features in the Redis lightning online store using the cache_nulls parameter in the feature method. By default, null values are persisted in the online store for features defined as Optional. If you set cache_nulls=False, null values will not be persisted in the online store. ``` from chalk import feature from chalk.features import features, Primary, Optional @features class RestaurantRating: id: str cleanliness_score: Optional[float] = feature(cache_nulls=False) # null values will not be persisted service_score: Optional[float] = feature(cache_nulls=True) # null values will be persisted. This is the default behavior. overall_score: float # null values are not persisted for required features ``` ### Feature value metrics from gRPC server and feature table updates Customers running the gRPC server can now reach out to enable feature value metrics. Feature value metrics include the number of observations, number of unique values, and percentage of null values over all queries, as well as the running average and maximum of features observed. Please reach out if you'd like to enable feature value metrics. Feature value metrics Additionally, the feature table in the dashboard has been updated to allow for customization of columns displayed, which enables viewing request counts over multiple time ranges in the same view. ### September 30, 2024 ### Compute cosine similarity between two vector features chalk.functions now offers a cosine_similarity function: ``` import chalk.functions as F from chalk.features import _, embedding, features @features class Shopper: id: str preferences_embedding: Vector[1536] @features class Product: id: str description_embedding: Vector[1536] @features class ShopperProduct: id: str shopper_id: Shopper.id shopper: Shopper product_id: Product.id product: Product + similarity: float = F.cosine_similarity(_.shopper.preferences_embedding, _.product.description_embedding) ``` Cosine similarity is useful when handling vector embeddings, which are often used when analyzing unstructured text. You can also use embedding to compute vector embeddings with Chalk. ### Dashboard now shows metrics for offline query runs When looking at an offline query run in the dashboard, you'll now find a new Metrics tab showing query metadata, CPU utilization, and memory utilization. ### Configuration option for recomputing features on cache misses We have a new offline query configuration option for recomputing features only when they are not already available in the offline store. This option is useful for workloads with computationally expensive features that cannot easily be recomputed. Please reach out if you'd like to try this feature. ### September 23, 2024 ### Configurable retry policy for SQL resolvers Sometimes, a SQL resolver may fail to retrieve data due to temporary unavailability. We've added new options for configuring the number of retry attempts a resolver may make (and how long it should wait between attempts). If you're interested in trying out this new functionality early, please let the team know. ### Has-one join keys can be chained When creating has-one relationships, you can set the primary key of the child feature class to the primary key of the parent feature class. For example, you may model an InsurancePolicy feature class as belonging to exactly one user by setting InsurancePolicy.id's type to Primary[User.id]. Now, we've updated Chalk so that you can chain more of these relationships together. For example, an InsurancePolicy feature class may have an associated InsuranceApplication. The InsuranceApplication may also have an associated CreditReport. Chalk now allows chaining an arbitrary number of has-one relationships. Chalk will also validate these relationships to ensure there are no circular dependencies. Here's an example where we have features describing a system where user has one insurance policy, each policy has one submitted application, and each application has one credit report: ``` from chalk import Primary from chalk.features import features @features class User: id: str # :tags: pii ssn: str policy: "InsurancePolicy" @features class InsurancePolicy: id: Primary[User.id] user: User + application: "InsuranceApplication" + @features + class InsuranceApplication: + id: Primary[InsurancePolicy.id] + stated_income: float + # For the sake of illustrating has-one relationships, + # we're assuming exactly one credit report per + # application, which may not be realistic. A has-many + # relationship may be more accurate here. + credit_report: "CreditReport" + + @features + class CreditReport: + id: Primary[InsuranceApplication.id] + fico_score: int + application: InsuranceApplication ``` To query for a user's credit report, you would write: ``` client.query( inputs={User.id: "123"}, output=[User.policy.application.credit_report], ) ``` To write a resolver for one of the dependent feature classes here, such as CreditReport.fico_score, you would still reference the relevant feature class by itself: ``` @online def get_fico_score(id: CreditReport.id) -> CreditReport.fico_score: ... ``` As an aside, if your resolver depends on features from other feature classes, such as User.ssn, we instead recommend joining those two feature classes directly for clarity (which was possible prior to this changelog entry): ``` from chalk import Primary - from chalk.features import features + from chalk.features import features, has_one @features class User: id: str # :tags: pii ssn: str policy: "InsurancePolicy" + credit_report: "CreditReport" = has_one(lambda: User.id == CreditReport.id) # ... the rest of the feature classes + @online + def get_fico_score(id: User.id, ssn: User.ssn) -> User.credit_report.fico_score: + ... ``` ### User permissions page shows roles per user When you view Users in the Chalk settings page, you will now find a menu for viewing the roles associated with each user, whether those roles are granted directly or via SCIM. An example user with admin and owner roles ### September 16, 2024 ### New feature and resolver UI in the dashboard We have shipped a new UI for the Features and Resolvers sections of the dashboard! The new UI has tables with compact filtering and expanded functionality. You can now filter and sort by various resolver and feature attributes! The tables also provide column resizing for convenient exploration of the feature catalog. Resolver table with compact filtering and sorting The features table now includes request counts from the last 5 minutes up to the last 180 days, has built-in sorting, and has a Features as CSV button to download all the feature attributes in your table as a CSV for further analysis. Feature table with request count and csv export button ### New helper functions for feature computation The new chalk.functions module contains several helper functions for feature computation. For example, if you have a feature representing a raw value in GZIP-compressed value, you can use gunzip with an underscore reference to create an unzipped feature. The full list of available functions can be found in our expression documentation. ### JSON feature type You can now define features with JSON as the type after importing JSON from the chalk module. You can then reference the JSON feature in resolver and query definitions. You can also retrieve scalar values from JSON features using the json_value function. ### September 9, 2024 ### Configure Chalk to not cache null feature values By default, Chalk caches all feature values, including null. To prevent Chalk from caching null values, use the feature method and set cache_nulls to False. ### More static execution of certain Python resolvers We built a way to statically interpret Python resolvers to identify ones that are eligible for C++ execution, which has faster performance. For now, resolvers are eligible if they do simple arithmetic and logical expressions. If you're interested in learning more and seeing whether these new query planner options would apply to your codebase, please reach out! ### New tutorial for using Chalk with SageMaker We have a new tutorial for using Chalk with SageMaker available now. In the tutorial, we show how to use Chalk to generate training datasets from within a SageMaker pipeline for model training and evaluation. ### September 3, 2024 ### Feature catalog shows associated named queries In the August 19 changelog entry, we announced NamedQuery, a tool for naming your queries so that you can execute them without writing out the full query definition. This week, we've updated the dashboard's feature catalog so that it shows which named queries reference a given feature as input or output. Feature catalog showing links to named queries a feature is an input or output of ### August 26, 2024 ### View aggregation backfills in the dashboard We added a new Aggregations page to the dashboard where you can see the results of aggregate backfill commands. Check it out to see what resolvers were run for a backfill, the backfill's status, and other details that will help you drill down to investigate performance. For more details on aggregate backfills, see our documentation on managing windowed aggregations. ### August 19, 2024 ### Execute queries by name Instead of writing out the full definition of your query each time you want to run it, you can now register a name for your query and reference it by the name! Here's an example of a NamedQuery: ``` from chalk import NamedQuery from src.feature_sets import Book, Author NamedQuery( name="book_key_information", input=[Book.id], output=[ Book.id, Book.title, Book.author.name, Book.year, Book.short_description ], tags=["team:analytics"], staleness={ Book.short_description: "0s" }, owner="mary.shelley@aol.com", description=( "Return a condensed view of a book, including its title, author, " "year, and short description." ) ) ``` After applying this code, you can execute this query by its name: ``` chalk query --in book.id=1 --query-name book_key_information ``` To see all named queries defined in your current active deployment, use chalk named-query list. As Shakespeare once wrote, "What's in a named query? That which we call a query by any other name would execute just as quickly." ### Miscellaneous improvements - The offline query page of the dashboard now shows which table in your offline store contains the query's output values. ### August 12, 2024 ### Queries can reference multiple feature namespaces Previously, you could only reference one feature namespace in your queries. Now you can request features from multiple feature namespaces. For example, here's a query for a specific customer and merchant: ``` client.query( input={ Customer.id: 12345, Merchant.id: 98765, }, output=[Customer, Merchant], ) ``` ### Dashboard resources view shows allocatable CPU and memory The resources page of the dashboard now shows the allocatable and total CPU and memory for each of your Kubernetes nodes. Kubernetes reserves some of each machine's resources for internal usage, so you cannot allocate 100% of a machine's stated resources to your system. Now, you can use the allocatable CPU and memory numbers to tune your resource usage with more accuracy. ### Performance improvements We identified an improvement for our query planner’s handling of temporal joins! Our logic for finding the most recent observation for a requested timestamp is now more efficient. Happy time traveling! ### August 5, 2024 ### DynamoDB with PartiQL We now support DynamoDB as a native accelerated data source! After connecting your AWS credentials, Chalk automatically has access to your DynamoDB instance, which you can query with PartiQL. ### Expressions support references to the target window duration Expressions on windowed features can now include the special expression _.chalk_window to reference the target window duration. Use _.chalk_window in windowed aggregation expressions to define aggregations across multiple window sizes at once: ``` @features class Transaction: id: int user_id: "User.id" amount: float @features class User: id: int transactions: DataFrame[Transaction] total_spend: Windowed[float] = windowed( "30d", "60d", "90d", default=0, expression=_.transactions[_.amount, _.ts > _.chalk_window].sum(), materialization={"bucket_duration": "1d"}, ) ``` ### Offline queries allow resource overriding - offline_query now supports the resources parameter. resources allows you to override the default resource requests associated with offline queries and cron jobs so that you can control CPU, memory, ephemeral volume size, and ephemeral storage. ### July 26, 2024 ### Dashboard improvements - The offline query page of the dashboard now shows live query progress. After query completion, the query page will also show how long each resolver took to run." - The Kubernetes resource page in the dashboard shows which kinds of hardware resources are currently running. It also allows you to group resources by application, component, and other common groupings. ### July 19, 2024 ### Datasets and dataset revisions now support previews and summaries Datasets and DatasetRevisions have two new methods: preview and summary. preview shows the first few rows of the query output. summary shows summary statistics of the query output. Here's an example of summary output: ``` describe user.id ... __index__ shard_id batch_id 0 count 1.0 ... 1.0 0 0 1 null_count 0.0 ... 0.0 0 0 2 mean 1.0 ... 0.0 0 0 3 std 0.0 ... 0.0 0 0 4 min 1.0 ... 0.0 0 0 5 max 1.0 ... 0.0 0 0 6 median 1.0 ... 0.0 0 0 [7 rows x 14 columns] ``` ### Create isolated node pools for your resource groups (in AWS) Chalk resource groups create separate independent deployments of the query server to prevent resource contention. For example, one team may want to make long-running analytics queries and another may want to make low-latency queries in line with customer requests. We have updated the Cloud Resource Configuration page! You can now configure resource groups to use completely independent node pools to ensure your workflows run on separate computer hardware. The configuration page also allows you to specify exactly what kind of hardware will be available in each resource group so you can optimize the balance between cost and performance. This feature is currently available for customers running Chalk in EKS, but will be available soon for customers using GKE. ### Performance improvements - We've significantly improved SQL runtime in our query planner by executing eligible queries in C++ instead of SQLAlchemy. Chat with our support team if you'd like to update your query planner options. - We improved the performance of some expressions by executing count() operations as native dataframe operations. ### Miscellaneous improvements - Our feature catalog now lets you filter features by their context (online or offline). Additionally, you can now search features by their name, description, and owner. - We fixed an issue where some underscore expressions had incorrect typechecking. ### July 10, 2024 ### Feature catalog You can now view and filter features in the feature catalog by their tags and owners. Feature catalog filtering by tag and owner ### July 1, 2024 ### Chalk gRPC We shipped a gRPC engine for Chalk that improved performance by at least 2x through improved data serialization, efficient data transfer, and a migration to our C++ server. You can now use ChalkGRPCClient to run queries with the gRPC engine and fetch enriched metadata about your feature classes and resolvers through the get_graph method. ### Spine SQL query With ChalkPy v2.38.8 or later, you can now pass spine_sql_query to offline queries. The resulting rows of the SQL query will be used as input to the offline query. Chalk will compute an efficient query plan to retrieve your SQL data without requiring you to load the data and transform it into input before sending it back to Chalk. For more details, check out our documentation. ### Static planning of expressions We shipped static planning of expressions. expressions enable you to define and resolve features from operations on other features. When you use expressions, we now do static analysis of your feature definition to transform it into performant C++ code. Expressions currently support basic arithmetic and logical operations, and we continue to build out more functionality! See the code snippet below for some examples of how to use expressions: ``` @features class SampleFeatureSet: id: int feature_1: int feature_2: int feature_1_2_sum: int = _.feature_1 + _.feature_2 feature_1_2_diff: int = _.feature_1 - _.feature_2 feature_1_2_equality: bool = _.feature_1 == _.feature_2 ``` ### June 28, 2024 ### Chalk deployment tags You can now add tags to your deployments. Tags must be unique to each of your environments. If you add an already existing tag to a new deployment, Chalk will remove the tag from your old deployment. Tags can be added with the --deployment-tag flag in the Chalk CLI: ``` chalk apply --deployment-tag=latest --deployment-tag=v1.0.4 ``` ### Resource configuration management in dashboard We updated our UI for resource configuration management in the dashboard! You can now toggle your view between a GUI or a JSON editor. The GUI exposes all the configuration options available in the JSON editor, including values that aren't set, and allows you to easily adjust your cluster's resources to fit your needs. resource configuration management ### June 19, 2024 ### New data sources and native drivers We added integrations for Trino and Spanner as new data sources. We've also added native drivers for Postgres and Spanner, which drastically improves performance for these data sources. ### May 29, 2024 ### Heartbeating We now have heartbeating to poll the status of long-running queries and resolvers, which will now mark any hanging runs that are no longer detected as "failed" after a certain period of time. ### May 14, 2024 ### Data source and feature-level RBAC We expanded the functionality of our service tokens to enable role-based access control (RBAC) at both the data source and feature level. On the datasource level, you can now restrict a token to only access data sources with matching tags to resolve features. On the feature level, you can restrict a token's access to tagged features either by blocking the token from returning tagged features in any queries but allowing the feature values to be used in the computation of other features, or by blocking the token from accessing tagged features entirely. datasource and feature level rbac ### May 8, 2024 ### Incremental Status We shipped statuses during incremental runs such that users can get a signal of the current high water mark of data being updated. ``` chalk incremental status --scheduled_query get_some_data__daily ✓ Fetched resolver progress state Resolver: N/A Query: run_this_query_daily Environment: chalk12345 Max Ingested Timestamp: 2024-07-01T16:01:46+00:00 Last Execution Timestamp: 2024-07-01T00:01:27.421873+00:00 ``` ### April 18, 2024 ### Miscellaneous improvements - Windowed resolvers have expanded to allow for hourly cadences. ### April 9, 2024 ### Miscellaneous improvements - SQL resolvers have improved error reporting for failures related to type conversion (e.g., if your resolver selects an int column, but the feature's type is string) ### March 29, 2024 ### Miscellaneous improvements - SQL file resolvers have spellcheck (based on Levenshtein distance) - Failed annotation parsing raises a type error with a more helpful error message ### March 19, 2024 ### Scheduled Queries Chalk now supports executing an offline_query on a schedule. Effectively, this extends the existing "scheduled resolver" functionality and allows you to execute more complicated data ingestion or caching workflows without needing to use Airflow or other external schedulers to orchestrate resolver execution. Here's an example of a scheduled query that caches the number of transactions a user has made in the last 24 hours into the online store: ``` from chalk import ScheduledQuery ScheduledQuery( name="num_transactions_last_24h", output=[User.num_transactions_last_24h], schedule="0 0 * * *", # every day at midnight store_online=True, # store the result in the online store store_offline=False, # don't store this value in the offline store ) ``` ### Bugfixes and improvements - offline_query(...) now accepts sample_features: list[Feature] as an argument. This works in conjunction with recompute_features, and allows you to write something like: ``` ChalkClient().offline_query( input={User.id: [...]}, output=[User.full_name], recompute_features=True, # means "recompute all features sample_features=[User.first_name, User.last_name] # but sample these features from the offline store ) ``` This is useful when you have a large number of features that you want to recompute, but only a few that you want to sample. - ChalkClient.offline_query now accepts run_asynchronously: bool to explicitly opt a query into running on an isolated worker. - DataSet.to_polars()/.to_pandas() now accept output_ts: str and output_id: str to customize the name of the timestamp and id columns in the output dataframe. - Feature and resolver discovery during chalk apply is roughly twice as fast as of chalkpy v2.33.9. - Dataset downloads no longer have any dependency on locally registered features, which resolves crashes for certain dataset management workflows. - ChalkClient.query now supports request_timeout: float, which is passed to the underlying requests.request call. ### March 8, 2024 ### Bugfixes and improvements - A persistent issue with chalk drop has been resolved. Now, chalk drop will allow you to reset a feature whose deletion has been deployed to the active deployment, which will allow you to re-deploy the feature. Previously, it was possible to get into a state that was impossible to recover from without support. - tags(...) allows you to extract the tags of a @features class or a property (Feature) of that class. - DataSet.to_polars()/to_pandas() now raises an error if the dataset computation had errors. This prevents the user from accidentally using a dataset that was not computed correctly. If you wish to use the dataset anyway, you can use DataSet.to_polars(ignore_errors=True). ### March 1, 2024 ### Support for custom SQL sampling in offline query You can now specify a custom SQL sampling query for offline queries. This allows you to use a native SQL query to compute the query's entity spine for offline queries. This is useful when you have a complicated sampling policy (i.e. class-based sampling). Additional non-primary key features can be provided as well. ### January 22, 2024 ### required_resolver_tags for queries You can now specify required_resolver_tags when querying. This allows you to ensure that a query only considers a resolver if it has a certain tag. This is useful for guaranteeing that a query only uses resolvers that are cost-efficient, or for enforcing certain compliance workflows. In this example: ``` @offline() def fetch_credit_scores() -> DataFrame[User.id, User.credit_score]: """ Call bureaus to get credit scores; costs money for each record retrieved. """ return requests.post(...) @offline(tags=["low-cost"]) def fetch_previously_ingested_credit_scores() -> DataFrame[User.id, User.credit_score]: """ Pull previously retrieved credit scores from Snowflake only """ return snowflake.query_string("select user_id as id, credit_score from ...").all() ``` querying with required_resolver_tags can be used to enforce that only 'low-cost' resolvers are executed -- ``` # This query is guaranteed to /never/ run any resolver that isn't tagged "low-cost". dataset = ChalkClient().offline_query( input={User.id:[1,2,3]}, output=[ User.credit_score ], recompute_features=True, required_resolver_tags=["low-cost"] ) ``` ### October 24, 2023 ### Support for Python 3.11 You can now use either of Python 3.11 or 3.10 on a per-environment basis. ``` project: my-project-id environments: default: runtime: python310 develop: runtime: python311 ``` See Python Version for more information. ### October 23, 2023 ### Quality of Life Improvements - ChalkClient.query_bulk(...) and multi_query no longer require that references features be defined as Python classes, and string names for inputs and outputs can now be used instead. ### October 11, 2023 ### Alert descriptions Alerts now support descriptions, which can be used to provide more context about the alert. ``` from chalk.monitoring import Chart, Series Chart(name="Request count").with_trigger( Series .feature_null_ratio_metric() .where(feature=User.fico_score) > 0.2, + description="""*Debugging* + + When this alert is triggered, we're parsing null values from + a lot of our FICO reports. It's likely that Experian is + having an outage. Check the . + """ ) ``` These descriptions can also be set in the Chalk dashboard via the metric alerts interface. Alert description interface: ### October 5, 2023 ### query_bulk support for notebooks The query_bulk method is now available in the ChalkClient class. This method allows you to query for multiple rows of features at once. This method uses Apache Arrow's Feather format to encode data. This allows the endpoint to transmit data (particularly numeric-heavy data) using roughly 1/10th the bandwidth that is required for the JSON format used by query. This method has been available in beta for a few months, but is now available for general use, and as part of this release is now supported when querying using notebooks without access to feature schemas. ### September 26, 2023 ### Improve scheduled resolver runs list The list of scheduled resolvers now shows which resolvers are actually scheduled to run in the current environment, based on the environment argument to @online and @offline. Scheduled Resolvers List: Resolvers that are annotated with an environment other than the current environment are labeled with the environment in which they are configured to run. ### August 23, 2023 ### Improved chalk query output The chalk query command now has improved output for errors. Previously, errors were displayed in a table, which meant that stacktraces were truncated: ``` > chalk query --in email.normalized=nice@chalk.ai --out email Errors Code Feature Resolver Message ───────────────────────────────────────────────────────────────────────────── RESOLVER_FAILED src.resolvers.get_fraud_tags KeyError: 'tags' ``` Now, errors are displayed in a more readable format, and stacktraces are not truncated: ``` > chalk query --in email.normalized=nice@chalk.ai --out email Errors Resolver Failed src.resolvers.get_fraud_tags KeyError: 'tags' File "src/resolvers.py", line 30, in get_fraud_tags return parsed["tags"] KeyError('tags') ``` ### August 19, 2023 ### Query plan trace viewer The query plan viewer now includes a flame graph visualization of the query plan's execution, called the Trace View. Precise trace data is stored for every offline query by default and for online queries when the query is made with the --explain flag. Trace View: ### August 11, 2023 ### Override now in online query - Support now= for .query, --now, etc. ### Query plan viewer improvements - Redesigned query plan viewer - Support viewing execution time per operator - Support viewing data processing metrics per operator - Query plans saved for all queries by default ### No-input online and offline query improvements - offline_query now supports running downstream resolvers when no input is provided. Query primary keys will be sampled or computed, depending on the value of recompute_features. - online_query now support running a query without any input. Query primary keys will be computed using an appropriate no-argument resolver that returns a DataFrame[...] ### Misc - --local for chalk query, combines chalk apply --branch and chalk query --branch - The progress indicator in the chalk command line tool is no longer an off-brand magenta. ### August 5, 2023 ### Chalk Python SDK Improvements Added: .to_polars(), to_pandas(), and .to_pyarrow() accept prefixed: bool as an argument. prefixed=True is the default behavior, and will prefix all column names with the feature namespace. prefixed=False will not prefix column names. ``` DataFrame({User.name: ["Andy"]}).to_polars(prefixed=False) # output: # polars DataFrame with `name` as the sole column. DataFrame({User.name: ["Andy"]}).to_polars(prefixed=True) # output: # polars DataFrame with `user.name` as the sole column. ``` Added: include_meta on ChalkClient.query(...), which includes .meta on the response object. This metadata object includes useful information about the query execution, at the cost of increased network payload size and a small increase in latency. ### July 25, 2023 ### Freezing time in unit tests Chalk now supports freezing time in unit tests. This is useful for testing time-dependent resolvers. ``` from datetime import timezone, datetime from chalk.features import DataFrame, after from chalk.features.filter import freeze_time df = DataFrame([...]) with freeze_time(at=datetime(2020, 2, 3, tzinfo=timezone.utc)): df[after(days_ago=1)] # Get items after february 2nd ``` freeze_time also works with resolvers that declare specific time bounds for their aggregation inputs: ``` @online def get_num_transactions(txs: Card.transactions[before(days_ago=1)]) -> Card.num_txs: return len(txs) with freeze_time(at=datetime(2020, 9, 14)): num_txs = get_num_transactions(txs) # num transactions before september 13th ``` ### July 11, 2023 ### Explicitly time-dependent resolvers Chalk now supports resolvers that are explicitly time-dependent. This is useful for performing backfills which compute values that depend on values that are semantically similar to datetime.now(). You can express time-dependency by declaring a dependency on a special feature called Now: ``` @online def get_age_in_years(birthday: User.birthday, now: Now) -> User.age_in_years: return (now - birthday).years ``` In online query, (i.e. with ChalkClient().query), Now is datetime.now(). In offline query contexts, now will be set to the appropriate input_time value for the calculation. This allows you to backfill a feature for a single entity at many different historical time points: ``` ChalkClient().offline_query(input={User.id: [1,1,1]}, output=[User.age_in_years], input_times=[ datetime.now() - timedelta(days_ago=100), datetime.now() - timedelta(days_ago=50), datetime.now() - timedelta(days_ago=0), ]) ... ``` Now can be used in batch resolvers as well: ``` @online def batch_get_age_in_years(df: DataFrame[User.id, User.birthday, Now]) -> DataFrame[User.id, User.age_in_years]: ... ``` ### June 21, 2023 ### Testing your SQL File Resolvers SQL file resolvers are Chalk's preferred method of resolving features with SQL queries. Now, you can get your SQL file resolvers in Python by the name of the SQL file resolver. For example, if you have the following SQL file resolver: ``` -- source: postgres -- cron: 1h -- resolves: Person select id, name, email, building_id from table where id=${person.id} ``` you can test out your resolver with the following code. ``` from chalk import get_resolver resolver = get_resolver('person') # get_resolver('person.chalk.sql') will also work result = resolver('my_id') ``` ### June 15, 2023 ### Metrics Export Updates Now, Chalk supports exporting metrics about "named query" execution. These metrics (count, latency) join similar metrics about feature and resolver execution. Contact your Chalk Support representative to configure metrics export if you would like to view metrics about Chalk system execution in your existing metrics dashboards. Additional updates: - synthetic cache resolvers are now excluded - query_name is a tag on many metrics ### June 14, 2023 ### Branch deployment performance Chalk Branch Deployments provide an excellent experience for quick iteration cycles on new features and resolvers. Now, Chalk Branch Deployments automatically use a pool of "standby" workers, so there is less delay before queries can be served against a new deployment. This reduces the time it takes to run query or offline query against a new deployment from ~10-15 seconds to ~1-3 seconds. This impacts customers with more complex feature graphs the most. ### June 13, 2023 ### Expanded support for logical keying in streaming contexts Stream resolvers support a keys= parameter. This parameter allows you to re-key a stream by a property of the message, rather than relaying on the protocol layer key. This is appropriate if a stream is keyed randomly, or by an entity key like "user", but you want to aggregate along a different axis, e.g. "organization". Now, keys= supports passing a "dotted string" (e.g. foo.bar) to indicate that Chalk should use a sub-field of your message model. Previously, only root-level fields of the model were supported. ### DataFrame unit tests If you specify projections or filters in DataFrame arguments of resolvers, Chalk will automatically project out columns and filter rows in the input data. Below, we test a resolver that filters rooms in a house to only the bedrooms: ``` @features class Room: id: str home_id: "Home.id" name: str @features class Home: id: str rooms: DataFrame[Room] num_bedrooms: int @online def get_num_bedrooms( rooms: Home.rooms[Room.name == 'bedroom'] ) -> Home.num_bedrooms: return len(rooms) ``` Now, we may want to write a unit test for this resolver. ``` def test_get_num_rooms(): # Rooms is automatically converted to a `DataFrame` rooms = [ Room(id=1, name="bedroom"), Room(id=2, name="kitchen"), Room(id=3, name="bedroom"), ] # The kitchen room is filtered out assert get_num_bedrooms(rooms) == 2 # `get_num_bedrooms` also works with a `DataFrame` assert get_num_bedrooms(DataFrame(rooms)) == 2 ``` While we could have written this test before, we would have had to manually filter the input data to only include bedrooms. Also note that Chalk will automatically convert our argument to a DataFrame if it is not already one. ### June 12, 2023 ### Query Run Page Chalk's dashboard shows aggregated logs and metrics about the execution of queries and resolvers. Now, it can also show detailed metrics for a single query. This is useful for debugging and performance tuning. You can access this page from the "runs" tab on an individual named query page, or from the "all query runs" link on the "queries" page. You can search the list of previously executed queries by date range, or by "query id". The query id is returned in the "online query" API response object. ### May 15, 2023 ### BigTable Online Storage Chalk now supports BigTable as an online-storage implementation. BigTable is appropriate for customers with large working sets of online features, as is common with recommendation systems. We have successfully configured BigTable to serve 700,000 feature vectors per second at ~30ms p90 e2e latency. ### May 10, 2023 ### Enhancements to Offline Query The Offline Query has been enhanced with a new recompute_features parameter. Users can control which features are sampled from the offline store, and which features are recomputed. - The default value False will maintain current behavior, returning only samples from the offline store. - True will ignore the offline store, and execute @online and @offline resolvers to produce the requested output. - If, instead, the user passes in a list of features to recompute_features, those features will be recomputed by running @online and @offline resolvers, and all other feature values - including those needed to recompute the requested features - will be sampled from the offline store. ### Recompute Dataset The 'recompute' capability is also exposed on Dataset. When passed a list of features to recompute, a new Dataset Revision will be generated, and the existing dataset will be used as inputs to recompute the requested features. ### Developing in Jupyter Chalk has introduced a new workflow when working with branches, allowing full iterations to take place directly in any IPython notebook. When a user creates a Chalk Client with a branch in a notebook, subsequent features and resolvers in the notebook will be deployed to that branch. When combined with Recompute Dataset and the enhancements to Offline Query, users have a new development loop available for feature exploration and development: - Take advantage af existing data in chalk - Explore that data using familiar tools in a notebook - Enrich the data by developing new features and resolvers - Immediately view the results of adjusting features in the dataset - When exploration is complete, features and resolvers can be directly added back to the Chalk project ### May 5, 2023 ### View Deployment Source Code Deployments now offer the ability to view their source code. By clicking the "View Source" button on the Deployment Detail page, users can view all files included in the deployed code. ### April 21, 2023 ### Improved Deployment Utilities Users can now "redeploy" any historical deployment with a UI button on the deployment details page. This enables useful workflows including rollbacks. The "download source" button downloads a tarball containing the deployed source to your local machine. Deploy UI Enhancements ### April 18, 2023 ### Resolver error messages for incorrect types include primary keys When writing resolvers, incorrect typing can be a difficult to track. Now, if a resolver instantiates a feature of an incorrect type, the resolver error message will include the primary key value(s) of the query itself. ### April 11, 2023 ### Online query improvements The Online Query API can now be used to query DataFrame-typed features. For instance, you can query all of a user's transaction level features in a single query: ``` chalk query --in user.id --out user.transactions { "columns": ["transaction.id", "transaction.user_id", ...], "values": [[1, 2, 3, ...], ["user_1", "user_2", "user_3", ...] } ``` More functionality will be added to Online and Offline query APIs to support more advanced query patterns. ### April 6, 2023 ### Branch deployments When deploying with chalk apply a new flag --branch has been introduced which creates a branch deployment. Users can interact with their branch deployment using a consistent name by passing the branch name to query, upload_features, etc. Chalk clients can also be scoped to a branch by passing the branch in the constructor. Branch deployments are many times faster than other flavors of chalk apply, frequently taking only a few seconds from beginning to end. Branch deployments replace preview deploys, which have been deprecated. ### March 31, 2023 ### Speed improvements for deployments Deployments via chalk apply are now up to 50% faster in certain cases. If your project's PIP dependencies haven't changed, new deployments will build & become active significantly faster than before. Deploy Time Comparison: ### March 17, 2023 ### Offline TTL Introduces a new "offline_ttl" property to features decorator . Now you can control for how long data is valid in the offline_store. Any feature older than the ttl value will not be returned in an offline query. ``` @features class MaxOfflineTTLFeatures: id: int ts: datetime = feature_time() no_offline_ttl_feature: int = feature(offline_ttl=timedelta(0)) one_day_offline_ttl_feature: int = feature(offline_ttl=timedelta(days=1)) infinite_ttl_feature: int ``` ### Strict Feature Validation Adds the strict property to features decorator, indicating that any failed validation will throw an error. Invalid features will never be written to the online or offline store is strict is True. Also introduces the validations array to allow differentiated strict and soft validations on the same feature. ``` @features class ClassWithValidations: id: int name: int = feature(max=100, min=0, strict=True) feature_with_two_validations: int = feature( validations=[ Validation(min=70, max=100), Validation(min=0, max=100, strict=True), ] ) ``` ### March 7, 2023 ### Datasets in Offline Query The Dataset class is now live! Using the new ChalkClient.offline_query method, we can inspect important metadata about the query and retrieve its output data in a variety of ways. Simply attach a dataset_name to the query to persist the results. ``` from chalk.client import ChalkClient, Dataset uids = [1, 2, 3, 4] at = datetime.now() dataset: Dataset = ChalkClient().offline_query( input={ User.id: uids, }, input_times=[at] * len(uids), output=[ User.id, User.fullname, User.email, User.name_email_match_score, ], dataset_name='my_dataset' ) pandas_df: pd.DataFrame = dataset.data_as_pandas ``` Check out the documentation here. ### February 28, 2023 ### Deployment Build Logs Chalk now provides access to build and boot logs through the Deployments page in the dashboard. Build Logs ### February 16, 2023 ### Resolver timeouts Computing features associated with third-party services can be unpredictably slow. Chalk helps you manage such uncertainty by specifying a resolver timeout duration. Now you can set timeouts for resolvers! ``` @online(timeout="200ms") def resolve_australian_credit_score(driver_id: User.driver_id_aus) -> User.credit_score_aus: return experian_client.get_score(driver_id) ``` ### January 26, 2023 ### SQL File Resolvers SQL-integrated resolvers can be completely written in SQL files: no Python required! If you have a SQL source like as follows: ``` pg = PostgreSQLSource(name='PG') ``` You can define a resolver in a .chalk.sql file, with comments that detail important metadata. Chalk will process it upon chalk apply as it would any other Python resolver. ``` -- type: online -- resolves: user -- source: PG -- count: 1 select email, full_name from user_table where id=${user.id} ``` Check out the documentation here. ### January 12, 2023 ### Improved Logging Logging on your dashboard has been improved. You can now scroll through more logs, and the formatting is cleaner and easier to use. This view is available for resolvers and resolver runs. Logs Viewer ### January 9, 2023 ### Pretty Print Online Query Results Online Query Response objects now support pretty-print in any iPython environment. Pretty Print Query Response ### January 8, 2023 ### Linux docker containers on M1 Macs chalkpy has always supported running in docker images using M1's native arm64 architecture, and now chalkpy==1.12.0 supports most functionality on M1 Macs when run with AMD64 (64 bit Linux) architecture docker images. This is helpful when testing images built for Linux servers that include chalkpy. ### January 6, 2023 ### Docs Search Chalk has lots of documentation, and finding content is now difficult. We've added docs search! Documentation search Try it out by typing cmd-K, or clicking the search button at the top of the table of contents. ### September 27, 2022 ### Tags & Owners as Comments This update makes several improvements to feature discovery. Tags and owners are now parsed from the comments preceding the feature definition. ``` @features class RocketShip: # :tags: **team:identity**, **priority:high** # :owner: **katherine.johnson@nasa.gov** velocity: float ... ``` Prior to this update, owners and tags needed to be set in the feature(...) function: ``` @features class RocketShip: velocity: float = feature( tags=["**team:identity**", "**priority:high**"], owner="**katherine.johnson@nasa.gov**" ) ... ``` Feel free to choose either mechanism! ### July 28, 2022 ### Auto Id Features It's natural to name the primary feature of a feature class id. So why do you always have to specify it? Until now, you needed to write: ``` @features class User: id: str = feature(primary=True) ... ``` Now you don't have to! If you have a feature class that does not have a feature with the primary field set, but has a feature called id, it will be assigned primary automatically: ``` @features class User: id: str ... ``` The functionality from before sticks around: if you use a field as a primary key with a name other than id, you can keep using it as your primary feature: ``` @features class User: user_id: str = feature(primary=True) # Not really the primary key! id: str ``` ### July 25, 2022 ### DataFrame Expressions The Chalk DataFrame now supports boolean expressions! The Chalk team has worked hard to let you express your DataFrame transformations in natural, idiomatic Python: ``` DataFrame[ User.first_name == "Eleanor" or ( User.email == "eleanor@whitehouse.gov" and User.email_status not in {"deactivated", "unverified"} ) and User.birthdate is not None ] ``` Python experts will note that or, and, is, is not, not in, and not aren't overload-able. So how did we do this? The answer is AST parsing! A more detailed blog post to follow. ### July 22, 2022 ### Descriptions as Comments This update makes several improvements to feature discovery. Descriptions are now parsed from the comments preceding the feature definition. For example, we can document the feature User.fraud_score with a comment above the attribute definition: ``` @features class User: # **0 to 100 score indicating an identity match.** # **Low scores indicate safer users** fraud_score: float ... ``` Prior to this update, descriptions needed to be set in the feature(...) function: ``` @features class User: fraud_score: float = feature(description=""" **0 to 100 score indicating an identity match.** **Low scores indicate safer users** """) ... ``` The description passed to feature(...) takes precedence over the implicit comment description. ### Namespace Metadata You can now set attributes for all features in a namespace! Here, we assign the tag group:risk and the owner ravi@chalk.ai to all features on the feature class. Owners specified at the feature level take precedence (so the owner of User.email is the default ravi@chalk.ai whereas the owner of User.flaky_api_result is devops@chalk.ai). Tags aggregate, so email has the tags pii and group:risk. ``` @features(tags="group:risk", owner="ravi@chalk.ai") class User: email: str = feature(tags="pii") flaky_api_result: str = feature(owner="devops@chalk.ai") ``` ### July 14, 2022 ### Self-Serve Slack Integration You can configure Chalk to post message to your Slack workspace! You can find the Slack integration tab in the settings page of your dashboard. Slack integration Slack can be used as an alert channel or for build notifications. ### July 13, 2022 ### Python 3.8 Support Chalk's pip package now supports Python 3.8! With this change, you can use the Chalk package to run online and offline queries in a Python environment with version >= 3.8. Note that your features will still be computed on a runtime with Python version 3.10. ### July 8, 2022 ### Named Integrations Chalk's injects environment variables to support data integrations. But what happens when you have two data sources of the same kind? Historically, our recommendation was to create one set of environment variables through an official data source integration, and one set of prefixed environment variables yourself using the generic environment variable support. With the release of named integrations, you can connect to as many of the same data source as you need! Provide a name at the time of configuring your data source, and reference it in the code directly. Named integrations inject environment variables with the standard names prefixed by the integration name (ie. RISK_PGPORT). The first integration of a given kind will also create the un-prefixed environment variable (ie. both PGPORT and RISK_PGPORT). ### June 29, 2022 ### SOC 2 Report Chalk is excited to announce the availability of our SOC 2 Type 1 report from Prescient Assurance. Chalk has instituted rigorous controls to ensure the security of customer data and earn the trust of our customers, but we're always looking for more ways to improve our security posture, and to communicate these steps to our customers. This report is one step along our ongoing path of trust and security. If you're interested in reviewing this report, please contact support@chalk.ai to request a copy. ### June 3, 2022 ### Pandas Integration You can now convert Chalk's DataFrame to a pandas.DataFrame and back! Use the methods chalk_df.to_pandas() and .from_pandas(pandas_df). ### Migration Sampling The 1.4.1 release of the CLI added a parameter --sample to chalk migrate. This flag allows migrations to be run targeting specific sample sets. ### Feature/Resolver Health Added spark lines to the feature and resolver tables which show a quick summary of request counts over the past 24 hours. Added status to feature and resolver tables which show any failing checks related to a feature or resolver. # Streaming Resource Groups source: https://docs.chalk.ai/docs/streaming-resource-groups ## Assign streaming resolvers to dedicated resource groups for workload isolation. Streaming resolvers can be assigned to resource groups, enabling you to isolate streaming workloads onto dedicated infrastructure. This is useful for separating high-throughput streams from lighter workloads, or for running specific resolvers on nodes with different resource profiles. By default, streaming resolvers without a resource_group run on the Default resource group. When a resource_group is specified, the resolver only runs on the streaming server deployed for that resource group. ### Setting up a resource group ### 1. Configure infrastructure In the Chalk dashboard: - Navigate to Infrastructure in the left sidebar - Click Resource Configuration - Under Resource Groups, click Add Resource Group and give it a name - Add a Streaming Server to the new resource group and configure CPU, memory, and scaling - Add a gRPC Query Server so the streaming server can route computed features to the engine - Click Save and Apply Service to deploy We recommend starting with the same resource settings as your Default group for simplicity. ### 2. Assign resolvers to the resource group In your deployment source code, pass the resource_group parameter to make_stream_resolver: ``` from chalk.features import features, make_stream_resolver from chalk.streams import KinesisSource source = KinesisSource(name="transactions") @features(max_staleness="1d", etl_offline_to_online=True) class Transaction: id: str amount: float # This resolver only runs on the "high-throughput" streaming server transaction_resolver = make_stream_resolver( name="transaction_resolver", source=source, resource_group="high-throughput", output_features={ Transaction.id: _.transaction_id, Transaction.amount: _.transaction_amount, }, ) ``` Resolvers without a resource_group run on the Default streaming server. ### 3. Deploy Run chalk apply to deploy your updated resolver definitions. The streaming server in the new resource group will automatically pick up only the resolvers assigned to it. ### Example: isolating a high-throughput stream Suppose you have five streaming resolvers, but one handles significantly more traffic than the others. You can isolate it onto dedicated infrastructure: ``` # Runs on the "high-throughput" resource group only high_volume_resolver = make_stream_resolver( name="high_volume_resolver", source=high_volume_source, resource_group="high-throughput", output_features={...}, ) # Runs on the Default streaming server low_volume_resolver = make_stream_resolver( name="low_volume_resolver", source=low_volume_source, output_features={...}, ) ``` In this setup, the Default streaming server runs low_volume_resolver (and any other resolvers without a resource group), while the high-throughput streaming server runs only high_volume_resolver. # GPUs and Resource Hints source: https://docs.chalk.ai/docs/resource-hints ## Schedule workloads on differentiated hardware & execution pools Different workloads have different requirements. Some workloads are compute-intensive, while others are memory-intensive. Some workloads require GPUs, some perform blocking IO, and some are purely CPU-bound.and Chalk allows you to specify the hardware and execution pool requirements for your queries, so that they can be scheduled on the most appropriate resources. ### Executing on Hardware with GPUs Chalk allows you to execute specific resolvers on hardware with GPUs. To do this, you can use the "gpu" resource hint in your resolver definition. This will ensure that the resolver is scheduled on a machine with a GPU. ``` import torch from chalk.features import online @online(resource_hint="gpu") def my_gpu_resolver(memo: Transaction.memo) -> Transaction.cleaned_memo: """ Run a torch model to process data using a GPU. """ ... ``` Then, specify the Resource Group that has the GPU resources available: ``` CHALK_GPU_RESOURCE_GROUP_NAME=gpu-resolvers ``` This resource group should: - have a UDF Invoker Service deployed - this service should have a node selector that matches nodes with GPUs, e.g.: ``` cloud.google.com/machine-family: a2-highgpu ``` Chalk's query planner will execute the primary query plan on your Query Server service, and will transmit row data to the GPU resolver service for processing in compressed Arrow format. Response data will be transmitted back to the Query Server service, which will then return the final result to the client. ### CPU vs IO Pools Chalk allows you to specify whether a resolver is CPU-bound or IO-bound. This is useful for scheduling resolvers on the most appropriate resources. To do this, you can use the "cpu" or "io" resource hints in your resolver definition. ``` from chalk.features import online @online(resource_hint="cpu") def my_cpu_bound_resolver(memo: Transaction.memo) -> Transaction.cleaned_memo: """ Run a CPU-bound operation. """ fibbonaci = [0, 1] for i in range(2, 100): fibbonaci.append(fibbonaci[i - 1] + fibbonaci[i - 2]) return fibbonaci @online(resource_hint="io") def my_io_bound_resolver(memo: Transaction.memo) -> Transaction.cleaned_memo: """ Run an IO-bound operation. """ return requests.get("https://example.com/api/data").json() ``` Chalk uses CPU-count thread pools for CPU-bound resolvers, because additional threads do not help with CPU-bound workloads. Chalk uses variable sized thread pools for IO-bound resolvers, because additional threads can help with IO-bound workloads. Note: if you use async resolvers, Chalk will execute them on an asyncio event loop that is shared across all resolvers. This means that you should not use blocking IO operations in async resolvers, as they will block the event loop and prevent other resolvers from executing. # Error Handling source: https://docs.chalk.ai/docs/query-errors ## Run resolvers in the presence of upstream failures. Chalk returns an error channel in addition the requested feature values. There are several types of errors one could expect: ### Error categories ### Request error Request errors are raised before execution of your resolver code. They may occur due to invalid feature names in the input or a request that cannot be satisfied by the resolvers you have defined. ### Field error Field errors are raised while running a feature resolver for a particular field. For this type of error, you'll find a feature and resolver attribute in the error type. When a feature resolver crashes, you will receive null value in the response. To differentiate from a resolver returning a null value and a failure in the resolver, you need to check the error schema. ### Network error Network errors are thrown outside your resolvers. For example, your request was unauthenticated, connection failed, or an error occurred within Chalk. ### Error schema The online query interface for resolvers returns the following schema: ### Response Schema The outputs features and any query metadata (discussed in detail at Query Basics.) Errors encountered while running the resolvers. Each element in the list is a ChalkError. If no errors were encountered, this field is empty. ### ChalkError The type of error, matching one of the error codes. The category of the error, given in the type field for the error codes. This will be one of "REQUEST", "NETWORK", and "FIELD". A readable description of the error message. The exception that caused the failure, if applicable. The name of the class of the exception. The message taken from the exception. The stacktrace produced by the code. The fully qualified name of the failing feature, eg. user.identity.has_voip_phone. The fully qualified name of the failing resolver, eg. my.project.get_fraud_score. ### Error code The query contained features that do not exist. A resolver was required as part of running the dependency graph that could not be found. The query is invalid. All supplied features need to be rooted in the same top-level entity. A feature value did not match the expected schema (eg. incompatible type "int"; expected "str") The resolver for a feature errored. The resolver for a feature timed out. A crash in a resolver that was to produce an input for the resolver crashed, and so the resolver could not run crashed, and so the resolver could not run. The request was submitted with an invalid authentication header. The request has credentials that do not provide the required authorization to execute an operation. An unspecified error occurred. # Query Scheduling source: https://docs.chalk.ai/docs/query-cron ## Create and ingest datasets on a schedule Once you have defined features and resolvers in Chalk, you can schedule queries to run at regular intervals. This is useful for ingesting data from external sources, or for precomputing features that are expensive to compute in real-time. ### Scheduling a Query Query scheduling looks very similar to offline_query's usage. Imagine that we want to precompute the average temperature for each city in the world every day. First, we define a feature class for the temperature readings, and a feature class for the cities: ``` @features class TemperatureReading: id: int city: City.id temperature: float ts: datetime @features(max_staleness="1d", etl_offline_to_online=True) class City: id: str name: str average_temperature: float temperature_readings: DataFrame[TemperatureReading] ``` Then, we define a resolver that fetches the temperature readings for each city: ``` -- resolves: TemperatureReading -- source: snowflake SELECT id, city, temperature, ts FROM temperature_readings ``` Then, we define a resolver to compute the average temperature for each city: ``` @offline def compute_average_temperature(temps: City.temperature_readings[TemperatureReading.temperature, after(days_ago=1)]) -> City.average_temperature: return temps.mean() ``` Finally, we schedule the query to run every day at midnight: ``` ScheduledQuery( name="average_temperature", output=[City.average_temperature], schedule="0 0 * * *", store_online=True ) ``` Each day, this query will execute and cache the average temperature for each city in the world. This data will be available for real-time queries: ``` client.query(input={City.id: 713}, output=[City.average_temperature]) # 58 ``` # Sharing sensitive data with Chalk source: https://docs.chalk.ai/docs/public-key ## Encrypt data with Chalk's public key before sending it, using GnuPG. ### Overview If you need to send sensitive material to Chalk, you can use asymmetric encryption to transmit your data securely. Only Chalk will be able to view the data that you encrypt with Chalk's public key. ### Using GnuPG To begin, ensure that you've installed GnuPG (Mac, Ubuntu). GnuPG provides the gpg utility, which you will use to encrypt your data using Chalk's public key. Next, download and import Chalk's public key: ``` curl -s https://docs.chalk.ai/chalk-public-key.key > chalk-public-key.key gpg --import chalk-public-key.key ``` Next, encrypt your plaintext using Chalk's public key: ``` cat | gpg -e -r Chalk > ciphertext.gpg ``` Now you can transmit ciphertext.gpg via email or Slack to Chalk. Only Chalk's support staff will be able to decrypt and view the contents of ciphertext.gpg. # OpenAI source: https://docs.chalk.ai/docs/openai ## Integrate with OpenAI for embedding models. Chalk supports OpenAI as an embeddings provider. ### Adding OpenAI On the dashboard, you can provide your OpenAI API key. # Offline Query Heartbeat Timeout source: https://docs.chalk.ai/docs/offline-query-timeout ## Learn more about debugging offline query timeouts. Offline queries can consume significant amounts of system resources. While an offline query runs, it sends regular heartbeats to Chalk. If a query stops sending heartbeats for several minutes, Chalk marks it as failed and stops it. The query's error then says that it "failed from missing a heartbeat. This means it probably encountered an out-of-memory error." Typically, this is due to the query being too resource-intensive for the system to handle. If you are experiencing offline query timeouts, you may want to consider the following: - Check for out-of-memory errors: Check the 'metrics' tab of your query, and look at the reported memory utilization. If you are running out of memory, you may want to increase the memory requests for your offline query pods. You can do this by navigating to the "Resources" page in the Chalk UI and increasing the memory requests for offline query workers. - Use run_asynchronously: Add run_asynchronously=True to isolate your query to its own Kubernetes pod. This lowers the chance that your query will contend with other workloads on a single machine and may help prevent timeouts. - Adjust node selectors: If you are using node selectors to run your offline queries on specific nodes, ensure that the nodes you are selecting have enough resources to handle the query. Make sure that your workloads are not co-scheduled with other intense tasks. - Shard your query: If you are running a large query, consider breaking it into smaller pieces using num_shards. This can help prevent timeouts and improve the overall performance of your query. - Optimize your query: Look at the operations the query is processing. Check for Python resolvers that use heavy libraries, or that perform memory-intensive operations like Pandas transformations. If you are still experiencing timeouts, contact Chalk Support for further assistance. Please provide an example failed query and context on how to reproduce the issue. # Job Queue source: https://docs.chalk.ai/docs/job-queue ## Understanding Chalk's job queue and resource groups ### Overview The job queue in Chalk, together with resource groups, functions similarly to warehouses in analytical data platforms - they provide dedicated, configurable compute resources for processing workloads. A job queue consumer is a persistent worker process that consumes jobs from a queue and executes them one at a time. By configuring multiple resource groups with different job queue consumers, you can create isolated compute environments optimized for different workload types. In the Chalk dashboard, this is configured under the "Job Queue Consumer" service. Earlier documentation referred to this service as the "Job Queue Server"; both names refer to the same underlying component. ### What does the job queue process? The job queue handles two primary types of workloads: - Scheduled queries - Feature pipelines that run on a cron schedule (see ScheduledQuery) - Async offline queries - Large offline queries that run asynchronously when run_asynchronously=True is set ``` # This runs on the job queue client.offline_query( input={'user.id': range(1_000_000)}, output=['user.name'], run_asynchronously=True, # Runs as a task on job queue ) # This runs on the query server (NOT the job queue) client.offline_query( input={'user.id': [1, 2, 3]}, output=['user.name'], # run_asynchronously=False by default - runs as synchronous RPC ) ``` ### How job queues work ### FIFO Processing Jobs are processed in first-in, first-out (FIFO) order. Each job queue consumer processes one job at a time sequentially. ### Fixed Resources Each job queue consumer has a single, pre-configured resource allocation (CPU and memory). ### Automatic Fallback If a job requests resources larger than the job queue consumer can handle, Chalk automatically skips the queue and runs the job as a standalone Kubernetes pod with the requested resources. ### Resource Groups Resource groups allow you to create multiple job queue consumers with different resource configurations. This is useful for: - Isolating large scheduled queries from smaller workloads - Optimizing costs by right-sizing compute for different job types - Preventing resource contention between different teams or use cases ### Configuring Job Queue Consumers In the Chalk dashboard under Infrastructure > Resource Configuration, you can configure the "Job Queue Consumer" for each resource group: - Set the CPU and memory allocation - Configure autoscaling (min/max instances) - Select the nodepool to use All Chalk environments start with a Default resource group. ### Targeting Specific Resource Groups ### For Scheduled Queries ``` from chalk import ScheduledQuery ScheduledQuery( name="large-batch-job", schedule="0 0 * * *", output=[User.features], resource_group="large-jobs", # Runs on the "large-jobs" resource group ) ``` ### For Async Offline Queries ``` from chalk.client import ChalkClient, ResourceRequests client.offline_query( input={'user.id': range(1_000_000)}, output=['user.name'], run_asynchronously=True, resources=ResourceRequests( resource_group="large-jobs" # Runs on the "large-jobs" resource group ), ) ``` For spill-heavy or Iceberg-backed async offline queries, see Local SSDs for spilling and scan caching for the full setup of a dedicated LSSD-backed resource group. ### Job Queue Consumer vs Query Server | Aspect | Job Queue Consumer | Query Server | | ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------------ | | **Processes** | Scheduled queries, async offline queries | Synchronous offline queries, online queries | | **Execution** | One job at a time (FIFO) | Multiple concurrent requests | | **Resources** | Fixed per resource group | Requested per query | | **Scaling** | Horizontal (more instances) | Vertical (larger pods) | | **Workload Isolation** | Jobs run sequentially without resource contention | Multiple concurrent queries may compete for resources on the same server | | **Timeout Behavior** | Can run indefinitely beyond load balancer timeout | Will report an error if execution exceeds load balancer timeout | ### Best Practices - Create separate resource groups for jobs with significantly different resource requirementsExample: Small daily scheduled queries vs. large weekly batch jobs - Right-size your default job queue to handle typical workloadsConsider your most common scheduled query needsRemember that oversized requests will automatically get their own pods - Use resource groups for isolationPrevent one team's large jobs from blocking another team's scheduled queriesGuarantee resources for critical scheduled pipelines - Monitor queue depth and adjust max instances if jobs are waiting too longJobs will timeout and fail if they can't obtain resources within ~4 hours ### Example Configuration Here's a common setup with two resource groups: ``` # Default resource group: moderate sizing for typical scheduled queries # Configured in dashboard: 8 CPU, 16 GB memory ScheduledQuery( name="daily-features", schedule="0 1 * * *", output=[User.daily_features], # Uses default resource group ) # Large jobs resource group: high-memory machines for big batch processing # Configured in dashboard: 32 CPU, 450 GB memory ScheduledQuery( name="weekly-aggregations", schedule="0 0 * * 0", output=[User.historical_aggregates], resource_group="large-jobs", # Uses dedicated high-memory queue ) ``` # Getting Started source: https://docs.chalk.ai/docs/getting-started ## Your first time with Chalk ### Introduction Chalk's platform is a powerful tool that allows data engineers and data scientists to collaborate efficiently and effectively. In this project we'll create features related to Usersand their credit scores. We'll combine data from an API and a database to help us decide whether we should issue new loans. You will learn how to create a new Chalk project, create features and resolvers, deploy them to the Chalk environment, and query the environment using the Chalk CLI and a Jupyter Notebook. This tutorial will take about 45 minutes to complete end-to-end. ### Installing the Required Software - If you don't have a favorite IDE, install VS Code by visiting the download site and running the appropriate installer. - You'll also want to install the code CLI command. - If you haven't already, install Python. You can do this through Homebrew as follows: ``` # Install Homebrew $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Use Homebrew to install python3 $ brew install python ``` If you're having trouble installing Python, follow the instructions in Python's docs. - Install the Chalk CLI by running ``` $ curl -s -L https://api.chalk.ai/install.sh | sh ``` ### Creating your Chalk project Now that all of your software is installed, let's create a Chalk project where you will add your features and resolvers. - Create a folder and call it chalk-tutorial. - Run chalk init from inside the chalk-tutorial to initialize your project files. This will create two files for you: chalk.yaml and .chalkignore. The first file, chalk.yaml, contains configuration information about your project. The second file, .chalkignore, tells your Chalk CLI tool which files to ignore when deploying to your Chalk environment. You can edit this file and use it just like a .gitignore file. ``` $ mkdir chalk-tutorial $ cd chalk-tutorial $ chalk init Created project config file chalk.yaml Created .chalkignore file ``` - Now edit the chalk.yaml file and set the project field to the name of your Chalk project. This might be chalk-tutorial, but check by visiting the Projects page. - Login by typing chalk login. If you're using a dedicated environment, make sure you use the --api-host flag. Type y when prompted and login in the browser. Now you're ready to deploy code and query the environment! ### Create a virtual environment We're going to create a virtual environment so that you can download and use Python libraries in your Chalk project. - Run the following command from inside your Chalk project to create and activate your virtual environment. When you want to deactivate your virtual environment, type deactivate. ``` $ python3 -m venv chalk-tutorial-venv $ source chalk-tutorial-venv/bin/activate ``` - Your requirements.txt file tells your virtual environment (and Chalk) which libraries to install. We'll use a few libraries for this tutorial, so add the following to the requirements.txt. Feel free to add more libraries later. ``` chalkpy pydantic requests ``` - Install the requirements. ``` $ pip3 install -r requirements.txt ``` - Check that the libraries are installed by importing requests in Python. If there's no error message, it worked. Exit by typing exit() and hitting enter. ``` $ python3 >>> import requests >>> exit() ``` ### Use Curl and Jupyter to call APIs. Now that all the software is installed and ready to use, you're ready to start integrating an API. - First, lets test the API we're going to use. Run this command to hit the API from your command line. Here we're asking "give me the credit score for user 12" and the API gives us back a JSON object with a credit score. ``` $ curl https://credit-report.chalk.dev/rutter_score/12 {"score":98} ``` Now you should try to hit this API from Python using a Jupyter Notebook. - Open VS Code, and create a new file File -> New File. Select Jupyter Notebook. - Jupyter will probably ask you which interpreter to use. If it does, select the virtual environment you created before (chalk-tutorial-venv). - If not, you can select it manually. Select Jupyter virtual environment - Finally, let's query the API from Jupyter. Enter the following code and run it with shift+enter. ``` import requests requests.get( "https://credit-report.chalk.dev/rutter_score/12" ).json()["score"] ``` If you see 98, congratulations! You're all setup and ready to start developing your Chalk project. ### Building a Chalk project With all the pre-requisites completed, let's jump right into building your Chalk project. ### Creating your first Chalk feature We'll create a feature called User with a "Rutter score" property. The resolver will call the "Rutter" API we tested earlier, passing the User ID, and return the score. After you deploy the feature, you will test it in Jupyter. - Create a credit.py file and copy in the feature and resolver definitions below: ``` import requests from chalk.features import online, features, Features @features class User: id: str credit_score: int @online def get_credit_score( id: User.id, ) -> Features[User.credit_score]: return requests.get( f"https://credit-report.chalk.dev/rutter_score/{id}" ).json()["score"] ``` - Deploy your code. Pass the --branch flag with a name so that you don't affect the "actual" environment ``` chalk apply --branch test ``` - Now lets query for your feature from your Jupyter Notebook. Enter and execute the following code in your notebook. ``` from credit import User from chalk.client import ChalkClient ChalkClient().query( input={User.id: 'u_F6zY0tE4w8'}, output=[User.credit_score], branch="test" ) ``` You can also query for this feature in your terminal: ``` $ chalk query --in user.id=u_F6zY0tE4w8 \ --out user.credit_score \ --branch test ``` Congratulations! You've written your first Chalk feature, connected it to an external API, and queried it from a Jupyter Notebook. This is a big achievement! ### Adding a database to Chalk We have a database containing more information about users, and it would be good to combine that information together on our user feature. - If you have psql or similar tool installed, you can preview the database from the command line. When prompted, the password is postgres. ``` psql -U postgres -p 5432 -h 35.188.7.252 -d postgres postgres=> select * from users limit 10; ``` Notice that we know name, surname, email, birthday, and is_fraud, a flag that tells us whether we know this user has committed fraud in the past. - To use this in your Chalk project, we'll add the database through the UI. Navigate to your environment, and find "Data Sources" on the left hand navigation. - If you see the database already, great, you can skip the next step of adding the database. Otherwise, continue. - Click Add a Data Source and choose PostgreSQL. - Fill out the fields as shown here. The password is postgres. Postgres ### Write a resolver that resolves a database - Add the fields from the database to the User feature class you created before. The properties name, surname, and email are strings. Birthday is a date, and is_fraud is a boolean. (Hint: you may have to add from datetime import date to your credit.py file) - Add the datasource to your Python context. Add the below code to your credit.py file. This tells Python about your datasource. ``` from chalk.sql import PostgreSQLSource user_pg = PostgreSQLSource(name="User_PG") ``` - You can use a SQL File Resolver to pull this information. Create a new file called users.chalk.sql and copy the below contents: ``` -- type: online -- resolves: user -- source: postgres -- count: 1 select name, surname, email, birthday, is_fraud from users where id=${user.id} ``` - After deploying (remember to use --branch with the name you specified, for example "test"), query your feature. Supplying User as the output tells Chalk to give you back ALL the features of User. ``` res = client.query( input={User.id: 'u_F6zY0tE4w8'}, output=[User], branch="test" ) ``` You can also query for this feature in your terminal: ``` $ chalk query --in user.id=u_F6zY0tE4w8 \ --out user \ --deployment test ``` # Tutorial: Backtesting source: https://docs.chalk.ai/docs/fraud-6 ## Try out new feature values on historical data. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. After you've created some features and resolvers, you can use them to generate values for training. Chalk tracks all the values of the features you compute, and times at which those values were computed. First, we need to sample some user ids on which to build a dataset. ``` from datetime import datetime, timezone, timedelta from chalk.client import ChalkClient from src.models import User client = ChalkClient() now = datetime.now(tz=timezone.utc) ds = client.offline_query( output=[User.id], lower_bound=now - timedelta(hours=12), upper_bound=now, ) ``` ``` from chalk.client import ChalkClient client = ChalkClient() dataset = client.offline_query( input={ User.id: [], }, output=[], recompute_features=True, ) ds = client.offline_query( input={ "fraud_model.id": spine["fraud_model.id"].to_list() }, input_times=spine['__chalk__.CHALK_TS'].to_list(), output=[ "fraud_model.id", "fraud_model.card_created_at", ], recompute_features=True ) ``` # Tutorial: Inference source: https://docs.chalk.ai/docs/fraud-5 ## Integrate Chalk into your production decisioning systems. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. Now that we've written some features and resolvers and deployed them to Chalk, we're ready to integrate Chalk into our production decisioning systems. ### CLI Query As a sanity check, it can be helpful to use the Chalk CLI to query a well-known input and ensure that we get the expected output. We can use the chalk query command, passing in the id of a user, and the names of the features we want to resolve: ``` $ chalk query --in user.id=1 \ --out user.name \ --out user.email \ --out user.account.balance Results user.name "John Doe" email "john@doe.com" user.account.balance 2032.91 ``` ### API Client Query Once we're satisfied that our features and resolvers are working as expected, we can use a client library to query Chalk from our application. In this first example, we'll use the ChalkClient in the chalkpy package to query Chalk from our application: ``` from src.models import User from chalk.client import ChalkClient # Create a new Chalk client. By default, this will # pick up the login credentials generated after running # `chalk login`. client = ChalkClient() client.query( input=User(id=1234), output=[ User.id, User.name, User.fico_score, User.account.balance, ], ) ``` We use the same feature definitions for querying our data as we used for defining our features and resolvers. Chalk has API client libraries in several languages, including Python, Go, Typescript, and Elixir. ### Code Generation (Optional) All API clients can operate on the string names of features. However, in a production system, you may have many hundreds or thousands of features, and want to avoid hard-coding the names of each feature in your code. To help with this, Chalk can codegen a library of strongly-typed feature names for you. For example, say the service that calls into Chalk is written in Go. We can generate a Go library of feature names with the following command: ``` $ chalk codegen go --out ./clients/go/client.go --package=client ✓ Found resolvers ✓ Wrote features to file './clients/go/client.go' ✓ Please do not change the generated code. ``` This generates a file clients/go/client.go that looks like this: ``` package client /************************************** Code generated by Chalk. DO NOT EDIT. > chalk codegen go --out ./clients/go/client.go --package client **************************************/ import ( "github.com/chalk-ai/chalk-go" "time" ) var InitFeaturesErr error type Account struct { Id *int64 Title *string UserId *int64 Balance *float64 User *User UpdatedAt *time.Time } type User struct { Id *int64 Name *string Email *string Account *Account AccountNameMatch *float64 FicoScore *int64 CreditScoreTags *[]any } var Features struct { Account *Account User *User } func init() { InitFeaturesErr = chalk.InitFeatures(&Features) } ``` We can then use this library to query Chalk: ``` import ( "github.com/chalk-ai/chalk-go" ) // Create a new Chalk client. client := chalk.NewClient() // Create an empty struct to hold the results. user := User{} // Query Chalk, and add the results to the struct. _, err = client.OnlineQuery( chalk.OnlineQueryParams{}. WithInput(Features.User.Id, 1234). WithOutputs( Features.User.Id, Features.User.LastName, Features.User.FicoScore, Features.User.Account.Balance, ), &user, ) // Now, you can access the properties of the // user for which there was a matching `output`. fmt.Println(user.Account.Balance) ``` If your calling service is written in Python, but you don't want to take a dependency on the repository that contains your Chalk features, you can generate your Python features into a separate repository: ``` $ chalk codegen python --out ./clients/python/client.py ``` You can see the generated code in clients/python/client.py. If you are generating Python into a subdirectory of your Chalk project, be sure to add an entry to your .chalkignore containing the directory of your generated code (in the above example, clients/). Otherwise, Chalk will find duplicate definitions of your features. # Tutorial: Python Resolvers source: https://docs.chalk.ai/docs/fraud-4 ## Define resolvers in Python that call APIs and compute derived features. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. So far, we've mapped SQL tables into feature classes. But there's a lot more we can do with Chalk. In this step, we'll add features to our Account and User feature classes that are gathered from API calls and computed downstream of other features. ### Derived Features We've noticed that some fraudsters try to link stolen accounts to our platform and attempt to transfer money through our system. To detect this behavior, we want to compute a similarity score between the user's name and the account's title. We'll start by adding this new feature, account_name_match, to our User feature class. ``` @features class User: id: int name: str email: str account: "Account" + # The similarity between the user's name and the account's title. + account_name_match: float ``` Next, we'll define a resolver that computes this feature. We'll use Jaccard similarity to compute the similarity score. ``` from src.models import User from chalk import online @online def account_name_match( title: User.account.title, name: User.name, ) -> User.account_name_match: """Docstrings show up in the Chalk dashboard""" intersection = set(title) & set(name) union = set(title) | set(name) return len(intersection) / len(union) ``` The @online decorator tells Chalk that this resolver should be called in realtime when the User.account_name_match feature is requested. Our feature dependencies are declared in the function signature as User.account.title and User.name. Chalk will automatically retrieve User.account_id and User.name from our user.chalk.sql resolver. Then, using this account id, Chalk will retrieve Account.title from the online store, where it has been cached from our cron run of the balance.chalk.sql resolver. ### Testing Resolvers are callable functions, so we can test them like any other Python function. Let's test our new resolver by writing a unit test: ``` from src.resolvers import account_name_match def test_names_match(): """Resolvers can be unit tested exactly as you would expect. Here, the `account_name_match` resolver should return 1.0 because the `title` and `name` are identical. """ assert 1 == account_name_match( title="John Coltrane", name="John Coltrane", ) def test_names_completely_different(): """The `account_name_match` resolver should return 0 because the `title` and `name` don't share any characters. """ assert 0 == account_name_match( title="John Coltrane", name="Zyx", ) ``` You can read more about testing resolvers in the API docs. ### API Calls Any Python function can be used as a resolver. This means that we can call APIs to compute features. Let's add a feature that computes the user's FICO score from our credit scoring vendor, Experian. As before, we'll first add the features that we want to compute: ``` + from chalk.features import features @features class User: id: int name: str email: str account_name_match: float + # The fraud score, as provided by a third-party vendor. + fico_score: int = feature(min=300, max=850, strict=True) + + # Tags from our credit scoring vendor. + credit_score_tags: list[str] ``` We are adding strict validation to our fico_score feature to ensure that we only store and utilize valid FICO scores. Now, we can write a resolver to fetch the user's FICO score from Experian. ``` from src.models import User from src.mocks import experian from chalk.features import online, Features @online def get_fraud_score( name: User.name, email: User.email, ) -> Features[User.fico_score, User.credit_score_tags]: response = experian.get_credit_score(name, email) # We don't need to provide all the features for # the `User` class, only the ones that we want to update. return User( fico_score=response['fico_score'], credit_score_tags=response['tags'], ) ``` Here, we are returning two features of the user, User.fico_score and User.credit_score_tags. We use the Features type to indicate which feature we expect to return. Also note that we are initializing the User class with only the features that we want to update. This partial initialization is the primary difference between Python's @dataclass and Chalk's @features. ### Deploying Finally, we'll want to deploy our new resolvers. As before, we can check our work by using a branch deployment: ``` $ chalk apply --branch tutorial ✓ Found resolvers ✓ Deployed branch ``` We can then query our new features: ``` $ chalk query --branch tutorial \ --in user.id=1 \ --out user.name_match_score ``` # Tutorial: SQL Resolvers source: https://docs.chalk.ai/docs/fraud-3 ## Mapping data from SQL sources to feature classes. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. A primary source of data for many companies is a SQL database. Chalk can automatically ingest data from SQL databases and map it to feature classes. ### Configuring SQL sources In our example application, we have two databases: PostgreSQL and Snowflake. Our PostgreSQL database is the primary database used elsewhere in our codebase, and our Snowflake database is used for analytics, with tables populated from DBT views and batch jobs. To configure our SQL sources in Chalk, we'll create a datasources.py file that contains a SnowflakeSource and a PostgreSQLSource: ``` from chalk.sql import SnowflakeSource, PostgreSQLSource snowflake = SnowflakeSource() postgres = PostgreSQLSource() ``` These singleton variables can be used to query data in Python SQL resolvers. They're also necessary before we can write any .chalk.sql files, as we'll do below. ### Online data Chalk's preferred way to ingest data from SQL databases is to use SQL file resolvers. This allows us to write queries in the same language as our database, and to use the same tooling to test and debug them. To create a SQL file resolver, we create a file in our project directory with the extension .chalk.sql. We can then write a SQL query in this file, and add metadata to the top of the file to tell Chalk how to ingest the data. From our User feature class, we may want to resolve the name and email attributes from a PostgreSQL table. To do this, we can write the following SQL file resolver: ``` -- The features given to us by the user. -- resolves: user -- source: postgres select id, full_name as name, email from users; ``` The resolves key tells Chalk which feature class the columns in the select statement should be mapped to. Then, the target names of the query are compared against the names of the attributes on the feature class. If the names match after stripping underscores and lower-casing, the select target is mapped to the feature. In the example above, we aliased the full_name column to name, so it will be mapped to the name attribute on the User feature class. Chalk validates your SQL file resolvers when you run chalk apply. The source key tells Chalk which integration to use to connect to the database. Since we have only one PostgreSQL database, we can reference the source as postgres. If we had multiple PostgreSQL databases, we can use named integrations to reference different databases. Other comments in the SQL file resolver are indexed by Chalk and can be searched in the Chalk dashboard. ### Deploying! Now that we've written a resolver, we can deploy our feature pipeline and query our data in realtime. In testing, it can be helpful to deploy your feature pipeline to a branch, which allows you to test your changes without affecting the production feature pipeline. Branch deployments take only a few seconds to deploy. ``` $ chalk apply --branch tutorial ✓ Found resolvers ✓ Deployed branch ``` ### Querying Now that we've deployed our feature pipeline, we can query our data in realtime. One of the easiest ways to do this is from the Chalk CLI. ``` $ chalk query --in user.id=1 --out user.name --out user.email user.name "John Doe" email "john@doe.com" ``` This query will fetch the name and email attributes from the User feature class for the user with id=1, hitting the PostgreSQL database directly. ### Push-down filters Note that in SQL file resolver that we wrote, we didn't include a where clause. However, Chalk automatically pushes down filters to the database when querying features. So, the SQL that will execute against our PostgreSQL database will be: ``` select id, full_name as name, email from users + where id = 1; ``` Chalk can also push down non-primary key filters to SQL databases. For example, to fetch all transactions for a user, Chalk will modify the SQL-resolver query to include a where clause: ``` select id, account_id, amount, status, date from txns + where account_id = 38; ``` ### Offline data In addition to online data, we can also ingest data from SQL databases into Chalk's offline store. Offline data won't be queried in realtime, but can be used to train models and generate features. For our Account feature class, we may want to ingest data from a Snowflake table. We can write a SQL file resolver to do this: ``` -- Incrementally ingest account data from Snowflake. -- This comment will be searchable in the Chalk dashboard. -- -- resolves: account -- source: snowflake -- type: offline -- cron: 5m -- incremental: -- mode: row -- lookback: 1h select id, user_id, amount, updated_at from accounts; ``` There are a few differences between this SQL file resolver and the one we wrote for the User feature class. First, we've added a type key to the header. This tells Chalk that this resolver should be used to ingest data into the offline store. If we didn't include this key, Chalk would assume that this resolver could be queried in realtime. Second, we've added a cron key to the header. This tells Chalk to run this resolver on a schedule. In this case, we're telling Chalk to run this resolver every 5 minutes. Finally, we've added an incremental key to the header. This tells Chalk to only ingest new data from the database, and is helpful when you have an immutable events table. Also, notice the new updated_at column in the select statement. We'll map that column to a FeatureTime attribute in our feature class: ``` from chalk.features import feature, features, FeatureTime @features class Account: id: int user_id: int amount: float + updated_at: FeatureTime ``` Features with overridden observation timestamps are inserted into the offline store with the timestamp that you specify. The observation timestamp works like an "effective as of" timestamp. When you sample historical data, you can specify the observation timestamp at which you want to sample a feature value. Then, Chalk will return the most-recent feature value that was observed before that timestamp. This method of sampling ensures temporal consistency in your feature values. ### Reverse ETL While our offline data is useful for training models and generating features, we may also want to use these values for serving production queries. However, data warehouses like Snowflake and BigQuery are optimized for analytics and are not well-suited for transactional queries. We can have Chalk reverse-ETL our offline data into our online store by setting the max_staleness and etl_offline_to_online keyword arguments on our @features decorator: ``` - @features + @features(max_staleness="infinity", etl_offline_to_online=True) class Account: id: int user_id: int amount: float updated_at: FeatureTime ``` The max_staleness keyword argument tells Chalk how stale a feature value can be before it should be refreshed. In this case, we're telling Chalk that we'll tolerate arbitrarily old feature values. However, we could also specify a max_staleness of 1h or 1d to tell Chalk not to serve feature values that are older than 1 hour or 1 day. The etl_offline_to_online keyword argument tells Chalk to reverse-ETL our offline data into our online store. By default, data only enters the online store when it's queried in realtime. However, by setting this keyword argument, we're telling Chalk to reverse-ETL our offline data into our online store. # Tutorial: Data Modeling source: https://docs.chalk.ai/docs/fraud-2 ## Defining the features that we want to compute. If you want to skip ahead, you can find the full source code for this tutorial on GitHub. In this example, we'll consider a fintech use-case where we want to detect fraudulent credit card purchases. Our data consists of a list of credit card transactions, each with a timestamp, a location, and a purchase amount. We also have information about the cardholder and the accounts that the card is linked to. ### Define features We'll start by modeling the features we want for our the users in our system. We'll start simple with three scalar features: user.id, user.name, and user.email. First, we'll create a new file called models.py where we'll define a User class decorated with @features. ``` from chalk.features import features @features class User: id: int # The name the user provided to us at signup. # :owner: identity@chalk.ai # :tags: pii name: str # :tags: pii email: str ``` Note that at this point, we haven't defined how to compute these features. We are only thinking about the data that we would like to have. ### Primary keys There are a few things to note here. First, all our feature classes need to have a unique id field. By default, this is the field named id. However, if you want to use a different field as the primary key, you can specify it using the Primary argument to @features. ``` from chalk.features import features @features class User: - id: int + user_id: Primary[int] name: str email: str ``` ### Tags, Descriptions, & Owners In our features below, we've added some comments and annotations to our features. These are optional, but can be useful for documentation and for setting alerting policies. For example, you may wish to send PagerDuty alerts to different teams based on the owner of the related feature. Any of the comments and tags from the code also show up in the Chalk dashboard, and are indexed for search. For example, we've added a pii tag to the name and email fields. This means that these fields will be treated as personally identifiable information and will be subject to additional restrictions. ### Has-One Relationships Next up, we'll define a related feature class to our users. We'll call this class Account and it will represent a bank account that a user owns. ``` from chalk.features import features @features class Account: id: int # The name of the owner of the account. title: str # The id of the user that owns this account. user_id: int # The balance of the account, in dollars. balance: float ``` This should look much like what we did for the User class. However, we may want to link these two classes together. We can do this by adding a user field to the Account class. ``` @features class Account: id: int - user_id: int + user_id: User.id balance: float + # The user that owns this account. + user: User ``` This denotes that each account has one user, and that the Account.user_id and the User.id are equal and of type int, as described by Account.user_id. Once we've defined the relationship on one side of the join, we can define the inverse relationship on the other side without needing to specify the predicate again. ``` @features class User: id: int name: str email: str + # The account that this user owns. + account: "Account" ``` ### Has-Many Relationships The final feature entity that we'll define in this tutorial is for transactions. Each account has many transactions, and each transaction is linked to a single account. We'll define the Transaction class and link it to our Account class as follows: ``` - from chalk.features import features + from chalk.features import features, DataFrame + class TransactionStatus(str, Enum): + PENDING = "pending" + CLEARED = "cleared" + FAILED = "failed" + + @features + class Transaction: + id: int + + # The id of the account that this transaction belongs to, set to a join. + account_id: "Account.id" + + # The amount of the transaction, in dollars. + amount: float + + # The status of the transaction, defined as an enum above. + status: TransactionStatus + + # Because we define the join condition between + # `Transaction` and `Account` below, we don't + # need to repeat it here. + account: "Account" @features class User: id: int name: str email: str # The account that this user owns. account: "Account" @features class Account: id: int user_id: User.id balance: float user: User + transactions: DataFrame[Transaction] ``` This is the first time we're seeing the DataFrame type. A Chalk DataFrame models tabular data in much the same way that pandas does. However, there are some key differences that allow the Chalk DataFrame to increase type safety and performance. Like pandas, Chalk's DataFrame is a two-dimensional data structure with rows and columns. You can perform operations like filtering, grouping, and aggregating on a DataFrame. However, there are two main differences. - Lazy implementation - Chalk's DataFrame is lazy and can be backed by multiple data sources, where a pandas.DataFrame executes eagerly in memory. - Use as a type - Chalk's DataFrame[...] can be used to represent a type of data with pre-defined filters. You can read more about the Chalk DataFrame in the docs and API Reference. You might also notice that we've used an Enum feature here. Chalk supports many feature types, including Enum, lists and sets, and @dataclasses. # Feature Studio source: https://docs.chalk.ai/docs/feature-studio ## Define, deploy, and manage features and resolvers with Chalk's Feature Studio. Chalk's Feature Studio provides a no-code interface within the web dashboard for users to define, deploy, and manage features and resolvers. The Feature Studio provides intuitive forms and visualizations to help users create and edit features in collaboration either entirely independently or in conjunction with Chalk's traditional Python-based interface. After defining features and resolvers in the Feature Studio, users can deploy their changes either to branches for testing, or directly as mainline deployments. From there, users can use the Online and Offline Data Explorers in the dashboard to trigger online and offline queries to validate their new feature pipelines, and create datasets for machine learning workflows. This workflow is still in alpha--please reach out to the Chalk team if you would like to try it out. ### Getting Started To access the Feature Studio, navigate to the Feature Studio tab in the Chalk dashboard. The studio currently loads the graph of all features and resolvers defined in the current active deployment in the environment. All features are editable, and show their current definitions, types, and metadata (including description, owner, tags, and max_staleness). All resolvers show their definitions, metadata, and links to upstream and downstream features. Feature Studio Home Page ### Developing in the Feature Studio In the dashboard, users can click the Add New button to define a new feature namespace, feature within an existing namespace, or resolver. Similar to the Pydantic-style definitions in Chalk's Python-based interface, users can define typed features with additional metadata. Feature Studio: Add New Feature In addition to creating new features, users can also edit their features upon creation, or modify existing features in the current deployment. Feature Studio: Edit Feature After defining the feature schema, users can define resolvers that indicate to the Chalk engine how feature values should be computed. SQL resolvers enable users to write SQL queries to load and transform data from their integrated data sources. Python resolvers provide the flexibility to call API's, execute models, and perform arbitrary computations to generate feature values. Chalk Expressions enable users to write simple, intuitive computations referencing other features. Feature Studio: Add SQL Resolver The Feature Studio supports the definition of more complex features as well, such as Windowed Aggregations. Through the feature definition modal, users can define time-based aggregations with custom filters based on has-many relationships between feature classes. Feature Studio: Add Windowed Agg After defining the necessary features and resolvers for a new use case, users can preview all of the changes made in their current studio session, including all new, modified, and deleted components. Feature Studio: Preview Changes Once satisfied with the changes, users can deploy their new graph either to the branch server for testing, or directly as a mainline deployment for use in production. Feature Studio: Deploy Changes ### Collaboration in the Feature Studio The Feature Studio is designed to enable concurrent collaboration between multiple users, the same way that multiple users can work on branches in Chalk's Python-based interface. In addition, many teams have some users who prefer to work in the Feature Studio, while others prefer the code-based interface. The Feature Studio supports this hybrid workflow as well, enabling imports of graphs as defined in code, as well as code exports of changes deployed from the Feature Studio. # Support source: https://docs.chalk.ai/docs/enterprise-support ## Get in touch with our support team! Our documentation is designed to be as comprehensive and easy to understand as possible. In addition, we have a team of Support Engineers and Forward Deployed Engineers who are available to answer customer questions and help to troubleshoot any issues that arise. All Chalk customers have channels of communication available with them to get in touch with our production support team and file tickets. Finally, the Chalk on-call engineering team does have a 24/7 paging system and alerting configured for customer environments to ensure that we are able to respond and resolve outages in a timely manner. We ask that customers also configure alerting and paging accordingly such that the Chalk team can be most effective in responding to and mitigating these potential issues. There are two levels of support available to customers: Standard Support and Enterprise Support. All customers have access to Standard Support, which enables customers to contact the Chalk team, and create support tickets. The Chalk team is distributed across multiple time zones, and so strives to respond to all tickets and messages in a timely fashion. ### Standard Support Customers can create support tickets by starting a new thread and adding the 🎫 ticket reaction. Our production support team monitors all tickets and will respond in a timely manner. If you have an urgent issue or need support with an outage, please ensure to highlight the urgency of the issue in the ticket and our team will prioritize accordingly. All customers can also view the status of their tickets through the Pylon Customer Portal. ### Enterprise Support In addition to our timely and friendly support provided for all customers, we also have an Enterprise Support plan that customers can choose to include in their bundles. This allows for a 24/7 Pagerduty line to page our on-call engineer for urgent issues that require immediate attention. Enterprise support customers are provided with three ways to page our on-call engineering team, outlined below. Our on-call rotation is staffed 24/7 year-round. We ask that customers provide context about the issue or incident when paging our on-call team so that we can respond as effectively as possible, especially in case of outages. ### 1. Paging via the dashboard To page via the dashboard, click your username at the bottom left hand corner of the dashboard in the menu bar, then click Escalate Issue. This will lead to a modal that enables you to provide more information about the incident. ### 2. Paging via Slack Enteprise support customers can also page our on-call engineering team via Slack by adding the 🎫 ticket reaction to a message, marking the ticket as urgent, and then selecting the option to page our on-call team. The urgency option looks like this: This option to page our on-call will open another modal that allows you to provide more information about the incident: ### 3. Paging via email Enterprise support customers are given an emergency email address. Each message sent to this address immediately pages our on-call engineering team. # Enterprise Deployment source: https://docs.chalk.ai/docs/deployment ## Enterprise Deployment Model Chalk integrates with your data sources, transform this data with feature pipelines, store this data in online and offline storage, and provide monitoring on feature computation and distributions. ### Enterprise Service Architecture Chalk offers a hosted model ("Chalk Cloud"), or a customer-hosted model ("Customer Cloud"). There are a few main components of a Chalk deployment: - Management - serves non-customer data (like alert configuration and RBAC configuration). - Builder - builds containers that run your feature pipelines. - Compute - machines that run your feature pipelines. In both AWS and GCP, the compute runs on Kubernetes with Knative. - Customer Data - the online and offline stores for your features. - Secrets - environment variables and configuration for your data sources. These components are organized as follows: ### Data Isolation In the enterprise deployment, your API clients talk directly to the compute engine deployed into your cloud. Feature values, historical and online, do not escape your cloud project. ### Storage Chalk uses different storage technologies to support online and offline use-cases. The online store is optimized for serving the latest version of any given feature for any given entity with the minimum possible latency. Behind the scenes, Chalk uses a key-value store for this purpose. Chalk can be configured to use Redis or Cloud Memory Store for smaller resident data sets with high latency requirements, or DynamoDB when horizontal scalability is required. The offline store is optimized for storing all historical feature values, serving point-in-time correct queries, and tracking provenance of features. Chalk supports a variety of storage backends depending on data scale and latency requirements. Typically, Chalk uses TimescaleDB, Redshift, or BigQuery. # Chalk Machine Type defaults source: https://docs.chalk.ai/docs/chalk-machine-type-catalog ## Search default primary instances and ordered fallbacks by cloud provider, workload, and Chalk Machine Type. Use this catalog to find the default primary instance and ordered fallbacks for Chalk Machine Types across cloud providers. It includes online workloads without GPUs, offline workloads with disk spilling enabled, and infrastructure workloads. Fallbacks appear in descending preference order. Regional availability can remove candidates and promote a fallback to the primary instance. Cluster and environment overrides can replace the default list. The catalog shows built-in defaults before regional filtering and overrides. # Chalk Billing source: https://docs.chalk.ai/docs/billing ## Learn about Chalk's billing model and how to optimize costs. Chalk charges for machine uptime of nodes labeled as Chalk Managed. We calculate this by multiplying: - The instance type - Chalk’s credit utilization rate/hour for that instance type - The number of hours (i.e., uptime) that the node remains active. Rather than rounding to the nearest hour, Chalk meters usage in 10-second increments. Regardless of the load placed on a Chalk-managed node, credit usage remains the same. We charge for the node’s uptime, not based on the level of activity within the node. You can run unlimited queries or create multiple branches without incurring extra charges. Billing is based solely on the node’s active hours, not on how many features or branches are used. ### Finding Chalk credit rates You can see the full list of credit utilization rates per hour for all machine types in the Chalk dashboard under Settings > Usage > Utilization Rates. To view the full list of credit utilization rates in your terminal, log into Chalk in the terminal and use the Chalk CLI command to view usage rates. ``` $ chalk login $ chalk usage rates ``` ### Cost optimization tips There are two primary ways to optimize your resource configurations for cost optimization. ### Configure auto-shutdown periods The branch server shuts down after a period with no requests, which stops its costs. The default period is 1 hour. To change it, go to Infrastructure > Resource Configuration > Branch Server in the dashboard and set Auto Shutdown Period. See Auto-shutdown for the available values. When you use the branch server again, chalk apply --branch, chalk query --branch, ChalkClient, and the dashboard start it. Queries sent over gRPC do not. See Start and stop the branch server. ### Configure resource configurations For the other nodes in your cluster, you can configure the resource configurations according to your needs in the dashboard under Infrastructure > Resource Configuration. You can select the instance type for each server that is right for you, and set a scaling policy that ensures that you scale up when traffic is high, but notably that your nodes scale down when traffic is low. The instance type selected should have enough capacity to support the resource request multiplied by the number of max instances, plus additional overhead. ### Frequently asked questions ### Do I get charged for pods or for nodes? You are charged per node. Multiple Chalk pods on a single node does not change the amount of the credit utilization. ### What if we are running non-Chalk workloads on the same cluster? Chalk only applies charges to nodes labeled “Chalk Managed.” Other workloads on different nodes do not incur Chalk charges. You can also configure your cluster to have isolated nodepools to ensure that unrelated workloads are completely isolated from your Chalk Managed nodes. ### What do these labels in the nodepool configuration pane mean? You may notice labels like Chalk, Restricted, and Isolated in the nodepools settings pane. Different nodepool labels - Chalk: This label indicates that the nodepool makes Chalk-managed nodes, whose uptime contributes to credit usage. All nodepools created through the Chalk dashboard automatically have this label, although the dashboard will also display nodepools without it. - Isolated: This label indicates that the nodepool will not allow any workloads to schedule on its nodes unless they are explicitly configured to do so by setting the Nodepool field in the Resources settings pane. Controlled by the checkbox "Isolate this nodepool". - Restricted: This label indicates that the nodepool makes nodes that only allow Chalk workloads to schedule on them. This is useful in the case where Chalk is running in your cluster alongside other unrelated workloads, and it prevents those workloads from inadvertently consuming Chalk credits. Controlled by the checkbox "Restrict to Chalk workloads only". Chalk workloads are only allowed to schedule on nodes from nodepools with the Chalk label. They will not schedule on an Isolated nodepool unless configured to do so, and the Restricted label has no effect on whether Chalk workloads can schedule on the nodepool, but only affects unrelated non-Chalk workloads. ### Do more branches cost more money? No. Because one pod runs all branches, there is no extra cost for having multiple branches. ### Do I get charged if nobody is running queries but the node is still up? Yes, if the node is up, you are accumulating costs. You can mitigate this by setting the auto-shutdown period for the branch server for periods of inactivity, as well as by configuring a scaling policy for your other workloads. ### Does Chalk charge for storage or staging environments? Chalk only charges for compute resources (i.e., running nodes). If a staging environment is idle or shut down, it won’t incur costs. ### Chalk marketplace billing Chalk supports marketplace billing through both AWS and GCP. With this setup, your Chalk usage is billed directly to your cloud infrastructure provider, appearing as a line item on your existing bill and simplifying your payment process. If you’re interested in enabling marketplace billing, reach out to your Chalk representative. Our finance team will guide you through the required steps and coordinate a private offer. Once finalized, you’ll be able to receive and pay your Chalk charges directly through AWS Marketplace or GCP Marketplace. Depending on your Chalk agreement, we can accommodate: - Prepaid bundles, invoiced up front. - Standard usage-based billing, invoiced monthly in arrears. For example, your April Chalk usage will appear on your April cloud provider bill. The benefits of marketplace billing include: - No extra invoices to manage - Chalk charges consolidated with your existing infrastructure spend - Available on both AWS and GCP # Benchmarking source: https://docs.chalk.ai/docs/benchmark ## Preflighting scale to production After setting up your features, resolvers, and queries, you'll want to confirm that your deployment meets your latency and throughput requirements. Chalk ships a benchmarking tool that runs load against a query host from inside your cluster and reports latency percentiles, throughput, and per-request data. To see the most up-to-date options, run chalk benchmark --help (or chalk benchmark run --help) in your terminal or navigate to our CLI docs. Benchmarking is an alpha feature. Flags and defaults may change between CLI releases — always cross-check against --help. ### Command Overview The benchmark tool is exposed as a group of subcommands: | Command | Purpose | | ------------------------------ | ------------------------------------------------------------------- | | `chalk benchmark run` | Launch a benchmark run against a query host on Kubernetes. | | `chalk benchmark upload` | Upload local `.parquet` / `.json` files to use as benchmark inputs. | | `chalk benchmark list-inputs` | List the input files available for benchmarking. | | `chalk benchmark list-results` | List result files from past runs. | | `chalk benchmark download` | Download one or more result files to the current directory. | | `chalk benchmark kill` | Cancel an in-flight benchmark run by id. | Every run produces a server-generated benchmark id. The CLI prints it on submission: ``` ✓ Benchmark workflow submitted with ID 7f3c1e0a-... ``` That id is the key to retrieving your results — result artifacts are named with it as a prefix (see Retrieving results). ### Best Practices ### Statically Compile Resolvers To get the fastest latencies, ensure your resolvers are statically compiled. - Static DataFrame resolvers: Resolvers defined with @online(static=True) are symbolically compiled into chalkdf / libchalk expressions and executed as a vectorized, native plan. See the Static Resolvers Tutorial. - Feature expressions: Use Chalk's underscore (_) and F.* expressions (for example F.json_value, F.http_post, arithmetic on features). These compile to native operators. - Python resolvers: Chalk will attempt to statically optimize all python resolvers. See Static Resolver Optimization for more detail and best practices. ### Named Queries When interested in a particular query, it's best practice to benchmark against Named Queries. - Simplifies benchmark command - Named Queries provide a single name to represent a repeatable and potentially large number of input and output features - Production encouraged - Named Queries are planned on engine boot which is encouraged for production to prevent first query latency spikes on new deployments. - Plan cache hashing - Queries with a larger number of requested inputs and outputs will need to be hashed to grab the pre-computed plan from the cache. Named Queries simplify the hashing and cache retrieval. ### Scale It's important to isolate and scale your resources to match the level of benchmark you are performing. The larger the QPS, the larger and/or more machines you'll need. With more cached features, you may also need to increase your online store's capacity. In the case of Valkey/Redis, this could mean increasing your cluster size and/or node capacity. The engine layer scales roughly linearly with replicas. Other services will possibly need to be scaled to keep pace (online store, data sources, envoy). You can read more about adjusting these resources on the Resource Configuration page. ### Tutorial: Benchmarking with a Custom Input Distribution If wanting to benchmark against a specific distribution of keys, this can be done by constructing an input parquet file and using the --in-file flag. ### Step 1: Build the Input Parquet File Each column in the Parquet file is a fully-qualified input feature name (the same string you would pass to --in, e.g. user.id). Each row is the input for one query. The benchmark client reads rows from this file to drive the load. Below, we generate one million user.id values drawn from a distribution rather than a uniform range — here a Zipfian-style skew so that a small set of "hot" users appears far more often, which exercises your cache the way production traffic does. ``` import numpy as np import pandas as pd N_ROWS = 1_000_000 N_USERS = 200_000 # Skewed (Zipf-like) sampling: a few users are very hot, most are rare. rng = np.random.default_rng(seed=42) raw = rng.zipf(a=1.3, size=N_ROWS) user_ids = (raw % N_USERS) + 1 # map into the valid id range [1, N_USERS] df = pd.DataFrame( { # Column name MUST be the fully-qualified input feature name. "user.id": user_ids.astype("int64"), } ) df.write_parquet("user_inputs.parquet") ``` If your query takes multiple inputs, add one column per input feature: ``` df = pd.DataFrame( { "user.id": user_ids, "merchant.id": merchant_ids, } ) ``` Match the feature types exactly — if user.id is a string in your schema, write a string column, not an integer column. ### Step 2: Upload the Input File Upload the file to your environment's benchmark storage. The CLI uses the file's base name as the input name you'll reference later. ``` chalk benchmark upload --file-path user_inputs.parquet ``` ``` ✓ Input files uploaded under the names [user_inputs.parquet] ``` Only .parquet and .json files are accepted. You can pass --file-path multiple times to upload several files at once. Confirm it's available (optionally filtering by a name prefix): ``` chalk benchmark list-inputs chalk benchmark list-inputs --prefix user_ ``` ### Step 3: Run the Benchmark Point the run at the uploaded file with --in-file, choose your outputs, and set the load shape. ``` chalk benchmark run \ --in-file user_inputs.parquet \ --out user.email \ --out user.risk_score \ --qps 500 \ --duration 5m \ --warmup-qps 200 \ --warmup-duration 1m \ --percentile 50 --percentile 95 --percentile 99 \ --result-target html --result-target parquet ``` The CLI prints a configuration table, asks for confirmation (skip it with --force), then submits the run and prints the benchmark id: ``` ✓ Benchmark workflow submitted with ID 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f ``` Note the id — you'll use it as the prefix to find your results. ### Step 4: List the Result Files Results are written once the run completes. List them, filtering by your benchmark id prefix so you only see artifacts from this run: ``` chalk benchmark list-results --prefix 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f ``` ``` Name Updated At Size 7f3c1e0a-..._result.tar.gz Mon, 12 Jun 2026 10:04:11 PDT 18234 7f3c1e0a-..._result.parquet Mon, 12 Jun 2026 10:04:11 PDT 402114 ``` Result artifacts are named _result.: - *_result.tar.gz — the HTML report bundle (always produced). - *_result.parquet — per-request latencies and metadata (when --result-target parquet). - *_result.json — a structured summary report (when --result-target json). The --prefix filter is a path prefix match on the result file name. Passing the full benchmark id returns every artifact for that single run; a shorter prefix matches multiple runs. ### Step 5: Retrieve the Result File Download by full name (including the extension) into the current directory: ``` chalk benchmark download \ --name 7f3c1e0a-..._result.tar.gz \ --name 7f3c1e0a-..._result.parquet ``` ``` ✓ Result file(s) downloaded to current directory ``` Unpack the HTML bundle to view the report, or load the Parquet file to analyze per-request latency yourself: ``` tar -xzf 7f3c1e0a-..._result.tar.gz ``` ``` import pandas as pd df = pd.read_parquet("7f3c1e0a-..._result.parquet") ``` ### Cancelling a Run If you need to stop an in-flight benchmark: ``` chalk benchmark kill --id 7f3c1e0a-3b2a-4f9c-9d1e-1a2b3c4d5e6f ``` ### Path to Success Benchmarking is most useful as an iterative loop: characterize a single engine first, then scale out. A reliable progression: ### 1. Get Your Pipeline Accelerated Convert resolvers to static=True and push logic into feature expressions (see Statically Compile Resolvers). ### 2. Benchmark One Engine in Isolation Configure your gRPC resources to a single engine instance to understand the baseline performance of your specific query: ``` chalk benchmark run \ --in-file user_inputs.parquet \ --out user.email \ --dedicated-engine \ --dedicated-engine-replicas 1 \ --qps 100 \ --duration 3m \ --warmup-qps 50 --warmup-duration 1m \ --percentile 50 --percentile 95 --percentile 99 ``` ### 3. Run on the Latest Compute-Optimized Instances For the most latency-sensitive applications, schedule the engine on current-generation compute-optimized instances — for example AWS c8i (or the newest c-family generation available in your region), or the equivalent compute-optimized SKUs on GCP/Azure. The newest generations give the best single-core performance and memory bandwidth. Depending on your cost vs. latency preferences, using an older compute optimized instance can work as well - for example AWS c7i. ### 4. Find the Optimal QPS for One Engine Sweep QPS upward, holding everything else constant, and watch your target percentile. Increase QPS until p99 starts to climb past your SLO or throughput stops scaling linearly with offered load — that knee is the sustainable QPS for one engine. Record the engine configuration (CPU/memory, env vars, instance type) and that per-engine QPS. Keep the benchmark client itself from becoming the bottleneck. Give it enough CPU/memory (--cpu-request/--memory-request) and connections so that it can actually offer the QPS you ask for. ### 5. Scale Out Roughly Linearly with Engines Once you know a single engine sustains, say, 500 QPS at an acceptable p99, you can scale to your target throughput by adding engines roughly linearly: to serve ~2,500 QPS, run ~5 replicas of the same engine configuration. Verify with a larger run rather than assuming perfect linearity: ``` chalk benchmark run \ --in-file user_inputs.parquet \ --out user.email \ --dedicated-engine \ --dedicated-engine-replicas 5 \ --qps 2500 \ --duration 5m \ --warmup-qps 1000 --warmup-duration 1m \ --percentile 50 --percentile 95 --percentile 99 ``` If p99 holds, you've validated your scaling factor. If it degrades, a shared dependency — such as the online store or envoy — is likely the new bottleneck; see Scale. # AWS Cloud Deployment IAM Requirements source: https://docs.chalk.ai/docs/aws-cloud-deployment ## Deploying Chalk to your AWS account. Chalk's feature platform with best-in-class developer experience enables machine learning teams to focus on building the unique products and models that make their business stand out. Chalk provides a feature store so that you can deploy production machine learning pipelines for real time data in minutes. Chalk is both a framework and a platform — developers can write code using familiar Python packages, and deploy their feature and data pipeline definitions to Chalk’s platform. In the Customer Cloud deployment, Chalk runs & administers its platform on the customer’s cloud account. Chalk's managed infrastructure then executes the customer defined pipelines to compute feature data for machine learning applications. Chalk then serves this data back to customer applications for online inference and to customer data teams for training set generation. ### Architecture ### IAM Role Permissions In order to manage infrastructure in your cloud account, Chalk requires certain IAM permissions. At a high level, Chalk needs the ability to provision the key components of your infrastructure: - Storage resources (buckets for Dataset storage and bulk insertion into offline storage) - Networking resources (LBs, VPCs, etc.) - IAM resources (e.g. creating service accounts for workload identity, etc.) - Kubernetes resources (EKS) - Online storage (RDS, or Elasticache, or Dynamo) - Offline storage (typically Snowflake, Iceberg, or Redshift) ### Setup Instructions We typically recommend the following steps for enabling Chalk to provision and manage AWS infrastructure in your cloud account: - Create a new account in your AWS organization. - Create a new role in the new account that enables Chalk's server to perform actions in this account. Ensure that AssumeRole is granted. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::754784422779:role/chalk-api-server" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` - Create an IAM policy using the JSON document below and attach this IAM policy to the created role. ``` { "Statement": [ { "Resource": "*", "Action": [ "sqs:*", "sns:*", "acm:*", "secretsmanager:*", "s3:*", "redshift:*", "redshift-data:*", "redshift-serverless:*", "rds:*", "logs:*", "kms:*", "kafkaconnect:*", "kafka:*", "kafka-cluster:*", "iam:CreateServiceLinkedRole", "iam:UploadServerCertificate", "iam:UploadSSHPublicKey", "iam:UpdateServerCertificate", "iam:UpdateRoleDescription", "iam:UpdateRole", "iam:UpdateOpenIDConnectProviderThumbprint", "iam:UpdateAssumeRolePolicy", "iam:RemoveRoleFromInstanceProfile", "iam:RemoveClientIDFromOpenIDConnectProvider", "iam:PutRolePolicy", "iam:PassRole", "iam:ListSSHPublicKeys", "iam:ListRoles", "iam:ListRolePolicies", "iam:ListRoleTags", "iam:ListPolicyVersions", "iam:ListPolicyTags", "iam:ListPolicies", "iam:ListOpenIDConnectProviders", "iam:ListOpenIDConnectProviderTags", "iam:ListAttachedRolePolicies", "iam:GetServerCertificate", "iam:GetSSHPublicKey", "iam:GetRolePolicy", "iam:GetRole", "iam:GetPolicyVersion", "iam:GetPolicy", "iam:GetOpenIDConnectProvider", "iam:GetInstanceProfile", "iam:DetachRolePolicy", "iam:DeleteServiceLinkedRole", "iam:DeleteServerCertificate", "iam:DeleteRolePolicy", "iam:DeleteRole", "iam:DeleteOpenIDConnectProvider", "iam:DeleteInstanceProfile", "iam:CreateServiceLinkedRole", "iam:CreateRole", "iam:CreatePolicyVersion", "iam:CreatePolicy", "iam:CreateOpenIDConnectProvider", "iam:CreateInstanceProfile", "iam:AttachRolePolicy", "iam:TagPolicy", "iam:TagRole", "iam:TagOpenIDConnectProvider", "iam:AddRoleToInstanceProfile", "iam:AddClientIDToOpenIDConnectProvider", "iam:ListInstanceProfilesForRole", "iam:DeletePolicy", "iam:UntagPolicy", "iam:UntagRole", "iam:UntagOpenIDConnectProvider", "elasticloadbalancing:*", "eks:*", "ecr:*", "ec2:*", "cloudwatch:*", "autoscaling:*", "dynamodb:*", "dax:*", "application-autoscaling:*", "elasticache:CreateCacheSubnetGroup", "elasticache:DeleteCacheSubnetGroup", "elasticache:DescribeCacheSubnetGroups", "elasticache:ModifyCacheSubnetGroup", "elasticache:CreateReplicationGroup", "elasticache:DeleteReplicationGroup", "elasticache:DescribeReplicationGroups", "elasticache:ModifyReplicationGroup", "elasticache:DescribeCacheClusters", "elasticache:AddTagsToResource", "elasticache:RemoveTagsFromResource", "elasticache:ListTagsForResource", "glue:*", "route53:*" ], "Effect": "Allow" }, { "Resource": "*", "Action": [ "iam:GetContextKeysForPrincipalPolicy", "iam:SimulatePrincipalPolicy" ], "Effect": "Allow" } ], "Version": "2012-10-17" } ``` ### What are these permissions for? - cloudwatch: The metrics viewer in the web UI - logs: Logs for the web UI - acm: Provisioning SSL certs for client<>server encryption - rds: RDS Online store - redshift: Redshift offline store - redshift-data: Redshift offline store - redshift-serverless: Redshift offline store - kafka: Asynchronous persistence queues for metrics & feature storage - kafka-cluster: Asynchronous persistence queues for metrics & feature storage - dynamodb: DynamoDB online store - dax: DynamoDB online store - application-autoscaling: DynamoDB online store, if auto-scaling is required - ec2: ALB and EKS node pool management - ecr: ECR image management for deployments - eks: EKS cluster management for running feature engineering workloads - kms: Encryption keys for secrets - secretsmanager: Encrypting data source secrets from the web UI - glue: Iceberg Glue offline store - iam:GetContextKeysForPrincipalPolicy, iam:SimulatePrincipalPolicy: Checking the effective permissions of Chalk's IAM role without executing the simulated actions ### Ongoing permissions Chalk's support team will work with you to scope permissions down to what are needed for ongoing maintenance. The precise details depend on the level of ongoing support that your team needs and your compliance requirements. In principle, Chalk does not require ongoing access to data, or to the ability to edit IAM permissions, but Chalk requires ongoing access to update the software deployed in your environment. For managing resource configurations such as autoscaling, the Chalk team can use an execution role that is scoped to the specific resources that Chalk manages. The scoped execution role uses resource tagging to restrict permissions to only resources managed by Chalk (tagged with chalk.ai/managed-by: chalk). This provides a secure way to limit Chalk's access to only the infrastructure it creates and maintains. For AWS maintenance, Chalk plans and applies with Terraform refresh disabled, using the last recorded state to detect configuration changes. These runs do not detect out-of-band changes to managed resources, and Terraform still reads data sources. Changes that create, update, or delete IAM resources require an IAM-capable execution role; the scoped role below covers runs without IAM changes. This policy assumes setup is complete. It supports upgrades to existing EKS clusters and node groups, Helm upgrades, and ongoing access to container images, storage, and secrets. Setup and discovery, IAM permission simulation, CloudWatch metrics and logs, and DynamoDB and ElastiCache dashboard metadata are outside this scope. The kms:Decrypt permission supports reading existing Kafka credentials encrypted with customer-managed keys. Creating or changing KMS keys requires a separately authorized role. The IAM policy for the scoped execution role is composed of the following statements: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "ChalkManagedResources", "Effect": "Allow", "Action": [ "eks:*", "ecr:*", "ec2:*", "secretsmanager:*", "kms:Decrypt" ], "Resource": [ "arn:aws:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:*", "arn:aws:ecr:*", "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:*", "arn:aws:secretsmanager:${AWS_REGION}:${AWS_ACCOUNT_ID}:*", "arn:aws:kms:${AWS_REGION}:${AWS_ACCOUNT_ID}:*" ], "Condition": { "StringEquals": { "aws:ResourceTag/chalk.ai/managed-by": "chalk" } } }, { "Sid": "ChalkUntaggedResources", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::chalk-*" ] }, { "Sid": "ChalkGlobalDescribe", "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:Describe*", "ec2:Describe*" ], "Resource": "*" }, { "Sid": "ChalkRunInstances", "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": [ "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:instance/*", "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:volume/*", "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:network-interface/*", "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:subnet/*", "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:security-group/*", "arn:aws:ec2:${AWS_REGION}::image/*", "arn:aws:ec2:${AWS_REGION}::snapshot/*" ] }, { "Sid": "ChalkTagOnCreate", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": [ "arn:aws:ec2:${AWS_REGION}:${AWS_ACCOUNT_ID}:instance/*" ], "Condition": { "StringEquals": { "ec2:CreateAction": "RunInstances" } } } ] } ``` Most permissions are gated by the aws:ResourceTag/chalk.ai/managed-by condition. The following statements use other restrictions: - S3 buckets (ChalkUntaggedResources): restricted by the existing chalk-* name pattern. - ECR and EC2 reads (ChalkGlobalDescribe): permit image authentication and metadata lookup, and reads of existing network configuration. - ec2:RunInstances (ChalkRunInstances): the instance, volume, and network interface it creates have no tags yet. EKS AMIs and their snapshots are AWS-owned. Subnets and security groups are included to support untagged network resources. - ec2:CreateTags (ChalkTagOnCreate): limited to tagging instances during RunInstances through the ec2:CreateAction condition. ### Kubernetes Chalk requires the following Kubernetes roles to manage the resources in your cluster: ### Managed Components Depending on cluster settings and enabled Chalk features, AWS customer-cloud deployments manage the following Kubernetes platform components in EKS. ### EKS Add-ons - CoreDNS - kube-proxy - Amazon VPC CNI ### Helm Charts - Argo Workflows - CloudNativePG - Envoy Gateway - Karpenter - KEDA - external-dns - metrics-server - aws-mountpoint-s3-csi-driver - cert-manager - AWS Load Balancer Controller ### EKS Version Support Chalk supports EKS versions up to two Kubernetes minor versions back, and Chalk updates managed clusters approximately one month before the end of AWS standard support for the in-use EKS version. ``` kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: chalk-cluster-management-role rules: - apiGroups: - "" resources: - nodes # required to track usage verbs: - get - list - watch # For allowing the web UI to manage cluster scaling - apiGroups: - "karpenter.sh" resources: - nodepools verbs: - get - list - create - update - patch - delete - apiGroups: - "karpenter.k8s.aws" resources: - ec2nodeclasses verbs: - get - list - create - update - patch - delete # For read/list access to Karpenter NodeClaims - apiGroups: - "karpenter.sh" resources: - nodeclaims verbs: - get - list - apiGroups: - "" resources: - persistentvolumes verbs: - get - list - create - update - patch - watch - delete - apiGroups: - "storage.k8s.io" resources: - storageclasses verbs: - get - list --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: chalk-management-role namespace: # replace with your actual namespace rules: - apiGroups: - "" resources: - configmaps - pods - services verbs: - get - list - create - update - patch - watch # For support w/ debugging & rendering logs in the dashboard. - apiGroups: - "" resources: - pods/log verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses - ingresses/status - ingressclasses verbs: - get - list - watch - delete - update - create - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses # can be self-managed if necessary verbs: - get - create - delete - update - list - patch - apiGroups: - "" resources: - secrets verbs: - get - list - watch - create - patch - update - delete - apiGroups: - "" resources: - events verbs: - get - list - watch - apiGroups: - "apps" resources: - replicasets - statefulsets - deployments - daemonsets verbs: - get - list - update - patch - create - delete - apiGroups: - "batch" resources: - cronjobs verbs: - get - list - create - update - patch - watch - delete # For managing Jobs - apiGroups: - "batch" resources: - jobs verbs: - get - list - create - update - patch - watch - delete # For durable storage management. - apiGroups: - "" resources: - persistentvolumeclaims verbs: - get - list - create - update - patch - watch - delete # For managing KEDA objects for autoscaling. - apiGroups: - keda.sh resources: - scaledobjects - scaledjobs verbs: - get - list - create - update - patch - watch - delete # For managing PodDisruptionBudgets. - apiGroups: - policy resources: - poddisruptionbudgets verbs: - get - list - create - update - patch - watch # OPTIONAL: For showing thread dumps & profiling for batch backfills + some support use-cases. - apiGroups: - "" resources: - pods/exec verbs: - create - get # OPTIONAL: For support with debugging k8s rbac, grant read-only access to this namespaces' rbac configuration. - apiGroups: ["", "rbac.authorization.k8s.io"] resources: ["roles", "serviceaccounts", "rolebindings"] verbs: ["get", "list"] ``` Additionally, Chalk workload service accounts require the following role to be able to manage batch workloads: ``` apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: chalk-job-reader namespace: # replace with your actual namespace rules: - apiGroups: ["batch"] resources: ["jobs"] verbs: ["get", "list", "watch"] ``` If you use in-cluster docker image building powered by kaniko and Argo, Chalk requires this role on the namespace where the docker image building is happening: ``` apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: argo-workflows-role namespace: # replace with your actual namespace rules: - apiGroups: ["argoproj.io"] resources: ["workflows"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] ``` This role must be granted to the management service account that Chalk uses to manage the cluster, and to the workload service account that runs the docker image building. ### Logging In order to display logs in the Chalk web UI, Chalk requires permissions to be able to read Cloudwatch logs. If you use a separate logging service, you can set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs. Then, to view these logs we need the following permissions: ``` { "Statement": [ { "Action": [ "logs:TestMetricFilter", "logs:StartQuery", "logs:StartLiveTail", "logs:List*", "logs:Get*", "logs:FilterLogEvents", "logs:Describe*" ], "Effect": "Allow", "Resource": "arn:aws:logs:us-east-1::log-group::*", "Sid": "readlogs" }, { "Action": [ "logs:StopQuery", "logs:StopLiveTail" ], "Effect": "Allow", "Resource": "*", "Sid": "stopActions" } ], "Version": "2012-10-17" } ``` ### VPC CIDR Blocks Chalk will create the underlying VPC and subnets for the EKS cluster(s) for your Chalk deployment. Typically, Chalk will request a /16 address block and work within it. However, the number of addresses will depend on the size of your implementation. The Chalk team will work with you to determine the appropriate CIDR sections to use. Our default configuration is: ``` vpc_cidr_block = "10.130.0.0/16" vpc_subnets = [ { name = "primary" cidr_block = "10.130.0.0/20" az = "a" public_cidr_block = "10.130.16.0/20" }, { name = "secondary" cidr_block = "10.130.32.0/20" az = "b" public_cidr_block = "10.130.48.0/20" }, { name = "tertiary" cidr_block = "10.130.64.0/20" az = "c" public_cidr_block = "10.130.80.0/20" } ] ``` # Ambiguous Resolvers source: https://docs.chalk.ai/docs/ambiguous-resolvers ## Ambiguous Resolvers ### Resolving Ambiguous Resolver Errors In standard operation, Chalk assumes that each feature is un-ambiguously resolved by a single "in-scope" resolver. If this assumption is violated, Chalk's planner will raise an "ambiguous resolver" error that looks like -- ``` Chalk was unable to determine which resolvers to run for feature 'user.id'. There were multiple candidates .... ``` This error typically indicates that there is a mistake in resolver definitions, but it may arise in other situations as well. This document describes common causes of ambiguous resolver errors and how to resolve them. ### Multiple DataFrame-Returning Resolvers Often, ambiguous resolver errors arise when there are multiple resolvers that return dataframes that include the same feature. This occurs most often when multiple offline resolvers return the "primary feature" for a given feature class. For example: ``` @online def get_users() -> DataFrame[User.id, User.name]: ... @online def get_user_registration_dates() -> DataFrame[User.id, user.registration_date]: ... ``` This tends to arise when some of the features for a particular entity are resolved from one data source, and other features for the same entity are resolved from another data source (i.e. a different SQL table). This causes ambiguity because the query planner cannot determine which resolver returns "all" of the user rows. To resolve this ambiguity, you can define a single "primary" resolver that returns all the features for the entity in question by adding the total=True keyword argument -- ``` from chalk import online + @online(total=True) def get_users() -> DataFrame[User.id, User.name]: ... ``` or for SQL resolvers: ``` -- resolves: User -- source: snowflake -- type: offline + -- total: true select id, name from users ``` The total keyword argument indicates to the Chalk query planner that all primary keys for the entity are returned by this resolver. Other resolvers which return subsets of the entity's features can still be defined, but they will be combined with the "total" resolver to ensure that all rows are returned via a left-outer-join operation. ### Multiple Implementations of Resolvers Sometimes, overlaps in resolver definitions arise when multiple implementations of the same feature are defined. Common scenarios include: - if an A/B test is being run - a new version of a feature is being rolled out - offline query workloads with multiple different source parquet/csv datasets - switching between a production or sandbox version of a service call In these cases, you can resolve the ambiguity by adding tags to resolvers in order to break scoping ties -- ``` @online(tags="**api:mock**") def simulate_no_fraud(id: User.id) -> User.fraud_score: return 10 @online(tags="**api:sandbox**") def sandbox_score( name: User.name, email: User.email, ) -> User.fraud_score: return sandbox.fraud_score(name, email).score @online def real_score( name: User.name, email: User.email, ) -> Features[User.fraud_score, User.fraud_tags]: r = prod.fraud_score(name, email).score return User(fraud_score=r.score, fraud_tags=r.tags) ``` In this example, the simulate_no_fraud and sandbox_score resolvers can be selected by specifying the appropriate tag when making a feature request. The real_score resolver will be used when no tags are specified. For more information, see resolver tags. ### Online and Offline Resolvers with Overlapping Features Often, a single feature may have a natural "online" source (i.e. PostgreSQL, or a microservice API), and a natural "offline" source (i.e. Snowflake, or another bulk warehouse system). This case is handled automatically by Chalk's planner, which will prioritize @online resolvers over @offline resolvers when both are available. # AIMD Queue Server source: https://docs.chalk.ai/docs/aimd-queue-server ## Adaptive concurrency control for maximizing query throughput within a latency budget ### Overview Chalk's AIMD Queue Server is a pull-based query processing system that decouples request ingestion from query execution. Instead of each query server directly accepting gRPC traffic, a lightweight Rust frontend receives requests, enqueues them into a Redis Stream, and query server pods consume work at a rate governed by an AIMD concurrency controller. The goal is straightforward: maximize throughput given a particular latency budget. The AIMD controller continuously adjusts how many queries each consumer processes concurrently. When p99 latency is well below the SLO, concurrency increases. When p99 latency exceeds the SLO, concurrency drops. The result is a system that self-tunes to the capacity of the underlying hardware without manual concurrency tuning. ### Architecture Requests flow through three stages: the Rust frontend, a Redis Stream, and the C++ consumer running inside each query server pod. ``` ┌──────────────────────┐ │ gRPC Clients │ └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ AIMD Queue Server │ │ (Rust) │ │ │ │ Receives gRPC RPCs, │ │ encodes to protobuf, │ │ XADD → Redis Stream │ └──────────┬───────────┘ │ XADD ┌──────────▼───────────┐ │ Redis Stream │ │ (consumer group: │ │ "query_servers") │ └───┬──────┬──────┬────┘ XREADGROUP │ │ │ XREADGROUP ┌────────────┘ │ └────────────┐ │ │ │ ┌──────────▼──────┐ ┌─────────▼───────┐ ┌─────────▼───────┐ │ Query Server 0 │ │ Query Server 1 │ │ Query Server N │ │ (C++ consumer) │ │ (C++ consumer) │ │ (C++ consumer) │ │ │ │ │ │ │ │ AIMD controls │ │ AIMD controls │ │ AIMD controls │ │ concurrency │ │ concurrency │ │ concurrency │ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ SET │ SET │ SET └───────────┐ │ ┌────────────┘ │ │ │ ┌────────▼───────▼───────▼────────┐ │ Redis (results) │ │ pull_query_result:{query_id} │ └────────────────┬─────────────────┘ │ GET (poll) ┌────────▼─────────────┐ │ AIMD Queue Server │ │ polls for result, │ │ returns gRPC response│ └──────────────────────┘ ``` 1. Ingestion. The Rust AIMD Queue Server implements the full QueryService gRPC interface. When a request arrives (e.g. OnlineQuery, OnlineQueryBulk), the server serializes it to protobuf, generates a unique query_id, and publishes the message to a Redis Stream via XADD. The message includes the serialized request, the query method name, and forwarded metadata (environment, deployment, project, team IDs, and OpenTelemetry trace headers). 2. Distribution. Query server pods form a Redis consumer group. Each pod calls XREADGROUP to pull messages in batches. Redis guarantees that each message is delivered to exactly one consumer within the group, so work is automatically load-balanced across pods. If a consumer crashes, unacknowledged messages are reclaimed by other consumers after a configurable idle timeout. 3. Execution. When a query server pod receives a message, it first attempts to acquire an AIMD concurrency slot. If the slot is acquired, the query executes. On completion, the result (a status byte + serialized protobuf) is written to a Redis key at pull_query_result:{query_id} with a TTL. The Rust frontend polls this key and returns the response to the original gRPC caller. ### AIMD concurrency control Each query server pod runs an independent AIMD controller that governs how many queries it will process concurrently. The controller maintains a sliding window of recent query latencies and adjusts the concurrency window on a periodic evaluation interval. The algorithm: - Additive increase: If the observed p99 latency is below p99_headroom × latency_slo and there was demand pressure (i.e., a try_acquire call was rejected), the window increases by alpha. - Multiplicative decrease: If the observed p99 latency exceeds latency_slo, the window is multiplied by beta (typically 0.7). - Fast circuit breaker: If any single query takes longer than fast_decrease_multiplier × latency_slo, the window is immediately multiplied by beta, without waiting for the next evaluation interval. The demand pressure requirement prevents the window from growing when there is no backlog — concurrency only increases when the queue has more work than the current window can handle. ### AIMD parameters These parameters are configured on the gRPC query servers that consume from the queue: | Parameter | Default | Description | | -------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `latency_slo` | *(required)* | Target p99 latency. The controller keeps p99 below this value. | | `alpha` | `1.0` | Additive increase step. Higher values ramp concurrency faster. | | `beta` | `0.7` | Multiplicative decrease factor. Lower values back off more aggressively. | | `min_concurrency` | `1` | Floor on the concurrency window. The controller will never drop below this. | | `max_concurrency` | `64` | Ceiling on the concurrency window. | | `eval_interval` | `1000ms` | How frequently the controller evaluates p99 and adjusts the window. | | `window_size` | `100` | Number of latency samples in the sliding window used for p99 calculation. | | `warmup_queries` | `10` | Minimum completed queries before the controller will allow a decrease. Prevents over-reaction on cold starts. | | `p99_headroom` | `0.85` | The controller only increases concurrency when p99 < `p99_headroom × latency_slo`. This provides a safety margin against oscillation. | | `fast_decrease_multiplier` | `3.0` | If a single query exceeds this multiple of the SLO, the controller immediately decreases the window. Acts as a circuit breaker for anomalous queries. | ### Redis Streams consumer configuration The consumer group on each query server pod is configured with: | Parameter | Default | Description | | -------------------- | -------- | ------------------------------------------------------------------------------------------------ | | `stream_key` | — | The Redis Stream key to consume from (must match the queue server's `CHALK_AIMD_QS_STREAM_KEY`). | | `consumer_group` | — | Consumer group name. All query server pods in a resource group should share the same group. | | `consumer_name` | — | Unique consumer identity, typically `{pod}:{pid}`. | | `batch_size` | `10` | Number of messages fetched per `XREADGROUP` call. | | `block_timeout` | `1000ms` | How long `XREADGROUP` blocks when the stream is empty. | | `claim_timeout` | `60s` | Idle time after which a consumer's pending messages are reclaimed by another consumer. | | `max_delivery_count` | `5` | Maximum delivery attempts before a message is routed to the dead letter queue. | ### Failure modes ### Queries exceeding the latency budget When individual queries consistently take longer than latency_slo, the AIMD controller will continuously decrease the concurrency window. In the extreme case, the window drops to min_concurrency (default 1), meaning the pod processes queries one at a time. This is by design — the controller prioritizes latency over throughput. However, if queries are inherently slower than the configured SLO (e.g. the SLO is 100ms but the query itself takes 200ms at minimum), the system will operate at minimum concurrency permanently. In this situation you should either increase the latency_slo to a realistic target, or optimize the underlying query to fit within the budget. The fast circuit breaker provides an additional safeguard: if a single query takes longer than fast_decrease_multiplier × latency_slo (default 3x), the controller immediately reduces concurrency without waiting for the next evaluation interval. ### Queue overload When the rate of incoming queries exceeds the aggregate processing capacity of all consumer pods, the Redis Stream will grow. The Rust frontend polls for each query's result with a configurable timeout (CHALK_AIMD_QS_RESULT_TIMEOUT_MS, default 30 seconds). If the result is not written before this timeout, the frontend returns a gRPC DEADLINE_EXCEEDED error to the caller. When the consumer pods are at capacity and the AIMD controllers have all been driven down to their minimum concurrency windows, the consumers will nack messages they cannot accept, causing Redis to redeliver them. At this point, the system is saturated and callers will see elevated latency and timeout errors. To mitigate queue overload: - Scale the consumer pool. Add more query server replicas so the consumer group has more aggregate capacity. - Lower the batch size. Smaller batch_size values reduce per-pod queuing depth. - Set max_concurrency appropriately. This caps each pod's contribution. If pods are resource-constrained, a lower ceiling prevents them from accepting more work than they can handle. ### Consumer failures If a query server pod crashes or becomes unresponsive, its pending messages in the consumer group will go unacknowledged. After claim_timeout (default 60 seconds), other consumers in the group will reclaim these messages via XCLAIM. If a message exceeds max_delivery_count delivery attempts across all consumers, it is routed to a dead letter queue for inspection. ### Queue server configuration The Rust AIMD Queue Server is configured via configuration variables: | Variable | Default | Description | | --------------------------------- | ------------------------ | ---------------------------------------------------------------------------------- | | `CHALK_AIMD_QS_REDIS_URL` | `redis://localhost:6379` | Redis connection URL. Supports both standalone and cluster mode. | | `CHALK_AIMD_QS_STREAM_KEY` | `pull_queries` | Redis Stream key for enqueuing queries. | | `CHALK_AIMD_QS_RESULT_KEY_PREFIX` | `pull_query_result` | Prefix for result keys. Results are stored at `{prefix}:{query_id}`. | | `CHALK_AIMD_QS_LISTEN_ADDR` | `[::]:50052` | gRPC listen address. | | `CHALK_AIMD_QS_RESULT_TIMEOUT_MS` | `30000` | Maximum time (ms) to poll for a query result before returning `DEADLINE_EXCEEDED`. | ### Metrics The AIMD Queue Server and its consumers publish metrics to Datadog: Queue server (Rust): - aimd_queueserver.open_connections — gauge of in-flight requests, tagged by method. - aimd_queueserver.request_duration_ms — end-to-end latency distribution, tagged by method, status, and reason. - aimd_queueserver.requests_total — request count, tagged by method, status, and reason (ok, backpressure, enqueue_failed, timeout, result_error, decode_error). - aimd_queueserver.redis.used_memory_bytes / used_memory_rss_bytes / maxmemory_bytes / available_memory_bytes — Redis memory usage gauges polled from INFO memory. - aimd_queueserver.redis.cpu_sys_seconds_total / cpu_user_seconds_total — cumulative Redis CPU seconds polled from INFO cpu. - aimd_queueserver.redis.cpu_usage_ratio — Redis CPU seconds consumed per wall-clock second since the previous poll (~1.0 == one fully-used core). Consumer (C++ AIMD controller): - chalk.engine.pull_query.aimd.concurrency — current concurrency window size. - chalk.engine.pull_query.aimd.p99_latency — observed p99 latency from the sliding window. - chalk.engine.pull_query.in_flight — number of queries currently being processed. - chalk.engine.pull_query.aimd.adjustments — counter of window adjustments, tagged by direction (increase, decrease, fast_decrease). - chalk.engine.pull_query.completed — counter of completed queries, tagged by status (success, rejected, transient_error, permanent_error).