Deployment
Make rapid changes and explore features with branches.
A branch deployment lets you try changes to features and resolvers without changing your mainline
deployment, the deployment that chalk apply creates without the --branch flag.
When working in a branch, you can:
recompute datasets to see the effects of changesA branch does not persist results to the online or offline store, and some settings always come from the mainline deployment. See Branch limitations.
To create a branch, run chalk apply --branch <branch_name>. Chalk creates a new branch in the environment that you can interact with. You can then make queries against your branch with chalk query --branch.
If you’re using the Python API, you can set the branch when creating your client, and all subsequent commands will execute against your branch.
The environment needs a mainline deployment before you can create a branch. If you have not deployed
to the environment yet, run chalk apply without --branch first.
You can ask the Chalk CLI to continuously deploy changes to your branch using the watch flag.
Chalk will automatically keep the branch up to date as you make changes locally.
A branch deployment can add or change:
Code on a branch can refer by name to data sources and Chalk Catalog entries that already exist. Everything else comes from the active mainline deployment, including:
To change any of these, run chalk apply without --branch.
A branch deployment cannot:
max_staleness,Chalk still records each query run against a branch, tagged with the branch name.
By default, online queries against a branch read values from the mainline deployment’s online store.
To give branches their own online store, go to Infrastructure > Resource Configuration > Branch Server in the dashboard and turn on Deploy Branch Online Store. The branch online store is separate from the mainline deployment’s online store. With it turned on:
upload_features are
stored for that branch and returned by later queries against it,Each chalk apply --branch <branch_name> replaces the code on that branch. To keep one version
running while you try another, deploy the second version to a new branch name. Calling
ChalkClient(branch=True) uses the name of your current git branch, with any / replaced by -,
which gives each git branch its own Chalk branch.
Run chalk apply without --branch when:
Deploying to mainline does not change your existing branches. Each branch keeps the code you deployed to it, and you do not need to deploy it again. See After a mainline deployment.
Once you’ve deployed a branch, you can iteratively edit features and resolvers and see the effects of these updates in a dataset.
Create a Chalk client and set the branch parameter equal to your branch. Then, any time you execute a cell that contains a class annotated with @features it will automatically be updated in your branch.
Similarly, executing cells that contain a function annotated with @online or @offline will automatically deploy the resolver to your branch.
@features
class NewFeatures:
id: int
name: str
greeting: str
@online
def new_resolver(name: NewFeatures.name) -> NewFeatures.greeting:
return f"Hello {name}!"As you adjust features and resolvers, you can iteratively see how your changes affect your feature values with the Dataset.recompute method. Just pass any features you want to be re-computed as arguments to the features parameter, and Chalk will generate a new Dataset revision using the latest definitions of features and resolvers.
When used in conjunction with the ability to adjust your features and resolvers in the notebook, this tool allows developers and data scientists to rapidly experiment and productionize their work.
In the following example we add an oversize feature to an existing dataset of loans.
We start out with a dataset. We may have produced this dataset manually, by calling an external API, or by executing offline_query.
dataset
shape: (786, 3)
┌───────────────┬──────────────┬─────────────────────────┐
│ loan.amount ┆ loan.id ┆ loan.event_time │
│ --- ┆ --- ┆ --- │
│ f64 ┆ str ┆ datetime[μs, UTC] │
╞═══════════════╪══════════════╪═════════════════════════╡
│ 165435.647396 ┆ l_G3Mc6bi9y4 ┆ 2023-04-08 10:03:09 UTC │
│ 405006.796909 ┆ l_O9OK3us7t2 ┆ 2022-02-07 20:46:08 UTC │
│ 680377.427817 ┆ l_L0Gg0Bd1v3 ┆ 2023-02-16 16:48:29 UTC │
│ 562678.545344 ┆ l_D1Rb5Jq4U5 ┆ 2022-06-21 06:15:19 UTC │
│ … ┆ … ┆ … │
│ 750583.279013 ┆ l_W4ZY9OK5N7 ┆ 2021-12-16 21:40:27 UTC │
│ 71698.15609 ┆ l_H9tG2yJ5B6 ┆ 2023-01-04 11:33:54 UTC │
│ 488697.890372 ┆ l_L4fd4xu4w2 ┆ 2022-08-22 03:55:16 UTC │
│ 769665.198436 ┆ l_k4Dq8bl7b3 ┆ 2022-10-31 07:15:52 UTC │
└───────────────┴──────────────┴─────────────────────────┘In our Jupyter notebook, we can execute this in a cell to set up our notebook to point to our branch, then update our branch with the new feature and resolver.
client = ChalkClient(branch="<our_branch>")
# We want to introduce an `oversize` detection feature
@features
class Loan:
id: str
event_time: datetime
status: str
amount: float
oversize: bool
# Detect any loans bigger than $250,000
@online
def is_oversize(amt: Loan.amount) -> Loan.oversize:
return amt > 250000Finally, we can recompute our dataset, telling it to calculate Loan.oversize, to get a dataset back with our new feature values.
# Recomputing the dataset will add the oversize column to our result
dataset.recompute(features=[Loan.oversize])
shape: (786, 4)
┌───────────────┬───────────────┬──────────────┬─────────────────────────┐
│ loan.oversize ┆ loan.amount ┆ loan.id ┆ loan.event_time │
│ --- ┆ --- ┆ --- ┆ --- │
│ bool ┆ f64 ┆ str ┆ datetime[μs, UTC] │
╞═══════════════╪═══════════════╪══════════════╪═════════════════════════╡
│ true ┆ 165435.647396 ┆ l_G3Mc6bi9y4 ┆ 2023-04-08 10:03:09 UTC │
│ true ┆ 405006.796909 ┆ l_O9OK3us7t2 ┆ 2022-02-07 20:46:08 UTC │
│ true ┆ 680377.427817 ┆ l_L0Gg0Bd1v3 ┆ 2023-02-16 16:48:29 UTC │
│ true ┆ 562678.545344 ┆ l_D1Rb5Jq4U5 ┆ 2022-06-21 06:15:19 UTC │
│ … ┆ … ┆ … ┆ … │
│ true ┆ 750583.279013 ┆ l_W4ZY9OK5N7 ┆ 2021-12-16 21:40:27 UTC │
│ true ┆ 71698.15609 ┆ l_H9tG2yJ5B6 ┆ 2023-01-04 11:33:54 UTC │
│ true ┆ 488697.890372 ┆ l_L4fd4xu4w2 ┆ 2022-08-22 03:55:16 UTC │
│ true ┆ 769665.198436 ┆ l_k4Dq8bl7b3 ┆ 2022-10-31 07:15:52 UTC │
└───────────────┴───────────────┴──────────────┴─────────────────────────┘All branches in an environment run on one server, the branch server. The branch server is separate from the servers that handle queries against your mainline deployment, and adding branches does not add servers. See Billing for how the branch server is billed.
The branch server does not start when a query arrives. These tools start the branch server if it is not running, and wait for it to be ready before they deploy or query:
chalk apply --branchchalk query --branchChalkClient in chalkpyQueries sent over gRPC, for example with ChalkGRPCClient, do not start the branch server. Before
you send gRPC queries to a branch, start the branch server with
ChalkGRPCClient.await_branch_server_start
or one of the tools above.
You can also start and stop the branch server from the Branches page of your environment in the dashboard, with Start Branch Server and Stop Branch Server.
The branch server shuts down after a period with no requests. The default period is 1 hour. To change it, go to Infrastructure > Resource Configuration > Branch Server in the dashboard and set Auto Shutdown Period to one of these values:
Queries sent over gRPC do not count as activity, so the branch server can shut down while you are
sending gRPC queries to it. If you query a branch only over gRPC, set a longer Auto Shutdown
Period, or call await_branch_server_start before each set of queries.
Each mainline deployment restarts the branch server. Your branches keep the code you deployed to them, and you do not need to deploy them again. Each branch reloads on its first query after the restart, so that query can take around 30 seconds. Later queries return at their usual speed.