Feature Engine
Build training sets for windowed features by merging materialized aggregation buckets and offline tiles.
Building a training set for a windowed aggregation feature reads every event inside each spine row’s window. The work grows with both the number of spine rows and the number of events per window.
For a feature defined with a materialized aggregation, Chalk can split historical events into fixed-width buckets, store a partial aggregate for each bucket as a tile, and compute each spine row’s value by merging the tiles that cover its window.
A training-set query for a materialized aggregation uses the planner options described in
Offline queries. The examples on this page use
the User.total_transaction_amount feature from
Materialized Windowed Aggregations, which has 1d, 7d, and
30d windows and one-day buckets.
Write tiles to offline storage that cover your spine’s input times and the windows before them.
Run chalk aggregate backfill with --store-offline, or set up a
scheduled backfill that targets
offline storage:
chalk aggregate backfill \
--feature user.total_transaction_amount \
--store-offline \
--lower-bound 2025-01-01T00:00:00Z \
--upper-bound 2026-04-14T00:00:00Z
Query with use_tile_store, use_materialized_offline_query, and recompute_features=True:
from datetime import datetime, timezone
from chalk.client import ChalkClient
client = ChalkClient()
ds = client.offline_query(
input={"user.id": [1, 2, 3]},
input_times=[datetime(2026, 4, 1, tzinfo=timezone.utc)] * 3,
output=[
"user.total_transaction_amount__604800__",
"user.total_transaction_amount__2592000__",
],
recompute_features=True,
planner_options={"use_tile_store": True, "use_materialized_offline_query": True},
)
The tiles must cover each row’s whole window, not only its input time. For example, tiles from
2026-03-30 to 2026-04-01 can answer the 1d window at an input time of 2026-04-01, but not the 7d
window. If the tiles do not cover a requested window, the query fails. For an aggregation with a
continuous buffer, the query aggregates the
trailing buffer span from raw events and the rest of the window from tiles. See
Reading tiles from offline storage.
With use_materialized_offline_query, with or without use_tile_store, each row is computed by
merging buckets. The two differ in where the buckets come from.
Consider merchant.count_transactions_last_7d on a historical spine where some merchants have
millions of transactions per day. With the default planner, each spine row for such a merchant
aggregates every transaction in the seven days before its timestamp. Both the number of spine rows
and the number of events per window grow with the merchant’s activity, so the work grows roughly
with their product.
With use_materialized_offline_query, Chalk aggregates each merchant’s events into buckets once per
query, and each row merges at most window / bucket_duration buckets, rounded up. Rows for the same
merchant whose bounds fall in the same buckets are computed once. With use_tile_store, Chalk reads
those buckets from tiles instead of aggregating the raw events.
With use_tile_store, a "14d" window added to a feature that already declares "7d" and "30d"
reads the tiles already written for those windows, as long as they cover its range, so it needs no
new backfill. A new aggregation needs its own backfill. See
Reading tiles from offline storage.
Tiles are aligned on fixed bucket boundaries (see bucket_duration in
MaterializationWindowConfig).
When an offline query requests a window that does not fall exactly on a bucket boundary,
Chalk must decide how to treat the partial bucket at the leading or trailing edge of the
window. With use_materialized_offline_query=True, Chalk aggregates whole buckets, from the
one that contains the window’s lower bound up to, but not including, the one that contains its
upper bound, which means:
window_interval requested at both ends.input_time falls in the same bucket.Shorter buckets bring each row’s effective window closer to the requested window, at the cost of more tiles to store and merge.
To match how the online store rounds instead, use
align_offline_chalk_window_with_materialization, described in
Offline queries.
This example uses a feature with a three-hour window and one-hour buckets. One user has eight transactions on 2026-10-20 (UTC), and each amount is a power of two, so every sum identifies exactly which transactions were included.
| Time (UTC) | 09:10 | 09:40 | 10:20 | 11:05 | 11:50 | 12:10 | 12:20 | 12:40 |
|---|---|---|---|---|---|---|---|---|
| Amount | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
from datetime import datetime, timezone
from chalk import DataFrame, Windowed, offline, windowed
from chalk.features import _, features
@features
class Transaction:
id: int
amount: int
user_id: "User.id"
timestamp: datetime
@features
class User:
id: int
transactions: DataFrame[Transaction]
spend: Windowed[int] = windowed(
"3h",
materialization={"bucket_duration": "1h"},
expression=_.transactions[
_.amount,
_.timestamp <= _.chalk_now,
_.timestamp > _.chalk_window,
].sum(),
default=0,
)
EVENTS = [
("09:10", 1), ("09:40", 2), ("10:20", 4), ("11:05", 8),
("11:50", 16), ("12:10", 32), ("12:20", 64), ("12:40", 128),
]
@offline
def get_transactions(
uid: User.id,
) -> DataFrame[Transaction.id, Transaction.user_id, Transaction.amount, Transaction.timestamp]:
rows = []
for i, (hhmm, amount) in enumerate(EVENTS, start=1):
hour, minute = map(int, hhmm.split(":"))
rows.append(
Transaction(
id=i,
user_id=uid,
amount=amount,
timestamp=datetime(2026, 10, 20, hour, minute, tzinfo=timezone.utc),
)
)
return DataFrame(rows)Query the feature at four times, once with the default planner and once with
use_materialized_offline_query:
from datetime import datetime, timezone
from chalk.client import ChalkClient
client = ChalkClient()
times = [
datetime(2026, 10, 20, 12, 34, tzinfo=timezone.utc),
datetime(2026, 10, 20, 12, 59, tzinfo=timezone.utc),
datetime(2026, 10, 20, 12, 0, tzinfo=timezone.utc),
datetime(2026, 10, 20, 12, 1, tzinfo=timezone.utc),
]
for planner_options in [None, {"use_materialized_offline_query": True}]:
ds = client.offline_query(
input={"user.id": [1] * len(times)},
input_times=times,
output=["user.spend__10800__"],
recompute_features=True,
planner_options=planner_options,
)
print(ds.get_data_as_pandas()["user.spend__10800__"].tolist())| Query time | Exact window | Exact sum | Buckets merged | Materialized sum |
|---|---|---|---|---|
| 12:34 | 09:34 to 12:34 | 126 | 09:00, 10:00, 11:00 | 31 |
| 12:59 | 09:59 to 12:59 | 252 | 09:00, 10:00, 11:00 | 31 |
| 12:00 | 09:00 to 12:00 | 31 | 09:00, 10:00, 11:00 | 31 |
| 12:01 | 09:01 to 12:01 | 31 | 09:00, 10:00, 11:00 | 31 |
At 12:34, the lower bound is 09:34 and the upper bound is 12:34. Aligned down to the hour, they are 09:00 and 12:00. Chalk merges the 09:00, 10:00, and 11:00 buckets. The 09:00 bucket includes the 09:10 transaction, which happened before the lower bound, and the 12:00 bucket is left out, so the 12:10 and 12:20 transactions are not counted even though they happened before the upper bound. The sum is 1 + 2 + 4 + 8 + 16 = 31. All four query times fall in the 12:00 hour, so they use the same buckets and get the same value.
A bound that is offset from the query time is shifted before it is aligned. This feature leaves out
the most recent hour by comparing against _.chalk_now - timedelta(hours=1). Add it to the User class above:
from datetime import timedelta
lagged_spend: Windowed[int] = windowed(
"3h",
materialization={"bucket_duration": "1h"},
expression=_.transactions[
_.amount,
_.timestamp < _.chalk_now - timedelta(hours=1),
_.timestamp >= _.chalk_window,
].sum(),
default=0,
)At 12:34, the bounds are 09:34 and 11:34, which align to 09:00 and 11:00. Chalk merges the 09:00 and 10:00 buckets, so the sum is 1 + 2 + 4 = 7. The exact window from 09:34 to 11:34 sums to 14. At 12:59, the bounds are 09:59 and 11:59, which align to the same buckets, so the materialized sum is again 7.
chalk aggregate backfill: the CLI command that
backfills aggregate tiles.