Online Store Capacity

​

Overview

The online store holds the latest computed feature values so queries can be served without recomputing them. It is a fixed-size cache: as you cache more features, for longer, across more entities, it fills.

This page explains what happens as the store fills, which parts of that behavior are specific to Chalk, and what you can do about it. The store runs in your own cloud account, so monitoring and alerting on it are yours to configure — the next section covers what to track.

​

Watching utilization

Two signals matter:

• Memory utilization — how full the store is, as a percentage of its limit.
• Evicted keys — how much data is being dropped to make room for new writes.

Your provider publishes both, and each one has its own monitoring and alerting stack. See the provider reference below for the metric names and monitoring guides for your platform.

We recommend setting an alert on utilization rather than checking it by hand. Provider guidance generally points at acting once sustained usage reaches roughly 75–80%, which leaves time to add capacity or trim what you cache before eviction becomes significant.

Watch the peak across nodes, not the average. In a clustered store, data is distributed across shards and they do not fill evenly. A single shard can be at its limit while the average across the cluster still looks comfortable — and as described in the next section, one shard filling is enough to matter.

To see what is consuming the space rather than how much is used, use Online Store Survey. The chalk.online_store.keys table reports key counts, memory usage, and remaining TTL broken down by feature namespace, which is the fastest way to find the features driving your footprint.

​

How eviction affects Chalk

When a Valkey or Redis store reaches its memory limit, the maxmemory-policy setting decides what gets removed to make room. The policies fall into two families:

• volatile-* — only keys that carry a TTL are eligible for eviction.
• allkeys-* — any key is eligible, whether or not it has a TTL.

Alongside your cached feature values, Chalk stores mapping metadata in the same instance. That metadata is what lets the engine interpret everything else in the store, and it is written without a TTL. Under a volatile-* policy it is never an eviction candidate, so it stays in place. Under an allkeys-* policy it is eligible like any other key, and Chalk depends on it being present to read the store.

This matters more than the share of memory it occupies. In a clustered store, the mapping keys are deliberately co-located so they resolve in a single round trip, which means they all live on one shard. One shard reaching its limit under an allkeys-* policy is enough to affect the mapping for the whole store — which is why the peak-across-nodes reading is the one to watch.

Chalk requires volatile-lru. For most customers this needs no action, because it is already the default everywhere:

• GCP Memorystore — default volatile-lru
• AWS ElastiCache — default volatile-lru
• Azure Managed Redis — default volatile-lru
• Azure Cache for Redis — default volatile-lru

Please keep the eviction policy set to volatile-lru. If you would like to explore a different policy, talk to us first and we will work through it with you.

​

Features cached indefinitely

The same rule applies to your own data. Under volatile-lru, only keys with a TTL can be evicted — so a feature cached with max_staleness="infinity" is never an eviction candidate. It stays resident whether or not anything is still reading it, and frees space only when it is overwritten or deleted.

@features
class User:
    id: int
    # never expires, never evicted — occupies memory until overwritten
    fraud_score: float = feature(max_staleness="infinity")

That is often exactly what you want. But if you have not looked recently, it is worth reviewing your pipelines for features written without a TTL that could reasonably carry one — those are the cheapest capacity you have. Online Store Survey will show you which namespaces are consuming the most memory and how much of their data carries a TTL.

See Caching indefinitely for the "infinity" setting, Feature Caching for how max_staleness works generally, and Garbage Collection for how expired values are reclaimed.

​

When the store is near full

Once the store is at its limit, every new write requires evicting something else. Two consequences follow.

Evicted features become cache misses. The next query for that feature recomputes it and reads from the underlying data source instead of serving from memory. Expect query latency to rise and load on your upstream sources to increase as the eviction rate climbs.

Eviction is driven by writes, not by time. If the working set no longer fits, data is replaced before the query that needed it arrives — features get computed, stored, and evicted without ever being read, so the work of populating the cache stops paying for itself.

There is also a floor on how much can be reclaimed. Because volatile-lru only evicts keys that carry a TTL, once those can no longer free space the store stops accepting new writes rather than evicting further. That boundary is what keeps the mapping metadata in place.

​

Adding capacity

Valkey and Redis stores scale two ways:

• Horizontally, by adding shards, which increases total memory and spreads writes across more primaries.
• Vertically, by moving to a larger node type.

Your provider documents both paths, and what to expect while the operation runs — see the provider reference.

Whenever modifying the cluster, consider the load your cluster is under and whether it will be significantly impacted by the modification operation.

​

Reducing what's cached

Lowering the resident set is the other lever, and it can be the cheaper one:

1. Survey first. Online Store Survey tells you which feature namespaces dominate, so you change the ones that matter.
2. Shorten max_staleness on high-volume features that do not need long retention.
3. Add a TTL to features currently cached indefinitely, where a bounded lifetime is acceptable.
4. Remove features from the online store that do not need to be served online at all.

These take effect as existing keys expire or are overwritten — a feature already written without a TTL keeps its current lifetime until the next write.

​

DynamoDB

If your online store is backed by DynamoDB, none of the eviction guidance above applies. DynamoDB is not memory-bound and has no maxmemory-policy; it does not drop data under memory pressure, and cached values are removed only by TTL expiry or explicit deletion. Capacity there is a throughput and storage question rather than an eviction one.

See DynamoDB deployment for capacity sizing, and Feature Caching for how expired values are cleaned up.

​

Provider reference

A note on reading these numbers: providers reserve part of an instance’s advertised memory for replication, fragmentation, and background saves, so the usable capacity is lower than the headline size. AWS documents this as reserved memory, and the percentage metrics on each platform already account for it — which is why a utilization percentage is a more reliable signal than raw bytes used.

Microsoft is retiring Azure Cache for Redis in favor of Azure Managed Redis. If your online store runs on Azure Cache for Redis, see the retirement FAQ for timelines and migration guidance.

​

See also

• Online Store Survey — find which feature namespaces are consuming memory.
• Feature Caching — how max_staleness controls what is cached and for how long.
• Online Store Setup — connecting and configuring an online store.
• Choosing an Online and Offline Store — store options per cloud.