Aug 13: Context layer webinar
Chalk home page
  1. Infrastructure
  2. Redis Configuration Variables

Chalk uses Redis Lightning, a custom Rust-based Redis client, as the online feature store backend. This page documents all environment variables that control Redis connection behavior, performance tuning, caching, and storage strategies.

​
Connection

VariableTypeDefaultDescription
REDIS_URIstring (URL)—Primary Redis URI for general Redis connections, used for the online store and windowed streaming.
REDIS_LIGHTNING_CLUSTER_URIScomma-separated URLs—List of Redis URIs for connecting to a Redis Lightning cluster.
REDIS_IS_CLUSTEREDbooltrueWhether Redis is running in cluster mode. When true, clustered=true is effectively set on every URI. When false, clustering is off unless the URI explicitly includes clustered=true.
REDIS_LIGHTNING_READ_FROM_REPLICASbooltrueDistribute read operations to replica nodes. Maps to read_from_replicas() on the underlying redis-rs client builder.
CHALK_PULL_QUERY_REDIS_URLScomma-separated URLs—Redis URLs for the pull query service. Falls back to the default online store Redis when unset.
CHALK_PULL_QUERY_REDIS_CLUSTEREDbool—Whether the pull query Redis is a cluster. Falls back to REDIS_IS_CLUSTERED when unset.
USE_REDIS_LIGHTNINGboolfalseFeature flag to enable Redis Lightning as the online store backend.

​
Thread Pool and Parallelism

VariableTypeDefaultDescription
REDIS_LIGHTNING_NUM_IO_THREADSintCPU cores × 3Size of the thread pool for threads that block on Redis queries. Best set to 1–3× logical core count.
CHALK_ENGINE_REDIS_LIGHTNING_THREAD_MULTIPLIERint1Maximum thread multiplier. Increasing above 1 allows oversubscription of threads to Redis connections.
REDIS_LIGHTNING_MIN_IO_THREADSint—Minimum number of Redis connections created at pool initialization. Set higher to pre-warm the pool and reduce cold-start latency.
CHALK_REDIS_LIGHTNING_WORKER_THREADSintsystem parallelismNumber of worker threads in the shared Rayon thread pool for chunk-level work (lookups, stores, has-many indexes). Set in the Rust layer.

​
TCP Connection Tuning

These variables take effect only when CHALK_REDIS_USE_CUSTOM_TCP_SETTINGS is set to a truthy value.

VariableTypeDefaultDescription
CHALK_REDIS_USE_CUSTOM_TCP_SETTINGSboolfalseEnable custom TCP socket settings for Redis connections.
CHALK_REDIS_TCP_CONNECT_TIMEOUT_MSint (ms)redis-rs defaultTCP connection timeout in milliseconds.
CHALK_REDIS_TCP_KEEPALIVE_TIME_MSint (ms)system defaultIdle time before TCP keepalive probes begin (SO_KEEPALIVE).
CHALK_REDIS_TCP_KEEPALIVE_PROBE_INTERVAL_SECSint (s)system defaultInterval between consecutive keepalive probes.
CHALK_REDIS_TCP_KEEPALIVE_PROBE_COUNTintsystem defaultNumber of failed probes before the connection is considered dead.

​
Local Caching (LRU)

Redis Lightning supports a per-namespace in-process LRU cache that avoids round-trips for frequently accessed features.

VariableTypeDefaultDescription
REDIS_LIGHTNING_PER_NAMESPACE_CACHE_SIZEint6000Maximum number of entries in the LRU cache per namespace.
REDIS_LIGHTNING_CACHE_TTL_SECONDSint (s)3600Time-to-live for each cache entry.
REDIS_LIGHTNING_LRU_CACHED_NAMESPACEScomma-separated strings—Feature namespaces (snake_case) to LRU-cache. Only listed namespaces are cached.
CHALK_REDIS_METRICS_LRU_CACHE_CACHE_MISSESboolfalseWhen enabled, cache misses are recorded in the LRU. If a primary key is absent from Redis, future lookups for it skip the round-trip.
CHALK_ONLINE_STORE_LRU_EVICT_AFTER_EXPIRATIONbooltruePromptly evict expired LRU entries. Disabling allows unbounded growth but improves throughput under high concurrency.

​
Load and Store Strategy

VariableTypeDefaultDescription
REDIS_LIGHTNING_LOAD_STRATEGYenum—Controls how features are loaded from Redis. Values: PIPELINE_ONLY_REQUIRED, PIPELINE_ALL_IN_NAMESPACE_FOR_PKEY, SEPARATED_ONLY_REQUIRED, SEPARATED_ALL_IN_NAMESPACE_FOR_PKEY.
REDIS_LIGHTNING_STORE_STRATEGYenumCHECK_TS_NO_PIPELINEControls how data is written to Redis (timestamp validation and pipelining behavior).
REDIS_LIGHTNING_WARMUP_FQN_MAPPINGbooltruePre-load the shortened FQN→key mapping at initialization. When false, mappings are lazily computed per query.
REDIS_LIGHTNING_SUPPORTS_HAS_MANYboolfalseEnable has-many relationship query support in Redis Lightning.
REDIS_MAX_CONNECTIONSint32Maximum concurrent Redis connections for the result bus online writer.

​
Feature Lookup Optimization

VariableTypeDefaultDescription
CHALK_REDIS_LIGHTNING_ENABLE_AUTO_HGETALLboolfalseUse HGETALL for wide projections (400+ required fields) instead of HMGET, reducing per-field request overhead. Set in the Rust layer.
LIGHTNING_REDIS_MIN_ITEMS_SIZEint8000Target number of features per chunk. Chunk size = value / num_target_features, clamped to [500, 2000]. Set in the Rust layer.
REDIS_LIGHTNING_STORE_FEATURES_MAX_ROWS_PER_CHUNKint1000Maximum rows batched per chunk during feature store operations. Set in the Rust layer.
REDIS_DF_INSERT_CHUNK_SIZEint20000Maximum features per batch for dataframe-based Redis inserts (Polars pipeline).

​
Streaming and Window Buffers

VariableTypeDefaultDescription
STREAMING_USE_REDIS_WINDOW_BUFFERbooltrueUse Redis as the window buffer backend for streaming aggregations.

​
Time Series Aggregations

VariableTypeDefaultDescription
REDIS_LIGHTNING_AGGREGATION_KEY_PROTOCOL_VERSIONintinternal defaultOverride the protocol version used for time-series key building.
REDIS_LIGHTNING_AGGREGATION_DEBUG_KEY_NAMESbool (presence)falseUse long human-readable key names instead of short FQN-mapped names. Useful for debugging.
REDIS_LIGHTNING_AGGREGATION_USE_LOAD_RANGE_SCRIPTboolfalseUse a Redis Lua script for LOAD_RANGE operations in time-series aggregations.

​
Pipeline Validation

VariableTypeDefaultDescription
CHALK_REDIS_LIGHTNING_VALIDATE_PIPELINE_SHAPEbooltrueValidate Redis pipeline command shapes at request time. Disable with 0, false, no, or off. Set in the Rust layer.

​
Observability

VariableTypeDefaultDescription
CHALK_REDIS_METRICS_HIGH_METRICS_CARDINALITYboolfalseInclude high-cardinality tags (e.g., namespace:<name>) in StatsD metrics. Disabled by default to limit metric volume.
REDIS_DEBUG_STORAGE_BUCKETstring—GCS bucket path. When set, bulk Redis upload command strings are stored here for debugging.

​
Aggregation Command Logging

VariableTypeDefaultDescription
REDIS_LIGHTNING_AGG_COMMAND_LOG_PATHstring (path)—Base directory for aggregation command logs. When set, 100% of aggregation commands are logged to files under this path.
REDIS_LIGHTNING_AGG_COMMAND_LOG_FLUSH_INTERVAL_MSint (ms)1000Batch flush interval for the aggregation command logger.
REDIS_LIGHTNING_AGG_COMMAND_LOG_MAX_BATCH_SIZEint10000Maximum commands per batch before a forced flush.

​
Debug HTTP Server

Redis Lightning can expose a debug HTTP endpoint for inspecting cluster topology, slot maps, and pool statistics.

VariableTypeDefaultDescription
CHALK_REDIS_LIGHTNING_DEBUG_HTTP_PORTint (port)—Port for the debug HTTP server. Server is disabled when unset.
CHALK_REDIS_LIGHTNING_DEBUG_HTTP_HOSTstring127.0.0.1Bind address for the debug server. Override with caution—no authentication is provided.

​
Azure Workload Identity

These variables are required when auth_mode=azure_workload_identity is set as a query parameter in the Redis URL.

VariableTypeDefaultDescription
AZURE_TENANT_IDstring—Azure tenant ID for federated token requests.
AZURE_CLIENT_IDstring—Azure client ID for federated token requests.
AZURE_FEDERATED_TOKEN_FILEstring (path)—Path to the federated token file for AKS workload identity.
AZURE_AUTHORITY_HOSTstring (URL)https://login.microsoftonline.comAzure OAuth2 authority host. Override for sovereign clouds.
CHALK_REDIS_AZURE_TOKEN_REFRESH_SECSint (s)2700 (45 min)Interval for refreshing Azure access tokens.

​
Testing and Development

These variables are dangerous in production and exist only for CI and local development.

VariableTypeDefaultDescription
CHALK_TESTING_REDIS_LIGHTNING_ENABLE_CACHE_PREFIXboolfalsePrefix all keys with the active environment name. Breaks production by making existing data inaccessible.
CHALK_TESTING_REDIS_LIGHTNING_SKIP_EXPIRATIONboolfalseSkip writing TTL data. Required in CI where tests use fixed timestamps in the past. Must be false in production.
CHALK_TESTING_USE_RUST_FOR_ONLINE_STORE_WRITEbooltrueUse the Rust writer for Redis Lightning writes.
REDIS_LIGHTNING_ALLOW_FLUSHDB_IN_TESTSbool (presence)falseAllow test utilities to flush the Redis database.

​
Notes

  • CHALK_REDIS_REPLICA_CONNECTION_POLICY is not currently implemented. Replica read behavior is controlled by REDIS_LIGHTNING_READ_FROM_REPLICAS.
  • Variables prefixed with CHALK_REDIS_ or CHALK_REDIS_LIGHTNING_ are set in the Rust Redis Lightning library. Variables without a CHALK_ prefix are typically set in the Python engine configuration.
  • Boolean variables in the Rust layer accept 0, false, no, off (case-insensitive) as falsy; anything else is truthy.