# Redis Configuration Variables source: https://docs.chalk.ai/docs/redis-config-vars ## Environment variables for configuring Redis Lightning, the high-performance online feature store backend. Chalk uses Redis Lightning, a custom Rust-based Redis client, as the online feature store backend. This page documents all environment variables that control Redis connection behavior, performance tuning, caching, and storage strategies. ### Connection | Variable | Type | Default | Description | | ------------------------------------ | -------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `REDIS_URI` | string (URL) | — | Primary Redis URI for general Redis connections, used for the online store and windowed streaming. | | `REDIS_LIGHTNING_CLUSTER_URIS` | comma-separated URLs | — | List of Redis URIs for connecting to a Redis Lightning cluster. | | `REDIS_IS_CLUSTERED` | bool | `true` | Whether Redis is running in cluster mode. When `true`, `clustered=true` is effectively set on every URI. When `false`, clustering is off unless the URI explicitly includes `clustered=true`. | | `REDIS_LIGHTNING_READ_FROM_REPLICAS` | bool | `true` | Distribute read operations to replica nodes. Maps to `read_from_replicas()` on the underlying redis-rs client builder. | | `CHALK_PULL_QUERY_REDIS_URLS` | comma-separated URLs | — | Redis URLs for the pull query service. Falls back to the default online store Redis when unset. | | `CHALK_PULL_QUERY_REDIS_CLUSTERED` | bool | — | Whether the pull query Redis is a cluster. Falls back to `REDIS_IS_CLUSTERED` when unset. | | `USE_REDIS_LIGHTNING` | bool | `false` | Feature flag to enable Redis Lightning as the online store backend. | ### Thread Pool and Parallelism | Variable | Type | Default | Description | | ------------------------------------------------ | ---- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | `REDIS_LIGHTNING_NUM_IO_THREADS` | int | CPU cores × 3 | Size of the thread pool for threads that block on Redis queries. Best set to 1–3× logical core count. | | `CHALK_ENGINE_REDIS_LIGHTNING_THREAD_MULTIPLIER` | int | `1` | Maximum thread multiplier. Increasing above 1 allows oversubscription of threads to Redis connections. | | `REDIS_LIGHTNING_MIN_IO_THREADS` | int | — | Minimum number of Redis connections created at pool initialization. Set higher to pre-warm the pool and reduce cold-start latency. | | `CHALK_REDIS_LIGHTNING_WORKER_THREADS` | int | system parallelism | Number of worker threads in the shared Rayon thread pool for chunk-level work (lookups, stores, has-many indexes). Set in the Rust layer. | ### TCP Connection Tuning These variables take effect only when CHALK_REDIS_USE_CUSTOM_TCP_SETTINGS is set to a truthy value. | Variable | Type | Default | Description | | ----------------------------------------------- | -------- | ---------------- | ----------------------------------------------------------------- | | `CHALK_REDIS_USE_CUSTOM_TCP_SETTINGS` | bool | `false` | Enable custom TCP socket settings for Redis connections. | | `CHALK_REDIS_TCP_CONNECT_TIMEOUT_MS` | int (ms) | redis-rs default | TCP connection timeout in milliseconds. | | `CHALK_REDIS_TCP_KEEPALIVE_TIME_MS` | int (ms) | system default | Idle time before TCP keepalive probes begin (SO_KEEPALIVE). | | `CHALK_REDIS_TCP_KEEPALIVE_PROBE_INTERVAL_SECS` | int (s) | system default | Interval between consecutive keepalive probes. | | `CHALK_REDIS_TCP_KEEPALIVE_PROBE_COUNT` | int | system default | Number of failed probes before the connection is considered dead. | ### Local Caching (LRU) Redis Lightning supports a per-namespace in-process LRU cache that avoids round-trips for frequently accessed features. | Variable | Type | Default | Description | | ----------------------------------------------- | ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `REDIS_LIGHTNING_PER_NAMESPACE_CACHE_SIZE` | int | `6000` | Maximum number of entries in the LRU cache per namespace. | | `REDIS_LIGHTNING_CACHE_TTL_SECONDS` | int (s) | `3600` | Time-to-live for each cache entry. | | `REDIS_LIGHTNING_LRU_CACHED_NAMESPACES` | comma-separated strings | — | Feature namespaces (snake_case) to LRU-cache. Only listed namespaces are cached. | | `CHALK_REDIS_METRICS_LRU_CACHE_CACHE_MISSES` | bool | `false` | When enabled, cache misses are recorded in the LRU. If a primary key is absent from Redis, future lookups for it skip the round-trip. | | `CHALK_ONLINE_STORE_LRU_EVICT_AFTER_EXPIRATION` | bool | `true` | Promptly evict expired LRU entries. Disabling allows unbounded growth but improves throughput under high concurrency. | ### Load and Store Strategy | Variable | Type | Default | Description | | ------------------------------------ | ---- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `REDIS_LIGHTNING_LOAD_STRATEGY` | enum | — | Controls how features are loaded from Redis. Values: `PIPELINE_ONLY_REQUIRED`, `PIPELINE_ALL_IN_NAMESPACE_FOR_PKEY`, `SEPARATED_ONLY_REQUIRED`, `SEPARATED_ALL_IN_NAMESPACE_FOR_PKEY`. | | `REDIS_LIGHTNING_STORE_STRATEGY` | enum | `CHECK_TS_NO_PIPELINE` | Controls how data is written to Redis (timestamp validation and pipelining behavior). | | `REDIS_LIGHTNING_WARMUP_FQN_MAPPING` | bool | `true` | Pre-load the shortened FQN→key mapping at initialization. When `false`, mappings are lazily computed per query. | | `REDIS_LIGHTNING_SUPPORTS_HAS_MANY` | bool | `false` | Enable has-many relationship query support in Redis Lightning. | | `REDIS_MAX_CONNECTIONS` | int | `32` | Maximum concurrent Redis connections for the result bus online writer. | ### Feature Lookup Optimization | Variable | Type | Default | Description | | --------------------------------------------------- | ---- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `CHALK_REDIS_LIGHTNING_ENABLE_AUTO_HGETALL` | bool | `false` | Use `HGETALL` for wide projections (400+ required fields) instead of `HMGET`, reducing per-field request overhead. Set in the Rust layer. | | `LIGHTNING_REDIS_MIN_ITEMS_SIZE` | int | `8000` | Target number of features per chunk. Chunk size = value / num_target_features, clamped to [500, 2000]. Set in the Rust layer. | | `REDIS_LIGHTNING_STORE_FEATURES_MAX_ROWS_PER_CHUNK` | int | `1000` | Maximum rows batched per chunk during feature store operations. Set in the Rust layer. | | `REDIS_DF_INSERT_CHUNK_SIZE` | int | `20000` | Maximum features per batch for dataframe-based Redis inserts (Polars pipeline). | ### Streaming and Window Buffers | Variable | Type | Default | Description | | ----------------------------------- | ---- | ------- | ------------------------------------------------------------------ | | `STREAMING_USE_REDIS_WINDOW_BUFFER` | bool | `true` | Use Redis as the window buffer backend for streaming aggregations. | ### Time Series Aggregations | Variable | Type | Default | Description | | --------------------------------------------------- | --------------- | ---------------- | ------------------------------------------------------------------------------------------ | | `REDIS_LIGHTNING_AGGREGATION_KEY_PROTOCOL_VERSION` | int | internal default | Override the protocol version used for time-series key building. | | `REDIS_LIGHTNING_AGGREGATION_DEBUG_KEY_NAMES` | bool (presence) | `false` | Use long human-readable key names instead of short FQN-mapped names. Useful for debugging. | | `REDIS_LIGHTNING_AGGREGATION_USE_LOAD_RANGE_SCRIPT` | bool | `false` | Use a Redis Lua script for `LOAD_RANGE` operations in time-series aggregations. | ### Pipeline Validation | Variable | Type | Default | Description | | ----------------------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------- | | `CHALK_REDIS_LIGHTNING_VALIDATE_PIPELINE_SHAPE` | bool | `true` | Validate Redis pipeline command shapes at request time. Disable with `0`, `false`, `no`, or `off`. Set in the Rust layer. | ### Observability | Variable | Type | Default | Description | | ---------------------------------------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------- | | `CHALK_REDIS_METRICS_HIGH_METRICS_CARDINALITY` | bool | `false` | Include high-cardinality tags (e.g., `namespace:`) in StatsD metrics. Disabled by default to limit metric volume. | | `REDIS_DEBUG_STORAGE_BUCKET` | string | — | GCS bucket path. When set, bulk Redis upload command strings are stored here for debugging. | ### Aggregation Command Logging | Variable | Type | Default | Description | | --------------------------------------------------- | ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `REDIS_LIGHTNING_AGG_COMMAND_LOG_PATH` | string (path) | — | Base directory for aggregation command logs. When set, 100% of aggregation commands are logged to files under this path. | | `REDIS_LIGHTNING_AGG_COMMAND_LOG_FLUSH_INTERVAL_MS` | int (ms) | `1000` | Batch flush interval for the aggregation command logger. | | `REDIS_LIGHTNING_AGG_COMMAND_LOG_MAX_BATCH_SIZE` | int | `10000` | Maximum commands per batch before a forced flush. | ### Debug HTTP Server Redis Lightning can expose a debug HTTP endpoint for inspecting cluster topology, slot maps, and pool statistics. | Variable | Type | Default | Description | | --------------------------------------- | ---------- | ----------- | --------------------------------------------------------------------------------------- | | `CHALK_REDIS_LIGHTNING_DEBUG_HTTP_PORT` | int (port) | — | Port for the debug HTTP server. Server is disabled when unset. | | `CHALK_REDIS_LIGHTNING_DEBUG_HTTP_HOST` | string | `127.0.0.1` | Bind address for the debug server. Override with caution—no authentication is provided. | ### Azure Workload Identity These variables are required when auth_mode=azure_workload_identity is set as a query parameter in the Redis URL. | Variable | Type | Default | Description | | -------------------------------------- | ------------- | ----------------------------------- | ----------------------------------------------------------- | | `AZURE_TENANT_ID` | string | — | Azure tenant ID for federated token requests. | | `AZURE_CLIENT_ID` | string | — | Azure client ID for federated token requests. | | `AZURE_FEDERATED_TOKEN_FILE` | string (path) | — | Path to the federated token file for AKS workload identity. | | `AZURE_AUTHORITY_HOST` | string (URL) | `https://login.microsoftonline.com` | Azure OAuth2 authority host. Override for sovereign clouds. | | `CHALK_REDIS_AZURE_TOKEN_REFRESH_SECS` | int (s) | `2700` (45 min) | Interval for refreshing Azure access tokens. | ### Testing and Development These variables are dangerous in production and exist only for CI and local development. | Variable | Type | Default | Description | | --------------------------------------------------- | --------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- | | `CHALK_TESTING_REDIS_LIGHTNING_ENABLE_CACHE_PREFIX` | bool | `false` | Prefix all keys with the active environment name. **Breaks production** by making existing data inaccessible. | | `CHALK_TESTING_REDIS_LIGHTNING_SKIP_EXPIRATION` | bool | `false` | Skip writing TTL data. Required in CI where tests use fixed timestamps in the past. **Must be `false` in production.** | | `CHALK_TESTING_USE_RUST_FOR_ONLINE_STORE_WRITE` | bool | `true` | Use the Rust writer for Redis Lightning writes. | | `REDIS_LIGHTNING_ALLOW_FLUSHDB_IN_TESTS` | bool (presence) | `false` | Allow test utilities to flush the Redis database. | ### Notes - CHALK_REDIS_REPLICA_CONNECTION_POLICY is not currently implemented. Replica read behavior is controlled by REDIS_LIGHTNING_READ_FROM_REPLICAS. - Variables prefixed with CHALK_REDIS_ or CHALK_REDIS_LIGHTNING_ are set in the Rust Redis Lightning library. Variables without a CHALK_ prefix are typically set in the Python engine configuration. - Boolean variables in the Rust layer accept 0, false, no, off (case-insensitive) as falsy; anything else is truthy.