Chalk supports migrating your online feature store from Cloud Memorystore for Redis to Cloud Memorystore for Valkey (clustered). This is an infrastructure upgrade that improves scalability, performance, and reliability for your feature serving infrastructure.
Benefits of Valkey cluster mode:
- Better horizontal scaling for high-throughput workloads
- Improved failover and replication
- Better performance for large feature vectors and concurrent reads
- Google Cloud is standardizing on Valkey for new deployments
What changes:
- Your online feature store moves from a single-node Redis instance to a clustered Valkey instance
- Connection credentials and endpoints will change; Chalk updates the connection strings and patch deploys your Chalk Deployment to pick up the new credentials automatically
- The migration can be done with minimal downtime (minutes of serving unavailability) or full data fidelity (hours of online store downtime), depending on your requirements
There are two migration scenarios. Scenario A is recommended for most customers. Choose based on your operational requirements:
| Consideration | Scenario A: Live Migration | Scenario B: Stop-the-World |
|---|
| Downtime for feature serving | <5 minutes (standard Chalk Apply cut-over) | 2–4 hours (no new feature data available) |
| Time until full data availability | 1–4 hours (cut-over + backfill pipeline) | 2–4 hours + 15–60 min queue catch-up |
| Data completeness during downtime | Features serve immediately after cut-over, but without backfilled data (1–4 hr backfill window) | Full data available after queue catch-up (15–60 min) |
| Backfill requirement | Optional — depends on use case; normal business processes may re-populate data as needed | NO — no backfill needed |
| Operational effort | Medium: coordinate backfill, monitor backfill pipeline | High: stop message processing, migrate instance to cluster, resume message processing |
| Best for | Most customers; online store is a cache with offline source of truth available | Customers without offline source of truth or unable to run backfill |
| Risk level | Lower: quicker cut-over, easier rollback | Higher: longer migration window, harder to roll back |
Use Scenario A if the following apply:
- You have an offline feature store (data warehouse, batch system) or other source of truth, OR normal business processes will re-populate feature data as needed
- You want to minimize downtime
- You can tolerate a brief gap in historical feature data (1–4 hours to backfill, if needed)
Use Scenario B if the following apply:
- Your online store is your primary or only source of truth
- You cannot run a backfill or the backfill is not feasible
- You can tolerate 2–4 hours of online feature serving unavailability
- You want zero data loss
Before migration day, Chalk will coordinate with you on the details below.
- Pull Redis details from records — Chalk identifies your Redis instance and current configuration
- Suggest Valkey cluster configuration — Based on your data size and throughput, Chalk recommends shard count and replica count for your approval
- Schedule migration window — Chalk works with you to find a suitable date and time
- Choose your migration scenario — Scenario A (minimal downtime, optional backfill) or Scenario B (zero data loss, extended downtime)
- Designate a migration contact — A person or team available during the migration window to acknowledge the cut-over and respond to any issues
For the migration to proceed, Chalk needs specific permissions in your GCP project.
Your GCP admin should verify that the Chalk management service account has these roles on your GCP project:
| Permission | Purpose |
|---|
Memorystore Admin | Create Valkey instances, manage Redis/Valkey clusters |
Compute Network Admin | Access your VPC network where Redis/Valkey run |
Storage Admin | Create/manage GCS bucket, manage permissions for Redis to export snapshots |
IAM Security Admin | Grant necessary permissions to the Redis service account for bucket access |
Secret Manager Admin | Create and update connection strings after migration |
Note: The Storage Admin and IAM Security Admin roles are needed because during the export, your Redis instance’s managed service account (automatically created by Google Cloud) must have write access to the GCS bucket. Chalk configures this automatically, but the management service account needs these roles to do so.
Ensure these APIs are enabled in your GCP project. You can check this in the GCP Console under APIs & Services > Enabled APIs:
- Cloud Memorystore for Redis API (
redis.googleapis.com) - Cloud Memorystore for Valkey API (
memorystore.googleapis.com) - Google Cloud Storage API (
storage-api.googleapis.com) - Secret Manager API (
secretmanager.googleapis.com)
If any are disabled, your GCP admin can enable them in the Enabled APIs section.
Chalk creates a temporary Google Cloud Storage (GCS) bucket to hold the Redis snapshot during migration:
- GCS bucket creation — In the same region as your Redis instance
- Automatic cleanup — Snapshots are automatically deleted after 7 days
- No ongoing cost — Storage is only used during migration; standard GCS pricing applies
If permissions are not set up, the migration cannot proceed. Chalk attempts to verify these during pre-migration checks. If any are missing:
- Chalk provides a specific list of what needs to be added
- Your GCP admin can grant permissions using the GCP Console or
gcloud CLI - Chalk re-verifies before proceeding
This typically takes 15–30 minutes to set up.
The migration process has a pre-migration verification phase followed by the main steps, which vary by scenario.
Before migration day, Chalk engineers will:
- Verify the current Redis instance is healthy and accessible
- Check available disk space in GCP for the RDB snapshot
- Confirm network connectivity to GCP resources
- Validate that all necessary GCP APIs and permissions are enabled
- Brief you on the expected timeline and what to watch for
| Phase | Duration | What happens | Your features |
|---|
| 1. Snapshot | 15–60 min | RDB export of Redis to GCP storage | Still serving normally |
| 2. Create Valkey | 30–60 min | New Valkey cluster initialized from snapshot | Still serving from Redis |
| 3. Cut-over | <5 min | Chalk Apply patches deployment with new Valkey connection string | Typically zero downtime (standard Chalk Apply) |
| 4. Backfill | 10 min–4 hours (if needed) | Optional: You run a batch job to recover features written during snapshot phase | Serving from Valkey (data populates via backfill or normal processes) |
| 5. Complete | Ongoing | Migration is complete; feature serving returns to normal | Serving from Valkey (fully operational) |
Your responsibilities during Scenario A:
- Acknowledge the cut-over window
- Have the backfill batch job ready and queued
- Start the backfill job immediately after cut-over (Chalk confirms when Valkey is ready and provides the start/end timestamps to cover)
- Monitor backfill pipeline for errors
| Phase | Duration | What happens | Your features |
|---|
| 1. Stop pipelines | 5 min | Disable feature pipelines writing to online store | No new features computed |
| 2. Snapshot + Create | 2–4 hours | RDB export and Valkey cluster initialization | No new feature data available |
| 3. Cut-over | <5 min | Chalk Apply patches deployment with new Valkey connection string | Connection switched to Valkey (zero downtime) |
| 4. Start pipelines | 5 min | Feature pipelines re-enabled, writing to Valkey | Pipelines process queued messages |
| 5. Queue catch-up | 10 min–4 hours | Queued input messages are processed and features computed | Serving catches up to real-time |
| 6. Complete | Ongoing | Migration complete; feature serving returns to normal | Serving normally |
Your responsibilities during Scenario B:
- Provide notice to end users (if applicable) that features will be unavailable
- Stop feature computation pipelines (or Chalk can do this for you)
- Wait for migration to complete
- Start pipelines or wait for Chalk to re-enable them
- Monitor pipeline output for anomalies
This is the recommended approach for most customers.
- Schedule the cut-over window with Chalk (date + time, 5–15 min duration)
- Prepare the backfill batch job with your data team
- This job re-fetches features from your offline source (data warehouse, etc.) for the time window during RDB export
- Chalk provides the exact time range needed
- Chalk engineers execute the migration:
- Export Redis RDB to GCP storage (you continue serving normally)
- Create the new Valkey cluster from the snapshot
- Update connection strings and re-point application to Valkey (brief cut-over)
- Provide you with the start and end timestamps of the migration window to cover
- You run the backfill job immediately after cut-over
- This populates the Valkey cluster with features that were written to Redis during snapshot export
- Backfill typically takes 1–4 hours depending on feature volume
- Verify — Compare backfilled data against your offline store to confirm completeness
The cut-over uses Chalk’s standard zero-downtime Apply process:
- Connection strings are updated to point to Valkey
- Deployment patches roll out with new credentials
- Existing connections gracefully drain and reconnect to Valkey
- No explicit downtime expected; feature serving continues uninterrupted
Best case: Zero downtime (standard for Chalk Apply)
Worst case: <5 minutes if any connection errors occur (rare)
- Feature serving resumes immediately (from Valkey, but without backfilled data)
- Backfill job runs in the background (1–4 hours)
- While backfill is running:
- Older features (written before the migration window) are in Valkey
- Recent features (written during RDB export) are being backfilled
- New features (written during backfill) are written directly to Valkey
- There may be a brief duplicate-key window where the same feature exists in both backfill data and live data
Some implementations don’t require explicit backfill, as normal business processes will re-populate the data as needed. Chalk will help you decide if backfills are necessary for your use case.
If backfill is required, Chalk provides:
- Start time — When RDB export began (snapshot captures data up to this point)
- End time — When Valkey cluster was ready
- Feature definitions — Which features need to be backfilled
- Assistance — Chalk can provide guidance on backfill query construction if needed, but you should build and own the query
Your responsibility (if backfill is needed):
- Build a backfill query that re-fetches features from your offline store for the provided time window
- This ensures consistency with your business logic and data requirements
- Chalk can provide assistance with query construction if needed
- Connect to your offline store (data warehouse, batch system, etc.)
- Run your query or re-compute for the time window
- Write results back to Valkey with correct timestamps
Use this approach if you cannot run a backfill or have no offline source of truth.
- Schedule the migration window with Chalk (date + time, 2–4 hour duration)
- Notify stakeholders that online feature serving will be unavailable
- Chalk engineers execute the migration:
- Stop all feature computation pipelines (or ask you to do so)
- Export Redis RDB to GCP storage
- Create the new Valkey cluster from the snapshot
- Update connection strings and re-point application to Valkey
- Restart feature computation pipelines
- Monitor — Watch pipeline health and feature consistency after restart
- Verify — Confirm all features are available and match pre-migration expectations
- Feature serving is fully unavailable during the migration window (instead of brief 5–15 min cut-over)
- No backfill job is needed (Valkey starts with the complete snapshot)
- Pipelines resume after Valkey is ready (instead of running backfill in parallel)
- All data written before the migration start is guaranteed in Valkey (no gaps)
- No new feature data: 2–4 hours while migration occurs
- Queued messages: Input events during the migration window are queued and processed after pipelines restart (15–60 min catch-up)
- Batch/offline pipelines: Also paused during the 2–4 hour window
- Downstream applications: May see feature requests fail or return stale data until catch-up completes
- Recovery: After queue processing, feature serving returns to normal with all queued data processed
For most customers: No code changes are required. Chalk handles connection string updates internally.
Most customers access the online feature store through Chalk’s query engines, not directly. In this case, Chalk updates the connection strings internally, and your applications continue to work without any code modifications. Feature serving functions continue seamlessly without modification.
For customers who access the online store directly: Chalk will direct you to the secret containing the new Valkey connection string in your environment after the migration completes.
If your application directly uses Redis commands to access the online store, note these Valkey cluster differences:
- Multiple endpoints: Cluster mode has multiple shards; clients may need to support cluster-aware libraries
- Key slotting: Keys are distributed across shards using consistent hashing
- No MULTI/EXEC across shards: Transactions only work on keys in the same shard
- Pipeline performance: Generally better due to sharding
Chalk provides guidance if your application needs client-side changes for Valkey cluster compatibility.
After the migration:
- Check connection — Verify your application can connect to the new Valkey endpoint
- Sample features — Fetch a few key features and confirm values match expectations
- Monitor latency — Valkey latency should be similar to Redis; flag any degradation
- Check pipeline health — If you stopped pipelines (Scenario B), confirm they’re catching up
Scenario A: If Valkey has issues after cut-over:
- Your application can be pointed back to Redis
- The backfill job can be stopped
- No data loss on Redis side (it was never fully stopped)
- You can restart the migration after investigating the issue
Scenario B: If Valkey has issues after start:
- Your application can be pointed back to Redis
- Redis was kept running and has the pre-migration snapshot
- No data loss
- Pipelines can resume writing to Redis while Valkey is investigated
Redis remains running during the migration specifically to allow quick rollback. Rollback typically takes 15–30 minutes.
Common questions about the migration process.
- Scenario A: <5 minutes of downtime (zero-downtime Chalk Apply cut-over). Feature serving resumes immediately, but full data availability takes 1–4 hours for backfill to complete.
- Scenario B: 2–4 hours with no new feature data. After pipelines restart, another 15–60 minutes for queued messages to be processed and catch up to real-time.
Consider Scenario B (stop-the-world) where serving is unavailable but clients aren’t surprised by errors. Alternatively, pause incoming traffic during the cut-over window.
Yes. Redis is kept running, so rollback is possible within 1 week (typically 15–30 minutes to execute). After 1 week, Redis is decommissioned unless you request to keep it longer.
The “lost” data is features written to Redis while the Valkey cluster was being created (RDB export to cluster init takes 30–60 min). This data usually exists in your offline store, so a backfill query recovers it. If backfill is needed, no permanent loss occurs — normal business processes will also re-populate data as required.
Yes. You can test the backfill query logic on your offline store before migration day. When you’re ready to run the actual backfill post-migration, Chalk provides the exact timestamps (start/end of the migration window) and feature list to use.
This is rare, but if it happens:
- You can re-run the backfill query or re-compute using the same start/end timestamps
- You can monitor backfill progress and see which features succeeded
- Failed features can be individually re-backfilled
- Chalk can provide guidance on the backfill query or troubleshooting if needed
No. Chalk uses a consistent library to interact with both Redis and Valkey, so your application code requires no changes. The migration is transparent to your feature definitions and queries.
Chalk provides:
- New Valkey connection string
- Confirmation that migration was successful
- Go/no-go for your team to stop monitoring
Yes, migrations can be rescheduled. Just notify Chalk.
Redis is kept running for 1 week after successful Valkey migration as a safety net. After 1 week, the old Redis instance is deleted. You can request to keep it longer if needed.
- Decide — Scenario A or B?
- Provide information — Send Chalk the details requested in the “Pre-Migration” section above
- Schedule window — Coordinate with Chalk on date/time
- For Scenario A: Prepare the backfill batch job with your data team
- Day-of: Acknowledge the cut-over window 30 minutes before it begins
Reach out to your Chalk support contact with questions or to schedule your migration.