Blue-Green Deployments

Chalk offers a variety of tools and strategies for testing your features, resolvers, and queries in the process of deploying them to production. Testing through writing unit tests and integration tests can ensure that your code is working as expected. Chalk also offers blue-green deployments for a release strategy that can further minimize downtime and risk when deploying changes. If your team is interested in using blue-green deployments in your production environment, please reach out to the Chalk team.

Blue-Green Deployments with Chalk

Chalk offers blue-green deployments as a way to minimize downtime when deploying new code by running two deployments, one “blue” and one “green”, in parallel. The blue deployment is the current production deployment, while the green deployment is the new version of the code that is being released. Using deployment tags, Chalk can route a specified percentage of traffic to the blue or green deployment, allowing for a gradual rollout of the new code, as well as an easy path to rollback if any issues arise.

Releasing using Blue-Green Deployments

Once you have confirmed with the Chalk team that blue-green deployments are enabled in your Chalk environment, there are a few ways that you can start to use blue-green deployments.

Full rollouts using Blue-Green Deployments

The simplest way to start to use blue-green deployments is to use the --bluegreen tag when deploying your code. When you run a chalk apply --bluegreen, that will automatically tag your latest deployment with either blue if the latest deployment was tagged green, or green if the latest deployment was tagged blue. Then, when you’re ready to migrate traffic to your new deployment, you can run chalk traffic promote, which will migrate traffic from the deployment currently in use to the other tagged deployment. The following example demonstrates how to use these blue-green deployments.

Suppose you have made changes that you now want to release. You can then run

$ chalk apply --bluegreen

Based on whether the current deployment handling traffic is tagged blue or green, the latest deployment will be tagged with the other tag. Then, you can run the following to migrate traffic.

$ chalk traffic promote

Say your previous deployment was tagged blue and the latest deployment was tagged green. Then this migrates 100% of the traffic from the blue deployment to the green deployment. You can then monitor the green deployment traffic. If you encounter any errors with your new deployment, then you can once again run chalk traffic promote and it will migrate traffic from the green deployment back to the blue deployment. This allows you to easily rollback during releases.

Gradual rollouts using Blue-Green Deployments

If you would like to gradually roll out your new deployment with staged increments of traffic, you can use the --deployment-tag flag to specify a new deployment towards which to direct traffic. To get started with blue-green deployments, you can tag the current deployment with blue. To do so you can run the following Chalk CLI commands:

$ chalk apply --deployment-tag blue
$ chalk traffic set --tags blue=100

This will tag the current deployment with blue and direct 100% of traffic to the blue deployment. Then, when you are ready to begin rolling out the next deployment, you can deploy the new version of code with the tag green.

$ chalk apply --deployment-tag green
$ chalk traffic set --tags blue=90,green=10

This will direct 90% of traffic to the blue deployment and 10% of traffic to the green deployment. You can verify the traffic distribution by running chalk traffic get.

$ chalk traffic get

tags:
     - deployment_id: cm9n46thy000bwe6y784y4j0l
       tag: blue
       weight: 90
     - deployment_id: cm9n3vx7c0004we6yefnsw8kg
       tag: green
       weight: 10

Then, you can gradually increase traffic to the green deployment until it is at 100%, and for the next time you release, you can tag the latest deployment with blue and migrate traffic from the green deployment to the even newer blue deployment. This workflow enables you to have fine-tuned control over the traffic distribution between your deployments. Furthermore, Chalk will automatically scale down unused deployments to minimize unnecessary resource usage. If you run the command chalk traffic set and do not specify some amount of traffic for a tagged deployment, then Chalk will recognize that the deployment is no longer in use and will scale it down.

Should I use Blue-Green Deployments?

Blue-green deployments are a powerful tool for minimizing downtime through a gradual rollout deployment. However, running two deployments in parallel also has resource implications and may not be necessary for all teams. We would recommend blue-green deployments for teams that have a high volume of traffic in their production environments where the benefit of minimizing downtime risk is worth the cost of additional resources required.

If you are interested in using blue-green deployments, please reach out to the Chalk team and we can help you determine what configurations are best suited for your release strategy.

Mirror Traffic

Chalk also offers a mirror traffic feature that allows you to mirror traffic from your production deployment to a new deployment. This is useful for testing new features in a production-like environment without affecting the production traffic. To use mirror traffic, you can run the following command:

# Mirror 5% of traffic to the green deployment.
$ chalk traffic set --tags blue=100,green=0 --mirror green=5

Note: mirror traffic does not block responses for existing production traffic. It simply mirrors the traffic to the new deployment without waiting to receive a response from the new deployment.

Metrics, logs, and traces from the mirrored traffic will be available in the new deployment. Queries will not impact the online store, but will impact the offline store if the new deployment is configured to write to the offline store. This permits statistical analysis of the new deployment using tooling like the query log or the offline store.

Light-weight Cookbook

You can make use of only mirroring with minimal reference to blue-green by:

Deploying your new code with the --deployment-tag flag to tag the deployment.
Mirroring traffic to the new deployment using the --weight flag to specify the percentage of traffic to mirror.
Monitoring the new deployment for any issues.

Example commands:

# Make the new deployment -- it won't receive any traffic yet
$ chalk apply --deployment-tag green

# Mirror 5% of traffic to the green deployment
$ chalk traffic set --tags blue=100,green=0 --mirror green=5

# Monitor the new deployment for any issues. Once you're satisfied, you can promote it to production with:
$ chalk traffic promote

A key point is that the mirrored deployment does need to have 0% weight specified, which indicates to Chalk that the deployment should have pods running to handle the mirrored traffic, but not to handle any production traffic.

Failing to specify the --weight flag will result in Chalk not running any pods for the off-color deployment.