Common deployment strategies

 

Rolling out new software to a massive user base all at once is a risky move. Deployment strategies are ways to roll out an application considering different factors such as system downtime, roll back duration and costs.

Introduction

There are different application types and requirements (both technical and business). Therefore, choosing the right strategy is an important decision hence different strategies have different costs, time and quality attributes. In this post, the following popular five strategies are presented:

Destructive deployment approaches:

  • Replacing deployment
  • Recreate deployment

Constructive deployment approaches:

  • Rolling deployment
  • Blue/Green deployment
  • Canary deployment

Destructive approaches

The following deployment strategies are overwriting the production environment hence their name. These approaches include only one production environment and do not use load balancers.

Replacing deployment  strategy

Replacing deployment is the typical rollout method for in-house legacy (i.e. monolith) applications and when the user access to the application could be easily controlled (e.g. an in house application in a company for its employees).

In this method, all nodes (typically clients, servers and databases) within the production environment are overwritten with the new release. See Fig. 1.

Figure 1: Replacing deployment strategy

Advantages

  • simple: overwrites the old components with the new version 
  • cheap: does not require additional environments or load balancers
  • fast: due to the "bing bang" (all at once) changing approach

Disadvantages

  • difficult rollbacks: due to its destructive overwrite nature (though backups are often created before the deployment)
  • long downtime: it often requires many minutes or hours for the whole deployment process
  • high business continuity risk: considering rollbacks and downtime

Recommended when

  • the application is not business-critical
  • deployment could be carried out off office hours (e.g. at night on Fridays to ensure test and correction hours)
  • the application architecture is monolith

Variant: Partial replacing deployment strategy

Partial replacing deployment is a variant of the previous Replacing deployment. This is more adequate in multi-service (i.e. not monolith) applications when the user access could be easily controlled.

In this method, some nodes (typically service nodes) within the production environment are overwritten only.

Advantages (over the Replacing strategy)

  • fast: faster than the Replacing deployment strategy as only selected nodes are changed. 

Disadvantages (over the Replacing strategy)

  • long downtime: it often requires many minutes or hours for the whole deployment process, but less than the Replacing deployment strategy

Recommended when

  • the application is not business-critical
  • deployment could be carried out off office hours (e.g. at night on Fridays to ensure test and correction hours)
  • the application architecture is modular or more service-oriented

Recreate deployment strategy

Recreate deployment strategy aims at rectifying the difficult rollout characteristics of the Replacing deployment strategy. This is the typical deployment strategy for development and testing environments and is rarely used in production. For development and testing environments, it usually includes initial data-loads (testing data or the replication of the production data) also.

In this method, some nodes (typically client and service nodes) within the production environment are stopped and uninstalled (removed, deconfigured) then new versions are installed (copied, configured) and started. 

Figure 2: Recreate deployment strategy

Advantages

  • simple: overwrites the old components with the new version and the configuration settings are re-created using installation scripts.   
  • cheap: does not require additional environments or load balancers 
  • fast: deployment downtime equals the full reinstallation cycle time
  • more reliable than the replacing strategy: as the configurations are tested before the application is installed onto the production environment.

Disadvantages

  • moderately difficult rollbacks: equals with a full re-installation cycle
  • long downtime: it often requires many hours for the whole deployment process, even more than the Replacing deployment strategy (usually due to the data transformations or data migrations)
  • high business continuity risk: considering rollbacks and downtime

Recommended when

  • the application is not business-critical
  • when data migration or data transformation is needed before the new application runs 
  • deployment could be carried out off office hours (e.g. at night on Fridays to ensure test and correction hours)
  • it is often used when the deployment target is development or testing environments (copied the whole Production environment for the development team or the testing team)

Constructive approaches

The following deployment strategies are building up a new production environment hence their name. These approaches often have more than one environment (one for the actual and one for the new release) and use routers.

Rolling deployment

Rolling deployment (aka. ramped deployment) is a popular rollout method for multi-service applications using gradual updates.

In this method, all nodes (typically service nodes) within the production environment are updated sequentially - on condition the verification of the previous node is passed. 

Figure 1: Rolling deployment strategy

Advantages

  • cheap: does not require additional environments or routers
  • easy rollbacks: just the misbehaving node (i.e. failed node on verification) is replaced with the previous version
  • less business continuity risk: considering rollbacks and downtime

Disadvantages

  • complex
    1. checking mechanism: it updates the old nodes with the new release version but automatic testing is required before moving on to the next node update; 
    2. multi-version support: each other node has to support both the new and the old version of the other nodes API for seamless replacement 
  • moderately slow: due to its sequential updating and verification, it is much slower than the Replacing or Recreate deployments
  • moderate downtime: it often requires some seconds or minutes for one service update including the checkups
  • application multi-version support: each application should support two versions of other application nodes (the actual and the new release version - so it is required a very precise API version management (For more information please read the Git commit message guideline article.)

Recommended when

  • the application architecture is service-oriented (consists of many nodes)
  • small update rollouts (only a couple of nodes are affected per deployment)
  • the application is moderately business-critical
  • application node multi-version support could be implemented
  • deployment process and node verification are automatized 

Feature flagging

A feature flag approach is used to enable or disable functionality remotely without deploying code. With this approach, new features can be deployed without making them visible to users, therefore, deployment and release could be decoupled.

Blue/Green Deployments

Blue/green deployment is a common rollout method for multi-service applications using two identical switchable environments.

In this method, two identical environments are used for the deployment: the production (i.e. green) and the staging (i.e. blue). The green environment contains the old version of nodes, while the blue one contains the new version of the nodes. When all the automatic and manual tests are passed on the blue environment, the user traffic is redirected from the green to the blue environment. (As a result, the green becomes blue and the blue becomes the green environment.)

Figure 4: Blue/Green deployment strategy

Advantages

  • fast: if all the tests passed it is an ultra-fast method
  • easy implementation: it requires only a load balancer
  • easy rollbacks: with redirection, the previous version is available
  • no downtime: with the help of the redirection 
  • less business continuity risk: considering rollbacks and downtime

Disadvantages

  • expensive: it requires two identical production-grade infrastructures 
  • risky quality-level: testing in the blue environment is not able to detect some anomalies in the system
  • database multi-version support: each database node should support two versions of application nodes   

Recommended when

  • the application architecture is service-oriented (many nodes)
  • large update rollouts (many nodes are affected)
  • the application is business-critical
  • database node multi-version support (i.e. green and blue) could be implemented

Canary Deployments

Canary deployment is a popular rollout method for multi-service applications where the services are available gradually to a larger and larger subset of users.

Rolling vs. Canary deployment

In Rolling deployment, the application is upgraded by rolling out small instances of the new nodes one after the other (if the verification of the module is passed) until the whole application is updated. On contrary, the Canary put focus on traffic control (a subset of users) instead of nodes (a subset of services) that is a more relevant dividing factor. The Canary deployment could be considered as an improved version of the Rolling deployment.

In this method, the services are released (i.e. made available to end-users) incrementally to a subset of users (e.g. 2%, 5%, 10%, 25%, 50%, 100%). The canary deployment is the lowest risk-prone approach, compared to other strategies due to its gradual testing and release approach. Canary deployments allow doing controlled trials with real users.

Strategies for scoping users

There are different strategies to choose a subset of users, such as:
  • By geolocation: deploy the canary one geographical area at a time.
  • Early adopter program: giving users a chance to opt-in to the program.
  • Technical parameters: e.g. browser type, screen size, language, device type
  • Dogfooding: involve internal users and employees first (origin: “eating your own dog food”)

Figure 5: Canary deployment strategy

Advantages

  • fast and easy rollbacks: with redirection, the previous version is available
  • no downtime: with the help of the redirection 
  • less business continuity risk: considering rollbacks and downtime
  • high quality-level: because of its gradual testing and release approach

Disadvantages

  • slow and difficult: due to its gradual approach
  • expensive
    1. infrastructure: it needs to run two versions of the same services where the nodes should be scaled up and down depending on the subset size of the users. (It is less expensive than the Blue/Green strategy where both environments are equals.) 
    2. controlling: verification can take time and the monitoring and instrumentation for testing
  • database multi-version support: each database node should support two versions of application nodes   
  • deployment automation: script-based processes are not recommended (complex and error-prone)

Recommended when

  • minimal downtime and business continuity is important
  • the application architecture is service-oriented (many nodes)
  • the application is highly business-critical
  • small update rollouts (only a couple of nodes are affected)
  • database node multi-version support (i.e. green and blue) could be implemented
  • deployment process and node verification are automatized

Canary origin

The canary word originates from coal miners' caged canary bird, which is used in the coal mine to detect whether the level of toxic gas was too high.

A/B Testing and Shadowing are not deployment strategies

Some people classify A/B testing and Shadowing as deployment strategies. It's a misconception, as:

  • A/B testing's goal is completely different: it is used for experimenting with user interface changes and other features to get user feedback.
  • Shadowing's goal is also different: it is used to test copied traffic from production to test application performance in real-world scenarios.

Conclusion

There are multiple deployment strategies available. A commonly asked question may be, which deployment strategy should I use? The answer depends on the type of application you have and your technical and business requirements. A well-informed release manager has to choose from the available strategies (he/she often mixes and customizes them) that align best with the requirements.