What is a CI/CD pipeline?

A CI/CD pipeline introduces automation to your software delivery process. While it is possible to perform the steps of the CI/CD pipeline manually, the real value of CI/CD pipelines is achieved through automation.

Thus, you can look at a CI/CD pipeline as a series of automated tests that takes the source code all the way to production, allowing the delivery of software multiple times a day automatically.

Before we go into further details into the steps involved, it would be helpful to explain what CI and CD acronyms stand for.

What do CI and CD mean?

CI/CD is the acronym in software development which combines the concepts of continuous integration and continuous delivery. It may also encompass continuous deployment but in this case, CD will be mainly used to refer to continuous delivery.

Continuous integration is a software development practice where developers merge their code changes into a shared trunk multiple times a day.

The next step is continuous delivery, which enables these code changes to be released to end-users at any time. Thus, continuous delivery ensures that your code is always in a deployable state. Continuous deployment takes this even one step further deploying the code automatically rather than developers having to do it manually.

Each code that is released will need to pass through a series of automated tests to make sure that it is ready for release and to let developers know whether a software build passes or fails.

The actual release step can be done manually but the processes included within CI/CD are fully automated, which can be monitored by developers for feedback purposes.

Why is it important?

Such a pipeline is essential to incorporate within agile processes as it makes fixing bugs easier and releasing new software faster than ever before.

It can also be considered as the backbone of a modern DevOps environment by ensuring the quick delivery of high-quality releases.

As a consequence, the quality of the code is greatly improved because the CI/CD pipeline will automatically test the code and signal to developers if there is an issue before deploying to users.

CI/CD elements

A CI/CD pipeline is triggered by an event such as a pull request from a source code repository.

Each company has different requirements and processes so you can personalize the CI/CD pipeline steps to suit your needs.

Nevertheless, normally, the software release process goes through a series of pipeline stages as we will outline below. The typical stages are:

• Source– this first phase includes the creation of the source code. You may have multiple source portions from Java to C++ depending on the project being developed.
• Build- this is where developers put their code. In this stage, the build process draws the code from a repository. The build tools used will depend on the programming language used.
• Test– this stage ensures that the code is working as intended to prevent bugs from reaching end-users. This usually starts with unit testing
• Deploy– this is where you deploy the code to a testing environment. Often, the build automatically deploys if it passes the series of automated tests.
• Automated testing phase– the automation test phase will perform the final tests necessary to ensure there are no bugs before deploying to production.
• Deploy to production– after the code has passed all the tests, it can finally move to the production environment to reach end-users.

Throughout this entire process, feedback will be instantly sent to developers in case there are any issues so they can review them.

This is highly beneficial and preferable to manual testing as manual testing defeats the purpose of continuous development since it would take a lot of time to test making it difficult to deploy on a frequent basis. 

Feature flags as part of the CI/CD pipeline

Feature flags are especially useful to incorporate into the CI/CD pipeline. Feature flags are a software development technique that allows you to hide features that are not yet ready for release.

Therefore, feature flags help accelerate the purpose of CI/CD pipelines, which is to deploy code several times a day. You can look at feature flags as a safety net that would allow you to release features safely and quickly even when there are still unfinished changes.

Conclusion

A CI/CD pipeline is an important asset for any organization looking to increase the speed of delivery resulting in a shorter time to market and hence greater customer satisfaction.

Since it relies on automation, it minimizes the occurrence of manual errors. Instead, a CI/CD pipeline gives rise to a continuous feedback loop allowing developers to constantly improve products.

The automation aspect frees up your team to focus on other less tedious tasks and do more of what they’re good at, which is writing code.

To sum up, a well-designed CI/CD pipeline will allow your team to follow DevOps practices that will shorten time to market by pushing code more often, which results in improved code quality.

What is Continuous Deployment?

Part of being agile is the ability to react quickly and efficiently, particularly when it comes to software development.

Continuous deployment and continuous delivery are two terms that are seen so often together that they may be used interchangeably and so there’s much confusion between them. They both refer to how features are released to production.

There is, however, a major difference between the two. With continuous delivery, someone will need to approve the release of the feature to production but with continuous deployment, this process happens automatically upon passing automated testing. 

Continuous deployment is basically when teams rely on a fully-automated pipeline. This practice fully eliminates any manual steps and automates the entire process. Therefore, continuous deployment ensures that code is continuously being pushed into production. 

Real-time monitoring, however, would still be needed to track and address any issues that arise during the automated tests and to ensure that the builds passed these tests. 

CI/CD and CD process

Continuous deployment is part of a continuous process. The first step starts with continuous integration when developers merge their changes into the trunk or mainline on a frequent basis. This helps teams evade what is known as ‘merge hell’, which happens when developers attempt to merge several separate branches to the shared trunk on a less regular basis. 

Then comes continuous delivery when code is deployed to a testing or production environment. Here, developers’ changes are uploaded to a repository, where they are then deployed to a production environment. With continuous delivery, you can decide to release daily, weekly or monthly but it is usually recommended to release as often as possible in small batches to be able to easily and quickly fix any issue that arises.

Continuous deployment goes one step further and combines continuous integration and continuous delivery to make software automatically available to users without any human intervention. Hence, continuous deployment requires the complete automation of all deployments and so refers to the process of automatically releasing developers’ changes from the repository to production, bypassing the need for developer approval for each release. Consequently, this process relies heavily on well-designed test automation.

All these practices come together to facilitate and accelerate releases making deployment less risky.

Benefits of Continuous Deployment

In this sense, continuous deployment accelerates releases to market and it also quickens the feedback process with both developers and customers. The work of developers goes live almost immediately after they’ve finished working on it. This way, developers can respond to such feedback in real-time and immediately respond to any bug reports or if they want to test a new idea, they can quickly deploy and validate new features.

Continuous deployment also takes away release day pressure so developers’ sole focus is on building the software, which automatically goes live in a matter of minutes.

It also heightens productivity and allows developers to respond to rapidly shifting market demands.

Continuous deployment, however, requires a high degree of monitoring and maintenance to ensure it continues to run smoothly as well as a great capacity to respond to changes quickly.

Features flags and continuous deployment

We can conclude that teams practicing continuous deployment need to be very thorough in their automated test practices as all releases are happening automatically without prior approval. Therefore, any bugs that manage to bypass these tests will make their way to your end-users.

This issue can be solved through the use of feature flags as they add a layer of safety to the continuous deployment process. A kill switch can be placed on each feature so any bugs detected don’t need to be rolled back but instead you just switch off the feature toggle. This way, the parts of the feature that are not working as they should are not exposed to your users while the rest of the changes to the feature can be released without the need for rollbacks.

Quick review

In summary:

• Continuous integration (CI): integrate changes to a shared trunk several times a day.
• Continuous delivery (CD): continuous integration then deploy all code changes to production environment; deployment is manual.
• Continuous deployment (CD): step further from continuous delivery; automated deployment to production without any need for developer approval.

CI/CD, then, can be taken to refer to the connectedness of the continuous integration and delivery processes or it can even refer to all three practices of continuous integration, continuous delivery and continuous deployment.

In the end, what’s important to remember is CI/CD, visualized as a pipeline, automates the software delivery process by building code, running tests and deploying this code to a live production environment.

What is Continuous Delivery (CD)?

Continuous delivery is a software release strategy alongside continuous integration and continuous deployment. The idea behind continuous delivery is preparing the software for delivery, often multiple times a day and thus being able to deploy new releases to users at any time. To have an efficient continuous delivery process, it’s important that continuous integration is built into your pipeline. 

The code will still need to pass through a series of automated tests but the actual release into the production environment is done manually.

Therefore, continuous delivery ensures that your code is always in a deployable state to a production environment even while numerous changes are being made by multiple developers on a frequent basis.

Benefits of Continuous Delivery

Quicker release to market

Since your code is always in a state ready for deployment, this means that you can continuously release new features to production at any time. This is particularly crucial in order to respond to rapidly changing market demands so just as quickly that developers can make the necessary changes, they just as quickly can release those changes to end-users.

Reduced risk

Just like continuous integration, when changes are released more frequently, errors are easily spotted and fixed. Developers can easily roll back any faulty releases as they are being released in small batches which are less complicated to correct in the production process.

Jez’s Humble’s page on continuous delivery outlines four main principles to achieve lower risk deployments. This includes the following:

• Low risks releases are a step-by-step process. This means that the system needs to be constructed in such a way that it allows for release of small batches rather than one big bang release. This requires building versioned APIs and implementing patterns such as circuit breakers.

• Decouple deployment from release. Integrating feature flags into your continuous delivery pipeline. Using feature flags will allow you to decouple deployment from release so you could release some changes while keeping the ones that you don’t want to expose to users behind a flag with a simple on/off toggle. The idea is to deploy continuously but release new features on demand.

• Reduce batch size. Releasing in small batches actually reduces the risk of deployment as it becomes easier to spot and roll back accordingly. 

• Optimize for resilience. Basically, teams should always be prepared in case something goes wrong and to treat each setback as a learning opportunity. Thus, it is important that everyone on the team knows what procedures to follow to locate a problem and get things up and running again.

Enhanced products

Because you are continuously releasing your changes, you get faster feedback from your users. This way you can make adjustments as needed taking into account your customers’ feedback resulting in higher customer satisfaction with your products.

Better coordination and productivity

Continuous delivery reduces the stress associated with software release, particularly when it comes to rollbacks. Also, releasing more frequently not only helps the team work at a more efficient pace but also allows developers to actively engage with users as they are continuously monitoring any small changes they introduce to their releases. This allows them to eventually provide more value to their end-users.

The CI/CD process

The first step starts with continuous integration then continuous delivery builds on this process by performing the necessary steps to release the changes integrated into the shared trunk during continuous integration. Together, CI and CD deliver many benefits including shortening the software development process, allowing developers to receive instant feedback and features to be released frequently.

Continuous delivery is oftentimes confused with continuous deployment but there is a distinct difference between these two procedures. While continuous delivery, the software is ready for deployment but human intervention is required to trigger these deployments, with continuous deployment, you deploy the software immediately and automatically. 

This means that with continuous deployment, every change automatically goes into production. Meanwhile, when it comes to continuous delivery, you have the option to release to production but you may choose not to do so. However, to implement continuous deployment, you need to do continuous delivery.

CD and DevOps

DevOps brings together the concepts of development and operations, highlighting the collaboration between the two. Thus, the DevOps culture emphasises the shared responsibility between development and operations.

CD is often linked to DevOps because part of adopting a DevOps culture is increasing value and responsiveness through quick and high quality delivery. Such an approach necessitates the creation of a continuous delivery pipeline.

In other words, DevOps provides the conditions necessary to develop, test and deploy new software and so the outcome of implementing DevOps is a CI/CD pipeline supported by effective collaboration between development and operations teams. 

Continuous delivery and feature flags

Implementing continuous delivery is not without its risks and hurdles. However, using feature flags alongside continuous delivery makes it easier to maintain the momentum.

CD requires a great amount of speed. However, there are times when you find that you need to slow down to finish changes to a feature, which puts a damper on the continuous flow that you were trying to maintain, beating the whole purpose of continuous delivery.

With feature flags, though, you can maintain that momentum. If there are still incomplete changes to a feature, you can leave these changes turned off by using feature flags until it’s ready but you would still be able to proceed with the release if other changes are ready for deployment. 

The benefits of using feature flags with continuous delivery are clear. No longer would you need to wait for all changes to be ready to deploy your feature which means you can still deliver your features rapidly and safely to your end-users.

What is continuous integration?

Continuous integration (CI) can be considered as an alternative to feature branching. To put it simply, this process occurs when teams integrate their codes on a continuous basis into the shared branch or mainline paving the way for instant feedback. 

In this case, trunk-based development enables continuous integration or in other words, it enables teams to merge small changes more frequently into the main codebase or trunk, potentially several times a day. Trunk-based development essentially provides developers the means to divide their work into small batches and then merge their work into the trunk or mainline at least once a day. Meanwhile, feature branching can take days or weeks of work as developers make changes to long-lived branches and do not merge the branch until a feature is complete.

Build automation is one of the major prerequisites for continuous integration. Each integration made by a developer is verified by an automated build to detect any errors. A build tool thus will handle delivering the software in an automated way. In other words, each commit made by a developer should trigger a series of automated tests to ensure the quality of the code. These automated tests should give you an indication that your software is working as it should. There are an array of tools to choose from depending on your needs including Jenkins, Travis CI, Circle CI, TeamCity and CodeShip.

Benefits of CI

Faster deployment

Consequently, this technique allows developers to move fast. Unlike in feature branches, the master branch in the trunk-based development is the only long-lived branch while all the other branches have a limited life-span, reducing merging conflicts since no branches are hanging in development for too long. It allows for quicker feedback too, almost immediately, so there is increased transparency among developers instead of waiting for weeks on end for developers to integrate their changes from a private branch.

Risk mitigation

When developers continuously integrate their changes into the mainline, it’s much easier and faster to detect bugs and eliminate them from your code. Therefore, your chances of releasing a more stable feature with less bugs leading to a significantly improved product that is always ready to release. It also reduces the risk of merging long-lived branches that give rise to conflict and what is known as ‘merge hell’.

Higher quality releases

Continuous integration is essential if you want to release new features and make small changes quickly while helping to minimise what is known as ‘merge hell’ that often results from long-lived branches. As previously mentioned, with continuous integration, bugs can be detected much quicker since continuous changes are taking place resulting in increased efficiency and productivity. 

This means that the quality is vastly improved as changes are continuously made and you’re not waiting for everyone to be finished to integrate your changes allowing you to check your code more frequently resulting in greater confidence when releasing your new features.

Drawbacks of continuous integration

The main challenge with this kind of practice is that due to the frequent changes that are made, developers may be tripping over one another to integrate their code at the same time which may lead to increased waiting times. This practice also requires constant monitoring to quickly address any issue that arises. Nonetheless, it is certainly useful when you are looking to release new products quickly and safely.

Continuous integration and continuous delivery vs. continuous deployment

We often see the terms continuous integration and continuous delivery used together. In fact, the two go hand-in-hand where they are often referred to together as ‘CI/CD’. 

For continuous delivery to take place, you need continuous integration. As such, continuous delivery comes into play to enable the automated delivery of the integrated code from development to production stage. 

With continuous delivery, you are preparing the release for delivery on a frequent basis. Hence, code is delivered several times a day. It is responsible for testing the code and ensure it is delivered without bugs or delays by sharing any changes with a select number of users to get their feedback before these changes are rolled out to all users. 

Consequently, CI/CD represent the process of continuous development, testing and delivery of new releases.

The final stage of CI/CD is continuous deployment, often confused with continuous delivery. However, continuous deployment means that any changes made to the code base is deployed automatically to production whereas continuous delivery prepares the feature to be released to production but the when is not done automatically but dictated by the team. Taken together, all of these practices make any release a much less risky endeavour. 

CI and feature flags

There might be instances where one developer may have completed their changes and have integrated these changes into the mainline while another developer’s work is not yet completed. An issue arises where the feature cannot be released until all the developers have completed their changes possibly causing delays to the integration process. Here, the conflict can be resolved by merging all changes including unfinished ones but without exposing the half-changing changes. 

This can be done using feature flags, which helps enable or disable parts of the code so the feature will be off if it is incomplete. You are essentially placing these unfinished changes behind a flag until it’s ready for release. Then, once this change is ready it can be turned on with the simple switch of a button. Therefore, the use of feature flags in such a scenario helps keep the branch healthy

Final thoughts and best practices 

Continuous integration, then, allows for frequent deployment. It enables you to get your releases out to your consumers quicker and to get more feedback on these features. It also smoothens the collaboration process between developers increasing productivity and the quality of their releases.

We will end this article with some CI best practices to check whether your team is really practicing CI, as listed by Martin Fowler from his comprehensive article about CI:

• Maintain a single source repository
• Automate the build
• Make the build self-testing
• Commit to the baseline everyday
• Every commit should be built
• Fix broken builds immediately
• Keep the build fast
• Test in a clone of the production environment
• Make it easy for anyone to get the latest executable
• Everyone can see what’s happening
• Automate deployment