Many types of software testing are conducted along a software development’s life cycle to ensure that the software you release to your customers is stable and behaves as expected.

Read more: smoke testing and user acceptance testing

One such type of testing is performance testing which seeks to validate the performance of the system and that your application provides an optimal user experience. It seeks to determine how the components of a system perform under certain conditions and various workloads.

Performance testing will primarily highlight any improvements you need to make to your software to ensure its stability, responsiveness and speed before it’s released into production.

In this article, we will focus on a subcategory of performance testing known as soak testing. 

What is soak testing?

Soak testing, which is sometimes referred to as endurance testing, is a type of performance and load test that evaluates how a software application handles a growing number of users for an extended period of time.

Therefore, the goal of this type of testing is to ensure that the software application can handle a huge volume of load and determine whether it will exhibit abnormal behavior, or even crash, under such circumstances. 

In other words, as its name suggests and from the word ‘soak’, the software is ‘soaked’ in or subjected to high load for a specified period of time.

Soak testing will be able to reveal any issues that can only come up after hours, or even days, of usage and to observe what would happen outside its design expectations.

Soak testing is usually run before the build is deployed. The application will then undergo a series of load tests after which soak testing is performed. Soak testing will then be the last step to ensure the reliability of your software.

Since this testing is done for a long period of time, it should usually be done on the weekends or overnight to ensure minimal disruption to the development team.

Due to the long duration of this test, it will need to be conducted with the help of an appropriate automation tool and so your team will need to possess the technical expertise to use such tools to run a successful soak test.

Why is soak testing necessary?

The reason this type of testing is performed is to check how the system behaves when it’s used continuously and under sustained load. As a result, the primary purpose of this test is to validate system behavior in production use for extended periods of time.

The main issues that soak testing seeks to uncover include:

• Memory leaks that may occur after extended usage- developers can find any potential leaks which might negatively impact performance and fix them
• Response time degradation– to ensure that response time after continuous use remains the same and as good as it was at the beginning of the test
• System resources degradation that would usually surface when a test is run for a long period of time
• Database connection issues– failure to close database cursors under certain conditions may result in the application crashing
• Layer connections failure– soak testing will reveal whether there is a failure of close connections between the layers of a system which could interrupt the modules of the system

It may behave normally for a couple of hours or so during other stages of software testing but soak testing will reveal whether the system remains stable if it’s used for a longer period of time or whether it would crash or behave abnormally.

As we’ve mentioned, the main problems this testing seeks to uncover are ones such as memory leaks or degradation in performance that could occur over time due to continuous usage.

Consequently, for this test, teams will seek to mimic real-world usage.

This means that soak testing gives developers the peace of mind that the system would run smoothly for longer periods of time without any maintenance required, thereby reducing higher maintenance costs if not identified during the testing stages.

As with other types of testing, there are some disadvantages associated with soak testing. For one, as this test usually runs for a long period of time, it can then be a time-consuming (and costly) process which could delay the release of the software.

It is also sometimes hard to determine how long to run the test exactly. You may run it for a good period of time and it passes the test but perhaps if you had just waited an hour longer, the application might end up crashing. In short, soak testing requires very careful planning.

Additionally, there’s the issue of having to run this test in a separate environment rather than the live environment to avoid major issues such as data loss and this will require extra resources and additional costs.

Soak vs load vs stress testing

There are many different types of performance tests and in this section we will discuss the differences between soak testing and other types of testing such as load and stress testing because while they may seem similar on the surface, they’re not exactly the same.

Load testing tests how a system behaves with a large number of users. The system is tested by slowly adding more load to it until it reaches the threshold limit. Thus, load testing is used to identify how many users a system can handle without any degradation to its performance.

Meanwhile, the purpose of stress testing is to investigate how a system behaves in extreme conditions, under high levels of stress. Stress testing may go beyond the limits of what the system can withstand, beyond the peak levels covered in load testing, and go so far as to purposely break it down. 

Therefore, it looks to test the stability of the system beyond its bandwidth capacity.

This will help teams investigate how the system recovers under such heavy stress, especially if you’re expecting unusually heavy traffic at a certain period and you want to test whether your system can withstand this sudden influx of very high traffic. If the stress test is successful then that means the system was able to return to normal after a breakdown.

Soak testing, for its part, is concerned with system reliability over a longer period of time. It’s a means to monitor and verify that the system can sustain continuous high levels of load.

To sum up:

• A soak test focuses on determining the reliability of a system over an extended period of time
• A load test seeks to monitor performance of the system under increasing load up to a certain threshold
• Meanwhile, a stress test measures the system’s stability beyond its maximum level, under extreme load

 

The following graphs illustrate the aforementioned points:

Load testing (load is gradually increased until peak load which is maintained for a short period)

Soak testing (gradual load over an extended period)

Stress testing (increasing load until the breaking point)

Conclusion

As we’ve seen, soak testing is a powerful form of load and performance testing that helps to ensure the reliability of your software.

Though it can be a time-consuming process and requires careful planning, it is useful in identifying performance issues that may arise during continuous use of an application and to determine whether an application can handle continuous load.

How imperative soak testing is will depend on your own circumstances. If, for example, you’re an e-commerce website that is running an online sale for a few days then it would be important to run a soak test in this scenario to ensure you’re prepared beforehand by making sure that your website can handle such an unexpected high load without crashing.

As with any type of software testing, it’s always best to uncover any underlying issues and bugs before it’s launched into production, after which it becomes more costly to fix these issues.

In modern software development, continuous delivery is a must. However, it is important to have a testing infrastructure set up to ensure your existing and main functionality retains normal operations despite new changes introduced.

This is where smoke testing comes in, or sometimes referred to as ‘Build Verification Testing’.

What is smoke testing?

Smoke testing is a software testing technique and form of regression test that helps to verify the stability of the recently deployed software build. It helps determine whether your new code has had any effect on your core functionality. 

This type of test is usually carried out by QA engineers, who will determine the most important aspects of the software’s features. Whenever there is a new build, the QA team determines the major functionality in order to perform smoke testing. Then, this team will uncover any faults in the application functionality with the introduction of new code.

Surprisingly enough, the name comes from plumbing where a smoke test is carried out on pipelines to know whether they’re working or not. In this context, smoke is fed into pipes which helps verify that there are no leaks.

It has the same underlying purpose in software development by ensuring the stability of your major functionality whenever a new build is deployed.

In that sense, this test is the first testing method you should implement whenever new changes are introduced to your software to determine whether it is stable enough to go through further testing.

Thus, smoke testing occurs at the beginning of software development testing life cycle.

Implementation methods

Smoke testing can be done manually or through automation. 

In the former case, the smoke tests are run manually each time a new feature is introduced while in the latter case, a set of automated tests is used allowing developers to check the build immediately.

However, some organizations may use a combination of manual and automated testing.

The process of smoke testing is pretty straightforward: first, builds are deployed by developers and sent to the testing team for testing.

Then, in the process of testing, if a bug is detected, it would fail the smoke test and this build is rejected; otherwise, if no major issues are uncovered, then the build can go on to further testing. 

To build this test, you will need to determine which parts of your application constitute the major functionality. Afterward, automated tests can be set up to test these major parts.

Benefits

Smoke testing helps detect early issues and enables your team to locate any errors or bugs before it reaches your user-base. Thus, this type of testing ensures that any new features you’re releasing do not affect your existing functionality.

This instant feedback means you can make changes quickly before they are accessed by your users so you can fix any issues, saving both time and effort later on.

Smoke tests are usually quick and simple and show that the feature is ready for testing. This means smoke testing is usually done in production just as you’re about to release your new features.

Therefore, the major advantage to this test is the early detection of major errors thereby reducing risks when testing later on on your users. It also saves time and cost which would have been wasted on further testing with a faulty build.

Because this test targets key features, it can be a cost-effective way to detect major errors and software defects by ensuring that high-quality software is being built.

Without this test, major bugs will be left undetected and could prove to be a major obstacle as you go further in the testing process.

Conclusion

After looking at the benefits of smoke testing, it is clear that this type of testing is vital as it locates defects in early stages and so it increases the quality and stability of your application.

Therefore, you should be carrying out this test on any new build or changes to avoid finding these errors when it’s more risky and costly.

To sum up, smoke testing is a must to ensure build stability at an early stage so you can move on to the next stage in testing with less risk.

The software development life cycle (SDLC) is a series of stages that helps organizations develop software in a well-structured way. It refers to a framework with clearly defined processes for creating high-quality software.

These days, many teams are moving towards an adaptive methodology, such as Agile, and moving away from more conventional ones such as the Waterfall methodology. 

The Waterfall methodology usually follows a strict series of steps or phases that details how the project from beginning to end, sticking to the original requirements.

Therefore, such a methodology is not adaptive to change and it usually takes a long time to deliver working software.

Agile software development grew out of frustration with the traditional Waterfall methodology to accommodate change and bring about faster software releases.

Agile practice helps you carry out continuous iterations of software development, with the end-goal to deliver high-quality software to users as quickly as possible. 

Read more on the differences between Agile and Waterfall methodologies.

Software development life cycle stages

Despite the differences in methodologies, each development method uses the basic principles of SDLC. The only difference is that your team adapts each phase to suit the chosen methodology.

Before we discuss the steps of an Agile life cycle, we will provide a general overview of the stages in a traditional software development life cycle. The 7 stages of the software development life cycle to develop software in a well-structured way are:

1. Planning
2. Requirements
3. Software design
4. Software development
5. Testing
6. Deployment 

The planning phase starts with defining the terms of the project by, for example, getting input from all stakeholders, as well as its purpose and scope, followed by determining the requirements of the application; in other words, what it’s supposed to do and the problem the software to be developed is addressing.

The design phase turns the software specification into a design including its architecture and user interface. Then, the software will be developed. Developers will start the coding process so there must be proper guidelines in place to implement the process.

Afterward, the software built will be tested before making it available for end-users to make sure it’s working as it should. Some of the testing can be automated. At this stage, developers must ensure that the software is free from bugs. 

Finally, the software can be released or deployed to the production environment and made available to users. 

There may be a final ‘maintenance’ phase, where bugs that weren’t discovered during testing will need to be resolved. The software may also need to be updated and additional features added in the future.

Phases of the Agile life cycle

In an Agile SDLC methodology, work is done in regularly iterated cycles known as sprints, often lasting for two to four weeks. It is largely driven by customer feedback and so helps developers build software based on this feedback.

1. Concept– this is where the product manager will determine the scope of the project. The key requirements are discussed and outlined.
2. Inception– this is the stage where the software development team is put together and the requirements are fleshed out.
3. Iteration– this is the longest phase as most of the work into the software is carried out here. Developers start working on the first iteration of the software to deliver the bare functionality of a product; this product is not fully functional and not the final version as it will require numerous revisions to end up with the final required functionality. However, additional features can be added in later iterations.
4. Release– QA tests are carried out to ensure the software is fully functional as well as user training which will require documentation then the product’s final iteration can be released into production. End-users can use this product and feedback will be gathered accordingly.
5. Maintenance– the software at this point is fully deployed and made available to customers. During this phase, developers will provide ongoing support to keep things running smoothly and remove any bugs. They would also offer additional training to users so that they know how to use the product. Further iterations may be introduced if necessary to improve the existing product and introduce new features.
6. Retirement– the final phase is where the product may be replaced with new software or the system itself is no longer in use by the organization so they will notify users and, where relevant, help them migrate to the new system. Developers will then carry out end-of-life activities.

The Agile model is more flexible and employs iterative methods, which is creating software by modifications and changing requirements to satisfy customer needs and increase productivity. Unlike a traditional model, which does not allow for major changes in later stages, it allows for rapid changes after the initial stages of the project.

Additionally, since there are frequent builds within an Agile model, it offers the ability to detect any deviations from customer requirements, which can be resolved early on.

Conclusion

In the end, the methodology and framework you decide to adopt will largely depend on your project needs.

An Agile methodology has clear benefits as it aims to produce rapid solutions, which can enhance productivity as well as customer satisfaction by providing them with high-quality software suited to their needs.

Before starting your software development process, it is important to understand which methodology best fits your requirements. The size of the team, complexity of the software and engineering capability are some of the factors that you will need to consider to make an informed decision.

A site reliability engineer (SRE) creates a bridge between development and IT operations by taking on the tasks typically done by operations. Instead, such tasks are given to these types of engineers who use automation tools to solve problems by creating scalable and reliable software systems.

Standardization and automation are at the heart of what an SRE does, especially as systems migrate to the cloud. Thus, they often have a background in software or system engineering or system administration with IT operations experience.

What is site reliability engineering?

We will start with a definition of what this type of engineering is before we move onto the role and responsibilities of a site reliability engineer.

Site reliability engineering is a term that was first coined by Google, where it is described as “when you treat operations as if it’s a software problem.”

The main purpose of SRE is developing software systems and automated solutions for operational aspects. Thus, SRE does the work traditionally done by operations but instead using engineers with software expertise to solve complex problems.

Therefore, site reliability engineering can be considered a set of practices that incorporates aspects of software engineering into operations thereby increasing the efficiency and reliability of software systems and improving workflow.

SRE and DevOps

Site reliability engineering is closely related to DevOps, another concept that links software development and operations, and can be seen as a generalization of core SRE principles. Consequently, SRE plays a large part in successfully implementing DevOps practices.

Additionally, both DevOps and SRE seek to bridge the gap between operations and development teams to deliver software faster.

However, an article by Google makes a distinction between the two terms stating that SRE “happens to embody the philosophies of DevOps, but has a much more prescriptive way of measuring and achieving reliability through engineering and operations work. In other words, SRE prescribes how to succeed in the various DevOps areas.”

Read more about DevOps and what a DevOps engineer does.

What does a site reliability engineer do?

A site reliability engineer (SRE) works between development and operations. The SRE, then, is a software developer with experience in and knowledge of IT operations.

A lot of this role revolves around writing and developing code to automate processes, such as analyzing logs, testing production environments and responding to any issues, so this engineer will be an expert in writing code.

Such automation allows developers, in turn, to focus exclusively on feature development enabling them to bring new features to production as quickly as possible.

The operations team, for their part, will find their workload decreasing as a SRE will automate solutions for any recurring problem.

Thus, he/she will be shifting between development and operations work and maintaining a balance between them.

Because an SRE engineer’s main focus is on automation, this means that he/she enhances performance, efficiency and monitoring of software development processes.

Required skill set

SREs dedicate their time to creating software that will improve the reliability of systems, fixing issues and responding to incidents and issues. As such, they will need various technical skills. 

They will need to have knowledge of various automation tools as they are usually responsible for building and integrating software tools to enhance an organizational system’s reliability and scalability.

As mentioned above, the SRE will require knowledge of coding and most of the common programming languages including Ruby, Javascript and PHP.

He/she will also need to have expertise in the major cloud providers such as AWS and Google Cloud.

Daily roles and responsibilities of an SRE

Automation

As mentioned previously, SRE engineers build tools for automation to manage IT operations. Thus, instead of manually performing these functions, their aim is to automate them. Such functions include:

• Continuous integration and continuous delivery
• Monitoring 
• Incident response
• Alerts 

Monitoring

SRE engineers are responsible for ensuring that the underlying infrastructure is running smoothly and that systems and tools are working as expected. 

They also monitor critical applications and services to minimize downtime and ensure their availability.

Issue resolution

These engineers work closely with developers, especially when issues arise so they will collaborate with developers to help with troubleshooting and provide consultation when alerts are issued.

This engineer will investigate and then resolve the issue in the event that a developer runs into a problem.

Following the incident resolution, the engineer will revisit the issue and determine the cause to ensure it doesn’t happen again.

Cross-team collaboration

Based on the above, SREs work across different teams, mainly operations and development. Building reliable systems and providing support to these teams, will give these teams more time to divert their attention to building new features and hence get these out faster to customers.

Common tools used by SREs

• Monitoring: such tools include AWS CloudWatch and NewRelic 
• Incident management/on-call: such as PagerDuty and VictorOps
• Project management and issue tracking: such as Jira and Trello
• Infrastructure orchestration: including Terraform and SaltStack

To find out more tools from project management tools to infrastructure and container orchestration used by site reliability engineers, check out this curated list of SRE tools.

How much does an SRE make?

According to payscale, this type of engineer makes a salary anywhere between $76,000 to $158,000 a year in the United States with the average being $117,768 per year.

Conclusion

A site reliability engineer is becoming an increasingly important role within organizations. It is a challenging role that requires a passion for coding and automation.

Having such engineers in your organization will help reduce your operational costs while improving the reliability of your systems.