Continuous testing as part of a CI/CD (Continuous Integration/Continuous Development) pipeline is a pillar of all Agile approaches to software development but especially DevOps. Agile development is confined to short iterations that improve and expand software systems, often based on feedback and user experience data.
Agile’s tight feedback loops and development loops have been shown to increase software quality in terms of its technical performance as well as better serving users’ needs. However, it is also more risky to have software down while in production.
Software updates are often accompanied by bugs and errors. If software is not working as expected, it can lead to production shutdowns. There is more chance of bugs and errors being missed during testing, the larger the new version.
Agile’s small, frequent iterations reduce the risk of errors and make fixing those that do occur much easier. DevOps is an extension to Agile software development that introduces automated testing in CI/CD pipelines. This reduces risk and makes testing more efficient.
Automation in testing relieves a lot of the stress of manual testing. This is both time-consuming, expensive and intrinsically unreliable since humans don’t excel at repetitive tasks. It also has limited ability to predict the effects of software system changes.
Although automated testing can’t replace manual testing completely, it does reduce the time and effort required for any phase of software delivery cycle testing. It reduces the amount of work required and enhances its strengths.
Google Cloud suggests that software developers “continue to run both automated and manually tested throughout the delivery process in order to validate the functionality of the system under construction”.>
Continuous testing, both automated and manual, is valuable, but developers must execute it well in order to reap its full benefits. Avoiding common pitfalls is key to doing so.
Avoid these common pitfalls when you introduce continuous testing to an Agile DevOps cycle for software development
Google Cloud also identifies the common failure points in continuous testing that is part of a DevOps/CI/CD pipeline.
1. Separate Development and Testing Functions
DevOps is a well-evolved approach to software development that combines development, testing and operations.
This can make it difficult to achieve that goal if developers don’t test their code continuously as they write it, and instead just pass it along to dedicated testers.
Involving developers in testing by making them responsible for the creation and maintenance of automated testing pipelines makes it easier for them to fix failures. Taking testing responsibility away from developers means they are more likely to write code that is harder to test.
This can lead to code that is not up to standard and expensive, high-maintenance test suites. If updates to the codebase are required, tests must be performed by an independent testing team.
2. It is not possible to pair developers with testers & QA experts
Dedicated testers and QA groups still play a critical role, especially in larger DevOps organizations. They bring a different perspective to how software works and have a better understanding of how users interact.
Developers and testers can learn from one another, which is key to a healthy DevOps culture. They also have the ability to solve any issues in real time. Developers can also use specialist testers to help them curate test suites and perform UI/UX testing.
3. Failure to Cure Test Suites
Development teams often fall into the trap of not curating test suites and constantly reviewing and improving them to improve their effectiveness. The acceptance test suites must not only be able to automate the collection of acceptance criteria, but also replicate real user journeys through the software system.
These scenarios should evolve with the software system in order to be optimally efficient.
As the codebase changes, unit tests will need to be updated. However, if many have to change with each code update then there may be an over-reliance upon mocking.
Test suites that are expensive and difficult to maintain can indicate problems in the software architecture. Refactoring can be a costly investment, but it will ultimately result in software that is more easy to use and saves significant resources over the life of a software system.
4. False Ratio between Unit and Acceptance Tests
Automated test suites are designed to identify flaws and errors early on in the process. This is why you will find a variety of testing methods, from automated unit tests to slower-running acceptance testing to manual testing.
The aim should be that the unit tests find errors. If they are not discovered before acceptance or even manual exploratory testing, a new unit test should be added to catch such errors earlier in the testing cycle next time. The sooner they are found, the easier and cheaper they are to fix.
5. Toleration Of Unreliable Tests
Unreliable tests should not be tolerated by DevOps teams. Software that passes the testing pyramid must always be released with high quality. It is easy to assume that an error in the testing pyramid is an isolated one. This is not true. The most common flaw of DevOps teams that are poorly developed and perform poorly is the failure to improve the test suite.
Continuous Testing is more than lip service
A majority of DevOps teams don’t have the technical expertise to be considered highly advanced. Software development teams are now required to adhere to a specific, often Agile, method of software development.
There is a big difference between teams that only pay lip service to a method and those who integrate it into their culture at every stage of the development process.
To be a highly developed DevOps team, or organisational approach, you must insist on continuous testing and optimize it. This article will highlight the most common pitfalls in continuous testing and help teams make significant progress towards a more mature, efficient, and effective DevOps culture. This will ultimately lead to software systems that can be built and maintained at a lower cost and higher quality.