Recent *studies have shown that 75% of companies use both manual and automated testing.
This marked a significant change from 2010, when I first started my blog. It is dedicated to helping people move towards more automation in their testing efforts.
This number was likely between 5-10% back then.
Although automation investments have increased substantially in recent years, there is still a significant gap.
Most engineers I spoke with were still focused on automated functional testing.
Although it’s a good start, many people miss out on a huge opportunity.
End to end automation in DevOps
If a process is possible to automate in the software development lifecycle, it should.
Automation is a key opportunity for many to save time and money in their development, build, or production phases.
This is why I believe that more automation engineers should push for end-to-end test, which not only includes functional automation but also automation to automate things like:
This last point is where I see the potential for teams to be more focused on the user.
We’re doing this because we want to make our app better.
This is where testers are the most qualified to lead it; they spend more time exploring their app than anyone else.
These areas are being expanded by automation tooling vendors to increase automation coverage.
Look at mabl
Let’s take mabl as an example.
The firm was still in beta when I first spoke with them. They initially focused on functional UI automation tests.
However, they have evolved over the years to become a more unified platform for software testing.
Recently, I spoke with Dan Belcher , founder of mabl. He explained that they were only focused on browser testing when they first started the company. They learned that automation testing is difficult, especially for those who don’t have programming backgrounds, over time.
They created a low-code solution to automate end-to-end testing.
Low-Code Test Automation
Some might wonder, “How realistic can low-code solutions be?”
It’s feasible, according to other automation experts like Paul Grossman. Low code can be very useful because it optimizes your time.
You don’t have to know complex languages like JavaScript, Python or Ruby if you are working with low-code tools. You don’t have to spend hours learning how to create your test. It is possible to navigate your app as a user, and then record the steps as a test.
Clicking on the Add to Cart button will take you to your test.
This allows people who may be a little hesitant about programming to participate in the automation efforts. This can allow other members of the team to understand the users and create and run reliable, realistic and efficient test automation scenarios.
Low-code solutions are still available for those who have coding experience. JavaScript allows you to control what JavaScript supports even if it doesn’t.Here’s an example of low-code test creation in mabl. You can record tests through the mabl Trainer and add assertions without needing to write code.
A low-code automation solution such as mabl offers the additional benefit of greater test coverage at the user’s experience level, not only for functional but also for other tests.
Machine learning is another way to increase coverage.
Machine Learning
The best thing about mabl is the fact that it runs primarily on Google Cloud. This allows them to see a lot of information while tests are executed and then process them efficiently and quickly.
This allows you to manage all aspects of test orchestration. Writing scripts and managing a framework is difficult. It can be challenging to analyze traces, metrics and visual information as well as logs, accessibility and test output.
If you roll your own stuff, it’s something most testers will have to deal with.
Cloud service providers offer many services that can help you open up more data for testing.
All this data can also lead to more reliable tests.
Auto-healing is a great example.
Auto Healing Feature
Mabl’s auto healing has heuristics that have different weights for different attributes. This allows backtesting to see if specific changes will result in more reliable tests.
Mabl’s team also said that they work directly with customers to refine their ML engine, based on their architecture.
FYI–According to a recent study (buildkite), 59% developers handle daily, weekly or monthly brittle testing.
Consider how many brittle tests you and your team would be able to pass.
Waiting for elements to display is another example of intelligence that testers can benefit from. Cloud tests usually run very quickly. Sometimes, the tests are executed so fast that they run faster than your application can load. Fixed waits are often used by testers to avoid problems. This can lead to more time-consuming delays.
Mabl is able to learn the behavior of your system, and will wait only for what it needs. The tests are executed exactly when they are finished. Mabl may, for example, wait for half a second instead of waiting 5 seconds by default. This is because it knows historically that that’s how long it took for an element to render. This speed ups execution and reduces false negatives.
Machine Learning to Replace Engineers?
One misconception about machine learning is that it will replace software developers. But it’s not about replacing; it’s more about augmenting.
Machine learning provides insight that can be used to guide a team in making decisions about how to proceed.
Let’s face it, I have been involved in automated testing for nearly 25 years and sometimes it still is a struggle.
It’s even more important to include your entire team.
As development velocity increases, I have seen teams after teams trying to maintain functional quality. There’s also the desire to expand testing for non-functional quality but not enough time.
Let’s say you have more advanced AI machine-learning techniques. It should give you core automation coverage, enough to let you focus on high-value activities such as:
- UX
- Strategy
- Collaboration
- Performance
- Ensure that your software is easily accessible
It is important that testers are allowed to focus on these tasks instead of reactive work on which many teams have been stuck.
How can teams approach new areas such as accessibility or performance?
Coverage for Non-Functional Testing
Mabl has added new features to make it easier to create end-to–end tests.
They have added accessibility checks and performance insights to their low-code, end-to–end tests over the years. Reusing existing functional tests allows teams to get more bang for their buck.
Dan noted that many mabl customers had reported that they had to rebuild their tests in another tool after they built them out in mabl. This was to include accessibility testing.
This feedback can be baked into mabl along with the requested features to allow users to leverage existing functional tests and perform critical non-functional checks.
Without creating a new test in mabl, add an accessibility check for common accessibility issues, such as color contrast.Understanding all of the accessibility issues across your application, categorized by failure reasons, and trend that data over time.
This is a trend that I have seen other vendors adopt, as I mentioned previously.
Another trend that I have seen is the way all these components are combined into a quality engineering design.
Quality Engineering Mindset
Quality engineering is made up of three concepts.
1. Quality Engineering-Focused Team
For one, a quality-engineering-focused team is building quality from the minute a developer begins working on a branch locally through to production. The traditional QA team receives a deliverable from developers sometimes ready to be tested.
Mabl provides functionality that aids the development build stage. For example, it offers a CLI. We’ll discuss this later.
2. Insurance
The second reason is that a quality-engineering-focused team considers coverage much more ambitiously.
They want enough automated coverage to be able to trust the coverage. It is the difference between feeling confident and having excellent coverage.
mabl provides advanced reporting to help analyze production and inform the team of gaps in coverage. Machine learning is used to extract insights from all data in CI/CD.
3. Data
We now reach the third area, data.
Quality-engineering-focused teams are measuring, changing their processes, and measuring again to see what’s going to improve. These same data are also used to identify non-functional requirements such as accessibility, performance, or other.
These three pillars provide incredible coverage and quality throughout the pipeline. Then, you can use data to improve your work as well as non-functional quality.
Reporting and Machine Learning Enhanced Insights
Data is crucial to great quality engineering practices and high-quality testing. Let’s examine how machine learning and data interact.
CTOs are the most important people I talk to.
Many companies are undergoing digital and DevOps transformations. They want to know which KPIs to use for each team member, as well as how to combine them to improve the overall initiative.
Tests run in the cloud can capture a lot of data, which can lead to a wealth of insights that are impossible to find any other way.
This type of data is invaluable for understanding what has been done in relation to a release. You need to know what you have done in relation to a sprint if you are developing in a sprint.
- What percentage of my tests were successfully completed?
- What is the failure rate and pass rate?
- What percentage of the tests were updated?
- Are there any new tests that I have added to this release?
- Does my coverage suffice?
The mabl workspace quickly surfaces recent test execution activity.
Mabl helps you to understand your test coverage, but they also take it one more step.
Now, teams can also correlate usage coverage and test coverage.
You can view your most visited pages and see how many pages you have added to your application.Quickly identify where test coverage needs to be added, and prioritize that based on actual usage of your application.
As I mentioned above, Mabl can curate a lot data, but users can also stream data to BigQuery (Google’s data warehousing service).
It’s used by many people to connect to customized dashboards that are tailored to the needs of their teams.
Unified Platform
Dan shared another lesson from mabl with me: customers missed the chance to catch problems earlier by running end-to-end testing in later stages, such as a staging environment.
They discovered that it was almost late.
It is best to do as much testing before your branch merges with your main branch. This is when the branch can be put on the train for production.
Developers must verify them locally and within CI to make sure you have sufficient testing coverage before merging.
This is why mabl developed a local runner for developers to run the tests before they are posted up for review.
This is what mabl refers to as a Unified Runner.
Unified Runner
The Unified Runner lets you use the same technology for both local and cloud-based tests.
It is easy to feel confident in releasing your software knowing that both Cloud-based and local tests yield the same results.
The Unified Runner integrates your test automation in your development pipeline. This allows developers and QA engineers alike to quickly release high-quality product updates while testing effectively.
mabl has developed CI tool integrations as well as a CLI to execute tests without the need for a human being before it is deployed in an environment.Example of a test executing via the command line interface.
It’s fun to play with automated testing, but how can you get developers involved in the development and execution of these tests?
Developer Friendly
Collaboration is an essential element of DevOps. Testing is the same.
Testing in Agile and DevOps environments requires that entire teams collaborate on quality. This includes developers.
Although I am seeing more teams involve their developers in testing activities I have heard from them that they are letting their developers test within their current workflow.
This is an excellent example of the CLI I mentioned.
Developers can easily trigger tests in their source control and CI tools to increase the likelihood of adoption.Using GitHub, developers can trigger tests to run before pull requests are approved or code is merged to the main branch.
Mature teams can use mabl’s version control and branching capabilities to update or create new test cases on a particular branch. They can also promote the tests with the code, so they are available for testing with new features.
All this data is stored in the cloud, making it easy for teams and users to view what’s happening within their applications.
Everyone can access test results if more testing is done early in development. Developers can see what is being tested and QA can assist developers if they spot a gap in coverage.
Cloud solutions can pinpoint the steps that a bug failed to pass, so they are ready for production when it happens. Solutions such as mabl can be integrated with issue tracking software, like Jira to create issues right from your test failure. This includes all the data necessary to quickly get to the root cause.Creating a Jira issue from a failed mabl test. The exact step a test failed is captured in Jira tickets, along with logs and screenshots developers need to reproduce issues
Automation Innovation
Vendors such as mabl are rising to the challenge in modern software development by offering tools that can help you or your team keep up with the rapid pace.
Vendors like mabl can be used to secure internal applications behind firewalls even for enterprises that have traditionally adopted on-prem environments.
