The nightmare of any early-stage founder: when you’re building traction and suddenly your product breaks. Fixing it takes time and money that you don’t have. What you do have is a lot of angry customers and a failing business. Larger companies can also face costly setbacks due to problems in their software. Malfunctions in parts of a solution that worked before are called regressions. Regressions are harder to catch but are typically more expensive to fix. It’s therefore always better to find a solution before any problems occur.
Regression bugs create struggles. At Squads, we help customers mitigate this risk by automating regression testing. We automate this because as the feature set grows, the regression test suite grows with it. Manually running a regression test suite, and maintaining it separately from the code base is expensive, and error-prone.
In this article, I’ll outline our roadmap towards full test automation. A road always takes us from A to B. So, let’s define a starting point and an ultimate goal and then zoom in to the steps in between.
The most rudimentary software setup
A single developer writes code. Testing is done by running the code against production data. When it works, it’s live. Any tech-savvy computer user knows and uses this strategy for their first automation efforts. It’s the starting point for most webshops, even your favorite webshops, and Facebook. Moreover, my own scripts for personal use are maintained this way. Probably most software is maintained and run this way and there’s nothing wrong with that. As long as it is not used where it doesn’t apply.
The perfect software setup
A team of developers writes code. Their communication is effective and efficient. At least, that is what you’re aiming for. This means interruptions are minimal, while the response to changes is quick. In this set-up, the following rules should govern.
Rule 1: only test failures are allowed to cause interruptions for developers
This means that in order to earn the right to interrupt any developer, you must submit proof that something is not working, that the dev team agreed that should be working. You can extend this rule to other creative work. For example, I may be interrupted while working on this article, but only by someone that has proof that their issue is more important. I’m not very strict on this rule, because I welcome the chance to be nice to any community member while writing, but I understand that frequent interruptions result in lower productivity. For difficult development work, the choice is always in favor of no interruptions.
Rule 2: tests are well defined and reproducible
To make rule 1 fair, you should define a test for each feature before it’s built. If this doesn’t happen, there would be no way to interrupt the developer, and in that case, there would be no point in having them on staff. Define a test before you start paying for development (hiring a QA specialist for this can be a great idea), and then extend it before you interrupt a developer working on that feature.
Rule 3: everything is automated
Smart humans are lazy and they make mistakes in repetitive work. Computers are less likely to make mistakes in predictable work, and they are very good at repeating tasks. Once you have a well-defined test, reproducing it should be cheap. I’ve almost never seen automation fail to pay off. It’s usually cheaper than we think, and once in place, it mitigates risks that are usually much more costly than we think. Without well-defined tests, test automation is meaningless.
Rule 4: done means live
We automate the tests and the deployments they are run against – it’s the only way to implement rule 3 completely. Once the deployment is automated, it’s easy to apply automation to the production environment as well.
So in a perfect setup: everything is automated, there’s a test for everything, and we’re only interrupted when something is proven to be broken. I’ve never seen perfection, but you should strive for continuous improvement. A utopian goal can be a good one, and I think in this case it is.
Advice: take a close look at the current reality, and focus on pain points where the above rules are not met. It’s a simple but very effective recipe.
Text continues below photograph
Stage 1: from rudimentary to safe
Automating the wrong behavior is a disaster waiting to happen, so let’s first make it right and then automate it. First, we introduce a safety net of regression tests. We start executing those tests manually. At this stage, I usually invite product owners to put demo and test scripts in their issues before the planning meetings. The demo script is how they’d like to see the issue demoed during a review, the test script is how they’d like the edge cases to work.
A QA specialist can expand edge cases from the demo script and then go back to the PO for approval before the development work starts. In this way, tests become part of the design, as they should be. I’m a strong believer in BDD, and I think that a good designer is also a tester, and a good tester is also a designer. They are both focused on perfecting the user journey, and we should only develop things that are thought through sufficiently from both a holistic and detailed perspective.
Once there is enough functionality, the manual work of testing for regressions will become more and more costly. The impact of regressions will also increase with the user base. When the costs and risks are high enough to outweigh the costs of automation, start automating the regression tests with the highest risk first. Keeping a good balance between risk and mitigation is the essence of keeping your company safe.
Stage 2: from safe to fast
Once you’re safe and covered, you’ll want to speed up your organization’s response to change. In any market, moving faster means more success. This can be achieved in two ways:
- Reducing the time spent on a single change;
- Allowing changes to be developed in parallel.
To reduce the time spent on a change, we can automate even more. Automatic deployments allow a developer to share a URL to a working version of the software, instead of sharing on their local machine, or via a video-only. This reduces interruptions and helps you move towards full deployment automation.
To allow changes in parallel, we need to define smaller and more decoupled changes. We also need to decouple inside the code. This increases maintainability and makes the life of developers easier. To keep the automation working, we need to introduce automatic feature deployments. Inside Squads, each feature branch is automatically built and deployed, then automated tests can be run against them. Once green, I (the product owner) check the functionality knowing that I don’t have to do double work if I find a regression. This makes my life easier, and my relationship with the development team stays great.
Step 3: towards perfection
In this stage, things start to diversify per situation. In high volume B2C cases, it might be cheaper to let end users do the majority of regression testing. Then the marketing department can decide if something is working based on the conversion rate. In low volume B2B cases, it might be cheaper to record automatic tests from all end-users and apply them to all new features. These are two extremes on the risk vs. cost balance. Make sure that everything stays automated, and that the test: “no features used in the past 3 months by any user break,” or “our conversion doesn’t drop more than x%” is also applied automatically.
Text continues below video
Your tests should be designed according to your business goals. Eventually, it’s your stakeholders (investors, shareholders, key customers) that can define the tests that matter. The people in your team are merely automating those tests and making them pass. Using regression test automation has proven to reduce the risk of regression bugs, and cut down release times by an order of magnitude. At Squads, we’re aiming to be perfect at this, and we’ve come a long way. The video our Qualitas team made of the tests running proves to me that I’d have more sleepless nights if we didn’t invest in this.
Let me know if you have questions about implementing regression test automation in your company. Let us help you.
[button link=”http://dev.squads.com/talk-to-an-expert/” type=”big” newwindow=”yes”] Plan a call[/button]