What is Regression testing and why is it important?
Regression testing is an additive validation approach which confirms the new changes didn’t break the system or its components in spite of being it is still in development. This method is incremental in nature because every change or a feature addition brings a new set of test cases to add to the regression testing scope. The developers may not find it conducive as not only they have to fix issues reported via regression but also maintain a sync with the QA on changes impacting the system behavior. However, it also poses a challenge for the testers too for selecting cases that are more relevant, realistic, and repetitive.
Regression testing is useful in all type of testing models. However, it is more successful in Agile testing. It can significantly reduce the testing cost in the long run if applied correctly. It is one of its kind testing approach that aims to build confidence in software going through rapid changes without unintended side-effects.
Since the scope of regression is going to increase, so it’s not viable to do it manually. The best way is to choose an automation framework pertaining to your test requirements. And then create the test suite, start test case automation, and reduce manual testing efforts. To leverage such a test suite, integrate it with CI tools like Jenkins and prepare it to run on the nightly basis.
When is it useful to perform Regression testing?
We should adopt Regression testing approach in the following scenarios.
- In a product which continually requires new feature addition.
- For testing product enhancement where you want to minimize manual testing efforts.
- To validate fixes of defects reported by customers.
- When the product is expecting changes that relate to its performance.
What are some of the advantages of Regression testing?
Regression testing works the best if implemented correctly. It improves the quality of the product under test and has following benefits over traditional methods.
- Notify us of any side effect occurred due to a fix or an enhancement in the module or application.
- Ensure that the bugs found earlier don’t re-emerge.
- Not only can it be done manually but tools are available to automate it.
- It contributes towards improving the quality of the product.
- In products with long lifecycle, it can considerably reduce manual testing efforts with the help of automation.
What are some of the disadvantages of Regression testing?
Regression testing makes life easier for the testers while working on a large software project. However, it has some limitations which we can overcome with the steps mentioned in the next sections of this tutorial.
- Without automation, it is hard to manage the cost of regression testing as the testing scope grows with every new feature coming to the product.
- Automating regression requires skilled software engineers.
- Changes in old features lead to modifications in the related test cases which further requires versioning.
- Testing new features requires more cases to be added which increases maintenance cost.
- It impacts the overall cost of the project budget.
- Regression tests have to run upon any small or big change taking place in the code as the smallest of modifications could bring down the existing functionality.
What are the challenges with Regression testing?
Regression testing can be tough for the testers in the following scenarios.
- The large is the no. of features in a product, more is the no. of test cases needed for regression.
- Executing a big regression suite takes time and sometimes turns unfeasible due to time and budget constraints.
- Running a regression test suite on a nightly basis demands a dedicated infra or systems which incur additional hardware cost.
- Optimizing a test suite for reducing execution time and achieving maximum test coverage is not at all easy.
- Full utilization of regression testing suite is a challenge as it requires to know when to run the suite i.e. for every minor change or after every build or when a bunch of bug fixes is available.
How do you select test cases for Regression testing?
You already know how critical regression testing is for delivering a quality product. And test cases are the primary elements of a regression test plan that contributes the most to make it successful. Hence, it is inevitable to select the most appropriate test cases to get the best results. So here are a few ideas for you to ponder.
1. Select test cases for features that get the most defects.
Find out the areas in your product which receive the most bugs and cause to fail with just a small change in the code. By reviewing the weekly/monthly bug reports, it is easy for you to identify the areas leading to maximum no. of defects. First of all, you can add those defects to regression and then look for increasing the test coverage of that particular area.
2. Select test cases for features that are core to the product.
Before you start designing the test cases for regression, do find out the core areas of the product. So get to the requirements specs, review the product design documents and come up with the features that are most critical to the product. Consequently, you can proceed with the test case selection and cover the desired functionality. With the help of traceability matrix, you can confirm the test coverage.
For example, in a web application, the regression should cover areas like login, dashboard, reports, and other core features apparent on the home page.
3. Focus on test cases for areas updated recently in the product.
In an Agile world, requirements keep on changes frequently. But most of the times, the changes occur only in parts of the product. And once the product’s first version is ready, there could be changes as little as 20-30% due to enhancements or bug fixes. In such cases, try focusing on the recent changes and add cases which can break the existing functionality.
4. Select test cases that cover integration testing.
However, the integration testing happens as a part of the software testing process. But some of its tests should also run along with the regression testing. It helps rule out any chances that the product could miss an important functionality because of the last minute change.
For example, change in the auth protocol may cause the login API to fail, fixing an error message could lead to failure in the reporting API.
5. Select all end to end test cases.
Every product has some critical end to end business flows that require following a composite sequence of UI actions.
For example, to buy a product from an e-commerce website, first of all, a user needs to find the product from a particular category, select the product, add it to the cart, apply a coupon if available, select the payment method, supply contact/delivery details and proceed to pay.
By adding more actions in the sequence, you can increase the probability of finding a serious bug. If any of the actions stumbles from the chain, then the entire functionality could collapse. And that’s why we are advocating such complex test cases to become the part of the regression testing suite.
6. Filter test cases based on priority for regression testing.
We can’t have a regression that keeps adding indefinite no. of these cases. Somewhere we’ve to stop and that we should know by making an informed and well-thought decision.
So start classifying all the regression test cases. Have priority categories like P1 (Very High), P2 (High), P3 (Moderate) and so on. Alternatively, you can segregate test cases based on their functionality. And you can even add tags to filter test cases. It could be a release tag, a label for Service Pack or Patch.
The ideas behind such a classification of test cases into several priorities should derive from the importance and customer impact.
Here are a few rules which a software tester can apply to customize the regression test execution.
1. If the Severity and the Impact of a bug are Low, then executing a bunch of tests from P1, P2 & P3 priorities would suffice.
2. If the Severity and the Impact of a bug are Medium, then execute all P1 & P2 test cases. However, the tester can also run P3 test cases if required. By the way, if the bug fix requires adding new test cases, then they should also run as a part of the regression.
3. If the Severity and the Impact of a bug are High, then execute all P1, P2 test cases and include a few of the selected P3 cases.
7. Select test cases to update when an old functionality gets changed.
It’s not often that a customer requests re-writing of an old feature. However, such things do happen. And the developers have to amend it. And so the tester has to respond accordingly.
- A significant shift in the product functionality.
- Build procedure/pre-requisites have changed.
- Some part of the regression test cases never got executed.
- Regression test cycle includes only a few selected test cases.
- Expect a steep deviation in the test results from the previous execution.
What are the steps required to perform Regression testing?
The purpose of regression testing is to uncover bugs during the various stages of the product lifecycle. And to achieve this objective, the QA team along with developers should devise an effective regression testing strategy starting from its onset. Here, we are laying down a list of steps that would help in carrying out a successful regression testing.
Step 1: Establish the requirements and target components
It is important to identify whether the product is developing from scratch or it’s the part of the product which is under development. Once you filter the first part, then go after looking a further deeper and isolate the components/modules getting changed. That’s how one can determine what’s exactly should become the part of the regression testing. It’s a step which you should repeat whenever a module has got a bugfix or a new feature added to the product.
Step 2: Select automation tool for regression testing.
Select a bunch of automation tools meeting your test requirements. Evaluate and identify their pros/cons. Discuss with business stakeholders, developers, and software testing engineers. And identify the appropriate tool for your project or product. Come to an agreement for the current and future cost with all the parties involved. It’s a step that you need to perform once so you must be very clear in choosing the right tool for regression testing.
Step 3: Define entry criteria for regression test cases.
The entry criteria outline the minimum eligibility or the least of conditions to satisfy before you begin the testing. So a test engineer should take care of the following.
- Make sure the test or the defect is repeatable and has proper documentation.
- If it’s a defect to cover in regression, then do check its history to identify and track the regression testing efforts.
- Add a regression test which targets the defect or the test requirement.
- Get the tests reviewed and approved by stakeholders.
Step 4: Define exit criteria for regression test cases.
Since the scope of regression keeps increasing with the arrival of new features and defects, so it’s important to set an exit point. Like the entry criteria, exit criteria also define the minimum eligibility or the least of conditions to satisfy before you declare the testing phase as closed.
- A software test engineer should complete this step during the planning phase and get it approved timely. Here are a few tips to follow.
- Ensure that regression test completes the full cycle.
- Check if the desired code coverage is ready.
- Don’t miss to check any severe bugs or defer it after approval.
- Finally, verify that regression has not skipped any “high-risk” areas.
Step 5: Define a schedule for execution.
After finalizing the above steps, it’s time to decide the frequency and schedule of test execution. Usually, the best practice says regression should run after any commit happens in the code. However, it’s a bit overkill to launch all the tests for every small change. So you can segment the test cases and classify few tests for ensuring sanity. You can run the sanity frequently. However, you should prepare for executing the complete Regression at least once in a day.
Since it’s not feasible to run the regression or part of it manually, so prefer using a continuous integration tool like Jenkins. It will make your life easier. You can use it to configure the tests to run any no. of times. It will let you control the regression the way you wishes it to be.
Summary – Regression testing.
While we tried to simplify the idea of regression testing above, we wish this information would help you implement an effective regression testing strategy.
It is one of the most important concepts of software testing and also practically relevant to the test requirements of the modern development.