[ad_1]
As the Latin proverb says: “Quis custodiet ipsos custodes”. Roughly translated, it means: Who monitors the watchmen? This paradox can also be applied to your testing strategy and is possibly one of the biggest reasons why an organization’s testing approach fails.
In this blog, I will try to explore the paradox a little more, dig a little deeper into the causes, and brainstorm on the remedies.
What the developers develop is tested by the testers. But what do the testers do? Who checks this?
In most organizations, no one does. In some organizations, peer reviews or manager reviews are conducted for exemplary test cases. If a test case misidentifies a bug, the developers protest and the tester corrects the test case/test script. However, if the tester doesn’t look at an error, the error propagates through the system.
A multi-layered test strategy is usually followed. Eg unit tests, integration, system, smoke, sanity, etc. But if the same or a similar team generates these test cases, what is missed is likely to be missed by other levels of testing as well. While multi-layered testing is required, it doesn’t go too far in solving the paradox we’re discussing.
Tester errors are often not recorded.
This allows the bug to get into the production environment. And we all know that fixing a bug in production is much more expensive than fixing it at earlier stages.
The problem can be more serious, because how do you identify an error in production? Usually your users don’t report a bug, they take the easier route of avoiding the product themselves, and the bug sits in the production environment for a long time, silently adding the damage.
You’d wonder why don’t we fix this if it’s such a big problem? The reason lies hidden in its recursive argument. Because if we have reviewers for the testers, someone might ask why not the reviewer for the review and so on.
Of course you have to stop somewhere.
And hence the paradox.
Ensure that the quality control exercises themselves meet the quality levels by applying the following steps;
Institutionalize a verification process with evidence
Treat each test case as a work product that cannot be approved without evidence and testing. So the author of the test case is different from the person running it.
The executor has the right and obligation to log errors in the scripts as part of the test project. These deficiencies will be tracked to completion. The bugs in test scripts must be treated as formal bugs in the project, but at the same time they should not be mixed up with the bugs logged for the product, as they are not product bugs and will break the product’s bug database.
Perform reverse verification of randomly selected scripts
Next to strengthen the process; Reverse verification should take place, ie for a sample set of test cases, the developer should verify the test cases to certify the coverage of the functions. This step helps in several ways:
- It breaks the paradox and comes full circle with the developers instead of finding the keeper for the keeper.
- Developers understand their code best and are the best people to certify the completeness of test cases.
- At the same time, we should not make this a general practice as it will take up too much of the developer’s time and to some extent detract from the value that independent QA brings to the table. But limiting it to a random sample will statistically help us determine the overall quality of coverage.
Effectively define the coverage metrics and link them to the traceability matrix.
There are different types of coverage like functional coverage, test coverage, code coverage, requirements coverage etc. The metrics should be well defined and tracked. This helps to find gaps in the test case definition and whether the test scripts address all conditions and areas of testing.
Another critical step in verifying coverage is creating and maintaining a detailed Requirements Traceability Matrix (RTM). This helps track whether all aspects of the requirements are tested and whether untested requirements are not published.
Let statistics come to the rescue
Data is your best friend when it comes to monitoring the success of your testing strategy. As said, “data doesn’t lie” and therefore can be the most authentic monitoring tool for the effectiveness of your tests.
Some of the key metrics you can track and use to determine if your tests are working are as follows:
- Error Detection Rate (DDR): This is the rate at which errors are injected into the module at a given stage of development (such as design review, coding review, unit testing, integration testing, etc.).
To measure this metric, you must first agree on the metric that measures the greatness of your software. There are many options like story points, function points, lines of code, SMC rating (simple/medium/complex) for each individual module, etc. Once the sizing metric is done, you can measure the error detection rate as follows:
- Error detection rate = number of errors identified in this software module StageSize
- Error Injection Rate (DIR) – This slightly trickier metric tells you at what stage the detected error was injected. This is more difficult because it is difficult to know this deterministically. Also, the determination requires some time and cooperation of the QA and development teams. However, if a discipline is put in place for this, it helps to correlate the detected errors with injected errors.
If the level difference between detection and injection is higher, it means the test levels could not detect the error, and sheds a direct light on the test performance in that level. If this number is more significant than the threshold set for your project, this is a “Check Engine Light On” for your testing strategy and requires investigation and corrective action.
- Similar to the error metrics given above, scaled down to the module size, you can also use many other metrics to help you gain insights into the performance of your test module.
Deviations from the established norms (even positive deviations) require a reason to investigate and isolate causes that affect the effectiveness of your tests.
- These metrics are:
Total number of test cases per software size
Total number of test data per software size
The total number of first-time successful test cases per software size
The total number of failed test cases per software size by number of test cycles
The total number of accepted/rejected bugs per software size
It should be noted here that negative deviations are often reacted to, e.g. B. when the manager determines that the injected error for a module is higher than the norm for injected errors.
However, please note that such a deviation will most likely lead to coding inefficiencies (which are of course very important). But the deviation from the positive magnitude, like the defect injected for a module, is less than the established norm. While this may simply indicate that the development team did an excellent job, on the other hand, it may indicate that the testing team underperformed and found minor bugs that they should have.
Introduce easter egg bugs to check the quality of the tests
How the testers test whether the code works according to the specifications or not; The developers can test whether the testing works as expected or not. This is done by injecting “Easter Egg Defects”. These are bugs intentionally introduced into the software to see if the testing team can identify them.
Of course, the easter egg bugs are written in such a way that they can be easily and cleanly disabled. This is one of the most expensive and risky approaches to test quality detection, but also the most effective.
As we have already discussed, there are two types of test quality issues:
- Incorrect test definition or inappropriate test data
- Insufficient scenario coverage or ineffective test data
The first type of errors can lead to overlooked errors or extraneous errors. Foreign defects are easily detected because the development team protests against them quite vehemently. As a metric, this data is summarized, rejected bugs per software size and stage of software development and can be effectively monitored.
The missed errors or the errors missed due to the second problem are difficult to spot. Strategies such as data monitoring of bug injection rates, bug detection rates, review of test cases and data, and Easter egg bugs can help reveal these bugs.
When you mature the process by these means, you gain control over missed errors and can effectively reduce them, improving the overall effectiveness of your testing strategy.
To streamline your software testing process and make it more efficient, Wishup offers software testers who are skilled in automation testing. To find out more about them, email us at [email protected] or visit our website Wishup.co
[ad_2]
