Testing Efficiency – A Better View

The ISTQB defines Testing Efficiency as the number of defects resolved over the total number of defects reported.  This is meant to measure the test team by the relevance of the bugs they report.  A low efficiency would imply that the test team is reporting many bugs that are not worth fixing.

This view is pretty limited and simplistic.

A better approach would be to measure the “resolution category” for the bugs that are closed. The bugs, when resolved, are marked with a category like “fixed”, “Cannot Duplicate”, or “Duplicate of another bug”.  The categories can be graphed on a pie chart:

Pie Chart showing Resolved Bugs by Category

Pie Chart showing Resolved Bugs by Category

Now, you can have a conversation about the bugs being reported, and whether improvements are warranted. We had this exact issue in a team that I lead a while back.  We made a few adjustments:

Duplicate – we upgraded the bug tracking system to improve the search function. This allowed the testers to search for duplicates before submitting a new bug.  If they found a bug already, they reviewed it to see if they could add any new information.

Cannot Duplicate – for these, we did bug huddles with the developers. Showing them a demo of the bug before writing/submitting the bug. This practice really helped get the bugs fixed faster, by eliminating the back-forth that sometimes happens.

Business Decision – Many of these were closed by the developers without involving the Product Manager in the decision. We added the PM as the person to “verify” bugs closed with this resolution to make sure they agreed.

Pie chart after improvements.

Pie chart after improvements.

Want to learn more about leadership in software testing? Check out the Software Leadership Academy.