Category Archives: Quality Management

Vanity Testing Metrics

This is a preview of a topic that I will cover in the upcoming talk, Testing Metrics – Choose Wisely at STPCon.

Vanity metrics are popular in marketing. These are metrics that allow you to feel good, but aren’t directly actionable, and are not related to your (true) goals.  Vanity metrics are also easily manipulated.  An example would be a hit counter, measuring page views, on a web site.  What would really matter for a business web site would be the conversion rate (how many visitors actually purchase) or revenue per customer.

I’ve seen marketing campaigns that add a lot of page views, but actually cause a decrease in conversion rate. The advertising may find more viewers, but if the people are less interested in your product, its not really useful to drive up traffic.   (and who knows if those viewers are really people and not bots) Measuring the impact of advertising by measuring revenue or number of visitors that become customers is more powerful.

An example in software testing is measuring the Average Age of bugs.  You might start a campaign to reduce bug backlog or improve the velocity of fixing the bugs, and a measure might be the average age.  However, what you are really looking for is a quicker response to every bug, not the average bug.

The average age of bugs chart from JIRA shows trends in the average age, over time.

The average age of bugs chart from JIRA shows trends in the average age, over time.

This metric is often misleading in these efforts, as really old bugs can be fixed or closed and dramatically reducing the average age.  In the chart above,  the dramatic downward swings actually came from closing only a couple of bugs. Those bugs weren’t fixed, they were closed as obsolete.  But, they were open in the backlog for several years, so closing them had a dramatic impact on the average age.  Closing those, however, didn’t tell us anything about the responsiveness to current bugs.

Instead of Average age, tracking the median age.   The median measure would be much less affected by really old bugs.  Medians are a way to prevent outliers in having outsized impact on your metrics.  Even better, a more direct measure of our goal to improve velocity might be to set a target timeframe, say 30 days – then measure the percentage of bugs that are fixed within that target.

These views will more directly measure your goal (improved velocity) and be less susceptible to manipulation.


Eliminating biases in A/B testing

A/B testing is a powerful customer-driven quality practice, which allows us to test a variety of implementations and find which works better for customers.  A/B testing provides actual data, instead of the HIPPO.

The folks at Twitch found that the users in the test cell had higher engagement than the control group. They found that this higher engagement came from factors other than the new experience, which might cause a cognitive bias in their results.  Factors like the Hawthorne effect and new users break the randomness for the experiment.

They adjusted the data to reduce the impact of these effects, and provided a great case study on how they did it


Testing Myopia

Horse and Buggy by Patrick Henson

Horse and Buggy by Patrick Henson

In school, marketing class. Concept of marketing myopia, where companies focus too much on the product they are currently producing. This near-sightedness keeps businesses from adapting to changing technology and market conditions. The classic example is the buggy whip manufacturing company.

The invention of the automobile is a threat to buggy whip producers, because the automobile technology will eliminate the need of buggy whips. If, instead, they see themselves as “transportation initiation” companies, then the automobile is an opportunity to serve a new set of customers, those who don’t want to hand crank their engines.

The idea behind marketing myopia is to take a step back, take a longer view of your product. A good way to do this is to think about what service you provide, rather than what product you produce.

What does this have to do with software testing and quality leadership?  Well, sometimes we get focused on our tools and processes. These tools of the trade are an important part of our craft, but the larger organization is not really looking for great automation suites, exploratory tests, or exit criteria. We talk about bug counts and severities. We talk about test pass/fail rates. We talk about code coverage.

Our product is testing, test results, and bug reports.

But, what service do we provide?  Why does the organization “purchase” testing?

Our organizations hire us to provide a service, that service is confidence. Confidence that our products will meet or exceed customer expectations. Confidence that our systems will be secure, available when needed, and perform as advertised.  Confidence that when its time to ship that the product is ready to ship.

How does taking a different look at our role influence what we do? What do we do differently if we are in the confidence business rather than the testing business?