Category Archives: Customer Driven Quality

Eliminating biases in A/B testing

A/B testing is a powerful customer-driven quality practice, which allows us to test a variety of implementations and find which works better for customers.  A/B testing provides actual data, instead of the HIPPO.

The folks at Twitch found that the users in the test cell had higher engagement than the control group. They found that this higher engagement came from factors other than the new experience, which might cause a cognitive bias in their results.  Factors like the Hawthorne effect and new users break the randomness for the experiment.

They adjusted the data to reduce the impact of these effects, and provided a great case study on how they did it


The Testing Show Podcast – Making QA Strategic

I recently found a new podcast about testing. The Testing Show podcast. This show appears to come out every 2 weeks, and is a panel discussion hosted by Matt Heusser and Michael Larsen.

The show usually starts with a news segment, where the panel discusses some major software bug that happened in the previous couple of weeks. Then, the panel moves onto a topic for that session. The topic that spurred me to write this was “Making QA Strategic”.

At first, the session sounded like it was going to be a complaint about how testers should be taken more seriously. “We have the knowledge, if they would only listen to us”, is too often a refrain amongst the testing community – delivered as a complaint.

However, that sentiment quickly faded and the bulk of the podcast was about testing professionals giving their advice on how to make the testing team more strategic. Here are a few examples:

Josh Assad: “I try to build partnerships with my customers”. He told a story where he traveled to the customer site and spent a week building relationships and figuring how to optimize his testing to fit into the customers acceptance practices.

Jared Small described the value of his team as staying focused on customers and helping the entire team stay focused on the customer.

Jarad also mentioned that a big part of his role is to stay on top of industry trends and what is happening in the software test community.

Matt Heusser described an interesting model, the Swiss cheese model of risk. No one technique will eliminate all risk, in any economic fashion, but combined efforts for automated tests, manual tests, and production monitoring – when taken together will greatly reduce overall risk.   I’m not sure that I can describe how this relates to Swiss cheese, but it sounded good. Maybe I should eat lunch before listening to podcasts.

Matt also described the SWOT (Strengths, Weaknesses, Opportunities, and Threats) model, as applied to doing a gap assessment of the team. Identifying the gaps between your existing team and the ideal is a great way to determine improvement goals.

I also really liked Erik Davis’s approach to innovation. He stresses the importance of trying new things and keeping the ones that work. I believe testing & quality is a field with tons of opportunities for innovation. As I like to say, there is literally an infinite amount of testing that could be done, but a (very) finite amount of time.

Take a listen if you enjoy podcasts. I’ve subscribed and will be browsing their archives. Another cool feature, they have the full transcript on the web.

Well done.

Quality Hierarchy: Relating Agile Testing with Customer-Driven Practices

Emily Bache has an excellent post describing a Maslow-like hierarchy for quality, inspired by Gojko Adzic. She relates the quality hierarchy with Lean Startup test concepts. I found the integration of these two models to be a very useful way to think about an overall quality strategy, combining agile practices with customer-driven practices.

Emily Bache's diagram relating Quality Hierarchy with Agile Testing Quadrants & Lean Startup Testing Concepts

Emily Bache’s diagram relating Quality Hierarchy with Agile Testing Quadrants & Lean Startup Testing Concepts

The Quality Hierarchy was developed, and explained well, on Gojko’s blog.  The Quality Hierarchy relates quality attributes to the Maslow Hierarchy of needs. In this model, attributes like deployability are equivalent to Maslow’s Physiological needs (such as breathing, food/water). The higher level needs, like successful products equate to self-actualization.   In this model, if you can’t build you software, there is no need to worry about how many stars your app earned in the app store.

Emily extended this model by relating testing practices from the Agile Testing Quadrants and testing practices derived from Lean Startup concepts.

Agile Testing Quadrants

Agile Testing Quadrants

Recently, I’ve been writing about practices for engaging with customers, and found Emily’s post to be a great way to integrate this customer-driven thinking with all of the other things we do in software development to ensure high quality. I’ll be using these ideas to build (and communicate) the context for our quality strategies.

Book Review: Interviewing Users by Steve Portigal

I recently read Interviewing Users by Steve Portigal. I chose this book to brush up on my skills for interviewing customers. I develop software for a living, and feel strongly that learning directly from customers is vital for building the best product for them, and this book did not disappoint. Interviewing Users: How to Uncover Compelling Insights (Author: Steve Portigal, Publisher: Rosenfeld Media, Released: May 2013)

Interviewing Users by Steve Portigal

Interviewing Users by Steve Portigal


Steve Portigal is a user researcher, sharing his framework, and experience in this eBook, for conducting interviews as part of an overall user research program.

The core of the eBook is chapter 2, which presents the framework for conducting a user interview: preparation, actually conducting the interview, recording information gathered, analyzing and decomposing the information into insights, finally presenting the findings.

He covers the context for each topic, why that topic is important and how it relates to the bigger picture of user research. He also shares many examples and stories that happened in real interviews. I also got a lot of value from Steve’s tips and tricks, which he obviously learned through experience.

Additional good things about this eBook:

  • I felt like I actually learned from his experience. I felt like I was conversing with Mr. Portigal, with him teaching me how to interview customers. His many stories and tips made this book feel conversational.
  • Companion websites: all of the photos from the book are available on, which make it easy to incorporate those in any subsequent presentation that I would like to make. The photos are copyrighted.
  • The companion website has links to videos for more in-depth learning, and templates to use. (Look for the resources link, after wading through the sea of links that are advertising the book itself). The companion sites will help me actually apply these lessons back in my profession.
  • Right up front, the eBook contains a FAQ section. Each question/answer has a couple of paragraphs and a link to the most relevant chapter associated with that question.

I received a free copy of this eBook in order to review it. If I had purchased it at the eBook price, I think it would have been a good value. I was able to read this in an afternoon, and gain from Mr. Portigal’s experience.  As a software engineer, not a professional user researcher, I would have been disappointed though, at the value had I paid full price for the physical book. However, it’s a great eBook value.

Besides the free review copy, I will not receive any additional compensation from this review.

I review for the O'Reilly Blogger Review Program