Monthly Archives: August 2016

The Testing Show Podcast – Making QA Strategic

I recently found a new podcast about testing. The Testing Show podcast. This show appears to come out every 2 weeks, and is a panel discussion hosted by Matt Heusser and Michael Larsen.

The show usually starts with a news segment, where the panel discusses some major software bug that happened in the previous couple of weeks. Then, the panel moves onto a topic for that session. The topic that spurred me to write this was “Making QA Strategic”.

At first, the session sounded like it was going to be a complaint about how testers should be taken more seriously. “We have the knowledge, if they would only listen to us”, is too often a refrain amongst the testing community – delivered as a complaint.

However, that sentiment quickly faded and the bulk of the podcast was about testing professionals giving their advice on how to make the testing team more strategic. Here are a few examples:

Josh Assad: “I try to build partnerships with my customers”. He told a story where he traveled to the customer site and spent a week building relationships and figuring how to optimize his testing to fit into the customers acceptance practices.

Jared Small described the value of his team as staying focused on customers and helping the entire team stay focused on the customer.

Jarad also mentioned that a big part of his role is to stay on top of industry trends and what is happening in the software test community.

Matt Heusser described an interesting model, the Swiss cheese model of risk. No one technique will eliminate all risk, in any economic fashion, but combined efforts for automated tests, manual tests, and production monitoring – when taken together will greatly reduce overall risk.   I’m not sure that I can describe how this relates to Swiss cheese, but it sounded good. Maybe I should eat lunch before listening to podcasts.

Matt also described the SWOT (Strengths, Weaknesses, Opportunities, and Threats) model, as applied to doing a gap assessment of the team. Identifying the gaps between your existing team and the ideal is a great way to determine improvement goals.

I also really liked Erik Davis’s approach to innovation. He stresses the importance of trying new things and keeping the ones that work. I believe testing & quality is a field with tons of opportunities for innovation. As I like to say, there is literally an infinite amount of testing that could be done, but a (very) finite amount of time.

Take a listen if you enjoy podcasts. I’ve subscribed and will be browsing their archives. Another cool feature, they have the full transcript on the web.

Well done.

What Pokémon Go Teaches Us About SaaS Quality

By far, the most popular mobile game this year has been Pokémon Go. As you would expect, with a release of this magnitude, there have been some glitches. These issues, and the reaction from the player base, illustrates two core tenants of software quality for SaaS offerings:

  • Availability is vital, the most important thing to get right in SaaS
  • Regression bugs, which remove functionality, are especially painful for customers

Pokémon Go probably doesn’t require an overview, but just in case… Pokémon Go is a mobile, augmented reality game where players travel the real world to find creatures (called Pokémon), resources to help play the game (at Pokéstops), and places to compete with other player’s Pokémon (Gyms). Players are called trainers, and the objective is to train the most powerful Pokémon and to collect all of the different types.

Niantic, the company behind Pokémon Go, has rolled-out Pokémon Go in many countries very quickly. In terms of popularity, the launch was extremely successful. In the US, Pokémon Go was released on July 6th, and by July 11th, the number of daily active users surpassed Twitter. Take another look at those dates: it took only 5 days to go from 0 to Twitter scale.

Availability is the most important feature

They were not prepared for that load and the servers showed it. Customers responded in kind with reviews and very public complaints:

iTunes reviews for Pokemon Go

1 star


Even 5-star reviews complain about availability: 5star

Once the servers seemed stable in the US, Niantic released Pokémon Go in new countries, only to bring the instability back for existing users. As usual in these times, there were even new Twitter accounts created to reflect the frustration:

Pokemon Go Servers Tweet

Players were frustrated because many of the resources in the game have timers associated with them. The player would spend on a resource, only to have the servers go offline.

Pokemon Go splash screen

Bugs are bad, regression bugs even worse

Many players are also complaining about the Nearby Pokémon feature that was available in the first launch. This feature gave players an indication on how close a Pokémon is to their current location, so they could hunt the Pokémon down. This was represented by the number of footprints next to the image, the fewer the footprints, the closer the creature:

The nearby feature shows the number of steps to the closest Pokemon


One of the updates introduced a bug, which caused all of the nearby Pokémon to be shown with 3 footprints, regardless of actual distance:

A bug shows all Pokemon as 3 steps away

In the latest update, Niantic removed the footprints. I’m assuming they followed a solid software principle that “its better to show no information rather than false information”, however the players were livid. Players, rightly, were frustrated that a feature was removed from the app.

Really? You deleted the footprints?

In hindsight, the players would have had a better experience if the footprint tracker was not included in the early launch, and instead added later when all of the glitches were fixed.  This feature was a relatively small part of the game, but removing it intensifies the player’s reaction. Regression bugs that remove functionality are very painful in SaaS offerings.

Overall, Pokémon Go is so far a great success, and my guess is that Niantic will fix these issues. And once they are fixed, they will be forgotten. The players are very passionate about the game and they will continue to play. But, it will take some time for the players to forget, and the app store ratings will remain.

The popularity of Pokémon Go will allow Niantic to survive these glitches, but your app might not have as much of a passionate customer base. For any other app/game, how many of the players would have deleted it and never returned?