Category Archives: Testing

Testing Pi day, which is more accurate?

In the US, today is Pi day.  March 14, or as we write it with numbers 3.14.  Happy Pi day everyone!

However, in Europe, they tend to write out dates starting with the day, then the month.  So, today is 14.3 over there.  No-where near Pi.  Instead, the closest to Pi day in Europe would be 22/7 (July 22nd), where 22/7 is a common approximation of Pi.

Which is more accurate?

Testing both approximations is pretty easy with Wolfram Alpha. The error in the approximation is determined by taking the absolute value of the difference between Pi and the approximation. So, the following screen shows the result, asking if the error in the US version is greater than the European version of Pi day:

Comparing the US version of Pi day (3.14) to the European version (22/7) with Wolfram Alpha

Comparing the US version of Pi day (3.14) to the European version (22/7) with Wolfram Alpha

Europe wins this time. 22/7 is a better approximation than 3.14.

A story of exploratory testing – 30 bugs

Me: “This week, we ran 250 test cases.  245 passed and 5 failed.”

Flip the slide.

Me: “Here is our bug find/fix graph. This week, we found 30 bugs and fixed 42. This is the first week where fixes outnumbered finds.”

VP: “Wait, go back one slide.  How come you only had 5 tests fail, but found 30 bugs?”

30 bugs?

On this project, we created test cases based on the written requirements, and could show traceability from requirements down to test results.  These test cases were intended to show that we met the customer’s requirements – and the customers had full access to this data.

In addition to the official test cases, we also ran many exploratory tests. The testers spent time experimenting, not following a direct script.  In terms of finding bugs, this exploratory approach was much more productive than following the script.

One reason, the scripts were written from the requirements, the same document that influenced design and code. We should have been surprised if the  prepared test cases found any bugs.  Professional, smart, testers following their nose found many of the issues that would have frustrated customers.

(these events happened a long time ago, on a product where we had an annual release cycle and 3 months of system test – that feels like forever ago)

Manual Tests or Automated Tests? The answer is “yes”.

A line from the new movie Hidden Figures reminds me of an adage that we’ve developed.  When the question is: “Should we do this or that?”  The right answer is usually, “Yes, do this and that.”.

The same is true for automated tests or manual tests.  Unless a project is completely a one time use, throw away, it will almost certainly benefit from developing some tests that are repeatable and executed automatically.  On the flip side, any project that has real humans as users should have real humans making sure it works.

The line from the movie was from John Glenn, “You know you can’t trust something that you can’t look in the eyes”.  He was asking Katherine Johnson to double check the calculations that came from the computer.  The computers did the math faster and with more accuracy than the humans, yet it still takes a smart human to make sure its right.


The Value of Exploration in Software Testing

This is a story about how I came to value Exploratory testing. But first, a story about this story: Every Saturday morning, we go through the same ritual. I have a couple cups of coffee, and Jake, my German Shepherd watches my every move, and follows me everywhere, waiting for me to put on my shoes. He knows we are going on a walk.

Today, on our favorite trail, he started to limp a little. Instead of the full 3.5 mile hike, I turned off a side trail to loop around and cut it short.  This new trail (new to us) was great. We found a small stream and surprised (and were surprised by) a flock of wild turkeys along the ridge line. If we kept to the standard loop, we would not have found this cool area of the hill.

Exploring with Jake and Rose

Exploring with Jake & Rose

While walking, I remembered another value of exploration, in software testing. Our test team had naturally evolved a style, where we would perform targeted and purposeful “ad-hoc” testing on a new feature until we were comfortable enough with the functionality. Then, run the official test cases for “score”. Once we had the score, we would fill in the rest of available time trying different ways to break the system.  These practices worked for us, and I really didn’t think about it until one day presenting status to the executive team.


The status (recreated here to protect the innocent) for this week showed 327 test cases passing with 5 failures. We also opened 30 new bugs, and received 43 bug fixes.  A pretty average week. However, one of the directors asked a question. How could we have only 5 test failures and yet find 30 bugs? His point of view was that our written tests should find all of the bugs, and if we were finding most of the bugs through other means, this pointed to inadequacy of our written tests. I explained how this happens, the written test cases actually find few issues. Most of the bugs are found with the ad-hoc and negative tests, and especially in system with multiple test endpoints (several APIs, several UIs), its prohibitively expensive to script all of the possibilities. This conversation piqued my curiosity, though. Were we behaving optimally? Should we invest more in test case definition?

One thing that I came to realize, our test cases were generated using requirements and design documentation. The developers also used the same requirements and design documents to create the code. Developer testing will generally ensure proper operation, at least for the happy path. So, test cases generated by the test team, from the same original source, tended to have a high pass rate. Problems and bugs with the system had to be found through other means.

After some research, I found that our practices had a name, Exploratory Testing, and ET is used by many organizations in different industries and software types. The exploratory testing approach emphasizes the creative engagement by the tester (as opposed to following a test script), to contemporaneously design and execute tests. The test team was not following the test scripts, but using their experience, creativity, and observations to find new tests to try.

I valued the time we spend on exploratory testing method more than spending more time on written test cases for two reasons. Exploratory testing was far more productive in finding bugs and errors in the code, plus I had more confidence in release readiness based on the testers judgement, rather than the quantitative results of test case execution.

My research into exploratory testing lead to several great resources:

These resources helped our team refine and improve our practices, by learning new techniques and attacks from the authors. Some of these methods are called tours. Our team got better at testing by studying these new tours.

We even found a tool to help manage exploratory testing sessions, called Session-Based Test Management, which could help put a measurement (i.e. test case count) to the testing effort. This measurement could have eased some the questions raised about the quality of our test cases.

All in all, I’m glad that question about the quality of our test cases came up. Our team learned that we were following industry best practices and we learned how to improve our practices.