Category Archives: Testing

A story of exploratory testing – 30 bugs

Me: “This week, we ran 250 test cases.  245 passed and 5 failed.”

Flip the slide.

Me: “Here is our bug find/fix graph. This week, we found 30 bugs and fixed 42. This is the first week where fixes outnumbered finds.”

VP: “Wait, go back one slide.  How come you only had 5 tests fail, but found 30 bugs?”

30 bugs?

On this project, we created test cases based on the written requirements, and could show traceability from requirements down to test results.  These test cases were intended to show that we met the customer’s requirements – and the customers had full access to this data.

In addition to the official test cases, we also ran many exploratory tests. The testers spent time experimenting, not following a direct script.  In terms of finding bugs, this exploratory approach was much more productive than following the script.

One reason, the scripts were written from the requirements, the same document that influenced design and code. We should have been surprised if the  prepared test cases found any bugs.  Professional, smart, testers following their nose found many of the issues that would have frustrated customers.

(these events happened a long time ago, on a product where we had an annual release cycle and 3 months of system test – that feels like forever ago)

Manual Tests or Automated Tests? The answer is “yes”.

A line from the new movie Hidden Figures reminds me of an adage that we’ve developed.  When the question is: “Should we do this or that?”  The right answer is usually, “Yes, do this and that.”.

The same is true for automated tests or manual tests.  Unless a project is completely a one time use, throw away, it will almost certainly benefit from developing some tests that are repeatable and executed automatically.  On the flip side, any project that has real humans as users should have real humans making sure it works.

The line from the movie was from John Glenn, “You know you can’t trust something that you can’t look in the eyes”.  He was asking Katherine Johnson to double check the calculations that came from the computer.  The computers did the math faster and with more accuracy than the humans, yet it still takes a smart human to make sure its right.


The Value of Exploration in Software Testing

This is a story about how I came to value Exploratory testing. But first, a story about this story: Every Saturday morning, we go through the same ritual. I have a couple cups of coffee, and Jake, my German Shepherd watches my every move, and follows me everywhere, waiting for me to put on my shoes. He knows we are going on a walk.

Today, on our favorite trail, he started to limp a little. Instead of the full 3.5 mile hike, I turned off a side trail to loop around and cut it short.  This new trail (new to us) was great. We found a small stream and surprised (and were surprised by) a flock of wild turkeys along the ridge line. If we kept to the standard loop, we would not have found this cool area of the hill.

Exploring with Jake and Rose

Exploring with Jake & Rose

While walking, I remembered another value of exploration, in software testing. Our test team had naturally evolved a style, where we would perform targeted and purposeful “ad-hoc” testing on a new feature until we were comfortable enough with the functionality. Then, run the official test cases for “score”. Once we had the score, we would fill in the rest of available time trying different ways to break the system.  These practices worked for us, and I really didn’t think about it until one day presenting status to the executive team.


The status (recreated here to protect the innocent) for this week showed 327 test cases passing with 5 failures. We also opened 30 new bugs, and received 43 bug fixes.  A pretty average week. However, one of the directors asked a question. How could we have only 5 test failures and yet find 30 bugs? His point of view was that our written tests should find all of the bugs, and if we were finding most of the bugs through other means, this pointed to inadequacy of our written tests. I explained how this happens, the written test cases actually find few issues. Most of the bugs are found with the ad-hoc and negative tests, and especially in system with multiple test endpoints (several APIs, several UIs), its prohibitively expensive to script all of the possibilities. This conversation piqued my curiosity, though. Were we behaving optimally? Should we invest more in test case definition?

One thing that I came to realize, our test cases were generated using requirements and design documentation. The developers also used the same requirements and design documents to create the code. Developer testing will generally ensure proper operation, at least for the happy path. So, test cases generated by the test team, from the same original source, tended to have a high pass rate. Problems and bugs with the system had to be found through other means.

After some research, I found that our practices had a name, Exploratory Testing, and ET is used by many organizations in different industries and software types. The exploratory testing approach emphasizes the creative engagement by the tester (as opposed to following a test script), to contemporaneously design and execute tests. The test team was not following the test scripts, but using their experience, creativity, and observations to find new tests to try.

I valued the time we spend on exploratory testing method more than spending more time on written test cases for two reasons. Exploratory testing was far more productive in finding bugs and errors in the code, plus I had more confidence in release readiness based on the testers judgement, rather than the quantitative results of test case execution.

My research into exploratory testing lead to several great resources:

These resources helped our team refine and improve our practices, by learning new techniques and attacks from the authors. Some of these methods are called tours. Our team got better at testing by studying these new tours.

We even found a tool to help manage exploratory testing sessions, called Session-Based Test Management, which could help put a measurement (i.e. test case count) to the testing effort. This measurement could have eased some the questions raised about the quality of our test cases.

All in all, I’m glad that question about the quality of our test cases came up. Our team learned that we were following industry best practices and we learned how to improve our practices.


Model-Based Testing: Code examples

Last week, this post described an example implementation of model-based testing, using a finite state machine to represent a web site, and randomly generated tests. The python code is open source, and available on gitHub.

Contains the class fsm, which implements the finite state machine. The fsm.mach attribute holds a representation of the machine in memory. There are 3 constructors, 2 for testing and 1 operational. The operational constructor takes a filename for the xml model, parses that file to read the model into memory.

The methods are used by the generate module to implement the test strategy. Several methods are used to get various attributes from the machine (the current state, next state, and list of possible transitions from the current state). The fsm.set_next_state() implements the transition.  These methods are likely to be present in any finite state machine class.

One unique method is fsm.get_current_state_oracle(). This method is used by the generate module to create the success criteria of the test.

The generate module creates an instance of fsm, and implements a navigation strategy across the finite state model, and generates the test script.  test_setup() writes the initialization code for a webdriver test program: imports webdriver and creates/initializes the driver object.

random_walk(m, n) takes a state machine, m, and an iteration count, n, as parameters. As the name implies, it walks through the state machine choosing a random transition at each state, and generates a test step that corresponds to the transition.

The basic test is: find the link represented by the next transition, click it, then verify that navigation was successful. The oracle, in this simple example, checks that the page title matches the oracle attribute from the model. is the generated test file. This file is the one that is executed to implement the tests.

Process notes:

You’ll notice the file has extensive unit tests. I used Test Driven Development for this project, where I wrote a test, then wrote the code necessary to pass the test. I found that TDD was an effective way to create the fsm class, implementing only the functionality needed. I could see that if I started with a pure specification of a finite state machine, I might have ended up with more code than required.

I did not use TDD for In retrospect, one glaring difference is the lack of tests. I find that disturbing.

I find your lack of tests disturbing