Category Archives: Testing

Manual Tests or Automated Tests? The answer is “yes”.

A line from the new movie Hidden Figures reminds me of an adage that we’ve developed.  When the question is: “Should we do this or that?”  The right answer is usually, “Yes, do this and that.”.

The same is true for automated tests or manual tests.  Unless a project is completely a one time use, throw away, it will almost certainly benefit from developing some tests that are repeatable and executed automatically.  On the flip side, any project that has real humans as users should have real humans making sure it works.

The line from the movie was from John Glenn, “You know you can’t trust something that you can’t look in the eyes”.  He was asking Katherine Johnson to double check the calculations that came from the computer.  The computers did the math faster and with more accuracy than the humans, yet it still takes a smart human to make sure its right.

 

The Value of Exploration in Software Testing

This is a story about how I came to value Exploratory testing. But first, a story about this story: Every Saturday morning, we go through the same ritual. I have a couple cups of coffee, and Jake, my German Shepherd watches my every move, and follows me everywhere, waiting for me to put on my shoes. He knows we are going on a walk.

Today, on our favorite trail, he started to limp a little. Instead of the full 3.5 mile hike, I turned off a side trail to loop around and cut it short.  This new trail (new to us) was great. We found a small stream and surprised (and were surprised by) a flock of wild turkeys along the ridge line. If we kept to the standard loop, we would not have found this cool area of the hill.

Exploring with Jake and Rose

Exploring with Jake & Rose

While walking, I remembered another value of exploration, in software testing. Our test team had naturally evolved a style, where we would perform targeted and purposeful “ad-hoc” testing on a new feature until we were comfortable enough with the functionality. Then, run the official test cases for “score”. Once we had the score, we would fill in the rest of available time trying different ways to break the system.  These practices worked for us, and I really didn’t think about it until one day presenting status to the executive team.

30Bugs5TestFailures

The status (recreated here to protect the innocent) for this week showed 327 test cases passing with 5 failures. We also opened 30 new bugs, and received 43 bug fixes.  A pretty average week. However, one of the directors asked a question. How could we have only 5 test failures and yet find 30 bugs? His point of view was that our written tests should find all of the bugs, and if we were finding most of the bugs through other means, this pointed to inadequacy of our written tests. I explained how this happens, the written test cases actually find few issues. Most of the bugs are found with the ad-hoc and negative tests, and especially in system with multiple test endpoints (several APIs, several UIs), its prohibitively expensive to script all of the possibilities. This conversation piqued my curiosity, though. Were we behaving optimally? Should we invest more in test case definition?

One thing that I came to realize, our test cases were generated using requirements and design documentation. The developers also used the same requirements and design documents to create the code. Developer testing will generally ensure proper operation, at least for the happy path. So, test cases generated by the test team, from the same original source, tended to have a high pass rate. Problems and bugs with the system had to be found through other means.

After some research, I found that our practices had a name, Exploratory Testing, and ET is used by many organizations in different industries and software types. The exploratory testing approach emphasizes the creative engagement by the tester (as opposed to following a test script), to contemporaneously design and execute tests. The test team was not following the test scripts, but using their experience, creativity, and observations to find new tests to try.

I valued the time we spend on exploratory testing method more than spending more time on written test cases for two reasons. Exploratory testing was far more productive in finding bugs and errors in the code, plus I had more confidence in release readiness based on the testers judgement, rather than the quantitative results of test case execution.

My research into exploratory testing lead to several great resources:

These resources helped our team refine and improve our practices, by learning new techniques and attacks from the authors. Some of these methods are called tours. Our team got better at testing by studying these new tours.

We even found a tool to help manage exploratory testing sessions, called Session-Based Test Management, which could help put a measurement (i.e. test case count) to the testing effort. This measurement could have eased some the questions raised about the quality of our test cases.

All in all, I’m glad that question about the quality of our test cases came up. Our team learned that we were following industry best practices and we learned how to improve our practices.

 

Model-Based Testing: Code examples

Last week, this post described an example implementation of model-based testing, using a finite state machine to represent a web site, and randomly generated tests. The python code is open source, and available on gitHub.

fsm.py

Contains the class fsm, which implements the finite state machine. The fsm.mach attribute holds a representation of the machine in memory. There are 3 constructors, 2 for testing and 1 operational. The operational constructor takes a filename for the xml model, parses that file to read the model into memory.

The methods are used by the generate module to implement the test strategy. Several methods are used to get various attributes from the machine (the current state, next state, and list of possible transitions from the current state). The fsm.set_next_state() implements the transition.  These methods are likely to be present in any finite state machine class.

One unique method is fsm.get_current_state_oracle(). This method is used by the generate module to create the success criteria of the test.

generate.py

The generate module creates an instance of fsm, and implements a navigation strategy across the finite state model, and generates the test script.  test_setup() writes the initialization code for a webdriver test program: imports webdriver and creates/initializes the driver object.

random_walk(m, n) takes a state machine, m, and an iteration count, n, as parameters. As the name implies, it walks through the state machine choosing a random transition at each state, and generates a test step that corresponds to the transition.

The basic test is: find the link represented by the next transition, click it, then verify that navigation was successful. The oracle, in this simple example, checks that the page title matches the oracle attribute from the model.

demo.py

demo.py is the generated test file. This file is the one that is executed to implement the tests.

Process notes:

You’ll notice the fsm.py file has extensive unit tests. I used Test Driven Development for this project, where I wrote a test, then wrote the code necessary to pass the test. I found that TDD was an effective way to create the fsm class, implementing only the functionality needed. I could see that if I started with a pure specification of a finite state machine, I might have ended up with more code than required.

I did not use TDD for generate.py. In retrospect, one glaring difference is the lack of tests. I find that disturbing.

I find your lack of tests disturbing

 

 

 

 

 

 

 

Model-Based Testing Example: Finite State Machine

This XML file is a model of my blog, representing the navigation between tags as a finite state machine.  My Model-Based Testing demo code randomly walks through this model and automatically generates the associated WebDriver test code.

blogModel

One benefit of Model-Based Testing is demonstrated here, we can simply modify the model to expand the test coverage. Adding more states or more transitions will make those available to the test generation algorithm without adding any new code.

The overall process starts with building the model. In my demonstration, I built the model by inspecting the web-page directly. In a new development, its more likely that we would be working with a specification.  Building the model based on the specification, in parallel with the development team building the software is a powerful method of testing the specifications themselves. I call building the model a second “instantiation of understanding”. The specification team & developers will review the model, and highlight mis-understandings early in the life-cycle.

FMSProcess

 

Once the model is built (and reviewed), we use a test selection method to automatically generate tests. In the demo, I used a random walk through the state machine, but there are other strategies that may be more appropriate.  (cringes while remembering languages & automata CS class)

Next, we extract implementation details from the system under test. In this example, the specific link tags provide the means for the automatic test generation algorithm to create the test input (clicking on a link), and the target page title to be used as the success criteria.

Finally, the demo automatically generates the WebDriver code to implement these tests. In the next post on this series, I’ll describe the code in more detail. For those who want to read ahead, the code is available on gitHub.

A brief word on the model: in the model, each page is represented by a “state”, and the links are a “transition”. The state has an “oracle”, which is the page title. The program uses the oracle to verify correct navigation. The state also has a list of transitions.

Each transition has an input attribute, which is the “class name” for that link, which is how WebDriver will find that link. Each transition also has a next state.  You can find the tag cloud on the side-bar of this blog. Right-click on the link and Inspect Element to see the class name.  (tag-link-3 for automation, for example)