Last week, this post described an example implementation of model-based testing, using a finite state machine to represent a web site, and randomly generated tests. The python code is open source, and available on gitHub.

fsm.py

Contains the class fsm, which implements the finite state machine. The fsm.mach attribute holds a representation of the machine in memory. There are 3 constructors, 2 for testing and 1 operational. The operational constructor takes a filename for the xml model, parses that file to read the model into memory.

The methods are used by the generate module to implement the test strategy. Several methods are used to get various attributes from the machine (the current state, next state, and list of possible transitions from the current state). The fsm.set_next_state() implements the transition.  These methods are likely to be present in any finite state machine class.

One unique method is fsm.get_current_state_oracle(). This method is used by the generate module to create the success criteria of the test.

generate.py

The generate module creates an instance of fsm, and implements a navigation strategy across the finite state model, and generates the test script.  test_setup() writes the initialization code for a webdriver test program: imports webdriver and creates/initializes the driver object.

random_walk(m, n) takes a state machine, m, and an iteration count, n, as parameters. As the name implies, it walks through the state machine choosing a random transition at each state, and generates a test step that corresponds to the transition.

The basic test is: find the link represented by the next transition, click it, then verify that navigation was successful. The oracle, in this simple example, checks that the page title matches the oracle attribute from the model.

demo.py

demo.py is the generated test file. This file is the one that is executed to implement the tests.

Process notes:

You’ll notice the fsm.py file has extensive unit tests. I used Test Driven Development for this project, where I wrote a test, then wrote the code necessary to pass the test. I found that TDD was an effective way to create the fsm class, implementing only the functionality needed. I could see that if I started with a pure specification of a finite state machine, I might have ended up with more code than required.

I did not use TDD for generate.py. In retrospect, one glaring difference is the lack of tests. I find that disturbing.

This XML file is a model of my blog, representing the navigation between tags as a finite state machine.  My Model-Based Testing demo code randomly walks through this model and automatically generates the associated WebDriver test code.

One benefit of Model-Based Testing is demonstrated here, we can simply modify the model to expand the test coverage. Adding more states or more transitions will make those available to the test generation algorithm without adding any new code.

The overall process starts with building the model. In my demonstration, I built the model by inspecting the web-page directly. In a new development, its more likely that we would be working with a specification.  Building the model based on the specification, in parallel with the development team building the software is a powerful method of testing the specifications themselves. I call building the model a second “instantiation of understanding”. The specification team & developers will review the model, and highlight mis-understandings early in the life-cycle.

Once the model is built (and reviewed), we use a test selection method to automatically generate tests. In the demo, I used a random walk through the state machine, but there are other strategies that may be more appropriate.  (cringes while remembering languages & automata CS class)

Next, we extract implementation details from the system under test. In this example, the specific link tags provide the means for the automatic test generation algorithm to create the test input (clicking on a link), and the target page title to be used as the success criteria.

Finally, the demo automatically generates the WebDriver code to implement these tests. In the next post on this series, I’ll describe the code in more detail. For those who want to read ahead, the code is available on gitHub.

A brief word on the model: in the model, each page is represented by a “state”, and the links are a “transition”. The state has an “oracle”, which is the page title. The program uses the oracle to verify correct navigation. The state also has a list of transitions.

Each transition has an input attribute, which is the “class name” for that link, which is how WebDriver will find that link. Each transition also has a next state.  You can find the tag cloud on the side-bar of this blog. Right-click on the link and Inspect Element to see the class name.  (tag-link-3 for automation, for example)

Model-Based Testing techniques allow the automatic creation of test cases, creating huge volumes of tests, for practically free.  However, many of these tests are shallow, and provide only a cursory check of your system.

I learned about Model-Based Testing from Harry Robinson, hearing his talk twice., once when he worked at Google and once when he worked at Microsoft.  One presentation demonstrated the use of Model-Based Testing for testing Google Maps routing, the second, Bing Maps.  Go figure.

In those examples, MBT was shown to be an effective method of performing a gross check of map routing, a very difficult application to test, as Apple learned the hard way.

The basic strategy is to retrieve routes between two locations (pair-wise between US Zip code centroid locations).  The routes are obtained automatically, but checking for correctness is the difficult problem. Harry showed several oracles to check the routes for reasonableness.  An oracle is an algorithm or method for determining the success criteria of a test.

Continue reading →

Google hosted GTAC 2013 this past week, with several very interesting presentations.  The videos are currently only available in one long stream. Here are the GTAC videos, along with the time stamp when the particular talk begins.

Day 1

• Time: 00:16:30, Keynote, Ari Shamash, Evolution from QA to Test Engineering
• Time: 1:06:00, James Waldrop, Testing Systems at Scale at Twitter
• Time: 2:20:00, David Burns & Malini Das, How Do You Test a Mobile OS?
• Time: 4:04:15, Igor Dorovskikh & Kaustubh Gawande, Mobile Automation in Continuous Delivery Pipeline
• Time: 4:47:00, David Rothlisberger, Automated Set-Top Box Testing with GStreamer and OpenCV
• Time: 5:03:00, Ken Kania, Webdriver for Chrome
• Time: 5:17:45, Vojta Jina, Karma – Test Runner for JavaScript
• Time: 5:32:50, Patrik Hoglund, Automated Video Quality Measurements
• Time: 5:47:10, Minal Mishra, When Bad Things Happen to Good Applications
• Time: 6:34:00, Tao Xie, Testing for Educational Gaming and Educational Gaming for Testing
• Time: 7:17:20, Simon Stewart, How Facebook Tests Facebook on Android

Continue reading →