Model-Based Testing techniques allow the automatic creation of test cases, creating huge volumes of tests, for practically free. However, many of these tests are shallow, and provide only a cursory check of your system.
I learned about Model-Based Testing from Harry Robinson, hearing his talk twice., once when he worked at Google and once when he worked at Microsoft. One presentation demonstrated the use of Model-Based Testing for testing Google Maps routing, the second, Bing Maps. Go figure.
In those examples, MBT was shown to be an effective method of performing a gross check of map routing, a very difficult application to test, as Apple learned the hard way.
The basic strategy is to retrieve routes between two locations (pair-wise between US Zip code centroid locations). The routes are obtained automatically, but checking for correctness is the difficult problem. Harry showed several oracles to check the routes for reasonableness. An oracle is an algorithm or method for determining the success criteria of a test.
One oracle, the route and the reverse route should be approximately the same length and drive time in both directions. A 10% buffer accounts for slightly different routes because of one-way streets. Another oracle, the route between two points is compared to the straight-line distance, determined by the latitude and longitude.
In Harry’s examples, MBT provided a way to automate the test script generation, both the route generation & calculation of success criteria, which allowed checking millions of route possibilities. His presentation shows several interesting bugs that were found using this method.
His presentation inspired me to experiment a bit with model based testing. I created a basic model of a web site, to navigate to various pages (perhaps as part of a load generation for a stress test). The model uses a Finite State Machine to represent states (each state is a web page), and the attributes include possible state transitions and an oracle to verify correct navigation.
My code is on gitHub, as open source. It uses an XML representation of the FSM, a Python class that I wrote to navigate the FSM, and generates Selenium WebDriver test code to implement the tests. My next post will describe the demo in more detail.