Accelerating Product Development through Test Automation

A couple of years ago, Todd Fitch and I collaborated on a paper and presentation for the Pacific Northwest Software Quality Conference.  This paper told the story how we adopted test automation in our organization. We shared our lessons learned, describing the adoption along the lines of process, organization, and areas for investment.

In hindsight, we could have called this:  Test Automation: A Managers Guide.

Here are the highlights:


Build software engineering skills in your team.

One theme you’ll throughout the paper, automated tests are software; so you’ll be best off treating test automation as a software development project from the start.  If you only have people that know how to test software (i.e. break it), you’ll also want to add the skills to design & build software.

Automation is their only job

I’ve done it several times, ask the automation team to take a few test areas; just to be part of the overall team.  Or, even worse, start automation with a 50/50 split: half of someone’s job is to automate and the other half is to test product.  Guess what; there are 168 hours in a week.  The 50% dedicated to testing will consume 84 of those hours.  When does the automation happen?

Its best to dedicate the automation engineers 100% to automation.

Start with a centralized team

Since automation is their only job, you’d benefit by having all of the automation engineers in a single, centralized, team to start your automation journey.  This will help make it easier to make key decisions around technology, framework, and infrastructure.  There are often several choices for tools and infrastructure, without a clear best choice.  Having all of the people working on the same team can help smooth this decision process.

Once your automated tests have reached critical mass , then it becomes more important to distribute the automation responsibility across the organization.  This helps increase the value of automation by building more and more tests, as well as building quality into your automated tests.  Automation engineers embedded with the feature teams are much more likely to be aware of product changes and adjust the tests accordingly.

Its best to keep some functions centralized, like infrastructure, training, tool license management, and the framework. These are shared resources and a centralized team can balance all needs.


Automated tests are software

Treat your automation project as a software development project.  Make sure you use all the practices for quality software development that you expect your product developers to use. In fact, this is a great chance to set an example.

Build a scalable and robust architecture, with separation of concerns to speed test development and ease of maintenance.  For example, keep test data and test logic separate. Also, its very useful to keep access identifiers like object IDs and XPATH expressions centralized to make maintenance easier.

Build a common function library.  You don’t want every test engineer to implement login as part of their setup.

Professional software developers do a great job at peer review to find defects the easy way. So do test automation engineers.

Test the tests.  Use techniques like automated unit tests, continuous integration, and automated smoke tests to keep your test framework healthy.

Use fail safe design techniques.  Tests that frequently provide false results are worse than useless. Consistently faulty tests will quicky erode the confidence in your tests and the stakeholders will start asking for manual tests to confirm the automated results, regardless of the initial results.  All that work to automate the tests is down the drain, you are back to running manual tests.


Don’t buy the “record and play back” sales pitch.  Script recording is useful to help discover how to create tests and UI object IDs, but your automation engineers will want to write the test scripts anyway.  The record/playback functions will not create maintainable and extensible tests that use the framework.

Invest in execution infrastructure. You can get away for a while with test automation engineers running the tests from their workstations, or a few dedicated test clients. But eventually, you’ll want to increase the number of times you execute your tests, and vary your clients.  We built our automation lab with physical PCs, but have replaced them with virtual clients.  The virtualization vendors now have excellent solutions.