Tag Archives: Management

Accelerating Product Development through Test Automation

A couple of years ago, Todd Fitch and I collaborated on a paper and presentation for the Pacific Northwest Software Quality Conference.  This paper told the story how we adopted test automation in our organization. We shared our lessons learned, describing the adoption along the lines of process, organization, and areas for investment.

In hindsight, we could have called this:  Test Automation: A Managers Guide.

Here are the highlights:

Organization

Build software engineering skills in your team.

One theme you’ll throughout the paper, automated tests are software; so you’ll be best off treating test automation as a software development project from the start.  If you only have people that know how to test software (i.e. break it), you’ll also want to add the skills to design & build software.

Automation is their only job

I’ve done it several times, ask the automation team to take a few test areas; just to be part of the overall team.  Or, even worse, start automation with a 50/50 split: half of someone’s job is to automate and the other half is to test product.  Guess what; there are 168 hours in a week.  The 50% dedicated to testing will consume 84 of those hours.  When does the automation happen?

Its best to dedicate the automation engineers 100% to automation.

Start with a centralized team

Since automation is their only job, you’d benefit by having all of the automation engineers in a single, centralized, team to start your automation journey.  This will help make it easier to make key decisions around technology, framework, and infrastructure.  There are often several choices for tools and infrastructure, without a clear best choice.  Having all of the people working on the same team can help smooth this decision process.

Once your automated tests have reached critical mass , then it becomes more important to distribute the automation responsibility across the organization.  This helps increase the value of automation by building more and more tests, as well as building quality into your automated tests.  Automation engineers embedded with the feature teams are much more likely to be aware of product changes and adjust the tests accordingly.

Its best to keep some functions centralized, like infrastructure, training, tool license management, and the framework. These are shared resources and a centralized team can balance all needs.

Process

Automated tests are software

Treat your automation project as a software development project.  Make sure you use all the practices for quality software development that you expect your product developers to use. In fact, this is a great chance to set an example.

Build a scalable and robust architecture, with separation of concerns to speed test development and ease of maintenance.  For example, keep test data and test logic separate. Also, its very useful to keep access identifiers like object IDs and XPATH expressions centralized to make maintenance easier.

Build a common function library.  You don’t want every test engineer to implement login as part of their setup.

Professional software developers do a great job at peer review to find defects the easy way. So do test automation engineers.

Test the tests.  Use techniques like automated unit tests, continuous integration, and automated smoke tests to keep your test framework healthy.

Use fail safe design techniques.  Tests that frequently provide false results are worse than useless. Consistently faulty tests will quicky erode the confidence in your tests and the stakeholders will start asking for manual tests to confirm the automated results, regardless of the initial results.  All that work to automate the tests is down the drain, you are back to running manual tests.

Technology

Don’t buy the “record and play back” sales pitch.  Script recording is useful to help discover how to create tests and UI object IDs, but your automation engineers will want to write the test scripts anyway.  The record/playback functions will not create maintainable and extensible tests that use the framework.

Invest in execution infrastructure. You can get away for a while with test automation engineers running the tests from their workstations, or a few dedicated test clients. But eventually, you’ll want to increase the number of times you execute your tests, and vary your clients.  We built our automation lab with physical PCs, but have replaced them with virtual clients.  The virtualization vendors now have excellent solutions.

It’s usually an “and”, not necessarily an “or”

Many times at work, we have discussions and debates whether to do one thing or another. One classic debate is between hiring testers or  quality engineers. Whether its better to perform manual tests or create automated tests. Or, my favorite, whether we need to “follow process” or “deliver on time”.

I remember one of these debates, the topic was testing verses quality engineering.  One one side, we had passionate defenders of testers who worked tirelessly and helped us deliver a much better product than we would have otherwise. On the other side, the thinking was that its much better to prevent problems in the first place, and to automate the testing so our precious time could be spent on more and more productive activities. Who is right?  Well, they both are.

No amount of prevention activities will replace having a real human using the system like our customers will.  The human brain and eyes are the best testing tools ever created, especially when paired with domain knowledge, customer empathy, and the knack of testing.  At the same time, it is much better to prevent problems before they happen. Second best is to find the problems right at the point of injection. Also, repeating the same mundane tasks over and over again is a good use of automation. The classic quality engineering functions.

The answer is almost always “both” when there is a debate over doing one thing or another.  We have to apply quality engineering skills and great testing skills; automation and manual testing; follow “process” and deliver on time.