What browsers do your customers use, and which browsers should you be testing? One way to answer this question is to consult the several services that track browser usage, and design your testing strategy to match (for example, here, here, or here). These sites are free and are definitely better than pure guessing. These sites do have some good use, you can see trends over time.
However, a better way is to use analytics within your application, and find out the browser usage for your customer base. The actual usage by your customers will likely vary from the published sites. For example, on my project we currently have 1.8% of our users connecting with IE6. While NetMarketShare shows nearly 17% world wide usage. We have a very different test strategy for a browser with 1.8% usage than one with 17%.
Analytics doesn’t have to be expensive. It may, in fact, be free. Google Analytics is free to use, and provides a lot of data about your customers. Another free source of browser analytics is parsing the log files from your web server, aggregating the User Agent Strings.
In school, marketing class. Concept of marketing myopia, where companies focus too much on the product they are currently producing. This near-sightedness keeps businesses from adapting to changing technology and market conditions. The classic example is the buggy whip manufacturing company.
The invention of the automobile is a threat to buggy whip producers, because the automobile technology will eliminate the need of buggy whips. If, instead, they see themselves as “transportation initiation” companies, then the automobile is an opportunity to serve a new set of customers, those who don’t want to hand crank their engines.
The idea behind marketing myopia is to take a step back, take a longer view of your product. A good way to do this is to think about what service you provide, rather than what product you produce.
What does this have to do with software testing and quality leadership? Well, sometimes we get focused on our tools and processes. These tools of the trade are an important part of our craft, but the larger organization is not really looking for great automation suites, exploratory tests, or exit criteria. We talk about bug counts and severities. We talk about test pass/fail rates. We talk about code coverage.
Our product is testing, test results, and bug reports.
But, what service do we provide? Why does the organization “purchase” testing?
Our organizations hire us to provide a service, that service is confidence. Confidence that our products will meet or exceed customer expectations. Confidence that our systems will be secure, available when needed, and perform as advertised. Confidence that when its time to ship that the product is ready to ship.
How does taking a different look at our role influence what we do? What do we do differently if we are in the confidence business rather than the testing business?
My first job was working in the aerospace industry, working for McDonnell Douglas (which is now part of Boeing). One tenet drilled into me during my tenure building military aircraft was the concept of “fail safe“. One example of a fail safe design is the control switch for the C-17 hydraulic pumps. The primary pump is directly connected to the engine, and is on by default. If the pilot wants to turn the pump off, he or she presses the button which applies power to the solenoid to disable the pump. Since hydraulic pressure is required to fly the plane, turning the pump off is an unusual condition, so active power is necessary to disable it. If any of the components in the chain fail, the pump stays on, which is the safest condition for the aircraft.
One question that this experience leads me to ask, during a design review is “how is this design fail safe”. But, this video reminded me of systems that are inherently in-stable and got me to thinking about fail safe processes. Which processes at work would go out of control without personal intervention?
The F/A-18 airframe is inherently instable. The flight control system keeps it flying straight. Obviously, in this example, multiple systems failed and thank goodness the pilot ejected safely. However, this gives an example of an inherently instable process that is held in place by a flight control system. The flight control system fails, the aircraft crashes.
At work, our bug management process is inherently instable. If the individuals on the change control board don’t play their role, then generally bugs will not be fixed. This is something that I should rethink. Also, what other processes are not fail safe?
The life of a cowboy would be perfect, if it weren’t for the cows.
Successful entrepreneurs know that its vital to talk to actual customers. The business plan may be solid, the product incredible, but without customers, the business is nowhere.
If that advice is sound for a small business, then why does it frequently break down in large businesses? Talking with actual customers is not the job of the software development team, but instead the role of the marketing, sales, and support teams. Job specialization has been the hallmark of big business since Henry Ford.
Here is a little secret, large businesses are really collections of small businesses. Senior management at large businesses look at each department as individual smaller businesses. They will construct a P&L (Profit & Loss statement) for each department, and praise those with earnings that improve the overall bottom line.
So, many of us work in larger businesses, but in an individual product line. Think of your product as a small business, one where funding comes from your success as a product.
What does this have to do with Software Quality? Well, as a software quality professional that follows the Context Driven School of thought, we believe that practices should be applied in the given context. Many practices that are common in small businesses seem less important in large businesses. Specifically, interacting directly with customers.
In a large business, interacting directly with customers is someone else’s job. The sales or support team are on the front lines, protecting the dev/test team from having to spend time with customers. Small businesses don’t have the luxury of separate departments, so the development & quality team get the benefit of interacting directly with customers.
For those of you in large businesses, what would you do differently? What are the artificial constraints that prevent you from talking directly to customers?
What does talking with customers have to do with cows and cowboys? I’ve frequently seen the similar attitude in software development & quality, where we really enjoy our jobs and practices, but find that working with customers can become a real inconvenience. Customers want changes that mess up our great specifications. Customers don’t like the cool feature we just developed. Customers found a bug by doing something they shouldn’t have been doing.
However, we need to remind ourselves that the customer is always right, and its all about how the customer defines quality.
A couple of years ago, Todd Fitch and I collaborated on a paper and presentation for the Pacific Northwest Software Quality Conference. This paper told the story how we adopted test automation in our organization. We shared our lessons learned, describing the adoption along the lines of process, organization, and areas for investment.
In hindsight, we could have called this: Test Automation: A Managers Guide.
Here are the highlights:
Organization
Build software engineering skills in your team.
One theme you’ll throughout the paper, automated tests are software; so you’ll be best off treating test automation as a software development project from the start. If you only have people that know how to test software (i.e. break it), you’ll also want to add the skills to design & build software.
Automation is their only job
I’ve done it several times, ask the automation team to take a few test areas; just to be part of the overall team. Or, even worse, start automation with a 50/50 split: half of someone’s job is to automate and the other half is to test product. Guess what; there are 168 hours in a week. The 50% dedicated to testing will consume 84 of those hours. When does the automation happen?
Its best to dedicate the automation engineers 100% to automation.
Start with a centralized team
Since automation is their only job, you’d benefit by having all of the automation engineers in a single, centralized, team to start your automation journey. This will help make it easier to make key decisions around technology, framework, and infrastructure. There are often several choices for tools and infrastructure, without a clear best choice. Having all of the people working on the same team can help smooth this decision process.
Once your automated tests have reached critical mass , then it becomes more important to distribute the automation responsibility across the organization. This helps increase the value of automation by building more and more tests, as well as building quality into your automated tests. Automation engineers embedded with the feature teams are much more likely to be aware of product changes and adjust the tests accordingly.
Its best to keep some functions centralized, like infrastructure, training, tool license management, and the framework. These are shared resources and a centralized team can balance all needs.
Process
Automated tests are software
Treat your automation project as a software development project. Make sure you use all the practices for quality software development that you expect your product developers to use. In fact, this is a great chance to set an example.
Build a scalable and robust architecture, with separation of concerns to speed test development and ease of maintenance. For example, keep test data and test logic separate. Also, its very useful to keep access identifiers like object IDs and XPATH expressions centralized to make maintenance easier.
Build a common function library. You don’t want every test engineer to implement login as part of their setup.
Professional software developers do a great job at peer review to find defects the easy way. So do test automation engineers.
Test the tests. Use techniques like automated unit tests, continuous integration, and automated smoke tests to keep your test framework healthy.
Use fail safe design techniques. Tests that frequently provide false results are worse than useless. Consistently faulty tests will quicky erode the confidence in your tests and the stakeholders will start asking for manual tests to confirm the automated results, regardless of the initial results. All that work to automate the tests is down the drain, you are back to running manual tests.
Technology
Don’t buy the “record and play back” sales pitch. Script recording is useful to help discover how to create tests and UI object IDs, but your automation engineers will want to write the test scripts anyway. The record/playback functions will not create maintainable and extensible tests that use the framework.
Invest in execution infrastructure. You can get away for a while with test automation engineers running the tests from their workstations, or a few dedicated test clients. But eventually, you’ll want to increase the number of times you execute your tests, and vary your clients. We built our automation lab with physical PCs, but have replaced them with virtual clients. The virtualization vendors now have excellent solutions.
Let’s face it, test data can be some of the dullest reading around. Communicating test results with impact can be difficult. The simplest way is to add color. The Red/Yellow/Green color palette works particularly well.
When you are one of the richest companies in the world, you can afford to do just a wee bit more. Here is an outstanding example of a performance report by our friends at Google.
Many times at work, we have discussions and debates whether to do one thing or another. One classic debate is between hiring testers or quality engineers. Whether its better to perform manual tests or create automated tests. Or, my favorite, whether we need to “follow process” or “deliver on time”.
I remember one of these debates, the topic was testing verses quality engineering. One one side, we had passionate defenders of testers who worked tirelessly and helped us deliver a much better product than we would have otherwise. On the other side, the thinking was that its much better to prevent problems in the first place, and to automate the testing so our precious time could be spent on more and more productive activities. Who is right? Well, they both are.
No amount of prevention activities will replace having a real human using the system like our customers will. The human brain and eyes are the best testing tools ever created, especially when paired with domain knowledge, customer empathy, and the knack of testing. At the same time, it is much better to prevent problems before they happen. Second best is to find the problems right at the point of injection. Also, repeating the same mundane tasks over and over again is a good use of automation. The classic quality engineering functions.
The answer is almost always “both” when there is a debate over doing one thing or another. We have to apply quality engineering skills and great testing skills; automation and manual testing; follow “process” and deliver on time.
It’s been a long time since I’ve contributed to this blog. My initial plan was to blog about fly fishing and software quality. I ended up getting more into the fly fishing side, so I just went with it on my other site, Whiskey Creek Flies.
I’ve been writing about software quality, but in blogs and email distribution lists at work. Now it’s time to revive this side of my blogging life. So, drop in occasionally, I’ll be writing about software quality and testing from the perspective that I’ve built over the past few years.
I saw a cooking show the other day where the chef was asked his secret for great food. His reply was essentially start with great ingredients and don’t screw them up.
That got me thinking about our software quality and my experiences in the workplace. It seems like a lot of time and effort are spent on managing to our exit criteria. We spend hours negotiating exit criteria, tracking progress of pass-rates, writing waivers, etc.
Thinking instead like the chef, our time may be better spent on the inputs to our processes instead. At least we should focus on the inputs as much as the outputs and current status. At our lessons learned, after the project is over, we do point to inputs (poor requirements, poor code delivered to system test, etc.).
Some ideas to concentrate on inputs:
Requirements Typical Exit criteria
Requirements document written in proper template
Reviewed and signed off by development, test, quality, etc.
Better to put more emphasis on activities that improve requirements
Actually talking with customers
Test our requirements with prototypes, models, etc
Anticipate change in requirements (build a prioritized backlog and develop in iterations)
Design & Coding Typical Exit Criteria
Have the reviews been conducted?
Do the developers tell us they executed unit tests?
Better to put more emphasis on activities that help make code better
Using design patterns
Reusing code
Are design and code reviews effective? How can we add value to the reviews?
Pair programming, Test Driven Development
Test Typical Exit Criteria
Test coverage
Pass rate
Quantity and severity of defects
Better to put more emphasis on activities that help find more and better defects
Customer feedback
Negative mode testing (fault injection)
Exploratory testing
Building the knowledge of our testers, in technology and domain knowledge
These lists seem obvious, but in the heat of projects, especially with schedule pressure, the temptation is to jettison any activity that doesn’t seem to contribute the exit criteria.
Being a test manager, I’ve been in the position a few times to explain why the test team did not catch a problem that our customers found. Early in my career, I would tend to defend the test team or test plan. However, that is not a very effective response to the situation. At the end of the day, we should be focused on delivering the highest quality products for our customers. Escapes happen, and we should use each as a learning opportunity.
In examining the escapes, I’ve rarely found a case where the answer was very simple. Its usually a “perfect storm“. A perfect storm is where multiple factors pile up to cause a critical problem. Its useful to spend a little effort in examining the root causes
To help guide the investigation, I’ve developed the following set of questions which help guide a comprehensive root cause. These questions help guide the review in more productive direction than finger pointing.
Describe Problem: (QA)
Describe circumstances and consequence of the problem.This description should be a summary of the information, with enough description for readers not familiar with the issue.Include a reference to CR number, or other tracking information.
Where was the problem introduced? (dev)
Describe the phase in which this was introduced, i.e. requirements, design, code, build, etc.
Describe the root cause of the problem: (dev)
For the phase where the problem was introduced, describe what actually happened. For example, the requirements could be missing, incorrect, unclear. Designs might not have provided for error handling, or consider the performance requirements. Coding errors may be logic, missing table entry, incorrect logic (“and” instead of “or”), etc.
The root cause is often difficult to get to. One tool to use is the “5-whys” approach. This is where you ask the question “why” 5 times or until you get to the root cause.
What reviews were held for the phase? (dev)
For example, requirements review for missing/incorrect requirements.Code reviews for coding errors.
How and why did the problem escape testing? (QA)
Examples, test case doesn’t exist, test was not run, defect introduced after test passed, etc. Also, describe why (for example, why didn’t a test exist, etc.)
What could be done to prevent this type of error in the future? (team)
What process improvement could have prevented this problem from occurring?
What could be done to find this type of problem in the future? (team)
What test or review should be implemented?Examples are adding a test case, implementing a new automated test, adding new exit criteria to peer review, etc.
Answering these questions does take some effort. If you find your self in the unfortunate position of having too many escapes to perform the full analysis, choose a subset driven by the severity of the problems. Start somewhere.
One customer was very helpful, finding many of our defects for us. To get some control of the situation, I performed the root cause analysis on all critical bugs. In a few months time, we had just a few Critical bugs that I extended the analysis to Major bugs. That customer gave us a letter grade (like in school). We went from an F to an A, in part because of this analysis.
My name is John Ruberto. I'm the Quality Leader for QuickBooks Online, which is the best web offering to help small business owners manage their money.
This is my professional blog, where I'll be sharing the lessons that I've learned over my 25 years of experience in software development.
I also enjoy fly fishing and fly tying. I have a blog for that too. It's called Whiskey Creek Fly Fishing.
