Worldwide Browser Share
What browsers do your customers use, and which browsers should you be testing? One way to answer this question is to consult the several services that track browser usage, and design your testing strategy to match (for example, here, here, or here). These sites are free and are definitely better than pure guessing. These sites do have some good use, you can see trends over time.
However, a better way is to use analytics within your application, and find out the browser usage for your customer base. The actual usage by your customers will likely vary from the published sites. For example, on my project we currently have 1.8% of our users connecting with IE6. While NetMarketShare shows nearly 17% world wide usage. We have a very different test strategy for a browser with 1.8% usage than one with 17%.
Analytics doesn’t have to be expensive. It may, in fact, be free. Google Analytics is free to use, and provides a lot of data about your customers. Another free source of browser analytics is parsing the log files from your web server, aggregating the User Agent Strings.
Test for your customer, not a generic customer.
I saw a cooking show the other day where the chef was asked his secret for great food. His reply was essentially start with great ingredients and don’t screw them up.
That got me thinking about our software quality and my experiences in the workplace. It seems like a lot of time and effort are spent on managing to our exit criteria. We spend hours negotiating exit criteria, tracking progress of pass-rates, writing waivers, etc.
Thinking instead like the chef, our time may be better spent on the inputs to our processes instead. At least we should focus on the inputs as much as the outputs and current status. At our lessons learned, after the project is over, we do point to inputs (poor requirements, poor code delivered to system test, etc.).
Some ideas to concentrate on inputs:
Typical Exit criteria
- Requirements document written in proper template
- Reviewed and signed off by development, test, quality, etc.
Better to put more emphasis on activities that improve requirements
- Actually talking with customers
- Test our requirements with prototypes, models, etc
- Anticipate change in requirements (build a prioritized backlog and develop in iterations)
Design & Coding
Typical Exit Criteria
- Have the reviews been conducted?
- Do the developers tell us they executed unit tests?
Better to put more emphasis on activities that help make code better
- Using design patterns
- Reusing code
- Are design and code reviews effective? How can we add value to the reviews?
- Pair programming, Test Driven Development
Typical Exit Criteria
- Test coverage
- Pass rate
- Quantity and severity of defects
Better to put more emphasis on activities that help find more and better defects
- Customer feedback
- Negative mode testing (fault injection)
- Exploratory testing
- Building the knowledge of our testers, in technology and domain knowledge
These lists seem obvious, but in the heat of projects, especially with schedule pressure, the temptation is to jettison any activity that doesn’t seem to contribute the exit criteria.