Here are a couple of funny examples of Customer-Driven Testing. Testing your software in the same conditions that customers will be using your product.
Bump for iPhone allows customers to exchange contact information. This app can be used in the office, instead of exchanging business cards, or may be used in another context. Imagine the late night, after several drinks, exchange of phone numbers.
During user testing in bars, Bump Technologies found that the feature created to selectively enable a variety of contacts (business vs personal, for example), was too complicated for those users with fuzzy-navel driven fuzzy minds.
Another set of examples is provided by Three Sheets Research, a user research company that specializes in app design for for this very condition. Windows 8 after 8 tequilas?
Several of the Customer-Driven Quality practices involve testing customer behavior while the customers are not aware they are part of a test. These practices include:
Meeting with and engaging customers is very valuable, but sometimes you hear what you want to hear. Customers are people too, and sometimes they tell you want you want to hear. This video shows a great example where interviews can sometimes lead to misleading results:
As you see in the video, these people answered questions about an event that never happened. Typical customer engagements, like design sessions, interviews, and usability and beta tests do have a Heisenberg effect. The test affects the output.
Its important to also observe and test with customers, while they don’t realize they are part of the test.
One design concept drilled into my head, back when I worked in the military aerospace industry, was the concept of fail-safe design. This concept requires the system to react in a safe manner, even if it fails. Safety is built-in, by design, into the system.
One example, aircraft hydraulic systems typically have multiple hydraulic pumps to supply pressure. The hydraulic control computer will sense flight conditions and turn on the number of pumps required to maintain proper system pressure, and turn off pumps when not needed to save fuel. The fail safe design has two elements: the main pump is not actually controllable by software. That way, no software failure can possibly result in the main pump being turned off. It’s impossible. Another element of fail-safe design, the secondary pumps are turned off by the control system applying power to the solenoid, rather than turning power on to activate the pump. This means that full hydraulic pressure will be available even if the electrical power is disabled.
Likewise, the landing gear can be deployed by hydraulic pressure, or in case of emergency, by gravity. The systems are designed to fail in a safe manner.
I once worked on a project to see if we could improve the UI indication to fighter pilots when firing a missile. The conversations drifted towards the possibility of automating the entire process. The pilots said that we can do whatever we want in software, but to always leave the trigger decision to them. Having a human in the loop was the fail-safe design principle.