Several of the Customer-Driven Quality practices involve testing customer behavior while the customers are not aware they are part of a test. These practices include:
- A/B Testing
- Dry Testing
- Usage Analytics
- Behavior Tracking
- Staged Deployments
Meeting with and engaging customers is very valuable, but sometimes you hear what you want to hear. Customers are people too, and sometimes they tell you want you want to hear. This video shows a great example where interviews can sometimes lead to misleading results:
As you see in the video, these people answered questions about an event that never happened. Typical customer engagements, like design sessions, interviews, and usability and beta tests do have a Heisenberg effect. The test affects the output.
Its important to also observe and test with customers, while they don’t realize they are part of the test.
While these are effective techniques from the point of view of the developer, you have to be careful about your relationship with the buyer.
The fact that it furthers your needs doesn’t mean you are free to track people without their permission in many jurisdictions. And user experiences that vary unpredictably can lead to very unhappy users.
At least disclose somewhere without your faqs and/or privacy policy that you may use these techniques.
And of course some software contracts would prohibit some of these behaviors completely.
Tread lightly. Ask permission.
Hi Joe, thanks for your comment. You make some good points. There are obviously contexts & test types where “testing without customer knowledge” doesn’t work. These techniques are definitely for learning about customer behavior, in a way that is safe, legal, and ethical. I’ve done this type of testing on web applications, in a way that conforms to the terms of service. And, in no way tracking personally identifiable information.
I also used to work on software that controls the hydraulic system on military cargo planes. In that context, no way would we install test software without telling the pilots. However, even in that world, the “over the shoulder” camera video was analyzed to understand pilot behavior.
The important principle here is when working with customers and learning about behavior, its important to try to eliminate any source of bias created by test conditions. These techniques also complement all of the other testing that we do. These techniques do not replace other tests.