Monthly Archives: July 2013

Build in Quality for Customers

Once the product is defined, the development team designs and codes the software. This is called the build phase of Customer-Driven Quality. In this phase, the development team can use practices like A/B testing to test alternate designs.

Build phase of software development life-cycle

Testing Alternate Designs

Making design choices often face the same issues as in the product definition phase; the choice is often influenced heavily based on the opinion of the most authoritative, or influential, person in the decision.  Instead of relying on one person’s opinion, customers can vote with their behavior using A/B testing.

In an A/B test, two designs (or more) are presented to customers and a tracking metric is used to test which design better helps the customer succeed with the desired behavior.  For example, Google famously tested 41 different shades of the color blue on web links to find the best color. In this case, the best color was chosen by how frequently customers clicked on an advertisement link.

Personalized Development

We have several methods for developers to interact directly with customers during the implementation phase. These boil down to putting a name, face, and engagement with actual customers to build the right product in the right way for our customers.

The Adopt a Customer approach has a developer choosing a small group of customers. The developers then build the feature collaborating directly with the customers. The developers have access to customers to ask questions, give frequent informal demonstrations, and brainstorm ideas. This process provides focused problem solving, rather than guessing intent from the requirements (or asking HIPPOs).

We maintain a list of customers willing to provide frequent interactions, through the Inner Circle program. Customers opt in to this program, and are available for consultations.

Default Behavior

Behavioral economics, and web analytics, have shown that when customers are presented a choice, they quite often choose the default selection.  This behavior should be considered when making design decisions. For example, a user interface design where the customer should choose from multiple options (State, for example), should not pre-select a default, especially if the list if alphabetical. I’ve seen customer data where Alaska has an unusual high number of customers. This is because, I believe, Alaska is the default selection for customers who just click through that screen.

Reviewing the default options is a specific line item in the design review checklist.

Build with Customer’s Platform

Many self-inflicted errors come from cross platform development. To the degree possible, this should be avoided. The product should be built on the platform used by customers.

In the current state of web application development, most developers prefer to develop with Chrome, because of the superior development and debugging plug-ins. These plug-ins were developed by developers, for developers (talk about Customer-Driven Quality). However, most customers use Internet Explorer as their browsers. Ideally, the web UI should be developed with Internet Explorer, or at least tested by the developers before they check in.

Continue to Customer-Driven Testing or return to the Customer-Driven Quality page for more practices.

Product Definition for Software Quality

In the Define stage of Customer-Driven Quality, your team identifies what should be built, which seems to be a natural fit for customer-driven practices. The marketing, product management, or product owners usually lead this phase; each of these roles are dedicated to understand customer wants and needs. So, where does the test/quality team play a role?

Product Definition Phase of Software Development

The test/quality team can help the product definition team in two key areas:

  • Make meaningful and mindful investments in three categories: growth of the business, satisfying current customers, and investing in technology for the long term.
  • Test the requirements with customers, before expending the time and effort to build the product.

Investment Level for Customer Quality

One of the most powerful decisions that can be made during the definition phase is deciding how much of the limited development bandwidth to apply towards three broad categories: product infrastructure or technology, satisfying current customers, and building new value for current or future customers.

These questions are not easy to make, but each category must be considered. Often, the product definition phase is dominated by building value for new customers, but product infrastructure and current customer pain points are important as well.

The product infrastructure must be maintained and improved to prevent future pain points, such as performance or availability issues. The quality team can make a difference by measuring the technical debt and communicating with all of the stakeholders, including the product definition team.

It’s important to communicate with business stakeholders in business language. They might not understand or appreciate the lack of test automation coverage, but will appreciate the opportunities to reduce the development cycle-time. Learn to speak the language of your stakeholders.

Measuring the customer feedback mainly drives solving current customer pain points. Satisfaction surveys, Net-Promoter scores, and Voice of the Customer are the measures, and we should dedicate some effort to satisfy current customers (who are likely to be influencing future purchase decisions).

A few ways of managing this prioritizing the current customer activities include:

  • Applying a “tax” of development time before considering new features; keeping some development resources in reserve so they can concentrate on satisfying current customers.
  • Managing requirements and features in a common prioritized backlog along with infrastructure and customer issue. Having a common repository will help the team make these tradeoffs.
  • Periodically, a “customer love” or “net promoter” release can be useful, where the entire focus is just solving pain points.

Testing requirements before building product

Customer-Driven Quality does have a natural predator, the HIPPO. Not the aquatic mammal from Africa, but the “Highly Paid Person’s Opinion”. The HIPPO is a term we use to remind ourselves that it’s the customer’s opinion that counts, not the boss’s.

Hippopotamus the animal. Hippo = Highly Paid Person's Opinion

Brainstorming is also used to bring out the best ideas from the whole team. Too often, the results of the brainstorming sessions are not the best ideas, but driven from the most charismatic participants.

Instead of building the feature set that is desired by the influential people in the organization, the customers could help decide what we build. One set of practices that features customer learning comes from the Lean Startup arena.

The Lean Startup process can be thought of as a product management process, and being tangentially related to “quality.” However, a better way to think about it: testing the requirements before the product is built. This is where the test team can play a strong role.

If you think about the idea for a new product or a new feature as a hypothesis, you can use the scientific method to validate the hypothesis through testing, and using Lean Startup terminology, the testing results in Validated Learning.

Validated Learning is all about achieving results from customer behavior, not just what they say they would do. Too often, when talking with a customer (or potential customers), and asking if they would value a feature, the answer is “yes”. Especially in face-to-face encounters, the customers (who are nice people) agree with the idea. Early surveys show tremendous interest, but eventual sales fail to get even close to those projections. Learning from actual customer behavior is intended to eliminate Heisenberg or Hawthorne effects from the learning.

Illustrating customer-driven testing with an example, suppose the product team believes customers want to import their contact list from their email account.

The HIPPO way: The VP uses outlook for email, so we build a utility to extract contacts from Outlook and an upload service. We build a training video and help guide for customers. This launches in 3 months.

The traditional way: The marketing team sends a survey out asking customers if they want to import their contact list from email, and asks which email service they use.  In one week, the team knows the percentage of customers who say they will import contacts, and a distribution of email services.

The customer-driven way: The team adds a link which says “import contacts from your email”, and a landing page with a drop-down list of email providers. In one day, you have information from customers who actually tried to import their contacts. Of course, if no one ever clicked the link, you learned sometime extremely valuable: customers don’t care about importing contacts. In this case, you saved 3 months of development (and all of the test cases and bugs that come with that much development).

The customer-driven way is more powerful because it provides information on actual customer behavior, not their intent, and not influenced by trying to be nice. It also has the chance to provide results much quicker than a survey, and almost certainly quicker than building the boss’s solution.

In the preceding example, the validated learning came from customers clicking the “import contacts” link, not knowing it was a test.  The relative power of validated learning varies with the level of investment or commitment made by the customer. In our example, the customer invested some time to attempt the import contacts and to select their provider from a drop down. This is a relatively low level of investment. Other examples, in descending order of commitment are:

  1. Pre-purchasing a product or upgrade.
  2. Intending to purchase/upgrade by entering a credit card. Of course, if you don’t have a product to sell yet, don’t actually charge the card. (also called a dry test)
  3. Providing personal information, like email, phone, or login credentials, and signing up to be contacted later. This can come in a message like, “Thanks for your interest, please fill out the form to learn when this is available”
  4. Clicking the link, and filling out a survey.
  5. Clicking a link, then clicking a second “learn more” page.

Using the “dry test” method can get tedious if over-used. Use caution and only use this when you are experimenting for a very large investment.

User Stories not Requirements

In the traditional model of software development, a special team of people, called business analysts or product managers, talk to customers, understand their needs, and document those needs in the formalized language of requirements.

Great care is taken to write the requirements in a manner that facilitates traceability, completeness, and abstracts any hint of implementation. When reading these formal requirements, it’s frequently easy to miss the point completely.

Documenting the product requirements in the form of User Stories helps keep the focus on customers, and removes an opportunity for ambiguity. User Stories can also be read and understood by customers

In the product definition stage, the quality/test team are usually recipients of the definition, not participants in the process. These practices show several ways in which the quality/test team can make a difference in defining the right product. The next post in this series will show several practices for including customers in the design and construction phase.

Visit the main page for more customer-driven quality practices or continue on to learn about building quality into your product.

Software Life-Cycle Model for Customer-Driven Quality

The software life-cycle model for Customer-Driven Quality is somewhat simplified, intended to be replaced by the actual life-cycle model used by your company. The purpose of this definition is not to  define a life-cycle, but to describe the customer-driven practices during each phase of your particular life-cycle.  These practices work with both Agile and waterfall life-cycles.

Life-cycle model for Customer-Driven Quality

Life-cycle Model for Customer-Driven Quality

The Define stage is where the product requirements and scope are defined. The customer-driven practices are intended to help build the right product for our customers.

The Build phase of the life-cycle includes design and coding. The Customer-Driven practices described help developers build the product for your customers.

The Test phase usually refers to system test, where activities like performance, regression, beta, and deployment validation testing occurs. Customer-Driven Quality helps optimize the test program.

The Support phase happens post-release and customers can help determine if the product was successful, and to find opportunities to improve the product for the next release.

Define, Build, Test, and Support phases of Customer-Driven Quality each have their own page, or return to the main page.