The previous post described tools and investments for implementing Customer-Driven Quality. Incoming support calls, feedback widgets, and customer usage logs are great ways to get information about your customers, but there are many more insights to be gained by engaging directly with customers.

Social Media

Product specific blogs allow conversations with customers. New features can be announced, articles written to help customers, and conversations happen through the comment feature of popular blog engines. One example where we used blogs effectively in software quality was to influence our customers to upgrade their browser. One quality constraint that we were living with was the support for Internet Explorer 6. A lot of special code had to be written for IE6, and testing multiple browsers reduced the amount of test on any specific browser. Newer alternatives had better security, faster performance, and better compatibility with standards. All aspects of better experience for our customers.

We wrote a series of articles on the blog to influence our customers to upgrade. Each time a new article was published, we saw a reduction in IE6 usage in the web analytics logs. Several customers complained on the blog about being forced to upgrade, but an interesting thing happened, other customers jumped into the conversation to explain the value of upgrading. By upgrading, customers had the benefit of a better internet experience, not just with our app.

Influencing our customers to take some action at the client side is an example of customer-driven quality.

Facebook, Twitter, Reddit, and numerous others are other venues for learning about customer’s usage of our products, and an opportunity to communicate with them. Often, customers are posting on these sites their frustrations and joys with our products.  The development and quality teams should be participating in social media.

One huge tip, if you use these social media sites yourself, you should create a new login dedicated to your professional persona. One time, I saw a customer complaining about not being satisfied by the customer support team. I sent him a private message introducing myself as part of the engineering team and offered my help. He never answered me, but the customer support supervisor received a blistering email about me. He saw my personal time posts about german shepherds and fly-fishing, and thought I was goofing off instead of working his problem.  Fair enough.

Now, I’ve created accounts associated with my product team and keep any posts and interactions to business. Keeping my business and personal internet personas is a little extra work, but reduces risk of customers being dissatisfied. (of course, I wish everything would work perfectly for every customer)

Search Engine Strategies

We also use search engines (Google, Bing) to search for our products and what our customers are potentially saying online about our products. Doing this exposed several community forums dedicated to using our product.

Customers will post their likes and dislikes about our products, questions about how to perform a task, and desired new features. This is a good way to gauge the customer’s opinion of our quality.

The search engines have a feature called alerts, which will automatically perform the search specified, and email the results in either a daily digest, or real time. To monitor the “blogosphere,” we’ve set up alerts with Google Alerts. I initially created a search term “<product> sucks” to look for negative sentiment, but that particular search rarely found anything. Instead, I search for just the product name.

Often these searches lead to blogs or forum entries. This allows you to communicate directly with customers experiencing problems (or who may have suggestions to improve).

Talk with Customers

Many of the customer feedback mechanisms provide both the feedback and contact information. One thing our team does is to regularly contact a sample of customers that gave us the gift of feedback. First, we thank the customer for the feedback (positive or negative) then ask follow up questions.  Doing this often fills in the gaps between the brief written note (often 140 characters or less) and what the customer really meant. These conversations can be scary, and I tend to be introverted and don’t have a good gift for gab, but usually these are the best 10 minutes of the day. I come away from these calls energized to build the best product for our customers.

What’s Next

The next few posts on Customer-Driven Quality will describe the practices for each phase of the software-development lifecycle (Define, Build, Test, Support)

Customer-Driven Quality Recap

Customer-Driven Quality is a framework for building software that is loved by your customers. It starts with the premise that customers define quality and every phase of the software development life-cycle should include engagement from actual customers. This post in one a series and describes some of the tools and infrastructure that most likely already exists in software development organizations, which are necessary to implement a Customer-Driven Quality program.

Customer Care Organization

I created this framework in a consumer-facing software company. The Customer Care organization is a huge investment intended for helping customers, but should also be used as a valuable source of feedback to the software development team. This function may be called customer support or tech support in your company. Alternatively, you may not have an organization, but online support forums. Some business-to-business organizations don’t have a support organization  but perhaps this function is handled by “sales engineers”.

Whatever the form of customer support, it’s extremely important to build relationships with the care organization and have a deep engagement with them. There is a wealth of information about quality (well, usually quality misses), in these organizations. Invest time and effort to learn how your company supports customers, and build a network of contacts within the support team.

Feedback Mechanisms

Besides the customer care channel, it’s very useful to get direct feedback from customers in the context of using our products. We’ve build a feedback widget that can be placed on any of our pages.

Screen shot of a feedback widget, which allows customers to provide feedback directly for the feature that is currently active on their window.

The feedback widget allows our customers to give the feature a rating, from 1 to 5 stars, and to type in some comments, and optionally contact information if they are willing to have a conversation. The product managers, developers and testers who created the feature monitor the feedback. This has been an extremely useful mechanism to learn what is working, and find opportunities to improve the feature. One example, to illustrate how valuable the feedback mechanism can be, the feature development team is not “done” with the feature until they have achieved at least a 4-star rating.

Here is some example feedback, where customers provide valuable insights in how we can make the feature even more useful.

Example customer feedback for a feature

Services such as uservoice.com provide this type of functionality for relatively low investment. This particular example has a benefit in that customer can vote for the most popular feedback. Customers define what they want, and also provide a priority for the team.

Screen shot shows another example of a customer feedback mechanism used in product. This one allows customers to vote on other customer?s feedback, helping to identify which of the new requests are most popular with existing customers.

Text Analytics

Much of the feedback that we receive is in text form. The feedback widgets, support emails, support chat transcripts, and call summaries all come in plain text. While useful to read the raw input to build empathy, it’s also important to aggregate the raw data to gain insights on the most important issues faced by your customers.

At my current company, we are lucky to have many customers, so we get a lot of feedback. Automating the analysis of feedback is efficient and gives an opportunity to analyze it more frequently.

A tool that we’ve found to be very useful is Sentiment Analysis. Sentiment analysis is a semi-automated process that measures both the frequency of occurrence of each issue type, but also measures the emotional intensity.

Text frequency analysis allows us to count the number of times our customers have a particular type of problem (i.e. how many calls are about login). Where sentiment analysis shines is adding customers intensity. For example, “I really hate your login” is generally worse than “Your login is difficult.”

Tools are available to help with this analysis, for example Clarabridge Analyze. Homebrewing your own solution is sometimes the right answer as well. I’m a fan of Python and NLTK for these tasks. One of my previous posts described several tools for analyzing customer sentiment on social networks.

Monitoring

Our software has many monitoring mechanisms built in to make sure its operating properly and to alert the support team in the event things start to go wrong. There are action logs, exception logs, and performance monitors. Each of these logs also can provide some insights into actual customer behavior and experience.

The quality teams should learn what logs are available, how to get access, and how to analyze these logs. One example where we used the log files, we pulled the action logs across 30 days of use, and identified the top 10 transactions. This information helped us to prioritize the test automation program.

Successful web apps likely are monitored by tools like Google Analytics or Site Catalyst. Sometimes, this information is managed by the marketing team. Build a relationship with the marketing team and use this data to understand how your customers are using the information.  

Customer Satisfaction Surveys

Your marketing or customer support team likely survey’s your customers to understand their satisfaction level. A very popular method is to use the Net Promoter survey. Your development and quality teams should read these survey results and the verbatim comments from your customers.

In case your organization does not survey customers, having the capability to do so is very useful. I’ve had success with SurveyMonkey, though there are many alternative tools available. Even if your organization regularly surveys your customers, it still may be useful to have this ability to ask targeted questions.

Wrap-up

This post listed a variety of tools that are used for implementing Customer-Driven Quality. The next post will describe ways to engage directly with customers (cutting out the middle-persons). Please leave a comment if you have any questions or your are gaining value from these posts. Return to the main page for Customer-Driven Software Quality.

Most software products are developed for “real people” not software developers. Sometimes those of use that are creating software every day, using technology in every aspect of our lives and are savvy computer users sometimes forget that our customers don’t have the same skills and thought processes.

Examples where we forget that real people are using our products:

• Technical knowledge about the operating system: We may think it necessary to “run as administrator” to install an update. Most customers don’t what this means or how to change their access levels.
• UI short cuts: We use the context (right-click) menu every day, but most customers don’t. 
• Better browsers: Awareness of Firefox and Chrome are on the rise, but many customers still use the default browser that came with their OS. 

A few of these examples are not likely to be specified by the product marketing team. To build software that is usable by your customers, its important for the developers and quality team to understand the customer. They need to have empathy with their customers to fill in the gaps between written requirements.

These are a few practices that have proven useful for building empathy with customers. For learning how they think, and what they think about, and what they expect to be taken care of for them.

Customer Care rotation for new engineers

Where I work, we have a practice for all new college hires. They spend a month with customer care, taking support calls on our product.  In this program, they are provided the same training that our care agents receive then get on the phone to help our customers. The learnings are amazing. After 4 years of academic learning, steeped in algorithms and theory, they get exposed to the day-to-day issues that our customers encounter. An example of an insight that an engineer passed on to me, “I was amazed how many people didn’t know about the right-click menu.”

The Customer Care rotation program is outstanding, but it’s also pretty expensive. It’s mostly been applied to new college graduates, and it’s combined with a more general orientation to our company and corporate life. Another method that is used by experienced professionals as well as long time employees is to listen into customer calls.

Listening to calls

Many of our customers call in for support. As we all know, “those calls may be monitored for quality control purposes.” The monitoring infrastructure provides an outstanding opportunity to get engineers and testers exposed to these calls. In fact, listening to both sides of the conversation while not an active participant gives some deeper insight to the customer’s issue and how we resolve those issues.

Many of our staff meetings start with listening to a call, followed by a discussion about the insights each of us learned by the process. Another practice, we call it the call listening lunch. Once a month, the entire team is invited to have lunch, while listening to prerecorded calls. After hearing the call, we discuss what could be done in product to mitigate the call. Many improvements came from these sessions.

Follow Me Home

The Follow Me Home process started a long time ago, when the main distribution channel for our products was a retail store. Engineers and marketers would hang out at a local retailer and watch people shop for our products. Marketers were interested in how the brand new customer chose our product from the alternatives on the shelf, but the fun stuff happened when they “followed” the customer home.

The team would approach the customer and ask to follow them to their home or place of business and just observe the first time use process. Did they read the instructions? Have trouble installing? Accept defaults or investigate each option?

The process continues, but is usually set up ahead of time via email contact. It’s very instructive to observe the customer use our products, in their environment. You notice things that are not spoken or written in surveys or support tickets.

One example insight, on a follow me home that I conducted with a customer that had complained about our service being slow. We checked the logs; all the transactions were in normal tolerances. We checked her computer, network connection bandwidth, network latency to our data center, etc. No slowness there.

In observing her, though, she had a stack of bills that needed to be paid. This stack was on a credenza behind her desk. She would take a bill, swivel her chair around and enter the data on our page. Then hit save, swivel and get another bill. She said “there, see, the screen is not yet ready to enter a new bill. It used to be ready by time I swung back.” Ah, we now have a new definition on how fast is fast enough: it can’t impede her work flow. (The real insight, why are we making this poor woman type in information, the bills should be automatically imported.)

Usability Studies

Web design is about a lot more than fonts and style sheets. Ultimately, it’s about ease of use. To test ease of use, and gain insights about customer behavior, the UI designers invest a lot of time in formal usability studies. They will invite customer (or potential customers) in house and observe them using our product or prototypes. This is an excellent opportunity to get an engineer or tester some insights as well. Make friends with the design team, and get some invitations to observe as well.

Voice of the Customer

Our customer care team handles many calls, emails, and online chat sessions each week. They produce a call drivers report, which lists the top 10 issues that generate a support call, with the goal of fixing product or process so the call is not required. This aggregated and abstracted data helps focus priorities for product changes, and the raw data that feeds these reports are a great resource for gaining insights about our customers. We call this raw data the Voice of the Customer.

Our support system has a feature that pulls out 50 random issue summaries, and emails it to the entire team. The first item on our weekly development staff meeting is to review the call drivers and discuss insights learned from reading the verbatim comments.

Customer Satisfaction Survey Call Backs

Our marketing team conducts a customer satisfaction survey every quarter, called the Net Promoter survey. (Net Promoter is a measure of the willingness of our customers to recommend our product to their friends and colleagues). Part of this process is to call every customer that is a detractor (not willing to recommend our product), and a random sample of the promoters.

The marketing team loves getting help in making these follow-up calls. The team that builds and tests the product is an excellent choice for this task. Each engineer and tester is expected to make 3 of these calls per year. It’s not a huge investment in time and the insights gained are very valuable.

Product Challenge

One way to understand our customers is to act like one. We hold frequent “challenges” where we put ourselves in the shoes of our customers and try to use our product. This is a great way to introduce new team members to the product.

To make this realistic, we have a document that describes a typical customer scenario. We ask the participatants to follow along with the scenario and only rely on information provided externally (web searches, product documentation, etc.) Putting yourself in the customer’s shoes is very helpful to understand how your product comes across to them.

Summary

These practices are helpful for developers and testers to build empathy for customers. In the next post, I’ll describe the infrastructure that is useful to implement customer-driven quality.

Photo Credit

In previous posts, I described the how the Customer-Driven Quality framework can improve your software quality by making sure quality & testing efforts are focused in the right area: customer satisfaction. The framework contains two main sections, a life-cycle view (Define, Build, Test, Support) and the surrounding practices (Focus, Empathy, Engagement, Learning). The first, and most important of these surrounding practices is Focus.

In this context, Focus means to place customers and customer-driven quality at the center of your teams activities. Providing your team with the overall context for customer focus, and showing them how their activities fits in the bit picture, shows them how their jobs and activities really matter.

Values

Values are the foundation upon which customer-driven quality is built.

Many organizations publish a statement of values, and these documents usually state a commitment to providing a value to your customers. Here are a few examples:

• Google: “Focus on the user and all else will follow”. 
• Apple: “Apple is committed to bringing the best personal computing experience to students, educators, creative professionals and consumers around the world…”
• Intuit: “We put customers at the heart of everything we do. We work together to deliver end-to-end experiences so profound that customers love using our products and services, and actively recommend them.”

Of course, software is developed in many other organizations than commercial software development companies. Here are a few examples from government agencies.

• Health and Human Services: “The Department of Health and Human Services (HHS) is the United States government’s principal agency for protecting the health of all Americans and providing essential human services, especially for those who are least able to help themselves.”
• Internal Revenue Service: “The IRS role is to help the large majority of compliant taxpayers with the tax law, while ensuring that the minority who are unwilling to comply pay their fair share.”
• National Security Agency: “Our customers range from U.S. decision makers to service personnel who are in harm’s way.” 

Principles

Once you have a connection to the organizational values, its time to define your operating principles. The principles are one click down from the values, and are designed to help your team make decisions.  Two principles we use for customer-driven quality are:

• Customers Define Quality: We have a variety of definitions of software quality: bug-free, performant, meets requirements, etc.  These definitions are mostly inwardly focused, based on our processes.  Explicitly stating that customers define quality leads us to think about how to learn from our customers.
• When in doubt, test with customers: Many decisions, like feature content or quality decisions, are made based on opinions. Too often, the decision is made by the person with the loudest voice, or the highest paid person in the room (we call them hippos)

Team/Group Goals

After identifying your organization’s values and defining your operating principles, its time to include customer-driven quality practices in your team’s goals. Team goals are vital, as these goals lie side-by-side with other priorities for your team, and can be seen as competing with your other goals. Its important that your stakeholders, like product owners, marketing, and your leadership, agree with allocating time and effort for customer-driven quality activities. In my experience, the stakeholders are usually interested in focusing the quality team on building better quality for customers. I’ve found it useful to declare these goals along-side of the other goals for the team.

If you manage via scrum or other Agile methodology, you’ll want to create a set of user stories associated with customer-driven quality activities.

Individual Goals

Organizations differ in the degree of formality of Performance Management systems. Some have very formal goals that are frequently updated, others are less formal. Regardless of the culture of goal setting or monitoring progress towards these goals, the activities associated with Customer-Driven Quality should play a large role.

Everyone in the organization can participate in the customer-driven activities. Many have customer interactions as an explicit part of their job, while others need to take the initiative to engage in customer interaction. I expect everyone on the team to participate, even the test automation framework developers.

One of the most powerful ways to influence any activity is to model that behavior yourself. If your team sees you investing your time talking to customers and gaining insights from that activity, they are very likely to participate as well.

This post showed how I focus on implementing a customer-driven quality program. I start with the organizational goals, define several principles, get buy-in from stakeholders through the goal setting process, and allocate those goals to individual people. I also use customer affinity/empathy as part of the skills needed to compose the team.

Future posts in this series will spell out more of the practices for customer-driven quality, organized by elements in the framework.