Authors note: I wrote this article in 2002.  At that time, Agile methods had definitely not crossed the chasm, and many organizations were struggling to obtain customer and user feedback prior to acceptance testing or beta testing.  This article describes a simple survey I had used to gage satisfaction–and take action–on projects that weren’t delivering incrementally.

Re-reading the article in 2010, it seems that there’s still a place for such a satisfaction survey. Even if you are delivering incrementally, it’s useful to get a sense of trends in the response to your product over time.

It’s 10 a.m. You’re about to ship to five beta sites. You’ve met the date, you’re within budget, and the defect counts have been steadily declining for the last four weeks. Still, you’re a little nervous. How will the customers

react to this new release?

Carrie was unhappy to learn during beta testing that the product her team had been building for insurance underwriting reps had drifted from what the users had wanted.

“It’s too slow,” the beta test group said. “It takes forever to pull up the records when I have a customer on the phone!”

“But in test it performed within the parameters in the requirements,” Carrie protested. “What changed?”

Just a small design tradeoff that affected the default display order of the records. It had seemed like a reasonable decision at the time.

If you’re a project manager, you know how important it is to have visibility into the product and the project. It’s bad news to find out in a beta test that you have built a product that doesn’t meet your customer’s expectations. How can you tell if you’re still on track to meet customer expectations?

Agile methods are one option. The agile school advocates close customer contact and frequent deliverables that are directly tied to business value. Still, many projects follow more traditional models: the customers (or their surrogates) are involved up front in defining features and then come back in during testing. A lot can happen during the time in between. If you’re not using an agile method and don’t have the luxury of having a customer assigned full-time to your team, how can you stay in touch with what the customer expects? One option is to measure customer satisfaction as you design and build the product.

Take a page from marketing and use a survey to catch a glimpse of customer reactions before the product ships. During development, ask the people who are closest to the product—sponsors, designers, and customers (or surrogates)—how they feel about the project.

You probably identified a set of system attributes during initial project definition. Suppose your customers identified these four qualities that were most important to them: easy to use, fast, always available, and secure. Of course, these descriptions are too vague to be helpful in actually building a system. You probably negotiated more specific targets for the product as a whole and for specific functions within the product. But for the purpose of a satisfaction survey, these more general attributes are about the right level. For each attribute, create a bipolar scale, with the desired attribute on one end of the scale and its opposite at the other. For example: the opposite of “fast” is “slow”; the opposite of “easy to use” is “difficult to use.” Here’s what a simple survey might look like:
____________________________________________
How do you rate the WidgetMaster system at this time?

What pleases you about the system at this time?

What displeases you about the system?

Comments:

____________________________________________
Then approach the cross-section of people you’ve chosen to fill in the survey with their opinions, and ask them about the version of the system under development. Be sure to leave plenty of room for comments at the bottom of the survey.

You won’t obtain scientifically precise data from your survey. However, you can find out how people feel about the project, and that ties directly to their expectations for the system.

When you first start using a survey, the comments section will provide the best clues. You may also want to investigate if the ratings vary widely. Big variations may be a sign that one stakeholder or group is unhappy with the direction the system is going. Follow up. You may learn something valuable. Jodi used a survey as a window on satisfaction for an application she was building for customer service reps. Jodi was surprised when the average ratings on “ease of use” took a dive, and she decided to investigate. It turned out that the customers had recently reviewed the screen and dialogue designs for the customer record retrieval function.

Supervisors were happy with the screen navigation because it was directive—helpful for new employees who weren’t familiar with the system. For them, it meant fewer questions and fewer errors from new customer service reps. Experienced customer service reps weren’t as happy. They were forced to respond to prompts that they no longer needed. For them the detailed navigation was a burden. They wanted to be able to take shortcuts and skip through prompts for functions that were second nature to them.

Armed with this information, Jodi was able to implement an alternate navigation scheme that better met the needs of experienced customer service agents. On the next survey, satisfaction was back up.

Best of all, Jodi was able to discover the dissatisfaction long before the system went into beta testing, when it would have been difficult to make the changes the agents wanted.

You can’t always “fix” the cause of dissatisfaction…but you can begin to manage expectations so that neither you nor the customer has an unpleasant surprise on
release day.

Why wait to discover how your customers will react to the system? Finding out how they are reacting to the work of building the system can give you valuable clues to help steer your project toward meeting expectations. This simple tool can help develop visibility into customer satisfaction.

This column originally appeared on Stickyminds.com in 2002.

© 2010 Esther Derby

Understanding the market and your customers (and potential customers) is the first step in building products that will sell and keep the business in business. You need to know enough about both so that you can plan your product investments and know which features to deliver when.

If you aren’t sure what your customers think about your current products, it’s time to find out. This doesn’t mean you have to launch a huge market research or competitive intelligence effort. You can begin by reviewing customer support data, interviews, and watching your customers use your products.  These simple acts of analysis, listening, and observation will reveal how your customers currently use your product, which features need improvement, and identify unmet needs.

Read about Your Customers’ Experience

If you have any sort of technical support line, you already have information about how your customers experience your software.

If your support calls are in a data base that doesn’t allow you to sort, slice, and dice the data, move it to one that can.  But don’t make it such a big project that it gets in the way of learning about your customer’s experience.  You can always take a sample, start with one product, or look one months worth of calls as a starting point.

The first step is a rough sort by feature or module.  Is there an area of the software the generates more calls than other areas?  Start there to understand the pattern and content of the calls. Look at your sample as a time series.  Is there any pattern in when the calls occur or spikes in the number of calls? Investigate.

Put yourself in the customers shoes, and try to see the issue from a customers point of view.  The data from support calls may have information on how problems affected the customer, but that information is filtered by the questions the support person asks and the from used to capture data. You need that information pure, straight from the customer. Follow up with customers who have called support recently to learn about the disruption or irritation they suffered.

Talk to Your Customer

Interview a sampling of customers, a dozen or so. You may choose a representative sample, or focus on people who have reported problems or called technical support.  Design a short interview—no more than 45 minutes.  After all the interviews, analyze the data. Look for themes related to difficult to use features or missing features.  Both will give you clues on where to focus future efforts.

I think it’s important for all people involved in building software to get to know their customers.  However, some managers worry that developers lack the social skills and polish to speak with customers.  It does take skill to interview customers, so give everyone a crash course on the three iron rules of customer interviews:

1) Never argue with the customer.  Even if you disagree with your customers assessment, do not argue the point.  If you win you lose because the customer will feel discounted. If you lose you lose because you’ve shown you don’t understand your own software.  In both cases, you’ve damaged the relationship with the customer and destroyed a chance to learn.

2) Don’t turn the meeting into a training session. If the customer is using the software incorrectly, take that as information about the product, training materials, or support.  At the end of the interview, offer to arrange a follow up call for training.  If you start instructing during the interview, you’ll miss out on learning from the customer.

3) The interviewee should talk more than the interviewer.  My rule of thumb is that the interviewer should talk no more than 30% of the time and spend at least 70% of his time listening to what the customer has to say. This is impossible if the interviewer relies on closed questions, which can be answered with yes or no.  Open-ended questions and probing questions invite the customer to provide deeper answers and reveal more about his experience with the software. (For more on how to interview your customers, see Building a Requirements Foundation with Customer Interviews, insights Vol. 2 No. 1.)

Observe Your Customer

Visit your customers and observe them in their context.  Watch them using the most common use features, those that the find easiest to use and those that they find difficult.  Take notes, and take pictures. User Experience expert Jared Spool advises that people should invest time every few weeks observing customers use the software. You say you can’t visit?  What about skype or video conference? Webcams are cheap and getting better every day. There’s really no excuse.

As you go about learning how your customers experience you product, notice how easy or how difficult it is to talk to your customer.  One client of mine had to go through multiple levels of approval in order to make a follow up call to a customer. Unless you work in an industry with high security or secrecy, it’s a big red flag if you can’t gain access to your customers. It signals that the customer’s point of view is distant, diluted, or discounted by the people who manage software development. How can you satisfy your customers when the people who build the software are prevented from getting to know them?

When everyone involved in writing software knows the customer, benefits abound.  People understand the customer’s context and concerns. They may find out that the feature they thought was wonderful is unwieldy. They’ll see the pain of duplicate data entry, foot-long drop down lists, and navigation that feels like searching for a needle in a hay-stack.  Feedback is critical for improved performance.

Are you getting the feedback you need to build great products?

***

If you would like to learn more about how your customers experience your software, my Customer Interviews workshop will give you the skills to prepare for and conduct effective interviews.