improve your customers' experience by listening to unstructured feedback

Improve Your Customers’ Experience By Listening to Unstructured Feedback

Good experiences. On brand. On budget.

©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • [email protected] • www.kana.com

PAGE 2

Executive Summary .................................................................................................................................................................................3

Finding the Golden Nuggets ...................................................................................................................................................................4

Capitalizing on Fresh Insight ...................................................................................................................................................................5

Emerging Market Appraoches Tipping Point ..........................................................................................................................................8

Making the Business Case .......................................................................................................................................................................9

Smart Practices for Success ..................................................................................................................................................................10

About the Author and Sponsor .............................................................................................................................................................11

Appendix: Text Mining Fundamentals ..................................................................................................................................................12

Acknowledgments

This paper was made possible with the generous sponsorship of Island Data Corp. and in-depth interviews

with its customers, including Meredith Sime, associate director, Customer Experience, for AT&T’s U-verse;

Jill Trecker, guest loyalty manager for Garden Fresh Restaurant Corp.; and Devon Child, senior product

manager for Yahoo!.

We also sincerely thank CustomerThink community members for their input in an online survey about

managing unstructured customer feedback, which provided most of the statistical information included in

this paper. Those seeking more details on text mining are encouraged to read our key background

sources: The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge University Press,

2007) and Machine Learning in Automated Text Classification by Fabrizio Sebastiani (ACM Computing

Surveys, Vol. 34, No. 1, March 2002).

Table of Contents



PAGE 3

Customer-centric businesses must, above all else, be committed to listening to the voice

of customers and acting on that feedback. But, these days, that “voice” includes

unstructured text that can’t be handled with conventional data analysis.

If your business is like most, you hear from your customers all the time. They send you

email. They unload earfuls to your customer service reps. They write in comments on your

product surveys. They leave feedback on your web site.

You probably would like to find out what customers are saying, but in practice, you don’t

cull through the massive amount of feedback for a variety of reasons: You’re

understaffed, there’s too much of it and you have other fires to put out.

Forward-looking business leaders are now recognizing that this unstructured feedback

contains the key to amplify customer listening programs. And they recognize that the

effort to mine their unstructured customer feedback is worth the rewards: early warnings

about calamities, insights about pain-points that could cause customer defection and a

view into future customer trends.

In this paper, I’ll look at the business case for mining your unstructured customer

feedback, and I'll discuss the basics of text mining, looking in simple terms at how it

works. I’ll examine the key findings from a 2007 CustomerThink survey on how businesses

are managing unstructured customer feedback, and I’ll discuss what executives from

some successful businesses say about what text mining has meant for their strategy.

Finally, I’ll discuss the practices that will work best when you do take the plunge and

adopt a text-mining solution for your organization.

A 2007 CustomerThink survey of members found that business leaders are certainly

aware of the importance of mining unstructured customer feedback, but they also

recognize they are, by and large, not doing enough about it. Here are some key findings

from that survey:

The majority of businesses surveyed received unstructured customer feedback in the forms of comments from surveys, unsolicited email and online feedback forms.

An overwhelming majority (80 percent) of businesses surveyed said managing unstructured customer feedback would help improve their business performance.

More than half of the businesses surveyed said that managing unstructured customer feedback is important to top executives, and that importance will only increase in the future.

Executive Summary

“This kind of feedback is

giving us what the

customer wants to tell

us, not what we want to

ask them.”

- Survey respondent



PAGE 4

Yet few (less than 25 percent) of the businesses we surveyed felt that their companies were doing an “excellent” or “very good” job of capturing, analyzing, reporting or acting on unstructured customer feedback. Only 12 percent said they used a software tool of any kind.

Managers surveyed felt that the two most important potential benefits to effectively managing unstructured feedback were improving products and services and improving customer loyalty.

One emerging best practice stands out, from both executive interviews and survey

respondents. For success in managing unstructured customer feedback, the top priority is

to act on the feedback received. Otherwise, why should customers invest their time in

telling you what they think?

Finding the Golden Nuggets Text mining, like its geological counterpart, is sifting through vast amounts of debris to

find the gold. As Ronen Feldman and James Sanger, the authors of The Text-Mining

Handbook (Cambridge University Press, 2007) note, what we call “unstructured data” isn’t

completely unstructured. Text follows some basic tenets of natural language, and text

mining involves analyzing text to 1) determine what the original author was trying to say

or 2) learn something completely new.

Like data mining, the idea is simple, but what’s “under the hood” in text mining

applications is complex. One common technique is called “categorization,” which simply

means classifying text into categories. An example would be deciding whether customer

emails represented “happy,” “unhappy” or “neutral” customers, based on the types of

words used in those emails. You can imagine that each category should be handled

differently, and it might be useful to track the percentages in each category over time. For

more details on text mining methods, please see the Appendix or read our reference

sources.

Categorization has been used since the late 1960s in areas such as medicine and news

services, where there were large stores of documents, a strong desire or need to make

sense of it and the financial wherewithal to handle the expense of the computing power

necessary to crunch through the data.

Text Mining Comes of Age

In recent years, text mining algorithms have been more widely adopted by businesses,

thanks to “Moore’s Law,” which drives continued computer performance improvements.

But equally importantly, as you’ll learn in this paper, the setup and usage of text mining

systems have become much easier with the adoption of more “turnkey” implementation

approaches, including hosted or “on-demand” applications.

Executives at companies that are using text mining to analyze their unstructured

customer feedback are enthusiastic about the benefits. Early adopters usually are. For



PAGE 5

them, not only did text mining help them meet the challenge they knew they had, but also

they found numerous other uses and benefits. Consider these illustrations:

In one case, a floral company was getting a high volume of returns of one particular floral arrangement. The company thought that it was because customers were disappointed in the look of the flowers. But after mining the unstructured customer comments, they discovered that wasn’t true after all.

In another example, a telecommunications company introducing a new service was able to analyze unstructured feedback over a period of time. This enabled executives to see the nature of complaints progress from predominantly service-related to encompassing more aspects of the customer experience as the product evolved in the marketplace.

And in yet another case, restaurant chain executives were able to see that the new buildings they were designing for additional restaurants had some critical issues. Executives were able to make design changes before replicating the faulty design to multiple locations.

In every one of those cases, executives are thankful for what they see as a robust tool for

looking into the minds and experiences of the customer. In general, they see text analytics

for customer feedback as a necessary part of a customer-centric enterprise. It’s not about

just automating a process to gain efficiency, although technology does enable text

analysis that is impractical to do manually. The real key is understanding what customers

are saying in their own words and then acting on that insight as quickly as possible.

Capitalizing on Fresh Insight If you’ve eaten at a restaurant or shopped at a store lately, you’ve no doubt seen an

invitation on your receipt to call a toll-free number and respond to a survey about your

recent experience. Surveying customers is a popular way to quickly gauge the thoughts of

motivated customers, whether they’re interested in venting about a problem or praising

your business for a great product or outstanding service.

Surveys are now commonplace tools for companies to solicit customer feedback. In a

2007 CustomerThink survey, nearly 80 percent of respondents said they were conducting

customer surveys at least annually, up from 70 percent in 2004. It’s a safe bet that nearly

all of those surveys include some questions for customers to write in their comments.



PAGE 6

In another survey in late

2007, we learned that,

sure enough, customer

surveys are a popular

source of unstructured

feedback. But unsolicited

emails and web site

forms are also popular

sources of customer

feedback text for the

majority of respondents.

Other sources cited

included call center agent

logs, transcripts from

recorded phone calls, text

messages, chat sessions

and posts on discussion

forums or blogs.

Now, the fact that this feedback is available does not mean that it is analyzed and

acted upon! It turns out that not enough are taking advantage of it, according to

our survey. When we asked people how effective their companies were managing

unstructured customer feedback, more said they were best at capturing customer

comments. But the average ranking they gave to that activity was only 2.66, when

a rating of 3.0 is “good.” And 46.2 percent rated their efforts as less effective than

that. It's clear that businesses have a long way to go.

What about taking advantage of the feedback? Well, the story there is somewhat worse,

with fewer than 25 percent of those surveyed rating their effectiveness as “excellent” or

“good” and more than 50 percent giving their organizations a “fair” or “poor” rating.

These findings are not at all surprising, given the emerging market for text analytics

solutions. Judging from the comments from early-adopter executives, however, there’s a

tremendous opportunity for a wide variety of companies to listen better to customers and

act on the insights gained.

Listening to Vocal Customers

If you give customers a chance, they’ll communicate with you in many ways. “It’s worth

noting our guests are very vocal,” says Jill Trecker, guest loyalty manager for Garden Fresh

Restaurant Corp., a chain of 104 buffet-style restaurants known as Souplantation in

Southern California and Sweet Tomatoes in other parts of the United States’ Sun Belt. The



PAGE 7

restaurants boast 55-foot salad bars and a selection of eight soups, baked goods and

frozen yogurts—on an “all you care to eat” basis for one price.

In addition to inviting customers on restaurant receipts to take surveys, Garden Fresh

posts a toll-free phone number in its restaurants, inviting customer feedback. They listen

to the voice prompts, and they rate the restaurant experience. Guests are also given the

opportunity to record a 60-second message. A service management company handles the

responses to the structured questions and transcribes the unstructured recordings to be

processed by Island Data.

The company also has an online “eat club.” Six weeks after customers have joined the

club, they receive a short survey with structured questions and a place for free-form

responses.

The philosophy at Garden Fresh is that “no news is bad news,” but with so many vocal

customers, the company was overwhelmed by the volume of customer feedback. The

company receives about 10,000 pieces of unstructured feedback a month from the

different channels. Going through them manually would be, in Trecker’s words,

“prohibitive.”

“There were things we knew that we were probably missing that we didn’t have a

structured question about,” Trecker says. Additionally, Trecker and company President

Kenneth J. Keane worried that if they were soliciting feedback and not responding to

important information in that feedback, they really were wasting people’s time—or

worse—risking angering customers.

Sifting Through Mountains

Garden Fresh contracted with Island Data in 2005 to find meaning in all that feedback. It

took the company about three weeks to set up shop, beginning by taking about 1,200

records of the 60-second transcribed comments and using them to train the system to

identify different categories of praise, criticism or requests.

Now the company has dashboards and receives monthly “praise” and “complaint”

reports, both showing trends and drilling down to individual customer comments. The

data is used mainly by the marketing team, but Trecker regularly shares reports with

Keane and the rest of the executive team.

They now can look at trends “month to month to month,” says Trecker, and drill down to

investigate trends. One “aha!” moment early on was in responding to customers’

demands for more soup varieties, including vegetarian soups. The company expanded the

number of soup varieties offered, and complaints about variety lessened.

“You’re one of our

favorite places because of

the healthy benefits of

your food. BUT, we have a

few suggestions: I would

like to see raspberry

vinaigrette as a regular

dressing and see you

improve the blue cheese

dressing, which is much

too ‘mayonnaisey.’”



PAGE 8

The company also was finally

able to confirm why it

typically received more

complaints in February and

March, traditionally the

months when Garden Fresh

restaurants have a higher

volume of activity. The

hunch was that when

volumes increased, attention

to cleanliness dropped, as

employees struggled to keep up with the load. Sure enough, the Island Data reports

showed people were complaining about cleanliness. In another example, as mentioned

earlier, Garden Fresh found that customers were complaining that the new restaurant

design seemed too crowded and noisy. Because of its window into customer feedback,

executives had time to make changes before it was too late.

Emerging Market Approaches Tipping Point

It’s early yet in the market for unstructured customer feedback solutions, according to

Lane Michel, executive vice president and managing director of the Marketing

Performance Management unit at Quaero Corp. The most active applications are in

customer experience assessment and monitoring; new or upgraded product feedback; call

center performance measurement and efficiency; and competitive intelligence gathering.

Call center executives, for example, are looking to a new generation of more effective

tools to address customer experience and operational performance issues. “Marketing

executives cannot continue to wait for the cumbersome 360-degree customer databases

and slow analytics to generate intelligence, early warnings and pragmatic segmentation

schemes,” Michel says. “My guess is that we should see a steep climb in text mining tool

sales in the next 18 months, with a shift out of early adopters predominately purchasing

today.”

In most cases, companies were already gathering feedback. The decision to actually

implement a system came when the volume of that feedback reached a tipping point. One

retailer was receiving from 4,000 responses in a slow week to 150,000 to 200,000

responses a week during peak holiday seasons.

When creating customer feedback tools, “we took a pretty decent stab at anticipating

what our customers might want to tell us," said Meredith Sime, associate director,

Customer Experience, with AT&T’s U-verse, a new platform for services provided over

“Marketing executives

cannot continue to wait

for the cumbersome

360-degree customer

databases and slow

analytics to generate

intelligence, early

warnings and pragmatic

segmentation schemes.”

- Lane Michel, Quaero



PAGE 9

Internet Protocol. "But we did not initially invest enough resources in mining the wealth

of information from an unstructured format.” And, she said, those instincts were borne

out once they started analyzing the data. “We learned a tremendous amount of

information that helped us drive improvement plans, and it taught us to ask better

questions.”

Customers, Sime said, “raised issues we had not previously considered because the

products were new to us.” Like others I spoke with, Sime would love to get near a real-

time response in the future. “We would have a first-response mechanism if any quality of

service issues were to arise,” she said.

In fact, according to Scott Austin, Island Data’s chief technology officer, the company

found immediate reception to its “early-warning” alerts, notifications that there was a

spike in complaints about a product or service. When Island Data first rolled out the

service, it was temporarily offered to customers on a complimentary basis. “Within a

week and a half, [customers] said, ‘We absolutely have to have it. We will pay you money

to make it a permanent part of what we have.’”

Making the Business Case We asked survey respondents to rate the potential benefits of effectively managing

unstructured customer feedback. “Improve products and services” and “improve

customer loyalty” tied for top ranking, followed closely by “gain insight hidden in

customer comments” and “listen to customers in a more natural way.”

However, when we asked people

how they would justify an

investment, the top three factors in

their rationale were: 1) improving

the customer experience, 2)

improving customer loyalty and 3)

increasing revenue. Collectively,

these factors accounted for 53

percent of the weighting. Other

factors included: improving

product/service quality, supporting

a customer-centric strategy,

improving decision-making and

reducing operational costs.

Considering the survey results along with executive interviews, it seems clear that the ROI

from managing unstructured customer feedback is geared toward improving customer



PAGE 10

retention by focusing on trends, “listening” to customers and improving the customer

experience.

One of those early-warning alerts came in handy at Yahoo!, according to Devon Child,

senior product manager for the portal’s internal feedback platform. It highlighted a

problem that product managers knew was festering but didn’t consider a priority. “They

had it on their radars.” The alert came back, and they realized that it was more serious

than they previously thought. That’s a great “aha!” moment, but does it justify

implementing a text-mining solution?

How do you convince the rest of your company that managing unstructured feedback is

important? I asked executives this question, and they were almost puzzled by it. Perhaps

these are stellar firms when it comes to customer-centricity, but these executives did not

encounter internal resistance to text mining.

Their companies wanted to know what customers were thinking, which, again, is why they

were collecting the feedback in the first place. The question for each was: Do we do it in-

house or do we outsource? Executives I interviewed said money wasn't the issue. Their

aim was to do what was best for their companies—and their customers.

At U-verse, which performed an IT assessment of the costs, comparing outsourcing with

in-house text mining, Sime saw no resistance to the question of whether to analyze the

text. “We had all this data coming in, but we had no idea what to do with it.” The ability to

cull through all the unstructured feedback has “been a godsend to us,” Sime said. “Before,

the best we could do was paste the information in quotes.”

What gems of information are hidden within the customer feedback your organization

collects? “It’s hard to argue with wanting to listen to the customers—our customers,”

Child said. “Their opinions matter.”

Smart Practices for Success It may be a bit premature to declare the definitive list of “best practices” for success with

projects to manage unstructured customer feedback. For now, let’s call them “smart

practices.” The chart below shows the opinions of CustomerThink members, based on our

2007 online survey. Given this input, along with my interviews with early adopters, here

are five practices that will help you succeed.

1. First and foremost, it’s critical that you are truly committed to act on the feedback you receive, whether it’s unstructured. Nothing is more disillusioning to a customer than taking the time to provide constructive praise or criticism and then have it be ignored.

“It’s hard to argue with

wanting to listen to the

customers – our

customers. Their opinions

matter.”

- Devon Child, Yahoo!



PAGE 11

2. Make sure the insights are correct. “Best algorithm” was not rated highly in our survey, but it's very important to the results you get. Garbage in means garbage out. The analysis-and-categorization algorithm must be reasonably effective; otherwise, you’re acting on bad information. That said, there is probably a point of diminishing returns in striving for the “perfect” algorithm.

3. You must get the “golden nuggets” of insights to the right people, at the right time, in a format that is easy for business people to use. You can see these priorities clearly in the survey, and several interviewees commented that ease-of-use was extremely important.

4. Make sure the project has senior level sponsorship. Odds are you won’t be able to create a business case that “proves” in a spreadsheet how your company will increase revenue or cut costs by investing in text mining. You need an executive’s vision of a customer-centric organization.

5. Finally, pick a solution that is cost effective and fits your business and IT environment. Consider the total cost of ownership when comparing an installed to on-demand solution. Keep in mind, however, that cost is probably not going to be the central issue if your business strategy dictates that better listening to customers is a critical success factor.

Follow those recommendations, and you, too, can reap the benefits of text mining. As

Sime says, “The value of the unstructured feedback is phenomenal.” You already have the

content. It’s time to put it to work to improve your customers’ experience—and your

business performance.

About the Author and Sponsor About the Author – Bob Thompson, CustomerThink Corp.

Bob Thompson is CEO of CustomerThink Corp., an independent customer management

research and publishing firm. He is also founder of CustomerThink.com, the world’s

largest online community dedicated to helping business leaders improve customer-centric

business strategies.

Since 1998, Thompson has researched the leading industry trends, including partner

relationship management, customer value networks and customer experience

management. In January 2000, he launched CRMGuru.com (renamed CustomerThink.com

in 2007) which now serves 300,000 business leaders monthly through its web site and

email newsletters.

“The value of the

unstructured feedback is

phenomenal.”

- Meredith Sime, AT&T’s

U-verse



PAGE 12

Thompson is a popular keynote speaker at conferences worldwide and has written

numerous articles and papers, including his most recent report, Customer Experience

Management: A Winning Business Strategy for a Flat World. Before starting

CustomerThink, he had 15 years of experience in the IT industry, including positions as

business unit executive and IT strategy consultant at IBM.

For more information, visit www.customerthink.com or contact Thompson at

[email protected].

About the Sponsor – Island Data Corp.

Island Data Corp., the innovator in Customer Analytics solutions, delivers real-time market

and business intelligence for global online enterprises. Island Data’s software-as-a-service,

Insight RT™, allows customer-centric organizations to listen to the voice of their

customers by managing the unstructured customer feedback. Insight RT enables

executives to track key performance indicators, collect actionable intelligence and

optimize the customer experience at every touch-point. Companies employ Insight RT for

timely discovery of critical facts and issues that support continuous improvement of

product management, product quality, market and online presence, brand loyalty,

customer retention, customer service and customer satisfaction. Headquartered in

Carlsbad, California, Island Data is funded by top-tier venture capital firms Dolphin Equity

Partners and ABS Ventures.

For more information, visit www.islanddata.com.

Appendix: Text Mining Fundamentals Text mining is the discovery of information by analyzing text. Sources could include any

written form of communication captured electronically or verbal communication

transcribed into text. Insights gleaned from this text could reflect the meaning of what the

author intended or provide entirely new information.

Whereas data mining extracts information from structured databases, text mining

extracts information from natural language text. There are two approaches: linguistic and

statistical.

The linguistic approach involves identifying linguistic elements of language and structures

that relate them to each other. Those elements are the keys to meaning. This is much like

parsing or diagramming a sentence to identify parts of speech. If you have an adjective,

for example, what noun does it modify?

You don’t need human intervention to train a classifier, but you do want to identify parts

of speech (which is done easily with a conveniently packaged list known as a “dictionary”);

words that are commonly used by people who are angry (which can come from a



PAGE 13

dictionary or thesaurus); and information about sentence structure (if you have two

nouns, which is the subject and which is the object? Did John throw the ball or did the ball

throw John?).

In the statistical approach,

words and phrases are treated

as abstract objects. You use

purely their mathematical

relationship to each other. This

approach most often involves

machine-learning. Is your

customer angry? Is he pleased?

Is another customer talking

about your latest product?

Using a sample of the text and

assignments in a number of

categories, the computer scans

them for common elements.

The machine learns by example

based on training data assigned

by human beings. If a business

receives a million emails a day,

you would take a small

sample—say, 500 to 2,000

emails—and manually classify them. Then the computer would scan the sample to

identify relationships in the text that hold clues for whether a particular email may be

from a happy customer. People using obscene language tend to be unhappy, so simply

scanning for profanity in your sample can distinguish email from irate customers.

There are many ways to train a classifier without human intervention. One important

method is clustering, in which you look at what words naturally go together, automatically

identifying common themes just by looking at one or two emails, rather than 200,000, for

instance.

For more information, please read:

The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge University Press, 2007).

Machine Learning in Automated Text Classification, by Fabrizio Sebastiani (ACM Computing Surveys, Vol. 34, No. 1, March 2002)



PAGE 14

Copyright 2011 KANA Software, Inc. KANA and the KANA logo are registered trademarks of KANA.

Other company, product and service names may be service marks of their respective owners.

840 W California Avenue, Suite 100

Sunnyvale, CA 94086

T 650.614.8300 | F 408.736.7613

www.kana.com

Contact us at http://www.kana.com/contact-us/contactsem.php

Twitter @KANASoftware

Linkedin: http://www.linkedin.com/groups/KANA-Software-1129?mostPopular=&gid=1129

Facebook: https://www.facebook.com/pages/KANA-Software-Inc/146154198748782

All Rights Reserved. 0110-01