flawed tools: the efficacy of group research methods to generate customer ideas

Flawed Tools: The Efficacy of Group Research Methods toGenerate Customer Ideas*Gary R. Schirr

Group customer and user research methods, brainstorming, and focus groups continue to be used in innovation effortsto uncover customer needs, generate new product and service ideas, and evaluate decisions, despite extensive empiricalevidence that group methods are ineffective for such purposes. This paper summarizes the strong evidence of theineffectiveness of group research methods for these purposes, much of which has been published outside of the newproduct development or business literature. The paper shows that the most common rationalization for the continueduse of group methods—cost and speed advantages—are questionable, and then proposes an organizational marketlearning framework for evaluating the use of group methods. This framework provides guidance for the proper use ofthese research tools and suggests areas for future research on research methods for product innovation.

U nderstanding customer needs and generatingideas from good customer research are impor-tant antecedents of success in new products

and service (Cooper, Easingwood, Edgett, Kleinschmidt,and Storey, 1994; Cooper and Edgett, 1996; Henard andSzymanski, 2001). A succession of research streams hasfocused on engaging customers to understand their deepneeds and to generate ideas for product and service inno-vation, including (1) the customer active paradigm of pro-duct development (von Hippel, 1978); (2) lead users (vonHippel, 1986); (3) the voice of the customer (Griffin andHauser, 1993); (4) probe and learn (Lynn and Morone,1996); (5) experimentation (Thomke, 2001); and (6)cocreation by customers (Payne, Storbacka, and Frow,2008; Prahalad and Ramaswamy, 2004). These methodsfocus on engaging users or customers at the individuallevel to uncover deep needs and sticky information(Szulanski, 2003; von Hippel, 1994); for example, theauthors of “The Voice of the Customer” stressed thatindividual customer interviews were much more effective

for new product ideas than focus groups (Griffin andHauser, 1993).

This preference for engaging customers individuallyinstead of through group research methods is consistentwith 50 years of research that casts doubt on the efficacyof group research methods for customer ideas or needs(Fern, 1982; Griffin and Hauser, 1993; Paulus and Dzin-dolet, 1993; Taylor, Berry, and Block, 1958; Thompson,2003). The two primary face-to-face group techniquesthat may be employed in the ideation phase of innovationare focus groups, generally a group of 4–9 customers orusers assembled to explore user needs, and brainstorminggroups, which may include engineers, customers, con-sultants and/or marketing staff, as well as users, to gen-erate new product or service ideas (Fern, 2001). Bothfocus groups and brainstorming groups are discussed inthis paper because (1) as noted, needs and ideas are keyantecedents of new product and service innovation andare gathered and generated at the onset of the effort; (2)both share the characteristics of face-to-face group inter-action, which have been considered a source of problemswith the techniques; and (3) there is overlap in thesegroups: both may include customers and users, and,despite the definitional distinctions, focus groups are fre-quently used to generate new product or service ideasthrough brainstorming techniques (Durgee, 1992; Fern,2001; McQuarrie and McIntyre, 1986).

Despite strong and consistent evidence denouncinggroup ideation methods, some new product developmentresearchers and consultants seem to consider all customerresearch to be effective. In addition, firms and consultantsare using focus groups as part of new product and serviceideation. This continued use of brainstorming groupsand focus groups to uncover valuable customer needs

Address correspondence to: Gary R. Schirr, Radford University, P.O.Box 6917, Radford VA 24142. E-mail: [email protected]. Tel: 540-808-6630.

* For their time and efforts contributing to this paper, a debt is owed to(1) Dr. Abbie Griffin for her encouragement and thoughtful suggestions; (2)Dr. Edward F. Fern and Dr. Albert L. Page for their encouragement in thiseffort; (3) The American Bankers Association, and Dr. Yao and MaggieKelly, who sponsored 2007 Bank Innovation Survey; (4) The ProductDevelopment and Management Association Research Foundation and toThomas Hustad and Stephen K. Markham of the Foundation for grantingthe author access to the comparative performance assessment study data set,and to Dr. Abbie Griffin, University of Utah, and Dr. Marjorie Adams-Bigelow for their help in analyzing the data; (5) anonymous conferencereviewers who made useful suggestions on earlier versions of this manu-script; and (6) an anonymous reviewer at JPIM who made valuable sug-gestions to improve the paper.

J PROD INNOV MANAG 2012;29(3):473–488© 2012 Product Development & Management AssociationDOI: 10.1111/j.1540-5885.2012.00918.x

and ideas is puzzling given strong evidence illustratingthe ineffectiveness of group production. Why do groupmarket research methods continue to be used in productand service development despite five decades of researchshowing such methods to be inferior to more individual-ized research techniques? The research questions for thispaper are vital to innovation in organizations:

Why are group research methods, focus groups, andbrainstorming used in customer research for NPD ideasdespite extensive evidence of ineffectiveness for ide-ation? Under what circumstances might an organizationuse group market research methods to support innova-tion and new product development?

Two recent articles have reviewed the strong evidenceagainst group brainstorming and then suggested proce-dural changes, expressing hope that electronic groupsmight solve the group problems (Furnham, 2000; Thomp-son, 2003). This paper takes a more fundamentalapproach to the issue: the evidence against groupmethods is reviewed, propositions about application ofgroup methods in research for innovation are developed,and a framework for analyzing new group or hybridmethods for new product and service development (NPD)research is presented.

A Conceptual Study

This paper is a conceptual study to provide propositionsto guide future research about the use of group marketresearch methods as part of a new product developmenteffort, and to provide a theoretical framework for choos-

ing between group and individual ideation methods.The literature on the effectiveness of focus groups andbrainstorming for generating ideas and uncoveringcustomer needs in the context of product developmentis discussed. Possible benefits to organizational marketlearning from group techniques are explored. Ideas fromthe literature of new product development and marketlearning are used to frame the issue of when to employgroup research. The objective of the effort is to developpropositions for future research and tentative guidelineson the proper use, if any, of these research tools.

The analysis emerges from a review of the literatureon group research methods and focus groups in NPD.Most of these articles are empirical studies or meta-analyses of past empirical studies, although conceptualdiscussions are frequent in the meta-analyses and litera-ture reviews. Meta-analyses are theory building as theyderive key effects and themes from previous empiricalstudy. Given the increased acceptance of qualitativemethods in marketing research, a conceptual paper basedin part on empirical studies should not seem unreason-able: theory can be created from data (Glaser, 1992;Glaser and Strauss, 1967; Strauss and Corbin, 1998).Past conceptual work is, of course, also discussed andutilized.

Two secondary data sets, which the author had accessto, are used to illustrate and frame the issues being dis-cussed in the paper. This is not an empirical study; thearticle could have been produced without the use of thesetwo data sets. In the author’s opinion, the statistics andexploratory data help the understanding of the issues,illustrate the problems, and frame choices. The two datasets used are the comparative performance assessmentstudy (CPAS) 2003 data set from the Product Develop-ment and Management Association (Adams-Bigelow,2004) and the results of the 2007 Bank New ServiceDevelopment Survey conducted by the author in coop-eration with the American Bankers Association (Schirrand Page, 2009). Additional information about the twodata sets is available in Appendix A.

The first section of this paper reviews the empiricalcase against group research methods and the use of focusgroups in need discovery, ideation, or idea evaluation forinnovation. The case against traditional group methods isstrong with 50 years of evidence and hundreds of empiri-cal studies, 208 articles in the subperiod 1967–1994 alone(Sutton and Hargadon, 1996). Much of the empiricalevidence is outside the marketing literature, in psychol-ogy and other social science areas, and may have there-fore escaped full notice by marketing researchers andpractitioners; this may explain why group research

BIOGRAPHICAL SKETCH

Dr. Gary Schirr is an assistant professor of marketing at Radford Uni-versity (Va.). He is a member of the Product Development and Manage-ment Association and reviews articles for the leading academic journalof new product development, JPIM. With the help of the AmericanBankers Association, Gary conducted the 2007 Bank New ServiceDevelopment Survey. Before beginning his academic career, Gary wasactively involved in new product and service development in financialservices. Professor Schirr formerly served on the new product commit-tees of two international financial futures exchanges, headed servicedevelopment for an online clearing subsidiary of a major bank, and hadbeen in charge of product development for a leading software vendorserving online trading. From 2001 to 2004, he consulted with start-upfirms involved in online trading and risk management. Gary has a Ph.D.in marketing from the University of Illinois at Chicago. He also has anM.S. in finance from Carnegie Mellon and a B.S. in mathematics fromMiami University. He is coauthor with Albert L. Page of a JPIM articlethat was awarded a 2008 Thomas Hustad Best Paper award.

474 J PROD INNOV MANAG G. R. SCHIRR2012;29(3):473–488

methods are still advocated for early-stage innova-tion without fully acknowledging the proven pitfalls(Furnham, 2000). The second section of the paperdemonstrates that despite the evidence against groupmethods, focus groups are still used in NPD to uncovervaluable customer ideas and needs. Section three dis-cusses efforts to explain the persistence of the use ofgroup methods in the face of the dismal evidence of theirefficacy. In the fourth section, rationales for the use ofgroup methods are linked to an organizational marketlearning model of NPD success. The final section beforethe conclusion discusses the alternatives to face-to-facegroup methods that have been promoted throughthe years to reduce groupthink problems.

Several propositions that emerge in the discussion arepresented in the later sections. These propositions willideally guide future research and provide some insight toinnovators.

Group Research Methods Fail to UncoverNeeds and Kill Ideas

This section presents the extensive evidence against theeffectiveness of focus groups and brainstorming groupsfor the key research missions of product and serviceinnovation: (1) uncovering user needs; (2) generatingideas for innovation; and (3) evaluating alternative ideas.Many studies, primarily empirical tests or meta-analysesof empirical studies of focus group and brainstormingeffectiveness, are cited in the discussions. A summarytable of selected past research articles is included inAppendix B.

Uncovering User Needs Through Focus Groups

Griffin and Hauser (1991) in their working paper thatpreceded the 1993 “Voice of the Customer” article,note that there are often key “exciting needs . . . not voiced. . . (by most customers who) . . . do not consider theirfulfillment a realistic possibility” (p.7) and claim thatthese needs are often key to the success of a project. Theconsensus process within a focus group may inhibitraising key issues and be a partial explanation for thelower quality of ideas generated in focus groups. Consis-tent with the evidence on idea generation in brainstorminggroups, “The Voice of the Customer” found that individu-als were less effective at uncovering user needs in a groupsession (Griffin and Hauser, 1993). A 2003 article sug-gests that the appropriate use of focus groups in innova-tion is to uncover “thoughts held in common” and “lessidiosyncratic ideas” (Bristol and Fern, 2003, p. 447–8).

Especially for more innovative product development,uncovering customer needs and problems is hampered bytheir contextual knowledge and inability to express orunderstand the needs away from the context (Szulanski,2003; von Hippel, 1994). Focus group or brainstormingparticipants are gathered at a central location away fromtheir work or home environment where they normally usethe organization’s goods or services. An article written byan experienced focus group leader declared that focusgroups should not be used for idea generation or screen-ing in innovation (McQuarrie and McIntyre, 1986, 1987).McQuarrie (1991, 1998) later wrote a number of publi-cations on the merit of customer site visits for gatheringideas—visiting a user where the good or service would beused and observing and interviewing one-on-one.

Such sticky or contextual information is likely lost inthese group methods. IDEO, a well-known innovationand design consultancy, is known as an active user ofbrainstorming groups for ideation, as will be discussed ina later section (Hargadon and Sutton, 1997; Kelley,Littman, and Peters, 2001). However, even IDEO advo-cates are using ethnographic methods rather than focusgroups to uncover customer and user needs, to get at thedeep or really interesting needs (Higgins and Koucky,2003; Kelley, 2005).

Group Ideation

Brainstorming—and the underlying techniques of groupresearch employed in brainstorming, focus groups, or“group creativity” exercises—was proposed as a methodto generate ideas (“ideation”) in a book written by anadvertising executive and adjunct professor (Osborn,1957). Osborn claimed that he had empirical evidencethat a group brainstorming by his four key rules—(1)criticism not welcome; (2) freewheeling welcome; (3)quantity of ideas welcome; and (4) combining/improvingideas welcome—could produce twice as many ideas asindividuals brainstorming alone and also produce betterideas (Osborn, 1957; Thompson, 2003). Osborn (1957)indicated that number of ideas alone is a sufficient metricto measure the performance of group ideation becausebetter ideas necessarily result from more ideas. Osbornnever presented the evidence to back his claims for brain-storming’s superiority, but he laid down a gauntlet andclear metrics—number of ideas and quality of thoseideas—that has led to 50 years of research and hundredsof studies testing group ideation methods against indi-vidual ideation. Taylor et al. (1958) conducted experi-ments, finding that the combined results of individualbrainstorming outperformed groups in terms of the

EFFICACY OF GROUP RESEARCH METHODS J PROD INNOV MANAG 4752012;29(3):473–488

number, quality, and uniqueness of the ideas generated.Several studies have found support for two of Osborn’sideas—that the four rules of conduct enhance creativityand that quantity of ideas is linked to quality of ideas(Litchfield, 2008; Mullen, Johnson, and Salas, 1991;Thompson, 2003). However, a wide range of studies havesupported Taylor et al.’s (1958) findings that brainstorm-ing groups produce fewer and lower average quality ideasthan methods employing users individually (Bouchard,1972a, 1972b; Bouchard, Barsaloux, and Drauden, 1974;Bouchard and Hare, 1970; Bristol and Fern, 2003; Diehland Stroebe, 1987; Fern, 1982; Griffin and Hauser, 1993;Mullen et al., 1991; Taylor et al., 1958).

As noted in the introduction of this paper, multiplestreams of research have pointed to the importance ofuser and customer ideas for effective product and serviceinnovation. Experimental results from customer researchreported in the Journal of Marketing Research (Fern,1982, p. 11) found that individual one-on-one interviewswere superior to focus groups in terms of (1) the numberof ideas generated; (2) the quality of ideas generated;(3) dealing with complex concepts; and (4) “thought-generation tasks.” Fern’s (1982) list of four areas inwhich focus groups were inferior to individual interviewsseems to correspond well to what new product develop-ment managers would seek in generating ideas fromcustomer research.

Focus groups reduce the number of unique ideasper subject involved and reduce the quality of theideas generated; the effect is predicted to be more severein complex products (Fern, 1982). Fern’s (1982) find-ings that focus groups reduce ideas and eliminate themost creative ideas are consistent with research resultsabout the effectiveness of group market researchmethods—focus groups and group brainstorming—fromthe two decades before he wrote his article and in thedecades since (Diehl and Stroebe, 1987; Griffin andHauser, 1993; Taylor et al., 1958; Thompson, 2003). Theeffectiveness trade-off is all too clear: for a standard timeperiod of around an hour, individual interviews averaged28 unique ideas (unique ideas within a randomly selectedgroup), members of a four-person focus group averaged20 unique ideas, and members of a six-person groupaveraged only 14 unique ideas (Fern, 1982, p. 7). On thetwo measures of quality—“summed quality” and numberof “good ideas”—the individual interviews scored 50%better than moderated focus groups (p. 8). Fern (1982)was not surprised by these results, noting that they wereconsistent with a series of earlier studies that found thatbrainstorming groups—students, scientists, or businessmanagers—generated fewer ideas and lower quality

ideas than by working alone (Campbell, 1968; Dunnette,Campbell, and Jaastad, 1963; Taylor et al., 1958).McQuarrie and McIntyre (1986) recommended that focusgroups not be used for idea generation or idea screeningin a new product development effort (p. 41). Past researchpresents clear evidence that focus groups are less effec-tive in idea generation or idea testing (Fern, 1982), indefining customer needs (Griffin and Hauser, 1993) or indiscerning customer attitudes toward existing products(Bristol and Fern, 2003).

Quality of Ideas and Best Ideas

Osborn (1957) stressed that quality of ideas was linked toquantity generated, so to measure the effectiveness of anideation technique measuring the quantity of ideas gener-ated was sufficient. A large number of studies have mea-sured both quantity and quality and have supportedOsborn’s quantity–quality link, while strongly rejectinghis claims for the performance of group brainstorming. Ameta-analysis of data from 20 published studies showedthat “quantitative productivity loss is not compensated forby an increase in quality of productivity” and in fact that“productivity loss . . . is highly significant and of strongmagnitude” measured by either quantity or quality of ideasgenerated (Mullen et al., 1991, p. 18). The link betweenquantity and quality is not a simple linear link because ofthe percentage of quality ideas staying constant; empiricalevidence shows that the average quality of ideas generatedthrough individual interviewing or brainstorming is higherthan in groups (Diehl and Stroebe, 1991; Fern, 1982;Lamm and Trommsdorff, 1973; Mullen et al., 1991;Taylor et al., 1958; Thompson, 2003).

A current working paper examines group and hybrid(individual research first then group brainstorming) ide-ation for the quantity, average quality, and very best ideasgenerated (Girotra, Terwiesch, and Ulrich, 2010). Girotraet al. (2010) conclude that an organization interested ingenerating ideas would be “better off relying on the asyn-chronous idea generation of individuals” rather thanusing groups (p. 602). Girotra found that hybrid groupswere better than traditional groups in:

1. Generating more ideas;2. Generating better ideas;3. Generating the best ideas; and4. Discerning the best ideas.

Furthermore, Girotra et al. (2010) found that all of thebenefits of the hybrid technique came from individualideation: idea add-on and building, alleged advantages ofgroup efforts, resulted in lower average quality ideas. The


group effect in ideation does not simply kill ideas, but itdisproportionately eliminates better ideas and the verybest ideas. This is a major problem for group research: ifthe shortfall was only in quantity of ideas—or if thequality of ideas was even proportional to quantity—anorganization could compensate for the ineffectiveness ofgroup ideation by holding additional sessions. However,this quality idea devastation cannot be cured by simplyholding more group sessions. The only way to compen-sate for the reduction in higher quality and the very bestideas is to undertake research at the individual level.

Idea Devastation

The effect of 50 years of empirical evidence on face-to-face group brainstorming methods is summarized inquotes from earlier researchers who have examined theempirical literature. Thompson (2003, p. 100) notes: “40or so years of research on brainstorming has shown thatgroup brainstorming is significantly worse in terms offostering creativity than comparable numbers of solitarybrainstormers.” “Research shows unequivocally thatbrainstorming groups produce fewer and poorer qualityideas than the same number of individuals working alone”(Furnham, 2000, p. 21). Furnham (2000, p. 23) furtherstates that: “The evidence from science suggests thatbusiness people must be insane to use brainstorminggroups.” Academic meta-analyses use more restrainedstatements, but the meaning is still clear: “[Past andcurrent studies] . . . suggest that there is little justificationfor group brainstorming in organizations” (Paulus, Larey,and Ortega, 1995, p. 262).Ameta-analysis of group versusnominal brainstorming studies state that “the long-livedpopularity of brainstorming techniques is unequivocallyand substantially misguided” (Mullen et al., 1991, p. 18).

Before developing the propositions of this paper, it isworth noting a law of group ideation that has been sup-ported by testing Osborn’s (1957) criterion for successfulidea generation in hundreds of studies and multiple meta-analyses (Diehl and Stroebe, 1987, 1991; Lamm andTrommsdorff, 1973; Mullen et al., 1991; Mullen,Symons, Hu, and Salas, 1989):

L1: Group methods, focus groups, or brainstormingproduce fewer and lower average quality ideas than indi-vidual brainstorming or interviews.

Why Do Group Methods Kill Ideas,Especially Good Ideas?

Diehl and Stroebe (1987, 1991) believed that group inter-actions caused the destruction of ideas in a group, spe-

cifically via three hindrances: (1) free-riding byindividuals; (2) evaluation apprehension; and (3) produc-tion blocking (limited time and need to wait for one’sturn). Mullen et al. (1991) argued that it was not groupprocesses but social psychology that affected group ideaproductivity. They noted that multiple studies had shownthat limiting individual brainstorming to the time eachwas allowed in a group did not negate the superiority ofindividual ideation to group efforts in both quantity andaverage quality of ideas. Other efforts to procedurallydeal with the three hindrances of Diehl and Stroebe(1991) similarly did not eliminate the idea gap. Mullenet al. (1991) urged that researchers and practitionersfocus on issues such as self-attention and drive arousalthat may be impacted by a group setting.

In a recent review of group literature, Thompson(2003) states that the key problem in group methods isnot the group or team, “but rather the social-cognitiveprocesses that operate in teamwork” (p. 100). She listsfour major problems (pp. 100–1), the original three fromDiehl and Stroebe (1991), plus an additional factor:

1. Social loafing—people in a group tend not to work ashard as they would individually, especially if theirpersonal contributions will not be obvious in the finalproduct.

2. Conformity—group members desire to be accepted,which leads them to respond with “traditional conser-vative and highly similar ideas.”

3. Production blocking—a group member does not enjoyan “uninterrupted flow of thought . . . they must waittheir turn . . . idea production is blocked.”

4. Downward norm setting—individuals working ingroups “tend to match their performance to the leastproductive member” (Camacho and Paulus, 1995;Paulus and Dzindolet, 1993).

Her first and third factors would seem to predict thelower quantity of ideas from groups, while the second andfourth may explain the lower quality of group ideas. It isperhaps beyond the talents of even the most gifted mod-erators to overcome these group barriers as well as thesocial psychology barriers discussed by Mullen et al.(1991). Certainly, the moderators used in 50 years ofresearch under carefully controlled experimental condi-tions have not succeeded.

Despite Strong Evidence, Group ResearchMethods Are Used in NPD

Despite 50 years of evidence that group methods under-perform by Osborn’s (1957) standards, group methods are


still used in early-stage innovation—to uncover needs,generate ideas, and evaluate ideas. Thompson (2003, p.100) notes that “companies who use brainstorming don’tlike to hear . . . [about the empirical evidence]” and“Despite the empirical evidence for its ineffectiveness,group brainstorming remains popular in business andindustry.” As noted, IDEO argued for individual methodsrather than focus groups to uncover customer needs, butconsultants, including IDEO, continue to promote thebenefits of ideation in brainstorming sessions (Kelleyet al., 2001). A recent article on ideation by prominentNPD academics and consultants even classified focusgroups as a technique to understand the “Voice of theCustomer,” despite the fact that Griffin and Hauser (1993)had marshaled considerable evidence against focus groupsin their famous article (Cooper and Edgett, 2008). A 2005business press article urged companies to “Kill the FocusGroups (BusinessWeek, 2005). Often-benchmarked P&Gonly recently de-emphasized focus groups in favor ofindividual methods such as individual interviews and eth-nography in their innovation efforts (Lafley and Charan,2008).

Table 1 summarizes usage data from the two second-ary data sets referred to in this study. A third of theProduct Development and Management Association(PDMA) membership that participated in the CPASsurvey indicated that their organization uses focus groupsfor customer research in at least half of their NPD efforts;69% use them in at least 25% of the projects. Over half ofthe banks surveyed in the 2007 Bank New Service Devel-opment Study indicated that focus groups were employedin at least half of their service innovation programs.Despite 50 years of evidence about the ineffectiveness ofthe group methods to uncover needs and to generateideas, they are still widely used in customer research forproduct innovation.

Why Are Group Methods Still Used inProduct Innovation?

After 50 years of evidence that groups are ineffective togenerate ideas, evaluate ideas, or uncover customerneeds, several ideas have been proposed to explain whybrainstorming and focus groups have continued to beused for those purposes. Certainly, there are times whenfocus groups can have real value, for example, (1) inconducting research with hard to reach or sensitive sub-jects such as immigrants, children, prisoners, personswith disabilities, or anyone uncomfortable in a one-to-one interview situations; and (2) in situations wheregroup decision making is relevant (Fern, 2001; Stewartand Shamdasani, 1998).

The focus of this paper is the use of group customerresearch for product innovation. The literature showingthat group ideation is ineffective has been discussed.McQuarrie and McIntyre (1986) recommended thatfocus groups not be used for idea generation or ideascreening in a new product development effort (p. 41);they may instead may be useful for studying the adoptionand diffusion of selected ideas (p. 42), far later in theNPD process. Two follow-up articles in response to theoriginal article provided further support for the argumentthat focus groups were not good instruments for the frontend of new product development (McQuarrie and McIn-tyre, 1986; Seymour, 1987). Group interaction in a focusgroup leads to a “shared perspective” on the issues dis-cussed in the group (Calder, 1977, p. 362). McQuarrieand McIntyre (1986) and Morgan (1996) suggest thatthe research topics should fit the group dynamics, whereideas and attitudes move toward a consensus. A possibleeffective use of the focus group procedure might be tohelp group customers into different categories (Bristoland Fern, 2003). This approach—using the group natureof these methods—is logical but would limit their useto very specific applications. Aside from very specificapplications (1) when customers are a protected group or(2) when product use will be a group decision, theresearch question of this paper remains: why are groupresearch methods still widely used in innovationresearch?

Empirical Results Have Not Been Disseminated toBusiness Researchers and Practitioners

As noted in the introduction to this paper, most of thehundreds of papers and multiple meta-analyses that haveshown empirical evidence have appeared in journalsoutside of the literature of marketing, new product devel-

Table 1. Use of Focus Groups for Customer Research inNew Product Development

CPAS-PDMA ABA 2007

Diverse Goods Bank Survey

Almost never 31% 47%Around 25% of the time 36% 24%Around 50% of the time 13% 15%Around 75% of the time 10% 7%Almost always 10% 7%Mean on 1–5 scale 2.3 2.0

CPAS, comparative performance assessment study; PDMA, ProductDevelopment & Management Association; ABA, American BankersAssociation.


opment, or business; many have been published in socialpsychology or applied psychology publications. Busi-nesspersons as well as business researchers may not befully aware of the strong evidence against the effective-ness of group ideation methods. Furthermore AdrianFurnham (2000, p. 26) notes that “most textbooks seemreluctant to discuss the damning research.” Furnham(2000) suggests that the ignorance of businesspersons ofthe extensive empirical evidence of the ineffectiveness ofgroup methods, especially when combined with the ten-dency of group participants to overestimate group effec-tiveness (Paulus, Dzindolet, Poletes, and Camacho,1993), may be a potent combination sustaining groupresearch.

Low Cost or Faster Innovation?

A common rationale for the use of group methods togenerate ideas is low cost or speed in collecting informa-tion: lower cost or faster results may compensate for lesseffective ideation and fewer good ideas than from indi-vidual interviews (Fern, 1982; Krieg, 1981; McQuarrieand McIntyre, 1986). However, a review of the groupresearch literature casts doubt on this rationale. In “TheVoice of the Customer,” a field experiment (Silver andThompson, 1991) to generate ideas for a complex pieceof office equipment was cited (Griffin and Hauser, 1993,p. 7) as well as the ideation experiments from Fern(1982). As a general rule, Griffin and Hauser (1993)found that groups of any size, 1–8, generated roughly thesame number of ideas per hour. A study comparing indi-vidual interviews and focus groups suggested that evenwith the reduced quantity and quality of ideas, “a numberof logistical factors, such as location of the interviews,the mobility of the participants, the flexibility of theirschedules, would determine which study would actuallybe easier to accomplish” (Morgan, 1996).

Girotra et al. (2010) suggested an “asynchronousidea generation of individuals” for idea generation (p.602). Because individual brainstorming or interviewingrequires little training (Fern, 1982), ideas could be col-lected by salespersons or customer services representa-tives during normal calls. When speed is essential, itmay be faster to gather six to eight customers in onelocation for a couple hours than to visit each one at theirsites. However, the cost of an experienced moderator,taping, the facilities, and time reviewing the tapes mustalso be considered, as well as the loss of better and thevery best ideas means the group effort is likely moreexpensive and less effective. If speed is more importantthan effectiveness, there is still the option to employ

the nominal group technique (NGT), which is indivi-dual brainstorming within and group and which hasshown to be almost as effective as individual interviewsor individual brainstorming; furthermore, NGT doesnot require a high-priced expert moderator (Claxton,Ritchie, and Zaichkowsky, 1980; Delbecq, Van de Ven,and Gustafson, 1975). Thus, neither the cost nor thespeed is a justification for the use of traditional face-to-face group research methods in innovation.

The lower cost argument is really only convincingwhen an organization chooses to have a managementteam actively participate in the research: for example,sometimes, a management team will observe a focusgroup session behind two-way mirrors. There is no evi-dence that such observation improves the number orquantity of ideas generated in group sessions. If anorganization is determined to have a team of executivesobserve the idea generation, the cost and opportunitycost of their time may make individual one-on-oneinterviews prohibitively expensive. However, even underthe constraint that an organization wanted multipleexecutives to monitor ideation, some of the quasi-individual techniques such as nominal groups haveproven to be more effective than focus groups or groupbrainstorming. So again, even with management teaminvolvement, a cost argument does not mandate idea-killing group sessions.

It is worth repeating that earlier cited evidence showedthe effectiveness of nominal groups or NGT with a singleleader, or even leader-less individual idea generation.Organizations that choose to have management teamsdirectly observe ideation efforts might have a rationalefor the involvement of the teams other than the Osborn(1957) criteria of number and quantity of ideas generated,such as eliciting the help of the team members in thedissemination and implementation of ideas within theorganization. These issues are discussed later, in the orga-nizational market learning discussion.

Ideas Are Lost, But Consultants Profit fromModerated Group Research

An ethnographic study of IDEO, a renowned design-consulting firm that uses group brainstorming, acknowl-edges the compelling evidence against the effectivenessof group brainstorming for idea generation (Sutton andHargadon, 1996). Sutton and Hargadon (1996) exploredhow and why IDEO uses brainstorming, believing thatOsborn’s idea generation success metrics must be inad-equate in light of the continued use of group research forideation. They found “six important consequences . . . not


labeled as effectiveness outcomes [in prior research]: (1)supporting the organizational memory of design solu-tions; (2) providing skill variety for designers; (3) sup-porting an attitude of wisdom (acting with knowledgewhile doubting what one knows); (4) creating a statusauction (a competition for status based on technical skill);(5) impressing clients; and (6) providing income for thefirm” (p. 685).

Sutton and Hargadon’s (1996) “consequences” areconsistent with past findings on the illusion of groupeffectiveness and organizational credibility of groupoutput, and individual positive affect for group informa-tion (Paulus, 2006; Paulus et al., 1993; Thompson,2003). Again, one way to read this list of group ideationconsequences is to worry that consultants are takingadvantage of the ignorance of businesspeople of thecompelling evidence against group methods (Furnham,2000): Consultants make money moderating groupresearch (Sutton and Hargadon’s consequence number6); illusion of effectiveness makes the service easy to sell(number 5); and the sessions advance the skills of theconsultants that they can apply on future assignments(numbers 1 and 2).

While the author believes from experience and anec-dotal evidence that this cynical view of overselling byconsultants is sometimes correct, organizational memoryand skills (numbers 1 and 2) and confidence in the infor-mation (numbers 3 and 5) may help a firm’s innovationefforts in some circumstances. There is likely value intraining your consultants and taking advantage of theirother expertise. In addition, organizational memory couldbe valuable within an organization that has ongoing inno-vation efforts. Furnham (2000) suggested that a benefit ofgroup methods might be organizational acceptance ofresults. Indeed, “the illusion of productivity might be avaluable affective outcome in some work contexts”(Litchfield, 2008, p. 652; Paulus and Yang, 2000; Suttonand Hargadon, 1996).

The Illusion of Group Effectivity

The illusion of group effectivity is thus one possi-ble rationale for the continuing popularity of groupresearch—brainstorming and focus groups are enjoyable,and participants feel that they have been creative(Stroebe, Diehl, and Abakkoumkin, 1992). Group partici-pants expect their efforts to be productive before com-mencing [they have not reviewed the empirical evidence],believe after the fact that they were productive, and takeexcess ownership of the results: they take individualcredit for the group results (Paulus and Dzindolet, 1993;

Paulus et al., 1993, 1995). Thompson (2003) studied thetapes of multiple brainstorming sessions and found apattern of positive feedback to all participants that likelyleft everyone with a good feeling and confirmed Pauluset al.’s (1993) findings that individuals seemed to takepersonal credit for the group results. Fern (2001) notesthe same effect in focus group participants involved inideation or needs discovery: evidence shows that theywere not as productive as working alone, but they enjoythe process more and have more confidence in the infor-mation obtained. Fern (2001) also notes that even orga-nizational members not involved in the group researchbut merely observing or viewing the tapes give greatercredibility to group output.

Even though ideas are being lost through groupresearch methods, individuals involved in innovation inorganizations feel that the efforts are successful, believein the information collected, and enjoy the process. Com-bined with ignorance of the empirical evidence againstgroup methods (Furnham, 2000), this credibility andaffect for group methods may lead to overuse of themethods. However, the credibility and affect for groupmethods by participants and nonparticipants in an orga-nization (Fern, 2001), even if not merited, might provideadvantages to an organization. A few articles have sug-gested including group methods as a supplement to indi-vidual ideation due to the organization affect andcredibility (Paulus et al., 1993; Paulus and Yang, 2000),but they have not suggested the exact benefit to innova-tion or how to evaluate the trade-off between good ideasand organizational affect and credibility. Furnham (2000)suggests that it is reasonable for firms to use groupmethods to “increase decision acceptance” and getdiverse groups in an organization on board (p. 26).

These suggested benefits of group methods seem to fitan organizational market learning model of innovationsuccess that has been used in recent new product devel-opment research. The next section of this paper exploreshow the illusion of group effectiveness, belief in theinformation collected, and organizational learning andskill-building, might fit into an organizational marketinglearning model of innovation success. L1 should alwaysbe kept in mind, however: group methods will producefewer and lower quality ideas.

Market Learning and Assimilation of Ideas

An organizational market learning model stressing theimportance of information acquisition, dissemination,and implementation has been applied to new productdevelopment (Lynn, Skov, and Abel, 1999; March, 1991;


Slater and Narver, 1995; Veldhuizen, Hultink, andGriffin, 2006). A process that gathers customer informa-tion can affect new product development directly throughthe needs collected and the ideas generated and indirectlythrough its effect on dissemination and acceptance of thenew ideas (Berchicci and Tucci, 2010). Three recentstudies applying the organizational market learningapproach to new product development have found that (1)a stage-gateTM type process may hinder learning andtherefore negatively impact long-term organizationalinnovation (Sethi and Iqbal, 2008); (2) organizations canenhance high-tech innovation by “influencing project pri-ority and flexibility, and by reducing interdepartmentalconflict” (Veldhuizen et al., 2006, p. 353); and (3) teamvalues may affect NPD by affecting how the informationis shared (Berchicci and Tucci, 2010).

This organizational market learning frameworkexplains the value of fun, credibility, and idea dissemina-tion from group research techniques. In a group process,information is disseminated to multiple members of anorganization at once: in the case of internal brainstorm-ing, multiple members of the organization participate andshould buy into the results (Thompson, 2003). Similarmultiple members of the organization will be involved infocus group sessions, and tapes of sessions are oftenshown to organization members. Even organizationalnonparticipants in group research seem to have positiveeffects to grant increased credibility to group researchresults (Fern, 2001). Adapting figure 5 from Berchicciand Tucci (2010, p. 987), Figure 1 contrasts the effect of1:1 methods on NPD success through a market learningmodel to the effect from group methods.

In Figure 1, the use of 1:1 customer researchmethods has a strong positive direct effect on both

information acquisition and the ultimate NPD perfor-mance; group methods have only moderate direct posi-tive effects on acquisition and performance. However,group methods also have positive effects on the twolater stages of market learning, information dissemina-tion and sharing and implementation due to the cred-ibility and positive effects for the group process withinthe organization. Because project priorities and theinnovation process is driven by teams, disseminationand implementation of innovation ideas is important.The illusion of group effectivity (Stroebe et al., 1992)may have a positive effect in promoting the implementof ideas. The model depicted in Figure 1 provides arationale for employing group methods of ideation, atleast on a limited basis:

P1: Group methods for ideation enhance the dissemina-tion of information in an organization.

P2: Group methods for ideation enhance informationsharing in an organization and lead to faster implemen-tation of new ideas.

P1 and P2 provide a possible rationale for continueduse of group methods of market research in NPD: Anorganization may want to first engage in individualresearch activities and then use group methods. Satura-tion is a term used in qualitative analysis to describe thepoint at which most of the good ideas have been gener-ated and further research is high cost relative to newdiscovery (Berg, 2004). Griffin and Hauser (1993) foundthat 20 subjects were generally sufficient to saturatenew product ideas; in narrow markets with very similarusers, as few as eight subjects might be sufficient. Thus,a logical approach to group methods might be to firstgather ideas from 1:1 methods and then hold some focusgroups or brainstorming sessions to aid market learning:“saturate, and then meet.” Saturate, then meet wouldseem to be particularly well suited for internal organiza-tion brainstorming.

P1 and P2 and the idea of saturation suggest somepossible specialized uses for group ideation methodsincluding:

1. When the gain from (1) enhancing the dissemi-nation of information and (2) the sharing and imple-mentation of ideas is believed to outweigh the costsof lost ideas and more creative ideas from employingthe methods;

2. To advance the dissemination and implementation ofthe information, after individual ideation methodshave reached saturation;

Figure 1. Market Learning Framework for Weighing GroupResearch versus 1:1 Customer Research


3. To validate decisions—for example, when aninnovation has already been decided upon bysenior management and cooperation or organization“buy-in” is sought; and

4. When the product innovation is small orincremental.

Future research, both qualitative and quantitative,should explore whether group idea-generation methodsare being used for these specialized uses, and if so, howeffective they are in these uses.

Trade-off of Group versus Individual Techniques inDifferent Innovation Situations

Focus groups and brainstorming methods kill ideas, andeliminate the better ideas (Diehl and Stroebe, 1987; Fern,1982; Griffin and Hauser, 1993; Thompson, 2003) andthe very best ideas (Girotra et al., 2010; Taylor et al.,1958). Group gatherings remove the ideation effort fromuseful context and reduce the chances of collecting stickyor contextual information, which is often essential tomore innovative ideas (Szulanski, 2003; von Hippel,1994). Ideation groups are not good at evaluating ideas(Fern, 2001; Fern 1982; Girotra et al., 2010; McQuarrieand McIntyre, 1986). Clearly, an NPD team looking forideas for more innovative projects should be skeptical ofthese methods.

However, as suggested in the preceding propositions,there may be circumstances when the positive affect,credibility, and organizational acceptance from groupmethods compensate for reduced quantity and quality ofideas generated. This trade-off could differ in varyinginnovation situations. For example, consultants are oftenhired to “sell” ideas that senior management has alreadyaccepted or that are viewed as obvious needed changesto line managers and staff (O’Shea and Madigan, 2002).The credibility of group processes could be used as atool to achieve employee buy-in for such engagements:what could be more powerful than to note that the ideascame from a brainstorming session of the managers orfocus groups that they were involved in? Similarly, theloss of the better ideas and the very best ideas may bemore costly to a more innovative project than an incre-mental new product or service. This discussion leads tothe following proposition:

P3: Group ideation methods will be less frequently usedin more innovative or radical new product developmentand explorative innovation efforts than for more incre-mental innovation.

User Input as a Scarce Resource:Business-to-Business (B2B) Firms

Small firms, startup firms, and some B2B organizationshave a limited number of customers or potential users tosolicit ideas, needs, and problems (McQuarrie, 1991;Oliva, 2005). Even firms with larger customer bases pur-suing novel or more innovative ideas may find availablehours of lead users (von Hippel, 1986) to be limited.There is actually a reduction of productivity as the groupsget larger (Mullen et al., 1991). This idea devastationshould be even more alarming when customer time isscarce. Using this framework of scarcity of user time forideation, another proposition emerges:

P4: Group ideation methods will be less frequently usedas a part of a new product development effort by B2Bfirms or firms targeting a small number of customers.

The CPAS data set allows an exploratory look at thisproposition. The B2B members of the PDMA who par-ticipated in the survey were indeed less likely to use focusgroups than business-to-consumer participants. This isillustrated in Table 2.

Removing the Group from GroupIdeation Methods

Osborn’s original four principles of brainstorming andsimilar rules for focus groups are in part attempts toovercome problems caused by group dynamics (Osborn,1957). As evidence continues to collect that groupmethods kill ideas, other rules or approaches havebeen proposed to improve creativity. Thompson (2003,pp. 102–3) summarized some of the more promisingapproaches, which include:

1. Diversity—participants with different expertise willtend to look at things differently. In addition, diverseindividuals may be less likely to bond into tight teams.

Table 2. Use of Groups Methods for Innovation for B2Bversus B2C Firms Group Methods Usage by B2B versusB2C Firms in CPAS Data Set

B2B Firms B2C Firms

Focus groups: mean use (1–5 scale) 2.0 3.0Focus groups: “almost never use” % 39% 13%Brainstorming: mean use (1–5 scale) 1.5 2.1Brainstorming: “almost never use” % 67% 36%

B2B, business-to-business; B2C, business-to-consumer.


2. Analogical reasoning—this cognitive approach mayhelp to access deep or contextual knowledge.

3. Brainwriting—group members individually writedown ideas silently. This is an attempt to gather indi-vidual ideas before group pressures kill them.

4. NGTs—groups engage in brainwriting to collectideas, and after discussing them, they individually,silently rate the ideas.

5. Delphi—Delphi is the ultimate NGT: group membersnever meet face-to-face.

According to Mullen et al. (1991), totally eliminatingthe face-to-face interaction is key: defining “alonenominal groups” (standard nominal groups) as indi-vidual brainstormers and “together nominal groups” asgroups sitting together doing individual activity, thismeta-analysis found that together nominal groups hadless productivity loss than group brainstorming butstill underperformed the alone nominal groups. Theseefforts to enhance group methods raise some interest-ing definitional problems: when has a method ceasedto be a group method? An important topic worthy offuture study is whether or to what degree the groupbenefits of improved dissemination and implemen-tation of ideas persist in “nongroup group approaches”such as nominal groups, Delphi, and markets for ideas(Armstrong, 2006b; Green, Armstrong, and Graefe,2007).

Researchers and practitioners should investigate theuse of nongroup group techniques, often electronic orvirtual techniques enabled by the internet, such asDelphi and idea markets (Armstrong, 1996, 2006a,2006b; Thompson, 2003). Perhaps nongroup grouptechniques—Delphi, idea markets, or online limited-interaction brainstorming or focus groups—can providea superior trade-off of ideas generated and dissemina-tion and implementation of the ideas. However, ifthese techniques completely remove group effects,then they may be merely considered to be more effi-cient electronic or online interviews or individualbrainstorming.

Evaluating New, Hybrid, and/or OnlineIdeation Methods

The discussion of organizational market learning sug-gests that group methods may aid in the disseminationand implementation of ideas. If these benefits to groupmethods are shown to exist, than it may be possible thatsome of the hybrid techniques combine benefits of indi-vidual and group ideation methods. These new or hybrid

techniques should thus be evaluated against benefitsfrom both individual techniques and traditional grouptechniques:

Electronic groups, nominal groups and any proposedprocedures to reduce idea loss from group researchshould be evaluated by three criteria: (1) Osborn’smeasures—quantity and quality of ideas relative to indi-vidual brainstorming; (2) cost by time and resourcesused; and (3) to what extent the new methods advancedissemination and implementation by being fun, attract-ing management attention, and aiding firm retention,compared to traditional face-to-face group methods.

Conclusion

A review of selected empirical studies, and multiplemeta-analyses and reviews covering hundreds of empiri-cal studies over the past 50 years, established that grouptechniques are not effective for key early inputs into orga-nizational innovation: (1) uncovering user needs; (2) gen-erating new product or service ideas; and (3) evaluatingideas. Group ideation has been shown to be inferior toindividual ideation whether measured by the number ofideas generated, the average quality of ideas generated,generating the very best ideas, or evaluating ideas. Thisreview of the empirical research, including the summarytable in Appendix B, should address the need to furtherdisseminate the strong evidence against group methodssuggested by Furnham (2000).

This paper also showed that a review of the empiricalevidence did not support the most common justificationfor the continued use of group methods for ideation andneeds discovery, the alleged low cost and speed of groupmethods. Individual methods that should be at least asquick and low cost as traditional group research methodsare more productive in generating ideas and uncoveringuser needs. Cost and speed do not justify group methods;the use of group methods can only be justified by factorsother than the output of ideas and information.

Group methods have been shown to be fun andbelieved productive by participants who take undocredit for the results; even nonparticipant observers findgroup results credible. Applying an organizationalmarket learning model to an organization’s innovationeffort, it seems likely that group methods may promotedissemination and implementation of information. A setof propositions are presented about the trade-offs ofindividual and group methods for ideation and need dis-covery that should suggest future research and guidancefor practitioners.


Group methods may have a role in the early stages ofinnovation to promote dissemination and implementationof user information, but ideally only after information hasbeen gathered through individual research methods.Online, virtual, and pseudo-group methods should beevaluated both on idea-generation versus individualbrainstorming and interviews and on organizational affectand credibility to further the implementation and dissemi-nation of information compared with traditional groupmethods.

Implications for Future Research

The (fuzzy) front end of product innovation, where userneeds are discovered and product ideas generated andpartially screened, is recognized as a vital process ofinnovation that is relatively unknown and understudied(Koen et al., 2001; Page and Schirr, 2008; Reid and deBrentani, 2004). A recent study of the front-end processfound that enhancing inputs positively impacted theprocess while formal process controls inhibited an effec-tive front-end effort (Poskela and Miia, 2009). A primarycontribution of this paper is to summarize the extensiveresearch from social psychology on the efficacy of groupresearch for ideation for the front-end innovation process.This suggests that individual methods be substitutedfor—or at minimum be used to augment—traditionalgroup methods for soliciting user inputs. In addition, anorganizational market learning framework suggestedother areas of research on ideation methods and measuresto supplement the Osborn (1957) metrics of ideationsuccess.

From the organizational market learning discussion,there emerged four propositions about the proper use ofgroup methods for ideation in a product innovation effort.These propositions should translate into testable hypoth-eses in experimental or quantitative research efforts. Inaddition, within the discussion of the organizational mar-keting learning model, a series of potential niche uses forgroup methods was suggested that could be explored inqualitative studies of organizational product innovation.For example: Do group methods suit incremental productinnovation?

Individual interviews, individual brainstorming,or even the NGT outperform group methods forideation—this should be considered a law of idea genera-tion for NPD. There are arrays of individual engagementtechniques (ethnography, lead user analysis, experimen-tation, etc.) that have been presented in the product inno-vation literature to elicit deep or sticky information fromusers (Szulanski, 2003; von Hippel, 1998). These tech-

niques have generally been observed in a limited numberof case studies. It is time to test these promising methodsagainst individual interviews as the group methods havebeen tested. Product innovation researchers and practitio-ners would benefit from experimental and quantitativetests and ultimately meta-analyses of the efficacy of theseindividual engagement methods. Similarly, the online,electronic, or virtual “group” methods need to be sub-jected to rigorous testing. It seems reasonable that Delphiresearch for experts and various forms of online brain-storming for users that minimizes group interaction mayproduce results similar to individual interviews ornominal groups, but empirical evidence of their effective-ness is needed.

Finally, the discussion of organizational market learn-ing suggested supplemental measures of organizationalmembers “buy-in” to aid in the dissemination and imple-mentation of ideas. As suggested, group methods shouldbe evaluated for these benefits. More importantly, someof the online and virtual group methods should be evalu-ated as well. Do tapes and records from nominal groupsessions lead to better organizational dissemination thando output from individual interviews or solo brainstorm-ing? If so, is there a benefit to the product innovationeffort beyond the quality of ideation? Does Delphiresearch and forms of online brainstorming that limitgroup interaction still lead to increased acceptance fromparticipants and observers? Research on these questionscould provide further illumination of need discovery andidea generation at the front end of product innovation.

Implications for ProductInnovation Practice

Managers of NPD or organizational innovation should bewary of dependence on group techniques, whether focusgroups of potential customers or brainstorming of stake-holders, to generate or evaluate ideas for innovation. Agroup brainstorming session among organizationalmembers may be a good way to build enthusiasm forchange, but it is unlikely to generate the quantity of ideasor more importantly the most innovative ideas. Organi-zational members involved in NPD should visit customersites and engage in one-to-one interviewing to understandcustomer needs and elicit their ideas (Griffin and Hauser,1993; McQuarrie, 1998). When group techniques areused, urge the research leaders to employ techniques suchas brainwriting—individually generating ideas beforethe group session—that have been shown to reduce theidea devastation resulting from the use of group researchmethods.


The evidence on the relative efficacy of researchmethods indicates that product innovation teams shouldemploy one-on-one discussions with users and perhapstry out online interview and “individual brainstorming”techniques to directly solicit ideas for innovation. Asnoted in the previous section, additional research isneeded on new one-on-one engagement techniques, butNPD managers might experiment with methods designedto more deeply engage users in ideation such as ethnog-raphy (Rosenthal and Capper, 2006), experimentation(Thomke, 2003), and lead user methods (von Hippel,1986). If an innovation team has funds allocated tohire “expert moderators” for group sessions, it shouldconsider diverting some of the funds to instead hirean anthropologist to train employees on ethnographictechniques.

References

Adams-Bigelow, M. 2004. Comparative performance assessment study(CPAS). In Foundation report, ed. M. Adams-Bigelow. Chicago:Product Development and Management Association Foundation.

Adams-Bigelow, M., and D. Boike. 2004. PDMA foundation’s comparativeperformance assessment. Visions 28 (3): 26–29.

Armstrong, J. S. 1996. Factors affecting new product forecasting accuracyin new firms: William B. Gartner, and Robert J. Thomas, 1993, Journalof Product Innovation Management, 10, 35–52. International Journalof Forecasting 12 (2): 321–22.

Armstrong, J. S. 2006a. Findings from evidence-based forecasting:Methods for reducing forecast error. International Journal of Forecast-ing 22 (3): 583–95.

Armstrong, J. S. 2006b. How to make better forecasts and decisions: Avoidface-to-face meetings. Foresight 5: 3–8.

Berchicci, L., and C. L. Tucci. 2010. There is more to market learning thangathering good Information: The role of shared team values in radicalproduct definition. Journal of Product Innovation Management 27 (7):972–90.

Berg, B. L. 2004. Qualitative research methods for the social sciences (5thed.). Boston, MA: Pearson.

Bouchard, T. J. 1972a. A comparison of two group brainstorming proce-dures. Journal of Applied Psychology 56 (5): 418–33.

Bouchard, T. J. 1972b. Training, motivation, and personality as determi-nants of the effectiveness of brainstorming groups and individuals.Journal of Applied Psychology 56 (4): 324–31.

Bouchard, T. J., J. Barsaloux, and G. Drauden. 1974. Brainstorming pro-cedure, group size, and sex as determinants of the problem-solvingeffectiveness of groups and individuals. Journal of Applied Psychology59 (2): 135–38.

Bouchard, T. J. Jr., and M. Hare. 1970. Size, performance, and potential inbrainstorming groups. Journal of Applied Psychology 54 (1): 51–55.

Bristol, T., and E. F. Fern. 2003. The effects of interaction on consumers’attitudes in focus groups. Psychology and Marketing 20 (5): 433–54.

BusinessWeek 2005. Shoot the focus group. Business Week. New York:McGraw-Hill.

Calder, B. J. 1977. Focus groups and the nature of qualitative marketingresearch. Journal of Marketing Research 14 (3): 353–64.

Camacho, L. M., and P. B. Paulus. 1995. The role of social anxiousness ingroup brainstorming. Journal of Personality and Social Psychology 68(6): 1071–80.

Campbell, J. 1968. Individual versus group problem solving in an industrialsample. Journal of Applied Psychology 52 (June): 205–10.

Claxton, J. D., J. R. B. Ritchie, and J. Zaichkowsky. 1980. The nominalgroup technique: Its potential for consumer research. Journal of Con-sumer Research 7 (3): 308–13.

Cooper, R. G., C. J. Easingwood, S. Edgett, E. J. Kleinschmidt, and C.Storey. 1994. What distinguishes the top performing new products infinancial services. Journal of Product Innovation Management 11 (4):281–99.

Cooper, R. G., and S. J. Edgett. 1996. Critical success factors for newfinancial services. Marketing Management 5 (3): 26–39.

Cooper, R. G., and S. J. Edgett. 2008. Ideation for product innovation: Whatare the best methods? Visions 32 (1): 12–17.

Delbecq, A., and A. H. Van de Ven. 1971. A group process model forproblem identification and program planning. Journal of AppliedBehavioral Science 7: 466–92.

Delbecq, A. L., A. H. Van de Ven, and H. Gustafson. 1975. Guidelines forconducting NGT meetings. In Group techniques for program planning,ed. A. L. Delbecq, A. H. Van de Ven and H. Gustafson. Glenview, IL:Scott Foresman.

Diehl, M., and W. Stroebe. 1987. Productivity loss in brainstorming groups:Toward a solution of a riddle. Journal of Personality and Social Psy-chology 53: 497–509.

Diehl, M., and W. Stroebe. 1991. Productivity loss in idea-generatinggroups: Tracking down the blocking effect. Journal of Personality andSocial Psychology 61 (3): 392–403.

Dunnette, M. D., J. Campbell, and K. Jaastad. 1963. The effect of groupparticipation on brainstorming effectiveness for two industrial samples.Journal of Applied Psychology 47 (January): 30–37.

Durgee, J. 1992. New product ideas from focus groups. Journal of Productand Brand Management 1 (3): 24–29.

Fern, E. F. 1982. The use of focus groups for idea generation: The effects ofgroup size, acquaintanceship, and moderator on response quantity andquality. Journal of Marketing Research 19 (1): 1–13.

Fern, E. F. 2001. Advanced focus group research. Thousand Oaks, CA:Sage.

Furnham, A. 2000. The brainstorming myth. Business Strategy Review 11(4): 21–28.

Girotra, K., C. Terwiesch, and K. T. Ulrich. 2010. Idea generation and thequality of the best idea. Management Science 56 (4): 591–605.

Glaser, B. G. 1992. Basics of grounded theory analysis: Emergence vs.forcing. Mill Valley, CA: Sociology Press.

Glaser, B. G., and A. L. Strauss. 1967. The discovery of grounded theory:Strategies for qualitative research. Chicago: Aldine Publishing.

Green, K. C., J. S. Armstrong, and A. Graefe. 2007. Methods to elicitforecasts from groups: Delphi and prediction markets compared. Fore-sight 8 (1): 17–20.

Griffin, A., and J. R. Hauser. 1991. The voice of the customer (WP #3449-92). Sloan Working Papers, Cambridge.

Griffin, A., and J. R. Hauser. 1993. The voice of the customer. MarketingScience 12 (1): 1–27.

Hargadon, A., and R. I. Sutton. 1997. Technology brokering and innovationin a product development firm. Administrative Science Quarterly 42(4): 716–49.

Henard, D. H., and D. M. Szymanski. 2001. Why some new products aremore successful than others. Journal of Marketing Research (JMR) 38(3): 362–80.

Higgins, A., and S. Koucky. 2003. Forget focus groups. Just watch users,innovator says. Machine Design 75 (6): 28.

Kelley, T. 2005. The ten faces of innovation. New York: Doubleday.

Kelley, T., J. Littman, and T. Peters. 2001. The art of innovation: Lessonsin creativity from IDEO, America’s leading design firm. New York:Doubleday Business.


Koen, P., G. Ajamian, R. Burkart, A. Clamen, J. Davidson, R. D’Amore, C.Elkins, K. Herald, M. Incorvia, A. Johnson, R. Karol, R. Seibert, A.Slavejkov, and K. Wagner. 2001. Providing clarity and a commonlanguage to the “fuzzy front end.” Research-Technology Management44 (2): 46–55.

Krieg, P. 1981. A look at focus groups. Chicago, IL: Bank MarketingAssociation.

Lafley, A. G., and R. Charan. 2008. The game-changer: How you can driverevenue and profit growth with innovation. New York: Crown Business.

Lamm, H., and G. Trommsdorff. 1973. Group versus individual perfor-mance on tasks requiring ideational proficiency (brainstorming): Areview. European Journal of Social Psychology 3 (4): 361–88.

Litchfield, R. C. 2008. Brainstorming reconsidered: A goal-based view.Academy of Management Review 33 (3): 649–88.

Lynn, G. S., and J. G. Morone. 1996. Marketing and discontinuous inno-vation: The probe and learn process. California Management Review38 (3): 8–37.

Lynn, G. S., R. B. Skov, and K. D. Abel. 1999. Practices that support teamlearning and their impact on speed to market and new product success.Journal of Product Innovation Management 16 (5): 439–54.

March, J. G. 1991. Exploration and exploitation in organizational learning.Organization Science 2 (1): 71–87.

McQuarrie, E. F. 1991. The customer visit: Qualitative research business-to-business marketers. Journal of Marketing Research 3 (2): 121–35.

McQuarrie, E. F. 1998. Customer visits (2nd ed.). Thousand Oaks, CA:Sage.

McQuarrie, E. F., and S. H. McIntyre. 1986. Focus groups and the devel-opment of new products by technologically driven companies: Someguidelines. Journal of Product Innovation Management 3 (1): 40–7.

McQuarrie, E. F., and S. H. McIntyre. 1987. What focus groups can andcannot do: A reply to Seymour. Journal of Product Innovation Man-agement 4 (1): 55–60.

Morgan, D. L. 1996. Focus groups. Annual Review of Sociology 22 (1):129–52.

Mullen, B., C. Johnson, and E. Salas. 1991. Productivity loss in brainstorm-ing groups: A meta-analytic integration. Basic and Applied Social Psy-chology 12 (1): 3–23.

Mullen, B., C. Symons, L.-T. Hu, and E. Salas. 1989. Group size, leader-ship behavior, and subordinate satisfaction. Journal of General Psy-chology 116 (2): 155–69.

O’Shea, J., and C. Madigan. 2002. Dangerous company: The consultingpowerhouses and the businesses they save and ruin. Darby, PA: DianePublishing.

Oliva, R. A. 2005. Business marketing “straight up.” State College, PA:ISBM, Pennsylvania State University.

Osborn, A. F. 1957. Applied imagination: Principles and procedures ofcreative problem solving. New York: Charles Scribner’s Sons.

Page, A. L., and G. R. Schirr. 2008. Growth and development of a body ofknowledge: Sixteen years of new product development research 1989–2004. Journal of Product Innovation Management 25 (3): 233–48.

Paulus, P. B. 2006. Putting the brain back into brainstorming. AssociationsNow 2 (12): 24–28.

Paulus, P. B., and M. T. Dzindolet. 1993. Social influence processes ingroup brainstorming. Journal of Personality and Social Psychology 64(4): 575–86.

Paulus, P. B., M. T. Dzindolet, G. Poletes, and L. M. Camacho. 1993.Perception of performance in group brainstorming: The illusion ofgroup productivity. Personality and Social Psychology Bulletin 19 (1):78–89.

Paulus, P. B., T. S. Larey, and A. H. Ortega. 1995. Performance andperceptions of brainstormers in an organizational setting. Basic andApplied Social Psychology 17: 249–65.

Paulus, P. B., and H.-C. Yang. 2000. Idea generation in groups: A basis forcreativity in organizations. Organizational Behavior and Human Deci-sion Processes 82 (1): 76–87.

Payne, A. F., K. Storbacka, and P. Frow. 2008. Managing the co-creation ofvalue. Journal of the Academy of Marketing Science 36 (1): 83–96.

Poskela, J., and M. Miia. 2009. Management control and strategic renewalin the front end of innovation. Journal of Product Innovation Manage-ment 26 (6): 671–84.

Prahalad, C. K., and V. Ramaswamy. 2004. Co-creation experiences: Thenext practice in value creation. Journal of Interactive Marketing 18 (3):5–14.

Reid, S. E., and U. de Brentani. 2004. The fuzzy front end of new productdevelopment for discontinuous innovations: A theoretical model.Journal of Product Innovation Management 21 (3): 170–84.

Rosenthal, S. R., and M. Capper. 2006. Ethnographies in the front end:Designing for enhanced customer experiences. Journal of ProductInnovation Management 23 (3): 215–37.

Schirr, G. R., and A. L. Page. 2009. Antecedents of service developmentsuccess: A culture-tools-process model. In American Marketing Asso-ciation Winter Conference, ed. K. Reynolds and J. C. White, 27. Tampa,FL: American Marketing Association.

Sethi, R., and Z. Iqbal. 2008. Stage-gate controls, learning failure, andadverse effect on novel new products. Journal of Marketing 72 (1):118–34.

Seymour, D. T. 1987. Focus groups and the development of new productsby technologically driven companies: A comment. Journal of ProductInnovation Man 4 (1): 50–54.

Silver, J. A., and J. C. Thompson Jr. 1991. Understanding customer needs:A systematic approach to the “voice of the customer.” Master’s thesis,MIT: Cambridge, MA.

Slater, S. F., and J. C. Narver. 1995. Market orientation and the learningorganization. Journal of Marketing 59 (3): 63–74.

Stewart, D. W., and P. N. Shamdasani. 1998. Focus group research: Explo-ration and discovery. In Handbook of applied social research methods,ed. L. Bickman and D. J. Rog, 505–26. Thousand Oaks, CA: Sage.

Strauss, A. L., and J. Corbin. 1998. Basics of qualitative research (2nd ed.).Thousand Oaks, CA: Sage.

Stroebe, W., M. Diehl, and G. Abakkoumkin. 1992. The illusion of groupeffectivity. Personality and Social Psychology Bulletin 18 (5): 643–50.

Sutton, R. I, and A. Hargadon. 1996. Brainstorming groups in context:Effectiveness in a product design firm. Administrative Science Quar-terly 41 (4): 685–718.

Szulanski, G. 2003. Sticky knowledge: Barriers to knowing in the firm.Thousand Oaks, CA: Sage.

Taylor, D. W., P. C. Berry, and C. H. Block. 1958. Does group participation,when using brainstorming, facilitate or inhibit creative thinking?Administrative Science Quarterly 3 (June): 23–47.

Thomke, S. 2001. Enlightened experimentation: The new imperative forinnovation. Harvard Business Review 79 (2): 67–78.

Thomke, S. 2003. Experimentation matters. Boston, MA: Harvard BusinessSchool Press.

Thompson, L. 2003. Improving the creativity of organizational workgroups. Academy of Management Executive 17 (1): 96–111.

Veldhuizen, E., E. J. Hultink, and A. Griffin. 2006. Modeling market infor-mation processing in new product development: An empirical analysis.Journal of Engineering and Technology Management 23 (4): 353–73.

von Hippel, E. 1978. Successful industrial products from customer ideas.Journal of Marketing 42 (1): 39–49.

von Hippel, E. 1986. Lead users: A source of novel product concepts.Management Science 32 (7): 791–805.

von Hippel, E. 1994. “Sticky information” and the locus of problemsolving: Implications for innovation. Management Science 40 (4): 429–39.

von Hippel, E. 1998. Economics of product development by users: Theimpact of “sticky” local information. Management Science 44 (5):629–42.


Appendix A. Data Sets Employed to IllustrateKey Points

The CPAS data set was collected from the membership ofthe Product Development and Management Association(Adams-Bigelow, 2004; Adams-Bigelow and Boike,2004). The CPAS data set was created from a single-respondent survey of members who had recently beeninvolved in new product development. Due to the natureof the PDMA and some advance screening, most of themembers contacted were qualified to take the survey. Theresponse rate was 17%. Respondents represented a broadrange of industries and an even mix of B2B and B2Ccompanies. Most of the companies were goods manufac-turers, service organizations were underrepresented.

The 2007 Bank New Service Development Surveywas conducted by the author with the help of the Ameri-can Bankers Association (Schirr and Page, 2009). Thissingle service industry study included 211 banks ofdiverse sizes. Solicitations were sent to an unqualifiedlist of bankers, so that the gross response rate was 3.3%,however using the estimates from the ABA of thepercentage of recipients who were qualified (had per-sonally been actively involved in a new service devel-opment project within the past 24 months) the responserate among eligible recipients was estimated to be over30%.

More detailed information about these data sets isavailable from the cited publications or from the author.

Appendix B. A Sampling of Articles on the Efficacy of Group Methods versus Individual Research Methods

Publication(s) Overview—Group Research Explanation for Problems Improvements

Osborn (1957)Applied ImaginationGuide to Brainstorming

1. Four key rules of brainstorming2. More ideas linked to quality3. Individuals are more productivein groups

Expert moderator must follow 4 keyrules of brainstorming.

Four keys: (1) Judgment withheld;(2) Freewheeling; (3) Quantity isgoal; (4) Combine and improve

Taylor et al. (1958)Admin Science QuarterlyExperiments—group versus

individual

“Nominal groups”—combinedindividual brainstorming—outperformed groupbrainstorming in terms of thenumber, quality, and uniquenessof ideas generated.

Group interaction kills ideas. Use individual brainstormingand individual interviews insteadof groupbrainstorming.

Bouchard and Hare (1970)Bouchard (1972a, 1972b)Bouchard et al. (1974)Journal of Applied PsychologyImproving group thinking

“group brainstorming . . . inhibitsrather than facilitates creativethinking, and pooled individualeffort is a far more productiveprocedure than group effort.”(1970, p. 51)

Blocking of individual inputs ingroups. Group creativity does notimprove with size; nominalindividual does.

Forced participation and useof analogies (synectics) in groupresearch improves productivitybut not to thelevel of individual research.

Delbecq and Van de Ven (1971)J of Appl. Behavioral ScienceClaxton et al. (1980)J of Consumer ResearchNominal Group Technique

Focus groups require a highlytrained leader and groupdynamics still affect ideation.

An individual or subset maydominate a focus group.

Use the Nominal Group Technique.NGT leadersneed little training. Indi-vidual ideas are gathered. Groupidea loss is avoided.

Lamm and Trommsdorff (1973)European J of Social Psych.Review of empirical studies

Nominal groups outperform groupideation in terms of quantity andquality of ideas generated.Brainstorming rules are effective.

Group dynamics, especiallyproduction blocking, socialinhibition, and cognitiveuniformity affect group results.

More study needed on uniquenessand variety of ideas (adjusted fornumber) generated. More researchon psychological factorssuggested.

Edward F. Fern (1982)J of Marketing ResearchFocus groups for ideation

If a goal is quantity of ideas, qualityof ideas, or relevance of ideas,individual brainstorming orinterviews outperform focusgroups or groups in general.

Group dynamics are the main factorreducing effectiveness.Moderators did not have asignificant effect. Groupparticipants did enjoy theexperience more than individuals.

The “relatively low cost as well asthe speed” (p.1) of group ideationmust be weighed against the lossof quantity, quality and relevanceof ideas from individual methods.

McQuarrie and McIntyre (1986)J of Product Innovation ManFocus Groups and NPD

For new product ideas or to screennew ideas it is better to interviewmarket participants individuallythan in focus groups. (p. 41)

Group research may have a role inevaluating and testing concepts orstudying diffusion of products.

Use individuals for generating andevaluating ideas.


Appendix B. Continued

Publication(s) Overview—Group Research Explanation for Problems Improvements

Diehl and Stroebe (1987)J of Personality & Social Psy.Diehl and Stroebe (1991)Interpersonal relationsMeta-analysis & experiments

Idea generating groups andbrain-storming are “widely usedin industry . . . despiteoverwhelming evidence for itsineffectiveness.” (1991, p. 392)

Group ideation is hindered by:1. Free-riding,2. Evaluation apprehension,3. Production blocking

Writing down ideas is a minor aid togroups. Groups suffer significantloss of quantity and quality evenif individual speaking time isequal.

Mullen et al. (1989)Journal of General PsychologyMullen et al. (1991)B & Appl. Social PsychologyMeta-analysis of studies

For quantity or quality of ideas,“productivity loss inbrainstorming groups is highlysignificant and of strongmagnitude.” It is “difficult tojustify brainstorming . . . [by] . . .any performance outcomes.”(1991, p. 18)

Group productivity decreases withsize. Working individually ingroup (NGT) is better than groupbut not as good as trulyindividual. Social psychologyissues self-attention, drive arousalneed to be studied.

Neither incentives for productivitynor interruption-avoidance willhelp group ideation. Socialpsychology adjustments to deal toencourage self-focus or reduceanxiety might.

Griffin and Hauser (1993)Marketing ScienceVoice of the Customer

The “group synergies expected fromfocus groups do not seem to bepresent.”

Focus groups are less effective thanindividual interviews inuncovering needs fromindividuals.

Groups are so much less effective tocall into question whether thereare any cost efficiencies.

Paulus et al. (1993)Personality and Social Psych.Illusion of Group Productivity

Groups produce fewer ideas, butgroup research remains popular.One basis may be perceivedproductivity of groups.

Group participants:1. Expect to be productive2. Believe they were productive3. Take individual credit

Future study to look for basis ofgroup perceptions.

Sutton and Hargadon (1996)ASQ: Adm. Science QuarterlyEthnography of IDEO

Groups are less effective thanindividuals working alone.

There are objectives for groupmethods other than quantity orquality of ideas generated. IDEO(as consultant) benefits fromgroup use

Other objectives may include:(1) organizational memory;

(2) skill variety; (3) attitude ofwisdom; (4) impressing clients;(5) profit

Furnham (2000)Business Strategy ReviewLit review and justifications

“Research shows unequivocally thatbrainstorming groups producefewer and poorerideas . . .” (p. 21) “businesspeople must be insane” to usegroup methods. (p. 23)

Business may be unaware ofgroup evidence and/orinfluenced by the illusion ofeffectiveness.

GOOD reasons include(1) decision acceptance(2) pool efforts.

Suggests “electronic meetings” aspossible alternative to face-to-facegroup ideation.

Fern (2001)Adv. Focus Group ResearchSummary of issues in f

groups

“Collecting unique or creativethoughts or ideas poses an almostinsurmountable obstacle for focusgroups.” (p. 168)

There is no evidence for creativesynergy in group efforts, butthere may be nonidea advantages.

Focus groups are fun.Both participants and observers

of focus groups tend to believethe results of the groups.

Thompson (2003)AoM ExecutiveReview of research

“Despite the empirical evidencefor its ineffectiveness, groupbrainstorming remains popular. . .” (p. 100)

Idea quantity does breed quality.

Reasons for group problems,including:

1. Social Loafing2. Conformity3. Production blocking, and4. Downward norm setting.

Procedural strategies (8)including:

(1) Team diversity(2) Brainwriting,(3) Nominal group technique,(4) electronic brainstorming,(5) building a playground

Bristol and Fern (2003)Psychology and MarketingF groups—interaction effects

“individuals working alone arebetter at generating ideas” . . . andbetter at uncovering attitudes ofindividuals. (p. 447)

Opinions in groups are swayed.Individual and NGT research ismore productive and betterrepresents individual attitudes.

Focus groups should perhaps belimited to the study of groupdecision making.

Litchfield (2008)Ac. of Management ReviewGoal-based view

Research indicates that individualsare more efficient and effectivethan groups, but Osborn’s fourrules for brainstorming areeffective and support goal-basedideation.

Improving ideation in groups orindividually will use managerialgoal setting as well aspsychological factors.

Goal-based rules may improveideation in groups andindividually.

Girotra et al. (2010)Management ScienceQuality of hybrid versus

group ideation

An organization is likely better off“relying on the asynchronous ideageneration of individuals” ratherthan hybrid groups or traditionalgroups. (p. 23)

Hybrid groups (individual thengroup analysis) outperformgroups:

1. 3 to 1 in quantity of ideas2. In average quality of ideas3. In the “very best” ideas

Hybrid groups are better thantraditional groups at evaluatingideas, but underperform outsidegroups. Building on others ideasdoes not result in better ideas.


flawed tools: the efficacy of group research methods to generate customer ideas

Documents