A study to analyse the effectiveness of multimedia

and text-only e-mail communications and discover

whether consumers attitudes towards e-mail reflect

their behaviour

By Owen Simon Ball

Submitted in partial fulfilment of the requirements for the Degree:

Bachelor of Arts in Marketing & Advertising

University of the Arts London

London College of Communication: School of Creative Enterprise

March 2007

1. AcknowledgementsThis Dissertation would not have come to fruition without the help, wittingly or

unwittingly, of a great many people too numerous to mention. This section provides

a platform to recognise and acknowledge those fortunate enough to spring to mind.

My dissertation tutor Jon Kitto should be commended for the guidance, support,

and wealth of ideas he provided in order to see this work completed.

I would like to express my gratitude to the BA Marketing & Advertising staff,

specifically Colin Watson for his introductory coffee on my first visit to the institution

and his subsequent help with the questionnaire designs for this study.

A huge thank-you Kenji Lim from Jail Clothing, without whom this study would not

have been devised or executed.

The staff at Apple, TextMate, Mellel & Campaign Monitor for developing exceptional

software that made my life a little easier, and saved me an overwhelming amount of

time and effort.

Luke Branley, Aimee Gillett and Reynaldo Canlas Jr. for kindly taking the time to

assist me with my research and distract me in equal measure.

All the unnamed people who kindly responded to the Questionnaire, I am indebted

to you.

Martin Hall in the Reprographics department at EHDC for providing the facilities to

print and bind this study.

Finally, my family for providing help, advice, and motivation when it was required.

- 1 -

2. AbstractTechnological developments have given rise to a global network of computers and

servers allowing its users to communicate with each other without geographic or

time constraints. The most popular, and established, medium of communication in

this global network is e-mail and it is this of communication that will be scrutinised

in this study. In its early days e-mail was constrained to plain text messages.

Advances in software and internet protocols have enabled the medium to carry a far

richer subset of information including, but not limited to, colour information, images

and varied typefaces. This new breed of multimedia e-mail, facilitated by HTML, has

polarised opinion. Proponents laud the new design capabilities made possible

whilst at the same time detractors lament the surfeit of rendering issues,

incompatibility and broken links they argue plague the format.

Placed in this context, the objectives of the research were two-fold. The first was to

identify the format of e-mail that was most effective. The second was to discover

whether recipients attitudes toward e-mail format reflected the behaviour observed

in relation to the first objective. In order to conduct the study three methods of

research were utilised. Desk research provided data for comparison and contrast,

an experiment was conducted to obtain metrics for the effectiveness of each format

of e-mail. Finally, a questionnaire was sent to the same sample used in the

experiment to determine whether their attitudes toward e-mail correlated with the

observed behaviour.

The study found that e-mail open and click-through rates both provided significant

evidence to show that multimedia e-mail was in fact more effective than text-only.

These findings were backed up by statistical analysis that showed that the

probability of the results being significant was extremely high. In addition to the

findings from the experiment the questionnaire responses suggested that there is a

strong correlation between the the attitudes of recipients and their behaviour.

Respondents preferred multimedia over text-only by a ratio of 3:1, a ratio that when

compared to click-through rates for each format was almost identical.

This study ultimately produced two noteworthy conclusions. The first conclusion is

that HTML e-mail is more effective than text-only e-mail, thereby rendering the

research hypothesis invalid. The second conclusion, relating to the second

objective, is that consumers attitudes show a strong correlation to their observed

behaviour, both showing a significant preference for multimedia e-mail.

- 2 -

3. Table of Contents1. Acknowledgements p. 1..............................................................

2. Abstract p. 2..............................................................

3. Table of Contents p. 3..............................................................

4. List of Appendices p. 8..............................................................

5. Abbreviations p. 9..............................................................

6. Introduction p. 10..............................................................

7. Literature Review p. 12..............................................................

7.1 Introduction p. 12..............................................................

7.2 E-mail Marketing p. 12..............................................................

7.2.1 E-mail p. 12..............................................................

7.2.2 E-mail Marketing p. 13..............................................

7.3 E-mail by Format p. 15..............................................................

7.3.1 Evidence for Text-only p. 15.....................................

7.3.2 Evidence for HTML-only p. 16.................................

7.3.3 Evidence for both formats p. 17................................

7.4 Further Theory p. 18..............................................................

8. Research Questions p. 21..............................................................

8.1 Area for Further Research p. 21........................................................

8.2 Research Aims p. 21..............................................................

8.3 Research Hypothesis p. 22..............................................................

- 3 -

9. Methodology p. 23..............................................................

9.1 Overview p. 23..............................................................

9.2 Research Considerations p. 23...........................................................

9.2.1 Quantitative vs. Qualitative p. 23..............................

9.2.2 Triangulation p. 24.....................................................

9.3 Research Methods p. 25..............................................................

9.3.1 Experiment p. 25........................................................

a. Process p. 25..............................................................

i. Sample p. 25..................................................

ii. Content p. 26..................................................

iii. Measurement p. 26.........................................

b. Limitations p. 26..............................................................

9.3.2 Survey p. 27..............................................................

a. Process p. 27..............................................................

i. Sample p. 28..................................................

ii. Content p. 28..................................................

iii. Measurement p. 28.........................................

b. Limitations p. 29..............................................................

10. Results & Findings p. 30..............................................................

10.1 JailMail Multimedia p. 30..............................................................

10.1.1 Metrics for Multimedia Campaign p. 30...................

10.2 JailMail Text-only p. 32..............................................................

10.2.1 Metrics for Text-only Campaign p. 32......................

10.3 Experiment Analysis p. 34..............................................................

10.3.1 Comparisons p. 34.....................................................

10.3.2 Chi Squared p. 35......................................................

10.4 JailMail Survey p. 39..............................................................

10.4.1 Survey Statistics p. 39................................................

10.4.2 Gender Analysis p. 40................................................

10.4.3 Age Analysis p. 42.....................................................

10.4.4 E-mail software analysis p. 43...................................

10.4.5 Browser based e-mail analysis p. 44..........................

- 4 -

10.4.6 Frequency respondents check e-mail analysis p. 45..

10.4.7 E-mail format analysis p. 46......................................

10.4.8 Reasons for choosing multimedia e-mail analysisp. 49..............................................................

10.4.9 Reasons for choosing text-only e-mail analysis

p. 50..............................................................

10.4.10 Respondent's preferred hyper-link format analysisp. 51..............................................................

10.4.11 Preferred frequency for e-mail communications analysis p. 52.............................................................

10.4.12 Behaviour towards unwanted commercial e-mail communication analysis p. 53....................................

10.4.13 The reasons respondents gave for not choosing to unsubscribe analysis p. 55.........................................

10.5 Survey Analysis p. 56..............................................................

10.5.1 Comparisons p. 56.....................................................

a. Gender Vs. E-Mail Format p. 56...............................

b. Age Vs. E-Mail Format p. 58....................................

11. Conclusions p. 61..............................................................

11.1 Key Conclusions p. 61..............................................................

11.1.1 Experiment p. 61........................................................

a. Open Rate p. 61.........................................................

b. Click-through Rate p. 62............................................

11.1.2 Questionnaire p. 63....................................................

11.2 Qualifying Statement p. 63..............................................................

11.3 Limitations p. 64..............................................................

11.4 Further Research p. 65..............................................................

12. Appendices p. 66..............................................................

13. Bibliography p. 73..............................................................

- 5 -

List of Tables

Table 1: Comparing unique click-throughs by format p. 18.....................................

Table 2: Effectiveness of Image & Text dominant web banner ads p. 19.................

Table 3: A Taxonomy of Research Methodologies p. 24..........................................

Table 4: Unique E-mail Opens Chi Squared Figures p. 35.......................................

Table 5: Unique E-Mail Clicks Chi Squared Figures p. 37.......................................

Table 6: Total E-mail Opens Chi Squared Figures p. 37...........................................

Table 7: Survey - Gender Chi Squared p. 40.............................................................

Table 8: Survey - Multimedia vs Text Chi Squared p. 46.........................................

Table 9: Survey - Revised Multimedia vs Text Chi Squared p. 48...........................

Table 10: Gender vs Preferred E-Mail Format p. 56.................................................

Table 11: Age vs Preferred E-Mail Format p. 58......................................................

Table 12: Further Research Possibilities p. 65..........................................................

List of Figures

Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign p. 30...

Figure 2: Total links clicked for multimedia campaign p. 31....................................

Figure 3: Opens, Bounces & Unopened numbers for text campaign p. 32...............

Figure 4: Total links clicked for text campaign p. 33................................................

Figure 5: Survey Statistics p. 39..............................................................

Figure 6: Q1: Respondent Gender p. 40..............................................................

Figure 7: Q2: Respondent Age p. 42..............................................................

Figure 8: Q3: Respondent's e-mail software p. 43.....................................................

Figure 9: Q3a: Respondent's browser based e-mail choice p. 44..............................

Figure 10: Q4: Frequency respondents check e-mail p. 45.......................................

Figure 11: Q5: Respondents preference of e-mail format p. 46................................

Figure 12: Click-through compared to Preferred Format p. 47.................................

Figure 13: Q5a: Reasons for choosing multimedia e-mail p. 49...............................

Figure 14: Q5b: Reasons for choosing text-only e-mail p. 50...................................

Figure 15: Q6: Respondent's preferred hyper-link format p. 51................................

- 6 -

Figure 16: Q7: Respondent's preferred frequency for e-mail communications

p. 52..............................................................

Figure 17: Q8: Behaviour towards unwanted commercial e-mail communicationp. 53..............................................................

Figure 18: Open & Unsubscribe Percentages comparison p. 54...............................

Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribep. 55..............................................................

Figure 20: Gender vs Preferred E-Mail Format p. 57................................................

Figure 21: Age vs Preferred E-Mail Format % by Age p. 59....................................

Figure 22: Age vs Preferred E-Mail Format Frequency by Age p. 59......................

- 7 -

4. List of AppendicesAppendix 1: JailMail Multimedia Format p. 66.......................................................

Appendix 2: JailMail Text-only Format p. 67..........................................................

Appendix 3: JailMail Questionnaire p. 68..............................................................

Appendix 4: Chi Squared Test Table p. 72..............................................................

- 8 -

5. Abbreviations & DefinitionsClick-through rate - The number of hyperlinks accessed from a campaign

E-mail - Electronic Mail

HTML - Hyper Text Mark-up Language

IPT - Interactive Prospect Targeting (company)

ISP - Internet Service Provider

Link(s) - Hyperlink (Method of navigating to content)

MIME - Multipurpose Internet Mail Extensions

Open-rate - The number of e-mails opened by recipients

Spam - Unsolicited Commercial E-mail

- 9 -

6. Introduction

The Global Reach research agency (2004) estimated that by the year 2005 there would be

over 1 billion users of the internet globally. Whilst it impossible to provide truly accurate

numbers this still represents a massive potential audience for marketing messages, and

because the internet transcends the limitations of time and space this audience can be reached

almost instantaneously. E-mail is one of the most established methods of communication

used online, according to IPT (2004), 99% of consumers use the internet for email which

must place the medium as a one of few communication methods with such a large reach.

If we accept then that e-mail has vast potential in terms of numbers, the next question to be

asked is what are the implications for marketers. According to the same IPT study, e-mail

marketing ranked second as the medium consumers felt was most effective. It is important

then, considering the numbers involved and consumers receptiveness to the format, that

marketers assess the best way with which to communicate their messages. There are currently

two clear options for e-mail messages. Traditional text formatted e-mail, or multimedia e-

mail facilitated by the use of HTML code. In some cases, it is very easy for a marketer to

make the choice to use one or the other, however, how does a marketer know whether to

spend the time and money required to develop an HTML based email campaign? This

research, in an attempt to provide empirical evidence on this issue, uses an experiment, and

questionnaire to discover what format of e-mail is most effective, and whether consumers

attitudes toward e-mail marketing reflect their behaviour in response to it. The aim of the

research then is to provide the marketer with quantifiable evidence that can be used to argue

in favour of either approach.

The dissertation is broken down into five broad section, beginning with a review of current

literature in the subject area. This is followed by the Research Questions and Research

Methodology chapters that define the area of research and how the research will be

- 10 -

conducted. The following section illustrates the findings from the experiment and

questionnaire. The dissertation is concluded by a chapter that draws together the key

conclusions from the findings.

- 11 -

7. Literature Review7. 1. Introduction

The relatively recent development of email, in particular multimedia based email facilitated

by HTML, as a marketing tool ultimately restricts the pool of previous academic research in

the field. This study concerns itself with the effectiveness of two broad categories of email

communication, that of graphically rich multimedia email and that of text-only email. It is

perhaps to be expected that the majority of information available regarding email and email

marketing is based on the world wide web rather than in books and journals, and it is true to

say that there is a large amount of subjective and anecdotal evidence that can be found

supporting the hypothesis that either type of email is more effective than the other. This

literature review will provide an objective view of the material available using, where

appropriate web-based and print-based material. The literature review will be segmented into

four logical sections. The first section will focus on email marketing, citing work from

figures and companies well-regarded in the industry alongside more anecdotal evidence. The

second section will be more empirical in nature addressing previous studies and research

supporting the case for either format. The third section will discuss relevant theory relating

primarily to the composition of communications. The final section will draw conclusions

from the research presented.

7. 2. E-mail Marketing

7. 2. 1. E-mail

The aim of this section is to define email marketing and give the reader an understanding of

the importance of email and why it is an important element of a marketing mix. According to

the Oxford American Dictionary e-mail stands for "messages distributed by electronic means

- 12 -

from one computer user to one or more recipients via a network." The internet, in its current

form, was not a prerequisite for e-mail. Indeed, the first crude form of e-mail was sent by Ray

Tomlinson of ARPANET in 1971 (Griffiths, R.T., 2002), however, in order for the subject

area of this research, email marketing, to be utilised by marketers, the internet along with

more advanced software and networking technology was required. With this backbone in

place Hoffman (2000) argues that e-mail has become "the most important innovation since

the development of the printing press" able to "transform not just the way individuals go

about conducting their business with each other, but also the very essence of what it means to

be a human being in society." This is certainly an extreme viewpoint, but there is some

justification to Hoffman's views. The following figures make impressive reading and

certainly suggest that e-mail has become an intrinsic part of many peoples lives. According to

Messaging Online (2000) "e-mail is the fastest-growing communications technology in

history. From only 2m e-mail accounts in 1985, this grew to 891.1m e-mail accounts at the

beginning of 2001." This number is now even higher, according to The Radicati Group

(2006) there are a staggering "668 million email users worldwide, with over 1.2 billion active

email accounts." When this is coupled with "worldwide email traffic per day total[ing] about

135 billion messages" the result is a behemoth of a communication medium.

7. 2. 2. E-mail Marketing

Purely in terms of metrics the previous paragraph illustrates just how valuable e-mail could

be to an online marketing program, and it is true to say this is being followed up by

marketers. An IPT email marketing survey (2004) found that 92% percent of professional

respondents currently used email marketing with over half (51%) declaring that the budget

would increase the following year. It is perhaps not surprising that these figures are so

positive given the findings from the consumer part of the survey. According to IPT, 99% of

consumers use the internet for email creating a vast potential audience for marketing

messages. Even more encouraging for marketers is the medium consumers felt was the most

effective for marketing communication. Email ranked second at 32%, a score only 7% behind

television, but markedly higher than press, radio and other forms. Clearly these statistics have

to be approached with caution, the answers consumers give, compared to their actual

- 13 -

behaviour can often be distinctly different. Still, the figures do show consumers are still

receptive to email communication despite that fact that "the average email user gets between

54 and 93 e-mails per week, a number that increases in direct proportion to the length of time

the person has been using email" (Brondmo, H. P., 2001).

Utilised as a marketing tool email has both advantages and disadvantages. The key

advantages of email marketing are two-fold and intrinsically linked. The first is low costs and

the second is the ability to utilise a digital work-flow. (Jackson & DeCormier, 1999; Peppers

& Rodgers, 2000). Actual figures vary by campaign but Dave Chaffey (2006) estimates that

"currently, e-mail costs range from $5 to $7 per thousand compared to $500 to $700 per

thousand for direct mail." If e-mail costs just 1% per thousand to reach customers compared

to direct mail the casual observer may question whether, in the words of the age-old idiom,

"you get what you pay for." According to Hans Peter Brondmo (2001) "the best email

marketing programs, in fact, routinely enjoy response rates above 30 percent, while even

fairly simple promotional marketing campaigns routinely see a 4 to 6 percent response

(compared to 1 to 2 percent response rate from traditional direct mail campaigns)." The

figures are a positive re-enforcement for email, but Brondmo continues with a note of

caution. He says that "with all the information consumers are exposed to on a daily basis, it's

getting harder and harder to separate the good from the bad." This a viewpoint expanded by

Windham (2000) who suggests that email marketing has one major disadvantage, the scourge

that is spam, or unsolicited email. Not only does this taint the reputation of email as a form of

communication, but it directly affects response rates to campaigns. In addition many counter

spam measures introduced by ISPs and email providers have had a significant detrimental

effect on email open rates and created rendering issues (DoubleClick, 2005).

- 14 -

7. 3. E-mail by Format

7. 3. 1. Evidence for Text-only

The following paragraph will assess evidence that suggests text-only email messages are

more effective than HTML. It is to perhaps to the credit of what Jeanne Jennings (2004)

describes as "the anti-HTML group", a "small but vocal minority" that there is any debate on

the issue of what format of email is most effective. Evidence in favour of text-only is

underwhelming and, in addition, predominately based on figures from the first couple of

years of the twentieth century. A report on consumers' preferred e-mail advertisement formats

by eMarketer (2002) discovered that a staggering 62% of respondents preferred text email

messages , with only 35% favouring HTML. These results supported findings from another

eMarketer report carried out in 2001 assessing response rates to e-mail marketing campaigns.

It found that text email enjoyed an 18.5% click-through rate compared to 15.6% for the

HTML equivalent. Research undertaken in the years proceeding the data above tend to show

a slight preference for HTML as we shall see in the following section, however there is some

small consolation for text email. Jupiter Research (2005) attempted to identify what impact

the content of an email communication has upon response behaviour. Out of a sample size of

1166 consumers, written copy with 40% of responses was identified as the second most

important factor for respondents. This compares with 12% and 9% for one large image and

multiple smaller images respectively. This result does have to be put in some context.

Because there is no distinction between HTML and text email, it is entirely possible that

readers preferred HTML email, but still felt the body copy was the most important factor for

initiating response. Finally, A DoubleClick (2005) email trend report for Q1 2005 notes that

"Despite performing the least well, text showed the greatest year-on-year increase in click-

through rates up 19.6%, from 5.6% to 6.7%", and "the UK, in particular, saw its text click-

through rates increase by 136.7%." The report suggests this may be influenced by a trend

towards email access on mobile devices. This is a view shared by Dave Chaffey (2006) who

notes that there are many email readers that cannot easily display HTML, such as Blackberry

and PDA devices.

- 15 -

7. 3. 2. Evidence for HTML-only

The previous section alluded to a trend where earlier research favoured text email and later

HTML. The metrics suggest that this is indeed the case, however, an eMarketer report (2001)

focused on consumer attitudes towards e-mail formats rather than email delivery statistics.

The results are of particular interest because they show a huge disparity with much of the

research at that time. The key finding was that 60% of respondents would rather receive

HTML over text email. This result supports more recent research suggesting that the case for

HTML email being the bastion of email marketing is stronger than that for text. This

viewpoint is supported by anecdotal alongside empirical data both of which will be detailed

in this section. To begin, a reader comments section on a text vs. HTML article by Jeanne

Jennings (2004 ) for Clickz resulted in "less than 3 percent of responses [being] anti-html."

This does not of course mean that 97 percent of respondents were pro HTML email, but it

does suggest that there is not a large swath of antipathy towards HTML. In addition

comments e-mailed to the author by marketers strongly suggest HTML is now the de-facto

standard for email. Paul Maloy (2004) says "the newsletter list that I maintain favors HTML

by about 12 to 1", a sentiment shared by Paula Skaper (2004) who adds that "more than 90

percent of readers choose the HTML version over text." And "until that metric changes

significantly, there's simply no solid business case for dumping HTML."

Data from DoubleClick (2005) echoes the views given in the preceding paragraph.

Discussing email format trends the report writes "HTML was, by far, the most popular email

format in Q1 2005 with 64.3% of e-mails being sent in this format. This was followed by

Multi-part ... at 20.9% and Text at 14.8%." The report continues "in Q1 2005, higher click-

through rates were achieved by formatted content with both HTML (11.5%) and Multi-Part

(11.0%) performing above the overall average of 10.0% for all types." However "Text

continued to perform least well at 6.7%". These results show that e-marketers clearly feel

HTML is the most effective method of communication, with the metrics backing up their

argument. Indeed the Q2 report from DoubleClick (2005) provides similar statistics. One of

the key findings in the report concluded that "although open rates declined from 36.0% in Q2

2004 to 27.5% in Q2 2005, click rates remained relatively stable, dipping slightly from 7.7%

- 16 -

to 7.2%". The report continues to say that "this relative stability in click rates, despite the

decline in opens, mirrors the HTML click-to-open ratio (which increased from 26.5% to

32.0% from Q2 2004 to Q2 2005)." A higher click-to-open ratio suggests that the content of

an email is effective in stimulating click-through once an email has been opened with HTML

the format that can facilitate this.

7. 3. 3. Evidence for both formats

In a survey of 50000 readers split evenly between HTML "lite", HTML and text based email,

it was found that almost no difference in effectiveness could be measured. The completion

rates were 7.2%, 7% and 7.5% respectively casting doubt on the supposition that either text

or HTML are more effective than the other (Marketing Sherpa, 2003). The article continues

to suggest that if a strong enough relationship between the business and consumer has been

forged, email format becomes a non-issue. In support of these findings, a survey of 600

people carried out by Lynda Partner (2003) for MarketingProfs discovered that just over half

preferred HTML email and just under half plain text.

Perhaps the most pertinent article is that of Tom O'Leary (2006) for GroupMail. O'Leary

proposes that due to technological advances in email composition software and readers the

best format to use is both. Using something called a multi-MIME message, it is possible to

send both HTML and text in the same communication. If a reader cannot read the HTML

correctly it can degrade gracefully down to the text version included, or alternatively the

reader can be programmed to show only the text-based version when available (Chaffey,

2006). To support the point O'Leary provides figures from a GroupMetrics report on unique

click-through from a list size of 10000.

- 17 -

Table 1: Comparing unique click-throughs by format

HTML Only 1436 Unique clicks

Text Only 1353 Unique clicks

HTML and Text 1518 Unique clicks

Source: GroupMetrics

Text is the worst performer, though not significantly below the performance of HTML. It is

clear however that the combination approach yields the best results. These results also

suggest that HTML would perhaps perform better without some of the technical and

rendering difficulties that can be associated with it, and which cause the reader to default to

text.

7. 4. Further Theory

A study conducted by Diameter for DoubleClick Media (2001) aimed to analyse the

effectiveness of each element of a web banner advertisement on ad recall. The study

concluded that in some situations visual images are second to brand logo in eliciting recall.

Whilst it did not discuss image versus text formatted advertisements, the findings suggest that

companies sending HTML formatted email should pay close attention to the branding of the

email above other image content. Two more studies, although assessing web banners as

opposed to email, can be applied to email formatting. The first of the studies carried out by

Electronic Telegraph/Ogilvy & Mather (1997) measured the effectiveness of each of the

creative elements incorporated into banner advertising. The findings of the study suggested

that image dominant adverts had a stronger brand recall than that of text dominant

advertising. However, closer scrutiny reveals only a slight difference. Image and text

dominated adverts had recall percentages of 60% and 56% respectively.

- 18 -

Utilising a similar comparison between image and text dominant web banner adverts, but

focusing on click-through-rate, BannerTips discovered that text dominant banners performed

better than their image dominant counterparts. In the context of email marketing these

findings suggest that the goal of each email communication should be reflected in the

formatting used, with the appropriate formatting utilised depending on whether the email was

designed to be a "call to action" or informative. The following table summarises these

findings.

Table 2: Effectiveness of Image & Text dominant web banner ads

Effectiveness

Attribute Brand-Reinforcement Click-through

Image Dominant Increase Decrease

Text Dominant Decrease Increase

Source: Park, M.H., 2002

It should be re-iterated that the findings in the studies preceding this paragraph provide data

based on the observation of banner adverts, not email. It is not then surprising that research

on the factors affecting click-through-rate in e-mail marketing (Chittenden, L. & Rettie, R.,

2002) produced contradictory results. Comparing click-through rate to creative information

they noted that "the number of images is less in the lower responding group, suggesting that

the more colourful and attractive e-mails generate greater response." If these results are

considered representative they provide a strong argument that not only are consumers

receptive to HTML email, but using HTML to provide colour and images actually improves

performance.

Eyetrack III (2004) undertook research studying the recall of information presented in text vs.

multimedia format. Although this study did not measure how effective each format was at

eliciting a physical response (click-through), it is important to remember not all email

marketing efforts are for that purpose. According to the IPT Email Marketing Survey (2004),

- 19 -

75% of marketers use email for customer retention, 15% for acquisition, 9% for cross selling

and 1% for branding. Retention may include information or news designed to keep the brand

in the customers mind rather than be a specific call to action. That said, the research shares

some similarities with email marketing and therefore it is possible to apply the results. The

aim of the research was to assess whether "the presentation of editorial content in multimedia

format [would] help readers understand and remember more [news] story information." Half

the participants viewed a news story in text only, and half in multimedia after which they all

completed the same recall quiz. Without delving too far into the data the study uncovered

three findings of relevance.

• Users that received information in text form seemed to have a better recall ofspecific factual information, names, places etc.

• Information about an unfamiliar process or procedure was more correctlyrecalled after viewing the multimedia.

• There was no significant difference between men and women regardless ofpresentation format.

Source: Eyetrack III, 2004

In the context of e-mail marketing the first finding suggests that important information should

be presented to customers in text regardless of the format used. The second suggests that new

or unfamiliar products can be better represented through multimedia or a combination of

both. The final finding suggests that there will be no discernible difference in the

effectiveness of HTML or text email with regard to gender. This is a stance supported by a

Pew Internet report (2005) looking at how men and women use the internet. The report

concludes that although there are differences, men and women are more similar than different

in their online lives. Further research would be needed to identify whether this trend follow

through to email usage and effectiveness.

- 20 -

8. Research Questions8. 1. Area for Further Research

The primary conclusion from the desk research was that although current research points to

HTML being the more effective format there is sufficient evidence to suggest that this is not

conclusive (see chapters 7.3.1 and 7.3.3). Indeed, if consider that the majority of evidence in

favour for HTML is based on metrics, and these metrics are not a "100% accurate measure"

(Patterson, 2007), then it would be ill-advised to infer that HTML is more effective. The

GroupMail report (2006) outlined in chapter 7.3.3, whilst concluding that sending e-mail in

both formats is preferable, also provide, numbers (Table 1) see that suggest there is virtually

no distinction in click-through rate between the formats. The lack of a unanimous agreement

on the topic provides scope for further research to be carried out. This study will attempt to

ascertain whether there is indeed a difference in effectiveness between HTML and text e-

mail. In addition, rather than focus entirely on metrics this study will explore consumer

attitudes toward the formatting of e-mail marketing communications and discover whether

these attitudes match the metric data.

8. 2. Research Aims

To determine if there is quantifiable difference in effectiveness obtained by utilising either

multimedia (HTML) or text-only email communications.

To identify if a sample groups attitudes toward multimedia (HTML) and text e-mail matched

their actions when presented with either type.

- 21 -

8. 3. Research Hypothesis

"There is no difference in effectiveness between HTML and text-only email"

- 22 -

9. Methodology9. 1. Overview

This dissertation tests the hypothesis "There is no difference in effectiveness between

multimedia and text-only email." Two complimentary types of primary research were used,

an experiment and a questionnaire. The goal of the experiment was to discover what format

of e-mail, HTML or text, was the most effective. A questionnaire was then used to assess

whether the measured results reflected their thought process or whether the two were in

opposition. For this purpose it was important to send the experimental e-mail campaign first

to prevent any bias corrupting the results after a questionnaire had been carried out. It is

important to set down in words that although participants were unaware that their actions

were being measured they remained completely anonymous in the experiment. Those

participants who took part in the questionnaire also remained anonymous. Upon the

conclusion of the primary research, both sets of data could be cross-analysed and patterns,

trends and other conclusions could be drawn from the results. This chapter then provides the

reader with an overview of the research utilised, and a justification for the choices taken for

this study. This chapter will be broken down into sections, each covering one part of the

process.

9. 2. Research Considerations

9. 2. 1. Quantitative vs. Qualitative

A positivist or scientific approach to research asserts that reality is stable and may be

observed objectively (Levin, 1988), this approach is in contrast to an interpretivist outlook

which proposes that "only through the subjective interpretation of and intervention in reality

can that reality be fully understood" (Davison, 1998). In an attempt to nullify the "paradigm

- 23 -

wars" between positivists and interpretivists, Benbasat (1984) and Pervan (1994) observe that

the methodology chosen for research should be suited to the problem being investigated and

the objectives of the research. The specific nature of the research and the ability to prove or

disprove the hypothesis with purely quantitative data influenced the decision to follow a

positivist rather than interpretivist approach. The two methods of primary research used to

collect data for the study have been identified by Galliers (1991) to be positivist (see table 3),

and these along with justification for there use are detailed in the following sections.

Table 3: A Taxonomy of Research Methodologies

Positivist Interpretivist

Lab Experiment Argumentative

Field Experiment X Reviews

Surveys X Action Research

Case Studies Case Studies

Theorem Proof Descriptive

Forecasting Futures Research

Simulation Role/Game playing

9. 2. 2. Triangulation

Triangulation identifies the process of using multiple forms research in order to get two or

more viewpoints on topic being studied, utilising this technique prevents anomalous results

from one set of data biasing the overall findings. This study has used three sources of data for

analysis. Desk research conducted prior to the primary research unearthed an abundance of

metrics directly related to the subject topic. This research was then supplemented by

quantifiable primary research in the form of an experiment and a survey.

- 24 -

9. 3. Research Methods

9. 3. 1. Experiment

The objective of the experiment used for the study was to discover whether there is a

measurable difference in effectiveness between multimedia HTML email and text-only email.

In order for this to be achieved, an email communication formatted in both HTML and text

was proposed. The sample was split in half with each half receiving one format of email. The

effectiveness of each format of email would be assessed on two criteria. The first is open rate,

the number of unique opens from the sample. The second criteria is click-through, how many

people, after opening the email, ultimately went on to click one of the enclosed links.

a. Process

In order for the experiment to be fully realised a set of variables first had to be defined. This

section will deal with each of these variables in turn which, in turn, will describe the process

undertaken to complete the experiment.

i. Sample

To carry out a live experiment using e-mail is problematic because US and EU anti-spam

legislation requires that all recipients must have opted-in to receive the communication. This

issue was circumvented by conducting the research on behalf of a company called Jail

Clothing. Jail Clothing has a house list of opted-in subscribers totalling 166 at the time of the

experiment.

- 25 -

ii. Content

The next issue was providing the content for both formats of email. It is difficult to obtain

measurements of effectiveness if there is no incentive for the viewer to open or respond to the

email. The partnership with Jail Clothing negated this problem. Jail send out regular email

newsletters, and it is one of these that was modified for the experiment. Recipients would

already recognise the sender ID and so the email was far less likely to be Blocked by an ISP

or filtered into junk or spam email folders by recipients. The two e-mails (see appendix 1 and

2) were crafted to have identical content as far as possible in order for the formatting of each

to be the only variable, and there-fore any difference in effectiveness between the two could

be measured. Both e-mails were hand-coded in TextMate and based on a template provided

by Campaign Monitor. Finally, because the text e-mail had to be sent in HTML format (see

Limitations below) a conscious decision was made to format hyperlinks in HTML so that

they were embedded in words. This way the links looked the same to the recipient regardless

of which format was delivered.

iii. Measurement

In order to measure the open and click through numbers for both e-mail campaigns Campaign

Monitor, an on-line e-mail service provider was utilised. The browser based software, already

in use by Jail for its e-mail communications, allows the user to measure, amongst other

occurrences, the open rate and click-through metrics that will used as a measurement of

effectiveness in this study. In addition Campaign Monitor generated and exported reports

directly saving the user the need to tackle this manually.

b. Limitations

There were three obvious limitation in the design of the experiment, one inherent to the

research methodology and the other two specific to this study. The inherent limitation of an

- 26 -

experiment is that it does not find out why the results have happened, in other words, it is

difficult to extrapolate meaning from quantified data. This limitation was bridged through the

use of triangulation (discussed above). The second limitation was a purely down to the

technology available. The text-only version of the e-mail had to be sent in HTML format. It

is impossible to measure open rates for text e-mail because in order for the measurement to

take place a tiny invisible image has to be requested from the host server, and this feature is

available only in HTML (Patterson, 2007). The final limitation is a limitation of format. It

was impossible for both versions of e-mail to be truly identical in everything except format

due to constraints with text-only formatting. Compromises had to be made, predominately

due to the lack of support for columns. The links down the left hand side of the multimedia e-

mail (appendix 1) had to be placed after the content of the e-mail for the text version

(appendix 2).

9. 3. 2. Survey

The objective of the survey questionnaire was to evaluate two things. The first, whether or

not the recipients views on e-mail format reflected the actions they had taken when receiving

the test campaign. And second, to identify if there were any other significant patterns or

trends that could be extrapolated from the answers given. These could have included

variables such as gender and e-mail provider.

a. Process

The following section will outline the processes undertaken to enable the questionnaire to be

carried out. In similar circumstances to the experiment there were a number of variables that

needed to addressed and a number of compromises that needed to be made.

- 27 -

i. Sample

In order for the sample used for the questionnaire to provide meaningful results it was

imperative that the recipients were the same ones who had received the email campaign. With

the permission of Jail Clothing the questionnaire was sent out to the subscriber list for Jail’s

newsletter. Due to the “live” nature of the study and to stay compliant to US and EU anti-

spam regulation it was necessary to send the survey only to those people who had opted in to

receive information from Jail. The questionnaire was sent by e-mail invitation to 166

recipients.

ii. Content

The final content of the questionnaire (appendix 3) was written to satisfy the needs of the

study but, due to the link to Jail Clothing, the language used and nature of certain questions

were written to appeal to Jail’s current subscribers. The content of the questionnaire covered

basic demographic information, namely gender and age, alongside email usage and reasoning

behind email usage questions. The questionnaire was composed and piloted using online

software from Question Pro. The digital nature of the survey allowed changes to be

implemented instantly in response to feedback from test recipients improving the quality and

accuracy of the results obtained. The live questionnaire was subsequently delivered using the

software's built in e-mail list manager using a recipient list imported from Campaign

Monitor.

iii. Measurement

Conducting the survey online enabled the questionnaire to be delivered to recipients directly,

and responses recorded and collated in real-time. In addition, analogous to the Campaign

Monitor tools, the software provided real-time customised reports negating the need to

complete this procedure manually.

- 28 -

b. Limitations

There were two limitations observed with the design and execution of the survey

questionnaire. The first was a relatively minor issue concerning the wording in certain

questions. Bartholomew (1963) suggested that uncommon words and jargon can be included

in questionnaires provided that the recipients are familiar with them. Piloting suggested that

the majority of terms were acceptable, however, "web-based e-mail" was deemed ambiguous

and this was changed in the live questionnaire to "browser-based e-mail" with examples

provided. In addition it was considered that "HTML"was too technical in nature and may not

be understood. "HTML" was subsequently substituted with "multimedia" to signify the use of

colour, formatting and images within an e-mail. This definition was provided to participants

along side the question.

The major limitation of the survey was the sample used and the consequences arising directly

from this decision. In order to discover a set of significant results it was necessary to use the

same sample group for the experiment and survey. Due to the remote nature of the sample the

only viable method of reaching them was to use e-mail. This in turn meant that the

questionnaire became vulnerable to the same problems that faced the e-mail campaign.

Indeed, the same metrics used to analyse the effectiveness of the e-mail experiment (open-

rate and click-through rate) would be significant factors in the response rate for the survey.

Walonick (1993) states that "Low response is the curse of statistical analysis" and it can

"dramatically lower confidence in the results." In order to aid response rates for the

questionnaire two techniques were used. The first was to place an article in the e-mail

campaign outlining Jails intention to conduct a survey. Participants who read the e-mail

would there-fore be expecting the questionnaire. The second technique was to provide the

recipient with an incentive to complete the survey. An agreement was made with Jail

Clothing for prizes to be provided for this purpose.

- 29 -

10. Results & Findings

10. 1. JailMail Multimedia

10. 1. 1. Metrics for Multimedia Campaign

Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign

Sent to 83 recipients on 15 Feb 2007 at 5:00 PM

Delivered 79 (95.18%)

Total Opens 47

Unique Opened 36 (43.37%)

Clicks 15 (41.67%) recipients clicked 25 unique links

Unsubscribed 0 (0%)

Bounced 4 (4.82%)

Total Unopened 47 (56.63%)

- 30 -

Figure 2: Total links clicked for multimedia campaign

Figure 2 show the total, not unique, clicks on hyperlinks from the multimedia campaign. 8

different links were accessed a total of 28 times.

- 31 -

10. 2. JailMail Text-only

10. 2. 1. Metrics for Text-only Campaign

Figure 3: Opens, Bounces & Unopened numbers for text campaign

Sent to 83 recipients on 15 Feb 2007 at 5:00 PM

Delivered 75 (90.36%)

Total Opens 32

Unique Opened 27 (32.53%)

Clicks 5 (18.52%) recipients clicked 5 unique links

Unsubscribed 1 (1.2%)

Bounced 8 (9.64%)

Total Unopened 56 (67.47%)

- 32 -

Figure 4: Total links clicked for text campaign

Figure 4 show the total, not unique, clicks on hyperlinks from the text-only campaign. 3

different links were accessed a total of 5 times. Significantly less than the multimedia

campaign.

- 33 -

10. 3. Experiment Analysis

10. 3. 1. Comparisons

There is no significant difference in the delivery rate between the two campaigns. This

suggests that some technological issues such as ISP image blocking (DoubleClick, 2005)

have not directly affected the results. The same conclusion can be drawn for the "unique

opens" numbers which show that although there is a declining trend for open rates

(DoubleClick, 2005), this has not influenced the results. The two most significant

comparisons are the "unique opens" and the "click-throughs". Both criteria show the most

disparate results between the two campaigns. Using just the data from the table these two

results strongly suggest that the null-hypothesis should be abandoned in favour of declaring

HTML more effective. However, further statistical analysis is required (see 10.3.2)

- 34 -

10. 3. 2. Chi Squared

The Chi Squared analysis is used "to test whether the number of individuals in different

categories fit a null hypothesis" (Deacon, J., n.d). The result, after being compared to a table

of values for Chi Squared, indicates the probability that the observations recorded in an

experiment are significant. If there is a statistically significant difference between the

observed numbers and the estimated numbers, this test provides the researcher with

supporting evidence to reject the null hypothesis. Conversely, a Chi squared value that is less

than the tabulated value for Chi squared gives the researcher no reason to reject the null

hypothesis. As a final note, according to Deacon (n.d) "When there are only two categories

(e.g. male/female) or, more correctly, when there is only one degree of freedom, the c2 test

should not, strictly, be used. There have been various attempts to correct this deficiency, but

the simplest is to apply Yates correction to our data." It is enough for this study to express

that this deficiency has been noted, and the Yates correction has been duly applied. Three Chi

Squared analysis will be presented in this section with accompanying explanation. Each of

the three tables represents one of the metrics extracted from the e-mail campaign that can be

used to assess effectiveness.

Table 4: Unique E-mail Opens Chi Squared Figures

- 35 -

When the Chi Squared Value of 1.02 is compared to the analysis table (appendix 4) it can be

seen that with 1 degree of freedom a value of 3.84 is required for a 95% probability that the

result is significant. The recorded value is distinctly lower than 3.84 and, with a probability at

less than 75% does not provide compelling evidence for the null hypothesis to be abandoned.

- 36 -

Table 5: Unique E-Mail Clicks Chi Squared Figures

In the case of Unique Clicks the Chi Squared value of 4.05 at 1 degree of freedom means that

we can say with a 95% probability that this result is a significant departure from the null

hypothesis. (Note: the values used represent the number of unique recipients who clicked on

a minimum of one link, not the total number of links clicked.)

Table 6: Total E-mail Opens Chi Squared Figures

The final table shows a Chi squared analysis for the total number of times the e-mail was

opened rather than just the unique open values. The Chi squared value of 2.48 along with 1

- 37 -

degree of freedom is lower than the value of 3.84 required for a 95% probability. However, it

does fall in the 90-95% probability range suggesting it is still a significant finding.

- 38 -

10. 4. JailMail Survey

Figure 5: Survey Statistics

10. 4. 1. Survey Statistics analysis

The statistics show that from a sample of 166 e-mail recipients 31 (18.67%) clicked through

to the questionnaire from the invitation. 21 (12.65%) respondents completed the

questionnaire. This a small response rate that jeopardises the reliability of the observations

recorded, however, if we consider the average click-through fell to 7.2% in 2005

(DoubleClick, 2005) then the click-through and response rate for this questionnaire is

statistically much greater than the average. Only 3 recipients dropped out of the survey before

completion suggesting that the majority of respondents felt comfortable with, and understood

the questions.

- 39 -

Figure 6: Q1: Respondent Gender

10. 4. 2. Gender Analysis

Table 7: Survey - Gender Chi Squared

- 40 -

The results show that there is a significantly higher proportion of male respondents to female,

and that this is significant. The value of 3.05 gives a 90-95% probability of significance.

There are a number of reasons this could be. The first possibility is for the JailMail list to be

gender biased. Gender statistics for the whole sample group were unavailable to test this

theory. The second possibility is that males were more likely to respond to the questionnaire

there-by biasing the results. The final explanation is that the sample size and resultant

response rate were too low for the gender sample to even out. Whatever the reason, this

finding will bias findings drawn from the questionnaire.

- 41 -

Figure 7: Q2: Respondent Age

10. 4. 3. Age Analysis

The key finding that two thirds of respondents are between the ages of 18 and 27 is

significant because it is likely to bias findings drawn from the questionnaire. Jail Clothing has

strong links with extreme sports, in particular snowboarding, and whilst a young age is not a

prerequisite for pursuing these sports a Mintel (2005) report into extreme sports suggests that

extreme sports "have much higher penetration levels amongst those under 25 ... due to the

physical nature of the sports in question."

- 42 -

Figure 8: Q3: Respondent's e-mail software

10. 4. 4. E-mail software analysis

It is clear from the results that accessing e-mail via a browser is the preferred method for the

sample group. It is possible that this factor would influence e-mail effectiveness. Most e-mail

clients provide superior rendering for HTML than their web-based counterparts and so this

would certainly not have a negative influence upon click through. However, many e-mail

clients have preview panes that automatically download tracking images thereby causing the

tracking software to register this as an open (Patterson, 2007). As 86% of the total

respondents (21) use browser-based e-mail, for the purpose of this research this factor can be

discounted as an influence on effectiveness.

- 43 -

Figure 9: Q3a: Respondent's browser based e-mail choice

10. 4. 5. Browser based e-mail analysis

Those users who selected browser-based e-mail in the previous question were asked to clarify

their choice. Over 60% chose Hotmail which is significantly high. As a comparison,

according to Emailcenter (2004) 40% of their test list email addresses were Hotmail or MSN.

It is certainly possible for the choice of e-mail provider to affect an e-mail campaign, in

particular a campaign utilising HTML code. A silverPOP (2005) report suggests that 40% of

e-mails contain missing or blocked images and this would negatively impact a multimedia

campaign. In the scope of this research there is not enough evidence to form a conclusion as

to whether the choice of e-mail provider directly affects campaign metrics.

- 44 -

Figure 10: Q4: Frequency respondents check e-mail

10. 4. 6. Frequency respondents check e-mail analysis

These results show that checking e-mail everyday has been selected by close to 100% of the

sample. The result is close enough to be classified as ubiquitous and accordingly it be

assumed that the presence of an e-mail will be detected within one day of its delivery. We

can also confidently predict that this factor will not have had an influence on the

effectiveness of the campaign.

- 45 -

Figure 11: Q5: Respondents preference of e-mail format

10. 4. 7. E-mail format analysis

Table 8: Survey - Multimedia vs Text Chi Squared

- 46 -

The Chi Squared table above uses the numbers of respondents from the questionnaire who

selected a preference for either multimedia (HTML) e-mail and text-only. The expected

numbers are, in this case, derived from the null hypothesis, assuming no difference between

the two formats of e-mail. The Chi squared value of 4.76 is very high signifying a 95-99%

probability that this is a significant result. It is interesting that this result is so distinctive from

the "total opens" numbers recorded in the experiment. If this result, on its own, is linked back

to the second research aim (8.2), we can see that there is a disparity. The sample claim to

prefer multimedia by a ratio of 3:1, however this is not born out in the total opens for each

format. The ratio is however almost identical to the "click through" numbers observed in the

experiment (Figure 12) which suggests a strong positive correlation between the two.

Figure 12: Click-through compared to Preferred Format

The previous section shows that there is a significant difference between recipients preferred

format of e-mail and the null-hypothesis that states there will be no difference in

effectiveness between the two. In order to fulfil the research aims (8.2) it is necessary to

- 47 -

repeat the Chi Squared analysis using estimated figures calculated from the numbers

observed in the experiment. In this way we can observe whether there is a significant

difference between the actions and attitudes of the sample group. Table 9 uses expected

numbers derived from the ratio of unique opens observed in the experiment.

Table 9: Survey - Revised Multimedia vs Text Chi Squared

We can see that there is still a difference between the observed numbers and the revised

expected numbers though not as evident. A Chi squared value of 2.42 suggests a 75-90%

probability that the result is significant.

- 48 -

Figure 13: Q5a: Reasons for choosing multimedia e-mail

10. 4. 8. Reasons for choosing multimedia e-mail analysis

The responses to this follow up question are distributed relatively evenly across the answers.

Taken in context with the recipients preferred choice of e-mail format these figures suggest

that Chittenden & Rettie (2002) were right to propose that "the more colourful and attractive

e-mails generate greater response." This factor could have been the driver behind the click-

through effectiveness of the multimedia campaign being dramatically higher than that for the

text-only message.

- 49 -

Figure 14: Q5b: Reasons for choosing text-only e-mail

10. 4. 9. Reasons for choosing text-only e-mail analysis

The number of responses recorded for a preference for text-only e-mail is too low to

extrapolate any significant findings from this follow up question. Of the answers received the

predominant reason for choosing text-only e-mail was to not be distracted by the colour and

images contained in a multimedia approach. This is perhaps due to the quantity of e-mails

received each week, 54 - 93 on average (Brondmo, 2000), limiting the time some users fell

they can spend reading each one. However, to reiterate the earlier point, there are not enough

responses to make confident assertions and the above analysis is conjecture only.

- 50 -

Figure 15: Q6: Respondent's preferred hyper-link format

10. 4. 10. Respondent's preferred hyper-link format analysis

The results show that respondents were split between preferring links to be shown in full, and

embedded in a word or phrase. These results are slightly contradictory to both the

observations taken in the experiment and the responses given thus far in the questionnaire.

One would expect the results for "embedded in a word or phrase" and "embedded in an

image" to be significantly greater than "link shown in full" which would then correlate with

the preference shown by respondents to multimedia e-mail and the higher click-through

numbers observed in the experiment. This is not the case suggesting one of two possible

causes. Firstly that respondents did not fully understand the question signifying an

inadequacy in the piloting of the questionnaire. Or, the second possibility is that there is a

difference between the sample groups actions and their attitude towards this question. Further

research would be needed to fully assess the influence the format of a hyperlink has on

effectiveness as it falls just outside of the scope of this study.

- 51 -

Figure 16: Q7: Respondent's preferred frequency for e-mail communications

10. 4. 11. Preferred frequency for e-mail communications analysis

The results show a clear preference in the sample group for monthly e-mail communications.

Jail currently send information to customers on a monthly basis but only if there is anything

new to say. The campaign sent preceding the experiment was delivered in October 2006

which left approximately 4 months between messages. It is possible this may have influenced

the campaign effectiveness, however it is unlikely. Response rates tend to drop due to a high

frequency of e-mail messages which can annoy recipients (Chaffey, 2006). This is definitely

an area that, given the time and resources, would be beneficial to study more closely. E-mail

is so easy to filter and delete that any actions leading to the disenfranchisement of customers

could be costly.

- 52 -

Figure 17: Q8: Behaviour towards unwanted commercial e-mail communication

10. 4. 12. Behaviour towards unwanted commercial e-mail communication analysis

This last question, and the follow-up (10.4.13) where applicable, primarily relate to the

formatting of future JailMail campaigns. There is one important observation that can be made

that has the potential to impact performance metrics. Over 50% of respondents did not select

unsubscribe as the option they would take if they no longer wished to receive messages. This

is significant because each would classify as a delivered message even though it were

unlikely to be read, and thus would lower the open rate accordingly. If we plot the figures for

the two campaigns (open % from delivered) next to the projected unsubscribe rate from the

table above there is a significant relationship between the two, in particular with the

multimedia campaign.

- 53 -

Figure 18: Open & Unsubscribe Percentages comparison

It is not in the remit of this study to explore this in depth although some indicators may be

found in the next section, these findings should be observed as stimulation for research

focusing solely on this factor.

- 54 -

Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribe

10. 4. 13. The reasons respondents gave for not choosing to unsubscribe analysis

The final question in the survey was a follow up to the previous question for those people

who chose one of the option that was not "unsubscribe." The even spread of answers suggests

that there is not a unanimous reason why people choose not to unsubscribe given the choice.

The answers given by the respondents in this section, could, combined with the data from the

last section be used as a starting point for more focused research.

- 55 -

10. 5. Survey Analysis

10. 5. 1. Comparisons

The key question in the survey that relates to the hypothesis is the question that asks the

respondent "which format e-mail communications they would prefer to receive." The

questionnaire research method provided extra variables that can be used to analyse this

question in more depth. This section will compare two sets of data at a time in an attempt to

discover trends or draw conclusions that the previous analysis has been unable to provide.

a. Gender Vs. E-Mail Format

Table 10: Gender vs Preferred E-Mail Format

- 56 -

The table above compares gender to preferred e-mail format. It was noted in section 10.4.2

that there is a gender bias in the sample. The table shows that 72.73% of respondents were

male. The table also shows no significant difference between the genders with regards to e-

mail format preference. A higher percentage of females preferred multimedia, but the sample

size is too small to infer anything of value from this. The male sample is larger and the 3:1

ratio in favour of multimedia strongly suggests that males prefer the format, however, this

cannot be assumed for females. As noted in the Research Methods chapter (9.3.2) the

inherent flaw in the research is the sample size used for the questionnaire, and in order to get

an accurate finding for females further research is required with a significantly larger sample

size. With this limitation in mind, it is still valuable to visualise the data, clearly showing a

correlation between the two genders responses.

Figure 20: Gender vs Preferred E-Mail Format

- 57 -

b. Age Vs. E-Mail Format

This section will assess what effect, if any, age has upon e-mail format preference.

Table 11: Age vs Preferred E-Mail Format

The most significant finding that can be drawn from the table is that 68.18% of the

respondents were aged between 18 and 27. Delving deeper into the data reveals that out of

this age group 80% prefer multimedia e-mail. These two results observed side by side suggest

that age is a factor in determining e-mail format preference, and whilst conjecture, is likely to

have been a significant factor in the results observed in the experiment.

- 58 -

Figure 21: Age vs Preferred E-Mail Format % by Age

Figure 22: Age vs Preferred E-Mail Format Frequency by Age

- 59 -

Figure 21 illustrates the percentages of each group that selected a preference for each format

of e-mail. These results should be viewed with caution, and in relation to Figure 22 which

shows the number of respondents who answered in each section. The three columns denoting

100% are misleading due to only having 1 or 2 respondents in each. The sample list used, and

the small list are the likely causes of this anomaly. Figure 22 clearly shows the correlation

between the age ranges and the format preferred. Jail Clothing's links to extreme sports is

likely to be the cause of the age bias in this research, however, there is room for additional

research in this area. It is possible that older age groups view the e-mail paradigm in a

different way to the younger audiences, and this would likely be an influencing factor in e-

mail effectiveness.

- 60 -

11. Conclusions

11. 1. Key Conclusions

11. 1. 1. Experiment

This section details the conclusions that can be drawn from the research experiment. They are

presented in such a way to build a convincing argument with respect to the null hypothesis

outlined in the Research Questions section (8.3) as the following;

"There is no difference in effectiveness between HTML and text-only email".

a. Open Rate

The "unique opens" observed for each format of e-mail appear to clearly favour the HTML

campaign, a result that, viewed alone, leads to the conclusion that the null hypothesis maybe

invalid. Further, statistical analysis using the Chi Squared method revealed a probability of

less than 70% that this finding was significant and so cast some doubt on the previous

finding. There is a high probability that the findings were obtained through chance instead of

indicating a trend, and therefore this study suggests that the data from "unique opens" is not

conclusive enough to disprove the hypothesis, but does suggest there may be some grounds

for this to occur.

The next conclusion that can be made is extrapolated from the "total e-mail opens" metric.

Unsurprisingly the figures show a strong correlation to the "unique opens" figures indicating

a significant trend in favour of HTML e-mail. The statistical analysis for these numbers

- 61 -

produced a 90-95% probability that the result is significant and so we can confidently assert

that the "total opens" figures give a strong indication that HTML is more effective. If this

assertion is assessed along side the conclusion outlined in the preceding paragraph then there

is now plausible evidence in place to conclude that HTML is indeed more effective rendering

the null hypothesis invalid.

There is an inherent problem with both conclusions drawn so far, they both rely on open

rates. Whilst being useful for measuring trends in a campaign, Mathew Patterson (2007)

from Campaign Monitor warns that "you should never take your open rate as a hard and fast

number, because you can never know the true figure." Accordingly, while the conclusions

drawn to date are valid, detached from subsequent findings they are not reliable enough to be

used to address the null hypothesis.

b. Click-through Rate

Click-through as a measure of effectiveness is significantly more accurate because a recipient

has to open, digest, and respond to an e-mail for a measurement to take place. The figures

used to analyse this factor were the number of unique recipients who clicked on a minimum

of one link. This method therefore produced results that showed for how many unique

recipients the e-mail campaigns had effected a response. The click-through numbers recorded

are compelling in their own right. 15 unique clicks to 5, or a 3:1 ratio in favour of HTML

formatted e-mail. This result is supported by the findings from a DoubleClick (2005) report

that found that the click-to-open ratio for HTML had increased from 26.5% to 32%. In

context "a higher click-to-open ratio suggests that the content of an email is effective in

stimulating click-through once an email has been opened with HTML the format that can

facilitate this." In support of this, Chittenden and Rettie (2002) found that "more colourful

and attractive e-mails generate greater response." Finally, when the Chi Squared value,

showing a >95% probability of significance, is factored in to the findings, this study

concludes that the results provide overwhelming evidence to render the null hypothesis

invalid. With the supporting evidence derived from the open-rate conclusions drawn above

- 62 -

the key conclusion from the experiment is that the null hypothesis has been confuted, HTML

is more effective than text-only e-mail.

11. 1. 2. Questionnaire

The second research question posed in this study related to the follow up questionnaire, and

asked whether the recipient’s attitude towards e-mail reflected the behaviour observed in the

experiment. The crucial question in the questionnaire asked the recipients to select whether

they prefer to receive multimedia or text-only e-mail (See 7.4.7). The results were definitive

and showed a strong preference for Multimedia e-mail. The Chi Squared analysis confirms

this and so this study concludes that the recipient’s attitude toward e-mail format is in accord

with their behaviour.

Finally, the results from the questionnaire showing gender and e-mail format preference were

cross-tabulated to determine if there were any correlation. This study concludes that gender

appears not to be an issue with respect e-mail format choice. Both genders favoured HTML

e-mail by a significant margin.

11. 2. Qualifying Statement

This study has concluded that HTML e-mail is more effective at eliciting a response,

regardless of recipient, than text-only e-mail. The null hypothesis proposed in the Research

Questions should be rejected and replaced with a hypothesis stating "HTML e-mail is more

effective than text-only e-mail".

- 63 -

11. 3. Limitations

Limitations anticipated prior to the research being conducted have been covered in the

Research Methodology Chapter and will not all be repeated here. However, a number of

limitations were identified during and post the research being conducted and they will be

included in this section. The first limitation was acknowledged earlier in the study, and it is

an important one. The sample size for questionnaire was very low and as a consequence this

factor is likely to have caused gender bias and age bias in the results.

The second limitation is a result of the constitution of recipients in the JailMail list that was

used for the sample. The market sector the company operates in attracts a certain

demographic of customer and this is likely to have created bias in the resulting findings.

- 64 -

11. 4. Further Research

There is a multitude of variables that could be used as the basis for further research, with

many highlighted in the questionnaire findings. The following table outlines a number of

areas where, given the time and resources, further research could have been used to obtain

more conclusive findings.

Table 12: Further Research Possibilities

- 65 -

12. AppendicesAppendix 1: JailMail Multimedia Format

- 66 -

Appendix 2: JailMail Text-only Format

- 67 -

Appendix 3: JailMail Questionnaire

03/08/2007 07:27 PMhttp://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 1 of 4http://www.questionpro.com/akira/frame.do?mode=editsurvey

Questions marked with a * are required

Hi, You are invited to participate in our survey for JailMail about your email usage.Approximately 150 of you will be asked to complete this survey to help Jail improve theiremail marketing. It shouldn't take any longer than five minutes to complete thequestionnaire. And, as a big thankyou to those who complete the survey, you will beentered into a draw to win some fabulous Jail Clothing goodies.

Taking part in this survey is completely voluntary, however, if you feel uncomfortableanswering any questions, you can withdraw from the survey at any point. Your opinionswill help us to make JailMail the best it can be, so we would really appreciate it if you canfinish the survey.

Don't worry, the responses you give and your information will all be kept strictlyconfidential and will be coded to remain private. If you have questions at any time aboutthe survey or the procedures, you may contact Owen@JailMail by email at the emailaddress specified below.

Thank you very much for your time and support. Please start with the survey now byclicking on the Continue button below.

*

What is your gender? *

Male

Female

How old are you? *

Under 18

18-22

23-27

28-32

33-37

38 and over

How do you receive email? (select all that apply) *

Browser based (Hotmail, Yahoo etc.)

Email Client (Outlook, Thunderbird, etc.)

- 68 -

03/08/2007 07:27 PMhttp://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 2 of 4http://www.questionpro.com/akira/frame.do?mode=editsurvey

Mobile Device (Blackberry etc.)

Other (please specify)

You selected that you use a browser-based email service, which do you use? (select allthat apply) *

Hotmail

Yahoo

Gmail

AOL

Other

How often do you you check your email? *

Everyday

2-3 times a week

Weekly

Bi-weekly

Monthly

Other

There are two main formats an email newsletter can take, multimedia or text. Multimediais defined as "using html to display pictures, images and colours in an email". Textemails are just that, text only.

Given the choice of one of the following options, which would you prefer to receive? *

Multimedia email

Text-only email

You selected a preference for multimedia (html) email. Would you be able to provide areason for your choice? Select as many options as you need. *

I like how multimedia (html) emails can be presented

I like the use of colour in emails

I like to see images and pictures in my emails

I like changes in text fonts and styles

Other

You selected a preference for text-only email, Can you provide us with some moreinformation about your choice? Select as many option as you need. *

My email client or reader does not render multmedia (html) correctly

I just want the "meat" of the email without the distraction of colour or images

- 69 -

03/08/2007 07:27 PMhttp://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 3 of 4http://www.questionpro.com/akira/frame.do?mode=editsurvey

I like to read my email off-line and don't like the broken image links

I have slow internet access and multimedia (html) emails take longer to download

Other

When you are reading an email, in which situation are you most likely to click on ahyper-link? *

When the link is shown in full

When the link is embedded in a word or phrase

When the link is embedded in an image or picture

Other (please specify)

How often do you prefer to receive email newsletters? *

Daily

Weekly

Bi-Weekly

Monthly

Quarterly

Other (please specify)

If you no longer wish to subscribe to an email newsletter which of the following are youmost likely to do? *

Unsubscribe

Bounce the email

Delete without reading

Filter in Junk/Spam

Other (please specify)

You didn't select unsubscribe as the option you would take if you no longer wanted toreceive email communications. What is the reason for this? *

It is difficult to find unsubscribe links in most emails

I don't trust the unsubscribe links given, they can lead to more spam

It takes less effort just to filter or block an email

Other

Is there anything you would like to add about email marketing that has not beencovered by the questionnaire? Please use the box below to inform us.

- 70 -

03/08/2007 07:27 PMhttp://www.questionpro.com/akira/frame.do?mode=editsurvey

Page 4 of 4http://www.questionpro.com/akira/frame.do?mode=editsurvey

Please contact owen@jailclothing.com if you have any questions regarding this survey.

Privacy | Security

Surveys | Email Marketing | Web Polls

- 71 -

Appendix 4: Chi Squared Test Table

Source: Adapted from (Deacon, J., n.d)

- 72 -

13. BibliographyBenbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) TheInformation Systems Research Challenge, Harvard Business School Press, Boston, MA,47-85.

Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496.

Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. NewYork: National Recreation Association.

Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis,MN: StatPac Inc.

BannerTips, 2001, Comparison between image and text dominant banners and click rate,http://www.bannertips.com/ BT025_answers.shtml

Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. NewYork: National Recreation Association.

Benbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) TheInformation Systems Research Challenge, Harvard Business School Press, Boston, MA,47-85.

Brondmo, H.P. (2001) The Eng@ged Customer: The New Rules of Internet DirectMarketing. Piatkus Books,

Chaffey, D., 2006, Total E-mail Marketing, Second Edition: Maximizing your results fromintegrated e-marketing (Emarketing Essentials), Butterworth-Heinemann Newton, MA,USA

Chittenden, L., and Rettie, R., 2003, An evaluation of email marketing and factors affectingresponse, Journal of Targeting, Measurement and Analysis for Marketing, 11(3), 203 – 218.

- 73 -

Davison, R.M., 1998, An Action Research Perspective of Group Support Systems: How toImprove Meetings in Hong Kong, http://www.is.cityu.edu.hk/staff/isrobert/phd/phd.htm

Deacon, J., n.d, The Really Easy Statistics Site, http://helios.bto.ed.ac.uk/bto/statistics/table2.html#Chi%20squared%20test

DoubleClick, 2001, Online brand Effectiveness Study, http://www.doubleclick.com/us/knowledge_central/documents/RESEARCH/dc_online_branding_0107.pdf

DoubleClick, 2005, DoubleClick EMEA Q1 2005 Email Trend Report, http://emea.doubleclick.com/WEB_ADMIN/documents/dc_Q1_05%20EMEA%20email%20trend%20report.pdf

DoubleClick, 2005, DoubleClick EMEA Q2 2005 Email Trend Report, http://emea.doubleclick.com/WEB_ADMIN/documents/dc_Q2_05%20EMEA%20email%20trend%20report.pdf

Electronic Telegraph/Ogilvy & Mather, 1997, How Internet Advertising Works, http://www.intelliquest.com/resources/ whitepapers/interadvert.pdf

Emailcenter, 2004, Can your customers read your email newsletters?, http://www.emailcenteruk.com/maxenews/ html_email_client_compatability.pdf

eMarketer, 2001, Response Rates of US E-Mail Marketing Campaigns, by Campaign Format,2001 (as a % of respondents), http://www.emarketer.com

eMarketer, 2001, US Internet User Attitudes toward Commercial E-Mail Formats, 2001 (as a% of respondents), http://www.emarketer.com

eMarketer, 2002, Consumers’ Preferred E-Mail Advertisement Formats Worldwide, 2002 (asa % of respondents receiving commercial e-mail), http://www.emarketer.com

Eyetrack III, 2004, Recall of Information Presented in Text vs. Multimedia Format, http://www.poynterextra.org/eyetrack2004/multimediarecall.htm

- 74 -

Galliers, R.D. (1991) Choosing Information Systems Research Approaches, in: Galliers, R.D.(Ed) (1992) Information Systems Research: Issues, Methods and Practical Guidelines, AlfredWaller: Henley-on-Thames, 144-162.

Global Reach, 2004, Evolution of Online Linguistic Populations, http://global-reach.biz/globstats/evol.html

Griffiths, R.T., 2002, Chapter Three: History of Electronic Mail, http://www.let.leidenuniv.nl/history/ivh/chap3.htm

Hoffman, D. I., 2000, The Revolution Will Not Be Televised: Introduction to the SpecialIssue on Marketing Science and the Internet, Marketing Science, Vol. 19, No. 1

IPT, 2004, IPT Email Marketing Survey 2004, http://www.ipt-ltd.co.uk/downloads/press/04_10_13_IPT_Research.pdf

Jackson, A. and DeCormier, R., 1999, ‘E-mail survey response rates: Targeting increasesresponse’, Journal of Marketing Intelligence and Planning, Vol. 17, No. 3, pp. 135–139.

Jennings. J., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http://www.clickz.com/showPage.html?page=3448841

Jupiter Research, 2005, E-mail Marketing Content Best Practices: Identifying the Impact ofContent on Response Behavior, http://www.jupiterresearch.com

Levin, W.C., 1988, Sociological Ideas: Concepts and Applications, Wadsworth: California.

Maloy, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http://www.clickz.com/showPage.html?page=3448841

Marketing Sherpa, 2003, Surprising Email Test Results: Text vs HTML Survey Versions,http://www.marketingsherpa.com/sample.cfm?contentID=2440#

Messaging Online, 2000, Year End 2000 Mailbox Report, http://www.messagingonline.com

- 75 -

Mintel, 2005, Extreme Sports - UK, http://academic.mintel.com/sinatra/academic/search_results/show&&type=RCItem&page=0&noaccess_page=0/display/id=114733/display/id=185037&section?select_section=185040

O'Leary, T., 2006, HTML vs. Text: The Debate is Over, http://www.group-mail.com/asp/common/articles.asp?id=191

Park, M.H., 2002, A Study of Effective Web Advertising Design to Maximize Click-Throughand Brand Awareness, http://www.idemployee.id.tue.nl/g.w.m.rauterberg/conferences/CD_doNotOpen/ADC/final_paper/130.pdf

Partner, L., 2003, Why Readers Prefer Text or HTML, http://www.marketingprofs.com/3/partner3.asp

Patterson, M., 2007, All about email open rates, http://www.campaignmonitor.com/blog/archives/2007/01/all_about_email_open_rates.html

Peppers, D. and Rogers, M., 2000, ‘Email marketing maximized’, Peppers and RodgersGroup, Stamford, CA.

Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496.

Pew Internet, 2005, How Women and Men Use the Internet, http://www.pewinternet.org/pdfs/PIP_Women_and_Men_online.pdf

Radicati Group, 2006, The Radicati Group, Inc. Releases Q4 2005 Market Numbers Update,http://www.radicati.com/uploaded_files/news/Q4-2005_PressRelease.pdf

silverPOP, 2005, SilverPOP's "2005 Broken Link Study", http://www.silverpop.com/downloads/BrokenLink_Study.pdf

Skaper, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http://www.clickz.com/showPage.html?page=3448841

Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis,MN: StatPac Inc.

- 76 -

Windham, L.,2000, ‘The soul of the new consumer’, Allworth Press, New York, NY.

- 77 -