application of fractional experiment in evaluation of ... · 2 application of fractional experiment...

16
2 Application of Fractional Experiment in Evaluation of Direct Marketing Effectiveness Sylwester Bialowas Aleksandra Kaniewska-Seba When testing tools and marketing messsages, the research methods not based on declarations, but on the buyers behavior measurement, are increasingly popular. One can observe the growing importance of obtaining data from observations (eg mystery shopping) and physiological measurements (eg neuromarketing). Experiments, according to the research conducted among Polish marketing managers and employees of advertising agencies, are much less used in this field. One of the obstacles may be a methodological gap in knowledge about the application (in particular about schemes with many variables) in testing and improving marketing communication tools and messages. In the paper we discuss the use of fractional experiment in optimizing direct marketing efforts. We present an example of experiment usage to evaluate versions of promotional e-mailing of Postgraduate Studies in Sales Management at the Poznan University of Economics. JEL: D12, M31 Keywords: fractional experiment, effectiveness of the direct marketing, Fractional experiment as a marketing research method Experiment with many factors has been rarely discussed in Polish literature on marketing research in terms of its use for measuring the results of marketing communications; additionally, if discussed, it was done in a rather perfunctory manner, and without being too detailed in terms of methodology. In the books addressed to students and practitioners of marketing most of the suggestions on how to test the effectiveness, have been limited to the simplest experimental designs with one independent variable and two treatment groups - a test one and a control one. These tests are called A/B splits, test-control or champion challenger testing (zob. Kall, 2000, Czupryna, 2004, Trojanowski, 2010). Meanwhile, there are numerous statistically-sophisticated experimental techniques which can be implemented to study the effects of advertising or direct marketing (Bell et al., 2006). The subject of using experiments with many variables in studies on marketing communications has a slightly better representation in the English literature. There are several publications dedicated exclusively to the use of experiments in the assessment of marketing activities, in which part of the discussion concerns strictly studying the effects of marketing communications. Special consideration should be given to a publication by Ledolter and Swersey (2007), who presented principles and examples of factorial experiments (both full and fractional) used e.g. to improve the effectiveness of direct marketing. Applications of factorial and fractional factorial design to advertising, direct marketing and retail can be also found in the papers published in marketing journals. For example, Curhan (1974) used fractional factorial experiment to examine the effects of advertising, display location, and price on the sales of fruits and vegetables in a supermarket. Wilkinson et al. (1982) described a factorial design for assessing the impact of promotion, price, and display on the sales of selected items in grocery stores. International Journal of Business and Management Vol. II (1), 2014 http://www.iises.net ISSN 2336-2197

Upload: hoangque

Post on 27-Feb-2019

218 views

Category:

Documents


0 download

TRANSCRIPT

2

Application of Fractional Experiment in Evaluation of Direct

Marketing Effectiveness

Sylwester Bialowas

Aleksandra Kaniewska-Seba

When testing tools and marketing messsages, the research methods not based on declarations, but

on the buyers behavior measurement, are increasingly popular. One can observe the growing

importance of obtaining data from observations (eg mystery shopping) and physiological

measurements (eg neuromarketing). Experiments, according to the research conducted among

Polish marketing managers and employees of advertising agencies, are much less used in this field.

One of the obstacles may be a methodological gap in knowledge about the application (in particular

about schemes with many variables) in testing and improving marketing communication tools and

messages. In the paper we discuss the use of fractional experiment in optimizing direct marketing

efforts. We present an example of experiment usage to evaluate versions of promotional e-mailing

of Postgraduate Studies in Sales Management at the Poznan University of Economics.

JEL: D12, M31

Keywords: fractional experiment, effectiveness of the direct marketing,

Fractional experiment as a marketing research method

Experiment with many factors has been rarely discussed in Polish literature on marketing research

in terms of its use for measuring the results of marketing communications; additionally, if

discussed, it was done in a rather perfunctory manner, and without being too detailed in terms of

methodology. In the books addressed to students and practitioners of marketing most of the

suggestions on how to test the effectiveness, have been limited to the simplest experimental designs

with one independent variable and two treatment groups - a test one and a control one. These tests

are called A/B splits, test-control or champion challenger testing (zob. Kall, 2000, Czupryna, 2004,

Trojanowski, 2010). Meanwhile, there are numerous statistically-sophisticated experimental

techniques which can be implemented to study the effects of advertising or direct marketing (Bell et

al., 2006).

The subject of using experiments with many variables in studies on marketing communications has

a slightly better representation in the English literature. There are several publications dedicated

exclusively to the use of experiments in the assessment of marketing activities, in which part of the

discussion concerns strictly studying the effects of marketing communications. Special

consideration should be given to a publication by Ledolter and Swersey (2007), who presented

principles and examples of factorial experiments (both full and fractional) used e.g. to improve the

effectiveness of direct marketing. Applications of factorial and fractional factorial design to

advertising, direct marketing and retail can be also found in the papers published in marketing

journals. For example, Curhan (1974) used fractional factorial experiment to examine the effects of

advertising, display location, and price on the sales of fruits and vegetables in a supermarket.

Wilkinson et al. (1982) described a factorial design for assessing the impact of promotion, price,

and display on the sales of selected items in grocery stores.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

3

A full factorial experiment tests and analyses the effects of two or more independent variables (also

known as factors) on at least two levels. A design of such an experiment includes all possible

combinations of the adopted levels across the examined factors. The influences of all factors are

estimated with the same precision as if it occurred in separate experiments with one independent

variable (Kirk, 2013, s. 431). Studying the influence of different factors on dependent variables is

easier and usually less costly, as it requires fewer runs than experiments with one variable1(Ledolter

and Swersey, 2007, s. 96, Diamond, 2011, s. 93). Moreover, factorial experiments can analyse two

kinds of effects, which is of cognitive value. First, the influence of individual factors on dependent

variables is measured in the form of main effects. Second, if the variables are not independent, then,

the interactions between them are examined (Shadish et al., 2002, s. 263, Kirk, 2013, s. 431).

According to Cochran and Cox (1992, s. 152) factorial experiments are useful in exploratory

studies, whose aim is to determine the effects of several factors at the same time, and in such

situations when principles/recommendations applicable in different conditions are looked for.

The simplest approach in a factorial experiment implies the establishment of fixed variables

independent on two levels. In this situation, the number of runs (tested combinations) is 2k, where k

– the number of factors. The number of combinations rises when we increase the number of

analysed independent variables and/or the number of levels on which we differentiate them. For

example, when a researcher wants to anylyze the influence of two factors on two levels, two factors

on three levels and one factor on four levels on a dependent variable, then, in order to test all the

possible combinations he/she should conduct 22 x 3

2 x 4

1 =144 runs.

An alternative solution is to test only selected combinations, namely to conduct a fractional

experiment which includes only a part - for example, 1/2, 1/3, 1/4, 1/8, 1/9 etc. of the combinations

of the variables occurring in a factorial experiment. If a full factorial experiment assumes the

analysis of the effect of four independent variables (each of them modified on three levels), then 81

different combinations are required to be tested. In a fractional experiment it is enough to analyse

only 1/3, namely 27 combinations (Kirk, 2013, s. 803).

The use of experiments in fractional replication was proposed by Finney. In 1945 he outlined how

to design fractional experiments 2n and 3

n, and 4 x 2

4 to be applied in agriculture (n - number of

independent variables). In 1946 Plackett and Burman created schemes which significantly reduce

the number of combinations in experiments with n factors modified on levels 2 to 7 (Cochran and

Cox, 1992, s. 244). In 1947 Kempthorne, basing on Finney’s works, extended the pk schemes,

where p is a prime number, and k is the number of variables (Kirk, 2013, s. 803).

Fractional experiments are used when conducting a factorial experiment requires too much financial

and/or time commitment. They are often used when many factors are studied, and the primary

objective of the researcher is to identify those that have a significant influence on the dependent

variable (so-called screening experiments). This way, one can efficiently analyse a relatively big

number of variables in the initial experiment and in the following ones, which are to focus on the

most significant variables and interactions, one can deepen the analysis of its results (Kirk, 2013, s.

804).

Decreasing the number of runs simplifies the process of design and organization of the study,

reduces its costs, however, it also decreases the amount of information on the interactions between

the independent variables. The design resolution (R), usually written in Roman numerals, informs

us about the degree to which estimated main effects are confounded with the interactions. And thus:

R=III means that the main effects are confounded with 2-factor interactions

1Friedman and Savage present arguments proving that a well-planned series of experiments with one independent

variable allows for faster finding the level of a tested factor which produces the best results. As in: COCHRAN, W. G.

& COX, G. M. 1992. Experimental Designs, Canada, John Wiley & Sons. p. 152

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

4

R=IV, main effects are confounded with 3-factor interactions, and 2-factor interactions with other

2-factor interactions

R=V, main effects are confounded with 4-factor interactions, and 2-factor interactions with 3-factor

interactions.

In practice, the most often used schemes are those with variables on two or three levels, as it

increases the probability of the interactions not being significant (Kirk, 2013, s. 803).

The background of the study

In times of economic slowdown, in the education sector there has also been a decline in interest in

paid postgraduate courses. This unfavourable trend has also been observed on those courses, which

constituted very popular choices a few years ago (marketing, sales, real estate, logistics). At that

time, good reputation of the university, a well-designed informative website, and recommendations

of graduates were enough to attract a relatively large number of new students. Additional

promotional activities were not necessary.

When the number of candidates started to fall, an attempt was made to identify the key

communication channels and to use them to promote the consecutive editions of the course (of

course, with a limited budget being taken into account). Conversations with the graduates of

previous editions confirmed the initial assumptions; namely, that the traditional promotional tools

(printed catalogues and press ads), which until recently were often used in the education sector,

have now become of marginal importance. A more effective and, very often, a cheaper solution are

online communication channels. The main source of information and, at the same time, the

encouragement to take advantage of the postgraduate courses offer are university websites - the

websites dedicated to postgraduate courses, to be precise. Some of the graduates of previous

editions assumed in advance that they would take up a postgraduate course on one or several

universities of their choice. The others searched the websites of various universities (usually from a

particular region), looking for an education product tailored to their current or future (anticipated)

professional situation.

The e-mailing campaign that was to promote the Sales Management Postgraduate Course at Poznan

University of Economics was undertaken with a view to the latter group in particular. These actions

were designed to build awareness regarding the abovementioned course, to inform about its frame

curriculum, as well as to encourage the prospective students to become familiar with detailed

information on the university website.

The aims of the study

While designing a direct marketing campaign, a key problem is to convince potential clients to take

advantage of the sent offer. According to the Direct Marketing Association in Poland , usually a

relatively small reaction of the recipients can be expected (BTL pl 2006, , 2005). Furthermore, by

analysing trends in West European markets, increasing difficulties and, what comes with it, costs of

reaching potential clients can be anticipated. Therefore, it is worth to identify the determinants of

the effectiveness of direct marketing tools, and to enhance actions in these areas.

In the literature there are examples of studies in which experimental design was to identify factors

determining the effectiveness of direct mail type advertising (przegląd tego typu badań prezentują

Feld et al., 2013). Studies showing which factors and to what degree determine the clients’

inclination to take advantage of the offers sent by e-mail are much more scarce (Magee, 2013,

Micheaux, 2011, Tezinde et al., 2002, White et al., 2008).

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

5

The present paper includes a description and results of a fractional factorial experiment conducted

to identify factors determining the effectiveness of an e-mailing campaign promoting the Sales

Management Postgraduate Course.

The e-mailing campaign was simultaneously an experiment. Its objective - in a cognitive sense -

was to verify the existing knowledge, and to acquire new information on the factors determining the

effectiveness of e-mailing, with the specificity of educational services being taken into account. The

methodological aim, on the other hand, was to determine the possibilities and limitations resulting

from the use of different experimental designs in the study of the conditions of the direct marketing

(e-mailing in this case) effectiveness.

Methodology and results2

The first stage of the study

The study was conducted in March 2013 in cooperation with the Pracuj.pl Group, which was

selected because of the size and structure of its e-mail address base. The selection criteria offered by

the abovementioned company, and target groups isolated on their basis, corresponded to the profile

of the participants of the previous editions of the course3.

The promotional e-mail informed about the spring edition of the Sales Management Postgraduate

Course - one that is usually less popular than the traditional autumn editions - and encouraged to

enrol.

The experiment was conducted on a sample of 16,200 subjects. Subjects who were selected

constituted the core target of the Sales Management Postgraduate Course: with university

education, related to sales, working as high-/mid-level managers, directors or specialists with at

least 2-year professional experience, residing in the Polish provinces: Wielkopolska, Lubuskie,

Zachodniopomorskie, Dolnośląskie, or Kujawsko-Pomorskie.

The content and the design of the e-mail was developed taking into consideration basic rules for

creating promotional electronic mails presented in the marketing literature. Due to a relatively small

number of academic publications, guidelines published in education publications and those for

practitioners were used (Trojanowski, 2010, s. 396-398, Sala, 2011a, Sala, 2011b, Sala, 2012). In

addition, general principles of creating an advertising message were employed, along with - in view

of the many similarities between the two tools - the principles of creating cover letters in traditional

mailings (m.in. Czupryna, 2004, s. 163-173). In order for the specificity of the promotion of

educational offer to be taken into account, the Heads and lecturers of postgraduate courses in sales

and marketing at Poznan University of Economics were consulted.

The main research dilemmas at the time of designing the experiment were on the selection of

independent variables. General principles regarding the creation of effective e-mail type mailings

were subject to verification, with the consideration of the specificity of the promotion of

educational offer. Based on the literature and conversations with experts, testing the following

factors was taken into consideration: name of the sender, subject line of the e-mail, personalization

(e.g. in the greeting), a graduate’s recommendation, the lecturers’ names, the number of links

(banners) included in the e-mail, the way the price is presented, the type of the photo, the technical

version of the e-mail (text version vs. html version). Due to the limitations on the database provider

2The study was financed from the resources assigned for the scientific research in the years 2010-2013 under a research

project N N112 3238 38 3The Pracuj.pl Group offers segmentation according to the following criteria: profile (student, graduate, specialist,

manager), region (by province and abroad), age, sex, level of education, date of graduation, type of completed

university programme, occupational status, industry in which they want to work in, knowledge of foreign languages

(prepared on the basis of “Educational e-mailing brief” sent by the Pracuj.pl Group)

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

6

and the mailing distributor’s side, we abandoned letter personalisation4 and differentiating the

technical version. Finally, seven independent variables on two levels were selected to test the

determinants of e-mailing effectiveness.

Sender - the name of the university (+) vs. the name of the promoted postgraduate course (-). The

name which appears in the sender field significantly influences the propensity to open the e-mail

type mailing. The research problem which appeared this time was as follows: Should good

reputation of the university (Poznan University of Economics) be used, or should the awareness of

the concrete course (Sales Management at Poznan University of Economics) be built through the

name of the sender?

Subject line - encouraging (+) vs. informative (-). Different experiences5 in this area led to the

preparation of two versions of subject lines to be tested. The “+” option was “Start a postgraduate

course in the spring”, and the “-”: version was: “A postgraduate course from March”

Recommendation – yes (+) vs. no (-). Recommendations play a significant role in the promotion of

services, including the educational ones. Hence, a decision was made to determine how a positive

opinion of a course graduate (and at the same time a Sales Manager in one of the leading FMCG

companies) would translate into the effectiveness of the promotional activities. Taking this factor

into consideration in the study was at the same time a test of the e-mail’s length. Removing one

paragraph with the recommendation, shortened the e-mail to such a degree, that the banner(s)

directing to the course’s website was/were visible right after opening the e-mail, without the

necessity to scroll the screen.

Name of the leading lecturer - yes (+) vs. no (-). People providing educational services influence

their heterogeneity and significantly determine the level of satisfaction from the service

(Kaniewska-Sęba and Kwiatek, 2009). Therefore, the experiment tested to what degree the name of

the leading lecturer of Poznan University of Economics (evaluated very well also for teaching

classes within the analysed course) may contribute to the success of the promotion.

Number of banners - two links (+) vs. one link. It was assumed that increasing the number of

banners may increase the their recipient’s propensity to click them, and that might lead to the rise in

the CTR value. That is why, half of the tested e-mail versions contained two banners (links) - to the

website of the promoted course and to the electronic registration form, while half of them contained

only a banner (link) directing to the website of the promoted course (-).

The price of the course per hour of class - the rounded amount (+) vs. the exact amount (-). Price is

one of the most significant factors determining the volume of demand, also when it comes to

educational services. There are many possibilities how to present it; that is why, this particular

independent variable proved the most difficult to operationalize. Because it was not possible to

interfere with the discount policy of the postgraduate course, the possibility of presenting two

different levels of price (one presented on the university’s website and the promotional one) was

excluded. In order to make this element of the offer more attractive, a decision was made to present

the price per hour of class. It was assumed that the rounded amount (in the expression: “less than

PLN25 per hour of class) will be a more attractive alternative than giving the exact fee per hour of

class (the expression: “PLN24.74 per hour of class”).

4An additional, substantial argument for abandoning the attempt to personalise the e-mails are inconclusive results of

previous experiments in this area. See more WHITE, T. B., ZAHAY, D. L., THORBJ√∏RNSEN, H. & SHARON, S.

2008. Getting too personal: Reactance to highly personalized email solicitations. Marketing Letters, 19, 39-50.

5For example, in their publications, Trojanowski and Sala encourage to address the recipient in a direct way, e.g. “Open

me. See what’s inside. It’s worth it.” See TROJANOWSKI, M. 2010. Marketing bezpośredni. Koncepcja-zarządzanie-

instrumenty, Warszawa, Państwowe Wydawnictwo Ekonomiczne. p. 397 i SALA, P. 2011a. Jak pisać efektywne e-

maile (cz. I). Marketing w praktyce, 10, 84-86.Micheaux, on the other hand, shows that subject lines of neutral content

can be effective. See MICHEAUX, A. L. 2011. Managing e-mail Advertising Frequency from the Consumer

Perspective. Journal of Advertising, 40, 45-66.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

7

Photo – people’s faces (+) vs. the university building (-). In publications on advertising, numerous

ways to attract the recipients’ attention are listed. According to Crompton (za: Janiszewska et al.,

2009, s. 266) there is a group of elements which have a strong influence on recipients, whatever the

region of the world is. Such elements that increase the interest in advertisement include e.g.

people’s faces (that is why photos showing faces of a few young, smiling people were thought to be

a better alternative).

The main objective of a study of this kind was to identify the factors significantly determining the

effectiveness of e-mailing campaigns promoting the Sales Management Postgraduate Course. That

is why a screening approach was used. A relatively large number of analysed factors led to a

decision that on this stage of the study a factorial experiment including all possible combinations of

the studied independent variables will not be used (it would require for 27 = 128 runs to be

conducted). Not only due to time and financial limitations, but also because of the limited number

of the target group’s e-mail addresses in the available database. Having 16,200 addresses at

disposal, each of the 128 creations could be sent only to 126-127 people (16,200/128 = 126.5).

While designing the screening experiment, a scheme which would give the possibility to estimate

the main effects in the fewest possible runs was looked for. A 27-4

fractional factorial design was

selected for the study.

Table 1.

A 27-4

fractional factorial design

A B C D (AB) E (AC) F (BC) G (ABC)

1 + + + + + + +

2 + - - - - + +

3 - + - - + - +

4 - - + + - - +

5 - - - + + + -

6 + + - + - - -

7 - + + - - + -

8 + - + - + - -

Source: own work based on (Ledolter and Swersey, 2007, s. 123)

The selection of dependent variables was a derivative of the possibilities provided by the Pracuj.pl

Group. The database provider and the e-mailing distributor evaluates its effectiveness basing on two

measures. These are measures commonly used to evaluate such actions:

opening rate (OR) - calculated by dividing the number of recipients who opened the e-mail

messages by the total number of recipients;

click through rate (CTR) - calculated by dividing the number of recipients who clicked the link(s)

put in an e-mail by the total number of recipients;

The former of the indicated measures had a limited application, as its value could be influenced by

only two out of seven examined independent variables - the sender and the subject line of the e-

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

8

mail. Therefore, the latter measure was adopted as the dependent variable, and its value can be a

potential derivative of each of the studied factors. The results of the experiment from the first run of

the study are depicted in Table 2.

Table 2.

The results of the experiment from the first stage of the study - promoting the spring edition of the

course

Creatio

n 1

Creation

2

Creation

3

Creation

4

Creation

5

Creation

6

Creation

7

Creation

8

OR 10.12% 7.35% 7.25% 7.50% 7.90% 6.56% 7.11% 16.24%

CTR 0.93% 1.03% 1.13% 1.08% 1.43% 0.34% 1.13% 2.27%

Source: own work based on data delivered by the Pracuj.pl Group

Main effects for each of the examined factors were estimated on that basis. The main effect

can be defined as the difference in the OR or the CTR when changing a factor’s level from (-) to

(+). In the case of the A effect (the sender’s name), this is the difference between the results

achieved with e-mails containing the name of the postgraduate course and the results achieved with

e-mails containing the name of the university. For the A factor and the CTR measure of success the

main effect is calculated in the following way (0.93% + 1.03% + 0.34% + 2.27%)/4 - (1.13% +

1.08% + 1.43% + 1.13%)/4, which equals -0.05%. By doing so when estimating the main effects, it

was demonstrated that the four independent variables (the subject line of the e-mail, the

recommendation, the lecturer’s name and the number of banners) had a significant influence on the

effectiveness of the e-mailing campaign measured by the CTR.

Table 3.

Main effects (in percentage points)

A -0.05

B -0.57*

C 0.37*

D -0.45*

E 0.55*

F -0.08

G -0.25

*the effect is statistically significant for the value of alpha=0.056

Source: own work

It can be stated that:

6On the basis of the average value of the OR and the CTR and the samples size, a mean error for each measure was

calculated. After multiplying the mean error by 1.96 (for alpha=0.05), values allowing to determine whether the

calculated results are statistically significant were produced.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

9

A neutral subject line of the e-mail (“Postgraduate course from March”) increases the CTR by 0.57

percentage point.

The significant role of recommendation was confirmed. Its use in the mailing campaign increases

the CTR by 0.37 percentage point.

However, putting the name of the leading lecturer in the e-mail, decreases the effectiveness of the e-

mailing campaign measured by the CTR by 0.45 percentage point. (maybe the name was not known

for the significant part of the recipients, as it is a person associated with giving lectures, trainings

and couseling more often in marketing than in sales; one may also think whether providing the

lecturer’s academic degree/title - professor, Ph.D. in this case - did not raise associations with rather

a theoretical than a practical character of the classes)

Placing two banners increases the value of the CTR by 0.55 percentage point.

The results of the screening experiment constituted the basis for formulating conclusions as for the

factors determining the effectiveness of e-mailing in the promotion of educational offer. It was

decided that the results of the study are going to be verified in the next autumn edition of the course.

Therefore, two creations of electronic letters were designed - the first one which was to guarantee

the highest possible CTR measure (hereinafter referred to as “the best” creation) and its opposite

(“the worst” creation).

Four out of seven analysed features of the mailing from the first stage had a significant influence on

the recipients’ inclination to click the link included in the e-mail. Hence, for these elements there

was no doubt as to which level of independent variable should be assigned to “the best” creation,

and which one to “the worst” creation. As for the other features, a decision was made for them to be

on the same level as in the most effective (as far as the CTR is concerned) mailing from the first run

of the study.

Table 4. Selected elements of the e-mailing campaign verifying the results of the first experiment

Variable “The best” creation “The worst” creation

Sender/Headline Poznan University of Economics Sales Management at Poznan

University of Economics

Subject Sales Management - enrolment

for the October edition of the

course has just started

Think about a postgraduate

course before the summer

Recommendation Yes (the graduate - the FMCG

sales manager)

No

Lecturers’ names No (no concrete names) Yes (the name of the leading

lecturer)

Link to the

postgraduate course

website and to the

form

Both links Link only to the postgraduate

course website

Cost of the course Per hour of class (exact amount) Per hour of class (rounded

amount)

Photo of the Poznan University of

Economics’ building

of people’s faces

Source: own work

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

10

The mailing was conducted in May 2013. A decision to conduct the mailing campaign before the

summer was made deliberately. Increased interest in taking up postgraduate courses at Poznan

University of Economics (autumn editions) is usually observed in the second half of August and in

September, as most of the prospective students postpone the decision and dealing with all the

formalities to the last moment.7 The May promotional action was to speed up the decision-making

process of the prospective students of the course.

The re-use of the creations used in the promotion of the next edition of the course required a change

in the subject lines and the content of the message (Table 5.). Layout and other graphic elements

remained unchanged.

Table 5. Differences between mailing campaigns from March and May 2013

Mailing campaign from

March 2013

Mailing campaign

May 2013

Informative

subject line

in “the best”

creation

A postgraduate course from

March

Sales Management - enrollment

for the October edition of the

course has just started*

Encouraging

subject line

in “the worst”

creation

Start a postgraduate course in

the spring

Think about a postgraduate

course before the summer

First paragraph You’re invited to take part in

the Sales Management

Postgraduate Course at

Poznan University of

Economics. Already on March

23rd this year we start the

12th edition of a course for

sales/commercial department

managers and for those

aspiring to work on such

positions.

You’re invited to the autumn

edition of the Sales Management

Postgraduate Course at Poznan

University of Economics. This

year in October we start the 13th

edition of a course for

sales/commercial department

managers and for those aspiring

to work on such positions.

Info about the

enrolment

process

The enrolment lasts until 20th

March this year

The enrolment has already

started

*In this case, Poznan University of Economics was the sender. Thus, the name of the postgraduate

course was added, so that the awareness regarding the name of the course was built already at the

pre-opening the letter stage.

Source: own work

7The information comes from the personal experience of the author, who held the position of the Head of the Sales

Management Postgraduate Course at Poznan University of Economics for several years.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

11

A decision to change the target group was also made. First, the campaign was not to be conducted

on the subjects who were once examined. Second, there was a plan to check the possibility of

transferring (generalising) the results of the study to other people who also constitute a target group

of the Sales Management Postgraduate Course, yet not the core target group. It was assumed that

the decision regarding conducting the experiment on a sample with different characteristics would

have its consequences. Achieving the results projected in the first treatment cannot be expected.

However, significant differences between the results achieved in “the best” and “the worst” creation

can be expected.

“The best” and “the worst” creation was sent to 3,587 people defined by the database provider as

“people with university education, from the following departments: management, sales, customer

service, purchasing, from the following Polish provinces: Kujawsko-Pomorskie, Lubelskie,

Zachodnio-pomorskie, Dolnośląskie, Wielkopolskie”.

Table 6. Effects of the mailing verifying the results of the first stage of the study

“The best” creation “The worst” creation

OR 9.21% 6.56%

CTR 0.82% 0.61%

The CTR projected on the basis of

the results of the experiment 2.08% 0.15%

Source: own work based on data delivered by the Pracuj.pl Group

As anticipated, the received CTRs were at a level lower than expected, but both for OR and CTR

the difference in the effectiveness of both mailings is statistically significant with alpha=0.05. It can

be assumed that one of the reasons may be expanding the target group. It can be observed that each

year the number of people who start a postgraduate course right after receiving their Bachelor’s or

Master’s, and who have little experience in sales and only begin to aspire to manage a team of

salespeople is growing (therefore, the requirement of at least 2-year professional experience was not

used as in the case of the first stage of the study). Moreover, the promotion was also addressed to

people who are not strictly related to sales, but who work in sales-related areas (customer service,

purchasing).

Another reason of lower than anticipated results of the campaign might be the time of the promotion

- unfortunately the attempt to fight the demand for postgraduate courses being seasonal without any

additional support (e.g. a discount, or a gift for early registration or early fee payment) proved

unsuccessful. Finally, one might look for other reasons in factors not controlled by the researcher

(e.g. actions of the competition).

The second stage of the study

The results of the first experiment constituted the basis for designing creations for the next two

experimental designs. First, a decision on the subject was made, as this was the factor which

significantly influenced the effectiveness of the mailing campaign in the first stage of the study.

Attempts were made for the subject to sound fairly neutral, as this was what the experiment

conducted in the first stage indicated. At the same time, on the basis of the observations of other e-

mailing campaigns from the commercial practice, it was decided to extend the subject line

according to the “information + encouragement to act” pattern.

In the first stage of the study it turned out that none of the sender’s names significantly influenced

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

12

the inclination to click the banners in the e-mails. Therefore, the choice of the sender in this stage

was a derivative of the decision on the subject line. It was assumed that if there is not the name of

the course in the subject line, it should be placed in the “sender” field. In all tested combinations the

sender was therefore unified: “Sales Management at Poznan University of Economics” and the

subject line: “The enrolment has started. Think about a postgraduate course before the summer.”

Next, a base version of the e-mail was created. It had the same layout and the previously identified

measures of success were included:

Recommendation of the graduate, information on the fact that the classes are taught by the Poznan

University of Economics’ lecturers and practitioners (however, without giving concrete names), two

links: to the course’s website and to the electronic application form.

At this stage of the study, the “cognitive needs” regarding testing further factors were smaller. Thus,

a factorial experiment with a fewer number of independent variables modified on two levels was

planned to be conducted. Two designs were chosen: factorial (23) and fractional-factorial (2

4-1), in

order to verify in practice the possibilities and limitations of both approaches.

In the case of the first experiment, three independent variables on two levels were selected to

examine the determinants of the e-mailing effectiveness:

The location of “Learn more” banner, which included a link to the website of the promoted

postgraduate course - in the upper (+) vs. lower (-) part of the body of the e-mail. The results of the

previous experiment indicated that e-mails with two links are more effective. In the original version

they were located in the lower part of the text, closer to each other. This time, it was decided to

examine to what degree a change in location of one of them - the banner with a link to the course

website - may contribute to the increase in the number of clicks. It was assumed that placing the

banner higher than before would be a better option, so that it could be visible right after opening the

email (without the necessity to scroll).

Tone of the letter - encouragement, e.g. become..., get..., take part... (+) vs. neutral (like in the

original version). In this case, it was decided to verify the idea presented in guidelines for

practitioners, that the tone of a promotional letter, similarly to its subject line, should be direct.

The price of the course - per hour of class (+) vs. amount total (-). The results of the first

experiment showed that there is no significant difference between an exact and rounded price for

hour of class. This time, it was decided to examine how an alternative and totally different way of

price exposition (PLN24.74 per hour of class vs. PLN4,700 for 190 hours of classes) would

translate into the interest in the course (in this case - clicking the banners in the email).

The choice of the full factorial scheme for 3 factors on two levels assumes testing all combinations

of the analysed features in 23 = 8 runs. This leads to the possibility of examining all main effects: A,

B, C, D, and possible interactions: AB, AC, BC, and ABC. The scheme of this experiment is

depicted in Table 7.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

13

Table 7. Scheme of the 23factorial experiment

Number of

creations

(number of

runs)

A

(location of the

banner)

B

(tone of the

letter)

C

(price presentation)

1 + + +

2 + + -

3 + - +

4 - + +

5 - - +

6 + - -

7 - + -

8 - - -

Source: own work based on (Ledolter and Swersey, 2007, p. 67)

Values of the success measures (OR and CTR), and calculated on that basis main effects and

interactions are depicted in Tables 8. and 9.

Table 8. Results of the 23 factorial experiment

Source: own work based on data delivered by the Pracuj.pl Group

Table 9. Main effects and interactions (in percentage points)

A -0.0001

B -0.0010

C -0.0030*

AB -0.0007

AC 0.0000

BC -0.0009

ABC 0.0000

*the effect is statistically significant for the value of alpha=0.05

Source: own work

Creation

1

Creation

2

Creation

3

Creation

4

Creation

5

Creation

6

Creation

7

Creation

8

OR 8.27% 7.46% 7.59% 6.53% 8.75% 9.62% 7.59% 6.52%

CTR 0.45% 0.25% 0.71% 0.53% 0.65% 0.32% 0.32% 0.26%

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

14

It can be stated that the two tested features of email + the location of the banner and the tone of the

email did not significantly influenced its effectiveness. Only the price influenced the increased

number of clicks. The value of CTR for the creations in which the price was given per hour was

statistically significantly bigger that the number of clicks in the creation which gave the total price.

In the second experiment four independent variables were differentiated. Each of them was

modified at two levels:

The location of the LEARN MORE banner - analogous to the factorial experiment

The tone of the email - analogous to the factorial experiment

The price of the course - no price (+) vs. amount total (-). The objective of this study was to check

how effective the lack of price exposition is.

Three expressions pointing to significant advantages of the Sales Management Postgraduate Course

(the possibility of acquiring up-to-date knowledge and practical skills, and the possibility of

participating in inspiring classes) - marked with bold, dark green font (+) vs. no highlighting of the

text (-).

A fractional-factorial scheme was selected for 4 variables on 2 levels in the 24-1

= 8 runs (1/2 of the

full scheme) version. In this way there is a possibility to examine all main effects: A, B, C, D8, but

there was a problem with estimating 2-factor interactions, as the AB effect is confounded with CD,

AC with BD, BC with AD.

Table 10. Design of the 24-1

fractional factorial experiment

Source: own work based on (Ledolter and Swersey, 2007, p. 123)

Values of the success measures (OR and CTR) as part of the experiment, and calculated on that

basis main effects and interactions are depicted in Tables 11. and 12.

8Factor D is confounded with the ABC interaction (but it is assumed that this interaction is close to zero) More:

Ledolter i Swersey 2007, p. 122

Number of

creation

(number of

run)

A

(location of the

banner)

B

(tone of

the letter)

C

(price)

D = ABC

(highlighting selected parts of

text)

9 + + + +

10 + + - -

11 + - + -

12 - + + -

13 - - + +

14 + - - +

15 - + - +

16 - - - -

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

15

Table 11. Results of the 24-1

fractional factorial experiment

Source: own work based on data delivered by the Pracuj.pl Group

Table 12. Main effects and interactions (in percentage points)

A -0.0001

B -0.0010

C 0.0030

ABC 0.0000

*the effect is statistically significant for the value of alpha=0.05, Source: own work

The results of the fractional-factorial experiment are similar to the previous one. Almost all

examined features of the e-mail (in this case: the location of the banner, the tone of the letter, and

marking selected expressions in bold) did not significantly differentiate its effectiveness. Again, the

price was the only variable which influenced the change in CTR. The CTR value for the creations in

which the price was not given at all, was statistically significantly larger than the CTR value for the

creations in which the total price was given.

Unfortunately, “weak” main effects did not allow for the estimation and interpretation of the

interactions among the factors - both in the first and in the second experiment. The obtained results

will be used in practice to promote the Sales Management Postgraduate Course in September 2013.

Table 13. Assumptions on the design and the content of the creations which are to be tested in

August-September 2013

Variable “The best” creation

from the first

experiment

“The best” creation

from the second

experiment

Price Price per one hour of

classes

No price

Tone of the e-mail Neutral Neutral

Recommendation Yes (the graduate, the FMCG sales manager)

Lecturers’ names No (no concrete names)

Link to the postgraduate

course website

Up

Highlighting selected

expressions

Bold green font

Photo Photo of the building

Source: own work

Creatio

n 9

Creatio

n 10

Creatio

n 11

Creatio

n 12

Creatio

n 13

Creatio

n 14

Creatio

n 15

Creatio

n 16

OR 5.83% 6.51% 6.72% 7.38% 7.39% 5.97% 7.05% 6.64%

CTR 0.45% 0.25% 0.71% 0.53% 0.65% 0.32% 0.32% 0.26%

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

16

Conclusions

Summing up the results of the field experiments, it can be stated that they showed peculiar

conditions of the effectiveness of the e-mailing campaigns promoting a postgraduate course in a

situation in which there was no possibility to reduce the prices and to use material incentives of call

to action type. The following features proved significant: neutral tone of the letter, a

recommendation of the course’s graduate - a person on a manager position, no concrete names of

lecturers (at leach those with academic degrees), larger number of links, and not exposing the total

price of the course.

Possibilities and limitations if the field experiment were also verified. It is worth underlining that

using 3 different experimental designs, including one factorial and two fractional-factorial, enabled

to test a relatively large number of factors in relatively few runs. Unfortunately, “weak” main

effects in the second and third experiment limited the possibilities of analysing the interactions.

Determining how much it “costs”, in an informational sense, to add an additional factor to the

experiment (and simultaneously moving from a factorial design to a fractional design) was not

successful.

For the majority of the examined independent variables (with the exception of the way the price was

presented), there were no problems with their operationalization. The assumption that each variable

can be tested only on two levels was not perceived as a limitation.

In the future, it is worth to consider examining a larger number of factors influencing the graphic

layout (in the present study only the effects of the type of photo were testes), as it was them that

may have erred on the effectiveness of the e-mailing. Unfortunately, it is difficult to operationalize

independent variables in the area of graphic design. What is more, it should be noted that the

decisions in this respect are the result of subjective decisions and tastes (even if the creations are

designed by a professional graphic designer and are consulted with experts).

Possibilities and limitations of the field experiment were also verified. Above all, there was a

feeling of lack of control over all the factors which can influence the results of the experiment (e.g.

the actions of competition). Furthermore, because of the study being conducted by an external

entity, blocking variables could not be applied, and certain factors could not be examined. The use

of more complex schemes was limited not only due to the cost of the study, but also due to the

number of e-mail addresses.

References

2005. BTL pl 2006, c. II, raport - agencje - dyskusja. Marketing w praktyce [Online], 5. Available:

http://www.marketing.org.pl/nowe/mwp-5-2006-65-75.pdf.

BELL, G. H., LEDOLTER, J. & SWERSEY, A. J. 2006. Experimental design on the front lines of

marketing: Testing new ideas to increase direct mail sales. International Journal of Research in

Marketing, 23, 309-319.

COCHRAN, W. G. & COX, G. M. 1992. Experimental Designs, Canada, John Wiley & Sons.

CURHAN, R. C. 1974. The Effects of Merchandising and Temporary Promotional Activities on the

Sales of Fresh Fruits and Vegetables in Supermarkets. Journal of Marketing Research, 11, 286-294.

CZUPRYNA, K. 2004. Skuteczny marketing bezpośredni. Narzędzia, techniki, operacje, czyli jak

zarobić 17 milionów dolarów, Kraków, IFC Press.

DIAMOND, W. D. 2011. Practical Experiments Design for Engineers and Scientists, John Wiley

& Sons.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197

17

FELD, S., FRENZEN, H., KRAFFT, M., PETERS, K. & VERHOEF, P. C. 2013. The effects of

mailing design characteristics on direct mail campaign performance. International Journal of

Research in Marketing, 30, 143-159.

JANISZEWSKA, K., KORSAK, R., KWARCIAK, B., LEWIŃSKI, P., LISOWSKA-

MAGDZIARZ, M., NIERENBERG, B., NOWIŃSKA, E. & ZIMNY, R. 2009. Wiedza o reklamie.

Od pomysłu do efektu, Bielsko-Biała, Wydawnictwo Szkolne PWN.

KALL, J. 2000. Reklama, Warszawa, Polskie Wydawnictwo Ekonomiczne.

KANIEWSKA-SĘBA, A. & KWIATEK, P. Influence of industry characteristics on satisfaction-

loyalty link on the Polish market. In: SPRINGER, R. & CHADRABA, P., eds. 17th Annual

Conference on Marketing and Business Strategies for Central nad Eastern Europe, 2009 Vienna.

Vienna University of Economics and Business Administrations Institute of International Business.

KIRK, R. E. 2013. Experimental design: procedures for the behavioral sciences, Los Angeles, Sage

Publication.

LEDOLTER, J. & SWERSEY, A. J. 2007. Testing 1-2-3: Experimental Design with Applications in

Marketing and Service Operations, Stanford, Stanford University Press.

MAGEE, R. G. 2013. Can a print publication be equally effective online? Testing the effect of

medium type on marketing communications. Marketing Letters, 24, 85-95.

MICHEAUX, A. L. 2011. Managing e-mail Advertising Frequency from the Consumer

Perspective. Journal of Advertising, 40, 45-66.

SALA, P. 2011a. Jak pisać efektywne e-maile (cz. I). Marketing w praktyce, 10, 84-86.

SALA, P. 2011b. Copywriting w e-mail marketingu: treści (cz. II). Marketing w praktyce, 11, 80-

83.

SALA, P. 2012. Mailing wysoce personalizowany. Marketing w praktyce, 1, 72-75.

SHADISH, W. R., COOK, T. D. & CAMBELL, D. T. 2002. Experimental and Quasi-Experimental

Designs for Generalized Caual Inference, Belmont CA, Wadsworth Cengage Learning.

TEZINDE, T., SMITH, B. & MURPHY, J. 2002. Getting permission: Exploring factors affecting

permission marketing. Journal of Interactive Marketing, 16, 28-36.

TROJANOWSKI, M. 2010. Marketing bezpośredni. Koncepcja-zarządzanie-instrumenty,

Warszawa, Państwowe Wydawnictwo Ekonomiczne.

WHITE, T. B., ZAHAY, D. L., THORBJ√∏RNSEN, H. & SHARON, S. 2008. Getting too

personal: Reactance to highly personalized email solicitations. Marketing Letters, 19, 39-50.

WILKINSON, J. B., MASON, J. B. & PAKSOY, C. H. 1982. Assessing the Impact of Short-Term

Supermarket Strategy Variables. Journal of Marketing Research, 19, 72-86.

International Journal of Business and Management Vol. II (1), 2014

http://www.iises.net ISSN 2336-2197