application of fractional experiment in evaluation of ... · 2 application of fractional experiment...
TRANSCRIPT
2
Application of Fractional Experiment in Evaluation of Direct
Marketing Effectiveness
Sylwester Bialowas
Aleksandra Kaniewska-Seba
When testing tools and marketing messsages, the research methods not based on declarations, but
on the buyers behavior measurement, are increasingly popular. One can observe the growing
importance of obtaining data from observations (eg mystery shopping) and physiological
measurements (eg neuromarketing). Experiments, according to the research conducted among
Polish marketing managers and employees of advertising agencies, are much less used in this field.
One of the obstacles may be a methodological gap in knowledge about the application (in particular
about schemes with many variables) in testing and improving marketing communication tools and
messages. In the paper we discuss the use of fractional experiment in optimizing direct marketing
efforts. We present an example of experiment usage to evaluate versions of promotional e-mailing
of Postgraduate Studies in Sales Management at the Poznan University of Economics.
JEL: D12, M31
Keywords: fractional experiment, effectiveness of the direct marketing,
Fractional experiment as a marketing research method
Experiment with many factors has been rarely discussed in Polish literature on marketing research
in terms of its use for measuring the results of marketing communications; additionally, if
discussed, it was done in a rather perfunctory manner, and without being too detailed in terms of
methodology. In the books addressed to students and practitioners of marketing most of the
suggestions on how to test the effectiveness, have been limited to the simplest experimental designs
with one independent variable and two treatment groups - a test one and a control one. These tests
are called A/B splits, test-control or champion challenger testing (zob. Kall, 2000, Czupryna, 2004,
Trojanowski, 2010). Meanwhile, there are numerous statistically-sophisticated experimental
techniques which can be implemented to study the effects of advertising or direct marketing (Bell et
al., 2006).
The subject of using experiments with many variables in studies on marketing communications has
a slightly better representation in the English literature. There are several publications dedicated
exclusively to the use of experiments in the assessment of marketing activities, in which part of the
discussion concerns strictly studying the effects of marketing communications. Special
consideration should be given to a publication by Ledolter and Swersey (2007), who presented
principles and examples of factorial experiments (both full and fractional) used e.g. to improve the
effectiveness of direct marketing. Applications of factorial and fractional factorial design to
advertising, direct marketing and retail can be also found in the papers published in marketing
journals. For example, Curhan (1974) used fractional factorial experiment to examine the effects of
advertising, display location, and price on the sales of fruits and vegetables in a supermarket.
Wilkinson et al. (1982) described a factorial design for assessing the impact of promotion, price,
and display on the sales of selected items in grocery stores.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
3
A full factorial experiment tests and analyses the effects of two or more independent variables (also
known as factors) on at least two levels. A design of such an experiment includes all possible
combinations of the adopted levels across the examined factors. The influences of all factors are
estimated with the same precision as if it occurred in separate experiments with one independent
variable (Kirk, 2013, s. 431). Studying the influence of different factors on dependent variables is
easier and usually less costly, as it requires fewer runs than experiments with one variable1(Ledolter
and Swersey, 2007, s. 96, Diamond, 2011, s. 93). Moreover, factorial experiments can analyse two
kinds of effects, which is of cognitive value. First, the influence of individual factors on dependent
variables is measured in the form of main effects. Second, if the variables are not independent, then,
the interactions between them are examined (Shadish et al., 2002, s. 263, Kirk, 2013, s. 431).
According to Cochran and Cox (1992, s. 152) factorial experiments are useful in exploratory
studies, whose aim is to determine the effects of several factors at the same time, and in such
situations when principles/recommendations applicable in different conditions are looked for.
The simplest approach in a factorial experiment implies the establishment of fixed variables
independent on two levels. In this situation, the number of runs (tested combinations) is 2k, where k
– the number of factors. The number of combinations rises when we increase the number of
analysed independent variables and/or the number of levels on which we differentiate them. For
example, when a researcher wants to anylyze the influence of two factors on two levels, two factors
on three levels and one factor on four levels on a dependent variable, then, in order to test all the
possible combinations he/she should conduct 22 x 3
2 x 4
1 =144 runs.
An alternative solution is to test only selected combinations, namely to conduct a fractional
experiment which includes only a part - for example, 1/2, 1/3, 1/4, 1/8, 1/9 etc. of the combinations
of the variables occurring in a factorial experiment. If a full factorial experiment assumes the
analysis of the effect of four independent variables (each of them modified on three levels), then 81
different combinations are required to be tested. In a fractional experiment it is enough to analyse
only 1/3, namely 27 combinations (Kirk, 2013, s. 803).
The use of experiments in fractional replication was proposed by Finney. In 1945 he outlined how
to design fractional experiments 2n and 3
n, and 4 x 2
4 to be applied in agriculture (n - number of
independent variables). In 1946 Plackett and Burman created schemes which significantly reduce
the number of combinations in experiments with n factors modified on levels 2 to 7 (Cochran and
Cox, 1992, s. 244). In 1947 Kempthorne, basing on Finney’s works, extended the pk schemes,
where p is a prime number, and k is the number of variables (Kirk, 2013, s. 803).
Fractional experiments are used when conducting a factorial experiment requires too much financial
and/or time commitment. They are often used when many factors are studied, and the primary
objective of the researcher is to identify those that have a significant influence on the dependent
variable (so-called screening experiments). This way, one can efficiently analyse a relatively big
number of variables in the initial experiment and in the following ones, which are to focus on the
most significant variables and interactions, one can deepen the analysis of its results (Kirk, 2013, s.
804).
Decreasing the number of runs simplifies the process of design and organization of the study,
reduces its costs, however, it also decreases the amount of information on the interactions between
the independent variables. The design resolution (R), usually written in Roman numerals, informs
us about the degree to which estimated main effects are confounded with the interactions. And thus:
R=III means that the main effects are confounded with 2-factor interactions
1Friedman and Savage present arguments proving that a well-planned series of experiments with one independent
variable allows for faster finding the level of a tested factor which produces the best results. As in: COCHRAN, W. G.
& COX, G. M. 1992. Experimental Designs, Canada, John Wiley & Sons. p. 152
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
4
R=IV, main effects are confounded with 3-factor interactions, and 2-factor interactions with other
2-factor interactions
R=V, main effects are confounded with 4-factor interactions, and 2-factor interactions with 3-factor
interactions.
In practice, the most often used schemes are those with variables on two or three levels, as it
increases the probability of the interactions not being significant (Kirk, 2013, s. 803).
The background of the study
In times of economic slowdown, in the education sector there has also been a decline in interest in
paid postgraduate courses. This unfavourable trend has also been observed on those courses, which
constituted very popular choices a few years ago (marketing, sales, real estate, logistics). At that
time, good reputation of the university, a well-designed informative website, and recommendations
of graduates were enough to attract a relatively large number of new students. Additional
promotional activities were not necessary.
When the number of candidates started to fall, an attempt was made to identify the key
communication channels and to use them to promote the consecutive editions of the course (of
course, with a limited budget being taken into account). Conversations with the graduates of
previous editions confirmed the initial assumptions; namely, that the traditional promotional tools
(printed catalogues and press ads), which until recently were often used in the education sector,
have now become of marginal importance. A more effective and, very often, a cheaper solution are
online communication channels. The main source of information and, at the same time, the
encouragement to take advantage of the postgraduate courses offer are university websites - the
websites dedicated to postgraduate courses, to be precise. Some of the graduates of previous
editions assumed in advance that they would take up a postgraduate course on one or several
universities of their choice. The others searched the websites of various universities (usually from a
particular region), looking for an education product tailored to their current or future (anticipated)
professional situation.
The e-mailing campaign that was to promote the Sales Management Postgraduate Course at Poznan
University of Economics was undertaken with a view to the latter group in particular. These actions
were designed to build awareness regarding the abovementioned course, to inform about its frame
curriculum, as well as to encourage the prospective students to become familiar with detailed
information on the university website.
The aims of the study
While designing a direct marketing campaign, a key problem is to convince potential clients to take
advantage of the sent offer. According to the Direct Marketing Association in Poland , usually a
relatively small reaction of the recipients can be expected (BTL pl 2006, , 2005). Furthermore, by
analysing trends in West European markets, increasing difficulties and, what comes with it, costs of
reaching potential clients can be anticipated. Therefore, it is worth to identify the determinants of
the effectiveness of direct marketing tools, and to enhance actions in these areas.
In the literature there are examples of studies in which experimental design was to identify factors
determining the effectiveness of direct mail type advertising (przegląd tego typu badań prezentują
Feld et al., 2013). Studies showing which factors and to what degree determine the clients’
inclination to take advantage of the offers sent by e-mail are much more scarce (Magee, 2013,
Micheaux, 2011, Tezinde et al., 2002, White et al., 2008).
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
5
The present paper includes a description and results of a fractional factorial experiment conducted
to identify factors determining the effectiveness of an e-mailing campaign promoting the Sales
Management Postgraduate Course.
The e-mailing campaign was simultaneously an experiment. Its objective - in a cognitive sense -
was to verify the existing knowledge, and to acquire new information on the factors determining the
effectiveness of e-mailing, with the specificity of educational services being taken into account. The
methodological aim, on the other hand, was to determine the possibilities and limitations resulting
from the use of different experimental designs in the study of the conditions of the direct marketing
(e-mailing in this case) effectiveness.
Methodology and results2
The first stage of the study
The study was conducted in March 2013 in cooperation with the Pracuj.pl Group, which was
selected because of the size and structure of its e-mail address base. The selection criteria offered by
the abovementioned company, and target groups isolated on their basis, corresponded to the profile
of the participants of the previous editions of the course3.
The promotional e-mail informed about the spring edition of the Sales Management Postgraduate
Course - one that is usually less popular than the traditional autumn editions - and encouraged to
enrol.
The experiment was conducted on a sample of 16,200 subjects. Subjects who were selected
constituted the core target of the Sales Management Postgraduate Course: with university
education, related to sales, working as high-/mid-level managers, directors or specialists with at
least 2-year professional experience, residing in the Polish provinces: Wielkopolska, Lubuskie,
Zachodniopomorskie, Dolnośląskie, or Kujawsko-Pomorskie.
The content and the design of the e-mail was developed taking into consideration basic rules for
creating promotional electronic mails presented in the marketing literature. Due to a relatively small
number of academic publications, guidelines published in education publications and those for
practitioners were used (Trojanowski, 2010, s. 396-398, Sala, 2011a, Sala, 2011b, Sala, 2012). In
addition, general principles of creating an advertising message were employed, along with - in view
of the many similarities between the two tools - the principles of creating cover letters in traditional
mailings (m.in. Czupryna, 2004, s. 163-173). In order for the specificity of the promotion of
educational offer to be taken into account, the Heads and lecturers of postgraduate courses in sales
and marketing at Poznan University of Economics were consulted.
The main research dilemmas at the time of designing the experiment were on the selection of
independent variables. General principles regarding the creation of effective e-mail type mailings
were subject to verification, with the consideration of the specificity of the promotion of
educational offer. Based on the literature and conversations with experts, testing the following
factors was taken into consideration: name of the sender, subject line of the e-mail, personalization
(e.g. in the greeting), a graduate’s recommendation, the lecturers’ names, the number of links
(banners) included in the e-mail, the way the price is presented, the type of the photo, the technical
version of the e-mail (text version vs. html version). Due to the limitations on the database provider
2The study was financed from the resources assigned for the scientific research in the years 2010-2013 under a research
project N N112 3238 38 3The Pracuj.pl Group offers segmentation according to the following criteria: profile (student, graduate, specialist,
manager), region (by province and abroad), age, sex, level of education, date of graduation, type of completed
university programme, occupational status, industry in which they want to work in, knowledge of foreign languages
(prepared on the basis of “Educational e-mailing brief” sent by the Pracuj.pl Group)
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
6
and the mailing distributor’s side, we abandoned letter personalisation4 and differentiating the
technical version. Finally, seven independent variables on two levels were selected to test the
determinants of e-mailing effectiveness.
Sender - the name of the university (+) vs. the name of the promoted postgraduate course (-). The
name which appears in the sender field significantly influences the propensity to open the e-mail
type mailing. The research problem which appeared this time was as follows: Should good
reputation of the university (Poznan University of Economics) be used, or should the awareness of
the concrete course (Sales Management at Poznan University of Economics) be built through the
name of the sender?
Subject line - encouraging (+) vs. informative (-). Different experiences5 in this area led to the
preparation of two versions of subject lines to be tested. The “+” option was “Start a postgraduate
course in the spring”, and the “-”: version was: “A postgraduate course from March”
Recommendation – yes (+) vs. no (-). Recommendations play a significant role in the promotion of
services, including the educational ones. Hence, a decision was made to determine how a positive
opinion of a course graduate (and at the same time a Sales Manager in one of the leading FMCG
companies) would translate into the effectiveness of the promotional activities. Taking this factor
into consideration in the study was at the same time a test of the e-mail’s length. Removing one
paragraph with the recommendation, shortened the e-mail to such a degree, that the banner(s)
directing to the course’s website was/were visible right after opening the e-mail, without the
necessity to scroll the screen.
Name of the leading lecturer - yes (+) vs. no (-). People providing educational services influence
their heterogeneity and significantly determine the level of satisfaction from the service
(Kaniewska-Sęba and Kwiatek, 2009). Therefore, the experiment tested to what degree the name of
the leading lecturer of Poznan University of Economics (evaluated very well also for teaching
classes within the analysed course) may contribute to the success of the promotion.
Number of banners - two links (+) vs. one link. It was assumed that increasing the number of
banners may increase the their recipient’s propensity to click them, and that might lead to the rise in
the CTR value. That is why, half of the tested e-mail versions contained two banners (links) - to the
website of the promoted course and to the electronic registration form, while half of them contained
only a banner (link) directing to the website of the promoted course (-).
The price of the course per hour of class - the rounded amount (+) vs. the exact amount (-). Price is
one of the most significant factors determining the volume of demand, also when it comes to
educational services. There are many possibilities how to present it; that is why, this particular
independent variable proved the most difficult to operationalize. Because it was not possible to
interfere with the discount policy of the postgraduate course, the possibility of presenting two
different levels of price (one presented on the university’s website and the promotional one) was
excluded. In order to make this element of the offer more attractive, a decision was made to present
the price per hour of class. It was assumed that the rounded amount (in the expression: “less than
PLN25 per hour of class) will be a more attractive alternative than giving the exact fee per hour of
class (the expression: “PLN24.74 per hour of class”).
4An additional, substantial argument for abandoning the attempt to personalise the e-mails are inconclusive results of
previous experiments in this area. See more WHITE, T. B., ZAHAY, D. L., THORBJ√∏RNSEN, H. & SHARON, S.
2008. Getting too personal: Reactance to highly personalized email solicitations. Marketing Letters, 19, 39-50.
5For example, in their publications, Trojanowski and Sala encourage to address the recipient in a direct way, e.g. “Open
me. See what’s inside. It’s worth it.” See TROJANOWSKI, M. 2010. Marketing bezpośredni. Koncepcja-zarządzanie-
instrumenty, Warszawa, Państwowe Wydawnictwo Ekonomiczne. p. 397 i SALA, P. 2011a. Jak pisać efektywne e-
maile (cz. I). Marketing w praktyce, 10, 84-86.Micheaux, on the other hand, shows that subject lines of neutral content
can be effective. See MICHEAUX, A. L. 2011. Managing e-mail Advertising Frequency from the Consumer
Perspective. Journal of Advertising, 40, 45-66.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
7
Photo – people’s faces (+) vs. the university building (-). In publications on advertising, numerous
ways to attract the recipients’ attention are listed. According to Crompton (za: Janiszewska et al.,
2009, s. 266) there is a group of elements which have a strong influence on recipients, whatever the
region of the world is. Such elements that increase the interest in advertisement include e.g.
people’s faces (that is why photos showing faces of a few young, smiling people were thought to be
a better alternative).
The main objective of a study of this kind was to identify the factors significantly determining the
effectiveness of e-mailing campaigns promoting the Sales Management Postgraduate Course. That
is why a screening approach was used. A relatively large number of analysed factors led to a
decision that on this stage of the study a factorial experiment including all possible combinations of
the studied independent variables will not be used (it would require for 27 = 128 runs to be
conducted). Not only due to time and financial limitations, but also because of the limited number
of the target group’s e-mail addresses in the available database. Having 16,200 addresses at
disposal, each of the 128 creations could be sent only to 126-127 people (16,200/128 = 126.5).
While designing the screening experiment, a scheme which would give the possibility to estimate
the main effects in the fewest possible runs was looked for. A 27-4
fractional factorial design was
selected for the study.
Table 1.
A 27-4
fractional factorial design
A B C D (AB) E (AC) F (BC) G (ABC)
1 + + + + + + +
2 + - - - - + +
3 - + - - + - +
4 - - + + - - +
5 - - - + + + -
6 + + - + - - -
7 - + + - - + -
8 + - + - + - -
Source: own work based on (Ledolter and Swersey, 2007, s. 123)
The selection of dependent variables was a derivative of the possibilities provided by the Pracuj.pl
Group. The database provider and the e-mailing distributor evaluates its effectiveness basing on two
measures. These are measures commonly used to evaluate such actions:
opening rate (OR) - calculated by dividing the number of recipients who opened the e-mail
messages by the total number of recipients;
click through rate (CTR) - calculated by dividing the number of recipients who clicked the link(s)
put in an e-mail by the total number of recipients;
The former of the indicated measures had a limited application, as its value could be influenced by
only two out of seven examined independent variables - the sender and the subject line of the e-
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
8
mail. Therefore, the latter measure was adopted as the dependent variable, and its value can be a
potential derivative of each of the studied factors. The results of the experiment from the first run of
the study are depicted in Table 2.
Table 2.
The results of the experiment from the first stage of the study - promoting the spring edition of the
course
Creatio
n 1
Creation
2
Creation
3
Creation
4
Creation
5
Creation
6
Creation
7
Creation
8
OR 10.12% 7.35% 7.25% 7.50% 7.90% 6.56% 7.11% 16.24%
CTR 0.93% 1.03% 1.13% 1.08% 1.43% 0.34% 1.13% 2.27%
Source: own work based on data delivered by the Pracuj.pl Group
Main effects for each of the examined factors were estimated on that basis. The main effect
can be defined as the difference in the OR or the CTR when changing a factor’s level from (-) to
(+). In the case of the A effect (the sender’s name), this is the difference between the results
achieved with e-mails containing the name of the postgraduate course and the results achieved with
e-mails containing the name of the university. For the A factor and the CTR measure of success the
main effect is calculated in the following way (0.93% + 1.03% + 0.34% + 2.27%)/4 - (1.13% +
1.08% + 1.43% + 1.13%)/4, which equals -0.05%. By doing so when estimating the main effects, it
was demonstrated that the four independent variables (the subject line of the e-mail, the
recommendation, the lecturer’s name and the number of banners) had a significant influence on the
effectiveness of the e-mailing campaign measured by the CTR.
Table 3.
Main effects (in percentage points)
A -0.05
B -0.57*
C 0.37*
D -0.45*
E 0.55*
F -0.08
G -0.25
*the effect is statistically significant for the value of alpha=0.056
Source: own work
It can be stated that:
6On the basis of the average value of the OR and the CTR and the samples size, a mean error for each measure was
calculated. After multiplying the mean error by 1.96 (for alpha=0.05), values allowing to determine whether the
calculated results are statistically significant were produced.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
9
A neutral subject line of the e-mail (“Postgraduate course from March”) increases the CTR by 0.57
percentage point.
The significant role of recommendation was confirmed. Its use in the mailing campaign increases
the CTR by 0.37 percentage point.
However, putting the name of the leading lecturer in the e-mail, decreases the effectiveness of the e-
mailing campaign measured by the CTR by 0.45 percentage point. (maybe the name was not known
for the significant part of the recipients, as it is a person associated with giving lectures, trainings
and couseling more often in marketing than in sales; one may also think whether providing the
lecturer’s academic degree/title - professor, Ph.D. in this case - did not raise associations with rather
a theoretical than a practical character of the classes)
Placing two banners increases the value of the CTR by 0.55 percentage point.
The results of the screening experiment constituted the basis for formulating conclusions as for the
factors determining the effectiveness of e-mailing in the promotion of educational offer. It was
decided that the results of the study are going to be verified in the next autumn edition of the course.
Therefore, two creations of electronic letters were designed - the first one which was to guarantee
the highest possible CTR measure (hereinafter referred to as “the best” creation) and its opposite
(“the worst” creation).
Four out of seven analysed features of the mailing from the first stage had a significant influence on
the recipients’ inclination to click the link included in the e-mail. Hence, for these elements there
was no doubt as to which level of independent variable should be assigned to “the best” creation,
and which one to “the worst” creation. As for the other features, a decision was made for them to be
on the same level as in the most effective (as far as the CTR is concerned) mailing from the first run
of the study.
Table 4. Selected elements of the e-mailing campaign verifying the results of the first experiment
Variable “The best” creation “The worst” creation
Sender/Headline Poznan University of Economics Sales Management at Poznan
University of Economics
Subject Sales Management - enrolment
for the October edition of the
course has just started
Think about a postgraduate
course before the summer
Recommendation Yes (the graduate - the FMCG
sales manager)
No
Lecturers’ names No (no concrete names) Yes (the name of the leading
lecturer)
Link to the
postgraduate course
website and to the
form
Both links Link only to the postgraduate
course website
Cost of the course Per hour of class (exact amount) Per hour of class (rounded
amount)
Photo of the Poznan University of
Economics’ building
of people’s faces
Source: own work
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
10
The mailing was conducted in May 2013. A decision to conduct the mailing campaign before the
summer was made deliberately. Increased interest in taking up postgraduate courses at Poznan
University of Economics (autumn editions) is usually observed in the second half of August and in
September, as most of the prospective students postpone the decision and dealing with all the
formalities to the last moment.7 The May promotional action was to speed up the decision-making
process of the prospective students of the course.
The re-use of the creations used in the promotion of the next edition of the course required a change
in the subject lines and the content of the message (Table 5.). Layout and other graphic elements
remained unchanged.
Table 5. Differences between mailing campaigns from March and May 2013
Mailing campaign from
March 2013
Mailing campaign
May 2013
Informative
subject line
in “the best”
creation
A postgraduate course from
March
Sales Management - enrollment
for the October edition of the
course has just started*
Encouraging
subject line
in “the worst”
creation
Start a postgraduate course in
the spring
Think about a postgraduate
course before the summer
First paragraph You’re invited to take part in
the Sales Management
Postgraduate Course at
Poznan University of
Economics. Already on March
23rd this year we start the
12th edition of a course for
sales/commercial department
managers and for those
aspiring to work on such
positions.
You’re invited to the autumn
edition of the Sales Management
Postgraduate Course at Poznan
University of Economics. This
year in October we start the 13th
edition of a course for
sales/commercial department
managers and for those aspiring
to work on such positions.
Info about the
enrolment
process
The enrolment lasts until 20th
March this year
The enrolment has already
started
*In this case, Poznan University of Economics was the sender. Thus, the name of the postgraduate
course was added, so that the awareness regarding the name of the course was built already at the
pre-opening the letter stage.
Source: own work
7The information comes from the personal experience of the author, who held the position of the Head of the Sales
Management Postgraduate Course at Poznan University of Economics for several years.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
11
A decision to change the target group was also made. First, the campaign was not to be conducted
on the subjects who were once examined. Second, there was a plan to check the possibility of
transferring (generalising) the results of the study to other people who also constitute a target group
of the Sales Management Postgraduate Course, yet not the core target group. It was assumed that
the decision regarding conducting the experiment on a sample with different characteristics would
have its consequences. Achieving the results projected in the first treatment cannot be expected.
However, significant differences between the results achieved in “the best” and “the worst” creation
can be expected.
“The best” and “the worst” creation was sent to 3,587 people defined by the database provider as
“people with university education, from the following departments: management, sales, customer
service, purchasing, from the following Polish provinces: Kujawsko-Pomorskie, Lubelskie,
Zachodnio-pomorskie, Dolnośląskie, Wielkopolskie”.
Table 6. Effects of the mailing verifying the results of the first stage of the study
“The best” creation “The worst” creation
OR 9.21% 6.56%
CTR 0.82% 0.61%
The CTR projected on the basis of
the results of the experiment 2.08% 0.15%
Source: own work based on data delivered by the Pracuj.pl Group
As anticipated, the received CTRs were at a level lower than expected, but both for OR and CTR
the difference in the effectiveness of both mailings is statistically significant with alpha=0.05. It can
be assumed that one of the reasons may be expanding the target group. It can be observed that each
year the number of people who start a postgraduate course right after receiving their Bachelor’s or
Master’s, and who have little experience in sales and only begin to aspire to manage a team of
salespeople is growing (therefore, the requirement of at least 2-year professional experience was not
used as in the case of the first stage of the study). Moreover, the promotion was also addressed to
people who are not strictly related to sales, but who work in sales-related areas (customer service,
purchasing).
Another reason of lower than anticipated results of the campaign might be the time of the promotion
- unfortunately the attempt to fight the demand for postgraduate courses being seasonal without any
additional support (e.g. a discount, or a gift for early registration or early fee payment) proved
unsuccessful. Finally, one might look for other reasons in factors not controlled by the researcher
(e.g. actions of the competition).
The second stage of the study
The results of the first experiment constituted the basis for designing creations for the next two
experimental designs. First, a decision on the subject was made, as this was the factor which
significantly influenced the effectiveness of the mailing campaign in the first stage of the study.
Attempts were made for the subject to sound fairly neutral, as this was what the experiment
conducted in the first stage indicated. At the same time, on the basis of the observations of other e-
mailing campaigns from the commercial practice, it was decided to extend the subject line
according to the “information + encouragement to act” pattern.
In the first stage of the study it turned out that none of the sender’s names significantly influenced
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
12
the inclination to click the banners in the e-mails. Therefore, the choice of the sender in this stage
was a derivative of the decision on the subject line. It was assumed that if there is not the name of
the course in the subject line, it should be placed in the “sender” field. In all tested combinations the
sender was therefore unified: “Sales Management at Poznan University of Economics” and the
subject line: “The enrolment has started. Think about a postgraduate course before the summer.”
Next, a base version of the e-mail was created. It had the same layout and the previously identified
measures of success were included:
Recommendation of the graduate, information on the fact that the classes are taught by the Poznan
University of Economics’ lecturers and practitioners (however, without giving concrete names), two
links: to the course’s website and to the electronic application form.
At this stage of the study, the “cognitive needs” regarding testing further factors were smaller. Thus,
a factorial experiment with a fewer number of independent variables modified on two levels was
planned to be conducted. Two designs were chosen: factorial (23) and fractional-factorial (2
4-1), in
order to verify in practice the possibilities and limitations of both approaches.
In the case of the first experiment, three independent variables on two levels were selected to
examine the determinants of the e-mailing effectiveness:
The location of “Learn more” banner, which included a link to the website of the promoted
postgraduate course - in the upper (+) vs. lower (-) part of the body of the e-mail. The results of the
previous experiment indicated that e-mails with two links are more effective. In the original version
they were located in the lower part of the text, closer to each other. This time, it was decided to
examine to what degree a change in location of one of them - the banner with a link to the course
website - may contribute to the increase in the number of clicks. It was assumed that placing the
banner higher than before would be a better option, so that it could be visible right after opening the
email (without the necessity to scroll).
Tone of the letter - encouragement, e.g. become..., get..., take part... (+) vs. neutral (like in the
original version). In this case, it was decided to verify the idea presented in guidelines for
practitioners, that the tone of a promotional letter, similarly to its subject line, should be direct.
The price of the course - per hour of class (+) vs. amount total (-). The results of the first
experiment showed that there is no significant difference between an exact and rounded price for
hour of class. This time, it was decided to examine how an alternative and totally different way of
price exposition (PLN24.74 per hour of class vs. PLN4,700 for 190 hours of classes) would
translate into the interest in the course (in this case - clicking the banners in the email).
The choice of the full factorial scheme for 3 factors on two levels assumes testing all combinations
of the analysed features in 23 = 8 runs. This leads to the possibility of examining all main effects: A,
B, C, D, and possible interactions: AB, AC, BC, and ABC. The scheme of this experiment is
depicted in Table 7.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
13
Table 7. Scheme of the 23factorial experiment
Number of
creations
(number of
runs)
A
(location of the
banner)
B
(tone of the
letter)
C
(price presentation)
1 + + +
2 + + -
3 + - +
4 - + +
5 - - +
6 + - -
7 - + -
8 - - -
Source: own work based on (Ledolter and Swersey, 2007, p. 67)
Values of the success measures (OR and CTR), and calculated on that basis main effects and
interactions are depicted in Tables 8. and 9.
Table 8. Results of the 23 factorial experiment
Source: own work based on data delivered by the Pracuj.pl Group
Table 9. Main effects and interactions (in percentage points)
A -0.0001
B -0.0010
C -0.0030*
AB -0.0007
AC 0.0000
BC -0.0009
ABC 0.0000
*the effect is statistically significant for the value of alpha=0.05
Source: own work
Creation
1
Creation
2
Creation
3
Creation
4
Creation
5
Creation
6
Creation
7
Creation
8
OR 8.27% 7.46% 7.59% 6.53% 8.75% 9.62% 7.59% 6.52%
CTR 0.45% 0.25% 0.71% 0.53% 0.65% 0.32% 0.32% 0.26%
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
14
It can be stated that the two tested features of email + the location of the banner and the tone of the
email did not significantly influenced its effectiveness. Only the price influenced the increased
number of clicks. The value of CTR for the creations in which the price was given per hour was
statistically significantly bigger that the number of clicks in the creation which gave the total price.
In the second experiment four independent variables were differentiated. Each of them was
modified at two levels:
The location of the LEARN MORE banner - analogous to the factorial experiment
The tone of the email - analogous to the factorial experiment
The price of the course - no price (+) vs. amount total (-). The objective of this study was to check
how effective the lack of price exposition is.
Three expressions pointing to significant advantages of the Sales Management Postgraduate Course
(the possibility of acquiring up-to-date knowledge and practical skills, and the possibility of
participating in inspiring classes) - marked with bold, dark green font (+) vs. no highlighting of the
text (-).
A fractional-factorial scheme was selected for 4 variables on 2 levels in the 24-1
= 8 runs (1/2 of the
full scheme) version. In this way there is a possibility to examine all main effects: A, B, C, D8, but
there was a problem with estimating 2-factor interactions, as the AB effect is confounded with CD,
AC with BD, BC with AD.
Table 10. Design of the 24-1
fractional factorial experiment
Source: own work based on (Ledolter and Swersey, 2007, p. 123)
Values of the success measures (OR and CTR) as part of the experiment, and calculated on that
basis main effects and interactions are depicted in Tables 11. and 12.
8Factor D is confounded with the ABC interaction (but it is assumed that this interaction is close to zero) More:
Ledolter i Swersey 2007, p. 122
Number of
creation
(number of
run)
A
(location of the
banner)
B
(tone of
the letter)
C
(price)
D = ABC
(highlighting selected parts of
text)
9 + + + +
10 + + - -
11 + - + -
12 - + + -
13 - - + +
14 + - - +
15 - + - +
16 - - - -
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
15
Table 11. Results of the 24-1
fractional factorial experiment
Source: own work based on data delivered by the Pracuj.pl Group
Table 12. Main effects and interactions (in percentage points)
A -0.0001
B -0.0010
C 0.0030
ABC 0.0000
*the effect is statistically significant for the value of alpha=0.05, Source: own work
The results of the fractional-factorial experiment are similar to the previous one. Almost all
examined features of the e-mail (in this case: the location of the banner, the tone of the letter, and
marking selected expressions in bold) did not significantly differentiate its effectiveness. Again, the
price was the only variable which influenced the change in CTR. The CTR value for the creations in
which the price was not given at all, was statistically significantly larger than the CTR value for the
creations in which the total price was given.
Unfortunately, “weak” main effects did not allow for the estimation and interpretation of the
interactions among the factors - both in the first and in the second experiment. The obtained results
will be used in practice to promote the Sales Management Postgraduate Course in September 2013.
Table 13. Assumptions on the design and the content of the creations which are to be tested in
August-September 2013
Variable “The best” creation
from the first
experiment
“The best” creation
from the second
experiment
Price Price per one hour of
classes
No price
Tone of the e-mail Neutral Neutral
Recommendation Yes (the graduate, the FMCG sales manager)
Lecturers’ names No (no concrete names)
Link to the postgraduate
course website
Up
Highlighting selected
expressions
Bold green font
Photo Photo of the building
Source: own work
Creatio
n 9
Creatio
n 10
Creatio
n 11
Creatio
n 12
Creatio
n 13
Creatio
n 14
Creatio
n 15
Creatio
n 16
OR 5.83% 6.51% 6.72% 7.38% 7.39% 5.97% 7.05% 6.64%
CTR 0.45% 0.25% 0.71% 0.53% 0.65% 0.32% 0.32% 0.26%
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
16
Conclusions
Summing up the results of the field experiments, it can be stated that they showed peculiar
conditions of the effectiveness of the e-mailing campaigns promoting a postgraduate course in a
situation in which there was no possibility to reduce the prices and to use material incentives of call
to action type. The following features proved significant: neutral tone of the letter, a
recommendation of the course’s graduate - a person on a manager position, no concrete names of
lecturers (at leach those with academic degrees), larger number of links, and not exposing the total
price of the course.
Possibilities and limitations if the field experiment were also verified. It is worth underlining that
using 3 different experimental designs, including one factorial and two fractional-factorial, enabled
to test a relatively large number of factors in relatively few runs. Unfortunately, “weak” main
effects in the second and third experiment limited the possibilities of analysing the interactions.
Determining how much it “costs”, in an informational sense, to add an additional factor to the
experiment (and simultaneously moving from a factorial design to a fractional design) was not
successful.
For the majority of the examined independent variables (with the exception of the way the price was
presented), there were no problems with their operationalization. The assumption that each variable
can be tested only on two levels was not perceived as a limitation.
In the future, it is worth to consider examining a larger number of factors influencing the graphic
layout (in the present study only the effects of the type of photo were testes), as it was them that
may have erred on the effectiveness of the e-mailing. Unfortunately, it is difficult to operationalize
independent variables in the area of graphic design. What is more, it should be noted that the
decisions in this respect are the result of subjective decisions and tastes (even if the creations are
designed by a professional graphic designer and are consulted with experts).
Possibilities and limitations of the field experiment were also verified. Above all, there was a
feeling of lack of control over all the factors which can influence the results of the experiment (e.g.
the actions of competition). Furthermore, because of the study being conducted by an external
entity, blocking variables could not be applied, and certain factors could not be examined. The use
of more complex schemes was limited not only due to the cost of the study, but also due to the
number of e-mail addresses.
References
2005. BTL pl 2006, c. II, raport - agencje - dyskusja. Marketing w praktyce [Online], 5. Available:
http://www.marketing.org.pl/nowe/mwp-5-2006-65-75.pdf.
BELL, G. H., LEDOLTER, J. & SWERSEY, A. J. 2006. Experimental design on the front lines of
marketing: Testing new ideas to increase direct mail sales. International Journal of Research in
Marketing, 23, 309-319.
COCHRAN, W. G. & COX, G. M. 1992. Experimental Designs, Canada, John Wiley & Sons.
CURHAN, R. C. 1974. The Effects of Merchandising and Temporary Promotional Activities on the
Sales of Fresh Fruits and Vegetables in Supermarkets. Journal of Marketing Research, 11, 286-294.
CZUPRYNA, K. 2004. Skuteczny marketing bezpośredni. Narzędzia, techniki, operacje, czyli jak
zarobić 17 milionów dolarów, Kraków, IFC Press.
DIAMOND, W. D. 2011. Practical Experiments Design for Engineers and Scientists, John Wiley
& Sons.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197
17
FELD, S., FRENZEN, H., KRAFFT, M., PETERS, K. & VERHOEF, P. C. 2013. The effects of
mailing design characteristics on direct mail campaign performance. International Journal of
Research in Marketing, 30, 143-159.
JANISZEWSKA, K., KORSAK, R., KWARCIAK, B., LEWIŃSKI, P., LISOWSKA-
MAGDZIARZ, M., NIERENBERG, B., NOWIŃSKA, E. & ZIMNY, R. 2009. Wiedza o reklamie.
Od pomysłu do efektu, Bielsko-Biała, Wydawnictwo Szkolne PWN.
KALL, J. 2000. Reklama, Warszawa, Polskie Wydawnictwo Ekonomiczne.
KANIEWSKA-SĘBA, A. & KWIATEK, P. Influence of industry characteristics on satisfaction-
loyalty link on the Polish market. In: SPRINGER, R. & CHADRABA, P., eds. 17th Annual
Conference on Marketing and Business Strategies for Central nad Eastern Europe, 2009 Vienna.
Vienna University of Economics and Business Administrations Institute of International Business.
KIRK, R. E. 2013. Experimental design: procedures for the behavioral sciences, Los Angeles, Sage
Publication.
LEDOLTER, J. & SWERSEY, A. J. 2007. Testing 1-2-3: Experimental Design with Applications in
Marketing and Service Operations, Stanford, Stanford University Press.
MAGEE, R. G. 2013. Can a print publication be equally effective online? Testing the effect of
medium type on marketing communications. Marketing Letters, 24, 85-95.
MICHEAUX, A. L. 2011. Managing e-mail Advertising Frequency from the Consumer
Perspective. Journal of Advertising, 40, 45-66.
SALA, P. 2011a. Jak pisać efektywne e-maile (cz. I). Marketing w praktyce, 10, 84-86.
SALA, P. 2011b. Copywriting w e-mail marketingu: treści (cz. II). Marketing w praktyce, 11, 80-
83.
SALA, P. 2012. Mailing wysoce personalizowany. Marketing w praktyce, 1, 72-75.
SHADISH, W. R., COOK, T. D. & CAMBELL, D. T. 2002. Experimental and Quasi-Experimental
Designs for Generalized Caual Inference, Belmont CA, Wadsworth Cengage Learning.
TEZINDE, T., SMITH, B. & MURPHY, J. 2002. Getting permission: Exploring factors affecting
permission marketing. Journal of Interactive Marketing, 16, 28-36.
TROJANOWSKI, M. 2010. Marketing bezpośredni. Koncepcja-zarządzanie-instrumenty,
Warszawa, Państwowe Wydawnictwo Ekonomiczne.
WHITE, T. B., ZAHAY, D. L., THORBJ√∏RNSEN, H. & SHARON, S. 2008. Getting too
personal: Reactance to highly personalized email solicitations. Marketing Letters, 19, 39-50.
WILKINSON, J. B., MASON, J. B. & PAKSOY, C. H. 1982. Assessing the Impact of Short-Term
Supermarket Strategy Variables. Journal of Marketing Research, 19, 72-86.
International Journal of Business and Management Vol. II (1), 2014
http://www.iises.net ISSN 2336-2197