Adobe® Marketing Cloud

Controlled Experiments

Contents

Understanding Controlled Experiments........................................................................4How Does Site Identify Visitors?...................................................................................................................................4

How Do Controlled Experiments Work?....................................................................................................................4

What Should I Test?...........................................................................................................................................................5

What Are the Requirements?.........................................................................................................................................6

How Do I Implement a Controlled Experiment?....................................................................................................6

Enabling Controlled Experimentation............................................................................7Modifying the ExpFile Parameter.................................................................................................................................7

Modifying the ExpCookieURL Paramter (Optional)...............................................................................................7

Modifying the ExpPartialMatch Parameter (Optional).........................................................................................8

Creating a Controlled Experiment..................................................................................9Defining the Objective.....................................................................................................................................................9

Forming a Hypothesis....................................................................................................................................................10

Determining the Number of Visitors Needed.......................................................................................................12

Creating the Test Content............................................................................................................................................12

Configuring and Deploying the Experiment.........................................................................................................12

Validating the Experiment.............................................................................................17

Viewing the Results........................................................................................................19Configuring the Dataset...............................................................................................................................................19

Modifying Log Processing.cfg..............................................................................................................................................................19

Modifying Transformation.cfg.............................................................................................................................................................20

Viewing the Experiment Results................................................................................................................................21

Evaluating the Experiment..........................................................................................................................................22

Experiment Design Spreadsheet...................................................................................25

Controlled ExperimentsLast updated 12/11/2013

Contact and Legal Information.....................................................................................30

Controlled ExperimentsLast updated 12/11/2013

Contents

Understanding Controlled ExperimentsControlled experiments are tests that enable you to compare results obtained from an experimental sample group with thosefrom a standard control group.

Site enables you to implement, measure, and analyze controlled experiments and their results as they relate to different aspectsof your website. Doing so enables you to test hypotheses regarding the improvement of website performance before spendingsignificant time or money fully implementing the proposed changes.

Note:

Site experiments can be analyzed only in datasets where the only method of visitor identification in use is the Sensor setpersistent cookie method. Sensors running on J2EE servers (JBoss, Tomcat, WebLogic, and WebSphere) do not supportcontrolled experimentation. For more information, see the following section.

Using Site, you can implement A/B, A/B/A, and multivariate controlled experiments to gather enough test data to providestatistically accurate data for a detailed evaluation of your hypothesis, without impacting current website performance.

How Does Site Identify Visitors?

A typical configuration of Site uses cookies to uniquely identify visitors to your website and track their behavior over time.

The first time that a particular browser (considered a visitor) makes a request of your website, Sensor works with your webserver to set a persistent cookie (by default cs(cookie)(v1st)), which is interpreted internally within the system asx-trackingid. This cookie is set only once, on the very first request made to your website by that visitor. It then is collectedfrom that visitor each time that browser makes a request (either page or embedded object request) of your website in the future.

Accepting a persistent cookie is at the browser’s discretion. If a user does choose to block persistent cookies, their page viewrequests are still logged, but the measurement data from those requests are not correlated to a particular visitor or their sessionson the website unless you implement an alternate method of visitor identification, such as using the Hash transformation onthe IP and UserAgent fields.

Note: Site experiments can be analyzed only in datasets where the only method of visitor identification in use is the Sensorset persistent cookie method. Sensors running on J2EE servers (JBoss, Tomcat, WebLogic, and WebSphere) do not supportcontrolled experimentation.

During a controlled experiment, users who do not accept cookies could be placed in different experiment groups from one clickto the next. This becomes an issue only if you perform your test analysis with the Broken Session Filter turned off in Insight,which Adobe does not recommended.

For more information about the Broken Session Filter, see the Insight User Guide.

If a visitor clears the cookie during an experiment, the visitor is assigned a new cookie and potentially could be assigned to adifferent group. Because Adobe identifies the visitor as new, the experiment is not invalidated.

How Do Controlled Experiments Work?

In an experiment, you can define any number of test groups in addition to the control group.

4Understanding Controlled Experiments

When an experiment is running, all visitors to your website become part of the experiment, either as part of a test group or ofthe control group, as soon as they access any page involved in the experiment. Visitors are allocated to your experiment groupsrandomly, in proportions defined during the experiment configuration.

Controlled experiments are implemented using the Sensor software that is installed on each of the content servers in your webcluster. As the content servers receive requests, Sensor randomly selects visitors for your test groups and redirects their pagerequests to the experimental content. When Sensor selects a visitor to view the test content, the address bar continues to list theoriginally requested URI but the visitor is routed to the test URI. Because this process takes place internally in the serverapplication, users are not aware of when they are being tested, which is an important consideration for unbiased experimentation.

Sensor passes the test URIs, not the original URI displayed to the user, to the log files for use in analysis.

The results of the experiments can be analyzed easily using Insight to determine whether the experimental hypothesis that youwere testing is correct.

Note:

Adobe strongly recommends that controlled experiments be coordinated and performed with input from those individualsin your organization who are responsible for configuring and maintaining your analysis datasets.

What Should I Test?

Test results must be clear and meaningful so that you can feel confident making large dollar decisions based on those results.

Although you can test various page layouts with Sensor and Site, Adobe suggests that you focus on testing high-value, strategicbusiness initiatives, or new or redesigned website functionality that address the goals that you have set for your website as wellas for your business. You can test for such issues as best price guarantees, personalization functionality, market offers (forexample, packages or bundles), creative design, and application processes.

The following concepts are most important when developing your controlled experiment:

• Understand the right changes to make. This requires some research into how your website functions and the business processesunderlying the front-end website. You want to make changes that provide the most impact and can be tested easily.

• Small changes can have significant impact. Not all of the changes that you test need to be drastic to have a significant impacton your business. Always be open to making small, but very important changes.

Supported Methodologies

Many types of experiments with many different goals can be performed using Site. The following list provides a few examples:

• Altering pages, content, and website processes to improve conversion rates.• Changing marketing campaigns, promotions, cross-sells, and up-sells to increase revenue.• Varying page load times to understand customer quality of service and the actual value of infrastructure performance.

To reach these goals, Site supports the following types of methodologies for controlled experimentation and testing:

• Page Replacement: Replace static URL X with static URL Y. This methodology is of limited use in a dynamic environment.• Dynamic URI Replacement: This is a variant of Page Replacement that replaces static page X with dynamic page Y to render

dynamic content.• Object Replacement: Replace fixed object X with fixed object Y.• Content Replacement: Replace content set X (multiple objects, pages, table, and so on) with content set Y.

5Understanding Controlled Experiments

• Experiment Variable Replacement: Replace JavaScript object /writeCookie_X.js with JavaScript object /writeCookie_Y.js towrite a cookie that can be used by a back-end system to serve particular content.

Note:

Controlled experiments are based on URI replacement, not query string replacement. The URI within a particular URL ishighlighted in the following example:http://www.omniture.com/index.asp?id=1

For example, in your controlled experiment you could specify that the control group URI index.asp be replaced with thetest group URI index2.asp to determine which page design would result in more value.

What Are the Requirements?

To perform a controlled experiment on your website using Site, you must meet the following requirements:

• Sensor must be installed and working properly on each web or application server that supports the website content that youare testing.

Note: Sensors running on J2EE servers (JBoss, Tomcat, WebLogic, and WebSphere) do not support controlledexperimentation.

• You must have a process in place for pushing content to all of your web or application server(s), such as a content managementsystem.

How Do I Implement a Controlled Experiment?

Controlled experimentation with Site and Sensor is designed to be simple, feasible, and actionable.

The rest of this guide describes how to fully implement a controlled experiment:

• Enabling Controlled Experimentation• Creating a Controlled Experiment• Validating the Experiment• Viewing the Results

6Understanding Controlled Experiments

Enabling Controlled ExperimentationTo enable controlled experimentation, someone with administrator access to your web or application servers must modify theExpFile parameter in the Sensor configuration file (usually named using txlogd.conf) on each web or application serverin your web cluster on which a Sensor is installed.

In addition, two other parameters in this file can be modified to implement a testing tool (ExpCookieURL parameter) or toremap large sections of your website (ExpPartialMatch parameter). This chapter provides more information about theseparameters.

To edit the txlogd.conf file

If you have administrator access, complete the following steps. If you do not have administrator access, contact your systemarchitect to request the changes, providing them with the following steps.

1. Navigate to the Sensor installation folder on a web or application server in your web cluster on which a Sensor is installed.2. Open the Sensor configuration file (usually named using txlogd.conf) using a text editor and edit the file as indicated

in Modifying the ExpFile Parameter and optionally in Modifying the ExpCookieURL Paramter (Optional) and Modifying theExpPartialMatch Parameter (Optional).

3. Save and close the file.4. Repeat this procedure for each web or application server in your web cluster on which a Sensor is installed.

Modifying the ExpFile Parameter

The ExpFile parameter points to the location of the experiment configuration file, which defines your experiment.

Setting this parameter enables you to run experiments. For steps to create the experiment configuration file, see Configuringand Deploying the Experiment.

Following is an example of the ExpFile parameter:ExpFile /home/experiment.txt

This tab delimited text file (.txt) can be located anywhere in the Sensor folder and can have any convenient name.

Make sure you record the location of the experiments directory and the name of the configuration file that you specify becauseyou need to save your experiment configuration file (to be described later in this guide) using this name and in this directory.

Note: If you do not set this parameter identically on each machine in your web cluster on which a Sensor is installed,controlled experimentation does not work.

This entry can be preconfigured and remain in the Sensor configuration file on an ongoing basis with no adverse effect. If theexperiment configuration file name specified is not found by Sensor or it is blank (that is, it exists but has no content), Sensordoes not conduct the experiment, logs an error event on the HTTP server, and continues to operate normally in all other respects.

Modifying the ExpCookieURL Paramter (Optional)

The ExpCookieURL parameter can be used to test that your controlled experiment is working properly.

This parameter defines the URL of a virtual page that when requested places you into a specified experiment and group andthen redirects you to the root of your website. From that point through the end of the experiment, you are part of the specifiedexperiment and group.

7Enabling Controlled Experimentation

The default page for this parameter is setcookie.htm, but you can use any valid virtual URL.

Note: This must be a virtual URL and must not be a real page or piece of existing content. There should be no file on theweb server at the specified path with the specified name.

Following is an example of the ExpCookieURL parameter:

ExpCookieURL /setcookie.htm

Modifying the ExpPartialMatch Parameter (Optional)

If you want to enable your controlled experiments to remap your entire website or an entire subdirectory of your website toanother location, you can set the ExpPartialMatch parameter in the txlogd.conf file to “on.” The default is “off.”

Following is an example of the ExpPartialMatch parameter:

ExpPartialMatch off

Be very careful when setting this parameter to “on” because it can result in an inadvertent remapping of your entire website.

8Enabling Controlled Experimentation

Creating a Controlled ExperimentThe overall objective of running a controlled experiment on a website is to determine whether and to what degree a definedchange or set of changes produces an effect on the users of the website.

Defining a controlled experiment involves the following steps:

1. Defining the Objective2. Forming a Hypothesis3. Determining the Number of Visitors Needed4. Creating the Test Content5. Configuring and Deploying the Experiment

Defining the Objective

When defining your objective, consider the purpose of the website or website process you are analyzing: What is its primaryfunction? Who is its target audience?

Common website objectives include converting more visitors into customers or increasing the average amount of revenue gainedfor all visitors within an experiment over the duration of that experiment.

Common website process objectives include improving the steps or pages in a process that cause visitors to abandon the process,removing unnecessary and confusing options in the process that tend to stop visitors from reaching the end of the process, orconsolidating or expanding the process to eliminate or add steps or pages.

Make sure to think carefully about what it is specifically that you want to understand about your website. Carefully planningyour experiment makes the results much more meaningful to your business.

Objective: To increase the number of visitors to our website who request a demo of our product using the “Request a Demo”graphical link, as shown in the following image:

9Creating a Controlled Experiment

Forming a Hypothesis

A hypothesis is an assumption, which can be taken as true for the purpose of argument, that provides a tentative explanationthat can be tested by further investigation.

Try to think about your hypothesis in terms of alternative pages, images, or processes that could be substituted easily for existingones.

Your hypothesis must be able to produce a result with statistical significance. This can be achieved by increasing the percentageof visitors included in the test, or by running the test for a longer period of time.

At this point, you also should define your visitor-based success metrics, either as part of the hypothesis or as an additionalmatrix.

Hypothesis: Moving the “Request a Demo” graphical link closer to the top of the page results in a Visitor Conversion increaseof at least 1.5%.

In our example hypothesis, we have defined the success criterion for this experiment as an increase in Visitor Conversion of atleast 1.5%.

10Creating a Controlled Experiment

11Creating a Controlled Experiment

Determining the Number of Visitors Needed

To create a statistically significant experiment, you must determine how long you need to run the experiment to include enoughvisitors to effectively evaluate the results of the changes to your website.

If you need help determining the minimum length of your experiment, you can use the experiment design spreadsheet providedby Adobe as a tool to help you design your experiment. This file, named VS Controlled Experiment Design.xlsby default, functions not only as a worksheet but also as a record of your decisions about the experiment. For more informationabout this file, see Experiment Design Spreadsheet.

Note: The experiment design spreadsheet can provide useful statistical inferences only when the metric in question is definedas a percentage of visitors that meet some criteria. That is, it is useful only when trying to test a visitor-based metric hypothesis.

Creating the Test Content

Before you configure the experiment, you should create the alternate content that you want to use in the experiment.

The control group is sent to the original URI, while the test group is sent to the new, alternate URI.

To avoid confusion, do not reuse test group file names. For example, if you run an experiment using a test group file namedtest2.asp, do not use test2.asp as the name for the test file in your next experiment.

For the hypothesis that moving the “Request a Demo” graphical link on your home page impacts Visitor Conversion, we createthe alternate home page containing the “Request a Demo” graphical link in the new position. The following section describeshow you then specify that the control group URI index.asp be replaced with the test group URI index2.asp for a certainpercentage of visitors.

Configuring and Deploying the Experiment

After you have defined your objective, hypothesis, and experiment details as well as created your test content, you must configureSensor to deploy the controlled experiment.

You do this by completing the following two steps:

Configuring the Experiment Configuration File

Deploying the Configuration File and Test Content

Configuring the Experiment Configuration File

To configure the experiment, you must complete the experiment configuration spreadsheet provided by Adobe (namedTestExperiment.xls by default). This file configures Sensor to perform the experiment and is the Excel version of thetext file that you specified in Modifying the ExpFile Parameter.

This file can contain information about multiple experiments, which can run at the same or at different times and use differentgroups and percentages, but these experiments are not correlated in any way.

Users are placed into a group for each experiment listed in the file that is configured to be running at this time.

Note: Each experiment is independent of all other experiments. Changes you make to one experiment do not affect anyother experiment, and although visitors may be in multiple experiments, the results do not relate to one another. If you think

12Creating a Controlled Experiment

a correlation exists between the changes in multiple experiments, you must create a new experiment that tests these changestogether.

To configure your experiment

You should complete this file before the experiment begins and not modify the information while the experiment is running.

Note: Any experiment is promptly invalid if the definition of the experiment changes after the experiment has begun.

1. If you have administrator access to your web or application servers, navigate to the Sensor installation folder on any Sensormachine in your web cluster to access the TestExperiment.xls file. If you do not have administrator access, contactyour Adobe account manager to request the TestExperiment.xls file.

2. Open the TestExperiment.xls file (you can rename this file if desired) and complete the following fields:

DescriptionField

A descriptive name for the experiment. Each experiment name must be unique and cannot containspaces.

Experiment

Experiment names are used when displaying the results of experiments in Insight. The names appearas the first half of the element names in the controlled experiment dimension. The second half of theelement name is the group name from the Group field in this file. Each group is named in the followingformat using the experiment name followed by the group name:

ExperimentName.Group Name

For example: New_Homepage.Control

The date and time that you want the experiment to begin. If you do not enter values, the experimentbegins immediately after the file is deployed.

Start

Format: MM/DD/YYYY H:MM

Note:

• If you leave the start and stop times empty, the experiment runs indefinitely.• You can predefine start and stop times well in advance; therefore, you can configure all of your

experiments for the next year at once if desired.• Start and stop times are based on the system time of the web server. If that clock changes for

any reason, your experiment may start or stop unexpectedly.• If you would like to add an experiment as a configuration file entry but do not want the

experiment to run in the near future, you can comment out the experiment information usingthe number sign “#” or define start and stop times in the past.

The date and time that you want the experiment to end. When the stop date and time occurs, Sensorwill stop sending the cookie values identified as a test group to the test URIs and will send all cookiesto the control group URIs.

Stop

Format: MM/DD/YYYY H:MM

13Creating a Controlled Experiment

DescriptionField

Note: See the notes for the Start field.

A descriptive name for each group of visitors in the experiment. Group names cannot contain spaces.Group

Group names are used when displaying the results of experiments in Insight. For more information,see the Experiment field description.

A control group can be implicitly or explicitly defined based on the value entered in the Percentagefield.

Note: To meet the number of visitors needed during the defined time period for the experimentto be statistically valid, you may need to either decrease the confidence level or increase the timeperiod. For example, if your time frame is five days, your confidence level is 98%, and your numberof visitors needed exceeds the number expected for that time period, you need to either increasethe time period or decrease the confidence level until the number of visitors expected exceeds thenumber needed to run a statistically valid experiment.

The percentage of website visitors to include in each defined group. These values can be expressedas either percentages or decimal values. In addition, both values must be either greater than or lessthan one.

Percentage

For example:

33.3% and 66.7%

.99 and .01

If the sum for all groups is less than 100, the undefined excess defaults to a control group.

The URI of the content to be remapped, followed by $. This value is case-sensitive.Original URL

Format: index.asp$

Original URIs can be specified using a dollar sign ($) at the end of the URI to denote that an exactmatch of the file name is required. For example, the expression /product/product_view.asp$matches that exact page only, while /product matches any page in the /product directory andcould be used to remap that entire sub-tree. Original URL entries that do not specify the $ characterat the end of the file name are ignored by the experiment unless the ExpPartialMatch parameter hasbeen set to “on.” For more information about this parameter, see Modifying the ExpPartialMatchParameter (Optional).

Note:

The controlled experiment functionality ignores any query strings appended to the URI stem. Forexample, the page

/product/product_view.asp?productid=53982 is not a valid URI, but the page/product/product_view.asp is a valid URI.

14Creating a Controlled Experiment

DescriptionField

The URI of the alternate content.Remapped URL

Format: index2.asp

Note: See the notes for the Original URL field.

The following is an example of a completed TextExperiment.xls spreadsheet:

Note: Do not modify the column positions in the spreadsheet.

This example indicates that the “New_Homepage” experiment starts on June 1, 2006, ends on June 30, 2006, and containsa control group with 50% of the visitors and a test group with 50% of the visitors, who see different content for one URI.

Note: Although the sample file above has an explicit control group defined, it is not necessary to explicitly define a controlgroup — the experiment automatically creates the control group. If the sum of the percentages for all groups in anexperiment is less than 100%, an implicit control group is assigned to users that do not fall into one of the explicit groups.

3. To insert comments to provide additional information about specific experiments, begin the cell with a number sign (#) andfollow with your comments. Comments can be inserted anywhere in the file.

4. After you have completed the variables in the experiment configuration spreadsheet, save the changes, then save the file intab-delimited text format (*.txt) using the name that you specified in the ExpFile parameter in the Sensor configurationfile. See Modifying the ExpFile Parameter.

The following is an example of an experiment configuration text file:

Note: Because of the tabs required in this file, do not edit the experiment configuration text file by hand. If you need tomake changes to the file, make the changes in the experiment configuration Excel file and re-save the file as a tab-delimitedtext file.

15Creating a Controlled Experiment

If you have defined Start and Stop times, there is no reason to ever delete an experiment from the experiment configuration file.Keeping all of your experiments listed in the experiment configuration file is actually a good way to keep a record of how youdefined each of your experiments.

Deploying the Configuration File and Test Content

You must deploy the experiment configuration file to each machine in your web cluster that is running a Sensor and servingthe pages involved in the experiment. You can do so using either a manual procedure or your existing content managementsystem.

To deploy your test content

• On each application or web server running a Sensor that is serving pages involved in the experiment, use your existing publishingprocess to deploy the test content to the appropriate location.

For example, if you want to publish the test group page index2.asp to the test folder for your website (mysite.com),you would publish the file to www.mysite.com/test.

Note: Do not link to any of your test files directly from a page on your website. Doing so invalidates your test results andyour index scores.

To deploy your experiment

• On each application or web server running a Sensor that is serving pages involved in the experiment, place the experimentconfiguration text file in the directory that you specified in the ExpFile parameter in the Sensor configuration file. See Modifyingthe ExpFile Parameter.

Sensor randomly selects website visitors for each group based on the percentages that you defined in the file and serves the testor control group content to them as appropriate.

16Creating a Controlled Experiment

Validating the ExperimentAfter you have deployed your experiment, you should validate that the experiment is working properly.

As discussed in Modifying the ExpCookieURL Paramter (Optional), the page specified in the ExpCookieURL parameter in theSensor configuration file can be used to place yourself in a specific experiment group.

The default virtual page is /setcookie.htm, but you must use the value that you set in the ExpCookieURL parameter.

Requesting the Test Page

To test a specific experiment group for your website, your browser must be configured to accept cookies and you must notalready have a cookie for this website.

Each time you want to test a new group, make sure to clear your cookies for the website.

To place yourself into a specific group within a specific experiment, request the test page with a query string in the followingform:

http:// <sitename/?Experiment Name=Group Name>

For example:

http://www.omniture.com/setcookie.htm?New_Homepage=index2

When the virtual URL request is sent to the server, Sensor identifies you as a member of the specified group within the specifiedexperiment and then redirects you to the root of the website. You now can navigate to the appropriate location on the websiteto validate whether the correct content displays for that experiment and group.

If you were to type the following into your browser, the browser would display the home page of the website and place you intothe index2 group within the New_Homepage experiment:

http://www.omniture.com/setcookie.htm?New_Homepage=index2

When visitors in the index2 group request the home page, the “Request a Demo” graphical link displays higher on the page, asin the following graphic:

17Validating the Experiment

18Validating the Experiment

Viewing the ResultsYou can view the results of an experiment in a visualization within Insight.

Configuring the Dataset

Before you can view controlled experiment data in Insight, you must modify your dataset configuration files to include a fieldand an extended dimension to contain information about your experiments, which enables you to view the results of an experimentin a table within Insight.

To do so, you must complete the following steps:

• Modifying Log Processing.cfg• Modifying Transformation.cfg

Note: Editing these configuration files causes your dataset to begin reprocessing immediately. It is important to schedulethis for a time that has the least effect on the users of your system. For more information about reprocessing a dataset, seethe Server Products Installation and Administration Guide as well as the Dataset Configuration Guide.

Modifying Log Processing.cfg

You must add the x-experiment field to the Log Processing.cfg file, which is used to create an extended dimension.

See Modifying Transformation.cfg.

To modify Log Processing.cfg

1. In Insight, open the Profile Manager by right-clicking within a workspace and clicking Admin > Profile Manager, or byopening the Profile Management workspace on the Admin tab.

2. In the Profile Manager, click Dataset to show its contents.3. Right-click the check mark next to Log Processing.cfg and click Make Local. A check mark for this file appears in

the User column.4. Right-click the newly created check mark and click Open > in Insight. The Log Processing.cfg window appears.5. Click Fields to show its contents.6. Right-click the last field in the current list and click Add new > Field.7. Type x-experiment in the newly created field, as shown in the following example:

8. Right-click (modified) at the top of the window and click Save.

19Viewing the Results

9. In the Profile Manager, right-click the check mark for Log Processing.cfg in the User column, then click Save to ><profile name> to save the locally made changes to the working profile.

Note: The dataset begins reprocessing immediately.

For more information about Log Processing and Fields see the Dataset Configuration Guide.

Modifying Transformation.cfg

Now that the x-experiment field is available, you must create an extended dimension to include the x-experiment field in yourdataset, which allows you to view your results in Insight.

To do so, you must add a new dimension to the Transformation.cfg file.

If you plan to run multiple experiments, you also must add a new Split transformation to the Transformation.cfg file.This Split transformation separates the different experiment and group names so that the information is easier to interpret. Toavoid reprocessing your data again if you were to need to add additional experiments at a later date, Adobe recommends thatyou add the Split transformation even if you are not currently planning to run multiple experiments.

The following procedure includes the creation of both the new Split transformation and the extended dimension. If you do notwant to add the Split transformation, simply skip steps 5–7.

To modify Transformation.cfg

1. In Insight, open the Profile Manager by right-clicking within a workspace and clicking Admin > Profile Manager, or byopening the Profile Management workspace on the Admin tab.

2. In the Profile Manager, click Dataset to show its contents.3. Right-click the check mark next to Transformation.cfg and click Make Local. A check mark for this file appears in

the User column.4. Right-click the newly created check mark and click Open > in Insight. The Transformation.cfg window appears.5. Click Transformation to show its contents.6. Right-click Transformations and click Add new > Split.7. Complete the new split on comma transformation as shown in the following example:

Note: You can enter any value in the Name field.

20Viewing the Results

8. Right-click Extended Dimensions and click Add new > ManyToMany.9. Complete the new dimension as shown in the following example:

Note:

• You can enter any value in the Name field.• If you did not include the Split transformation, you must type “x-experiment” in the Input field.

10. Right-click (modified) at the top of the window and click Save.11. In the Profile Manager, right-click the check mark for Transformation.cfg in the User column, then click Save to >

<profile name> to save the locally made changes to the working profile.

Note: The dataset begins retransforming immediately.

For more information about Transformation.cfg and extended dimensions, see the Dataset Configuration Guide.

Viewing the Experiment Results

After you have added the new field to Log Processing.cfg and created the new Split transformation and extendeddimension, you can view the new extended dimension that you created as soon as the Fast Input stage of data reprocessing hasfinished.

This dimension, by default, displays the number of sessions for each of your experiment groups.

To view the experiment dimension

• Within any workspace in Insight, open a table with the experiment dimension that you created.

The experiment dimension elements, which represent each experiment you are currently running and each group within eachexperiment, display with the current number of sessions for each group. Each group is named in the following format usingthe experiment name followed by the group name:

Experiment Name.Group Name

For example: New Homepage.Control

The following table shows the Controlled Experiment Groups dimension that was created in Transformation.cfg andeach of the experiments and their groups.

21Viewing the Results

The New Homepage experiment is shown at the bottom of the table with its two groups: Control and index2.

You now can use the experiment dimension and any relevant metrics to explore and interpret the experiment results, as well ascreate useful reports detailing those results.

Evaluating the Experiment

After running the experiment until the required minimum number of visitors have participated in the experiment, you can beassured of sufficient statistical confidence to evaluate the results of your experiment.

Using Insight, compare whichever metrics or key performance indicators were defined as part of the hypothesis to determinewhether the experiment was a success (that is, the hypothesis was validated with the specified confidence.)

In our example experiment, our hypothesis is proven correct if the Visitor Conversion increases by at least 1.5%, which is thesuccess criterion we defined earlier.

The following workspace example shows that the Conversion for the index2 test group was actually 1.8% higher than for thecontrol group, proving our hypothesis.

22Viewing the Results

• Summarizing the Experiment Results• Taking Action Based on the Results• Monitoring Your Actions

Summarizing the Experiment Results

Using Insight, you can create detailed reports to summarize and illustrate the results of your experiment.

You then can use your reports, as shown in the following example, to make recommendations based on the results, which arebacked up by the visual information you have provided in your reports:

23Viewing the Results

Taking Action Based on the Results

After the results are clear, you are ready to act on those results by making production-level changes to the tested pages, applyingthese same changes to other areas of your website, and making sure to completely document the test, its results, and the changesthat you have made.

Monitoring Your Actions

After the controlled experiment is complete and you have implemented the appropriate changes, make sure to continue tomonitor the changes that you made by, for example, viewing validation metrics, creating control charts, and providing dashboardmetrics.

Always be prepared to re-test your hypothesis if you think the changes that you tested and made are not bearing out the originalresults.

24Viewing the Results

Experiment Design SpreadsheetThis file functions not only as a worksheet but also as a record of your decisions about the experiment.

If you need help designing your experiment, you can use the experiment design spreadsheet (named VS Controlled ExperimentDesign.xls by default) provided by Adobe.

The experiment design spreadsheet can provide useful statistical inferences only when the metric in question is defined as apercentage of visitors that meet some criteria. That is, it is useful only when testing a visitor-based metric hypothesis.

To design your experiment using the experiment design file

1. If you have administrator access to your web or application servers, navigate to the Sensor installation folder on any Sensormachine in your web cluster. If you do not have administrator access, contact your Adobe account manager to request thefile.

2. Open the VS Controlled Experiment Design.xls file. (You can rename this file if desired.)

The spreadsheet on the following page is an example of how you would complete the spreadsheet when preparing to test theexample hypothesis used throughout this guide.

25Experiment Design Spreadsheet

3. Enter text or values for all of the fields in blue in this file, which are described in the following table. The calculated fieldsare defined in the second table.

Fields to Define

SpecifyIn this field…

A descriptive name for your experiment.Experiment Title

A textual description of the experiment.Experiment Description

The name of the metric on which the experiment is based.Metric Being Studied

Example: Visitor Conversion

The definition of the metric on which the experiment is based.Metric Definition

Format: Visitors[X]/Visitors

Example: Visitors[URI='conversionpage.asp']/Visitors

The date and time you want the experiment to begin.Intended Start Time

The date and time you want the experiment to end.Intended End Time

(Optional) The dimension name and element set or range by which you want to furthersegment the dataset.

Applicable Selections

The URIs involved in your hypothesis. You define the current URIs for the control groupand the alternate URIs that you have created or will create for the test group(s).

Experiment URIs

Heading for the metric values that you expect for your website.Expected Metrics forApplication Selections

The average number of visitors to your website per day.Average Visitors per Day

The average visitor conversion rate for your website.Visitor Conversion

Heading for how the metric values should be compared.Experiment Will Determineif the metric name for theTest Groups is …

26Experiment Design Spreadsheet

SpecifyIn this field…

Set this field to True if you want the ability to conclude that the test group’s metric increasedduring the experiment. Set this field to False to reduce the number of visitors needed to drawconclusions. Adobe recommends that you set it to True.

Greater Than The Value Forthe Control Group?

Set this field to True if you want the ability to conclude that the test group’s metric decreasedduring the experiment. Adobe recommends that you set it to True.

Less Than The Value For theControl Group?

The percentage by which you want the metric for the test group to be higher or lower thanthat for the control group.

By at Least (Detection Level)

The desired confidence level for the test group values. The confidence level determines thenumber of false positives to measure the probability that the stated expectation is true.

With a Confidence Level ofat Least

The desired power level for the test group values. The power level determines the numberof false negatives.

and a Power Level of

Heading for the percent of visitors values.% of Visitors

Percent of visitors you want to include in the test group. You can play with this numberuntil the value in the Total (Usually 100%) field in the Visitors section is equal to or greater

Test Group

than the value in the Minimum Visitors Required (Test+Control Groups) field, both ofwhich are described in the following table.

Percent of visitors you want to include in the control group.Control Group

Any notes that you want to save for future reference.Other Design Notes

The remaining fields are calculated based on the values that you entered and are described in the following table.

Calculated Fields

DescriptionField

Heading for the metric values that you expect for your website.Expected Metrics forApplication Selections

This field is normally automatically calculated by the spreadsheet. It relies on the assumptionthat on most days the website receives many more new visitors than return visitors. If this is

Expected Visitors perPeriod

not the case, this cell’s calculation should be overwritten with the actual number of visitorsexpected during the experiment.

The Z score for a false positive result. This is an intermediate statistical calculation.Calculated Z Score for TypeI Error

The Z score for a false negative result. This is an intermediate statistical calculation.Calculated Z Score for TypeII Error

Minimum number of visitors needed in your experiment to meet your specified confidencelevel, power level, and Z score, expressed as a percentage of the value in the Expected Visitorsper Period field.

Minimum VisitorsRequired (Test+ControlGroups)

27Experiment Design Spreadsheet

DescriptionField

Minimum number of visitors needed in your experiment to meet your specified confidencelevel, power level, and Z score. This value must be less than or equal to the value in the Total(Usually 100%) field in the Visitors section.

Minimum VisitorsRequired (Test+ControlGroups)

Minimum number of days you need to run your experiment to meet your specified confidencelevel, power level, and Z score. This calculated number is subject to the same issues as discussed

Minimum ExperimentTime (Days)

in the Expected Visitors per Period field. In the case of a website with many returning visitors,the Minimum Experiment Time (Days) field is the expected number of days it takes to see anumber of unique visitors equal to the value in the Minimum Visitors Required field.

Heading for the visitors values.Visitors

Number of visitors needed in the test group.Test Group

Number of visitors needed in the control group.Control Group

Total number of visitors needed for the experiment. This value must be equal to or greaterthan the value in the Minimum Visitors Required (Test+Control Groups) field.

Total (Usually 100%)

Percentage indicating that there is a chance equal to the specified confidence level that themeasured value of the metric calculated for the test group will be within this percentage of itstrue value.

Test Group Accuracy (atTarget Confidence Level)

Percentage indicating that there is a chance equal to the specified confidence level that themeasured value of the metric calculated for the control group will be within this percentageof its true value.

Control Group Accuracy (atTarget Confidence Level)

Number of standard deviations a given value is from the test mean.Z Score (at TargetAccuracy)

The confidence level achieved for the experiment. The confidence level measures the probabilityof the stated expectation to be true.

Actual Confidence Level (atTarget Interval)

The confidence interval achieved for the experiment, which provides an estimated range ofvalues that is likely to include an unknown population parameter. This range is calculatedfrom a given set of sample data.

Actual Interval (at TargetConfidence Level)

You need to look at the value in the Minimum Visitors Required (Test+Control Groups) field . . .

and compare it to the value in the Total field in the Visitors column.

28Experiment Design Spreadsheet

For your experiment to be statistically valid, the value in the Total (Usually 100%) field must be equal to or greater than thevalue in the Minimum Visitors Required (Test+Control Groups) field.

Given the inputs provided, what the example worksheet shows you is that 10,475 visitors need to participate in this experimentto achieve the entered 95% confidence rate (which is the minimum suggested confidence for any controlled experiment,although you can increase this number). The experiment as currently designed includes 30,000 visitors, which is well overthe minimum number of visitors required.

If you keep the number of days the same, you could increase the confidence level as long as the total number of visitorscontinues the meet or exceed the required minimum.

4. Save the file for your records and then use the information from the file to configure the experiment using the experimentconfiguration spreadsheet. For more information about this spreadsheet, see Configuring and Deploying the Experiment.

29Experiment Design Spreadsheet

Contact and Legal InformationInformation to help you contact Adobe and to understand the legal issues concerning your use of this product and documentation.

Help & Technical Support

The Adobe Marketing Cloud Customer Care team is here to assist you and provides a number of mechanisms by which theycan be engaged:

• Check the Marketing Cloud help pages for advice, tips, and FAQs• Ask us a quick question on Twitter @AdobeMktgCare• Log an incident in our customer portal• Contact the Customer Care team directly• Check availability and status of Marketing Cloud Solutions

Service, Capability & Billing

Dependent on your solution configuration, some options described in this documentation might not be available to you. Aseach account is unique, please refer to your contract for pricing, due dates, terms, and conditions. If you would like to add toor otherwise change your service level, or if you have questions regarding your current service, please contact your AccountManager.

Feedback

We welcome any suggestions or feedback regarding this solution. Enhancement ideas and suggestions for the Analytics suitecan be added to our Customer Idea Exchange.

Legal

© 2013, AdobeAll rights reserved.Published by Adobe Systems Inc.

Terms of Use | Privacy Center

A trademark symbol (®, ™, etc.) denotes an Adobe trademark.

All third-party trademarks are the property of their respective owners. Updated Information/Additional Third Party CodeInformation available at http://www.adobe.com/go/thirdparty.

06052013

30Contact and Legal Information