Multivariate Testing

Sitecore gives you the ability to dynamically test the content of your website to find out which components or combination of components are the most effective. Multivariate tests can be used to see which variations of content work best with site visitors. You can track the results by pairing M/V tests with goals or campaigns to quantify a site visitor’s value resulting from a multivariate test.

The Page Editor is the most useful tool in setting up M/V testing. When you use the Page Editor to create a multivariate test, you have the following options:

  • Test existing content — prepare several test variations as content items and make them available in the content tree.
  • Clone a content item — duplicate items to test variations in content fields
  • Create new content — create new content to use in your test.
  • Change a component in the test —allows you to choose a different rendering or sublayout for the test.
  • Hide a component in the test — allows for a variation of the test to hide the component entirely
  • Create a combination test — select multiple pieces of content on the same page to run a combination test.

Creating an MV Test in the Page Editor

To create a multivariate test in the Page Editor:

1. Open the Page Editor and select a component on your website to test.

2. Click the Test the component button on the floating toolbar to open the testing dialog.

3. In the Test the Component dialog box, click New Variation to create a new test variation variable.

4. Content options for this variation include:

  • Browse for content in the content tree by clicking the ... button in the Test Content Item box
  • Change the component itself by selecting the Enable variation of component design checkbox
  • Hide the component by selecting the Hide Component checkbox

5. Click OK in the Test the Component dialog box once you have added all your test variable variations.

Running an MV Test in the Page Editor

To run a multivariate test in the Page Editor:

1. Select the Testing tab in the ribbon and click Start Test.

Note: After you click start test, you will not be able to make any modifications to the test. If components in your test contain personalization, this will be paused while the test is running.

2. Enter a name for the test and click Start the Test

3. To preview the variations, use the arrows on the component itself:

Or use the gallery on the ribbon to select variations and get a page overview of your test:

Note: You can add more than one component to a multivariate test so that you can decide which combination of controls works best on a page.

4. Click Stop Test in the ribbon to stop the test and view your results. Using DMS, Goals enable site visitors to accumulate value. You can decide which variation of test variables is the most successful by the value it has accumulated during the test. The variation with the highest engagement value is typically the winner.

You can use the marketing center to manage all of your tests as well as configure settings such as Test Strategy:

Test Strategy

Further Development

Test Strategy

You can customize tests by creating a new Test Strategy. For example, you could favor a test value by a certain percentage:

1. Create a class similar to the following:

  1. using Sitecore.Analytics.Testing;
  2. public class PercentageMultiVariateTestStrategy : MultiVariateTestStrategyBase, IMultiVariateTestStrategy
  3. {
  4. public TestCombination GetTestCombination(TestSet testset)
  5. {
  6. Random rnd = new Random();
  7. byte[] combination = new byte[testset.Variables.Count];
  8. for (int i = 0; i < combination.Length; i++)
  9. {
  10. combination[i] = (byte)rnd.Next(testset.Variables[i].Values.Count);
  11. // implement logic to select a test value
  12. }
  13. return new TestCombination(combination, testset);
  14. }
  15. }

2. Create a new test strategy item underneath /sitecore/system/Settings/Analytics/Multivariate Test Strategies

3. The type field of the created test strategy item should be:

YOUR_NAMESPACE.PercentageMultiVariateTestStrategy,YOUR_ASSEMBLY_NAME

Get test data from API

You can also retreive multivariate test data directly from the DMS API:

The following is an example of how to get statistics from a single MV Test Set. The item path should be the path to your MV test in the content tree:

  1. protected void Page_Load(object sender, EventArgs e)
  2. {
  3. var testDef = new TestDefinitionItem(Tracker.DefinitionDatabase.GetItem("/sitecore/system/Marketing Center/Test Lab/doMultiVariate"));
  4. var stats = TestManager.LoadTestStatistics(testDef);
  5. Response.Write("Runs: " + stats.TestRuns + "<br/>");
  6. foreach (var variable in stats.Testset.Variables)
  7. {
  8. Response.Write("Label: " + variable.Label + "<br/>");
  9. var varStats = stats.GetVariableStatistics(variable.Id);
  10. foreach (var value in varStats.Statistics)
  11. {
  12. Response.Write("value: " + value.TestValue.Label + ", ValuePerVisit: " + value.ValuePerVisit + "<br/>");
  13. }
  14. }
  15. }