Experiments

Experiments provide the option to show different versions of a campaign to your users so that you can collect data and make changes to optimize the user experience. Through experiments, you are able to test out different variants of a campaign on a real audience and collect data for each variant. Based on the collected data, you can make changes to your campaign to improve conversion rates and increase revenue.

Experiment Overview

When you first login to the Tonos admin page, you will be redirected to the Dashboards page. From here you have to go the Experiments page, which is located in the Optimization section.

When you enter the Experiments page, the entirety of the experiments you have added up until now are displayed like in the picture below:

To find an experiment in the list, you may use the search bar and type its name.

You can choose to view a certain experiment’s details, results, edit, delete or unschedule it.

Experiment Status

For each experiment you can see its status, the options for which are as follows:

• Draft - added but never ran (gray dot)
• Active - currently running (blue dot)
• Scheduled - has been scheduled to be run on a certain date
• Completed - has been run (green dot)
• Canceled - has been run but has been stopped manually (red dot)

Based on the experiment status, an experiment can be run (if status is Draft, Scheduled), ended (if status is Active), and its results can be downloaded (if status is Completed, Canceled).

Experiment permissions for roles

Different roles have different permissions on managing the experiments. The permissions for each role are as follows:

• System roles
  • Marketing Specialist - can Create, View, Edit, Delete experiments.
  • Support Specialist - can View and Delete experiments.
  • Viewer/Analyst - can view experiments.
  • Administrator - can Create, View, Edit, Delete experiments.
• Other roles (custom roles created for each application) - by default, do not have any access to experiments.

Experiments and Google Tag Manager

When an experiment begins running, data is collected on the experiment to be displayed as part of the experiment results. These results are used to make changes on the campaign to optimize the user experience and therefore increase conversion rates and revenue.

The results of the experiment are saved as events and can be shared with Google Tag Manager so that business intelligence analysts can analyze the data through GTM. The events saved for each experiment are:

• For the experiment: Experiment Id, Campaign Id, Offer Id, Variant Ids, Variants’ weights, Experiment start & end datetime.
• or the experiment audience: Audience Id, Size, Target Audience Percentage.

Adding an experiment

In order to add an experiment, click the Add Experiment button on the top-left part of the page.

After clicking the button, a form will pop up that will request you to fill out the necessary information needed to add an experiment.

After you have filled each of the boxes, click the blue Add button and with that an experiment will be added.

Filling out the experiment form

Status	Description
Name	This field requires you to write the name of the experiment.
Hypothesis	This field is an option for you to provide a hypothesis on what assumption you would like the experiment to validate.
Select Campaign	This field requires you to select a campaign you would like to experiment with. You may select campaigns with one offer and a Bisko audience.
Type	This field has been preselected as Offer Template as all experiments are based on changes made to the offer template variants.

Editing and deleting an experiment

An experiment can be edited if its Status is Draft or Scheduled. An experiment which is currently running (Active) or has been run (Completed, Canceled), cannot be edited. An experiment can be deleted by clicking the Delete option provided for each experiment on the list. If an experiment is currently running (active), it cannot be deleted.

Getting an experiment ready to run

To get an experiment ready to run, it is important to note that the experiment has three tabs to move between and make changes to - the Variants, Results and Details Tab. In addition, an experiment needs to have an audience who will be shown the variants within the experiment.

Experiment Details

On the Experiment Details, you can find the following information:

• Hypothesis - the hypothesis made when the experiment was added.
• Type - type of experiment. Preselected as Offer Template.
• Campaign Name - name of campaign being experimented with.
• Campaign Description - description of campaign being experimented with.
• Offer Template Name - name of offer template being experimented with.
• Offer Template Description - description of offer template being experimented with.

The Name of the experiment and the Hypothesis made can be changed here.

Experiment Results

The experiment results are displayed after an experiment has started running and data on the experiment has been collected.

• If an experiment has not been run, there are no results to show.
• The results of the experiment are updated every 12 hours.

The Results Tab contains the Schedule (experiment start and end date, time), Audience Size, Target audience, KPIs for all variants and a table which shows the weight and KPIs for each variant.

The KPIs for all variants are: Total Exposures, Total Unique Exposures. The KPIs for each variant are: Exposures, Unique Exposures, Clicks, CTR, Closes, Closure Rate, Conversions, Conversion Rate, Revenue.

Experiment KPIs (Key Performance Indicators)

Below, please find an explanation of each KPI measured through the experiments.

• Total Exposures - the total number of views for a campaign
• Total Unique Exposures - the total number of unique views for a campaign
• Exposures for a single variant - the total number of views for a variant
• Unique Exposures for a single variant - the total number of unique views for a variant
• Clicks - the total number of clicks that are not close actions for a variant
• CTR - click through rate for a variant
• Closes - the total number of close clicks for a variant
• Closure Rate - the closure rate for exposures which resulted in a close click for a variant
• Conversions - the total number of exposures which converted users into subscribers for a variant
• Conversion Rate - the conversion rate for exposures which converted users into subscribers for a variant
• Revenue - the total income generated through subscriptions for a single variant
  • The revenue currency is the same as the organization's currency

Downloading the experiment results

If an experiment has been completed successfully or it has been canceled, the results collected on the experiment can be downloaded.

If an experiment status is Completed, Canceled, on the top right corner of the page, a Download CSV button is displayed which on click downloads the experiment results as a CSV file.

Experiment Variants

The experiment variants tab is where the offer template is edited to be displayed as different variants to the campaign audience.

• Experiment variants can be added, edited and deleted while an experiment has not been run (its status is draft or scheduled).
• An experiment can have a maximum of 5 variants (the original variant and four variants created as part of the experiment).
  • An experiment needs a minimum of two variants to run (the original variant and a variant created as part of the experiment).
• Variants are named as variant A, B, C, D.
• The offer template variants can be either modal or inline based on the original variant (e.g. if the original variant is inline, the variants added are also inline).
• When an experiment is created, a variant is added by default as the original variant. The original variant is the offer template of the offer within the campaign we are experimenting with. The original variant cannot be deleted.
• If the experiment is active, completed, canceled, the variants of the offer template can only be viewed.

Adding and editing experiment variants

To add an experiment variant, click the “Add Variant” Button located on the Variants Tab.

• The newly added variant has the same prefilled details as the original until edited.
• Each variant is a dropdown which on click shows the variant details.
  • For each variant, including the original, an input field is displayed to add the weight of the variant (the percentage of audience that will see that experiment variant, Ex: audience size is 1000, variant A’s weight is 50%, 500 users will see variant A).
  • The total weight must add up to 100%.
  • Variant weight must be a round number bigger than 0. (that is, 1,2,3,...)
• For each variant, including the original, the administrator may add a description.

Viewing and editing the offer template

For each variant, as part of the variant details, the offer template may be viewed or edited.

• The original variant’s offer template can only be viewed and on click of View Template button the offer template is displayed (it can only be viewed, not edited).
• For other variants, the offer template is edited by clicking the Edit Template button.
  • On click of Edit Template, the offer template variant is opened and ready to be edited.
    • Changes are saved by clicking Save Template and discarded by clicking Cancel.

Experiment Audience Configurations

Each experiment must have an audience which has been selected as the one the experiment variants will be shown to. An audience can be configured by providing the information required in the audience configurations section as below:

• Audience - dropdown which displays audiences that the admin can select from.
  • The selected audience is applied to the campaign and shown on Tonos Admin / Campaigns / Campaign Details if experiment is run (Status = Active).
  • A campaign must have an audience to be able to run an Experiment.
  • If a campaign already has an audience selected, a different audience cannot be selected here.
  • When an audience is selected, the audience size displays the calculated audience size for the campaign.
    • Audience size must be 1000 and over to run an experiment.
• Target audience - input field where admin can target a specific percentage of the audience. Default option and maximum is 100 %.
  • Target audience must be a round number bigger than 0. (So, 1,2,3,...)
  • The percentage of the audience that can be targeted should not allow for audience size to fall under 1000. (Ex: audience size is 2000, lowest possible target audience percentage is 50%)

Running an experiment

To collect results for an experiment, an experiment needs to be prepared and run. Some of the restrictions when an experiment is running are:

• To run an experiment, the experiment needs to have at least two variants (the original variant and another variant). An experiment can have a maximum of 5 variants.
• Experiments can be run for a maximum of 10 weeks.
• The limit for active experiments is 25 per application.
• A campaign can only have one active experiment at a time.
• If an experiment is running, the campaign and offer cannot be edited, disabled or deleted on Tonos Admin.
• While an experiment is running it cannot be edited.
• While an experiment is running, the campaign is enabled and cannot be disabled.
• If an experiment has been scheduled for a campaign - that campaign is no longer available to add as part of another experiment.

An experiment can be run immediately or scheduled to be run in the future. To run an experiment, make sure that the experiment status is Draft and click the Run experiment button. A form will popup which needs to be filled in to run the experiment. After you have filled in the form, click Run and the experiment will be run.

Filling out the run experiment form

The run experiment form requests that the experiment is either run immediately or scheduled.

• Run now - an experiment end date is selected and the experiment is run immediately. Experiment status is changed to active.
• Schedule - an experiment start and end date is selected and the experiment is scheduled to run on the selected dates. The experiment status is changed to scheduled. The experiment may still be edited until it starts running and its status is changed to active.

Unscheduling an experiment

If an experiment has been scheduled to be run on specific dates, it can still be unscheduled through the Unschedule option provided for each experiment as part of the more options. From the Experiments page, in the *Optimization** section, on click of more options for scheduled experiments, the Unschedule option can be found. On click of Unschedule, a popup is shown to confirm that the experiment should be unscheduled. If confirmed, the experiment is unscheduled and it will no longer be run on the specified dates (its status is changed to draft).

Ending an experiment

An experiment can be stopped before it has completed its specified running period through the End Experiment option.

If an experiment is running on the top right corner of the page, an End Experiment button is displayed which on click opens a popup to confirm that the experiment should stop running.

If confirmed, the experiment stops running, its status changes to Canceled and data is no longer collected about the experiment. When an experiment is canceled, details are saved and displayed on who ended the experiment (Ex: Stopped by email@email.com) and the date, time the experiment was ended.

Applying the results of an experiment

When an experiment has been completed or canceled after running for a period of time, it has collected data which is shown as part of the experiment results. The results of the experiment can be applied to the campaign to improve the campaign according to a specific variant.

Selecting and applying a variant

On the results tab, a button is displayed when the experiment status is canceled or completed. The button is Apply a variant and on click a popup is displayed which shows the name and weight of each variant within the experiment. A radio button is provided to select the variant which will be applied to the campaign. After selecting the variant that will be applied, click Apply.

On click of Apply you are redirected to the Campaigns List where the campaign has been modified according to the selected variant and is ready to be used.

Campaign modification after a variant is applied

To modify the campaign, a new offer is created based on the selected variant. The offer can be found on the list of offers on Tonos Admin and has a label “A/B Testing” specifying that the offer was created as part of the experiments. This offer is applied to the campaign which was experimented with to modify the campaign based on the selected variant.

On the Campaigns List, for the modified campaign, you can find the “A/B Testing” label specifying that the campaign has been experimented with. For the modified campaign, you can find the versions of the campaign before and after the experiment. The modified version is shown as the first option and can be edited, enabled, disabled or deleted. To see the previous version of the campaign, click the dropdown icon and the previous version of the campaign will be shown. This version can be viewed and deleted only.

Applying the original variant

If the results of an experiment imply that the original variant is the most optimal for the campaign and therefore the original variant is selected to be applied to the campaign, the following actions are undertaken:

• The campaign has one version only as no changes were made.
• The “A/B Testing” label is displayed under the campaign's name specifying that the campaign was experimented with.
• A new offer is not created and the “A/B Testing” label is displayed under the name of the offer that was part of the campaign within the experiment.

As an admin I want the changes I make to campaign and offer on Tonos admin applied to experiment so that the experiment is relevant to the campaign and offer.

Handling specific cases for experiments

An experiment can have specific cases which need to be handled so that they may run and collect results as expected. Therefore, these cases and how they are handled are specified below (the campaign mentioned below refers to a campaign which is part of an experiment):

When an experiment is created with a certain campaign:

• The campaign cannot be added as part of another experiment.
  • A campaign can be part of two experiments if the initial experiment is deleted, completed or canceled.
• The campaign and offer cannot be edited or deleted on Tonos Admin unless the experiment is deleted, canceled or completed.

• If the campaign is deleted, the experiment is deleted.
• If the campaign name or description is changed, the changes are displayed on experiment details.
• If the offer name or description is changed, the changes are displayed on experiment details.
• If the offer within the campaign is deleted, the experiment is deleted.
• If the offer within the campaign is removed from the campaign, the experiment is deleted.
• If the offer is edited, the experiment is deleted.
• Editing the subscription plans, license request forms, checkout flow is not allowed.