Skip to content

A/B Testing

ChilliConnect's A/B Testing allows you to run experiments on your game to find out what particular features or configurations of your game are most effective. AB tests can be created to target specific segments of players based on country, platform, device type and device model. For each test, one or more test groups can be created, with players evenly distributed across groups in a round-robin fashion. Tests can also be restricted to a maximum number of players and can be configured to target new players only.

Players are assigned to test groups on login and initial creation. A player is considered to be part of a test group until they log in again, at which point they will be unassigned from the test if it has ended.

Creating

To create a new Test in ChilliConnect, select top level the Live Ops menu option followed by the test option. Use the "Add Test" button in the top right to bring up the New Test form.

AddTest

Details

The details tab captures basic configuration information for the test, including the test name, description and start and end dates.

AddTest

Name A descriptive name for the test
Key A unique Key that will be used to refer to the test in API calls
Description Optional additional text that can be used to capture more information about this test
Start Date The Date the test is due to start
End Date The optional End Date that the test should complete. If not provided, End Date can be changed once the test has started.
Total Installs The total number of installs of your game
Target KPI The KPI that should be tracked for this test. This can be changed after the test has been created.

Segments

The segments tab allows you to filter the test to specific groups of players. For more details see our Segmentation guide.

Test Groups

The Test Groups tab allows you to define how many players should be included in your Test, whether or not players should be restricted to new players only, how many players should be included and if the Test should retain it's players regardless of the Tests completion status.

AddTest

Player Limit An optional maximum number of players that should be included in this Test. Setting this limit allows you make the most of your player base and run multiple tests in parallel. If no limit is set, players will be allocated to the test until the test ends.
New Players Only Should the Test be restricted to new players only. When true, players will only be allocated to Test Groups on CreatePlayer calls and not on log in calls. This option is useful when you are testing fundamental changes to the game economy and apply these changes to in progress games would affect Test results.
Sticky When a Test has completed, players are usually unassigned from the Test on their next login call. With the Sticky flag enabled, players will not be unassigned from a Test once it completes - this will also prevent such players from ever being assigned to a new Test. This can be useful in situations where the player has been part of a Test that changes some aspect of your game that you would not want to revert once the Test has completed - for example, changes to IAP prices or game balancing. Note that players will still be removed from a Test if that Test is later deleted.
Test Groups The Test Groups that players should be allocated to. A Test must contain at least one test group. Players are allocated evenly in a round robin fashion. For each Test Group, a descriptive Name and a unique Key can be provided. The Key will be returned to the game when making the GetActiveCampaign call.

Overrides

Overrides allow you to easily define changes to the Catalog for specific test groups. Items can be added, removed and updated. Overrides are automatically applied for users in the test group they are part of when retrieving the item definitions or downloading the Catalog package file. This means that no additional logic is required in your game to support overrides. For a detailed example, see our A/B Testing tutorial

OverrideList

Prioritise

ChilliConnect will compute any possible conflicts with the Overrides and existing Campaign Overrides. If conflicts are found you will be presented with options on prioritising the Campaigns.

Override listing

More details on Campaign prioritisation.

Custom Data

Custom Data allows you to define arbitrary, free-form JSON to be attached to the test definition. This JSON can be retrieved at run time from your game via the GetActiveCampaign call, providing a convenient place to add any small, extra data to the test.

AddTest

Viewing Tests

All tests, active, upcoming and completed are displayed in the test views. Filters in the top right of the screen can be used to show tests in a specific state. Active tests are always shown in blue.

ViewTests

Details of a specific test can be viewed by selecting the test from the list.

ViewTests

The overrides tab will show you what overrides are defined for the current test.

ViewTests

The KPIs tab will show you all the KPIs collected for the test. The displayed KPIs can be changed to only show the ones that are required to be tracked for this test.

ViewTests

The Players tab can be used to view specific players that have been assigned to any given test group:

ViewTests

The edit button in the top right of the test view can be used to modify test details. Once a test is running, only some fields will be able to be modified.

Ending Tests

Rather than manually ending the test by setting an End Date in the past, Active tests can be ended using the End Test button in the top right of the test view.

ViewTests

Changing Player Assignment

Often, it is desirable to be able to change the test group a specific player is assigned to for testing purposes. This can be done from the player view, using the Edit button displayed next to the currently assigned test group:

ViewTests

Metrics Capture

ChilliConnect will automatically capture and track KPIs for each individual test group you define, making it easy to track how the changes between each test group affect KPIs. To view metrics for test groups, you can use the Segments menu on any of the analytics views:

Metrics

The two most recent and active A/B Tests are shown in the drop-down with the "More" option allowing you to select recently completed and older A/B Tests.