Guide to A/B testing with LiveSession

Qualitative data in A/B testing

Qualitative data is not an alternative to quantitative findings. It’s actually quite the contrary! These two methods are supposed to complement each other and help you gain a deeper understanding of user behavior.

While numbers will show you what is happening, qualitative insights will help you find the reasons why it’s happening.

For instance, you might have a landing page with quite a lot of traffic but unsatisfying results. While many people visit it, the conversion rates and bounce rates may not meet your expectations. Now, what should you do about it?

Instead of trying to take a shot in the dark, you can turn to qualitative data and see how the users interact with your landing page. Perhaps a button is not working, or they just can’t find what they’re looking for?

There is a method that is sure to help you out:

Get more context with session replays

You can easily access this kind of information through session replays. This qualitative user research method has a range of od advantages:

• It’s affordable – Conducting real-time interviews with your users comes with costs that can be hard to cover for some. Session replays, on the other hand, are much more budget-friendly. All you need to do is add a piece of code on your website and you’re ready to start.
• It happens in the background – Session replays don’t need any involvement from your side. They’re recording in the background and the material is ready for your analysis.
• It fights the social desirability bias – When faced with a researcher, the users tend to respond in a way that seems more desirable. In the case of session recordings, the users are not aware they’re being watched. Because of this, their behavior is more natural and not affected by this phenomenon, known as the social desirability bias.

Use qualitative research to design A/B tests

Qualitative data can suggest you potential solutions to the problem. When you watch how the visitors use your website, you start to imagine what can be done to solve the issue. Below we’re describing our own case study so you can see exactly how it works!

How LiveSession can help with A/B testing

LiveSession is a session recording and qualitative analytics platform. It can also be connected with a range of other tools, including Google Optimize, one of the most popular solutions for A/B testing.

When you connect your Google Optimize A/B test to LiveSession, you will be able to see sessions that illustrate how users interacted with different versions of the test. In other words, it’s a fast and easy way to enrich quantitative data with qualitative data.

Below you’ll find a tutorial on how to connect LiveSession and Google Optimize:

How to connect LiveSession with Google Optimize

We’ll start with an example so you can easily see how it works.

Let’s say that you want to measure how the button color affects conversion and bounce rate.

In our example, we’re going to test two different colors: green and red. Originally, the “Purchase Now” button on our site was green:

The best way to find out is to conduct an A/B test with different color variants. Here’s where Google Optimize comes in:

It’s really simple: just go to https://optimize.google.com/ and create your first experiment:

Define the name (1), the URL of the website you’ll be testing (2), and click Create (3):

Next, go to the Targeting and variants section. Click Add variant (1), define the name of the variant (2), and click Done (3):

Click Edit to set a variant directly on the website:

TIP: The easiest way to create a variant is with the official Google Optimize extension.
We’re going to use it in this tutorial.

Select the button (1). Find the background-color property (2), change it to red (3) and click Save (4).

In the next section, connect (1) your Google Analytics account and choose (2) a property for the Google Optimize Container. Remember that it has to be the same property that you’re using on your website.

Create an objective to measure:

It this example, we’re going to measure Bounces:

Go back to the top and click Start:

Click View instructions to get information about your Container ID:

Copy the selected ID:

Next, go to Google Tag Manager (https://tagmanager.google.com/), chose a container for the same website, and open the Tags section:

Click New to add a new tag:

Find Google Optimize on the list:

Paste your container ID and connect your Google Analytics settings (or create new ones) with the same property you’ve connected with Google Optimizer. Click Save.

Go back to Google Optimize and copy the Experiment ID from the Measurement and objective section:

In Google Tag Manager, add another Tag with a Custom HTML.

Paste this code:

<script>
  function gtag() {dataLayer.push(arguments)}
  gtag('event', 'optimize.callback', {
    name: '<EXPERIMENT_ID>',
    callback: function(variant) {
      __ls("setCustomParams", {params: {ab_variant: variant, ab_id: "<EXPERIMENT_ID>"}});
    }
 });
</script>

Paste the copied experiment ID in the <EXPERIMENT_ID> field. This is a connector between LiveSession and Google Optimize.

Click Save, and then Submit...

...and Publish the changes:

In Google Optimize, click Start to start testing!

From now on, your website visitors will see the two different versions of the Purchase Now button.

Go to LiveSession, and choose the Custom param filter:

Set the ab_id with the Experiment ID value. Then, set the ab_variant. In our case, 1 represents the first variant with the red button.

Save the results as a segment (1). It will appear in the segments section (2). Click at the ab_variant filter (3):

Change the original version’s ab_variant to 0 (1), apply filters (2), and click Save as a segment.

Use the created segments to compare two groups of users. With LiveSession, you can precisely check how the behavior varies between different versions. You can always preview the properties to see which variant the user has interacted with.

What’s more, you can also see the results for the segment in Google Analytics. Just create a segment in Google Optimize:

Here’s what it looks like in Google Analytics:

Case study: how to analyze test results with LiveSession and Google Optimize

To show you how our Google Optimize integration works in action, we’ve decided to use a real-life example!

We’ve set up a little A/B test on our home page. We’ve created two different versions of the hero section on our homepage. Each one of them was shown to half of the website’s visitors. The 0 variant had only one column, while the 1 version had two:

The what: Google Optimize test results

Here are our results from Google Optimize:

As you can see, the one-column version performed significantly better than the two-column one. The conversion (from visit to signup) was twice as high for the original version.

What’s more, the Optimize analysis comes with actionable statistical data. The Probability to be best section shows you how likely it is that the winning variant really is the best one. You can also see the range of conversion rates you can expect for both cases.

That’s it for the results, but here comes another question – how exactly do people interact with these websites? Why is one version better than the other?

LiveSession can help you find the answers.

The why: LiveSession recordings

The recordings of users who took part in the A/B test are saved as a separate segment:

It’s easy to say that with session recordings you can see how A/B test participants interacted with the website.

It’s harder, however, to imagine what exactly you can see in these recordings. We don’t want to leave it to your imagination only. Have a look at some of our findings from the described A/B test:

• Users tend to click on things that are not clickable, such as the “play” button on this illustration:
• You can see where people tend to stop when they’re browsing the page. For instance, they spend more time reading the section about data privacy:
• You can track the most common next steps, just like you’re sitting next to the user! In most cases, the visitors go to Features and Pricing after seeing the main page: