Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review

This article is part 8 of 12 reviewing the CRO Minidegree at the CXL Institute.

Image credit Jason Dent on Unsplash

More CXL Institute reviews. Take a look! 1|2|3|4|5|6|7|8|9|10|11|12

Testing, testing, and more testing is the focus of my review this week at the CXL Institute.

Testing Strategies

This course is taught by Peep Laja. One of the key things noted is that it is important to figure out exactly what to test — things that are actually going to make a difference in your conversion rate!

When we test the wrong things we waste time, effort, and most importantly money.

Whether you decide to test one website change or several should depend on your business goals and the amount of traffic your site has. If you plan to make many changes for testing consider the following two strategies for the best results:

Strategy #1

Ensure each change is directly connected to an identified problem.

Strategy #2

Each change on your website supports the same hypothesis.

But…What should we test?

A very common testing question in conversion optimization is:

“When should you use Multivariate testing, and when is A/B/n testing best?”

Before answering this question let’s take a look at what both A/B testing and multivariate testing can do.

With A/B testing the following can be achieved:

• Test dramatic design changes
• Install advanced analytics to evaluate for each variation (i.e., mouse tracking info, phone calls, etc.)
• Typically bigger gains are achieved because bigger changes are often tested.

Multivariate testing on the other hand requires a ton of traffic! If your website does not have a lot of traffic, high conversions can be substituted.

As a general rule of thumb:

Typically, if your traffic is under 100,000 unique visitors per month, you’re probably better off doing A/B testing instead of multivariate testing. The only exception would be the case where you have high-converting (10% to 30% CR) lead generation pages.

So the answer to our previous question:

A/B testing is ideal for testing dramatic changes, easy to implement, fast and insightful.

Bandits

Bandits testing is best utilized for short term seasonal campaigns such as Black Friday, Valentine’s Day, Mother’s Day, etc. In general, not much human involvement is required for bandits. They are ‘set it and forget it’ and great for automation at scale.

Statistics for A/B testing

Image credit GIPHY

This course is taught by Georgi Georgiev and it dives straight into statistics, no chaser (yikes!). A lot of information was shared in this course below. I’ll highlight the concepts that actually stuck with me (FYI: I really struggle with this topic, but I’m getting better each day with practice!).

Non-binomial data

Non-binomial data are continuous metrics that are non-discrete and their values span a portion of the entire real line.

Examples of non-binomial data:

• Average Revenue per user
• Average Order value
• Average Orders per user
• Average Session duration
• Average Pages per session

Key things to keep in mind when working with non-binomial metrics:

• Requires extra effort
• Can be more difficult or even impossible to extract data necessary for estimating the standard deviation from popular software such as Google Analytics
• Increased sample size requirements due to increased variance
• Often these metrics are closest to the business’s bottom line (e.g. ARPU*) and are more actionable.
• May obscure underlying changes (ARPU* is up BUT is it due to an increase in conversion rate, OR average order value, OR both?).

*average revenue per use

Image credit: GIPHY

Running Concurrent A/B tests

Concurrent A/B testing is a threat to the generalizability of the outcomes due to the test sample not being representative of the target population.

External validity may be compromised if a winner in test #1 was only winning due to variant B in test #2. If variant B is not implemented, later on, the result of test 1 will have poor predictive validity.

Two ways to solve the issue of concurrency:

1. Running tests one after another
2. Running tests on isolated user samples (separate “testing lanes”)

A/B Testing Mastery

This CXL course is taught by Ton Wesseling, here I learned how to really refine and/or troubleshoot A/B testing methods to achieve maximum wins.

A commonly asked question in A/B testing is:

When do we run how [X] many online experiments?

Wesseling shared the ROAR rule of thumb model. It consists of four

Optimization phases

1. Risk
2. Optimization
3. Automation
4. Re-think

It is important to note when using the ROAR model you have to have at least 1000 conversions per month (15% impact needed) because it’s really hard to find a winner if you too low on data.

Keep in mind that conversions can mean whatever goal you’re trying to optimize for (ex. email sign-ups, phone calls, transactions, etc.).

My thoughts

I always enjoy the in-depth articles included in Peep Laja’s courses, it was no different in Testing Strategies. I feel this content lays a solid foundation for what is learned in the rest of the Testing module.

Statistics for A/B Testing is where the wheels begin falling off the wagon for me 😳 and I am literally trembling in my boots! This course for me is like being thrust directly into a bullfight with the outfit and equipment but absolutely no idea what to do with them! 😬 Or like sitting in the classroom of an advanced level Albanian language course and you’ve never ever heard Albanian spoken or seen it written — but you are expected to be able to follow along.

Now I get it the concepts taught in this course are the foundational rules and principles for CRO! But I also have to be completely honest and transparent that this is not my strongest CRO area.

I don’t know what my problem is but I just can’t seem to wrap my head around the concept of p-values, standard deviation, and z scores. Let alone actually having to explain them to someone else on my team!

Image credit: GIPHY

For the life of me…

“I can’t make IT make sense!”

At least not right now, but I am going to continue to practice and study every day until it becomes second nature. Like this guy…

Image credit: GIPHY

On the bright side, I was able to learn the concepts enough to pass the final exam with 92% and earned another CXL certificate!

Now it did take me three times to pass (the most for me for any CXL Institute’s course final exam’s so far). But I am okay with that because this really isn’t my strongest area as previously mentioned. In all honesty, I thought it would take several more times than that to pass the exam!

Some of the concepts that flew way over my head in Statistics for A/B Testing made a little more sense to me in A/B Testing Mastery.

I really appreciated Wesseling’s teaching style and the lesson slide decks that were included with each lesson. I also appreciated the brief background history he gave on A/B testing, very interesting information indeed!

Thus far, I am super satisfied with the knowledge that I have obtained from studying the Conversion Optimization course at CXL. I may have a few headaches from the courses mentioned above, but I have absolutely no regrets! I can see the CRO light at the end of the tunnel and it is exciting!

Would you like to join me by learning from the top 1% of digital marketers in the whole entire world?

Check out more of my in-depth reviews of CXL Institute’s CRO Minidegree courses:

1. Intro to CRO and Best Practices Review
2. Intro to Conversion Copywriting, Product Messaging, Psychology, and Social Proof Review
3. Neuromarketing, Emotional Content Strategy, Influence, and Interactive Design Review
4. Google Analytics for Beginners Review
5. Landing Page Optimization, Conversion Research and Using Analytics to Find Conversions Review
6. Google Tag Manager for Beginners, User Research, Fast and Rigorous User Personas Review
7. Heuristics Analysis Frameworks for CO Audits, Google Analytics Audits, How to Run Tests
8. Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review
9. Optimizing for B2B, Customer Value Optimization, Digital Psychology, and Behavioral Design Training Review
10. Advanced Experimentation Analysis and Applied Neuromarketing Review
11. How to Design, Roll Out, and Scale an Optimization Program; Evangelizing for Optimization in Enterprise Review
12. Building Your Optimization Technology Stack and CRO Agency Masterclass Review