Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review

Image credit Jason Dent on Unsplash

Testing Strategies

But…What should we test?

“When should you use Multivariate testing, and when is A/B/n testing best?”

  • Test dramatic design changes
  • Install advanced analytics to evaluate for each variation (i.e., mouse tracking info, phone calls, etc.)
  • Typically bigger gains are achieved because bigger changes are often tested.

Statistics for A/B testing

Image credit GIPHY

Non-binomial data

  • Average Revenue per user
  • Average Order value
  • Average Orders per user
  • Average Session duration
  • Average Pages per session
  • Requires extra effort
  • Can be more difficult or even impossible to extract data necessary for estimating the standard deviation from popular software such as Google Analytics
  • Increased sample size requirements due to increased variance
  • Often these metrics are closest to the business’s bottom line (e.g. ARPU*) and are more actionable.
  • May obscure underlying changes (ARPU* is up BUT is it due to an increase in conversion rate, OR average order value, OR both?).
Image credit: GIPHY
  1. Running tests one after another
  2. Running tests on isolated user samples (separate “testing lanes”)

A/B Testing Mastery

When do we run how [X] many online experiments?

  1. Risk
  2. Optimization
  3. Automation
  4. Re-think

My thoughts

Image credit: GIPHY

“I can’t make IT make sense!”

Image credit: GIPHY
  1. Intro to CRO and Best Practices Review
  2. Intro to Conversion Copywriting, Product Messaging, Psychology, and Social Proof Review
  3. Neuromarketing, Emotional Content Strategy, Influence, and Interactive Design Review
  4. Google Analytics for Beginners Review
  5. Landing Page Optimization, Conversion Research and Using Analytics to Find Conversions Review
  6. Google Tag Manager for Beginners, User Research, Fast and Rigorous User Personas Review
  7. Heuristics Analysis Frameworks for CO Audits, Google Analytics Audits, How to Run Tests
  8. Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review
  9. Optimizing for B2B, Customer Value Optimization, Digital Psychology, and Behavioral Design Training Review
  10. Advanced Experimentation Analysis and Applied Neuromarketing Review
  11. How to Design, Roll Out, and Scale an Optimization Program; Evangelizing for Optimization in Enterprise Review
  12. Building Your Optimization Technology Stack and CRO Agency Masterclass Review




Queen👑of CRO💖 Conversion Rate Optimization Let's make even more money!

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

COVID-19 in Mexico: A data driven analysis

Sinusoid Regression: The Stats (Receipts)

Joyent CoPilot: Bringing Application Awareness to Cloud Infrastructure. Part V.

We’re tracking the most up-to-date information about the coronavirus in North Carolina.

SIR model for modelling COVID 19 mathematically

Data Science in Finance : Data Reduction

Using Proof of Stake (PoS) Consensus Algorithm on the ERC20 Platform for a Native School Coin —…

Visual representation of the potential of blockchain technology to connect educational institutions

K-mean Clustering -

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ashley Neal

Ashley Neal

Queen👑of CRO💖 Conversion Rate Optimization Let's make even more money!

More from Medium

Customer Churn Analysis

Product Management for AI and Data Science

Application Lifecycle Management

Study Notes on Online Controlled Experiments (A/B testing) — Part 1