Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review

Image credit Jason Dent on Unsplash

Testing Strategies

But…What should we test?

“When should you use Multivariate testing, and when is A/B/n testing best?”

  • Test dramatic design changes
  • Install advanced analytics to evaluate for each variation (i.e., mouse tracking info, phone calls, etc.)
  • Typically bigger gains are achieved because bigger changes are often tested.

Statistics for A/B testing

Image credit GIPHY

Non-binomial data

  • Average Revenue per user
  • Average Order value
  • Average Orders per user
  • Average Session duration
  • Average Pages per session
  • Requires extra effort
  • Can be more difficult or even impossible to extract data necessary for estimating the standard deviation from popular software such as Google Analytics
  • Increased sample size requirements due to increased variance
  • Often these metrics are closest to the business’s bottom line (e.g. ARPU*) and are more actionable.
  • May obscure underlying changes (ARPU* is up BUT is it due to an increase in conversion rate, OR average order value, OR both?).
Image credit: GIPHY
  1. Running tests one after another
  2. Running tests on isolated user samples (separate “testing lanes”)

A/B Testing Mastery

When do we run how [X] many online experiments?

  1. Risk
  2. Optimization
  3. Automation
  4. Re-think

My thoughts

Image credit: GIPHY

“I can’t make IT make sense!”

Image credit: GIPHY
  1. Intro to CRO and Best Practices Review
  2. Intro to Conversion Copywriting, Product Messaging, Psychology, and Social Proof Review
  3. Neuromarketing, Emotional Content Strategy, Influence, and Interactive Design Review
  4. Google Analytics for Beginners Review
  5. Landing Page Optimization, Conversion Research and Using Analytics to Find Conversions Review
  6. Google Tag Manager for Beginners, User Research, Fast and Rigorous User Personas Review
  7. Heuristics Analysis Frameworks for CO Audits, Google Analytics Audits, How to Run Tests
  8. Testing Strategies, Statistics for A/B Testing and A/B Testing Mastery Review
  9. Optimizing for B2B, Customer Value Optimization, Digital Psychology, and Behavioral Design Training Review
  10. Advanced Experimentation Analysis and Applied Neuromarketing Review
  11. How to Design, Roll Out, and Scale an Optimization Program; Evangelizing for Optimization in Enterprise Review
  12. Building Your Optimization Technology Stack and CRO Agency Masterclass Review

--

--

--

Queen👑of CRO💖 Conversion Rate Optimization Let's make even more money!

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Linear regression — All you need to know

YOLO: You Only Look Once

The New Role of the Data Scientist at Nonprofits

Tools for quick initial insight into structured data

Dash App: Experiment with the Best Multi-Armed Bandit Algorithms.

6 Best QGIS Courses & Tutorials [2020]

Hypothesis testing for data scientists

2020: A year of remote dataviz

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ashley Neal

Ashley Neal

Queen👑of CRO💖 Conversion Rate Optimization Let's make even more money!

More from Medium

Data in the Meta- Age

Metadata is just as important as the data itself

Netflix Discovery Journey | HCD

Drowning in Data