Conversion Rate Optimization — Week 6| CXL Review

This is the fifth week of Conversion Optimization Training. Let’s get straight into the courses I learned this week.

Course: Statistics for A/B Testing

Instructor: Georgi Georgiev (

Total time: 3hrs 30mins

Difficulty/Skill Level: Advanced

Key Takeaways

· Data may be a proxy for reality — we hope to know the underlying cause behind the info (human behaviour) through its output. However, not all data is made equal and it’s our job to be healthily sceptical of any data put ahead of folks with the claims of being ‘insightful’.

· The role of A/B testing in business deciding are: 1) Establishing causal links, 2) Estimation (including uncertainty estimate), and 3) Risk management

· Introducing an experimental design with an impact group means we’ve got a real-time baseline. Random allocation of the groups allows us to ignore all the factors that are unnoticed by modelling the metrics as random variables.

· Before you run an A/B test, run an A/A test and see if an equivalent page/experience serves different results (you are searching for false positives in your eventual A/B test).

· we will translate business questions into statistical hypotheses under a specified statistical model- this enables us to calculate probabilities and estimate the uncertainty of every, therefore managing the upper limit of business risk

· Decision logic for A/B tests: We should notice the significant difference from the null, then we should reject the hypothesis derived from statistics and adopt a new checkout experience or keep up the previous experience.

· Communicating one statistic is best than communicating many- this statistic should reflect the discrepancy of the info from a statistical model

· The Z-score may be a measurement of the difference between the mean of a distribution and a given observation expressed in many ordinary deviations. It accounts for the estimated variance, the observed distance from the model and therefore the sample size.

· The p-value is that the probability, under the required statistical model for the null hypothesis, of observing a statistic as extreme or more extreme than the observed.

· The outcome we obtain from the test is very important if the result p-
value gets a previously determined threshold known as the significance threshold(𝛂).

· Confidence Intervals are random intervals such as a particular percentage of them, constructed over many tests, would cover the true value of the parameter of interest.

· the sort II error rate is that the probability of observing a non-significant p-value at a particular threshold α if a real effect of a particular magnitude μ1 is present.

· The statistical power of a statistical test is defined because the probability of observing a p-value is statistically significant at a particular threshold α if a real effect of a particular magnitude μ1 is present.

· A/B tests are designed to unravel the difficulty of attribution by eliciting causal links. We don’t deny a conversion to an A/B test supported by a user coming from a selected traffic source, nor should we do so if that user is additionally simultaneously exposed to a special test.

· Expressing a result as a 3% lift is simpler to interpret than 0.00001. Percentage change outcomes are often easily translated into business results.

· Getting stakeholders buy-in on test design, reporting and action even before the test starts is vital

Course: A/B Testing Mastery

Instructor: Ton Wesseling (

Total time: 5hrs 10mins

Difficulty/Skill Level: Advanced

Key Takeaways

· Statistical power is that the likelihood that an experiment will detect an impact when there’s an impact there to be detected.

· Test against a high enough significance level (90% or 95%) otherwise you’ll declare a winner when there isn’t any impact (False positives).

· once you start: attempt to test on pages with a high Power (>80%) otherwise you don’t detect effects when there’s an impact to be detected (False negatives).

· instead of going for non-binary KPIs that don’t add up to your business objectives, agree on an ‘Overall Evaluation Criterion’ within your organisation.

· This criterion might be the sum of weighted factors like conversion/action, time to action, visit frequency, etc.

· We need to apply the 6V Conversion Canvas’ in our research-
5.Verified and Voice.

· Before you commence your experiment, make sure you have clear communication within your organisation on 1) the matter, 2) the proposed solution, and 3) the anticipated outcome.

· find out how to use a prioritisation model (e.g. PIPE) and transfer it to an A/B-test roadmap

· Mirror your A/B test design together with your hypothesis. Don’t be scared of making quite one change. Decide whether you’re doing research or optimisation.

· a suitable probability depends on your business and therefore the level of risk it’s willing to require (e.g. making changes that supported a false positive).

What happened in Week 6

What I Loved

· The instructors are clearly on top of their fields and their depth of data about statistics, analytics and A/B testing were awe-inspiring.

· The use-case scenarios and ‘what to not do’ examples are even as useful (if less so) as the recommendations on what to try to do. this is often the important value of the CXL course in my opinion- the non-intuitive, non-obvious stuff that you simply can only learn from years of trial and error experience.

What might be Better

· I did find the statistics course a touch hard to stay up with. It looked like we got into deeply complicated math with any introduction. However, which will just be my very own fear of math acting up :)

Week 6 Overview

This was the week where things got very advanced very quickly, and my fear of math hit me in the face.

The hero that I’m, I persevered and got through all the course material for the courses below. I’m under no illusion that I even have fully grasped even half all the concepts covered in these two advanced courses. That said, I feel I even have a minimum of laid the framework in my brain to be filled in with details after lots and much of revision of those key concepts.

Thanks for spending your valuable time reading my review, I will try improvising my writing and keep you posted about the course.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store