Issue 38: Get better at A/B test statistics

Not all tests are made equally

Apr 06, 2023

Follow us on YouTube
Help us finish our 2023 CRO Salary report
Avoid AB Testing Statistics Mistakes with Max LI
Poll of the week: Do you want to speak at our virtual conference?
CRO Short of the week: Just one more thing
We’re looking for Podcast guests!
The CRO meme of the week
Coming soon: Improve conversions with the Message-market Fit framework With Daphne Tideman

Follow us on YouTube

Don’t miss a single interview, CRO Short, or conference session by following us on YouTube. Not only does it help us out, but it also helps promote CROs from around the world.

Visit our YouTube Channel

📊 Help us finish our 2023 CRO Salary report

Are you being paid fairly? Help us find out. Experiment Nation is working on its 2023 CRO Salary Report. We will share a high-level summary of the results in APRIL while our newsletter subscribers will get access to the full report along with data explorer.

👉 Contribute to the community and fill in our short survey.

Fill out our short survey

Avoid AB Testing Statistics Mistakes with Max LI

Do you want to understand whether your work is better than what's currently out in the market? Just run an A/B test, right?! Whoa there. Not so fast, buster. There are many statistical aspects to consider!

Qike (Max) Li, a staff data scientist over at Wish shares many of the pitfalls you should avoid when running experiments - and walks through the math for us in a very approachable session.

Here’s a sneak peek:

“When we need to run a test to show that the treatment is no worse than control it's incorrect to run a regular AB test. And non-inferiority test is designed for those use cases.”

👉 Listen to the full episode here - and don’t forget to Subscribe it really helps us out!

Poll of the week: Do you want to speak at our conference?

YouTube CRO Short of the week

Early in-house Conversion Optimization / Experimentation teams (and let's admit it, many mature ones) are often understaffed and have to rely on the charity of other teams to get their tests built (even those with WYSIWYG tools).

If you find yourself in this situation, try some of these tips:

1) Frame your work as a way for others to achieve their goals - everyone is busy trying to get their work done but if you are able to demonstrate that you have skin in their game, you might find a new ally.

2) Earn trust - People like to work with people they trust. Try offering help and delivering on it. It doesn't have to be a big ask. Just be consistent in delivering on your word. Over time, folks will see you as someone worth helping.

3) Share the spotlight - Folks often like when they are mentioned in a positive light and often want to work closely with those that do so. This is not to say to be cheesy about it, or fake. Balance is key.

What advice do you have?

Subscribe to our YouTube channel

We’re looking for Podcast guests!

We are filling out Experiment Nation's podcast calendar and are looking for Conversion Rate Optimizers (CROs), Experimenters, and A/B Testers with interesting client stories - ones from which they have learned valuable lessons that others can learn from - to appear on the show!

If you're passionate about CRO and experimentation and you'd like to get your voice out there - let us know!

Apply here