#43: Ignore these A/B Testing Statistics "Issues"

Also: Submit your conference session!

May 11, 2023

Poll of the week: What kind of podcasts should we produce?
A/B Testing Statistics issues Experimenters should ignore with Timothy Chan
The Certified CROs of the week
Call for Conference Submissions
Be a guest on our podcast - Only a few spots left!
CRO Short of the week: Spellcheck your work
Next Week: Get Executive buy-in for your CRO program with Jonny Longden

Poll of the week: What kind of podcasts should we produce?

A/B Testing Statistics issues Experimenters should ignore with Timothy Chan

Experimenters obsess about many A/B Testing Statistics concepts - but not all of them are worth worrying about. Statsig's Timothy Chan shares his thoughts about test isolation, interaction effects and confidence levels - some of his thoughts may surprise you.

Here’s a sneak peek:

“I came to the sort of the conclusion that interaction effects tend to be overestimated. They're, they're number one, they're actually quite rare. And then when they're found, they're actually in most cases underwhelming, and in such a way that they generally still produce directionally accurate results. So it wouldn't change your decision over whether to ship a feature, it would just change your estimate of what that effect, true effect is. So I think in some ways, folks who are really worried about interaction effects are really making too much of a big deal.”

👉 Listen to the full episode here - and don’t forget to Subscribe it really helps us out!

If you’d prefer to read the AI-generated transcript or listen to the podcast version of this interview, visit our site:

The Certified CROs of the week

Congratulations to the amazing CROs for recently passing Experiment Nation's CRO Qualification Exam!

Andrej (Andy) Denkovski
Kiran Mahara

About the exam
The exam was crafted by industry veterans and covers the areas that CROs should be knowledgeable about. It takes less than 25 minutes to take. If you'd like to get certified, learn more here: https://experimentnation.com/about-the-experiment-nation-cro-qualification-exam/

The 2023 Conference: Early call for submissions

Our Early Call for Conference submissions is now open! This year, to help inspire potential speakers for our annual CRO conference held every September, we’ve asked our community for topics they’d like to see covered. We’ve compiled the list here. If you’d like to submit a session idea, fill out the form below.

Note: To help us achieve our goal of equal representation for this event, we will be prioritizing underrepresented populations such as women and visible minorities. Thank you for your support.

Submit your sessions

If you’ve never seen any of our past sessions check out these playlists:

If you like our content, consider subscribing to our YouTube channel!