#47 | A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi
Also: Build or buy your Google Optimize replacement with Kenneth Kutyn
In this issue
Get early access to our Fall Conference!
A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi
Build or buy your Google Optimize replacement with Kenneth Kutyn
The Certified CROs of the week
CRO Short of the week: Stats are hard
What’s your favorite CRO tool?
Next week: We announce our Conference lineup!
Get early access to our Fall Conference!
While our conference is only in September, you can secure FREE access to our event now! To get access to our DOZENS of virtual and on-demand sessions:
Subscribe to our YouTube channel
Leave a comment on one of our videos
Tell us you’ve subscribed via this form and we’ll send you the conference URL
A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi
Here’s a sneak peek:
“So we actually share the data of what happens when we run an experiment for six months. And we showed that the users that stay with us for six months are so different than most users on many, many attributes. So remember, you’re introducing bias here, there are users that you think are new, but they’re really older users. And so if you’re accumulating stuff over a long period, like I’m computing revenue per user, that metric is much less reliable. Or when you run an experiment for a long time.”
Bonus for our readers: For those who want to learn more in an interactive 10-hour class (5 sessions x 2 hours) with Ronny, it starts June 12: https://bit.ly/ABClassEN
There's a $500 off coupon code at the end of the interview.
Build or buy your Google Optimize replacement with Kenneth Kutyn
Here’s a sneak peek:
“Does it give you a competitive edge to build your own experimentation product? No companies out there building their own Google Docs, because if you are Nike, you’re not going to beat Adidas by having a better word processing tool. So the same thing if you are if you are a Nike and you want to experiment better, are you gonna get a competitive edge from that? Do you have the engineering resources to build that? Or are you just going to do it to try and save costs because it’s often the opposites true.”
The Certified CROs of the week
Congratulations to the amazing CROs for recently passing Experiment Nation's CRO Qualification Exam!
About the exam
The exam was crafted by industry veterans and covers the areas that CROs should be knowledgeable about. It takes less than 25 minutes to take. If you'd like to get certified, learn more here: https://experimentnation.com/about-the-experiment-nation-cro-qualification-exam/
YouTube CRO Short of the week
Stats are hard.
What is your favourite CRO tool of 2023?
What are some of your go-to CRO / Experimentation tools and resources? Ones that newcomers to the industry should check out? We'll share the top suggestions and shout out all the contributors at the end of the year!
While we are very selective about whom we partner with, for inquiries about sponsoring the podcast, or this newsletter, email: rommil+partners@experimentnation.com.