#71 | The CRO Statistical concepts all CROs should know featuring Ishan Goel
Also: Vote for your favorite CRO!
Welcome to Experiment Nation’s Newsletter! Every week Rommil Santiago, the founder of Experiment Nation, shares interviews with CRO experts, instructional conference sessions, reports, and events. We hope you find this useful for your own Conversion Rate Optimization practice!
Would you like to reach our >2K subscribers? Reach out.
In this issue
The CRO Statistical concepts all CROs should know featuring Ishan Goel
Treat Your Experimentation Program Like a Product
Experimentation Job Listings of the Week
Contribute to the CRO Salary report!
Experiment Nation’s CRO Awards: Time is running out!
The Certified CROs of the week
Goodbye to CRO Wordle
The CRO Statistical concepts all CROs should know featuring Ishan Goel
In this insightful video, VWO’s Ishan Goel, transitioning from a software engineer to a researcher, delves into crucial CRO statistical concepts. He sheds light on statistical significance in experimentation, explores A/B testing intricacies, emphasizes the efficiency of sequential testing, and discusses challenges in stopping tests prematurely. The talk concludes by highlighting the intersection of experimentation and statistics in the realm of Conversion Rate Optimization.
Here’s a preview:
“…the idea I think which has become accepted in the industry is let test run for two weeks but I also accept and say that if there are some crucial stakes involved like for instance if you understand why the performance is bad or you have some crucial guardrail metric being hurt very bad then you should go ahead and stop them but it should not be a general practice…”
Watch Richard’s chat with Ishan here.
Past interviews
Treat Your Experimentation Program Like a Product
Products go through several stages over their lifecycle (e.g. Introduction, Growth, Maturity, and Decline). Each stage requires different approaches, focuses, and strategies to maximize returns. For example, during the introduction phase, there is a lot of iteration, and trying to figure out what works. During the growth phase, this is where you ramp things up. Mature products often benefit from a focus on optimization and operational excellence. Finally, during their decline, products tend to become discontinued - giving way to something better. An Experimentation Program should be treated no differently.
The question about ROI
As the founder of Experiment Nation, a global community of conversion rate optimizers (CROs) and product experimenters, I get to hear a lot of the questions that folks have about setting up an experimentation program. One of the more recent topics that caught my eye was one about the goals of a program and whether ROI should be one of them. The argument, in short, is whether CROs should focus on a program’s ROI when experimentation should be about learning. From my experience having stood up programs in the past at various companies, the answer to this is nuanced and… Read more
Latest Experimentation Jobs
Looking for a new challenge in experimentation? Find 100+ experimentation-related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote, or hybrid. Take a look and start pivoting your career.
This week’s featured roles:
Senior Manager, Conversion-Rate Optimization at Seer Interactive (USA)
Senior Analyst-Experimentation at Dunelm (London or Syston, United Kingdom)
Senior Engineering Manager, Experimentation at 2U (Lanham or remote, USA)
Senior Software Engineer – Experimentation Platform at Apple (Santa Clara Valley, USA)
Data Scientist, Customer Experimentation at Eppo (USA)
Are you hiring? Submit here
Time is running out - vote now!
Experiment Nation is collecting nominations for the TOP TEN must-meet experimenters of 2023! We want to celebrate CROs who love to connect with others, share their advice, and introduce people to others! Vote here while you can!
We need your help! Contribute to the 2024 CRO Salary report
How do CRO salaries break down by seniority, country, minority status, team size, and gender in 2023? Experiment Nation is compiling its 2024 CRO / Experimenter salary report. If you've ever wondered whether you are paid fairly in your current job or for an upcoming one, you'll know how valuable this resource is for the community. To get your copy of the report when it is ready, contribute to the study and subscribe to our newsletter. Get your access here: https://lnkd.in/gwADBWTd
The Certified CROs of the week
Congratulations to the amazing CROs for recently passing Experiment Nation's CRO Qualification Exam!
About the exam
The exam was crafted by industry veterans and covers the areas that CROs should be knowledgeable about. It takes less than 25 minutes to take. If you'd like to get certified, learn more here: https://lnkd.in/gHdJbJV2
Goodbye to CRO Wordle
It’s been a good run, but we’re saying goodbye to our CRO Wordle game. Thanks to everyone who played it over the years! We will keep the leaderboard up, but the game will be taken down on December 31, 2023.