#164 | Try our free Bayesian Test Calculator
Also: Just because a test won once doesn’t mean it will win forever.
TL;DR
I get it. You don’t have time for another newsletter. Here’s the summary and the CTAs.
CRO Posts of the week: How to engage people, you can’t a/b test compliance.
Ask CROs Anything: Ask Koalatative‘s Gerda Thomas and Experiment Nation‘s Rommil Santiago anything here.
Our new Bayesian Calculator: Learn more about BayesianBuddy here.
CRO Tip of the Week: Just because a test won once doesn’t mean it will win forever. Learn from Emiliano Blanco here.
Jobs: Experimentation Job Listings of the Week include roles from Just Eat Takeaway, SimpliSafe, and Wolt.
Get Prove It or Lose It for less than $10 CAD on Amazon.ca. Only a few copies left.
Reminders: You can still watch our virtual conference! Register to get access. You can save 50% off of Rommil Santiago’s book Prove It or Lose It by buying the PDF version here.
📈 CRO Posts of the week
Every week, we find interesting LinkedIn posts about Experimentation and CRO. Check them out, comment, and feel free to reshare them!
“The difference? Just a few extra words. The impact? Clear as day.” from Alejandro Cabrera Yaksic | 👉 Read the post here
“People engage with relatable content.” from Danielle Peckskamp | 👉 Read the post here
“You can’t A/B test your way past compliance.” from Sandeep S. | 👉 Read the post here
📈 Ask CROs Anything. Seriously.
Koalatative‘s Gerda Thomas and Experiment Nation‘s Rommil Santiago are cooking something new! We will answer ANONYMOUS questions from the CRO / Experimentation community in a new YouTube series.
If you have ANY CRO / Experimentation questions you’ve always wanted to ask but were too shy to ask, here’s your chance. There are NO stupid questions. NO topic is too spicy. Ask as many as you’d like. We’ll give you the raw, unpolished truth. (I hope we don’t regret it)
Our Latest Free Tool: BayesianBuddy
If you’re in Conversion Rate Optimization (CRO) or Product Experimentation, sometimes Frequentist just isn’t it for you - maybe you don’t have a lot of sample size, and you just want to have a decently good betting chance of a winner - Bayesian could be a fit for you.
Based on facing this problem at several companies, we created BayesianBuddy: a simple Bayesian Calculator to help you figure out test winners, probability of winning, and risk of promoting a losing variant so you can make more informed decisions.
You can share results with others, compare multiple variants, set your own thresholds for risk, as well as success.
🎯 CRO Tip of the Week: Rerun Your Winning Tests After 12 Months
Just because a test won once doesn’t mean it will win forever.
As Emiliano Blanco shared on the Experiment Nation Podcast, if you’re working in a startup or fast-growing company, your users, traffic mix, and product experience can change dramatically in a year.
That means:
The context that made your A/B test successful last year might no longer hold true today.
Your audience composition and traffic sources may have shifted.
Your seasonality effects and competitive landscape may have evolved.
“If I’m running a test in a startup, I know I should rerun it again in 12 months — because ideally, the company will have grown.” – Emiliano Blanco
The takeaway
Treat every “winning” experiment as a snapshot in time, not a permanent truth. Schedule a rerun of key tests every 12 months (or sooner if traffic or behavior changes significantly).
Doing so ensures your decisions stay grounded in current customer behavior — not last year’s assumptions.
💼 Experimentation Job Listings of the Week
From ExperimentationJobs.Com
Looking for a new challenge in experimentation? Find 100+ experimentation-related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week’s featured roles:
Senior Python Data Engineer – Experimentation Platform at Just Eat Takeaway (London, United Kingdom)
Sr Analyst Site Analytics and Experimentation at SimpliSafe (Boston, USA)
Machine Learning Engineer, Personalization at Wolt (Berlin, Germany)
AI‑Powered Search & Content Optimization Lead at Empathy (New York, USA)
Experimentation Delivery Manager at Creative CX (London, United Kingdom)
Thanks for writing this. How about qualitativ data? Brilliant.