#155 | Ethical Testing & Design in Experimentation
Also: Checkout our pre-post analysis tool, LaunchBuddy, and our New CRO Coaches
In this issue
Our Latest Interview: Ethical Testing & Design in Experimentation: Leaks vs. Puddles Framework with Namrata Agarwal
LaunchBuddy is in Beta - Our Pre-Post analysis tool for low-traffic sites is live
CRO Posts of the week: Avoid Silos, You can win but have no business impact, Retention Is Cheaper Than Acquisition
Meet our new CRO Coaches: Lucas Vos and Rommil Santiago
Jobs: Experimentation Job Listings of the Week
Croct is a headless AB testing and personalization platform.
Here is what they offer:
Bayesian AB testing with great performance, low latency, and native integrations (works really well with React and Next.js, both client and server-side).
Feature flags for complex experiments and gradual rollouts.
Real-time personalization with cutting-edge user segmentation and built-in CDP and analytics (no need for third-party vendors).
Component CMS with dynamic content to give autonomy for the CRO and product teams while also ensuring the website will never break (tech and design teams love it).
Combine these things with their pricing packages (starting at $100/month), and it does sound compelling.
Create your forever-free account if you want to explore by yourself or book a meeting with Juliana Amorim (she is also in our community).
They’re also offering 10% OFF for Experiment Nation subscribers on both monthly and annual plans.
Ethical Testing & Design in Experimentation: Leaks vs. Puddles Framework with Namrata Agarwal
In this episode of Experiment Nation, host Jason Gossett sits down with Namrata Agarwal, a seasoned product designer with over 17 years of experience building digital products, websites, and apps.
Together, they explore:
Ethical testing: how to balance user needs with business goals.
Leaks vs. puddles framework: a practical way to identify urgent vs. non-urgent product problems.
The role of design in CRO and experimentation.
Frameworks, guardrails, and collaboration strategies for building an experimentation culture.
LaunchBuddy is now in Beta!
What's LaunchBuddy?
Let me ask the Product Experimenters out there: How many times have the product teams you work with launched a feature without an a/b test? And how many times have you been asked to figure out the impact of that launch?
I'm guessing more than once.
Now, you're forced to do a pre-post analysis, or an ocular analysis (with your eyes). And to make matters worse, if you're working on a site that has low traffic without an A/B testing tool, this analysis could be a nuisance.
That’s why I launched a cute little tool (yes, I vibe-coded it over the weekend for all you AI maniacs out there) for low-traffic sites to compare pre- vs post-launch metrics.
What are the stats under the hood?
It uses Bayesian Bootstrap and CUPED so that you don't need to worry about normality, sample size, and you can reduce some variance, though your sample should still be representative, of course. Before you folks think I'm a stats expert, I just play one on television. I bounced this approach off a Data Scientist from Meta and someone with a PhD in Experimental Psychology. But that's enough stats jargon.
What do you need to use it?
Just 2 CSVs (one for pre, one for post) that include dates, user IDs, and the metric to optimize (you may need BigQuery for this if you're on GA4).
Other than that, click the analyze button and bam, results show up in a few seconds. The tool never stores your data; everything is processed locally. But you can still share the results with others with a link or PDF.
I feel it's a pragmatic and better alternative to what is usually eyeballing results. With that said...
I would love feedback! I realize it's a tool that not many people need, but it solves a few issues for me, so I thought I’d share it with the experimentation community.
CRO Posts of the week
Every week, we find interesting LinkedIn posts about Experimentation and CRO. Check them out and feel free to reshare them!
"Looking only at website or app analytics can give us a siloed, sometimes even misleading view of our customers." from Sachita Agrawal | 👉 Read the post here
"You can run 52 tests a year, report big wins, and still drive zero business impact." from Friedolin Stockmeier | 👉 Read the post here
"Retention Is Cheaper Than Acquisition" from Fatihu Yakub | 👉 Read the post here
Meet our new CRO Coaches: Lucas and Rommil
About Lucas
"As a Lead CRO specialist with more than 500 A/B tests under my belt, I have had the pleasure to lead the programmes for multiple A-brands in building their experimentation culture and optimizing their most important customer journeys, from subscription flows, reach for media platforms and of course to eCommerce conversions and AOV."
Expertise
Subscription optimization, media organizations, change management, and in-house experimentation
Timezone
Amsterdam (GMT+1)
About Rommil
"With over 15+ years of experience in optimization, I’ve launched several Experimentation programs and helped organizations accelerate growth through data-driven experimentation. My work has generated over $110M in proven business impact to date and levelled up how companies work. I’ve also written an Amazon Canada Bestselling book entitled, Prove It or Lose which helps experimentation program owners survive the politics of launching a program.
Read more about me here - I’m open for opportunities as well."
Specialties
CRO, eCommerce, job hunting, career coaching, Product Experimentation, Growth, SaaS, PLG
Timezone
EST (from 10 AM to 4 PM only)
Have you ever wanted to be a CRO Coach?
Experiment Nation is launching a coaching program. Set your own schedule, choose your session length, and set your own prices (though free services will be prioritized). Get promoted through Experiment Nation’s site and newsletter. If that sounds interesting to you, sign up below.
Every week, our newsletter is delivered to over 2.5K CROs and Experimenters around the world. If you’d like to explore advertising opportunities with us, contact us here.
Experimentation Job Listings of the Week
From ExperimentationJobs.Com
Looking for a new challenge in experimentation? Find 100+ experimentation-related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week’s featured roles:
CRO Specialist at PM Digital Design (United Kingdom)
Software Engineer, Product Growth and Experimentation at Cloudflare (USA)
CRO Specialist at SQLI (Amsterdam, Netherlands)
Lead Product Designer – Growth at Paired (London, United Kingdom)
CRO Specialist at MSC Cruises (Naples, Italy)