#84 | How to learn from failures (from losing and winning tests) with Simon Girardin
Time is running out! Answer our CRO Salary Survey!
Welcome to Experiment Nation’s Newsletter! Every week Rommil Santiago, the founder of Experiment Nation, shares interviews with CRO experts, instructional conference sessions, reports, and events. We hope you find this useful for your Conversion Rate Optimization practice - and if you do, kindly tell a friend!
In this issue
Lastest Interview: How to learn from failures (from losing and winning tests) with Simon Girardin
Throwbacks: Emily Lonetto on powering past things that didn’t work and the importance of focus; and How Josh Sookman drives product development through Experimentation; Product Experimentation: Aligning the need to deliver and the need to learn with OKRs
Report: Help us with this year’s CRO Salary report!
Jobs: Experimentation Job Listings of the Week
Community: Experimentation Risk vs. Opportunity Cost
Shoutouts: The Certified CROs of the week
Be our guest: Appear on our show!
How to learn from failures (from losing and winning tests) with Simon Girardin
The latest episode of Experiment Nation's podcast is out! This week, Simon Girardin shares with our own Tracy Laranjo 🍊 about the importance of learning from every test, capturing value by reviewing data, segmenting results, and analyzing user behavior through session recordings and heatmaps. Watch the full episode below!
This week’s throwbacks
How Josh Sookman drives product development through Experimentation
For our first throwback, I have for you an Experiment Nation interview with Josh Sookman for which Josh shared how he drives product development through Experimentation. Here are some highlights:
How Josh manages stakeholder expectations and manages potential risks around riskier tests
How he identifies PMs that embrace experimentation
Using Experimentation to gauge market demand
Emily Lonetto on powering past things that didn’t work and the importance of focus
For our second throwback, I have for you an interview with Emily Lonetto in which Emily spoke about growth, startups, and of course experimentation. Here are some of the topics Emily covered:
How to drive growth in the early days of a startup when you have little traffic
How fast to iterate on your experiments - when runway is a concern
Lessons large corporations can learn from start-ups about experimentation
This is one of my favorite quotes from the interview:
"What works for one company might not work for yours, so don’t beat yourself up for trying something that seemed to kick it out of the park elsewhere and didn’t stick with your current audience or product."
Having worked a few years at startups, this quote and everything else Emily said hit home. If you find yourself at a startup wondering about how to leverage Experimentation - check out the full interview here:
Product Experimentation: Aligning the need to deliver and the need to learn with OKRs
Here’s a lovely chat about Product Experimentation between Jaya Gupta and Siddharth Taneja here are some of the takeaways:
Focus on Northstar Metrics: A good product experiment should align with the Northstar metric, evaluating the success and failure of the product. This ensures that the experiment contributes to the overall goal, such as the number of daily meals delivered in a restaurant delivery service.
Proper Segmentation is Crucial: Identifying and focusing on the right segments is essential. The experiment should consider different user groups and ensure that the chosen metrics are significant for the target audience. Neglecting proper segmentation can render the experiment inconclusive.
Technical Aspects and Consideration of Technical Debt: When conducting experiments, it's crucial to avoid negatively impacting the performance of the existing product. Additionally, managing technical debt, especially in terms of code base cleanliness and organization, is vital to ensure efficient experimentation without hindering development efforts.
Answer our 3-minute survey and get a copy of our CRO salary report!
Time is running out!
How do CRO salaries break down by seniority, country, minority status, team size, and gender? Experiment Nation is compiling its 2024 CRO / Experimenter salary report. If you've ever wondered whether you are paid fairly in your current job or for an upcoming one, you'll know how valuable this resource is for the community. To get your copy of the report when it is ready, answer our quick and ANONYMOUS survey here.
Latest Experimentation Jobs
Looking for a new challenge in experimentation? Find 100+ experimentation-related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote, or hybrid. Take a look and start pivoting your career.
This week’s featured roles:
Conversion Rate Optimization (CRO) Specialist at ConversionWise (Remote, United Kingdom)
Analytics & CRO Manager at Svea Solar (Stockholm, Sweden)
Senior Experimentation Specialist at Preply (Barcelona, Spain)
Optimisation Lead at Just Eat Takeaway (London, United Kingdom)
Product Analytics Manager at 7-Eleven (Irving, USA)
Are you hiring? Submit here
From the Community: Experimentation Risk vs. Opportunity Cost
A post from Hristian (Chris) Kambourov caught my eye this week:
“Experimentation Risk vs. Opportunity Cost… do we need it on our calendar invites?
Last year Shopify added a meeting cost estimate to their calendar invites.
The intended effect seems pretty clear: Cut out meaningless meetings. There is an actual cost to the company.
So…Why not do the same for experimentation. Remind meeting participants each day you are not actively experimenting…the company is falling behind and/or losing revenue.”
In my previous roles, I often estimated revenue lost. But I sometimes wonder if there is a way to measure the time lost to rework. Which got me thinking - how do YOU highlight opportunity cost?
The Certified CROs of the week
Congratulations to the amazing CROs for recently passing Experiment Nation's CRO Qualification Exam!
About the exam
The exam was crafted by industry veterans and covers the areas that CROs should be knowledgeable about. It takes less than 25 minutes to take. If you'd like to get certified, learn more here.
Appear on our show!
Are you a Conversion Rate Optimizer (CRO) or a Product Manager that runs Experiments on your features? If that's the case, Experiment Nation would love to have you on our podcast! If you'd like to appear on our show, sign up here: https://docs.google.com/forms/d/e/1FAIpQLScPLeH181W2FvE3icUa0ZfS4PHeMvNGgFpUSK4UVALjQWDYiA/viewform