Issue 37: Stop having bad test ideas
Fact: Half your test wins are not winners after all
In this issue
Follow us on YouTube
Help us finish our 2023 CRO Salary report
Stop having bad test ideas - with Bart Schutz
CRO Short of the week: Who needs CROs?
ICYMI: The Best CROs start with RESEARCH with Rommil Santiago
We’re looking for Podcast guests!
The CRO meme of the week
Stop having bad test ideas - with Bart Schutz
The math around Experimentation can be flustering. As Bart Schutz shares with Richard Joe on Experiment Nation's latest podcast, HALF of the ideas we think are winners are not winners after all.
But all hope isn't lost according to Bart!
Firstly, the cost of publishing non-winners is low. But more importantly, by structuring your experimental process to have fewer bad ideas you can increase your chances of success.
Here’s a sneak peek:
"...people are usually shocked when I tell them that half or usually even more of their winning screams are not winners at all, [it's] just based on statistics. But it’s not the problem because pushing a winner live has very little cost, usually, right?...how much did it cost, maybe four hours...maybe even $1,000 or euros, whatever...it’s the costs of a false winner tend to be very low."
👉 Listen to the full episode here - and don’t forget to Subscribe it really helps us out!
Follow us on YouTube
Don’t miss a single interview, CRO Short, or conference session by following us on YouTube. Not only does it help us out, but it also helps promote CROs from around the world.
📊 Help us finish our 2023 CRO Salary report
Are you being paid fairly? Help us find out. Experiment Nation is working on its 2023 CRO Salary Report. We will share a high-level summary of the results in APRIL while our newsletter subscribers will get access to the full report along with data explorer.
👉 Contribute to the community and fill in our short survey.
YouTube CRO Short of the week
A test didn’t go the way we wanted. We’ve all been there. Dust yourself off, learn, and iterate!
Subscribe to our YouTube channel
We’re looking for Podcast guests!
We are filling out Experiment Nation's podcast calendar and are looking for Conversion Rate Optimizers (CROs), Experimenters, and A/B Testers with interesting client stories - ones from which they have learned valuable lessons that others can learn from - to appear on the show!
If you're passionate about CRO and experimentation and you'd like to get your voice out there - let us know!
The CRO Meme of the week
ICYMI: The Best CROs start with RESEARCH with Rommil Santiago
Often at times, businesses look at their site's performance and wished it was better (who doesn't). The typical way site optimization plays out is that folks will start looking at KPIs and funnels, finding drop-off points, and then they'll start coming up with random ideas to test. Now, none of this is bad per se - but it isn't very efficient. This is because quantitative information is only half the story. Metrics will only inform you about what customers have done but not about WHY.
If you really want to rock Growth, Experimentation, or Conversion Rate Optimization - you absolutely need to start with RESEARCH.
Here’s a sneak peek:
Where should you start testing, like with most processes, starting is usually the most challenging part. You know the feeling, it’s when you don’t have your flywheel going quite yet, and you’re not sure at what stage to jump in. When it comes to experimentation. The usual places I see people jump in are at research, hypothesis creation, or setting up the test. And I’m going to tell you why the best CROs always start with research.
👉 Listen to the full episode here - and don’t forget to Subscribe it really helps us out!
Share our newsletter with someone interested in Experimentation and Conversion Rate Optimization - it helps us a lot!