#190 | When Conversion Rate Drops, Are You Investigating Noise?
Also: New job opportunities at Lyft and Canva
Follow us: Slack | LinkedIn | YouTube | Spotify | Substack
In this issue
Community thread: When Conversion Rate Drops, Are You Investigating Noise?
Jobs: New job opportunities at Lyft and Canva
News from Vendorland:
Community question of the week:
When Conversion Rate Drops, Are You Investigating Noise?
Experiment Nation’s Takeaways
“The goal is to avoid overreacting to every single change and only meaningful ones.” — Rommil Santiago
Segment before reacting: Breaking down CR changes by channel, page, device, and audience helps isolate where problems are actually occurring instead of treating the site as one monolithic funnel.
Not every dip matters: Using control limits or standard deviation thresholds can prevent teams from overreacting to normal fluctuation and wasting time chasing noise.
CR alone can be misleading: Several practitioners questioned whether conversion rate, in isolation, is even the right KPI to optimize or monitor this aggressively.
We need your opinion:
The community discussion
Juan Pablo: I have access to hotjar and GA4. What I usually do is look at the general conversion rate by channel to see what channels have been increasing and which have been decreasing. When the general CR has decreased, I try to filter by pages (URLs). Which page has increased CR? Which page has decreased CR? When I find a page that has decreased CR, I look at conversion by device, to see if CR has both decreased on mobile and desktop or only on a specific device. Then, I use this information to filter out session recordings on hotjar that match page + device that has decreased CR, and see if anything weird has happened.
Rommil Santiago: I’d also report on key segments/audiences. I’d also include actions to investigate and resolve the change (if negative). I’ve also used control limits to indicate if a change is truly odd. TLDR; they are statistical upper and lower limits where if the data fluctuates within them, any change is likely not that meaningful. The goal is to avoid overreacting to every single change and only meaningful ones
Claire More: @Rommil Santiago so like standard deviations? what timeframe do you typically do/would you suggest? 3-day for larger traffic sites and something like a week for smaller sites?
Rommil Santiago: Pretty much stdev yes. Actually I used to do 3 weeks. But you can play with historical to determine how sensitive you need to get.
Steven Hudspith: I would tell them CR is a pretty pointless metric to be tracking especially like this and do something much more constructive instead.
Join the conversation on our Slack
If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.
Our Slack is a place to:
Get answers in a judgement-free zone
Share learnings
Talk shop with other CROs
You can join by signing via this form and we will send you an invite within 48 hours.
Jobs of the Week
As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.
From Optimization Jobs by Experiment Nation
Find Optimization, Growth, and CRO jobs that other boards missed. With new jobs posted every hour - you will always find something new.
Staff Product Manager, Lyft Business Growth
📍 Lyft – New York, NYGrowth & Product Lead, Indonesia
📍 Canva – Jakarta, idAnalytics Lead, Full Stack
📍 Affirm – Remote USProduct Lead, Growth
📍 Enigmaio – New York, NYManager, Host & Growth Creative
📍 Turo
To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.
News from Vendorland
Target Target - New What’s new page
”Added a new What’s new page in the Adobe Target UI to help you discover the latest features, enhancements, and updates without leaving the product.”