What do you do when an experiment tanks conversion rate by 8% in less than a week?
In this episode, Gerda and Rommil break down real CRO and experimentation scenarios submitted by the community, including failed tests, executive pressure, guardrail metrics, knowledge management, developer friction, and why experimentation teams always seem to end up back in spreadsheets.
Check out: http://katsed.com/
We also discuss:
Root cause analysis for failed experiments
How to calm leadership during testing disasters
Guardrail metrics and control limits
Why experimentation knowledge banks become messy
Airtable vs spreadsheets vs enterprise tools
Why developers and designers struggle with experimentation culture
How to onboard technical teams into testing workflows
A practical conversation for experimentation leads, growth teams, product managers, CRO specialists, and anyone building an experimentation culture.
Chapters
00:00 Burger Menu Debate: Left vs Right
01:28 Mystery Meat Navigation & UX History
03:03 Why Modern Phones Break UX
05:08 Community CRO Scenarios Begin
06:18 Experiment Tanks Conversion by 8%
07:33 Root Cause Analysis & Leadership Pressure
09:17 QA, Tracking Errors & Failed Tests
10:03 Building a CRO Analysis Tool
12:18 Probabilistic Root Cause Detection
14:26 Managing Panic During Bad Experiments
15:21 Guardrail Metrics Explained
16:40 Using Control Limits in Experimentation
18:18 Building an Experimentation Knowledge Bank
20:05 Why You Don’t Need AI for Everything
21:08 Why Process Changes Create Admin Pain
23:08 Migrating Between Experimentation Tools
24:31 Why Teams Always Go Back to Spreadsheets
26:50 CRO Teams vs Developers & Designers
29:09 Why Early Experiments Should Be Simple
30:41 Why Developers Hate “Throwaway Code”
32:16 Designers vs Experiment Constraints
33:02 Outro & Next Community Questions
