#187 | Project the incremental lift of your experimentation program

Also: Statsig is being absorbed

Follow us: Slack | LinkedIn | YouTube | Spotify | Substack

---

In this issue

• Community thread: Project the incremental lift of your experimentation program

• Jobs: New job opportunities at Substack, Datadog + more

• Vendor updates: The latest news about Statsig

• Thanks to our paid members!

---

This newsletter is made possible with support from:

Convert Experiences is a privacy-first, AI-ready A/B testing & 1:1 personalization platform that helps lean marketing & growth teams, CRO agencies, and ecommerce stores in regulated industries build reliable experimentation programs at affordable monthly pricing.

---

Community question of the week: How do you project the lift of your program?

The question of the week:

After several test wins, how do you project the overall lift of your program?

Excerpts of what the community had to say

• Claire More: Estimated monthly impact for up to 6 months (added for all tests) with a decay model applied. It’s not the cleanest or most accurate but it’s a bit more modest than annualizing

• Shiva Manjunath:

  

• Geoffrey Bell: You can annualize the lift in the given test metrics and attempt to paint the picture that way…. Something like, these 5 tests will help contribute an additional XX orders per year, or $$ ARR, etc…. Assuming the test variants are shipped. Just a thought.

• Rommil Santiago: I used to annualize also. But it was more around giving directional information rather than true forecast information. The last time I contributed information for forecasting, I had to be very careful in terms of overlapping segments and stages, and even still, I handicapped it by a % to make up for potential market changes etc….There’s also annualized with a haircut based on significance / probability. I.e., discount the annualization by a factor.

• Matt Mulvey: Yeah, I’ve always used a diminishing / decay factor to account for the test effect fading over time when it comes to any projections beyond a quarter

• Will Critchlow: We have generally done a decay factor too. Experimenting with Bayesian cumulative impact modelling (to account for the winner’s curse). Anyone else got value from that?

• Tally Keller: Annualized benefit, sometimes expressed as a range (+\- std deviation) so the expectation is more realistic.

• Ryan Thomas: I think the important parts are:

  1. Use some model / calculation to decay the impact over time,

  2. Be transparent and up front about the fact that it’s just an estimate and how it’s calculated,

  3. Be very conservative. Our model has the impact decaying to zero over 6 months. Still no problem showing really good ROI from the program.

• Shiva Manjunath: So saying 1589% lift in AOV isn’t realistic?

• Rommil Santiago: Haircut that to 1500%

• Jess Vandenbruggen: Loving this banter. I think as long as you can show how you calculate it - and that you are expecting the return to decay over time it tells enough of the story to showcase the value of the program. We DID make it up, but we make it up the same way each time so it’s consistent

Takeaways

• Annualization Needs Guardrails: Most teams annualize wins directionally, then apply decay models, confidence discounts, or “haircuts” to avoid inflated projections.

• Conservative Forecasting Wins: Practitioners account for overlapping segments, diminishing effects over time, and changing market conditions when estimating program impact.

• Transparency Beats Precision: A consistent and clearly explained forecasting methodology is often more valuable than claiming highly accurate projections.

Join the conversation on our Slack

If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.

Our Slack is a place to:

• Get answers in a judgement-free zone

• Share learnings

• Talk shop with other CROs

You can join by signing via this form and we will send you an invite within 48 hours.

---

Jobs of the Week

As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.

From Optimization Jobs by Experiment Nation

Find Optimization, Growth, and CRO jobs that other boards missed. With new jobs posted every hour - you will always find something new.

1. Director, Growth & Strategy
   📍 Xenergy

2. Growth Marketing Manager (CRO & Ecommerce Optimization)
   📍 Happy Mammoth

3. Product Data Scientist – Growth
   📍 Gusto

4. Growth & CRO Strategist
   📍 Buck Mason

5. Growth Marketing Specialist | Full-Funnel CRO & Optimization
   📍 Kendago

To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.

From Experimentation Jobs

Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.

This week’s featured roles:

• Director of Conversion Rate Optimization at LawnStarter (USA - Remote)

• Conversion Rate Optimisation (CRO) Specialist at Glass Atlas (Manchester, United Kingdom)

• Product Manager, Relevance and Personalization at Airbnb (San Francisco, USA)

• Product Manager – Experimentation & Growth at Conversionista! (Stockholm or Gothenburg, Sweden)

• Digital Conversion Manager at Rabobank (Utrecht, Netherlands)

Are you hiring? Submit here

---

News from Vendorland

• Statsig and Amplitude partnership
  Statsig is being absorbed—its platform, brand, and customers folded into a broader offering—while continuing to power experimentation, feature management, and decision-making for AI-driven teams. As part of the arrangement, Amplitude will integrate and expand these capabilities across its platform.
  
  What our Slack community had to say about the news:

  • Elizabeth Gilbert: I haven’t seen a move like this before… Statsig was only with OpenAI for 8 months. There sure has been a lot of movement in the vendor space recently.

  • Johan: …the writing was on the wall ever since OpenAI acquired Statsig I suppose…This seems rushed…I cannot see a long-term future where Statsig exists as a standalone company given how much overlap there’s between the Statsig & Amplitude offerings.

  • Rommil: They compliment each other but there is a bit of overlap.

  • Elizabeth Gilbert: Great points. With the overlap, I wonder if Amplitude will merge their experimentation into Statsig, merge Statsig into Amplitude, or some other option. With how rushed it’s seeming, it’d be quite hard to be a Statsig customer right now.

    Ellie Hughes: You can’t have the amplitude exp tool without the analytics

    This opens them up to work with other warehousing and event collecting solutions?!? Eg GCP etc