#188 | Synthetic Users Catch Bugs. Humans Drive Revenue.
Also: Adobe Target now provides an MCP server
Follow us: Slack | LinkedIn | YouTube | Spotify | Substack
In this issue
Community thread: Can AI Synthetic Users Replace Real A/B Testing?
Jobs: New job opportunities at Airbnb and Manulife
Vendor updates: The latest news about Adobe Target
Save 50% off our Best Selling book, “Prove It or Lose It”
Thanks to our paid members!
Community question of the week: Can AI Synthetic Users Replace Real A/B Testing?
Recently, Ginny Forshaw asked:
Synthetic users - what is everyone thinking of it? I want to understand why practitioners are choosing this option (apart from the low cost)? I am so late to this topic but I want to understand it all, good, bad, ugly. I have some initial (strong) opinions but it would be good to hear from those who have done it.
Experiment Nation’s Take
“That’s like certifying a flight simulator instead of the aircraft.”
Synthetic users may be useful to identify obvious UX or experiment setup problems before exposing them to real traffic.
But most practitioners still shouldn’t trust synthetic users for production-level decision making because human behavior is irrational, emotional, inconsistent, and difficult to simulate reliably.
The current synthetic user wave feels more like a nice add-on for experimentation and research teams, not a replacement for real-world testing.
What the community had to say:
Ritvik Sharma: I think it’s a really interesting space and clearly gaining traction. The idea of getting instant A/B test results from synthetic users that mimic a brand’s real audience is cool. That said, it still feels very early. Aligning synthetic users with real user behavior is a key challenge. I ran into this while building a prototype to A/B test creatives using multiple AI agents with persona-driven scoring and the results were inconsistent.
Sharing an interesting paper on using synthetic users for website journey testing as well, in case helpful: AB testing with LLM agents.pdfRommil Santiago: IMHO, the use case is more to catch problems prior to doing it for real. It shouldn’t be a replacement altogether. That’s like certifying a flight simulator instead of an aircraft.
Eddie Aguilar: Synthetic users should not be used to make decisions. Humans are far more capable of breaking things without realizing than bots are. Bots follow the program, so are more likely to help find issues on a linear path, while humans aren’t generally linear when researching, or moving from page to page. Synthetic users take instructions, real users do not. From the research paper above too:
“Agent A/B introduces a new phase for agent-based piloting in the design life-cycle that complements traditional A/B testing and expands the scope of early-stage UX evaluation”
Basically used to find experiment design issues quickly, rather than make decisions.
What you’re describing as to why you see more people promoting it, is because there is a lot of “AI Slop” out there that created these “synthetic” users vibe coding a product and pitching. @Paul Randall would have a few thoughts on this too ; )Shiva Manjunath: Ton Wessling gave basically Rommil’s POV. Its literally 0% risk to ‘try’ stuff out on it. But i would never make a production-level decision on synthetic users.
Ritvik Sharma: As the web becomes more agentic (still early days), if agents increasingly shortlist on behalf of users, do we shift from optimizing for human behavior to optimizing for agent selection? Could change what we A/B test quite a bit.
Shiva Manjunath: Maybe i’m in the minority, but i don’t think agentic will be remotely close to the majority of traffic for like a decade
Ritvik Sharma: I dunno how popular it is, but my GPT/Claude fetch info from the web quite often. And I’m assuming less and less people visit websites organically.
Ginny Forshaw: But then the whole point is HUMAN behaviour. Its weird, crazy, unpredictable. AI cannot replicate that, even if it tried. I read this article today and sums up some examples well, any ai tool will tell us what we want to hear. In psychology its called participant bias, bur in AI world, its just a “people pleaser” https://www.nngroup.com/articles/synthetic-users/
On the point about agentic access to the sites, I’ve seen traffic come through the GPT etc but its 2-3% at most. I don’t see it becoming the majorityEzequiel Boehler: @Ginny Forshaw First, I think most of what we see on Socials is just hype / ads etc and very little is actually being done with it.
Second, I find it interesting that its being pushed so hard, while still today a lot of experimentation programs do not do much Research in general. Its for sure not a widely adopted practice yet, but somehow it wants to already be replaced with a synthetic version of it? How does a team that never done any normal (actual human) research knows the actual value they are getting from a Synthetic one they cant really validate on market? How do they know what are the trade offs against normal research? The common pitfalls?
In other words, why would they be so quick to look for the value of Synthetic users without never actually measuring the value of non synthetic one?
And third, and this one is more abstract but, a lot of research is used for measuring and hopefully understanding the Why of certain actions. Why did this experience not delivered what we though? Why might users prefer this over that? How a synthetic system that never actually experienced anything, be helpful in that area?
Join the conversation on our Slack
If you’re looking for a place where you can connect with other CROs either to talk shop, or simply shoot the sh*t - consider signing up to join our Slack.
Our Slack is a place to:
Get answers in a judgement-free zone
Share learnings
Talk shop with other CROs
You can join by signing via this form and we will send you an invite within 48 hours.
Jobs of the Week
As a THANK YOU for being an active subscriber of Experiment Nation’s newsletter, you get access to Optimization Jobs, a CRO-focused job board with 2000+ open roles from the last 30 days alone.
From Optimization Jobs by Experiment Nation
Find Optimization, Growth, and CRO jobs that other boards missed. With new jobs posted every hour - you will always find something new.
CRO Specialist_JR105989 1
📍 TVHGrowth Marketing Manager (Affiliate and Influencer Marketing) - France
📍 N26 – ParisHead of Growth Marketing, Asia
📍 Airbnb – SingaporeHead of Growth Marketing Commercialisation
📍 ManulifePrincipal Analyst, Growth Marketing
📍 Upstart – Remote
To see more jobs, visit the Optimization Jobs board and explore 3000+ jobs from 400 companies from just the past 30 days that other boards may have missed.
News from Vendorland
Adobe Target MCP server (Public Beta)
”Adobe Target now provides an MCP (Model Context Protocol) server that surfaces experimentation, personalization, and reporting operations directly inside any MCP-compatible application. With this integration, marketing and technical personas can inspect A/B tests, analyze performance reports, and explore audiences and offers — all using natural-language prompts instead of navigating multiple UI screens or writing queries against the Adobe Target REST API. This capability is currently available in Claude Web, Claude Desktop, Claude Code, Cursor, and ChatGPT. This capability is available to all customers in Public Beta.”
Get your copy of Prove It or Lose It (PDF Format) for $10 USD
The (mostly) no-nonsense guide to surviving experimentation program drama
Are you ready to prove the value of your Experimentation Program? In this actionable guide, Rommil Santiago distills over 15 years of experience and shares all you need to know to survive managing an Experimentation Program. Inside, you’ll learn:
The GROW framework for communicating value effectively
Strategies to win over stakeholders
How to build a brand for your program that lasts
Prove it or Lose it is laser-focused on delivering only the actionable advice you need. Your Experimentation Program is only as strong as the value you can demonstrate.
Are you ready to Prove it?