Every test type, run with the rigor to trust what wins.
We run A/B, multivariate and autonomous bandit tests, server-side and at the edge, with the same statistical discipline applied to every result. Whether a person is watching the dashboard or a bandit algorithm is shifting traffic on its own.
Only about 1 in 7 tests produces a significant win. That's exactly why sample size, significance and governance come before the test starts, not after we like what we're seeing.
Every test type, and when each earns its place.
The right method depends on your traffic and the question you're answering, not on which tool we happen to like.
A/B
Two full versions, compared head to head. Best for big, clear changes on lower-traffic pages.
A/B/n & multivariate
Three or more versions, or several elements tested together to see how they interact. Best for higher-traffic pages.
Split-URL
Entirely separate page templates, compared directly. Best for structural redesigns.
Multi-armed bandits
Traffic shifts to the winning variant automatically, in real time. Best for high-traffic, always-on funnels.
Not every test wins. Every test tells you the truth.
A/B testing isn't a bet on being right. A win gets shipped, a loss saves you from building the wrong thing at scale, and a flat result tells you to stop arguing and test the next idea instead. Calling any of those results early is how you end up trusting the wrong one, which is why sample size and significance are set before a test starts, not after we like what we're seeing.
Sample size, set upfront
Every test is powered for the effect size we are actually looking for, and we never resize the target once the data starts looking good.
95% confidence, 80% power
Applied whether a human is watching or a bandit is deciding. No lower bar for autonomous tests just because nobody is watching live.
No peeking, ever
The hypothesis and success metric are locked before launch. A result is called when it is real, not the moment it looks good.
No black boxes, not for the tests a human calls and not for the ones a bandit calls. A real edge where rigor is the difference.
How long until your test reaches significance?
A rough estimate of the sample size and time a test like yours would need, using the same 95% confidence, 80% power standard we test to.
Bandits capture the upside during a test, not after it.
Two related but different capabilities, both running with the same governance as everything else here.
Multi-armed bandits
The algorithm watches performance live and shifts more traffic to the variant that's winning, instead of splitting evenly until a human calls the test.
Predictive personalization
Machine learning reads live signals, scroll speed, referral source, device and intent, and serves the highest-probability variant before a visitor bounces. Done well, that's a 10% to 25% lift on top of static testing.
Governance
We set the objectives and guardrails, and a person on our team stays accountable for what the system learns and ships. Autonomous doesn't mean unsupervised.
Whichever way you test, it shouldn't cost you the speed you're protecting.
Client-side testing scripts can slow the page and cause visible flicker, the flash of the original page before a variant swaps in, if the implementation isn't careful. That's worth watching, because it hits Core Web Vitals and conversion, the exact things a test is meant to improve.
Client-side testing
- Fast to launch, works with almost any stack, no dev dependency to get started
- Variant decision happens in the browser, in the moment the page loads
- Flicker risk is real, but manageable with careful implementation
- The right starting point for most testing programs, including most of ours today
Server-side and edge
- Variant decided at the network edge, before the page ever renders
- Zero extra script weight added to the page
- No flicker, by construction, not just careful implementation
- The right upgrade once traffic or SEO stakes justify the engineering investment
Most programs start client-side, and that's the right call for most of ours today. We move a program to server-side and edge when the traffic or the stakes make the engineering investment worth it, not because client-side is broken.
Testing isn't just how you optimize. It's how you avoid building the wrong thing.
A full feature build is expensive before it ever proves its worth. Testing a lightweight version first turns that bet into a decision you can actually defend.
Validate the concept, not the full feature
A lightweight version of a new flow, tested against real traffic, tells you if it's worth building at all, before a single sprint goes into it.
Feature flags, doing double duty
The same mechanism that ships a feature safely can run it as a test first, at a fraction of the cost of the full build.
A flat result still saves a budget
A test that shows no meaningful impact isn't a failed experiment. It's the cost of a full build you didn't have to spend.
The cost of testing first is a fraction of the cost of building blind, which is why this conversation usually pulls Product and Engineering into the room, not just Marketing.
Before you talk to us.
Do you replace our current testing tool, or work with it?
How much traffic do we actually need?
What stops a bandit test from calling noise a winner?
Will server-side testing slow our site down?
What's the difference between A/B and multivariate, in practice?
Experimentation runs on evidence from elsewhere in the funnel.
None of this happens in isolation from research or data.
Bring us your traffic. We'll bring the rigor.
Tell us where your testing program stands today and we'll tell you honestly what's realistic, what needs fixing first, and where autonomous testing would actually help.
Talk to the experimentation team →
