The Hidden Pitfalls of A/B Testing: Six Key Mistakes to Avoid
With feedback from industry experts and analysis from "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing."
In high-stakes digital product development, A/B testing has been the gold standard for data-driven decision-making. But after seven years of observing dozens of product teams, it's clear that even the best, most forward-thinking organizations inevitably fall into the same traps and undermine the very conclusions they're striving to obtain.
-
The Impatience Problem: Peeking Too Early
At testing-disciplined Booking.com, a painful experience awaits former product director Lukas Vermeer to relate. "We once had an executive who was so delighted with early positive outcomes that he wanted to ship a new price display after just four days of testing," Vermeer said. "Six weeks down the line, we found that conversion had actually dropped by 3% across our European markets. That cost us millions."
This is a routine affair across the tech universe. Groups plan four-week experiments but break protocol when early numbers are heartening—or sobering.
"It's not an experiment if you don't show respect for statistical power," writes Ronny Kohavi, Airbnb's previous VP of Analysis & Experimentation and author of "Trustworthy Online Controlled Experiments." "Either hold to your intended sample size or use proper sequential testing methods with appropriate alpha spending functions." [Experiment Guide]
-
The Statistical Fishing Expedition: Chasing Too Many Metric
When Pinterest reworked its home feed algorithm in 2021, the team initially celebrated what seemed like a massive win: users interacted more with pins that were recommended to them, up 12%. But they had monitored 23 metrics.
"We ultimately fell into the trap of multiple comparisons," admitted one-time Pinterest data scientist Sarah Chen. "Once we applied good Bonferroni corrections and focused on our north star metric of weekly active pinners, we saw that the change truly was neutral at best."
Seasoned practitioners advise a single-stated strategy: a single metric upon which your strategy goal focuses, supported by a small group of guardrail metrics to ensure you're not destroying something important.
-
The Averaging Illusion: Segment Impact Forgotten
Spotify's 2019 redesign of its mobile player tallied a small 2% gain in listening time in total—but when product manager Gustav Söderström examined it more closely, he discovered new listeners were actually spending 15% less time on the platform.
"The total count covered up the harm we were inflicting on our acquisition pipeline," Söderström wrote in a case study last year. "Veteran users loved the shift, but new customers found it entirely bewildering." [Source: Podcast]
Segmenting outcomes by user cohorts—new versus repeat, mobile versus desktop, free versus premium—usually identifies key information hidden by overall averages.
-
The Moving Target: No Clear Success Criteria
When Microsoft's Edge browser team tested a new feature for tab management, they initially declared it a success because "users seemed to like it." There wasn't any measurable negative impact on key metrics, so they shipped it.
"Three months in, we'd discovered that our technical debt had increased and our iteration speed had slowed as a result of the complexity this feature added up," recalled program manager Jessica Lin at a recent product conference. "We'd never actually laid out what success even was beyond 'not breaking anything.'" [Source: Microsoft Article]
Having clear success criteria established before shipping any test—what metric to change, by how much, and over what period—prevents this post hoc rationalization.
-
The Tunnel Vision Danger: Focusing On One Metric
Uber's infamous surge pricing optimization algorithm is a cautionary tale. In 2017, it ran tests that were effective at increasing ride completion rates by 7%—a clear win by its primary metric. What they found out too late was that customer satisfaction scores crashed through the floor, and repeat usage imploded over the next few months.
"We maximized for exactly what we achieved," said ex-product lead Daniel Graf. "But we had not carefully considered the set of metrics that actually defined our business health."[Source: Article by Rice University]
Every experiment needs a system of measures: a primary measure to optimize against, as well as guardrail measures that protect key aspects of the user experience and business model.
-
The Paralysis Problem: Overfearing Experiment Interaction
Netflix runs hundreds of experiments all at once on its website at any moment, a practice that helped them fine-tune everything from thumbnail selections to recommendation algorithms.
"We lost nearly a year in the early days because we were afraid of the effects of interaction between experiments," confessed Todd Yellin, Netflix VP of Product. "After we built appropriate randomization systems and knew when experiments actually posed interference, we increased our learning velocity by a factor of ten."[Source: Article]
Except where tests directly modify the same user interface elements or compete for the same limited user attention, interaction effects tend to be low. Slowing down experimentation is often too expensive compared to the risk of occasional interactions.
The Real Goal: Learning Velocity
The best product companies don't measure success by how many tests they run but by how fast they achieve valid conclusions. Appropriately used, A/B testing creates a flywheel of learning that grows exponentially.
"Each test should have a particular question," Kohavi recommends. "But above all, it should generate three new questions. That's how you build institutional knowledge."
As digital products become more and more complex, sidestepping these all-too-familiar testing fallacies not only improves short-term outcomes but also redefines how organizations discover and evolve in an ever-changing digital universe.