You ran a test and saw something like this:
Revenue Per Visitor (RPV): +8% with 97% probability to beat original
Average Order Value (AOV): +2% with only 68% probability (or even “no winner”)
This happens all the time and is completely normal. Here’s why.
How the metrics are calculated
AOV = Total Revenue ÷ Number of Orders (only counts people who actually bought something)
RPV = Total Revenue ÷ Total Visitors (includes every visitor, even the 95–98% who bought nothing)
Because most visitors spend $0, the starting RPV number is very low (usually $2–$4). A small increase in total revenue creates a much larger percentage jump in RPV than in AOV.
Example
Same extra revenue, two very different percentage lifts:
Baseline | Variant | Uplift | |
Orders | 5,000 | 5,200 | |
AOV | $42 | $43.27 | +3.0% |
RPV | $2.10 | $2.33 | +11.0% |
That’s why RPV almost always reaches statistical significance faster and shows higher confidence than AOV, even when the actual business improvement is exactly the same.
What you should do
Trust RPV as your primary metric for deciding winners.
Companies like Amazon, Booking.com, and Shopify do exactly this for the same reasons:
It directly measures total revenue impact
It naturally includes both conversion rate and order value changes
It’s far more reliable and sensitive than AOV or conversion rate alone
Quick rule of thumb
RPV is clearly positive (95%+) → Ship the winner
RPV is up but AOV looks weak or flat → Still ship it (this pattern is expected)
Both RPV is flat → No real revenue impact
Bottom line: Strong RPV + weak-looking AOV is not a bug, it’s how the math works. Feel confident declaring the variant the winner.