This page names the statistical methods behind every Mida report, so you know exactly what you are looking at. You do not need any of it to run a test. It is here for when your analyst, your client or your procurement team asks.
Two analysis methods
Every test uses one of two methods. You choose it in Configuration → Analysis Method.
Bayesian (default). Gives you a plain probability: how likely the variant is to beat the original.
Frequentist. Gives you a p-value, for teams that report that way.
Bayesian (default)
For conversion goals, Mida uses a Beta-Binomial model. Each variant's conversion rate is estimated from its visitors and conversions, and the report shows three numbers:
Chance to Beat Original: the probability that the variant converts better than the original. Tests with more than one variant also show Chance to be Best.
Credible interval: the range the variant's true lift most likely falls in.
Risk: the expected loss. If you ship this variant and it turns out to be the worse one, this is how much you would give up on average. Lower is better.
Confidence and Risk answer different questions, so read them together. Risk vs Confidence explains why.
Frequentist
For conversion goals, Mida uses a two-proportion z-test. You can pick one-tailed (is the variant better?) or two-tailed (is it different in either direction?). The report shows the p-value against the confidence level you set.
When Mida recommends a winner
A variant being ahead is not enough. On Bayesian tests, the report only recommends a winner when all of these hold:
At least 7 days of data. That covers a full week, since weekdays and weekends often behave differently.
A higher bar in the first two weeks. Early results swing the most, so until day 14 Mida asks for more confidence than the level you set. The report shows the bar that applies today.
Low Risk. High confidence is not enough if the possible downside is still large.
A correction for multiple variants. Every extra variant is another chance of a lucky result, so with more than one variant the confidence needed goes up with the number of variants. This is a Bonferroni-style correction.
Mida also checks for the opposite case. If the difference between variants is very likely too small to matter, the report tells you there is no meaningful difference instead of calling a winner on a tiny gap. This is a ROPE (region of practical equivalence) check.
Data quality checks
Mida checks every test for sample ratio mismatch (SRM): whether the traffic each variant received matches the split you set. If it does not, the report warns you, because an uneven split usually means something in the setup is off. See Why isn't my A/B test result 50-50?
Multi-armed bandit
When you choose Automatically give more traffic to better-performing variants, Mida uses Thompson sampling. Traffic leans toward the variants most likely to be best, while the others keep getting some visitors so Mida keeps learning. See How to set up a Multi-Armed Bandit test.
Your settings
You choose the confidence level for each test. See How to configure test confidence and Interpreting test results in Mida report.