Sample Ratio Mismatch Calculator

What is a sample ratio mismatch?

In a randomised experiment you decide in advance how visitors are divided among the variants, for example 50% to the control and 50% to the treatment. A sample ratio mismatch (SRM) means the split you actually observe differs from that plan by more than random assignment can reasonably explain.

A mismatch is a warning about the experiment, not about the variants. Something in the way users were assigned, delivered the experience, or were counted is biased, so the comparison between groups is no longer a fair one. Treat any metric result from an experiment with an SRM as unreliable until you have found the cause.

The gap does not have to look dramatic. Imagine a 50/50 test with 10,000 users in which 4,850 land in one group and 5,150 in the other. That is only a 3-point difference, yet the chance of seeing a split at least that lopsided from a fair coin is about 0.27%, which is low enough to be flagged.

How to use the calculator

1. Under Sample sizes, enter how many units each group received, control included. Spaces and line breaks both work as separators, so you can paste a spreadsheet column directly; thousands commas are dropped automatically.
2. When the plan was an even split across all groups, keep Target allocation set to Equal.
3. For an uneven plan, switch to Unequal and enter each group's planned share under Target ratios, keeping the groups in the order you used for the sample sizes. Write the shares as fractions (0.5 0.25 0.25) or percentages (50% 25% 25%); they have to total 1.
4. Press Check for SRM.

The result has two lines: a verdict and the p-value. The verdict flags a mismatch when the p-value is below 0.01 and reports no mismatch otherwise. The p-value is printed so you can apply a threshold of your own.

Count the same unit you randomise on. If assignment is per user, enter users, not sessions or page views. At least two groups are required, and if the number of ratios and groups differ the missing side is treated as zero, which usually produces an error or a flagged mismatch.

How the check works

The check is a Pearson chi-square goodness-of-fit test. For N units in total and a planned share r for a group, the expected count is N × r. The statistic adds up how far each group is from its expectation, relative to that expectation:

χ² = Σ (observed − expected)² ÷ expected

With k groups the statistic is compared with a chi-square distribution with k − 1 degrees of freedom, and the p-value is the probability of a statistic at least that large if assignment really followed the plan. A tiny p-value means the observed split is very unlikely under the plan.

The 0.01 cut-off is stricter than the usual 0.05. SRM checks run on every experiment, so a looser threshold would raise false alarms constantly, and real assignment bugs normally produce extremely small p-values anyway. These are the chi-square values that correspond to p = 0.01:

Worked examples

Two equal groups. A test sends 20,000 users to A and 20,400 to B. The total is 40,400, so each group should hold 20,200. Each group is 200 away, giving χ² = 200² ÷ 20,200 + 200² ÷ 20,200 = 3.96 with 1 degree of freedom. The p-value is 0.0466: below 0.05 but above 0.01, so the check passes. This is exactly the situation where the stricter threshold matters.

Three unequal groups. A test is planned 50% / 25% / 25% and records 5,100, 2,400 and 2,500 users. Expected counts are 5,000, 2,500 and 2,500, so χ² = 100² ÷ 5,000 + 100² ÷ 2,500 + 0 = 2 + 4 = 6 with 2 degrees of freedom. For two degrees of freedom the p-value is e^(−χ²/2) = e^(−3) = 0.0498, so again no SRM is flagged at 0.01.

A clear mismatch. With 50,000 users in the control and 51,100 in the treatment of a 50/50 test, each group is 550 from the expected 50,550. χ² is 11.97 and the p-value is about 0.00054, so the experiment is flagged.

What to do if SRM is detected

Do not report the result, and do not try to fix the numbers by re-weighting the groups. The usual path is to find the cause, correct it and run the test again. Common sources of bias fall into a few families:

• Assignment: a bug in the randomiser or bucketing, hashing that is not independent of another experiment, an uneven ramp-up, or users being forced into a variant.
• Delivery: one variant crashes, loads slowly or redirects, so some users leave before they are counted.
• Tracking: events from one variant are dropped, blocked or logged late, or bots are filtered differently.
• Data processing: exclusion rules, deduplication or joins applied after the fact that remove more units from one group than another.

Slice the counts by day, platform, browser, country and traffic source and look for the slice where the ratio breaks. A mismatch confined to one browser or one date range points straight at its cause.

Limits of the check

• A pass does not prove the test is unbiased. A small bias may exist that the chi-square test lacks the power to detect.
• Very large samples make the test sensitive to small deviations. That is intended, because even a slight but real imbalance is worth investigating.
• The chi-square approximation works best when every expected count is at least about 5.
• Re-running the check many times while data accumulate raises the chance of a false alarm. Run it on the final data, or treat early flags as hints.

Once a test passes the SRM check, you can go on to compare outcomes with a significance test or build an interval around the effect. Planning a future test? The sample size calculator tells you how many users you need.