Severity Calculator

What is severity?

Severity is a post-data measure from the error-statistical school of Deborah Mayo and Aris Spanos. It answers a pointed question: given the data you actually observed, how well did a particular claim about the true effect survive a test that would probably have caught it if it were wrong?

The guiding idea is that a claim is warranted only to the extent that it passed a severe test, one that would very probably have turned out differently if the claim were false. Severity attaches a number between 0 and 1 to a specific inference, so you can compare how well different claims are supported by the same data.

That makes it different from its neighbours. A p-value speaks only about a null hypothesis, a confidence interval lists many values without ranking them, and power is calculated before the data exist. Severity is evaluated after the test and for the exact value you care about.

How to use the calculator

1. Choose whether the groups differ in proportions (a conversion or survival rate) or in means (a continuous measurement).
2. Pick the statistic: Z-test, or T-test when each group has fewer than about 30 observations.
3. Enter alpha, the significance threshold of the original test, such as 0.05.
4. Supply the data. For proportions give each group's size and its proportion, percentage rate or number of events. For means give raw observations or the size, mean and standard deviation of each group.
5. In Inference for, type the value μ1 you want to make a claim about, on the same scale as group B's proportion or mean, and press Calculate SEV.

Group A is the control or baseline and group B the treatment. The test is one-sided in the direction "B is larger than A". When the outcome is a percentage rate, type the inference value as a percentage too; for event counts, enter the number of events in group B.

Besides the severity value (and the same figure as a percent) the table shows the one-sided p-value, the Z or T score (with degrees of freedom for T), the significance level, which is one minus the p-value shown as a percent, and the observed difference B − A. For means it also lists the pooled standard error, and for raw data the sample sizes and means.

The severity criterion

Let SE be the standard error of the difference, d = B − A the observed difference and p the one-sided p-value. For proportions SE uses the pooled proportion p̄; for means it uses the pooled variance:

SE = √( p̄ (1 − p̄) (1/n_A + 1/n_B) ) or √( s²_pooled (1/n_A + 1/n_B) )

z = d ÷ SE, p = P(Z > z)

If p is below alpha the test rejected the null, and the claim that can be probed is that the true value is greater than μ1. Otherwise the claim probed is that it is at most μ1:

reject: SEV(μ > μ1) = Φ( (B − μ1) ÷ SE )

no rejection: SEV(μ ≤ μ1) = 1 − Φ( (B − μ1) ÷ SE )B is the observed value in group B. With the T-test, Φ is replaced by the Student t distribution with n_A + n_B − 2 degrees of freedom.

A useful check: when the test does not reject and μ1 equals the value of group A, SEV(μ ≤ μ1) is identical to the p-value.

How to interpret severity

The higher the severity, the better the claim has been probed by the data. A value of 0.5 means the data say nothing for or against the claim, and values below 0.5 mean the claim is poorly supported, because a result like yours would have been quite likely even if the claim were false. There is no official cut-off, so treat this informal reading as a starting point:

Severity changes with the claim, and that is the point. The same data can warrant "the effect is above zero" very well and "the effect is above the planned minimum" poorly. Reading a whole series of values shows which effect sizes the experiment has really excluded.

Worked example

An A/B test shows 2,500 visitors to each version. Conversion is 10% for the control (A) and 12% for the variant (B), and alpha is 0.05. The pooled proportion is 0.11, so SE = √(0.11 × 0.89 × 2 ÷ 2,500) = 0.00885, the z-score is 0.02 ÷ 0.00885 = 2.26 and the one-sided p-value is 0.012. The test rejects the null, so we examine claims of the form μ > μ1.

The claim that the variant converts above 10.5% is strongly supported, the claim of more than 11% much less so, and the claim of more than 12.5% fails because the observed 12% is itself below that level.

Now suppose the variant had only reached 11%. Then z = 1.153 and p = 0.124, which does not reject. The claim μ ≤ 10% has severity 0.124 (equal to the p-value, as noted above), while the claim μ ≤ 12% has severity 0.876.

For means, take 40 users per group with means 50 and 54 and a standard deviation of 10 in each, using the T-test. The pooled standard error is 2.236, t = 1.789 with 78 degrees of freedom and p = 0.039, so the claim μ > 52 has severity 0.813 and μ > 50 has severity 0.961.

Assumptions and notes

• The two groups are independent, and the pooled standard error assumes similar variability in both groups.
• Alpha is read exactly as typed, so enter 0.05, not 5.
• The test is one-sided, in the direction B greater than A. If B is below A the p-value is large and the claim examined is of the form μ ≤ μ1.
• Severity complements, rather than replaces, a confidence interval and an effect size estimate. For the underlying significance test see the p-value calculator.