Comparing two means with t and F tests

Empirical correlation and regression asks whether two measurement sequences move together. The second basic question is different: do the underlying random variables differ in a way that matters? This page checks the standard deviations first, then the means.

Both tests below assume the two series come from normally distributed random variables $X$ and $Y$, the case a probability-paper check is meant to clear. Write the series as $x_1,\dots ,x_{n_1}$ and $y_1,\dots ,y_{n_2}$, with empirical means and empirical standard deviations $\bar{x},\bar{y}$ and $\Delta x,\Delta y$.

Square the larger spread

Form the ratio of the larger empirical standard deviation to the smaller one, and square it:

$$F:=\{\begin{array}{ll}(\Delta x/\Delta y{)}^2& \text{if }\Delta x>\Delta y,\\ (\Delta y/\Delta x{)}^2& \text{if }\Delta x\le \Delta y.\end{array}$$

Look up the bold-faced table value $F_{0.01;m_1,m_2}$ with degrees of freedom $m_1=n_1-1$ and $m_2=n_2-1$.

• If $F>F_{0.01;m_1,m_2}$, the standard deviations are not the same; the difference has a deeper meaning.
• If $F\le F_{0.01;m_1,m_2}$, take the standard deviations as essentially the same.

Both statements carry an error probability of $0.02$: in about $100$ such situations, roughly two conclusions will be wrong.

Example. The medicine table below has $\Delta x=5$ and $\Delta y=4$, so $F=(5/4{)}^2=1.6$. For $m_1=m_2=14$ the table gives $F_{0.01;14,14}=3.7$. Because $1.6<3.7$, treat the two spreads as the same.

Drag $\Delta x$ and $\Delta y$. The blue point sits at $(F,0)$ on the axis; the gray mark is the critical value $3.7$.

A shared spread turns the mean gap into $t$

Now assume, in addition, that the two normal variables share one standard deviation (the F-test just checked that). Form

$$t=\frac{\bar{x}-\bar{y}}{\sqrt{(n_1-1)(\Delta x{)}^2+(n_2-1)(\Delta y{)}^2}}\sqrt{\frac{n_1{n}_2(n_1+n_2-2)}{n_1+n_2}}.$$

Choose a small error probability $\alpha$ and look up $t_{\alpha ,m}$ with $m=n_1+n_2-1$.

• If $|t|>t_{\alpha ,m}$, the means of $X$ and $Y$ differ significantly; the gap is not just random scatter.
• If $|t|<t_{\alpha ,m}$, the means do not differ significantly.

An error probability $\alpha$ means that in $100$ such tests, about $100\alpha$ conclusions will be wrong. Taking $\alpha =0.01$ therefore risks being wrong about once in a hundred times.

Example. Fifteen patients on medicine A and fifteen on medicine B give

(The means are days until recovery.) Substituting into the formula with the positive gap $\bar{y}-\bar{x}$ yields $t=3.6$. For $\alpha =0.01$ and $m=15+15-1=29$, the $t$-table gives $t_{\alpha ,m}=2.8$. Because $3.6>2.8$, the means differ significantly: medicine A is better than B.

Drag the mean gap and the two spreads. The blue point sits at $(t,0)$; the gray mark is the critical value $2.8$. Passing that mark to the right is the significant-difference case.

Drop normality and the t-test stops

The $t$-test applies only to normally distributed quantities. The Wilcoxon test is much more general: it can check whether two sequences come from variables with different distributions. A fuller account of that test comes later. The numerical $t$ and $F$ tables themselves belong to a later section.