Statistical Inference

Hypothesis Testing

Neyman--Pearson Lemma

The cleanest optimality result in hypothesis testing is for simple vs. simple tests, where both hypotheses fully determine the distribution.

Definition. A test is simple vs. simple if: \[ H_0:\theta=\theta_0 \qquad H_1:\theta=\theta_1. \]

Equivalently, the null gives one distribution $p_{\theta_0}$ and the alternative gives one distribution $p_{\theta_1}$.

Theorem. For a simple vs. simple test, the Neyman--Pearson lemma says the most powerful level-$\alpha$ test rejects for large likelihood ratio: \[ \Lambda(x)=\frac{p_{\theta_1}(x)}{p_{\theta_0}(x)}. \]

The rejection region has form:

\[ \Lambda(x)>c, \]

where $c$ is chosen so that:

\[ P_{\theta_0}(\Lambda(X)>c)\le \alpha. \]

NOTE Neyman--Pearson maximizes power against the fixed alternative $\theta_1$ while controlling type I error under the fixed null $\theta_0$.

Composite Hypotheses

Most useful tests are not simple vs. simple. The null or alternative usually contains many possible distributions.

Definition. A test is composite if: \[ H_0:\theta\in\Theta_0 \qquad H_1:\theta\in\Theta_1. \]

The null family is $\{p_\theta:\theta\in\Theta_0\}$ and the alternative family is $\{p_\theta:\theta\in\Theta_1\}$.

Definition. A test has level $\alpha$ if it controls the worst-case null rejection probability:

\[ \sup_{\theta\in\Theta_0} P_\theta(\text{reject }H_0) \le \alpha. \]

Definition. The power function is: \[ \beta(\theta)=P_\theta(\text{reject }H_0), \qquad \theta\in\Theta_1. \]

A test is UMP if it has maximal power for every $\theta\in\Theta_1$ among all level-$\alpha$ tests.

NOTE Neyman--Pearson does not directly apply because there is no single null distribution and no single alternative distribution.

UMP tests often do not exist for composite hypotheses. A test can be best for one alternative but worse for another. To recover optimality, we usually restrict the problem using extra structure.

NOTE Common sources of structure: monotone likelihood ratio, invariance, unbiasedness, or asymptotics.

Gaussian Mean Tests

For Gaussian mean testing, the nuisance parameters are the unknown common mean and possibly the unknown variance.

Definition. For two independent Gaussian samples: \[ X_1,\dots,X_n\sim N(\mu_X,\sigma^2), \qquad Y_1,\dots,Y_m\sim N(\mu_Y,\sigma^2), \]

the one-sided two-sample mean test is:

\[ H_0:\mu_X=\mu_Y \qquad H_1:\mu_X>\mu_Y. \]

Equivalently:

\[ H_0:\Delta=0 \qquad H_1:\Delta>0, \qquad \Delta=\mu_X-\mu_Y. \]

If $\sigma$ is known, the nuisance variance disappears by standardizing the difference in sample means.

Proposition. If $\sigma$ is known, define: \[ Z= \frac{\bar X-\bar Y}{\sigma\sqrt{1/n+1/m}}. \]

Under $H_0$:

\[ Z\sim N(0,1). \]

Reject at level $\alpha$ when:

\[ Z>z_{1-\alpha}. \]

The one-sided p-value is:

\[ p=1-\Phi(Z_{\text{obs}}). \]

Pooled and Welch t-Tests

If the variance is unknown but shared across both groups, we estimate it using the pooled sample variance.

Proposition. If: \[ \sigma_X^2=\sigma_Y^2=\sigma^2, \]

use:

\[ S_p^2= \frac{(n-1)S_X^2+(m-1)S_Y^2}{n+m-2}. \]

Then:

\[ T= \frac{\bar X-\bar Y}{S_p\sqrt{1/n+1/m}}. \]

Under $H_0$:

\[ T\sim t_{n+m-2}. \]

Reject at level $\alpha$ when:

\[ T>t_{n+m-2,1-\alpha}. \]

Definition. A statistic is pivotal if its null distribution does not depend on unknown parameters.
NOTE The pooled t-statistic is pivotal: under $H_0$, its distribution does not depend on the unknown common mean or unknown variance.

The pooled t-test is not directly a Neyman--Pearson test. Its finite-sample optimality comes from restricting to tests that respect the natural symmetries of the Gaussian problem.

Theorem. The one-sided pooled t-test is UMP invariant under common location-scale transformations:

\[ (X_i,Y_j)\mapsto a(X_i,Y_j)+b, \qquad a>0. \]

NOTE This means the pooled t-test is optimal among tests whose decision does not change after shifting or rescaling all observations.

If the variances are unequal, the exact finite-sample problem becomes harder.

Proposition. If: \[ X_i\sim N(\mu_X,\sigma_X^2), \qquad Y_j\sim N(\mu_Y,\sigma_Y^2), \]

use Welch's statistic:

\[ T= \frac{\bar X-\bar Y}{\sqrt{S_X^2/n+S_Y^2/m}}. \]

Approximate degrees of freedom:

\[ \nu\approx \frac{(S_X^2/n+S_Y^2/m)^2} {\frac{(S_X^2/n)^2}{n-1}+\frac{(S_Y^2/m)^2}{m-1}}. \]

Then approximately:

\[ T\sim t_\nu. \]

NOTE Welch's test is asymptotically valid and robust to unequal variances, but it is not an exact finite-sample UMP test. This is the Behrens--Fisher problem.

Confidence Intervals

Bernoulli Confidence Intervals

A confidence interval is a random set $C(X)$ designed to contain the true parameter with high probability.

Definition. For Bernoulli data: \[ X\sim \mathrm{Bin}(n,p), \qquad \hat p=\frac{X}{n}. \]

A confidence interval $C(X)$ has coverage $1-\alpha$ if:

\[ P_p(p\in C(X))\ge 1-\alpha \qquad \forall p\in[0,1]. \]

The simplest interval is the Wald interval, but it only has asymptotic justification.

Proposition. The Wald interval is: \[ \hat p \pm z_{\alpha/2} \sqrt{\frac{\hat p(1-\hat p)}{n}}. \]

It satisfies approximately:

\[ P_p(p\in C(X))\approx 1-\alpha \qquad n\to\infty. \]

NOTE Wald can undercover because it plugs in $\hat p$, uses a normal approximation, and ignores binomial discreteness. It is especially bad near $p=0$ or $p=1$.

Clopper--Pearson Interval

Clopper--Pearson fixes the finite-sample coverage issue by inverting exact binomial tests.

Definition. The Clopper--Pearson interval is the exact binomial confidence interval obtained from all $p$ values not rejected by an exact binomial test.
Theorem. Given $X=x$, the two-sided $1-\alpha$ Clopper--Pearson interval has endpoints: \[ L= \begin{cases} 0, & x=0,\\ \mathrm{Beta}^{-1}(\alpha/2;x,n-x+1), & x>0, \end{cases} \] \[ U= \begin{cases} 1, & x = n,\\ \mathrm{Beta}^{-1}(1-\alpha / 2; x+1,n-x), & x< n \end{cases} \]

It satisfies:

\[ P_p(L(X)\le p\le U(X))\ge 1-\alpha \qquad \forall p\in[0,1]. \]

NOTE Clopper--Pearson has guaranteed finite-sample coverage. It is usually conservative because binomial discreteness prevents the coverage from landing exactly at $1-\alpha$.