Hypothesis Testing¶
The subset of all posable hypotheses that have the property of falsifiability constitute the space of scientific hypotheses.
Roughly, a falsifiable statistical hypothesis is one for which a statistical experiment can be designed to produce data or empirical observations that an experimenter can use to falsify or reject it.
In the statistical decision problem of hypothesis testing, we are interested in empirically falsifying a scientific hypothesis, i.e. we attempt to reject a hypothesis on the basis of empirical observations or data.
Thus, hypothesis testing has its roots in the philosophy of science and is based on Karl Popper's falsifiability criterion for demarcating scientific hypotheses from the set of all posable hypotheses.
Introduction¶
Usually, the hypothesis we attempt to reject or falsify is called the null hypothesis or $H_0$ and its complement is called the alternative hypothesis or $H_1$.
For example, consider the following two hypotheses:
- $H_0$: The average waiting time at an Orbiter bus stop is less than or equal to $10$ minutes.
- $H_1$: The average waiting time at an Orbiter bus stop is more than $10$ minutes.
If the sample mean $\overline{x}_n$ is much larger than $10$ minutes then we may be inclined to reject the null hypothesis that the average waiting time is less than or equal to $10$ minutes.
Suppose we are interested in the following slightly different hypothesis test for the Orbiter bus stop problem:
- $H_0$: The average waiting time at an Orbiter bus stop is equal to $10$ minutes.
- $H_1$: The average waiting time at an Orbiter bus stop is not $10$ minutes.
Once again we can use the sample mean as the test statistic, but this time we may be inclined to reject the null hypothesis if the sample mean $\overline{x}_n$ is much larger than or much smaller than $10$ minutes.
The procedure for rejecting such a null hypothesis is called the Wald test we are about to see.
More generally, suppose we have the following parametric experiment based on $n$ IID trials:
$$
X_1,X_2,\ldots,X_n \overset{IID}{\sim} F(x_1;\theta^*), \quad \text{ with an unknown (and fixed) } \theta^* \in \mathbf{\Theta} \ .
$$
Let us partition the parameter space $\mathbf{\Theta}$ into $\mathbf{\Theta}_0$, the null parameter space, and $\mathbf{\Theta}_1$, the alternative parameter space, i.e.,
$$\mathbf{\Theta}_0 \cup \mathbf{\Theta}_1 = \mathbf{\Theta}, \qquad \text{and} \qquad \mathbf{\Theta}_0 \cap \mathbf{\Theta}_1 = \emptyset \ .$$
Then, we can formalise testing the null hypothesis versus the alternative as follows:
$$
H_0 : \theta^* \in \mathbf{\Theta}_0 \qquad \text{versus} \qquad H_1 : \theta^* \subset \mathbf{\Theta}_1 \ .
$$
The basic idea involves finding an appropriate rejection region $\mathbb{X}_R$ within the data space $\mathbb{X}$ and rejecting $H_0$ if the observed data $x:=(x_1,x_2,\ldots,x_n)$ falls inside the rejection region $\mathbb{X}_R$,
$$
\text{If $x:=(x_1,x_2,\ldots,x_n) \in \mathbb{X}_R \subset \mathbb{X}$, then reject $H_0$, else do not reject $H_0$.}
$$
Typically, the rejection region $\mathbb{X}_R$ is of the form:
$$
\mathbb{X}_R := \{ x:=(x_1,x_2,\ldots,x_n) : T(x) > c \}
$$
where, $T$ is the test statistic and $c$ is the critical value. Thus, the problem of finding $\mathbb{X}_R$ boils down to that of finding $T$ and $c$ that are appropriate. Once the rejection region is defined, the possible outcomes of a hypothesis test are summarised in the following table.
The outcomes of a hypothesis test, in general, are:
| 'true state of nature' |
Do not reject $H_0$
|
Reject $H_0$
|
|
$H_0$ is true
|
OK
|
Type I error
|
|
$H_0$ is false
|
Type II error |
OK |
So, intuitively speaking, we want a small probability that we reject $H_0$ when $H_0$ is true (minimise Type I error). Similarly, we want to minimise the probability that we fail to reject $H_0$ when $H_0$ is false (type II error). Let us formally see how to achieve these goals.
Power, Size and Level of a Test¶
Power Function¶
The power function of a test with rejection region $\mathbb{X}_R$ is
$$
\boxed{
\beta(\theta) := P_{\theta}(x \in \mathbb{X}_R)
}
$$
So $\beta(\theta)$ is the power of the test if the data were generated under the parameter value $\theta$, i.e. the probability that the observed data $x$, sampled from the distribution specified by $\theta$, falls in the rejection region $\mathbb{X}_R$ and thereby leads to a rejection of the null hypothesis.
Size of a test¶
The $\mathsf{size}$ of a test with rejection region $\mathbb{X}_R$ is the supreme power under the null hypothesis, i.e.~the supreme probability of rejecting the null hypothesis when the null hypothesis is true:
$$
\boxed{
\mathsf{size} := \sup_{\theta \in \mathbf{\Theta}_0} \beta(\theta) := \sup_{\theta \in \mathbf{\Theta}_0} P{\theta}(x \in \mathbb{X}_R) \ .
}
$$
The $\mathsf{size}$ of a test is often denoted by $\alpha$. A test is said to have $\mathsf{level}$ $\alpha$ if its $\mathsf{size}$ is less than or equal to $\alpha$.
Wald test¶
The Wald test is based on a direct relationship between the $1-\alpha$ confidence interval and a $\mathsf{size}$ $\alpha$ test. It can be used for testing simple hypotheses involving a scalar parameter.
Definition¶
Let $\widehat{\Theta}_n$ be an asymptotically normal estimator of the fixed and possibly unknown parameter $\theta^* \in \mathbf{\Theta} \subset \mathbb{X}$ in the parametric IID experiment:
$$
X_1,X_2,\ldots,X_n \overset{IID}{\sim} F(x_1;\theta^*) \enspace .
$$
Consider testing:
$$
\boxed{H_0: \theta^* = \theta_0 \qquad \text{versus} \qquad H_1: \theta^* \neq \theta_0 \enspace .}
$$
Suppose that the null hypothesis is true and the estimator $\widehat{\Theta}_n$ of $\theta^*=\theta_0$ is asymptotically normal:
$$
\boxed{
\theta^*=\theta_0, \qquad \frac{\widehat{\Theta}_n - \theta_0}{\widehat{\mathsf{se}}_n} \overset{d}{\to} Normal(0,1) \enspace .}
$$
Then, the Wald test based on the test statistic $W$ is:
$$
\boxed{
\text{Reject $H_0$ when $|W|>z_{\alpha/2}$, where $W:=W((X_1,\ldots,X_n))=\frac{\widehat{\Theta}_n ((X_1,\ldots,X_n)) - \theta_0}{\widehat{\mathsf{se}}_n}$.}
}
$$
The rejection region for the Wald test is:
$$
\boxed{
\mathbb{X}_R = \{ x:=(x_1,\ldots,x_n) : |W (x_1,\ldots,x_n) | > z_{\alpha/2} \} \enspace .
}
$$
Asymptotic $\mathsf{size}$ of a Wald test¶
As the sample size $n$ approaches infinity, the $\mathsf{size}$ of the Wald test approaches $\alpha$ :
$$
\boxed{
\mathsf{size} = P_{\theta_0} \left( |W| > z_{\alpha/2} \right) \to \alpha \enspace .}
$$
Proof: Let $Z \sim Normal(0,1)$. The $\mathsf{size}$ of the Wald test, i.e.~the supreme power under $H_0$ is:
$$
\begin{aligned}
\mathsf{size}
& := \sup_{\theta \in \mathbf{\Theta}_0} \beta(\theta) := \sup_{\theta \in \{\theta_0\}} P_{\theta}(x \in \mathbb{X}_R) = P_{\theta_0}(x \in \mathbb{X}_R) \\
& = P_{\theta_0} \left( |W| > z_{\alpha/2} \right) = P_{\theta_0} \left( \frac{|\widehat{\theta}_n - \theta_0|}{\widehat{\mathsf{se}}_n} > z_{\alpha/2} \right) \\
& \to P \left( |Z| > z_{\alpha/2} \right)\\
& = \alpha \enspace .
\end{aligned}
$$
Next, let us look at the power of the Wald test when the null hypothesis is false.
Asymptotic power of a Wald test¶
Suppose $\theta^* \neq \theta_0$. The power $\beta(\theta^*)$, which is the probability of correctly rejecting the null hypothesis, is approximately equal to:
$$
\boxed{
\Phi \left( \frac{\theta_0-\theta^*}{\widehat{\mathsf{se}}_n} - z_{\alpha/2} \right) +
\left( 1- \Phi \left( \frac{\theta_0-\theta^*}{\widehat{\mathsf{se}}_n} + z_{\alpha/2} \right) \right) \enspace ,
}
$$
where, $\Phi$ is the DF of $Normal(0,1)$ RV. Since ${\widehat{\mathsf{se}}_n} \to 0$ as $n \to 0$ the power increase with sample $\mathsf{size}$ $n$. Also, the power increases when $|\theta_0-\theta^*|$ is large.
Now, let us make the connection between the $\mathsf{size}$ $\alpha$ Wald test and the $1-\alpha$ confidence interval explicit.
The $\mathsf{size}$ Wald test¶
The $\mathsf{size}$ $\alpha$ Wald test rejects:
$$
\boxed{
\text{ $H_0: \theta^*=\theta_0$ versus $H_1: \theta^* \neq \theta_0$ if and only if $\theta_0 \notin C_n := (\widehat{\theta}_n-{\widehat{\mathsf{se}}_n} z_{\alpha/2}, \widehat{\theta}_n+{\widehat{\mathsf{se}}_n} z_{\alpha/2})$.
}}
$$$$\boxed{\text{Therefore, testing the hypothesis is equivalent to verifying whether the null value $\theta_0$ is in the confidence interval.}}$$
Example: Wald test for the mean waiting times at our Orbiter bus-stop¶
Let us use the Wald test to attempt to reject the null hypothesis that the mean waiting time at our Orbiter bus-stop is $10$ minutes under an IID $Exponential(\lambda^*)$ model. Let $\alpha=0.05$ for this test. We can formulate this test as follows:
$$
H_0: \lambda^* = \lambda_0= \frac{1}{10} \quad \text{versus} \quad H_1: \lambda^* \neq \frac{1}{10}, \quad \text{where, } \quad X_1\ldots,X_{132} \overset{IID}{\sim} Exponential(\lambda^*) \enspace .
$$
We already obtained the $95\%$ confidence interval based on its MLE's asymptotic normality property to be $[0.0914, 0.1290]$.
$$\boxed{\text{Since our null value $\lambda_0=0.1$ belongs to this confidence interval, we fail to reject the null hypothesis from a $\mathsf{size}$ $\alpha=0.05$ Wald test.}}$$
We will revisit this example in a more computationally explicit fasion soon below.
