Hypothesis testing is an inferential procedure that uses sample data to evaluate the credibility of a hypothesis about a population parameter. The process involves ...

1. Stating a hypothesis: an assumption that can neither be fully proven nor fully disproven. For example,

   • Not more than 5% of GM trucks breakdown in under 10,000 miles

   • Heights of North American adult males is distributed with $\mu =72$ inches

   • Mean year-round temperature in Athens (OH) is $>62$

   • 10% of Ohio teachers are Accomplished

   • Mean county unemployment rate is 12.1%

2. Drawing a sample to test the hypothesis

3. Conducting the test itself to see if the hypothesis should be rejected

The Null and the Alternative Hypotheses

• Null Hypothesis: $\left(H_0\right)$ is the assumption believed to be true

• Alternative Hypothesis: $\left(H_a\right)$ is the statement believed to be true if $\left(H_0\right)$ is rejected

  $H_0$: $\mu >72$ inches; $H_1$: $\mu \le 72$

  $H_0$: $\mu <72$ inches; $H_1$: $\mu \ge 72$

  $H_0$: $\mu \le 72$ inches; $H_1$: $\mu >72$

  $H_0$: $\mu \ge 72$ inches; $H_1$: $\mu <72$

  $H_0$: $\mu =72$ inches; $H_1$: $\mu \ne 72$

$H_0$ and $H_1$ are mutually exclusive and mutually exhaustive

• Mutually Exclusive: Either $H_0$ or $H_1$ is True $\cdots$ both cannot be true at the same time

• Mutually Exhaustive: $H_0$ and $H_1$ exhaust the Sample Space $\cdots$; there are no other possibilities unknown to us

Type I and Type II Errors

The probability of committing a Type I error $=\text{Level of Significance}=\alpha$
The probability of committing a Type II error $=\text{Level of Significance}=\beta$
The power of the test is $(1-\beta )$

We have to decide how often we want to make a Type I error (i.e., falsely Reject $H_0$). Conventionally we set this rate to one of the following $\alpha$ values:

$$\alpha =0.05\text{or}\alpha =0.01$$

Note the very cautious language ... Reject $H_0$ versus Do Not Reject $H_0$

The Process of Hypothesis Testing: One Example

Assume we want to know whether the roundabout on SR682 has had an impact on traffic accidents in Athens. We have historical data on the number of accidents in years past. Say the average per day used to be 6 (i.e., ${\mu }_0=6$). To see if the roundabout has had an impact we could gather accident data for a random sample of 100 days $(n=100)$ from the period after the roundabout was built.

Before we do that though, we will need to specify our hypotheses. What do we think might be the impact? Let us say the City Engineer argues that the roundabout should have decreased accidents.

• If he is correct then the sample mean $\overline{x}$ should be less than the population mean ${\mu }_0$ i.e., $\overline{x}<{\mu }_0$

• If he is wrong then the sample mean $\overline{x}$ should be at least as much as the population mean ${\mu }_0$ i.e., $\overline{x}\ge {\mu }_0$

The Sampling Distribution of $\overline{x}$

We know from the theory of sampling distributions that the distribution of sample means, for all samples of size $n$, will be normally distributed (as shown below)

Most samples would be in the middle of the distribution but by sheer chance we could end up with a sample mean in the tails. This will happen with a very small probability but it could happen!!

Scenario 1: Engineer says $\overline{x}<\mu$

If we believe the City Engineer, we would setup the hypotheses as follows:

• $H_0$: The roundabout does not reduce accidents, i.e., $\mu \ge {\mu }_0$
• $H_1$: The roundabout does reduce accidents, i.e., $\mu <{\mu }_0$
• Next set $\alpha$ (the probability of making a Type I error)
• We then calculate the sample mean $\left(\overline{x}\right)$ and the sample standard deviation $\left(s\right)$
• Next we calculate the standard error of the sample mean: $s_{\overline{x}}=\frac{s}{\sqrt{n}}$
• Now we calculate $t=\frac{\overline{x}-{\mu }_0}{s_{\overline{x}}}$ and this is what we call $t_{calculated}$
• Using $df=n-1$, find the area to the left of $t_{calculated}$
• If this area is very small we can conclude that the roundabout must have worked to reduce accidents
• How should we define very small? By setting $\alpha$ either to 0.05 or to 0.01

We Reject $H_0$ if $P\left(t_{calculated}\right)\le \alpha$; the data provide sufficient evidence to conclude that the roundabout has reduced accidents

If $P\left(t_{calculated}\right)>\alpha$ then we Fail to reject $H_0$; the data provide insufficient evidence to conclude that the roundabout has reduced accidents

Rejection rule

Reject $H_0$ if calculated $t$ falls in the green region (i.e., calculated $t\le -1.6603$)

Failure to reject rule

Do Not Reject $H_0$ if calculated $t$ falls in the grey region (i.e., $t_{calculated}>-1.6603$)

Scenario 2: Engineer says $\overline{x}\ne \mu$

If we believe the City Engineer, we would setup the hypotheses as follows:

• $H_0$: The roundabout has no impact on accidents, i.e., $\mu ={\mu }_0$
• $H_1$: The roundabout has an impact on accidents, i.e., $\mu \ne {\mu }_0$
• We then calculate the sample mean $\left(\overline{x}\right)$ and the sample standard deviation $\left(s\right)$
• Next we calculate the standard error of the sample mean: $s_{\overline{x}}=\frac{s}{\sqrt{n}}$
• Now calculate $t_{calculated}=\frac{\overline{x}-{\mu }_0}{s_{\overline{x}}}$
• Using $df=n-1$, find the area to the left/right of $\pm t_{calculated}$

If this area is very small then we can conclude that the roundabout must have worked to reduce accidents

How should we define very small? By setting $\alpha$ either to 0.05 or to 0.01

We can then Reject $H_0$ if $P(\pm t_{calculated})\le \alpha$ ; the data provide sufficient evidence to conclude that the roundabout has reduced accidents

If $P(\pm t_{calculated})>\alpha$ then we will Fail to reject $H_0$; the data provide insufficient evidence to conclude that the roundabout has reduced accidents

Rejection rule

Reject $H_0$ if calculated $\left|t\right|$ falls in the green region (i.e., calculated $t\le -1.98$ or calculated $t\ge 1.98$)

Failure to reject rule

Do Not Reject $H_0$ if calculated $\left|t\right|$ falls in the grey region (i.e., $-1.98<\text{calculated}t<1.98$)

The process revisited ...

1. State the hypotheses

   • If we want to test whether something has changed, or is different, or had an impact, etc. then $H_0$ must specify that nothing has changed $\dots H_0:\mu ={\mu }_0;H_1:\mu \ne {\mu }_0\dots$ two-tailed
   • If we want to test whether something has increased, or has risen, or is more then $H_0$ must specify that it has not increased $\dots H_0:\mu \le {\mu }_0;H_1:\mu >{\mu }_0\dots$ one-tailed
   • If we want to test whether something has decreased, or has reduced, or is less then $H_0$ must specify that it has not decreased $\dots H_0:\mu \ge {\mu }_0;H_1:\mu <{\mu }_0\dots$ one-tailed

2. Collect the sample and set $\alpha =0.05$ or $\alpha =0.01$

3. Calculate $s_{\overline{x}}=\frac{s}{\sqrt{n}}$, $\overline{x}$, $df=n-1$

4. Calculate the $t$

5. Reject $H_0$ if calculated $t$ falls in the critical region; do not reject $H_0$ otherwise

But of course ... beware Type I and Type II errors

• Type I Error: You rejected $H_0$ but it should not have been rejected $(\text{level of significance}=\alpha )$
• Type II Error: You failed to reject $H_0$ but it should have been rejected

Example 1 (One-sample t-test)

[From Harrell & Slaughter] We want to test if the mean tumor volume is 190 $m{m}^3$ in a population with melanoma

$H_0:{\mu }_0=190$ versus $H_1:{\mu }_0\ne 190$

$\overline{x}=181.52,s=40,n=100,{\mu }_0=190$

$s_{\overline{x}}=\frac{s}{\sqrt{n}}=\frac{40}{\sqrt{100}}=4$

$t=\frac{\overline{x}-{\mu }_0}{s_{\overline{x}}}=\frac{181.52-190}{4}=-2.12$

$p-value=0.037$, leading us to reject $H_0$ if $\alpha =0.05$

The data do not conform to the pattern predicted by the null hypothesis.

Example 2 (Paired t-test)

[From Harrell & Slaughter] To investigate the relationship between smoking and bone mineral density, Rosner presented a paired analysis in which each person had a nearly perfect control which was his or her twin. Data were normalized by dividing differences by the mean density in the twin pair. Computed density in heavier smoking twin minus density in lighter smoking one.

Mean difference was -5% and standard error was 2%, with $n=41$

$H_0:\text{mean difference is}{\mu }_0=0$ versus $H_1:\text{mean difference is}{\mu }_0\ne 0$

$t=\frac{-5-0}{2}=-2.5$

$p-value=0.0166$

The data do not conform to the pattern predicted by the null hypothesis.

Comparisons of Means from Common Parent Populations

Comparisons of Means from Different Parent Populations

Separate Parent Populations

We often need to compare sample means across two groups. For example, are average earnings the same for men and women in a specific occupation? Perhaps we suspect (a) women are underpaid or (generally) that (b) their salaries differ from those of men.

Let the population and sample means be ${\mu }_m,{\mu }_w$ and ${\overline{x}}_m,{\overline{x}}_w$, respectively

(a) $H_0:{\mu }_m\le {\mu }_w$ and $H_1:{\mu }_m>{\mu }_w$, $\therefore H_0:{\mu }_m-{\mu }_w\le 0$ and $H_1:{\mu }_m-{\mu }_w>0$

(b) $H_0:{\mu }_m={\mu }_w$ and $H_1:{\mu }_m\ne {\mu }_w$, $\therefore H_0:{\mu }_m-{\mu }_w=0$ and $H_1:{\mu }_m-{\mu }_w\ne 0$

Standard Error of the difference in means: $s_{{\overline{x}}_m-{\overline{x}}_w}=\sqrt{\frac{s_m^2}{n_m}+\frac{s_w^2}{n_w}}$

Confidence Interval estimate: $({\overline{x}}_m-{\overline{x}}_w)\pm t_{\alpha /2}\left(s_{{\overline{x}}_m-{\overline{x}}_w}\right)$

The Test Statistic: $t=\frac{({\overline{x}}_m-{\overline{x}}_w)-({\mu }_m-{\mu }_w)}{\sqrt{\frac{s_m^2}{n_m}+\frac{s_w^2}{n_w}}}=\frac{({\overline{x}}_m-{\overline{x}}_w)-D_0}{\sqrt{\frac{s_m^2}{n_m}+\frac{s_w^2}{n_w}}}$

The degrees of freedom for this test: $df=\frac{{(\frac{s_m^2}{n_m}+\frac{s_w^2}{n_w})}^2}{\frac{1}{(n_m-1)}{\left(\frac{s_m^2}{n_m}\right)}^2+\frac{1}{(n_w-1)}{\left(\frac{s_w^2}{n_w}\right)}^2}$

Note: We usually round down the $df$ to the nearest integer

We have two ways of calculating the estimated standard error $\left(s_{{\overline{x}}_1-{\overline{x}}_2}\right)$ and the degrees of freedom $df$

(1) When the population variances are assumed unequal
(2) When the population variances are assumed equal

(1) Unequal Population Variances

Standard Error will be: $\left(s_{{\overline{x}}_1-{\overline{x}}_2}\right)=\sqrt{\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2}}$

Degrees of Freedom will be: $df=\frac{{(\frac{s_m^2}{n_m}+\frac{s_w^2}{n_w})}^2}{\frac{1}{(n_m-1)}{\left(\frac{s_m^2}{n_m}\right)}^2+\frac{1}{(n_w-1)}{\left(\frac{s_w^2}{n_w}\right)}^2}$

• Use this when $n_1$ or $n_2$ are $<30$ and

• Either sample has a standard deviation at least twice that of the other sample

(2) Equal Population Variances

Standard Error will be: $\left(s_{{\overline{x}}_1-{\overline{x}}_2}\right)=\sqrt{\frac{n_1+n_2}{n1\times n_2}}\sqrt{\frac{(n_1-1)s_{x_1}^2+(n_2-1)s_{x_2}^2}{(n_1+n_2)-2}}$

Degrees of Freedom will be: $df=(n_1+n_2)-2$

• Use when the standard deviations are roughly equal, and

• $n_1$ and $n_2$ $\ge 30$

Example 3 (Two-Sample t-test)

[From Harrell & Slaughter] Two soporific drugs to be tested, Durg 1 versus Drug 2. Which of these is more effective?

$H_0:{\mu }_1={\mu }_2$ versus $H_1:{\mu }_1\ne {\mu }_2$

Assuming unequal variances: $t=-1.8608,df=17.776,p-value=0.07939$ so fail to reject $H_0$
Assuming equal variances: $t=-1.8608,df=18,p-value=0.07919$ so fail to reject $H_0$

Binomial and Chi-Square Distributions

The Binomial

Given a certain number of independent trials $\left(n\right)$ with an identical probability $\left(p\right)$ of success $\left(X\right)$ in each trial we can easily calculate the probability of seeing a specific number of successes.

For example, if I flip a coin 10 times, where $X=Head$ with $p=0.5$ then what is probability of seeing exactly 2 heads, exactly 4 heads, 7 heads, etc.? The answer is easily calculated as:

Radiologists and Missing Sons

Assume that they are just as likely to have boys as girls. This then generates the following hypotheses:

$H_0:\text{Radiologsts are just as likely to have sons as daughters}(p=0.5)$ $H_1:\text{Radiologsts are not as likely to have sons as daughters}(p\ne 0.5)$

Let $\alpha =0.05$

The $p-value=0.005014$ so we can reject $H_0$; the data provide sufficient evidence to conclude that radiologists are not as likely to have sons as daughters.

What if we suspected, a priori that radiologists are less likely to have sons? In that case we would have done the following:

$H_0:\text{Radiologsts are at least as likely to have sons as daughters}(p\ge 0.5)$ $H_A:\text{Radiologsts are less likely to have sons than daughters}(p<0.5)$

Again, the $p-value=0.002507$ and we can easily reject $H_0$; the data provide sufficient evidence to conclude that radiologists are not at least as likely to have sons as daughters.

The ${\chi }^2$

The ${\chi }^2$ distribution is used with multinomial data (i.e., when the categorical variable has more than two categories) to test whether observed frequency counts differ from expected frequency counts.

The Goodness-of-Fit Test for a single variable

$H_0$: Proportions are all the same $H_A$: Proportions are \textit{not} all the same

$${\chi }^2=\sum _i\frac{{(Observe{d}_i-Expecte{d}_i)}^2}{Expecte{d}_i}$$

${\chi }^2$ distributed with $(\text{no. of categories}-1)$ degrees of freedom $\left(\text{df}\right)$

Reject $H_0$ if $p-value\le \alpha$; Do not reject $H_0$ otherwise

As $df\to \infty$ you need a larger ${\chi }^2$ to Reject $H_0$ at the same $\alpha$

The plot below shows how the theoretical ${\chi }^2$ distribution varies with the degrees of freedom. As the distribution shifts right the degrees of freedom are getting smaller. The first is for 3 degrees of freedom, which means we have a total of 4 categories, then we have 4 degrees of freedom (so 5 categories), then 5 degrees of freedom (so 6 categories), and finally 6 degrees of freedom (i.e., 7 categories). Note what happens; The more the degrees of freedom, the larger the ${\chi }^2$ value needed to reject $H_0$ with $\alpha =0.05$ or $\alpha =0.01$.

An Example

Assume that there are four gourmet meats placed before 100 subjects. In a blind taste test each subject is asked to pick the item they liked the most. Do subjects exhibit indifference between the four items? If they do, then we would expect about 25% to pick item A, 25% to pick item B, 25% to pick item C, and 25% to pick item D. If $H_0$ were true then expected frequencies would be as follows for A, B, C, and D:

## [1] 25 25 25 25

Now assume that the 100 subjects actually indicated the following preferences for A, B, C, and D:

## [1] 30 10 40 20

Some of the difference between the observed frequencies and expected frequencies if $H_0$ were true (i.e., there were no clear preferences) could be by chance. Therefore we test whether the overall difference is enough to suggest that this could not happen by chance very often or if it could happen very often. In other words, do the data suggest that subjects do prefer some items over the others. We will set $\alpha =0.05$ and then conduct the test.

Carrying out the test

## 
##     Chi-squared test for given probabilities
## 
## data:  observed
## X-squared = 20, df = 3, p-value = 0.0001697

Given that the $p-value$ is less than $\alpha =0.05$ we can reject the $H_0$ that the items are equally preferred. That is, the data suggest that some items are preferable to others.

${\chi }^2$ test of Association\Independence

Often you are interested in testing for a relationship between two categorical variables

• is smoking associated with lung cancer?
• is a woman's educational attainment associated with fewer pregnancies?
• does diabetes differ by race\ethnicity?

Calculate, for each cell in the contingency table, $\frac{(f_{ij}-e_{ij}{)}^2}{e_{ij}}$

Add the resulting value over all cells. This yields $${\chi }^2=\sum _i\sum _j\frac{(f_{ij}-e_{ij}{)}^2}{e_{ij}}$$

${\chi }^2\sim df=(r-1)(c-1)$ where ... $r=$ number of rows, and $c=$ number of columns

If you have a $2\times 2$ table or small samples then Fisher's Exact test may be preferable

Pigeons and Predation

It has been hypothesized that the white rump of pigeons serves to distract predators like the peregrine falcons, and therefore it may be an adaptation to reduce predation. To test this, researchers followed the fate of 203 pigeons, 101 with white rumps and 102 with blue rumps. Nine of the white-rumped birds and 92 of the blue-rumped birds were killed by falcons.

##           
##            blue white Sum
##   killed     92     9 101
##   survived   10    92 102
##   Sum       102   101 203

##           
##                  blue      white
##   killed   0.91089109 0.08910891
##   survived 0.09803922 0.90196078

The table suggests that 91% of the blue-rumped pigeons were killed versus only 9% of the white-rumped pigeons.

Do the two kinds of pigeons differ in their rate of capture by falcons? Carry out an appropriate test.

$H_0:$ Predation by falcons is independent of rump-color
$H_A:$ Predation by falcons is not independent of rump-color
$\alpha =0.05$

## 
##     Pearson's Chi-squared test
## 
## data:  tab.p
## X-squared = 134.13, df = 1, p-value < 2.2e-16

## 
##     Fisher's Exact Test for Count Data
## 
## data:  tab.p
## p-value < 2.2e-16
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##   33.70502 270.81972
## sample estimates:
## odds ratio 
##   89.50586

Regardless of the test used, we can safely Reject $H_0$ given that the $p-value\approx 0$. The data suggest that predation by falcons is not independent of rump-color

Coffee and Sex at Birth

Research Question: Are coffee preferences independent of sex at birth (i.e., is there any association between coffee preferences and gender)?

$H_0:$ Coffee preference is independent of sex at birth $H_1:$ Coffee preference is not independent of sex at birth

For each cell in the contingency table, calculate

$$e_{ij}=\frac{\text{Row}i\text{Total}\times \text{Column}j\text{Total}}{\text{Sample Size}}$$

$df=(r-1)(c-1)=(2-1)(3-1)=\left(1\right)\left(2\right)=2$

$p-value<0.05$; Reject $H_0$

Coffee preferences and gender are not independent

Understanding Infant Mortality (2014)

Can we predict infant mortality from income?

$$y=\alpha +\beta \left(x\right)$$

$$\text{Infant Mortality}=\alpha +\beta \left(Income\right)$$

$$\text{Infant Mortality}=53.23-0.0016\left(Income\right)$$

As Income increases by 1 US Dollar, Infant Mortality drops by 0.0016

If Income rises by 1000 US Dollars, Infant Mortality drops by $0.0016\left(1000\right)=1.6$

What if a country has Income of 20,000? What would be the predicted Infant Mortality?

$$\text{Infant Mortality}=53.23-0.0016\left(Income\right)=53.23-0.0016\left(20000\right)=53.23-32=21.23$$

How good is this linear regression?

(1) Income is a significant predictor ... $p-value=0.000000107$

(2) Adjusted $R^2=0.1884$ ... This linear regression model explains about 18.84% of the variaton in infant mortality

(3) Root Mean Squared Error $=35.09$ ... If we use this regression model to predict infant mortality, average prediction error would be $\pm 35.09$ infant deaths

High Adjusted $R^2$, small Root Mean Squared Error, and statistically significant predictor variable is desirable

Improving the Linear Regression Model

$$y=\alpha +{\beta }_1\left(x_1\right)+{\beta }_2\left(x_2\right)+\dots +{\beta }_k\left(x_k\right)$$

$$y=\alpha +{\beta }_1\left(Income\right)+{\beta }_2\left(\text{Female Youth Literacy Rate}\right)+{\beta }_3\left(\text{\% in Urban Areas}\right)$$

## 
## Call:
## lm(formula = U5MR ~ Income + FemaleYouthLR + Urban, data = sowc)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -41.570 -13.036  -4.449   4.948  94.270 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    1.810e+02  1.011e+01  17.901   <2e-16 ***
## Income        -3.374e-04  2.411e-04  -1.399   0.1642    
## FemaleYouthLR -1.418e+00  1.236e-01 -11.467   <2e-16 ***
## Urban         -2.686e-01  1.213e-01  -2.214   0.0286 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 23.22 on 129 degrees of freedom
## Multiple R-squared:  0.6525,    Adjusted R-squared:  0.6444 
## F-statistic: 80.74 on 3 and 129 DF,  p-value: < 2.2e-16

• Holding all else constant, as Female Youth Literacy Rate rises by a unit amount infant mortality drops by 1.418

• Holding all else constant, as the % of the population living in Urban areas rises by a unit amount infant mortality drops by 0.02686

• Income does not appear to have a statistically significant impact on infant mortality

• The model "explains" 64.44% of the variation in infant mortality

• Average prediction error would be 23.22

Potential Issues

• Biased sample (analysis will be worthless)

• Incorrect functional form $(y=\alpha +\beta (\frac{1}{x^3})\text{but you fit}y=\alpha +\beta (x\left)\right)$

• Influential outliers (extremely unusual data point may influence the linear regression line)

• Heteroscedastic errors (discretionary spending of poor families will have little variance while that of wealthy families will have more variance)

• Correlated errors (people in the same poor neighborhood are more likely to share adverse health outcomes)

• Measurement error (measurement error in the independent variables)

• High Multicollinearity (income and discretionary spending tend to be highly correlated so you cannot control for one while increasing the other by a unit amount)

• Non-normal distribution of errors (you forgot to include some independent variables or have the wrong functional form or both)

Logit (Logistic) and Probit Models

Goal is to model the probability of survival, drug impact, birth of a girl child, tumor shrinking, and so on

How good are the models?

## Bootstrapped (25 reps) Confusion Matrix 
## 
## (entries are percentual average cell counts across resamples)
##  
##           Reference
## Prediction    0    1
##          0 56.5 29.9
##          1  4.1  9.5
##                             
##  Accuracy (average) : 0.6599

## Bootstrapped (25 reps) Confusion Matrix 
## 
## (entries are percentual average cell counts across resamples)
##  
##           Reference
## Prediction    0    1
##          0 51.4 11.7
##          1 10.2 26.8
##                             
##  Accuracy (average) : 0.7814

Story

• Male's had a lower probability of survival than females, on average, and holding all else constant.

• Wealthier passengers had a higher probability of survival than other passengers, on average, and holding all else constant

Reference individual is a woman who paid the Median Fare. Compared to this individual,

(a) Probability that the average Male who paid the same Median Fare of 14.4542 survived = 0.1660

(b) Probability that the average Female who paid the same Median Fare of 14.4542 survived = 0.6919

(c) Probability of survival was 0.6760 for the average Female who paid 7.91 $\left(Q_1\right)$ and 0.7300 if she paid 31.00 $\left(Q_3\right)$

(d) Probability of survival was 0.1561 for the average Male who paid $Q_1$ Fare and 0.1934 if he paid $Q_3$ Fare

The Chivalry of the Seas