3.3.5 One sample $t$ tests

An SAT preparation company claims that its students’ scores improve by over 100 points on average after their course. A consumer group would like to evaluate this claim, and they collect data on a random sample of 30 students who took the class. Each of these students took the SAT before and after taking the company’s course, and so we have a difference in scores for each student. We will examine these differences $x_1=57$, $x_2=133$, …, $x_{30}=140$ as a sample to evaluate the company’s claim. (This is paired data, so we analyse the score differences; for a review of the ideas of paired data, see Section 3.1.) The distribution of the differences, shown in Figure LABEL:satImprovementHTDataHistogram, has mean 135.9 and standard deviation 82.2. Do these data provide convincing evidence to back up the company’s claim?

Answer. This is a one-sided test. $H_0$: student scores do not improve by more than 100 after taking the company’s course. ${\mu }_{{}_{diff}}=100$ (we always write the null hypothesis with an equality). $H_A$: students scores improve by more than 100 points on average after taking the company’s course. ${\mu }_{{}_{diff}}>100$.

Answer. This is a random sample from less than 10% of the company’s students (assuming they have more than 300 former students), so the independence condition is reasonable. The normality condition also seems reasonable based on Figure LABEL:satImprovementHTDataHistogram. We can use the $t$ distribution method. Note that we could use the normal distribution. However, since the sample size ($n=30$) just meets the threshold for reasonably estimating the standard error, it is advisable to use the $t$ distribution. Just as we did for the normal case, we standardize the sample mean using the Z score to identify the test statistic. However, we will write $T$ instead of $Z$, because we have a small sample and are basing our inference on the $t$ distribution:

If the null hypothesis was true, the test statistic $T$ would follow a $t$ distribution with $df=n-1=29$ degrees of freedom. We can draw a picture of this distribution and mark the observed $T$, as in Figure LABEL:pValueShownForSATHTOfOver100PtGain. The shaded right tail represents the p-value: the probability of observing such strong evidence in favour of the SAT company’s claim, if the average student improvement is really only 100.

Answer. We use 29 degrees of freedom. The value $t=2.39$ thus pt(2.39,df=29). Therefore the p-value is 1-0.9882165= 0.0117835 and is less than 0.05 (the default significance level), so we reject the null hypothesis. The data provide convincing evidence to support the company’s claim that student scores improve by more than 100 points following the class.

Answer. This is an observational study, so we cannot make this causal conclusion. For instance, maybe SAT test takers tend to improve their score over time even if they don’t take a special SAT class, or perhaps only the most motivated students take such SAT courses.