Lecture 7

Last time

How to compare 2 proportions

Playing for a sell out crowd

Sports teams prefer to play in front of their own fans rather than at the opposing team's site. Having a sell out crowd should provide even more excitement and lead to an even better performance right?

Playing for a sell out crowd

Sports teams prefer to play in front of their own fans rather than at the opposing team's site. Having a sell out crowd should provide even more excitement and lead to an even better performance right?

Simulating the null

Null hypothesis:
There is no association between playing in front of a sell out crowd and winning/losing a game.

If there was no association between the variables, what would be the probability of observing an association as strong as the one we observed ($RR=3.1$) between the fact that a game is a sell out or not and winning or losing?

To evaluate the statistical significance of the observed association in our groups, we will investigate how large the $RR$ in conditional proportions tends to be just from the random assignment of outcomes (win or loss) to the explanatory variable groups (sell out crowd or not).
=> we will do a simulation in which the chances of winning a game is the same whether it is a sell out or not.

Simulating the null

1. Pool all the 41 wins/losses together
2. Shuffle all 41 games and randomly redistribute into two stacks
   • One with 18 games (representing the sell out crowd games)
   • Another 23 games (representing the games without sell out crowd)
3. Calculate the relative proportion and relative risk for the simulation
4. Repeat 10000 times (we obtain 10000 RR that we could have observed if the presence or not of a sell out crowd did not matter)

Simulating the null

We have strong evidences supporting the idea that it is less likely for the Oklahoma City Thunder to win when they play in front of a sell out crowd.

95% confidence intervals

What is the true value of how much more likely the Oklahoma City Thunder is to win when there is not a sell out crowd compare to when they play in a packed stadium?

95% confidence intervals

What is the true value of how much more likely the Oklahoma City Thunder is to win when there is not a sell out crowd compare to when they play in a packed stadium?

Possible explanations for home court disadvantage

There are three possible explanations for this odd finding that the team had a better winning percentage with smaller crowds:

Possible explanations for home court disadvantage

Association does not mean causation

Two variables are associated (or related), if the value of one variable gives you information about the value of the other variable.
=> When comparing groups, this means that the proportions or means take on different values in the different groups.

A confounding variable is a variable that is related both to the explanatory and to the response variable in such a way that its effects on the response variable cannot be separated from the effects of the explanatory variable.

Always think critically when you observe a cause-and-effect conclusion.

Avoiding confounding variables

In a randomized experiment, you use a chance device to make the assignements. The role of the random assignement is to balance out potentially confounding variables among the explanatory variable groups, giving us the potential to draw cause-and-effect conclusions. (e.g.: you flip a coin before a game to assign if this will be sell-out or not)

In a double-blind study, neither the subjects nor those evaluating the response variable know which treatment group the subject is in. (ex: the teams and the statistician do not know if this is a sell out crowd game or not)

Avoiding confounding variables

The different possible study designs

Bicycle weight and commuting time

The UK government’s Cycle to Work scheme allows an employee to purchase a bicycle (up to a cost of $1560) at a significant discount by using tax incentives, provided the bicycle is used for commuting to and from work. The initiative aims to “promote healthier journeys to work and reduce environmental pollution.”

A British researcher, Jeremy Groves, decided to take advantage of this opportunity to buy a new bike (and publish a statistical study about it).

"Bicycle weight and commuting time: randomised trial" - BMJ, (2010) 341:c6801

Bicycle weight and commuting time

I purchased a bike at the top end of the cost allowed by the scheme and opted for a carbon frame because it was significantly lighter than my existing bicycle’s steel frame. The wheels were lighter and tyres narrower too. All were factors that made me believe that the extra £950 I had spent would get me to work in a trice.

One sunny morning, I got to work in 43 minutes, the fastest I could recall. My steel bike was consigned to a corner of the garage to gather dust—until I had a puncture. The next day I was back on my old steel bike. I fitted the cycle computer, set off . . . and discovered I had got to work in 44 minutes. “Hang on,” I thought, “was that minute worth £950 or was it a fluke?” There was only one answer: a randomised trial.

Bicycle weight and commuting time

Randomized experiment conducted by Jeremy Groves

Research question:
Groves wanted to know if bicycle weight affected his commute to work.

Experimental design:
For 56 days (January to July) Groves tossed a coin to decide if he would bike the 27 miles to work on his carbon frame bike (20.9lbs) or steel frame bicycle (29.75lbs).
He recorded the commute time for each trip.

What are the observational units? What are the variables? What type/kind of variables are those?

Bicycle weight and commuting time

The parameters of interest are:
$\mu_{\mathrm{carbon}}$ = Long term average commute time with carbon frame bike
$\mu_{\mathrm{steel}}$ = Long term average commute time with steel frame bike

Bicycle weight and commuting time

Remember:
The hypotheses are about the association between commute time and bike used, not just the 56 trips of Groves.
Hypotheses are always about populations or processes, not the sample data.

Bicycle weight and commuting time

The sample average and variability for commute time was higher for the carbon frame bike. Does this indicate a tendency? Or could a higher average just come from the random assignment?

Perhaps the carbon frame bike was randomly assigned to days where traffic was heavier or weather slowed down Dr. Groves on his way to work?

Bicycle weight and commuting time

Is it possible to get a difference of 0.53 minutes if commute time isn’t affected by the bike used?
Yes it’s possible, how likely though?
=> is the observed difference in commute time arising solely from the randomness in the of the frame type used each day?

Simulating the bike commute null distribution

Statistic of interest?
The observed difference in average commute time
$d=\mu_{\mathrm{carbon}} - \mu_{\mathrm{steel}}$

Shuffling assumes the null hypothesis of no association between commute time and bike.

Simulating the bike commute null distribution (difference in means)

Shuffling procedure

Simulating the random assignment of bike frame (difference in means)

Simulating the random assignment of bike frame (difference in means)

What does this p-value mean?

If the mean commute times for the bikes were the same in the long run, and we were repeating random assignment of the lighter bike (carbon) to 26 days and the heavier (steel) to 30 days, a difference at least as extreme as 0.53 minutes would occur in about 72% of the repetitions.
=> Therefore, we don’t have evidence that the commute times for the two bikes will differ in the long run.

Have we proven that the bike Groves chooses is not associated with commute time? (Can we conclude the null?)
No! A large p-value is not “strong evidence that the null hypothesis is true.” It suggests that the null hypothesis is plausible
=> There could be long-term difference just like we saw, but it is just very small.

Scope of conclusions of the study

Can we generalize our conclusion to a larger population?

Two Key questions:

Was the sample randomly obtained from a larger population?
No, Groves commuted on consecutive days which didn’t include all seasons.
=> We cannot generalize to all the commute trips

Were the observational units randomly assigned to treatments?
Yes, he flipped a coin to choose which bike to ride each day
=> We can draw cause-and-effect conclusions.

We can’t generalize beyond Groves and his two bikes. A limitation is that this study is not double-blind. The researcher and the subject (which happened to be the same person) were not blind to which treatment (bike) was being used. (Perhaps Groves likes his old bike and wanted to show it was just as good as the new carbon-frame bike for commuting to work.)

Simulating the random assignment of bike frame (difference in medians)

Statistic of interest?
The observed difference in median commute time
$d=m_{\mathrm{carbon}} - m_{\mathrm{steel}}$

Shuffling assumes the null hypothesis of no association between commute time and bike.

Simulating the random assignment of bike frame (difference in medians)

Simulating the random assignment of bike frame (difference in medians)

But isn't getting an estimate of the true value of the parameter what's really important?

I couldn't agree more!!!
You're right, let's compute the $95\%$ confidence intervals of the difference in median between the times obtained with the carbon frame and the times obtained with a steel frame

Resampling confidence intervals

Resampling confidence intervals

Resampling confidence intervals

=> Zero (the "no difference in median times") is in this interval. So we do not have evidence against the fact that Groves could have obtained the results he obtained if there were no difference in the times it would take for a trip using either bike.

Resampling for comparing 2 groups

Theory: Two-sample $t$-test

When comparing means between two groups, the null hypothesis is typically H$_0$ ∶ $\mu_1=\mu_2$ or, equivalently, H$_0$ ∶ $\mu_1-\mu_2=0$.
Thus the “Null parameter” is usually equal to zero and we use the difference in means for two samples, $\bar{x}_1 − \bar{x}_2$, as the “Sample statistic”.

If the underlying populations are reasonably normal or the sample sizes are large, we can estimate the standard error (standard deviation of the sampling distribution) of $\bar{x}_1 − \bar{x}_2$ with $SE = \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$, where $s_1$ and $s_2$ are the standard deviations in the two samples.

Note: When we use the samples standard deviations in estimating SE, we need to switch to a $t$-distribution rather than the standard normal when finding a p-value.

Theory: Two-sample $t$-test

For a two-sample $t$-test, you are making the (strong) assumption that your samples are coming from populations normally distributed. You can then compute the ($t$) distribution of the difference in means from 2 samples from these underlying distributions.

Theory: Two-sample $t$-test

To test H$_0$ ∶ $\mu_1=\mu_2$ vs H$_a$ ∶ $\mu_1\neq\mu_2$ (or a one-tail alternative $\mu_1>\mu_2$ or $\mu_1<\mu_2$) based on samples of sizes n$_1$ and n$_2$ from the two groups, we use the two-sample $t$-statistic.

Theory: Two-sample $t$-test

To test H$_0$ ∶ $\mu_1=\mu_2$ vs H$_a$ ∶ $\mu_1\neq\mu_2$ (or a one-tail alternative $\mu_1>\mu_2$ or $\mu_1<\mu_2$) based on samples of sizes n$_1$ and n$_2$ from the two groups, we use the two-sample $t$-statistic.

Theory: confidence intervals

If the underlying populations are reasonably normal or the sample sizes are large, we can estimate the standard error (standard deviation of the sampling distribution) of $\bar{x}_1 − \bar{x}_2$ with $SE = \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$, where $s_1$ and $s_2$ are the standard deviations in the two samples.

If the distribution for a statistic follows the shape of a $t$ distribution with standard error SE, we find a confidence interval for the parameter using:

$\textrm{Sample Statistic} \pm t^*\times SE$

where $t^*$ is chosen so that the proportion between −$t^*$ and +$t^*$ in the $t$-distribution is the desired level of confidence.

If the sample size in each group is large:

• $t^*\simeq2$ ($1.96$) for the $95\%$ confidence intervals
• $t^*\simeq1.645$ for the $90\%$ confidence intervals
• $t^*\simeq2.576$ for the $99\%$ confidence intervals

Theory: confidence intervals

If the underlying populations are reasonably normal or the sample sizes are large, we can estimate the standard error (standard deviation of the sampling distribution) of $\bar{x}_1 − \bar{x}_2$ with $SE = \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$, where $s_1$ and $s_2$ are the standard deviations in the two samples.

If the distribution for a statistic follows the shape of a $t$ distribution with standard error SE, we find a confidence interval for the parameter using:

$\textrm{Sample Statistic} \pm t^*\times SE$

$SE=\sqrt{\frac{6.25^2}{26}+\frac{4.89^2}{30}}=1.51$

Mean & $95\%$ confidence intervals:
$0.53\pm2\times1.51$
$0.53\pm3.02$

Theory: What about the median?

Can't we just get an equation to model the distribution of the sampling distribution of the difference in medians?

NO!!! Remember, the median is a procedure, not the result of a formula, so there is no theorem or formula that can predict what the sampling distribution of the difference in median would look like. There is no two-sample t-test for the median

Using resampling statistic, you are not limited to the the study of the mean for which formula and theorems are available.
Using resampling statistic, you can study any statistic that is relevant to your study, even if if is a very exotic statistic, since each time you can do a simulation of what the sampling distribution for this particular exotic statistic would look like.