From question to data plan

An investigative question worth collecting data for has three working parts. First, it is phrased in terms of the variables of interest, which tells you exactly what to measure. "Do students get enough sleep?" measures nothing; "What is the mean nightly sleep time of students at this school?" tells you to record hours per night. Second, it points at the analysis: a question about estimating a value calls for an interval around a parameter, while a question asking whether one group differs from another calls for a test with a stated direction. Third, it names the population the conclusion should reach, and the kind of conclusion, descriptive or cause-and-effect, that the study is being built to support.

Read the three parts back and the data collection plan almost writes itself: what to record, from whom, and by what selection mechanism. When a study goes wrong at the design stage, it is usually because one of these parts was never pinned down.

A census records information from every item or individual in the population. It answers the question outright, with no inference needed, and it is usually impractical: measuring all 2,000 students is a census, measuring 50 of them is a sample, and those are different claims with different strengths.

An observational study records the values of variables of interest without imposing anything on anyone. A sample survey is the human version: an observational study that collects data from people using a standard set of questions. Observational studies come in two time directions: a prospective study selects units now and gathers data going forward, while a retrospective study selects units now and gathers data from the past.

An experiment is the one design where the researcher intervenes: conditions, called treatments, are assigned to experimental units (called subjects or participants when they are people). The variable whose levels are imposed is the explanatory variable, and the outcome measured afterward on each unit is the response variable. The assignment is the defining act: choosing whom to study is not an experiment, imposing a condition on them is.

Randomness enters a study in two places, and they do different jobs:

1. Random selection decides who gets into the sample, using a chance mechanism such as a random number generator. It makes the sample representative of the population it was drawn from, so the results can be generalized to that population.
2. Random assignment decides which treatment each unit receives. It balances the treatment groups, so a difference in the response can be attributed to the treatment: a cause-and-effect conclusion.

A study can have either, both, or neither. Volunteers randomly assigned to treatments support a causal claim, but only about people similar to those volunteers. A random sample that is merely observed supports a generalization, but not a causal claim. Citing the wrong randomness for a conclusion is one of the most reliably penalized errors on the AP exam.

When units are not randomly selected, because they were deliberately chosen or volunteered themselves, generalization shrinks to a population of individuals similar to those studied. And the number computed from a sample is a statistic: an estimate of the population's parameter, never the parameter itself. "42 percent of our sample" and "42 percent of the school" are different sentences, and only one of them was measured.

In an observational study, the groups being compared assembled themselves, and whatever made them different in the first place travels with them. A confounding variable is a variable associated with both the explanatory variable and the response variable, which lets it provide an alternative explanation for the observed relationship.

Both links are required, and naming them in the specific study is what earns credit. "Maybe other stuff matters" identifies nothing. "Children who eat breakfast tend to have more involved parents, and parental involvement also tends to raise test scores" names one variable and traces both of its connections, so the breakfast-and-scores association no longer forces a causal reading.

This is not a defect to be patched with a bigger sample. However many units you observe, the self-assembled groups still differ in the same systematic ways. The only design that breaks the link between the explanatory variable and everything tangled with it is random assignment, which is the subject of Topic 1.13.