Data Collection Methods
Design a useful study: define the target, distinguish observation from intervention, and handle sampling, missingness and measurement.
On This Page
Define the claim before collecting the numbers
A useful study begins with a target population, observational unit, outcome and comparison. “Does a new checkout help?” is incomplete. “Does assigning eligible visitors to a shorter checkout increase the proportion who buy within seven days?” specifies a measurable question.
Write down who is eligible, how people enter the study, the measurement window and which outcome counts as success. Decide these before viewing results to limit opportunities to select a favorable analysis.
Observation and intervention support different conclusions
An observational study records what happens without assigning the exposure. It can describe patterns and associations, but groups may differ for reasons other than the exposure. Customers using express shipping may have larger budgets, more urgent orders and different expectations.
An experiment deliberately assigns a treatment. Random assignment can make treatment groups comparable on average, supporting a causal comparison when assignment, measurement and follow-up are sound. It does not guarantee perfect balance or fix differential missing outcomes, spillovers, attrition or inconsistent implementation.
Random sampling helps generalize to a population; random assignment helps compare interventions. A randomized experiment using volunteers is not automatically representative of every customer.
Worked example: design an onboarding comparison
Suppose a hypothetical site tests two tutorials. Assign each eligible new account once to A or B, retain that assignment on return visits, and define completion within seven days as the outcome. Randomizing individual pageviews would let one account see both versions and would treat dependent observations as independent.
If 80 of 200 assigned accounts complete A and 100 of 200 complete B, the observed completion proportions are 40% and 50%. The difference is 10 percentage points, or a 25% relative increase compared with 40%. These descriptive values alone do not quantify uncertainty or establish whether the study was conducted correctly.
Analyze using the assigned groups when appropriate for the intended treatment-policy question. Report missing outcomes and exclusions, and preserve the denominator: “100 of 200 assigned accounts,” rather than just completed accounts. Do not repeatedly inspect results and stop at the first small p-value without a valid sequential design.
Surveys need more than a large response count
Neutral wording, response options, question order and accessibility affect measurement. An optional feedback widget measures the views of responders, not the satisfaction of everyone who visited. Report invitation and response counts when available. A large voluntary sample can still be systematically biased.
Pilot the collection process: test missing values, duplicate IDs, timestamps, units and impossible values. Keep a small data dictionary and a log of changes. Prefer collecting the minimum information needed for the stated question.
Practice
A survey link is posted to loyal customers. Ten thousand respond. Does the size make its estimate representative of all customers?
Show the explanation
No. Coverage and participation can favor loyal customers even with many responses. Increasing the sample reduces random variability under a suitable design; it does not remove systematic selection or measurement bias.
Continue
Choose a sampling method and identify data types. OpenStax discusses sampling, variation and bias.
Did this explanation help?
Your response helps us identify lessons that need more work.