Careers
Part of Market research surveys beyond the obvious
Five public survey cases show how small design choices distort estimates
Five public survey cases, from the 1936 Literary Digest poll to the AAPOR 2020 review, show how wording, mode, speed cutoffs and weights distort estimates.
What to take away
- The clearest evidence on survey design error comes from public polls, academic reviews and government reports, not private client work.
- The 1936 Literary Digest forecast mailed 10 million ballots, received 2.4 million, and still missed Roosevelt's 523 to 8 electoral college win.
- Wording, mode, speed cutoffs and weights each move answers by enough to flip a close decision.
- The AAPOR review of 2020 election polls put the national overestimate of Biden's margin at 3.9 percentage points.
- Correct by narrowing the claim, testing overlap, and keeping the original record.
Five public cases, named
The cases below are real and checkable. None is a private study. Each one shows a single design choice shifting an estimate.
| Case | Design choice | Documented shift |
|---|---|---|
| Literary Digest 1936 | Sampling frame | Predicted Landon; Roosevelt won 523 to 8 |
| Pew Research Center label test | Question label | Several points; 5 to 10 is a typical range in wording experiments |
| U.S. Census Bureau mode report | Mode of collection | Comparable answers need mode-sensitive presentation |
| Pew Research Center bogus respondents study | Speed cutoff | Fraud signals mixed with valid fast completes |
| AAPOR 2020 poll review | Weighting | National margin off by 3.9 points |
Start with the market research surveys overview if you need the vocabulary before reading the cases.
Case 1: Literary Digest 1936 and a frame that was the market
In 1936 the magazine mailed 10 million ballots and got 2.4 million back. Its lists came from telephone owners, car registrations and magazine subscribers. It predicted Alf Landon. Roosevelt won 523 electoral votes to 8. Gallup's quota sample called it for Roosevelt.
Literary Digest 1936 forecast
- 10 millionballots mailed
- 2.4 millionballots returned
- 523 to 8Roosevelt electoral votes
A business that surveys only active email subscribers and calls the result market demand repeats the error. Reviewers narrow the population to reachable subscribers, disclose excluded groups, and commission separate research for the missing ones. Quota samples carry their own bias.
Case 2: Pew label test and context inside a question
Pew Research Center asked about the same law under two labels, "Obamacare" and "Affordable Care Act". Support moved by several points depending on the label. Wording that supplies an approval context before asking for an opinion does work the question did not intend.
Wording experiments commonly shift answers by about 5 to 10 points. A concept test that calls a product secure before asking about interest can move purchase intent the same way. Cognitive testing is one of the market research surveys best practices that catches loaded wording before it reaches the field.
Strip the promotional wording, define the concept, add comprehension checks, balance the alternatives, and withdraw any earlier comparison. Neutral wording still misses intensity of feeling.
Case 3: Census Bureau mode report and a fake trend
The U.S. Census Bureau's working paper on questionnaires across modes explains that equivalent meaning can require mode-sensitive presentation rather than identical formatting. The goal is comparable substantive answers, not identical layout.
When a tracker moves from interviewer-led calls to a self-administered web survey, sensitive answers change. Mode effects on sensitive items commonly run a few points and can reach 10 points or more. That shift is not customer sentiment.
Mode change correction steps
- Run overlap testing
- Review coverage and order
- Document a series break
- Start a new baseline
- Preserve earlier results
Correcting a mode change:
Census Bureau mode report
- Run an overlap test with both modes in the same period.
- Review coverage, question order and routing.
- Document a series break and keep the earlier results.
- Start a new baseline and label the date.
- Report the size of the mode effect beside the trend.
Overlap testing costs time and sample.
Case 4: Pew bogus respondents and the speed cutoff
Pew Research Center's study of bogus respondents found that a small share of online respondents may be fraudulent or automated, and it examined speeding. A vendor that removes every fast complete mixes true shortcuts with short routing paths, experienced respondents and assistive-technology patterns.
Replace the single cutoff with layered review: device checks, open-end reads and consistency tests. Document exclusions and publish a sensitivity range. Restore valid records only where the evidence supports it. No screen catches every fraudulent case.
Case 5: AAPOR 2020 review and weights that hide coverage gaps
AAPOR's review of the 2020 election polls found that national polls overestimated Biden's margin by 3.9 percentage points. State polls missed by more in some cases. The review pointed to nonresponse bias and education weighting.
The University of Michigan's survey-weighting precision appendix shows that variable weights can raise sampling variance even when they correct for coverage, nonresponse or selection. A sample can match age and region benchmarks and still miss offline buyers.
Report weight concentration, effective sample size and a narrow population claim. Teams that track concentration can compare their numbers against market research surveys benchmarks for coverage and precision.
What the cases share
Each case keeps the original record, names the error's effect on the estimate and on the decision, corrects the downstream output, and states the remaining limit. None claims a full repair.
Any public performance claim drawn from a survey needs a reasonable basis before it runs. The FTC advertising substantiation policy is the U.S. standard to check. Keep the underlying tables.
Customer research case studies follow which decisions held up years later, which is the test these public cases set.
Common questions
Are these real company case studies?
They are real public cases: the Literary Digest poll, Pew Research Center studies, a U.S. Census Bureau working paper and the AAPOR 2020 election review. No client names are used.
Why keep the original wrong analysis?
It shows what changed, which decisions were exposed and how corrections propagated. The record is evidence, not clutter.
When should a tracking series be restarted?
When a mode, population or frame change leaves an unresolved break in comparability. The same applies to a question, order or translation change.
How large a shift should trigger a correction?
Any shift you cannot explain by design deserves a test. A few percentage points is enough to flip a close decision.
