the roots of total survey design lars lyberg stockholm university qmms seminar leinsweiler, nov 7-9,...

The Roots of Total Survey Design

Lars LybergStockholm University

QMMS Seminar Leinsweiler, Nov 7-9, 2010

Early thinkers Hansen and colleagues, U.S.

Bureau of the Census Deming, U.S. Bureau of the Census

and consultant Kish, University of Michigan Dalenius, Statistics Sweden and

Stockholm University

What were they thinking about? Nonsampling errors Balancing errors and costs Design criteria The limitations of sampling theory Standards Similarities between survey

implementation and the assembly line

4

Deming (1944) “On Errors in Surveys”

American Sociological Review! First listing of sources of problems,

beyond sampling, facing surveys The 13 factors

Deming’s 13 factors

The 13 factors that affect the usefulness of a survey

-To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available

-To point out the futility of concentrating on only one or two of them

-To point out the need for theories of bias and variability that correlate accumulated experience

6

Their difficult position They had to promote Neyman’s theory But his theory basically assumes very

small nonsampling errors They were in a first-things-first situation They promoted vigorous controls

hopefully leading to small biases They discussed what a Bayesian

approach might offer

Lines of thought I “There is as yet no universally

accepted ‘survey design formula’ that provides a solution to the design problem (Dalenius 1967)

That’s why textbooks devote little space to design

Important to control specific error sources

Lines of thought II The U.S. Bureau of the Census is a

statistical factory. The main product is statistical tables (Deming and Geoffrey 1941)

Concentration on QC of error sources, evaluation, and survey models

Disentangling the design process

Lines of thought III Hansen-Hurwitz-Pritzker 1967

Take all error sources into account Minimize all biases and select a minimum-

variance scheme so that Var becomes an approximation of (a decent) MSE

The zero defects movement that later became Six Sigma

Dalenius 1969 Total survey design

The design process Criterion of effectiveness: Minimum MSE

per unit of cost while meeting other requirements such as timeliness of results (not just minimum variance)

Good survey design calls for reasonably effective control of the accuracy through appropriate specifications for survey procedures and adequate control of the operations, i.e. proper design of the total system

Mean squared error (MSE) MSE=Var+B2+(Relevance

error)2+Interaction MSEZ(y)=E(y-Y)2+(Y-X)2+(X-Z)2

+2(Y-X)(X-Z)Z is the ideal goal, X is the defined goal,

and y is the actual result Hansen-Hurwitz-Pritzker call them

requirements (Z), specifications (X) and operations (y)

Design issues X-Z is crucial in the design

situation Do we want an approximate

solution to the right problem or an exact solution to the wrong problem?

The design approach (Dalenius and Hansen et al) Specify the ideal goal Z Analyze the survey situation

(financial, methodological and information resources)

Construct a small number of alternative designs

Evaluate the alternatives by reference to associated MSE and costs

The design approach (cont’d) Make a decision

Use one of the alternatives Use a modification of one of them Do not conduct the survey

Develop the administrative design Feasibility The signal system A self-contained design document (tree) Plan B

What does this tell us?

All error sources should be taken into account

There is very little process talk such as the need for CQI

However, the common situation was: no process view, no controls

Concern about costs and effectiveness of all these controls

The user is a somewhat distant player

The user The user was hiding under terms such

as subject matter problem, study purpose or “the four key functions of a statistical system” (reporting, analytic, consulting, research) Tukey 1949

But there were federal statistics users conferences in the U.S. from 1957. Dalenius provides more than 200 references on users in a 1967 ISI paper

Who identifies the requirements? Usually seen as one fictive person

An official An administrator A statistician acting as a subject-

matter specialist Requirements define the population,

types of measurement, time dimensions and statistics needed

The designer’s role vis-à-vis the requirements To critique the suggested

requirements To suggest QC procedures, construct

dummy tables to check the decision-making and perform sensitivity analysis

To act as the devil’s advocate and discuss specific result interpretations with the user

Kish’s contributions The neo-Bayesian view

Appreciates the literature by Schlaifer, Ericson, Edwards, Lindman and Savage on Bayesian methods in survey sampling and psychometrics

For instance, judgment estimates of measurement biases may be combined with sampling variances to construct more realistic estimates of the total survey error

More from Kish Experiments and sample surveys

might not be sufficient. Other investigations “collecting data with considerable care and control” but without randomization and probability sampling might be necessary.

Kish’s view on design Multipurpose is great from an economical

point of view. If one principal statistic can be identified

that alone can decide the design If a small number of principal statistics

can be identified a reasonable design compromise is possible

If statistics are too disparate a joint design might not be possible

Kish on economic design Requires joint consideration of sampling

and nonsampling errors Sometimes demands prior or pilot

studies of sufficient size Requires information about unit

variance Emphasizes a small total error Appreciates the fact that a reduction of

one source might increase total error

Examples of decisions Frame needs updating? Reference period? Acceptable respondent rules? Number of callbacks? Allocation of callbacks? How much and what kind of editing? Mix of modes?

Kish summed up Get a good balance between different

error sources We need to know how error structures

behave under different design alternatives Relevant information should be recorded

during implementation (paradata) Many practical constraints The multipurpose nature calls for a

compromise

Hansen, Dalenius and colleagues on standards General standards

Measurable survey plan, self-contained plan, replications should generate similar results, cost-efficient, sufficiently simple plan

Standards for error control Relevance control, control of accuracy (should

be dominated by variance terms) Minimum performance standards Check that standards yield the results expected

Hansen, Dalenius and colleagues summed up One should be guided by common sense,

experience and theory Design and execution is a management and

systems analysis problem A survey is an economic production process Survey goals must be identified Standards must be dynamic End the practice that sampling error is viewed

as the total error They predicted the CASM movement

More from Hansen, Dalenius and colleagues The examination of design alternatives is costly

and time-consuming There is a risk of overcontrol and inadequate

control. Consequences of large errors must guide any relaxation but they don’t talk about CQI

One might have to compromise relevance to get controllable measurements or abstain from the survey

Keep bias near zero and allow variance at expected levels

What happened? Still no “design formula” General design principles exist for some areas Still a concentration on some error sources

more than others CASM happened We got standards The TSE paradigm accepted but has some

promotional problems Many of the early thoughts were just that, very

little practice, but still useful