quality control and measurement of nonsamplingerror in the...
TRANSCRIPT
DATA EVALUATION AND METHODS RESEARCH
Quality Control and Measurement of
Nonsampling Error in the
Health Interview Survey
A report describing the selection and training of interviewers, interviewer
observation program, a reinterview program, measuresof interviewer perform-
ance, the editing and coding of questionnaires, a response error study, and an
interviewer variability study. The report is based on the Bureau of the Censusexperiencewith the Health Interview Survey.
DHEW Publication No. (HSM) 73-1328
U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFAREPublic Health Service
Health Services and Mental Health AdministrationNational Center for Health Statistics
Series 2Number 54
Rockville, Md. March 1973
Vital and Health Statistics-Series 2-No. 54
For sale by the Superintendent of Documents, U.S. Government Printing Offlce, Washington, D. C. 2M02Price 85 cents domestic postpaid or 60 cents GPO Bookstore
NAT ONAL CENTER FOR HEALTH STATIST CS
THEODORE D. WOOLSEY, Director
EDWARD B. PERRIN, Ph.D., Deputy Director
PHILIP S. LAWRENCE, SC.D., Associate Director
OSWALD K. SAGEN, Ph .D., Assistant Director for Health Statistics Development
WALT R. SIMMONS, M.A., Assistant Director for Research and Scientific Development
JOHN J. HANLON, M.D., Medical Advisor
JAMES E. KELLY, D.D.S., Dental Advisor
EDWARD E. MINTY, Executive Officer
ALICE HAYWOOD, Information Officer
OFFICE OF STATISTICAL METHODS
MONROE G. SIRKEN, Ph.D., Director
E. EARL BRYANT, M.A., Deputy Director
DIVISION OF HEALTH INTERVIEW STATISTICS
ELIJAH L. WHITE, Director
ROBERT R. FUCHSBERG, Deputy Director
RONALD W. WILSON, Chief, Analysis and Reports Branch
KENNETH W. HAASE, Chief Survey Methods Branch
COOPERATION OF THE BUREAU OF THE CENSUS
Under the legislation establishing the National Health Survey, the publicHealth Service is authorized to use, insofar as possible, the services orfacilities of other Federal, State, or private agencies.
In accordance with specifications established by the National Center forHealth Statistics, the Bureau of the Census, under a contractual arrange-ment, participated in planning the survey and collecting the data.
Vital and Health Statistics-Series 2-No. 54
DHEW Publication No. (HSM) 73-1328
Library of Congress Catalog Card Number 72-600133
FOREWORD
This report is one in a series designed to document themethodology of the Health Interview Survey (HIS) and toinvestigate the quality of HIS statistics. In previously publishedreports, the emphasis was on questionnaire development (Series 1,Number 2) and on sample design (Series A, Number 2). Otherreports (e.g., Series 2, Numbers 6, 7, 18, 28, and others) presentfindings of methodological studies that investigated the accuracyof health data collected in household surveys.
Specifically, this report deals with the quality-control proce-dures for the data-collection operations of the Survey. It describesprocedures for selecting, training, supervising, and observinginterviews and measuring interviewer performance, and for editingand coding questionnaires. It also describes the reinterviewprogram. In this program, a staff of field supervisors and seniorinterviewers reinterview sub samples of households in the Survey.This report presents estimates of nonsampling error based on thereinterview program and estimates of the interviewer contributionto nonsampling variance based on results of a special studydesigned for this purpose. These statistics have a dual utility. Theyare useful in evaluating the quzdity of HIS data and in improvingthe design of the Survey.
Through contractual arrangements with the National Center forHealth Statistics, the Bureau of the Census prepares the sampleand conducts the field collection process of the Health InterviewSurvey, and, until 1968, also carried out the data-coding andinitial editing procedures. The particular quality-control proce-dures described in this report are essentially applications ofmethods that are used by the Census Bureau to monitor the fieldoperations of national household surveys. However, the findingspresented in this report relate only to the Health Interview Survey.
The work for this report was done under a special contract withthe Statistical Research Division, Bureau of the. Census, in closecollaboration with the Office of Statistical Methods and theDivision of Health Interview Statistics.
Elijah L. WhiteDirectorDivision of Health Interview Statistics
Monroe G. SirkenDirector
Office of Statistical Methods
...Ill
SYMBOLS
Data not available ---------------------------------------- ---
Category not applicable ------------------------------- . . .
Quantity zero ----------------------------------------------
Quantity more than O but less than 0.05 ----- 0.0
Figure does not meet standards ofreliability or precision ------------------------------ *
CONTENTS
Page
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Purpose of the Health Interview Survey . . . . . . . . . . . . . . . . . .Brief Description of the Survey . . . . . . . . . . . . . . . . . . . . . . .Control of the Survey Process . . . . . . . . . . . . . . . . . . . . . . . .Measurement of Nonsampling Error . . . . . . . . . . . . . . . . . . . . .
Part L Control of Data Collection and Data Processing . . . . . . . . . . . .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Selection and Training of Interviewers . . . . . . . . . . . . . . . . . . .
Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Initial Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Continuing Training . . . . . . . . . . . . . . . . . . . . . . . . . . .
Observation of Interviewers.. . . . . . . . . . . . . . . . . . . . . . . .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Types of Observations . . . . . . . . . . . . . . . . . . . . . . . . . .
Supervisory Reinterview Program . . . . . . . . . . . . . . . . . . . . . .Introduction . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Sample Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Content of Reinterview . . . . . . . . . . . . . . . . . . . . . . . . . .Field Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Quality Control of Interviewers’ Work . . . . . . . . . . . . . . . . . .
Editing and Coding of Completed Questionnaires . . . . . . . . . . . . . .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Regional Office Edit . . . . . . . . . . . . . . . . . . . . . . . . . . .Quality Control of Clerical Coding Operations . . . . . . . . . . . . . .Central Office Edit.......,.. . . . . . . . . . . . . . . . . . . .
Measures of Interviewer Performance . . . . . . . . . . . . . . . . . . . .
Part 11. Measurement of Nonsampling Error . . . . . . . . . . . . . . . . . .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Response Errors as Determined by a Reinterview Survey . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .HIS Reinterview Survey Results . . . . . . . . . . . . . . . . . . . . .
Interviewer Variability Study . . . . . . . . . . . . . . . . . . . . . . . .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Design of the Interviewer Variance Study . . . . . . . . . . . . . . . .Method of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . .Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
...m
11123
33334566688888
10101011121313
1616171717242424262832
33
v
CONTENTS–Con.
Page
Appendix I. Formula for Computing Error Rate . . . . . . . . . . . . . .
Appendix II. Time and Cost Model for HIS Interviewing . . . . . . . . . .General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Models for HIS Assignments.. . . . . . . . . . . . . . . . . . . . . . . .
Appendix III. Some Theory of Measurement Errors . . . . . . . . . . . . .Some Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . ...The Desired Measure or True Value . . . . . . . . . . . . . . . . . . . . .The General Conditions That May Affect the Results of a Survey . . . . . .An Estimate from a Survey (or Trial) Taken Under a Set of
General Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . .The Mean Square Error of an Estimate from a Survey (or Trial) . . . . . .Gross and Net Differences . . . . . . . . . . . . . . . . . . . . . . . . . .Gross and Net Differences as Evidence of Response Variance
and Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Index of Inconsistency . . . . . . . . . . . . . . . . . . . . . . . . . . .
Appendix IV. Least-Squares Solution . . . . . . . . . . . . . . . . . . . .
Appendix V. HIS Observation Report (NHS-HIS-406) . . . . . . . . . . .
Appendix VI. HIS Reconciliation Form (NHS-HIS-R-IX-T) . . . . . . . . .
Appendix VII. Summary Report of NHS-HIS Reinterview (NHS-HIS-R-401) .
Appendix VIII. Production Guide for NHS (11-102C) . . . . . . . . . . . .
35
363636
38383838
393940
4142
43
45
48
50
52
QUALITY CONTROL AND MEASUREMENT
OF NO NSAMPLING ERROR
IN THE HEALTH INTERVIEW SURVEY
David A. Koons, Statistical Research Division, U.S. Bureau of the Census
INTRODUCTION
This report presents a summary of proceduresused in the Health Interview Survey (HIS) tocontrol the quality of the data collection anddata processing operations. It also provides someresults of measurements related to the quality ofHIS statistics.
PURPOSE OF THE HEALTHINTERVIEW SURVEY
The Health Interview Survey is an integralpart of the program of the National Center forHealth Statistics (NCHS). This program is de-signed to provide continuing statistical measure-ments of the extent of disease, disability, andother health characteristics of the population.
The legislation authorizing the HIS, TheNational Health Survey Act,l contains the fol-lowing provisions:
(b) It is, therefore, the purpose of this Act to provide (1) fora continuing survey and special studies to secure on anon-compulsory basis accurate and current statistical infor-mation on the amount, dutribution, and effects of illness anddisability in the United States and the services received for orbecause of such conditions: and (2) for studying methodsand survey techniques for securing such statistical informa-tion, with a view toward their continuing improvement.
This provision for methodological researchhas strongly influenced the NCHS program.Since the HIS began, emphasis has been placedon improving statistical output rather than oncontinuity and comparability of estimates.Changes to improve the methods and proceduresused in the survey have been made since it beganin 1957.
BRIEF DESCRIPTION OF THE SURVEY
A description of the HIS is necessary tounderstand the quality control procedures usedduring the collection and processing of data.
The Health Interview Survey uses a question-naire to obtain information on injuries, acuteillnesses, chronic conditions, impairments, utili-zation of medical services, and other healthtopics, in addition to information about per-sonal and demographic characteristics. Thefindings from the survey are tabulated for theNation as a whole and published by NCHS.Separate reports are issued which cover one ormore of the specific topics.
The population covered by the sample for theHealth Interview Survey is the civilian, noninsti-tutional population of the United States living atthe time of the interview. Persons in long-stayhospitals, nursing and convalescent homes, andso forth are excluded from the universe to besampled.
The sampling plan of the survey follows amultistage probability design which permits acontinuous sampl~mg of the civilian populationof the United States. The first stage of thisdesign consists of drawing a sample of 357 fromabout 1,900 U.S. geographic divisions calledprimary sampling units (PSU). A PSU is acounty, a group of contiguous counties, or astandard metropolitan statistical area (SMSA).
Within PSU’S, ultimate stage units calledsegments, selected from clusters of 18 neigh-boring households or addresses, are defined sothat each one contains an average of six house-holds. (In July 1968, the average segment size
1
changed from nine to six households.) Twogeneral types of segments are used: (1) areasegments, which are defined geographical y, and(2) other segments, which are defined from a listof addresses from the 1960 Decennial Censusand a current Survey of Construction.
Prior to interviewing in area segments, inter-viewers make a list of the addresses of allhouseholds or dwelling units in the selectedsegments.
Wherever possible, the visit of the intervieweris preceded by a letter from the Director of theU.S. Bureau of the Census announcing that aninterviewer may be expected to visit and settingforth the general purposes of the survey. Theconfidential treatment that will be accorded anyinformation given is emphasized.
As a general rule any adult member, 19 yearsof age and over, of a family may be interviewedconcerning the characteristics of all the membersof the family. Persons in the household who arenot related to the head of the household areexpected to answer all questions about them-selves. Exceptions are made for persons who arenot competent to answer for themselves. Personsaged 17-18 may respond for themselves, whilepersons under 17 must be responded for by anadult.
The sample is evenly distributed throughoutthe year, so that interviews are conducted inapproximately 800 households each week. Sincehousehold members interviewed each week are arandom sample of the population, samples forsuccessive weeks can be combined into largersamples. Thus the design permits both con-tinuous measurement of characteristics of highincidence or prevalence in the population and,through the larger consolidated samples, moredetailed analysis of less common characteristicsand smaller categories. This continuous collec-tion of information has administrative, opera-tional, and technical advantages since it permitsfield work to be handled by an experienced,stable staff. In addition, this design eliminatesbiases due to the seasonal nature of certainconditions or the occurrences of short-runepidemics.
Approximately 100 interviewers, about halfof whom work each week, are used in the HIS.Each interviewer is assigned an average of threesegments (about 18 households) as a week’s
work. (As of July 1968, an average interviewerassignment changed from two nine-householdsegments to three six-household segments.)
The interviewers, as well as the entire fieldstaff for the HIS, are employees of the U.S.Bureau of the Census. Specifications for thesurvey are established by the NCHS. In accord-ance ‘with these specifications, the U.S. Bureauof the Census selects the sample, conducts thefield interviewing as an agent of the Center, andchecks questionnaire entries. Data preparation,consisting of the preliminary editing and thecoding of questionnaires, is carried out by theNCHS. Further editing and preparation of tabu-
lations is done by NCHS using electroniccomputers.
The Bureau of the Census has 12 regionaloffices located in 12 major cities where super-visors of the HIS are stationed. Each supervisorspends a great deal of time visiting the approxi-mately 30 PSU’S in his region in which theinterviewing is carried out. Since there are threeto four PSU’S per interviewer, many of theinterviewers are also required to do a consider-able amount of travel.
CONTROL OF THE SURVEY PROCESS
The quality control program for the HIS hastwo purposes: to minimize errors in the surveyresults and to provide data to evaluate theextent of bias caused by interviewers and re-spondents.
Nonsampling errors can occur at any stage ofa survey. They may result from the improperstatement of the objectives, from faulty con-cepts, or from improper definition of the popu-lation to be studied. They may arise during thesample selection, during the conduct of aninterview, or during the processing of the data,e.g., coding, editing, or tabulating. Both the fieldand office quality control programs of the HISstrive to minimize these errors and to maintainthe quality of the interviewing and of theediting, coding, and other data-processingoperations.
The quality control activities in the field areprocess controls rather than product controls.That “is, very little work in the field is done overagain because it does not meet quality controlstandards. To control errors contributed by
2
interviewers, the program seeks to identifyinterviewers whose work is defective in partic-ular areas so that remedial action can be takento improve future work. Such remedial actiongenerally takes the form of retraining the inter-viewers on those aspects of the survey in whichtheir performance was poor. Sometimes, how-ever, interviewers cannot be helped by retrainingand must be replaced. Some intemiewers resignwhen they dkcover through the field qualitycontrol program that their performance is notup to par. The turnover, inchxling interviewerswhose performance is satisfactory but whoresign for personal and other reasons, is about15 to 25 persons per year among the approxi-mately 100 interviewers assigned to the HIS.However, about 40 percent of the interviewershave been with the survey for at least 5 years.
MEASUREMENT OF NONSAMPLING ERROR
Nonsarnpling errors that arise during theinterview may have as their source the respond-ent, the interviewer, or the questionnaire. Theymay result from such causes as respondentmemory lapse, the misunderstanding of aquestion, improperly omitted questions, orincomplete answers.
Two of the manymade to measure the
attempts that have beennonsamplin~ error in the
HIS are discussed in this report: -One attempt has been the systematic super-
visory reinterview which consists of reinterviewsconducted by the field supervisory staff andsenior interviewers at a subsample of householdsincluded in the sumey. The resuIts of thereinterview survey are compared on a case-by-.case basis with the results of the original survey.Data from these comparisons are presented asnet and gross differences. Net differences aredifferences between the statistics produced fromregular HIS interviews and the statistics pro-duced from the reinterviews. Gross differencesare disagreements in individual classificationsmade by the interviewers and the reinterviewers.A more detailed discussion of these measures isincluded in this report in the section part IIResponse Errors as Determined by a ReintemiewSurvey.
Another approach has been an interviewervariability study in which interviewer assign-ments were randomked to obtain estimates ofbetween-iiterviewer variance. (See InterviewerVariability Study in this report for a moredetailed discussion.)
PART 1. CONTROL OF DATA COLLECTION AND DATA PROCESSING
INTRODUCTION
Quality control is commonly achieved bymeasuring performance and setting standards.Illustrations of performance measures are nonin-terview rates, item-response rates, editing-failurerates, and error rates in clerical and card-punching work. Such measures are frequentlythe basis for applying formal quality controls inthe conduct of surveys. These quality controlsare specified in terms of minimum performancestandards that maintain the quality of work invarious operations and thus contribute to theaccuracy of survey results.
This part of the report describes the controlsimposed on the collection and processing of HISdata. Quality control measures are applied atfive different stages in the HIS: (1) interviewerselection, (2) training of interviewers, (3) obser-
vation of interviewers, (4) supervisory reinter-view, and (5) editing of the completedquestionnaires.
In the HIS, about 30 percent of the total fieldbudget goes into quality control. Table 1 pro-vides a distribution of the costs incurred byactivity for calendar year 1968.
SELECTION AND TRAINING OFINTERVIEWERS
Selection
HIS interviewers are selected with great care.Because of the potentially delicate nature of anHIS interview, candidates must have not onlythe necessary qualifications for handling theinterview questionnaire but also unusual tactand sensitivity.
3
Table 1. Percent distribution of field costs by detail expense item: Heelth Interview Survey, 1968
Detail expense item
Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Interviewing . . . . . . . . . . . . . . . . . . . . . . . . ........c.m-,.-. ..
Initial training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-
Groupandhometraining. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
ReinterviewZ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..
Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
0fficework3 . . . . . . . . . . . . . . . . . . . . . . . . . . .:::::........”. . . . . . . . . . .
Another . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
‘ Includes overhead.
2 Includes a check for completeness of coverage.
3 Includes preparation of reinterview assignments.
4 Less than 0.05 percent.
EmploymentasHIS interviewers islimitedtowomen. The typical respondent, a housewife, isgenerally thought to be more willing to revealcomplete health information to another womanthan to a man. In addition, employment ofwomen with formal nursing or medical trainingas HIS interviewers is discouraged because inter-viewers with such training may tend to diagnoseor interpret rather than merely record theinformation obtained from respondents duringthe interview. Interviewers must pass atest thatmeasures reading comprehension, arithmeticability, and map-reading ability. An elaborateprogram of training and observation early inaninterviewer’s career also appears to aid in theselection process. The very low level of refusal,about 1 percent ofhouseholds contacted, andofcomplaints received seems to show that theinterviewers who are finally selected appear tobe doing a good job of gaining publiccooperation.
Initial Training
The initial training consists of five separatestages: preclassroom training, classroom training,postclassroom training, on-the-job training, andediting of questionnaires by the supervisor.
Preclassroom training. –Preclassroom trainingis designed to familiarize the new interviewerswith both the purpose, scope, and general usesof the HIS and the interviewing materials andthe interviewing job.
Total
100.0
63.3
4.1
5.3
8.3
4.4
12.1
2.2 7Salaries Travel
47.6 27.7=
29.5 20.8
2.0 0,,8
2.3 1,,7
3.3 2.7
1,8
7.5 Ji21.2 0.3
3theri
24.7
13.3
1,2
1.3
2.3
1.4
4.6
0.6
The usual method of Dresentirw such trainhw.is through self-study materials. The interview;is given a self-study package to complete beforereporting for classroom training. The contentsinclude administrative materials, a copy of theHIS questionnaire, an interviewer’s manual withinstructions to read certain sections, and copiesof the letters that the respondents receive.Occasionally, in order to become betteracquainted with the survey, the new intervieweralso spends 1 day observing an experiencedinterviewer.
Classroom training and practice interview-ing. —Classroom training consists of 5 days ofinstruction, which covers the interviewer’smanual, the questionnaire and related forms,and interviewing techniques. This training isusually conducted in one of the 12 CensusRegional Offices, permitting the trainee tobecome acquainted with some of the regionaloffice staff members and with general officeprocedures.
The classroom portion of the initial trainingcombines formal classroom training with mockinterviewing. Mock or hypothetical interviewsare created from situations the interviewer mayface. The formal classroom training is primarilyfor teaching survey concepts.
Several training techniques besides mockinterviewing are employed in the classroom.They include lectures by the trainer, readingportions of the interviewer’s manual, answeringquestions, participating in group discussions, andcompleting written exercises.
4
Practice field interviewing gives the trainee achance to apply her knowledge of the surveymaterials to actual interviewing situations. Thisserves to familiarize her as emly as possibIe withthe work she will be doing as well as to stimulateher Iearning of the concepts and techniques byusing them under actual conditions.
An observer accompanies the interviewerduring practice interviewing assignments. Hecoaches her on how to handle difficult situationsand explains interviewing techniques. During theinterview, the observer does not interrupt thetrainee unless she becomes very confused. Itmay occasionally be necessary for the observerto conduct the first interview so the interviewercan get an idea of how the interview should beconducted.
The observer uses an observation report torecord all pertinent details of each interview,including any errors the interviewer makes.After they leave the household, the observerdiscusses with the interviewer the points that hehas marked on his report and gives her somehints on how to improve her interviewingtechniques or to solve problems that aroseduring the interview. He encourages her to lookup the solutions to problems in the interviewer’smanual.
In addition to evaluating the interviewer’stechnical understanding of the rules and defini-tions that apply to subject matter, the observerchecks the interviewer’s performance in thefollowing specified areas:
1. Introduction at the doorstep2. Use of identification card3. Explanation of survey4. Getting settled in the household5. Interviewer’s ability to maintain a busi-
nesslike but friendly attitude with thehousehold members
6. Ability to handle unusual or difficuhsituations
7. Adeptness with forms (i.e., following skippatterns, probing where answer is incom-plete, asking questions as worded, record-ing answers as instructed)
8. Dress and posture
Postclassroom training. –Postclassroom train-ing is designed to familiarize the interviewer
with rules and procedures that, although impor-tant, are not as frequently used as those coveredduring the classroom training.
The fiist postclassroom assignment is com-pleted after classroom training and before thefirst interviewing assignment. It consists ofreviewing classroom topics, completing a lessonthat describes the persons to be included in thesurvey along with a description of housing andsample units, and reading a discussion of admin-istrative forms the interviewer will use.
The second postclassroom training assignmentis completed at home by the interviewerbetween her first and second interviewing assign-ments. This assignment is designed to help theinterviewer understand sample unit coverage inaxea segments, the use and background of’ theaddress Iists taken from the 1960 DecennialCensus, and the procedures to be followed atspecial dwelling places, e.g., motels or convents.
The third postclassroom training assignment iscompleted at home by the interviewer imme-diately before her first listing assignment. A listof all housing units in each area segment must becompiled before any addresses can be selectedfor interviewing in these segments. The inter-viewer travels around the segment and recordsthe addresses or other description of all placeswhere people live or might live within thesegment.
On-the-job training. –On-the-job training isconducted by the supervisor-trainer during theinterviewer’s first two interviewing assignmentsand her first listing assignment. This type oftraining is usually referred to as initial observa-tion and is discussed more fuIly in the section“Observation of Interviewers.”
Edit of questionnaires. –All work of newinterviewers is edited by the field supervisor.This includes questionnaires for about 70 house-holds from about four assignments conductedover an 8-week period. A complete check ismade of the questionnaire, errors are identifiedand tzdlied, and the number and description ofthe errors is given to the interviewer.
Continuing Training
Several different kinds of continuing trainingare used.
5
Group training. –Group training is used forexperienced interviewers and normally takesplace twice a year. Interviewers are broughttogether in the 12 Regional Office cities. Thisprovides an opportunity for the widely scatteredinterviewers to meet one another, exchangeviews, and receive formal training. There is onesession in December or January of each year tolearn the new questionnaire for the next cal-endar year, and a midyear refresher sessionusually held in June or July.
Home study and exercises. –There is a formalhome-study program for which the interviewersare paid. Home-study assignments, which gener-ally take an average of 3 hours to complete, aremade four times a year. These assignments covervarious aspects of the interviewer’s job, such asthe order in which the questions should beasked, when a question is necessary, etc. Theyalso emphasize the importance of the survey andof the interviewer’s vital role in its operation.
Feedback of errors. –Supervisors in the
regional offices edit 1 week’s assignment perinterviewer per quarter. Additional editing isdone as needed, i.e., when previous editingresults, observation, or reinterview indicate anyconsistent type or pattern of omissions orinconsistencies. On the average, about one-sixthof the completed questionnaires are edited eachquarter in the regional offices. Certain categoriesof errors are identified and tallied, and the errorsare recorded on forms that are forwarded to theinterviewer immediately upon completion of theediting. For example, the interviewer may berequired to give an explanation of each error orto make a written reference to the part of theinterviewer’s manual that describes the correctprocedure.
A second edit is carried out at an early stageof the data-processing operation in Wash-ington, D.C. Here the data from all question-naires are examined. This edit takes place,however, weeks or even months after the ques-tionnaires have been filled out by the inter-viewers. Although the immediate feedback thatis provided by the field edit is lost, the degree ofuniformity in the detection of errors made byinterviewers becomes much greater in the centraloffice edit. This procedure therefore provides abetter basis for the numerical error scores that
are an important part of each interviewer’sperformance record.
Other training. –Informal training takes placeevery time supervisors and interviewers gettogether in connection with the quality controlprograms discussed in the next two sections.
In addition to formal training and the infor-mal meetings with supervisors, interviewers areencouraged to use referral sheets for describingproblems. They can mail these sheets directly tothe regional offices where solutions for theirproblems can be quickly determined and mailedback to them.
OBSERVATION OF INTERVIEWERS
Introduction
An important part of the quality controlprogram for the HIS is the observation ofinterviewers. Either the HIS Program Supervisor,Alternate Supervisor, or Senior Interviewerobserves in each regional office. An intervieweris observed in a group of households in herassignment. The observation program contrib-utes to on-the-job training as well as evaluationof the interviewer’s performance. The mainfocus of the observation is to see how theinterviewers conduct themselves in obtaininginformation in their assigned households. Thetype of controls possible through the observa-tion program depends a great deal on the abilityof the supervisor to detect inadequacies andcorrect them. An HIS observation report (appen-dix V) is used as a guide while observinginterviews. On it are recorded the observer’simpression of the interviewer’s performance.Some of the items are entered on a person-by-person basis, and some relate to the day’s work.A copy is placed in the interviewer’s perform-ance file in the regional office, and the original issent to the U.S. Bureau of the Census.
Types of Observations
Observations are classed as initial, systematic,and special needs. Although the same procedureis followed for all kinds of observations, theyhave different purposes.
6
Initial observations.-The purpose of the ini-tial observations is to give new interviewerson-the-job training to correct weaknesses at thebeginning of their interviewing career. Initialobservations are conducted for each new inter-viewer for 2 full days on her first interviewingassignment, for 1 full day on her secondinterviewing assignment, and for part of a dayon her first listing assignment.
The new interviewer begins her day bylocating the segment in which she will work. Theobserver allows the interviewer to find thesegment without guidance, unless she becomescompletely confused in reading the maps. If thisoccurs, he assists her in finding the first house-hold.
On interviewing assignments, the interviewerintroduces herself, and the observer merelyobserves unless the interviewer asks for help ormakes errors. If the interviewer is having seriousdifficulties, the observer then assists by con-ducting as much of the interview as needed toshow the interviewer the proper procedures.
At the end of the interview, the observerreviews with the interviewer any general prob-lems that have arisen, such as misapplication ofdefinitions or poor interviewing techniques. Hediscusses his notes with her immediately afterleaving the interviewed household and beforegoing to the next. He also reviews (edits) thequestionnaire for completeness. At the end ofthe observation, he may also review the inter-viewer’s time and mileage records and discussany general points he noted for specialattention.
Systematic observations. –Most observationsare regularly scheduled visits by the supervisor.These are called systematic observations and aredesigned to serve three broad purposes: to giveon-the-job training in areas where specific weak-nesses are observed; to allow each interviewer aregular opportunity to discuss her work with hersupervisor, to make suggestions, and to com-municate in general with the regional office; andto provide information for evaluation of theoverall quality of the interviewing in the HIS.
Systematic observation assignments are madeby the regional offices. One-half of the expe-rienced interviewers are observed each quarter.A systematic observation is made of newlytrained interviewers in the quarter following
their initialducted whenreinterview.
training. No observation is con-assignments are aIso scheduled for
Before conducting a systematic observation,the observer reviews records of the interviewer’s
past performance. In addition, he edits recentexamples of her listing of households, reviewsthe office copy of the report of her lastobservation, and, in general, tries to determinewhich points should be observed most closeIy.
Special-needs observations. —Some inter-viewers need more contact with their supervisorthan is provided by the systematic observations,and the regional offices need the flexibility ofbeing able to give additional training to inter-viewers when it is needed. Special-needsobservations are used for this additional training.
A special-needs observation is usually madefor an interviewer whose work is rejected inreinterview. Rejection in reinterview is based onthe number of differences between the originalinterviewer’s resuhs and the reinterviewer’sresults. An interviewer’s work is rejected if thenumber of differences is in excess of specifiedlimits given in a table of acceptability. (See thenext section for further discussion of the reinter-view program.) The special-needs observation isscheduled for the interviewer’s first assignmentfollowing the reinterview.
A special-needs observation may also be madefor an interviewer whose work falls belowcertain minimum performance standards such asthe following: poor production, e.g., too fewcompleted interviews per day or too much traveltime; a high nonintexview rate; an excessivenumber of recording errors and omissions on thequestionnaire filled out by the interviewer; andpoor performance on recent observation.
The purpose of the special-needs observationis retraining. If, in the judgment of the super-visor, an interviewer needs retraining after falhgbelow the minimum standard for some aspect ofher work, an observation is conducted. Some-times a seemingly poor performance can beexplained,for example, a high noninterview ratein the summer, and no retraining is necessary.Except in the case of rejection in reinterview,the decision to conduct a special-needs observa-tion is made entirely in
The procedure forvation is the same as
the ;egional offices.the special-needs obser-that for the systematic
7
observation, but with special attention given tothe aspect of the interviewer’s work that needsimprovement.
SUPERVISORY REINTERVIEW PROGRAM
Introduction
The major purpose of the program of super-visory reintemiews is to control quality. Theprogram provides a process control on the workof individual interviewers so that interviewerswith high levels of error can be identified andremedial action taken. The remedial action isintended to improve the quality of the individ-ual interviewer’s work through retraining, obser-vation, and discussion of errors with the inter-viewer, as necessary. The reinterview also servesas a periodic check on interviewers to see thatassignments are carried out as instructed.
This process also provides assessments of thereliability and accuracy of the HIS because thequality control technique employed in thisprogram requires that the supervisors fill outcomplete questionnaires that can then bematched with the questionnaires filled out bythe interviewers. Since these questionnaires con-stitute, by design at least, a probability sampleof all HIS interviews, estimates bearing on thereliability and accuracy of HIS statistics can bemade.
Sample Design
Reinterview assigriments are made on thebasis of interviewer workload, that is, inter-viewers with larger workloads have more reinter-views in a given year. A reinterview assignmentconsists of one weekly work assignment, andthere is an average of three reinterview assign-ments per year per interviewer. Originally, onereinterview assignment was carried out for eachinterviewer per quarter. In January 1963, theprogram was reduced to three assignments peryear per interviewer. On July 1, 1965, theselection method was changed to reflect thevariability of interviewer workload.
An additional reinterview assignment is madefor interviewers rejected in the previous quarter.The number of interviewers for which this is
done is limited to not more than 10 percent ofthe total number of interviewers. Reinterviewassignments are evenly divided among the weeksof the quarter, and there is only one reinterviewassignment in a regional office area in any givenweek. A reinterview of 12 of the 18 householdsin a typical interview assignment is conductedwith one person reinterviewed in each of the 12households selected.
The reintemiew sample is divided into twoparts. The sample of households selected forreinterview is subdivided into an 8 O-percentsubsample and a 20-percent subsample. In the80-percent subsarnple of households, the super-visor carries out a reconciliation of reinterviewresults with the results of the original interview.No reconciliation is carried out for persons inhouseholds designated for the 20-percent sub-sarnple. The division of the reinterview sampleinto an 8 O-percent subsample and a 2 O-percentsubsarnple began in January 1959. Before that,reconciliation was carried out for the entirereinterview sample.
Content of Reinterview
The first part of the reinterview is a coveragecheck to see if all household members have beenproperly incIudcd in the survey. The second partof the reinterview deals with the reporting ofpersonal and health characteristics.
In general, the reinterview covers all questionsrelating to the reporting of health conditionsand their characteristics originally included inthe first interview. Supplements, such as hospitalinsurance, eyeglasses , or hearing aids, are notusually included in the reinterview.
Field Procedures
At the beginning of each quarter, regionalsupervisors are told which weeks will have areinterview assignment. One week before inter-view week, the y are told which assignments areto be reinterviewed and given instructions forselecting the subsample of households desig-nated for reinterview and the sample personswithin reinterview households. One person perhousehold is randomly selected for th(: part ofthe reinterview that covers personal and healthcharacteristics.
8
The reinterviewer is instructed not to look atthe original interview results before reinterview.The HIS reconciliation questionnaire (appen-dix VI) containing the transcribed informationfrom the original interview is given to him in asezded envelope. He does not open this envelopeuntil he completes We reinterview.
For the 20-percent subsample of householdsfor which reconciliation is not carried out, theoriginal questionnaires are not transcribed. Forthese households, the note “Omit Content Rec-onciliation” is placed on the reconciliationquestionnaire inside a sealed envelope. Thus thesupervisor is not supposed to know in advancethe households where he will not do reconcili-ation. The data from the 2O-percent group areused to test the extent to which accessibility oforiginal responses to the reinterviewed hasapparently affected reinterview results.
The reinterview fieldwork incIudes verifyingthe originzd interviewer’s work in the listing ofaddresses in area segments, checking househoIdcomposition in the sample households, andreinterviewing one person in each reinterviewsample household.
The reinterview is scheduled for the weekfollowing the original interview and must becompleted no later than 2 weeks after the dateof the original interview. Since the questions onthe HIS schedule refer to specific time periods,such as “last week or the week be fore,” “a yearago /’ and “past 12 months,” the reinterviewer,in asking these questions, must be certain to getinformation for the same time period used bythe original interviewer. In order to do this, thereinterviewer must specify the exact dates of thereference period used in the original interview.
The reintemiewer makes a personal visit toeach household selected for reinterview. Thequestions relating to coverage of persons withinthe household may be asked of any eligibIerespondent. For the health information foradults, the most acceptable respondent is theperson who provided the data in the originalinterview. If he is not available, however, thesample person may be interviewed. (Before July1, 1965, the only acceptable respondent was thesample person himself.) Information for childrenis obtained from parents or an adult responsiblefor the child’s care.
Responses are entered on the reintemiew
questionnaire and changes are not made afterthis part of the reinterview is completed. (BeforeJanuary 1967 , supervisors used a special ques-tionnaire containing only the subjects covered inthe reinterview. Since then supervisors use ablank HIS questionnaire for recording the rein-terview restits.)
In 80 percent of the reinterviews, differencesin responses from the two interviews are recon-ciled immediately after completion of the re-interview. For personal characteristics, the re-interviewed transcribes the information he hasobtained to the reconciliation questionnaire. Hethen compares these answers with the originalresponses and reconciles any that are different.Next, he compares the responses to the healthquestions on the reinterview questionnaire withthe reconciliation questionniare, which containsthe information from the orighud interview. Ifthe reinterviewer finds that differences exist, heattempts to determine from the respondent theproper response and any possibIe reasons fordifferences. The reconciliation questionnaireprovides space for recording reasons given by therespondent for differences between the originalinterview and the reintemiew on the reporting ofibesses and other health conditions andhospitzdizations.
Table 2 shows the number of persons reinter-viewed in fiscal years 1959 through 1967.
Table 2. Number and percent of persons interviewed and rein-
terviewed: Health Interview Survey, fiscal years 1959-67
Fiscal year
1959 . . . . . . . . . . . .
1960 . . . . . . . . . . . .
1961 . . . . . . . . . . . .
1962 . . . . . . . . . . . .
1963 . . . . . . . . . . . .
1864 . . . . . . . . . . . .
1965 . . . . . . . . . . . .
1966 . . . . . . . . . . . .
1967 . . . . . . . . . . . .
Number of
completed
interviews
(persons)
126,841
118,068
112,086
118,432
138,055
128,801
138,152
138,486
133,916
Number of
completed
reinterviews
(persons)’
3,478
3,061
3,206
2J338
2~96
2,391
2,081
2,053
1333
2.7
2.6
2.9
2.4
2.2
1.8
1.5
1.5
1.4
lThe decline in the number of completed reintewiews is the
result of a cutback in the reintewiew sample size.
9
Quality Control of Interviewers’ Work
One purpose of the supervisory reinterviewprogram is quality control. Specifically, theprogram is designed to check on coverage andcontent errors. Errors in coverage of the popula-tion can occur because of incorrect listing ofaddresses in sample segments, failure to conductinterviews at the correct addresses, and incorrectapplication of definitions of housing unit andhousehold member. Content errors are errors inthe data obtained by the interviewer concerningpersonal and health characteristics of membersof the sample household. Through the recon-ciliation of original interview and reinterviewresults, the supervisor tries to obtain, the bestanswers to the HIS questions.
A second purpose of the reinterview program,that of obtaining measures of nonsamplingerrors and biases, is discussed in part II of thisreport.
After a reinterview assignment has been com-pleted, the reinterviewer completes a summaryreport of the HIS reinterview (appendix VII)showing the number of differences for fivecategories of the interviewer’s work: listing;household composition; personal characteristics;characteristics of conditions and hospitaliza-tions; and number of conditions, hospitalization,and injuries. Tolerance limits a~e establishedfor each category. The interviewer’s work isrequired to meet the standards for each categoryseparately. The tolerance limits are listed in atable of acceptability, which shows for eachcategory separately the number of differencesthat are acceptable for a particular sample size.Only cases where the respondent was the sameon both interviews are used in the table ofacceptability. The acceptance numbers are set sothat a difference rate at a 5-percent level will beaccepted 95 percent of the time.
An interviewer’s work is rejected if thenumber of differences in any classification is inexcess of the numbers given in the table ofacceptability. An analysis of reintewiew assign-ments was carried out for the period July 1,1962, through June 30, 1967. During this time,1,554 original interview assignments were re-interviewed. Rejections were noted in 115assignments in one or more categories. These
115 assignments were rejected on the followinggrounds:
Category of rejection
All categories of rejection . . . . . . . . . . . . . .
Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Houshold composition . . . . . . . . . . . . . . . . . . . .
Personal characteristics . . . . . . . . . . . . . . . . . . . .
Characteristics of conditions and hospitalizations . . .
Number of conditions and hospitalizations . . . . . . .
Number
139
18
7
31
30
53
Of the 115 assignments that were rejected, 95were rejected on one category, 16 were rejectedon two categories, and 4 were rejected on threecategories. No assignments were rejected on morethan three categories.
Supervisors in the regional offices initiateretraining, observations, etc. of interviewerswhose work is rejected. A report of actionstziken is made to headquarters in Washing-ton, D.C. Table 3 presents the actions taken forthose interviewers whose assignments wererejected in reintemiew by reason for rejectionfor the period July 1, 1962, through June 30,1967.
EDITING AND CODING OFCOMPLETED QUESTIONNAIRES
Introduction
When completed assignments are received inthe regional offices from interviewers, the ques-tionnaires are edited for consistency and com-pleteness. A systematic edit is carried out for aspecified sample of assignments. The results ofthis edit are sent to the interviewers withidentification of errors and specific references tosections of the interviewer’s manual to review.Further editing is carried out during coding andprocessing operations at headquarters. TheNCHS assumed responsibility for the coding anddata preparation in 1968. A new questionnaireformat and new coding procedures were adoptedat that time. The coding and quality controlprocedures described in this report are thoseused by the Bureau of the Census before 1968.
About 800 household questionnaires are re-ceived each week for processing. After the
10
r
Table 3. Percent distribution of actions taken in cases of interviewer rejection by nature of rejection: Health Interview survey,
July 1, 1962-June 30, 1967
ICategory Of rejection
Action taken after rejectionHousehold
ListingPersonal
TablesHealth
composition characteristics conditions
Total rejections
Retrained . . . . . . .
Observed . . . . . . .
Retrained and observed
Errors discussed . . .
Resigned . . . . . . .
Dismissed . . . . . . ,
No action necessary .
Action not reported .
Percent distribution
. . . . . . . . . . . . . . . . . . . . . . . 100.0 100.0 100.0 100.0 100.0
. . . . . . . . . . . . . . . . . . . . . . . 27.6 14.3 3.2 3.3 3.8
. . . . . . . . . . . . . . . . . . . . . . . 33.3 14.3 64.5 40.0 41.5
. . . . . . . . . . . . . . . . . . . . . . . 5.6 14.3 12.9 30.0 15.1
. . . . . . . . . . . . . . . . . . . . . . . 22.2 14.3 9.7 3.3 9.4
. . . . . . . . . . . . . . . . . . . . . . . 0.0 0.0 0.0 0.0 1.9
. . . . . . . . . . . . . . . . . . . . . . . 0.0 0.0 0.0 3.3
. . . . . . . . . . . . . . . . . . . . . . . 0.0 28.6 6.5 10.0 &
. . . . . . . . . . . . . . . . . . . . . . . 11.1 14.3 3.2 10.0 7.5
] The supervisor may decide that no actions are necessary if, in his judgment, the differences are due to factors beyond the
control of the interviewer. For example, one confused respondent may contribute al I the differences because he misunderstood
questions.
questionnaires are checked in, they are groupedinto work units of approximately 25 question-naires each. The questionnaires are assigned inwork units to clerks who check the question-naires for completeness, assign codes to theinformation on the questionnaires, and tran-scribe aIl of the information on the question-naires to punch card transcription sheets. Ques-tionnaires go. through nonmedical coding andmedical coding operations. Nonmedical codingassigns codes to the demographic items anditems related to health conditions. Medicalcoding, which is a more complicated operation,assigns detailed diagnostic codes to the illnesses,injuries, and h“ospitalizations reported on thequestionnaires. Diagnostic codes are assigned,with some modification, according to Inter-national Classification of Diseases (ICD). Tocontrol the level of errors in these codingoperations, specifllc quality control proceduresare followed. These procedures are discussed inthe section on quality control of clerical codingoperations.
Regional Office Edit
Specifications for carrying out a regionaloffice edit are sent to each office at the
beginning of a year. These specifications definethe minimum editing that must be done. Addi-tional editing is carried out on the basis of need,i.e., if previous edit results, observation results,or interview resuIts show errors such as omis-sions and inconsistencies.
The specifications for editing the work ofexperienced interviewers generally provide formore editing at the beginning of the year whennew items are added to the questionnaire. Asinterviewers become more experienced with newitems, the amount of editing is. reduced. How-ever, the first four assignments of new inter-viewers are always edited.
There are two types of edits performed in theregional offices, diagnostic and nondiagnostic.The diagnostic edit must be done by the HISsupervisor. Errors are assigned for missing orinadequate entries for ilhesses, injuries, or hos-pitalizations. The nondiagnostic edit can bedone by a qualified clerk and consists ofidentifying omissions and incorrect entries inidentification and control items on the ques-tionnaire.
The results of the diagnostic and nondiag-nostic editing are sent to the interviewers andprovide some immediate feedback on errors. Acopy of the results is retained in the regional
11
office for comparison with the more intensiveedit and identification of errors made during thecentral office coding and processing operations.
Quality Control of Clerical CodingOperations
There have been a number of changes over theyears in the processing of HIS question-naires. From the beginning of the survey in 1957until November 1965, processing consisted oftranscribing and coding information from ques-tionnaires to document-sensing cards fromwhich IBM punchcards were mechanically pre-pared. The assignment of diagnostic codes toillnesses, injuries, and hospitalizations was inde-pendently verified on a 100-percent basis. Twocoders independently assigned diagnostic codeson the information in the questionnaire. Thesecodes were compared, and differences wereresolved by a supervisor. The coding and tran-scription of nonmedical entries was completelyverified by a second coder’s examining theentries on the document-sensing card to see ifthey had been correctly transcribed by the firstcoder.
In November 1965, a new schedule formatwas introduced into the survey. Entries on thisschedule could be read directly by machine, thusbypassing a large amount of clerical transcrip-tion. In addition some of the codes for non-medical items, such as age, were entered on theschedule by interviewers and required no furthercoding. However, a substantial amount ofediting and coding was still required, particularlyfor diagnostic entries that had to be medicallycoded. At the time this new schedule wasintroduced, it was decided that sample verifica-tion to control the quality of coding wouldprovide a better use of resources than 100-percent verification, particularly for medicalcoding.
Consequently, starting in November 1965, asample verification plan was introduced into themedical coding operations of the survey. Theplan provides for two stages of control in themedical coding operation: a training and quali-fication period during which the coders’ work isindependently verified 100 percent, and a post-training period during which the coder’s work isindependently verified on a 10-percent sample
basis. All errors detected during verification arecorrected.
During the qualification period the new codercodes to a work sheet. Then the coding is doneover again by a qualified coder independently onan HIS schedule. A comparison clerk matchesthe medical codes entered on the worksheetwith the medical codes on the correspondingschedules. Differences in medical codes arereviewed by an adjudicator. The adjudicatorassigns an error if, in his judgment, the originalcoder assigned the wrong code. If, however, thedifferences in codes are a matter of coderjudgment, a decision concerning the proper codeis made, but an error is not assigned.
In order to qualify for sample verification, acoder must code four consecutive work unitsout of a maximum of eight with an error rate of4 percent or less for each work unit. If a coderfails to qualify within the first sequence of eightwork units coded, a second sequence of eight forquahfication is started. A coder has a maximumof three sequences in which to qualify. If acoder fails to qualify in the third sequence ofeight work units, he is not considered for sampIeverification. Once a coder has qualified forsample verification, his work continues to beverified on a 10-percent sample basis. A recordof verification is maintained for’ each medicalcoder. When the cumulative number of verifiedcodes reaches 45, a decision is made to deter-mine if the coder’s work is still acceptable. If acoder’s work is rejected three or more times in10 decisions, he must requalify for sampleverification. During the requalification periodhis work is verified on a 100-percent basis. If thecoder fails to requalify, he is no longerconsidered for sample verification.
Additional changes have been made in theformat of the questionnaire since the sampleverification plan was introduced. However,essentially the same verification procedures con-tinue to be used, i.e., independent verification ofmedical coding on a sample basis and 100-percent dependent verification of nonmedicalcoding.
Records for the period April 1, 1967, toMarch 31, 1968, show an average error rate ofabout 2 percent in assignment of medical codesfor experienced coders on sample verification.
12
For the same period, the nonmedical codingerror rate was also about 2 percent.
Central Office Edit
As part of the coding operations describedabove, a comprehensive review of each question-naire is carried out to identify omissions, inade-quate entries, and inconsistencies. Error codesare entered on the processing record so that thenumber and identification of errors can betabulated and summarized for each interviewer.In addition, specific descriptions of errors areprovided on a separate document. These descrip-tions identify the number and type of errors fordifferent sections of the questionnaire, forexample, person page, condition page, hospitali-zation page, etc. These errors are divided intotwo main categories: diagnostic errors on condi-tions and hospitalizations, and nondiagnosticerrors. The forms used to record the number andtypes of errors are the same used in the regionaloffice edit. Weekly summaries of diagnosticerrors are sent to regional offices, and quarterlysummaries of both diagnostic and nondiagnosticerrors are sent to the regional offices which inturn notify individual interviewers. The quar-terly summaries also form the basis for com-puting an interviewer error rate, which is one ofthe measures used to evaluate interviewer per-formance as described in the next section.
Additional editing is done on the computer,which performs a series of adequacy and con-sistency edits. Individual records with errors areidentified, the original questionnaires arelocated, and corrections made, as necessary, tothe records.
MEASURES OF INTERVIEWERPERFORMANCE
In the preceding sections of this report theactivities for controlling the quality of surveyresults have been presented. Results from thesequality control activities are combined withother data to provide an overall evaluation ofinterviewer performance.
The measurement of interviewer performancein the HIS is a combination of subjective ratingsby supervisors and quantitative measures based
on an examination of an interviewer’s completedwork.
Minimum standards of performance on thequantitative measures are set up. A cumulativerecord of performance for each interviewer ismaintained in the regional office. If, at any time,this record indicates that an interviewer’s workhas fallen below the minimum standard, correc-tive action is taken. This corrective action mayconsist of retraining, observation, or, in somecases, replacement of the interviewer. In practicethe evaluation of interviewer performance isbased on the pattern of performance over timeand on different aspects of the interviewing jobrather than performance on any single aspect.
In the HIS, three quantitative measures ofperformance are computed on a continuingbasis. They are the error score, the noninterviewrate, and the production ratio.
The error score is computed as folIows:(number of errors)/(total conditions + total acci-dents + total hospitalizations). (See appendix I.)The numerator is the number of errors identifiedduring processing. Errors include omittedentries, missed conditions, missed hospitaliza-tions, and diagnostic errors. Missed conditionsand missed hospitalizations are those identifiedin the early or probing section of the interviewbut not followed up for additional informationin the latter section of the interview. Diagnosticerrors occur when the interviewer faik to recordsufficient information to allow a medical coderto assign diagnostic codes.
The noninterview rate is computed as follows:(number of noninterview households)/(numberof interviewed households + number of non-interview households). The nonintefiew house-holds are households eligible to be included inthe HIS, but for which no interview wasconducted. Included as noninterviews are thosethe interviewer has reported as “refusals,” “noone at home,” “ temporarily absent,” etc.
The production ratio is measured as follows:(estimated time based on production stand-ards)/ (actual payroll time charged by the inter-viewer). The numerator is estimated from amathematical equation (appendix II) that takesinto account such things as the average time perhousehold, the number of assigned households,and the distance to area of assignment frominterviewer’s home.
13
In addition to quantitative measures used toevaluate interviewer performance, the results ofsupervisory reinterviews and observations areused as much as possible. Also, if specificindividual interviewer errors are discoveredduring the processing operation at the centraloffice, they are noted and forwarded to theregional supervisor. He in turn informs theinterviewer of these errors and suggests means ofeliminating them.
After a probationary period of 6 months,each interviewer receives a report on her per-formance over the past quarter. The reportcontains both a descriptive rating and a numeri-cal score. The descriptive ratings are “Excel-lent,” “ Satisfactory, “ “Needs Improvement:’ or“Unsatisfactory.” If an interviewer receives arating of “Needs Improvement” or “Unsatis-factory,” she will receive a warning notice
unless, in the judgment of her supervisor, thereare extenuating circumstances.
In addition to the quantitative measures usedin evaluating individuzd interviewer performance,other measures are used to provide an overallsummary of performance. These include numberof conditions per person,a number of missedconditions and hospitalizations, number of diag-nostic errors, and proportion of reinterviewassignments accepted. (See “Regional OfficeEdit” for a discussion of diagnostic errors.)Tables 4 and 5 show the average rates for HISinterviewers over a 4-year period for five ofthese rates and over a 7%-year period for threeof these rates. The variation from quarter toquarter is small; the only apparent trend being inthe number of conditions per person, wlhich hassteadily increased throughout the survey.
Table 6 shows average rates for interviewers
Table 4. Average interviewer performance on various measures by survey quarters: Health Interview Survey, January 1962-June 1965
Survey quarter
1962:Jan.-Mar. . . . . . . . .
Apr.June . . . . . . .
July -Sept. . . . . . . .
Oct.-Dee. . . . . . . . .
1963:
Jan.-Mar. . . . . . . . .
Apr.-June . . . . . . .
July -Sept. . . . . . . .
Oct.-Dee. . . . . . . . .
1964:
Jan.-Mar. . . . . . . . .
Apr.-June . . . . . . .
July -Sept. . . . . . . .
Oct.-Dee. . . . . . . . .
1965:
Jan.-Mar. . . . . . . . .
Apr.-June . . . . . . .
Number
of inter-
viewers
107
128
120
117
112
104
99
104
98
98
102
101
100
97
Average
error rate
(percent)
3ate
4.6
5.8
I 0.0
8.3
5.8
5.9
5.0
6.9
7.3
6.9
6.8
6.9
7.8
6.4
N’
105
128
119
116
112
103
98
104
98
98
101
101
100
97
Average number of
missed conditions
per 100 persons
Rate
.10
.25
.30
.38
.32
.23
.09
.31
.40
.33
.28
.42
.16
.12
N
105
127
119
116
112
103
98
104
98
98
101
100
100
97
Average number
of missed hos-
pitalizations
per 1,000 persons
Rate
.64
.83
.45
.68
.49
.64
.24
.45
.42
.39
24.46
.42
.66
.58
N
105
127
118
115
112
103
98
104
98
98
101
100
100
97
Ratio
1.09
.91
.89
.80
.99
.97
1.01
.97
1.06
1.03
1.00
1.02
1.05
1.03
N Proportion
100 .95
114 .95
110 .95
103 .91
91 .94
87 .95
91 .85
98 .80,
92 .89
94 .95
100 .92!
106 .89
96 .90
96 .92
N
94
95
92
94
62
66
61
70
61
64
48
65
68
61
IN = number of interviewers included in the computations.
‘This high rate is due to one interviewer who interviewed 10 persons but had 4 missed hospitalizations. Excluding the inter-
viewers work, the measure is .54.
aBecause of the differences from one assignment to anothezin the population covered, the number of conditions per personis not used as a performance measure for indkidual int ewiewers.
14
Table 5. Average interviewer performance on various measures by survey quarters: Health Interview Survey, January 1958-
June 1965
Survey quarter
1958:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1959:
Jan.-Mar.2 . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1960:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1961:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1862:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1863:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July+ept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1964:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
July-Sept . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oct.-Dee . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1865:
Jan.-Mar . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Apr.-June . . . . . . . . . . . . . . . . . . . . . . . . . .
Number
of inter-
viewers
102103
104
108
87
92
94
88
85
93
93
98
103
108
109
108
107
128
120
117
112
104
99
104
98
98
102
101
100
97
Average
non interview
rate (percent:
Rate
4.8
4.3
5.6
4.6
4.4
7.4
4.7
4.0
3.4
6.2
3.8
4.7
4.3
5.7
4.5
4.7
4.5
7.0
4.9
3.6
4.3
5.3
3.5
3.7,
3.9
4.8
3.6
4.0
4.2
N’
1021021041Oe
84
93
85
82
92
92
98
103
108
108
108
105
128
119
116
112
103
99
104
98
98
101
101
100
97
Average number
of conditions
per person
Rate
.94
.92
.84
.90
.91
.86
.80
.87
.93
.95
.94
1.00
.97
.88
1.00
1.07
1.02
1.01
1.04
1.08
1 .!34
1.03
1.06
1.08
1.06
1 .m
1.12
1.12
1.14
N
102
102
104
106
84
92
84
82
93
92
98
99
105
108
109
105
128
119
116
112
103
99
104
88
88
101
100
100
J97
Average number of
diagnostic errors
per 100 conditions
Rate
4.8
4.6
5.1
4.2
4.4
4.2
3.7
2.4
2.1
2.1
1.4
2.3
2.7
3.3
3.8
3.4
5.0
6.0
5.9
4.2
4.1
3.0
3.6
3.9
3.6
5.3
4.0
4.1
3.7
N
102
102
104
106
84
91
84
82
93
92
98
99
105
109
108
105
128
119
116
112
103
99
104
98
98
101
100
100
97
‘N= number of interviewers included in thecomputetions.
2No information available for this quarter.
15
Table G. Amm@inmwieMr petiormance onvarious measurS bynumber afquartom of experience
—
Akarageerror’ rate[percent)
Averagenoninterview
rate (percmt)
Average number
of conditionsper person
Average numberof dkgnostic
errors per 1Wconditions
Average number
of missed
renditions per
100 persons
Awrage numberof missed hospi-
talizations par1,000 persons
Average propor-
tion of reintewiew
assignments accepted
—
Averageproduction
ratio
Number of
quarters of
expwien co
~umberof inter-viewers —
N’—
48
464641383838
33
3334
323439415655474844413837363430
2828241715—
Rate N Rate
—N—
41
424237333442
373337363541
424646
474844402$37363430
2828231715—
Rate N Rate N NRate Rate N N
97
9490827775
7467
6363
605858
575655
4848444138373634302928241715
12.3
12.99.47.56.05.35.7
4.14.44.6
4.94.44.4
4.93.74.24.43.94.54.73.53.44.04.34.4
4.44.34.23.83.5
—
5.24.14.44.36.04.04.4
3.53.83.3
3.73.14.2
2.94.23.84.13.24.13.64.03.93.73,5
3.02.86.32.33.73.2
—
97
918980627374
67
6363
605858
575655
474344413337%34302628241715
—
1.01.96.95
1.001.061.02
1.01
.991.011.02
1.021.011.05
.871.011.051.081.051.051.031.051.051.101.061.051.121.061.141.211.27
9589
8981627374
676363
605858
5766554748444138313634
302828241715
6.95.94.74.13.82.83.1
2.42.32.42.22.42.6
2.72.52.92.62.62.62.82.22.01.92.2
2.22,22.32.12.32.2
9589
8981527374
676363
w5858575855474844412837263430
2626241716
.40
.26
.21
.26
.20,14.10.12
.10
.12
.15
.16
.14
.19
.16
.12
.16
.03
.14
.17
.28
.15
.24
.24
.24
.26
.20
.22
.03
.03
45
48
48
44404040
393540
383942
4448&474844413837383430
2828241715
.761.51
.30
.781.511.00
.49
.28
.15
.28
.12
.62
.=
.49
.11
.20
.401.13
.00
.37
.28
.13
.26
.19
.23
.07
.59
.42
.00
.00
48
46474240534040
3s38
3s3840
4348484748444138373634
m2828241715
.71
.76
.90
.951.041.05
.991.00
1.051.07
1.051.061.10
1.051.101.051.101.101.031.011.031.071.101.081.02
1.051.061.051.071.12—
.79
.83
.69
,83“91“97.94
1.00
.86
.89
.931.001.00
,91.94
1.00
,88
.85
.95
.93
.87,96,96.86.95
1.001.00
.88
.s01.00
19
3535292329
31292436292331
323634413837303023252220
1918171010
18efore survey quarter July.Sept. 1961, the definition of error rate was actually an omission rate. Therefore rates for quarters of experience 1-14 exclude quarters before July-
Sept. 1961.‘N = number of interviewers included i“ comp”tmicms.
NOTE: Restricted to HIS interviewers employed during the period Apr.-June 1965.
by the number of quarters of experience in the achieve maximum performance in terms of errorHIS. As expected, the more experienced integ- rates, nonintewiew rates, and production. How-viewers have a better performance, on the ever, the data in table 6 do not represent a pureaverage, than new interviewers. From this table learning curve since the same interviewers areit appears that about eight quarters, or 2 years, not included in all quarters.
of experience are necessary for interviewers to
PART Il. MEASUREMENT OF NO NSAMPLING ERROR
INTRODUCTION Estimates of total sampling variance for im-portant statistics can be made more or less
A sample survey must take into account routinely. The estimation of response variancesnonsampling errors and methods of control as and variances contributed by other aspects ofwell as sampling errors. The allocation of the survey process, e.g., editing and coding, is
resources between control of nonsampling errors more difficult. Particularly difficult is the esti-and increasequestion, anhere.
16
in sample sizeanswer to which
is a c“om~licated mation of biases in the measurement process.is not attempted This part of the report describes two pro-
grams carried out in connection with the HIS to
obtain estimates of nonsampling error, includingbias. One is the supervisory reinterview programdescribed in part I, which, in addition to servingas a field quality control device, provides overallestimates of response variance and bias. Thesecond is a study designed to measure theinterviewer contribution to the variance ofestimates from the survey.
RESPONSE ERRORS AS DETERMINEDBY A REINTERVIEW SURVEY
Introduction
In the HIS, information is obtained by per-sonal inquiry or a self-administered questionnaireon age, number of chronic conditions, numberof hospital episodes, dlability, and other charac-teristics. The set of measurements or observa-tions recorded in the collection operationordinarily is examined for internal consistencyand acceptability, certain “corrections” aremade, and some of the entries coded to identifythem in a classification system. Results are thensummarized into totals, averages, correlations, orother statistical measures. Taken together, thecollection and processing operations constitutethe measurement process and are the source ofany measurement errors.
The interpretation of reinterview surveyresults or comparisons of results from a surveywith case-by-case matched responses or measure-ments from some other source has been thesubject of much research and study.z’9 Sometheory of measurement errors that may help @the interpretations of the results of two sets ofmeasurements i,s presented in appendix III. Thefirst set of measurements is obtained by theregular survey procedures. The second set isobtained from reinterviews or through matchingof survey results , unit by unit, with recordsproviding information similar to that obtained inthe survey.
HIS Reinterview Survey Results
The use of the reinterview program as a devicefor evaluating the. reliability and accuracy ofstatistics of the HIS is a byproduct use. There
are two important. respects in which the super-visory reinterviews do not meet the standardsthat are imposed for the original interviews.First, the supervisory reinterviews are by andlarge conducted by men, whereas it is a require-ment that HIS interviewers be women. Second,there is a longer time interval between thereporting and occurrence of heahh-relatedevents in the reinterviews than in the originalinterviews. The reinterviews occur at least aweek later than the original interviews. Thereare, however, some offsetting factors. For 80percent of the households selected for reinter-view, the reinterviewer has the benefit of theresults from the original interview. Where differ-ences exist, the reinterviewer is to determine theproper answer and also possible reasons for thedifference. It seems reasonable that in generalbetter responses would be obtained from recon-ciliation of two interviews than from a singleinterview.
The original survey data can be comparedwith reinterview data under three procedures.
Procedure 1.–The reinterview in 20 percent ofthe households in the reintefiew sample isconducted without the results of the originalinterview being available to the reinterviewer.No reconciliation of resuIts is carried out.
Procedure H. –The results of the originalinterview are available to the reinterviewer for80 percent of the households in the reinterviewsample. However, the reintewiewer is not toexamine the results of the original interviewuntil after a reinterview has been completed.
Procedure II is a comparison of the results of theoriginal interview with the reinterview beforeany reconciliation of responses in the twointerviews is carried out. If the reintemiewerfollows instructions, this comparison is the sameone as procedure 1.
Procedure III. –After conducting the reinter-view in 80 percent of the households, thereintemiewer compares the responses obtainedin the two interviews. Where differences exist,the reinterviewed tries, with the help of therespondent, to decide upon the proper response.Results of this reconciliation are compared withoriginal results under procedure III.
Summary measures. –To analyze the dataobtained from a case-by-case comparison of an
17
original and a reintemiew survey, certain sum-mary measures should be defined. Table 7compares the results of an original survey with areinterview survey. The total number of differ-ences affecting the tabulated figure for a givenclass is equal to the number of cases included inthat class in the original survey but not in thereinterview survey plus the n“umber of casesincluded in the reinterview survey but not in theoriginal survey. This sum is called the grossdifference for the class in question. In terms oftable 7, b + c is the gross difference, and(b+ c)/rz is the gross difference rate.
The net difference of the tabulated figure fora given class is the difference between the totalfor the class obtained in the reinterview and theoriginal surveys. The gross difference usuallyincludes differences in both directions thatpartly or substantially offset eaeh other. The netdifference is the nonoffsetting part of the grossdifference. In table 7, the net difference is b - c,and (b - c)/n is the net difference rate.
Net differences. –Table 8 summarizes the netdifference rates for procedure III for 7%+yearaverages. Since procedure III provided an oppor-tunity for reconciliation of differences, theestimated net differences obtained from it areregarded as the best estimates of bias that thesupervisory reinterview program can provide.Except for persons with one or more chronicconditions, the net difference rates would beregarded as small by almost any standard.However, as indicated in the table, all of the netdifference rates are statistically significant; i.e.,significantly greater than zero.
Many of the results from the HIS are pub-lished as rates per person. Table 9 presents rates
from the reinterview survey for the originalinterview and for the reinterview after recon-ciliation. The percent net differences shown inthe table can be considered as an estimate of therelative bias of the original survey results.According to the reintemiew, chronic conditionstend to be underreported by about 24 percent.Disability days are underreported by about 13to 18 percent. Table 9 shows that hospitalepisodes and hospital days are better reported.Estimates made from 1959-61 reinterviewsurvey results show percent net differences ofabout 8 percent for hospital episodes and about5 percent for hospital days.
Gross differences and the index of inconsist-ency. —Gross differences are differences in indi-vidual classifications between the original inter-view and the reinterview. As discussed inappendix III the gross difference rate can beused to estimate the simple response variance ofthe original survey estimates, that is, the basictrial-to-trial variability in survey responses.(Appendix III also shows the derivation of anindex of inconsistency based on the grossdifference rate. This index provides a measure ofthe unreliability or inconsistency of classifi-cation and is defined as the ratio of the simpleresponse variance to the total variance.)
Reinterview without reconciliation (proce-dure I) provides the best estimate of simpleresponse variance. However, the data from pro-cedure I were tabulated only for fiscal years1959-61. The gross difference rates and indexesof inconsistency shown in table 10 are based ondata from procedure III after reconciliation.
Table 11 shows consistent declines in movingfrom procedure I to procedure II to procedure
Table 7. General representation of results of original and reinterview surveys for identical persons
Results of reinterview survey
Number having the characteristic . . . . . . . . . . . . . . . . . . , . . . . . . . . .
Number nothaving the characteristic . . . . . . . . . . . . . . . . . . . . . . . . . .
Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results of original survey
Number Number
having not having
the charac-Total
the charac-
teristic teristic
a b a+b
c d C+cl
a+c b+d n=a+b+c+rj
18
Table 8. Estimated proportions, net difference rates, and standard error of net difference rate for procedure 1II after reconciliationfor a 7Kyear quarterly average
Survey item, persons with:
Oneormore chronic conditions . . . . . . . . . . . . . .One or more hospital episodes in past
12 months . . . . . . . . . . . . . . . . . . . . . . .One or more restricted activity days in pest
2 weeks . . . . . . . . . . . . . . . . . . . . . . . . .Ona or more bed days in past 2 waeks . . . . . . . . . . .One or more time-lost days in past 2 weeks . . . . . . . .
Percant inPercent in
class onclass on
originalreinterview
interview
42.3 49.2
9.3 10.0
10.6 12.15.6 6.43.4 4.2
Netdifference
rate(percent)
-7.0
-r).16
-1.5-0.8-0.8
Estimatad standarderror of average
net difference rate
Underestimate Overestimate
0.2 0.4
0.1 0.1
0.1 0.20.1 0.20.1 0.2
NOTE: Includas fiscal years 1959, 1960, 1961, 1963, 1964, 1965, one-half of fiscal year 1966 and all of fiscal year 1967. A fiscal
year runs from July 1 to June 30.
Table 9. Estimatad annual rates per 100 parsons, original interview and reinterview, and percent net difference for procedura III afterreconciliation
Characteristic
Chronic conditionsz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restricted activity days2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Bed days’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tima-lostdays2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospitalepisodes3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospitaldaysa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OriginalReinterview
after recon-interview
ciliation
Rateper100persons
82.0 107.41,383.6 1,596.7
466.3 544.5287.1 351.2
9.9 10.794.6 99.7
Percentnet
iifferencei
-23.6
-13.3
-14.4
-18.3
-7.5
-5.1
‘Original -reintewieWx ,OO.reinterview
27%-year averages.33-year averages.
Table IO. Estimated proportions, gross difference rates, andindexes ofinconsistency forprocedurelll after reconciliation fora7Z-yearquartarly average
Survey item,persons with:
Oneormorechronicconditions . . . . . . . . . . . . . . . . . . . . . . .One or more hospital episodes inpast
12months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .One or more restricted activity days in past
2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Oneormorebeddaysin past2weeks . . . . . . . . . . . . . . . . . . .
Oneormore time-lost days inpast2 weeks . . . . . . . . . . . . . . . . .
Percent in
class on
original
interview
42.3
9.3
10.6
5.6
3.4
ParCent in
class on
reinterview
49.2
10.0
12.16.44.2
Gross Index of
difference inconsistency
rate X 100 x 100
0.9 I 5.5
3.4 17.2
2.3 17.5
1.6 23.4
NOTE: Includes fiscal years 1959, 1960, 1961, 1963, 1964, 1965, Ona-half of fiscal year 1966, andalIof fiscal year 1967. A fiscalyear runs from Julyl to June30.
19
Table Il. Estimated indexes of inconsistency bythree procedures, 3-year averages, fiscal years l959%l
Survey item, persons with one or more:
Chronic conditions inpast12 months . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospital episodes inpast12 months . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restricted activity daysinpast2 weeks . . . . . . . . . . . . . . . . . . . . . . . . .
Baddays inpast2 weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Time-lost daysinpast2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospital daysinpast 2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Procedura I Procedure II
(no recon- (before rec-ciliation) onciliation)
30.9 22.2
7.6 7.0
44.5 28.6
41.1 26.6
37.6 32.9
12.8 19.5
=—
%ocedure I I I
(after recon-
ciliation)
17.4
6.0
18.3
15.8
21.4
19.1
III in the estimated indexes of inconsistency. Ifprocedures I and II were carried out as specified,then the expected difference between theindexes of inconsistency would be zero. Theestimated indexes for procedure H, however, areabout 20 percent smaller than the indexes forprocedure I.
The differences between procedures 11 and IIIare in the direction that would be expected:reconciliation reduces the gross difference rate.
Thus, the estimates in table 10 are an under-statement of the gross differences that wouldoccur if repetitions of the HIS were carried outwithout reconciliation.
Some values of indexes of inconsistency fordemographic items computed from other studiesare provided in table 12 by size classes andcompared with indexes for health items com-puted from the reinterview program of the HIS.
The index of inconsistency for hospital epi-sodes in the past 12 months is in the same sizeclass as the simpler demographic items such assex, color, and age.
Time-lost days compare with the more diffi-cult items to measure such as income andeducational attainment.
Comparison of self-respondents with proxyrespondents. —In the original interview foradults, the health questions are asked of theperson himself if he is home at the time of theinterview. If he is not at home, a related adultmay provide the information. The person who isnot present at the time of the interview isreferred to as a proxy respondent since theinformation on such a person is obtained byproxy. However , in reinterviews all adults areself-respondents (in all reinterviews conductedduring fiscal years 1959-67). Table 13 shows
20
3-year averages of proportions, net differencerates and ind-exes of-inc-onsistency for six surveyitems as reported by self-respondents and proxyrespondents in the original survey. The data arebased on results after reconciliation (proce-dure III).
Comparison of self-respondents and proxyrespondents are limited by the fact that theselection of the respondent on the originalinterview is not a random selection. Thus part ofthe differences in IeveI may be attributed toinherent differences between respondents whoare available to report for themselves and re-spondents who are not available at the time ofinterview and whose health conditions arereported by another member of the household.
For four of the six items, the reconciliationtends to bring the proportion in the class forproxy respondents closer to that of self-respondents, that is, the net difference rates aregreater for proxy respondents.
The estimated prevalence rates for chronicconditions per person from the reinterviewsurvey are presented in table 14 for self-respondents and proxy respondents. In attempt-ing to estimate what effect the respondent has,certain assumptions were made about the differ-ences. Specifically an assumption was made thatfor self-respondents the net difference betweenthe original rate and the reinterview rate can beconsidered as the difference due to the secondinterview. For proxy respondents, the net differ-ence consists of second-interview differencesand differences due to the use of a proxyrespondent and that these differences are addi-tive. Furthermore, an assumption was made thatthe differences due to the second interview arethe same for self-respondents and proxy
Table 12. Comparison of estimated indexes of inconsistency for Health Interview Survey items with d
Size of index (X 100)
<lo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11-20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1960 Decennial
Census Evaluation
Program’
SexColor
Age
Labor force
Mobility
Educational attai nment
Income
Current
Population
Suwey
Reinterviews
1961%62
Employed
In labor force
Unemployed
nographicitems
HIS
Reinterviews
Hospital episodes
Hospital days
Chronic conditions
Bed days
Restricted activity days
Time-lost days
‘ U.S. Bureau of the Census: E~aluation and Research Program of the U.S. Censuses of Population and Housing, 1960: Accuracyof Data on Popu Iation Characteristics as Measured by Reinterviews. Series ER 60-No. 4. Washington. U.S. Government Printing
0ffice,1964.
2 U.S. Bureau of the Census: The Current Population Survey Reinterview Program January 1961 through December 1966. Tech.
Paper No. 19. Washington. U.S. Government Printing Office, 1966.
respondents.’The last column of table 14 showsthe net difference expressed as a proportionofthe reinterview estimate, and the differencebetween these relative net differences is ameasure of the effect of proxy respondents, thatis, that about 16 percent of the relative netdifference for persons reported for by proxyrespondents can be attributed to the use of aproxy respondent.
Effect of nonreporting on estimates of magni-tude. –Differences in reporting of the number ofconditions, days, episodes, and so forth betweenthe original interview and the reinterview areclassified as folIows: differences due to a change
in the number of conditions, days, episodes, etc.,reported on the two interviews or as differencesdue to a report of no conditions, days, episodes,etc., on one interview and a report of one ormore conditions, days, episodes, etc., on theother interview (table 15).
The change from a report of none on oneinterview to a report of one or more conditions,days, episodes, etc., on the other interview has arelatively small effect on estimates of propor-tions, but with the exception of hospital days inthe past 2 weeks it has a major impact onestimates of magnitude. This section presentsestimates of the part of the net and gross
differences for estimates of magnitude that canbe accounted for by a change from a report ofnone on one interview to a report of one ormore on the other interview.
Table 16 shows estimates of the componentsof the total net difference and the ratio of eachcomponent to the total net difference.
Table 16 shows that for hospital episodes,restricted activity days, bed days, and time-lostdays, the net increase on reintemiew for esti-mates of magnitude is principally due to thechange from a report of none on the originalinterview to a report one or more on reinter-view. For hospital days the number of days
accounted for by changes from a report of noneon the original interview to a report of one ormore on reintemiew is about the same as thenumber of days involved in changes of a reportof one or more on the original interview to areport of none on reinterview and thus canceIfor estimates of net difference. For chronicconditions, the changes in the number of condl-
tions reported for cases which are one or moreon both interviews are close to the number ofconditions accounted for by changes of a reportof none on the original to a report of one ormore on reinterview and increase the volume ofconditions reported on reinterview. Changes
21
Table 13. Estimated proport~ons, net and gross difference rates, and indexes of inconsistency by subject and respondent on original
interview compared with proxy respondents, procedure I I 1, 3-year averages, fiscal years 1959-61
Survey item, subject and respondent
on original interview, persons with:
One or more chronic conditions:
Adultrself-respondent . . . . . . . . . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) . . . . . . . . . . . .
One or more hospital episodes in past 12 months:
Adult, self-respondent . . . . . . . . . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) . . . . . . . , . . . .
One or more restricted activity days in past 2 weeks:
Adult, self-respondent . . . . . . . . . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) . . . . . . . . . . . .
One or more bed days in past 2 weeks:
Adult, self-respondent .,... . . . . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) , . . . . . . . . . . .
One or more time-lost days in past 2 weeks:
Adult, self-respondent . . . . . . . .. . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) . . . . . . . . . . . .
One or more hospital days in past 2 weeks:
Adult, self-respondent . . . . . . . . . . . . . . . . . . . . .
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . .
(self-respondent - proxy respondent) . . . . . . . . . . . .
Persons
in class
Original
59.4
47.4
12.0
13.2
9.0
4.2
13.5
8.2
5.3
6.7
4.7
2,0
3.1
4.3
-1.2
0.7
0.4
0.3
Reinter-
view
64.8
58.5
6.3
13.5
10.0
3.5
14.3
11.0
3.3
7.1
5.5
1.6
3.5
4.7
-1.2
0.7
0.4
0.3
Average
net
difference
rate
-5.4
-11.1
5.7
-0.3
-1.0
0.7
-0.8
-2.8
2.0
-0.4
-0.8
0.4
0.4
–0.4
Average
gross
difference
rate
6.1
14.9
0.6
1.3
3.5
5.0
1,7
2.0
0.8
2.4
0.3
0.2
Index
of
Inconsistency
13,1
30.3
2.4
7.3
14.8
28.9
13.4
21.1
11.8
28.2
21.6
23.9
Table 14. Estimated prevalence rate of chronic conditions per person by subject and respondent on original interview, original and
reinterview estimates for a sample of identical persons, reconciled reinterviews, fiscal years 1959-61
Chronic conditions per person (12 months)
Subject and respondent on original interview mAdult, self -respondent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.26 1.52
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84 1.25
Self-respondent - proxy respondent . . . . . . . . . . . . . . . . . . . . . .
Estimated sampling error of difference . . . . . . . . . . . . . . . . . . . . . . .
NOTE: A similar comparison for a different time period and including all conditions, acute and chronic,
The magnitude of effect that the type of respondent has is about the same.
Net
cliff erence
(a) - (b)
-.26
-.41
(a) - (b)
(b)
-.17
-.33
.16
.02
presentad in table 15.
22
Table 15. Estimated rate of conditions per person by subject and respondent on original interview, original and reinterview estimates
for a sample of identical persons, reconciled reinterviews, fiscal years 1963-67
Subject and respondent
at original interview
I Conditions Pw person
mAdult, self-raspondent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.61 1.74
Adult, proxy respondent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.00 1.31Salf-respondent - proxy respondent . . . . . . . . . . . . . . . . . . . . . .
Estimated sampling error of difference . . . . . . . . . . . . . . . . . . . . . . .
Net(a) - (b)
difference —
(a) - (b)(b)
-.13 -.07
-.31 -.24
.17
.01
Table 16. Estimated total net difference between original interview results and reinterview results and components of net difference,
reconciled reinterviews, fiscal years 1959-51
Survey item
Chronic conditions in
past 12 months . . . . . . .
Hospital episodes in
past 12 months . . . . . . .
Hospital days in past
2 weeks . . . . . . . . . . . .
Restricted activity days
inpast2weeks . . . . . . .
Bed days in past
2weeks . . . . . . . . . . . .
Time-lost days in past
2weeks . . . . . . . . . . . .
Total net
difference
(x-y)’
(1)
-26,700
-881
185
-13,808
-3,592
-3,361
Components of total net difference
Due to change
in reporting
from presence on
original to absence
on reinterview
(2)
964
220
584
7,244
1,533
1,327
Due to change
in reporting from
absence on original
to presence on
reinterview
(3)
-12,541
-969
-552
-17,003
-4s1
-4,684
Due to change
in reporting of
magnitude when
present on
both interviews
(4)
-15,123
-132
153
-4,149
-204
-4
Ratio of components
to total net
difference
0,1difference
I(2) I - I(3) I
(1)
(5)
.43
.85
.16
.70
.94
1.00
ufagnitude
iifferance
~
(1)
(6)
.57
.15
.83
.30
.06
(’)
*x is the estimate of magnitude from the original interview, and y is the estimate of magnitude from the reinterview.
‘Algebraically this component is [column (1)] - [column (2) + column (3)1. A minus sign indicates a net increase on reinterview.
sLess than 0.005.
NOTE: The tabulations for hospital days, restricted activity days, bed days and time-lost days are in terms of 2day intervals. The
components of the net difference were estimated by using midpoints of the 2day intervals. Changes in tasponse that did not result in
a change of class interval have no effect on the estimates.
from a report of one or more on the original to a changes from a report of none to a report of onereport of none on reinterview do not have much or more in both directions. The following indexeffect on the estimates of chronic conditions. is an estimate of the proportion of the total
A large proportion of the gross differences in response variance that is accounted for by casesestimates of magnitude is accounted for by which either change from a report of none on
23
the original interview to a report of one or moreon reinterview or from a report of one or moreon the original interview to a report of none onreintemiew:
i LO(X - 0)’ +~foy(o- 3))2& = .=1 y=l
Ww
where x is the value on the original interview, yis the value on reinterview, fxo is the number ofpersons with x value on original and O onreinterview, ~.
2is the number of persons with O
on original an y value on reinterview, and ~XY isthe number of persons with x value on ongmalinterview and y value on reintenriew.
Table 17 shows that except for chronicconditions most of the response variance forestimates of magnitude can be accounted for bychanges in the reporting of the presence orabsence of the characteristic.
Tables 18-22 present estimates by year of thesummary measures discussed in previous sec-tions. All of these results are for procedure III,that is, a comparison of the original interviewwith reinterview after reconciliation. The yearly,estimates are subject to large sampling errorssince the reinterview sample in any one year isrelatively small. One additional summary meas-ure, the index of net shift, is presented. The
Table 17. Proportion of total response variance due to changes
in reporting of presence or atxence of characteristic pro-
cedure I I 1, reconciled results
Characteristic
Chronic conditions in past 12 months . . . . . . . . . .
Hospital episodes in past 12 months , . . . . . . . . . .
Hospital days inpast2 weeks , . . . . . . . . , . . . . .
Restricted activity days in past 2 weeks . . . . . . . . .
Beddays icpa$t2weeks . . . . . . . . . . . . . . . . .
G.
.41
.92
.96
.88
.80
NOTE: Hospital days, restricted activity days, bed days, and
time-lost days for the tabulations are in terms of 2-day intervals.
Midpoints of the 2day intervals were used for the x and y
values. Changes in response which did not result in a change of
class interval have no effect on the estimates.
index of net shift is simply the ratio of the netdifference rate to the percent in class onreinterview.
INTERVIEWER VARIABILITY STUDY
Introduction
The joint effects of sampling and nonsampIingerrors determine the accuracy of survey results.The mathematical model of response errors insurveys presented in appendix III shows how themean square error of a statistic is divided into itsvarious components: sampling variance, responsevariance, interaction, and square of bias.
The response variance can be further dividedinto simple response variance and correlatedresponse variance. In the preceding section,estimates were presented of the simple responsevariance and of response bias as measured byreinterviews. The usual estimates of samplingvariance include the simple response varianceand possibly a small part of the correlatedresponse variance due to field interviewers.However, the major part of the interviewercontribution to response variability is not in-cluded in the estimates of sampling variance.This section describes an interviewer variancestudy designed to measure the contributions ofinterviewers to the vanabilit y of health statistics.
Significant between-interviewer variance inthe reporting of health data has been observed ina number of studies. Data from other studiesalso indicate that interviewer effects may oper-ate differently for different statistics.
Design of the Interviewer Variance Study
The interviewer variance study was conductedover the 4-year period 1960-63. For the first 2years of the study, randomization of interviewerassignments was carried out in eight largeSMSA’S where there were two or more HISinterviewers. The study included 10 SMSA’Sduring the second 2-year period. The assign-ments (within each pair or triplet) of inter-viewers within an SMSA were randomized in aninterpenetrated design so that each interviewerof a pair would have produced results with thesame expected value if there were no between-interviewer variabilityy.
24
Table 18. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency forprocedure III after reconciliation for parsons with one or more chronic conditions in the past 12 months, by year
>I Percent in
Parcant inclasson
reinterview
Net Index ofnet shift 1
x 100
Gross
differencerate X 100
Index of
Fiscal year I class on
originalinconsistency
x 100rate X 100
-6.8
-6.7
-7.5
NA
-6.7
-7.7
-7.6
-7.7
-5.5
I intarview
1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1965 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18663 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1867 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Net difference rate
Percent in class on reinterview
z Not available.
‘2 quarters only (July 1, 1965 -Dec. 1965).
38.542.541.7NA2
41.542.843.748.546.3
45.248.349.2
NA48.250.551.358.251.9 1
-15.0 8.6-13.8 7.9-15.3 8.9
NA NA-13.8 8.3-15.2 9.5-14.8 8,7-13.8 8.9-10.7 6.7
17.9
16.2
18.1
NA16.819.317.617.913.4
Table 19. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for
procedure III after reconciliation for persons with one or more hmpital episodes in the pest 12 months, by year
Percent inPercent in
class onoriginal
classonreinterview
interview
Netdifferencerate X 100
Index of
net shifti
x Iw
Gross
differencerateX 100
Index of
inconsistency
x 100Fiscal year
1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1860 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1885 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1966Z . . . . . . . . . . . . . . . . . . . . . . . . . . . .1867 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.49.48.0NA
9.611.111.9
7.38.7
8.610.19.1NA
10.911.812.17.59.1
-0.1-0.7-1.1
NA-1.4-0.7-0.2-0.1-0.4
-1.2-6.9
-12.1NA
-12.8-5.9-1.7-1.3-4.8
0.90.91.2NA1.50.80.40.11.1
5.55.27.4NA8.33.61.61.06.9
1 Net difference ;ate
Percent in class on reintewiew
22 quarters only (Jul.{ 1, 1965-Dec. 1965).
For the first 2 years of the study, calendaryears 1960 and 1961, the data cover theworkin25 interviewer assignment areas. Six of the eightSMSA’S had a pair of interviewers, Los Angeleshad atriplet, and the New York SMSA hadfivepairs ofinterviewers.
For the second 2 years, calendar years 1962and 1963, the data cover the work in 30interviewer assignment areas for two quarters
and the work in 28 interviewer assignment areasfor the remaining six quarters. (The HIS samplewas redesignedin 1962, and one interviewer pairwas dropped in the New York SMSA.) Assign-ments in Chicago and Los Angeles were ran-domized among three interviewers during thesecond2-year period.
During the 4-year period, a total of 6,415segments of six to nine households were in-
25
Table 20. Estimated proportions, net difference rates, indexes of net shift, gross diffarance rates, and indexes of inconsistency for
procedure III after reconciliation for persons with one or mora restricted activity days in the past 2 waeks, by year
Fiscal yaar
1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1965 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1966’ . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Net difference ratePercent in class on reinterview
22 quarters only (July 1, 1965-Dec. 1965).
Percent in
class on
original
interview
10.4
10.5
12.4
NA9.5
10.911.1
6.410.9
Percent inclass on
reinterview
11.9
12.8
14.1
NA10.011.912.37.5
13.6
Net
difference
rate X 100
-1.4-2.3-1.7
NA-0.5-1.1-1.3-1.1-2.7
I ndax ofnat shift]
x 100
-11.8
-17.9
-12.1
NA-5,0-9.2
-10.6–14.7-19.9
TGross Index ofdifference inconsistency
rateX 100 x 100
14.0 20.64.4 22.03.3 14.6NA NA2.5 14.22.3 11.13.0 15.01.8 13.94.5 23.1
eluded in the assignments. Of these segments, each of 84 health and demomaphic character-1,204 were excluded from the analvsis for thefollowingreasons:
.
Percent
Raason for exclusion of total
segments
Total assigned segments not used in
analysis . . . . . . . . . . . . . . . . . .
Segments completed by “other than assigned
interviewers” . . . . . . . . . . . . . . . . . .Sagments in non-self-representing PSU’S . . . . . .
Segmants where it could not be determined who
complated the assignments . . . . . . . . . . .
Sagments containing “special dwelling places” . . .
18.8
12.43.1
1.22.0
The remaining 5,211 segments used in theanalysis contained about 34,000 householdswith 110,000 persons.
Method of Analysis
For this analysis, ratios of two variables wereused at all times for two reasons. First, most HISdata are presented as rates orproportions of thetotal population; and, second, this methodminimizes the effect of the variability in size ofinterviewer assignment on the analysis.
The starting point for the analysis was acomputer tape containing quarterly totals for
.istics for each of the intem~ewer assignmentareas.
The mathematical model usedto compute thetotal variance is basically acomparison of resultsbetween a pair of interviewers, extended to thewhole study area. The task was to estimate theexpected value of the difference between twointerviewers’ findings for a specified period oftime. In the model, the total relvariance for thestudy area is
(1)
where r= is the ratio computed from a randomhalf of the interviewers and r~ is the ratiocomputed from the other random half and theexpected value is taken over all possible half-samples. An estimate of this expected value wasmade by assigning each interviewer in a pair to aor b at random and then averaging over all pairsof interviewers.b Different permutations of thepairs give other estimates. Twenty-five permuta-tions were used to give the estimator:
bStrictly speaking, it is interviewer assignment area instead ofinterviewer. When there was a change in interviewers due toresignation, illness, or other administrative reason other than atemporary substitution, the work of the replacement interviewerwas treated es part of the same interviewer assignment area.
26
Table 21. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency forprocedure III after reconciliation for persons with one or more bed days in the past 2 weeks, by year
Fiscal yeac
Percent inPercent in
class onclass on
originalreinterview
interview
1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1863 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1965 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1966= . . . . . . . . . . . . . . . . . . . . . . . . . . . .1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Net difference ratePercent in class on reinterview
22 quarters only (July 1, 1965 -Dec. 1965).
L5.6 6.15.5 6.45.4 6.3NA NA5.3 6.06.0 6.45.1 6.04.5 5.06.8 8.8
Net
differencerateX 100
-0.5-0.9-0.9
NA-0.6-0.5-1.0-0.5-1.9
-8.2-14.1-14.3
NA-10.0
-7.8-16.7-10.0-21.6 -1-
3.8 15.82.1 18.31.7 I 5aNA NA2.1 19.91.6 13.81.2 12.71.7 22.73.4 25.6
Table 22. Estimated proportions, net difference rates, indexes of net shift, gross difference rates, and indexes of inconsistency for
procedure III after reconciliation for persons with one or more time lossdays in the pest 2 weeks, by year
Fiscal year
I Percent in
class onoriginal
interview
1959 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1960 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1961 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1962 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1963 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1964 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1966 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19662 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1967 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Net difference rate
Percent in class on reinterview
22 quarters only (July 1, 1965-Dec. 1965).
4.23.73.3NA3.12.63.43.03.6
25
where j denotes the permutation and r theestimated ratio computed from the work of allinterviewers.
This estimator has two components that mustbe identified separately :the between-interviewerrelvariance and a sampling relvariance arisingfrom the fact that each interviewer of a pair
Percent in Netclass on difference
reinterview rate X 100
4.9 -0.74.9 -1.23.8 -0.5NA NA3.8 -0.73.1 -0.54.2 -0.83.1 -0.15.2 -1.6
Index ofnetshifti
x 100
Gross Index ofdifference inconsistencyrate X 100 x 100
-14.3-24.5-13.2
NA-18.4-16.1-19.0
-3.2-30.8
1.41.72.0NA1.11.61.20.62.6
19.324.129.5NA
15.831.615.010.737.2
interviewed a separate random sample of house-holds. Estimate~of V; were mad: for asinglequarter of data, two quarters combined, threequarters, up to an estimate based on eightquarters of data.
In the analysis, the results were treated as ifthey were two independent studies of 2 yearseach. One reason for treating the results as twoobservations was to minimize the effect ofinterviewer turnover. Another reason was strict-ly practical; 2 years of data could rehandledmore easily than 4years.
If it is assumed that the sampling variance is afunction of sample size but that the estimate ofthe between-interviewer variance is a functiononly of the number of interviewers, then
(3)
A
where ~ is the estimat~ of the between-interviewer relvariance and WE/n is the estimateof the sampling relvariance for a single quarterdivided by the number of quarters used in theestimate. The parameters of this function wereestimated in terms of V$ by the method of leastsquares (appendix IV). This least-s~uares s~lu-tion was then used to determine ~ and V:.Figure 1 sh~ws an example of the expectedbehavior of ~~ as the sample size increases.
As the number of quarters included in theestimate increases, the sampling component ofthe variance decreases and the estimate of totalvariance approaches the between-interviewervariance asymptotically.
The assumptions of equation 3 are un-doubtedly not fully warranted. There is evidencefrom other studies conducted by the Bureau ofthe Census that the response variance cannot beviewed as a constant even if, as is not the case inthe present study, the interviewers did notchange over the 2-year period. To illustratefurther that the assumptions are not completelytrue, it can be shown that the response varianceis the sum of two terms.
The first is a simple response variance, whichexpresses the variability in repeated measures(interviews) on the same persons. This quantityvaries inversely with the sample size and thus
*‘*2VT
i ++ F
-)F+ F 3 : -1e *
--! ...-..-. . . . . . -------- ------- . . ------ . . . . . . .._.. - --4;
1 2 3 4 5 6 7 8
NUMBER OF QUARTERS
Figure 1. Expected reduction in I/; as sample size increases.
depends on the number of quarters of data. Inthe estimation scheme used in this study, thesimple response variance has been treated as partof the sampling varim~ce and subtracted out.Thus, the estimator ~ is somewhat of anunderestimate.
The second term is the product of twofactors: the simple response variance and acorrelation expressing the extent to which eachinterviewer tends to introduce her own system-atic response errors in her assignment (seeappendix III). An assumption of this study isthat this correlation is a constant over the 2-yearperiod, although there is evidence from otherprograms that the correlation may decline withan increase in experience because of training,increased proficiency, and the increasing hetero-geneity in the population included in theassignment.
Results
Tables 23 and 24 contain the results of thisstudy. These results are presented in two tables,one for each 2-year period. Each table showsresults for 67 items.
The estimated relvariances shown in the tablesrelate to annual estimates prepared from thework of approximately one-fourth of the ‘inter-’viewer staff. For some items the estimates ofbetween-interviewer relvariances are negative.These are items which presumably have a verylow interviewer variability.
The sample size is too smzdl to place muchreliance on the specific estimates of the relvari-ances; however, the last column of the tables isprobably sufficiently reliable to provide ageneral ranking of the characteristics. This lastcolumn expresses the between-interviewer rel-variance as a proportion of the total relvariance.Discussion of the results shown in tables 23 and24 is based on this last column.
The results shown in the tables indicate thatthe reporting of chronic conditions and activityrestriction associated with such conditions havethe highest ratios of between-interviewer vari-ance to total variance. In addition, as might beexpected, reporting of income also has a fairIyhigh ratio.
As has been observed in other studies ofbetween-interviewer variance, reported in papers
Table 23. Intewiewer variance study of estimates of cwnponents of relvarianm for annual estimntas of selectad characteristics based on 26 interviewer assignment areas in
eight SMSAW United States, 1S%0-61
Numerator
Health characteristics (Magnitude items):
Conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Chronic conditions for females . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions formales . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 or more ted days in last
12 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restricted activity days for chronic renditions
inlast2w3ek5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
R=tdctWatiivi~&Winlsti2Mh . . . . . . . . . . . . . . . . . .
Acute conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Chronic conditions with 1 or more bed days in last
2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Disability days in last 2 weeks fmm all
=idents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Restricted activity dcys in last 2 weeks for acute
conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restricted cctivity days in I&ct 2 weeks for ccute
conditions formdac. . . . . . . . . . . . . . . . . . . . . . . . . .Bed days for chronic conditions in k.t 2 weeks . . . . . . . . . . . . .
Bedd.cys inlast2vmks . . . . . . . . . . . . . . . . . . . . . . . . .Bed days for chronic conditions in last 12 months . . . . . . . . . . .
Days lost from school or work in Iaat 2 weeks . . . . . . . . . . . . . .
D8yslostfrom workinlast2weeks . . . . . . . . . . . . . . . . . . .
Baddaysfor acute conditions inlast2weeks . . . . . . . . . . . . . . .
Dayslostfrorn school inlact2weeks . . . . . . . . . . . . . . . . . . .
Hospital disclmges inlastl month . . . . . . . . . . . . . . . . . . . .
Hospitalizations inlast12months . . . . . . . . . . . . . . . . . . . .
Hospital dayainlast 12months . . . . . . . . . . . . . . . . . . . . . .
Hcspital days for ail discharged in last
12months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Health characteristic (Attributa items):
Persons with 1 or mom chronic conditions . . . . . . . . . . . . . . . .
Persons withlor moreconditions . . . . . . . . . . . . . . . . . . . .
Maleawith lormore chronic conditions . . . . . . . . . . . . . . . . .
Malmwithl ormoreconditions . . . . . . . . . . . . . . . . . . . . .Perxons limited in kind or amctmt of activities . . . . . . . . . . . . . .
Acute conditions rnadically attended . . . . . . . . . . . . . . . . . . .New cases ccute respimto~ conditions, 1 w more
baddays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospitalizations for tonsillectomy or adanoidectomy
formales inlsst12months . . . . . . . . . . . . . . . . . . . . . .
Hospitalizations for operations on the female genital
system inlast12 months . . . . . . . . . . . . . . . . . . . . . . .
Persons unable to carry on major activity . . . . . . . . . . . . . . . . .
.%ciceconomic characterictica:
Fmilieswith income >$6000 . . . . . . . . . . . . . . . . . . . . . .
PersOns in families withinc0nw>$5000 . . . . . . . . . . . . . . . . .
Families withincome <$20W . . . . . . . . . . . . . . . . . . . . . .
P.waons who are not employed and not keeping
house . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Employadpmons . . . . . . . . . . . . . . . . . . . . . . . . . . . .
PersOns in families with inc0rne>$2000 . . . . . . . . . . . . . . . .
Employed femalec . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Denominator
(total number of the following]
Househol&Females
Males
Persons
Chronic conditions
PersonsPersons
Persons
Accidents
Acute conditions
Acute conditions for males
PersonsPersons
Chronic conditions
Persons currently employed and
fEBOl15 a@d 5-16 YCWS
Persons currently emplcyed
Acute conditions
Persons aged 6-16 years
Persons
Households
Hospitalization in last 12
months
Hcspital discharges in lest
12 months
Per30ns
PafiOns
Males
MalesPersons
Acute conditions
Persons
Hospitalizations for operations,
males
Hospitalization for cfmations,
femdas, exclusive of deliveryPersons
Families
Persons
Families
Persons aged 17 years and over
Persons aged 17 years and over
Persons
Femalas aged 17 years and over
Ratio
2.563.769
.612
.115
.633
.502
.115
.024
1.182
3.119
2.696.136
.2274.246
.217
.206
1.261
.23s
.103
.333
10.732
10.456
.369
.450
.359
.426
.066
.731
.026
.147
.s46
.018
.521
.606
.117
.075
.561
.065
.330
l-otalra!-
variance~2
T
.00395.00463
.00434
.00476
.01328
.W411
.00285
.00661
.01269
.00273
.00524
.00623
.0026s
.0Q675
.00359
.00517
.00203
.01041
.0C049
.00045
.00236
.00183
.00196
.00165
.00230
.00166
.00263
.00031
.00424
.01540
.0U360
.00233
.00115
.0@39S
.00371
.00133
.OC@X
,00356
.00040 ~
Betwen-Interviewcr
relvariance~:
.00339.00442.0Q370
.oo4rM
.01045
.00296
.03202
.0Q604
.W642
AX1135
.W244
.00%1
.00115.00243
.00135
.0013s-.00007
-.00113
-.00007
-.OQO11
-.00133
-.0+31 70
.00174
.W146
.C4MS3
.00165
.W152
.000Q3
.W1C8
.00369
-.00107
-.-1
.m77
.0GQ66
.00127
.0003S
.Coxrl
.LIJ031
-.WJO03
—
Sampling
relvariawe
g
.00026.00041
.CK)W’f
.00072
.00283
.00115
.oom3
.043256
.C0447
.0013s
.00261
.00462
.00153
.0C332
.00224
.00365
.IX121 o
.01154
.W056
.CCI056
.W169
.00353
.00022
.wcr19
.00037
.00033
.W132
JYJ022
.00317
.01171
.00467
.00313
.00a29
.000S3
.00244
.mlcm
.00LW5
.~326
.00042
.934
.915
.664
.846
.7s7
.720
.706
.702
.663
.494
.465
.439
.430
.423
.376
.259-.036
-.109
-.1 2a
-.266
-.563
-.926
.6%
.666
.636
.632
.536
.2a3
.2M
.240
-.296
-.W6
.665
.664
.342
.246
.116
.066
-.069
29
Table 23. Interviewer variance study of estimates of componentsaf re[variance for annual estimates of selected characteristics based on 25interviewer assiqnmentareasineight SMSA’s: United States, 1660.61-Con,
Nonresponse Items:
Persons with amount ofeducationu”known . . . . , , . , . . . . . .Heads of households with amount of education
unknown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Families with unknown family income . . . . . , . . . . . . . . .
Conditions from amidems, u“knmvn if m~t~r WMcIe . . . . . . . , . .
Conditions from accidcmts, u“k”cwv” lEatiO” . . . . . . . . . . . .
Demographic characteristics (includhg health characteristics by age groups):
Chronic conditions with 1 or more &d days, perscms
aged5.14 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .Acute conditions, persons aged 25.44 years . . . . . . . . . . . . . . .
Chronic conditions with 1 or more bed days, persons
aged2544 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 15-24 years . . . . . . . . . . . . . .
Females, married, spouse present , . . . . . . . . . . . . . . . . .
Perscms ofather races . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic renditions with 1 or more bed days, persons
agedunder 5 years . . . . . . . . . . . . . . . . . . . . . . . . . . .
Perscmsof Negro race . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 5.14 years . . . . . . . . . . . . . . .
Chronic conditions with 1 or more &d days, persons
65years And over . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 65 years a“d over . . . . . . . . . .
Males aged17 years Andover.. . . . . . . . . . . . . . . . . . . . . .
Persons agedunderl year . . . . . . . . . . . . . . . . . . . . . . . .
Persons agedl yearand over.... . . . . . . . . . . . . . . . . . . .
Females aged17yearsandowr. . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 45.64 years . . . . . . . . . . . . .
Acute conditions, persons aged 5 years and over . . . . . . . . . . . . .Persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 or more bed days, persons
aged15-24 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 or more bed days, persons
aged45+4year5 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Permnsagedl?yearsa ndover. . . . . . . . . . . . . . . . . . . . . .
Persons agedunder17 years . . . . . . . . . . . . . . . . . . . . . . .
(total number of the following)
Perscms aged 17 years and over
HcmsehOl&
Families
Accidents
Accidents
Chronic conditions
Acute conditions
Chronic conditions
Acute conditionsFemalesr aged 17 years and over
Persons
Chronic conditions
Persons
Acute conditions
Chronic conditions
Acute conditionsPersons aged 17 years and over
Persons
Persons
Persons aged 17 years and over
Acute conditions
Acute conditions
Households
Chronic conditions
Chronic conditions
Persons
Persons
by Eckler and Hurwitz,l O and Hurley, Jabine,p.nd Larson,l 1 there are considerable between-interviewer variances in nonresponse rates. Note,for example, that the ratio of between-interviewer variance to the total variance isabout .8 for the number of persons witheciucat.ioz urdmown.
It is also of interest to note that the demo-graphic differentials in morbidity rates (distri-butions of acute and chronic conditions by agegyoups) are not subject to any significantbetween-interviewer variance.
?atio
.021
.026
.065
.003
.002
.016
.239
.049
.102
.643
.007
.007
.106
.241
.028
.075
.465
.020
.960
.535
.170
.173
3.167
.011
.053
.667
.333
Total rel-
variance
y
.01961
.02727
.01275
.37186
.53790
.00883
.00133
.00251
.00520
.00013
.03511
.01386
.01216
.00167
.00376
.00756
.00005
.00161
.00000
.00003
.00235
.00282
.OQoos
.00629
.00130
.00005
.00020
Between-
nterviewer
relvariance+2
I
.01535
.02064
.00798
.34642
.58952
.00545
.00036
.00050
.00068
.CK)ool
.80200
.00076
-.00011
-.00008
-.00024
-.00066
-.00001
-.00033
-.00000
-.00001
-.00074
-.00107
-.WO05
-.cf1430
-.00092
-J33004
-.00018
Sampling
,elvariance~;
.00426
.00663
,00477JJ~E44
-.05162
.00337
.00098
.00201
.00452
.00011
.03311
,01310
.01227
.00175
.00400
.00822
.00006
.00194
.00000
.00004
.00309
.00389
.00013
.01058
.W223
.moo9
.00036
.783
.757
.626
.932
1.096
.618
.267
.200
.131
.108
.057
.055
-.009
-.050
-.064
-.o87
-.164
-,203
-.214
-.222
-.316
-.380
-.677
-.683
-.706
-.646
-S96—
The estimates of interviewer variability fromthe two periods (1960-61 and 1962-63) differconsiderably for some items. The material forthe first 2 years was investigated to determinethe cause of the higher estimates of interviewervariability for these items. During the 11960-61period, one interviewer of the 25 in the studycontributed a disproportionate amount to theestimates of between-interviewer variability.However, the response variance study of the1960 censusl 2 demonstrated that the distribu-tion of individual interviewer contributions to
30
Table 24. Interviewer variance study of ettimates of components of relvariance for annual estimates of selected characteristics based on 2S interviewer assignment areas in
10 SMSA%: United States, 1962.S3
Numeramr
(total number of the following)
Health characteristic (Magnitude items):
COnditiOns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Chronic conditions for femal~ . . . . . . . . . . . . . . . ., . . . .
Chronicconditionsformales . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 ormorebed days in last12mon%hs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restricted activity days for chronic conditions in last
2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Restrictedactivitydaysinlast2weeks . . . . . . . . . . . . . . . . .
Acuteconditions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 ormorebed days in last
2weeks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Disabilitydaysinlast2weeksfromall accidems . . . . . . . . . . . .
Restricted activity days in last 2 weeks for acute
conditions . . . . . . . . . , . . . . . . . . . . . . . . . . . . . .
Restricted activity days in last 2 weeks for acute
conditions for males . . . . . . . . . . . . . . . . . . . . . . . . .
Beddays forchronic conditions inlast2weeks . . . . . . . . . . . .
Beddays inlast2weeks . . . . . . . . . . . . . . . . . . . . . . . .
Beddays forchronic conditimw inlast12 months . . . . . . . . . . .
Days lost from school orworkinlast2waeks . . . . . . . . . . . .
Dayslost fromwork inlast2weeks . . . . . . . . . . . . . . . . . .
Beddays foracute conditions inlast2v.eeks . . . . . . . . . . . . . . ,Dayslostfrom schmlinlest2 weeks . . . . . . . . . . . . . . . . . . .
Hmpital discharges inlast12 months . . . . . . . . . . . . . . . . . . .
Hc=spitalizations inlast12 months . . . . . . . . . . . . . . . . . . . ,
Hospital daysinlast 12 months”.. . . . . . . . . . . . . . . . . . . . .
Hospital days foralldischarges inlast12 months . . . . . . . . . . . . .
Health characteristics (Attribute items):
Persons withl ormorechronic conditions . . . . . . . . . . . . . . . .
Persons withlor more conditions . . . . . . . . . . . . . . . . . . . .
Males with 1 ormore chronic conditions . . . . . . . . . . . . . . . . .Males with lormoreconditions . . . . . . . . . . . . . . . . . . . . .
Persons limited inkindor amount of activities . . . . . . . . . . . . .
Acute conditions m%ficallyattended . . . . . . . . . . . . . . . . . . .Newcases acute respiratory conditions, l or more
bed day s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hospitalizations for tonsillectomy or adenoidectomy,
malasin last12 months... . . . . . . . . . . . . . . . . . . . . .
Hospitalizations foroperations, female genital
system inlast12 months.. . . . . . . . . . . . . . . . . . . . . .
Persons unable tocar~on major activity . . . . . . . . . . . . . . . . .
Socioeconomic characteristics:
Families withincome >$5000 . . . . . . . . . . . . . . . . . . . . . .
Persons in families with income >$50W . . . . . . . . . . . . . . . . .Families withincome <$20GQ . . . . . . . . . . . . . . . . . . . . . .
Persons whoarenotemploy edandnot keeping house . . . . . . . . . .Employed persons . . . . . . . . . . . . . . . . . . . . . . . . . . . .PersOns in famities with inc0me<$20W . . . . . . . . . . . . . . . . .
Employ ed females . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Denominator
[total number of the follom”ng}
Households
Females
Males
Persons
Chronic conditions
Pemons
Persons
PertOnsAccidents
Acute conditiom
Acute conditions for males
PemOns
Persons
Chronic conditionsPersons currently employed and
persons aged 6-16 years
Penons currently employed
Acute conditionsPemons aged 6.16 yeart
Persons
Households
Hospitalizations in Iatt 12
monthsHospital discharges in last
12 months
Persons
Persons
Males
Males
Pertons
Acute conditions
Persons
Hospitalizations for operation,
males
Hospitalizations for operations,
females, exclusive of del ivery
Persons
Families
Persons
Families
Persons aged 17 years and roverPersons ag?d 17 years and over
Persons
Female aged 17 years and over
Ratio
2.85c
.848
.667
.132
.6s3
.610
.120
.028
1.022
2.812
2.716
.172
.2s3
4.677
.231
.217
1.=
.240
.071
.382
10.s45
1.3s4
.4W
.46s
.3s7
.444
.06s
.76s
.030
.149
.347
.018
.573
.657
.115
.07S
.E60
.064
.324
Total rel
variance62
T
.000S4
.00117
.Oom
.00142
.010s9
.0+)314
.00054
.01029
.0W80
.00118
.00232
.01242
.00245
.00273
.00219
.00342
.00172
.00712
.00102
.00261
.00297
.C0766
.0CQ34
.OCHX%
.00046
.00U%
.00211
.0Ca315
.00272
.01129
.00235
.002!33
.0D035
.W27
.CW68
.000s5
.00004
.00351
.00015
Betwaen-
intwdewm
relvarianm
q
.Cn1024
.GO06B
.00029
.mw
.0QS39
.00204
-.W012
.CQ734
.IX1353
.00015
.00042
.00552
.06C65
-.0000s
.00Q05
.00012
.W1004
-.00270
.00034
.Wol 7
.00004
.C0327
.W016
.ml 1
.Wol 7
.m13
.00131
.CQOcE
.W016
-.00173
-.GQ020
.@3003
.C4)oo9
.Oooo1
.0010s
.0cw3
.Occm
.00078
-.00011
%mpling
relvariance$2E
.00039
.0Q048
.00064
.00077
.00250
.00110
.00066
.00295
.00327
.00103
.00190
.00s90
.00180
.C0279
.00214
.00330
.00168
JX1981
.000S8
.00044
.00292
.0@138
.00018
.00014
.0002s
.CQ023
.00060
.GQolo
.W266
.01302
.00255
.Cm250
.00026
.00026
.WY160
.00W2
.00004
.00272
.CO025
,3s4
,5%
.312
.455
.770
.649
-.230
.713 ‘
.519
.126
.1s0
.445
.265
-.021
.021
.036
.023
-.37s
.332
.281
.014
.428
.475
.446
.262
.354
.621
.35s
.058
-,153
-.0s4
.012
.280
.044
.4m
.269
.122
.224
-.721
31
Table 24. Intewiewer variance study of estimates of components of relvariance for annual estimates of selected characteristics based on 28intewiewer assignment areasin
I08M8A’s: United States, 1962-63-Con.
Numerator
(total number of the following)
Nonresponse Items
Persons with amount of education unknown . . . .
Heads of households with amount of education
unknown ..,...... . . . . . . . . . . . . . . . . . . . . .Families with unknown family income . , . . . . . . . . . . . . . .
Conditions from accidents, unknown if motor vehicle . . . . . . .
Conditions from accidents, unknown location . . . . . . . . . . . .
Demographic characteristics (including health characteristics by age groups):
Chronic conditions with 1 or more bed days, personsaged5.14 years... . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 2544 years . . . . . . . . . . . . .
Chronic conditions with 1 or more bed days, persons
aged2544 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 15-24 years . . . . . . . . . . . . . .
Females, married, spouse present . . . . . . . . . . . . . . . . .
Persons of other races . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 m more bed days, persons
agedunder 5years . . . . . . . . . . . . . . . . . . . . . . . . . . .
Persons of Negro race . . . . . . . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 5.14 years . . . . . . . . . . . . . . .
Chronic conditions with 1 or more &d days, personsaged65 years And over... . . . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 65 years and over . . . . . . . . . . .
Males aged 17 years And over.... . . . . . . . . . . . . . . . . . . .
Persons agedunderl year . . . . . . . . . . . . . . . . . . . . . . . .
Persons agedlyear And over.... . . . . . . . . . . . . . . . . . . .
Females aged 17yearsand over... . . . . . . . . . . . . . . . . . . .
Acute conditions, persons aged 45.64 years . . . . . . . . . . . . . .Acute conditions, persons aged under 5 years . . . . . . . . .
Persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Chronic conditions with 1 or more bed days, persons
aged 15-24 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chronic conditions with 1 or more bed days, personsaged45-64 years . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Persons aged17yearsandover . . . . . . . . . . . . . . . . . . . . .Persons agedunder17years . . . . . . . . . . . . . . . . . . . . . . .
Denominator
(total numb?r of thO following)
Persons aged 17 years and over
Households
Families
Accidents
Accidents
Chronic conditions
Acute conditions
Chronic conditions
Acute conditions
Females, aged 17 years and over
Persons
Chronic conditionsPersons
Acute conditions
Chronic conditions
Acute conditionsPersons agad 17 years and over
Per30ns
Persons
Persons aged 17 years and over
Acute conditionsAcute conditions
Households
Chronic conditions
Chronic conditions
PersonsPersons
Ratio
.016
.019
.055
.008
.004
.017
.246
.052
.123
.633
.011
.008
.118
.231
.029
.C64
.466
.020
.960
.534
.171
.166
3.231
.014
.052
.657
.342
Total rel-variance
92T
.02338
.02767
.01244
.08156
.18992
.00401
.00109
.00168
.00207
.00011
.C6787
.00896
.01480
.00134
.00279
.0C630
.00002
.00210
.00000
.00002
,00252
.00270
.0WC6
.00587
.00119
.00004
.00015
—
Botween-
interviewer
relvariance$/2
.01950
.02218
.00884-.00321
.0s211
.00040
.00023
.00015
-.0+3053
.00002
.01118
-.00147
.00543
.00016
-.00112
.00076
.00000
.00051
,00000
.00000
.00053
.00025
-.00001
-.00009
-.00004
-.00000-.00001
Samplingrelvariance
;2E
——
.00388
.00549
.003s0
.0s477
.10781
.003B1
.00086
.Ocll 53
.00260
.00009
.05669
.01043
.00937
.00118
.00391
.00555
.00002
.00159
.00000
.00002
.00199
.00245
.00007
.00596
.IY3123
.00004-.00015
.
fizJ-~z
T
.834
.802
.711
-.039
.432
.100
.208
,091
-.257
.208
.165
-.164
.367
.118
-.404
.120
.103
.241
.275
.152
.210
.093-.224
-.015
-.037
-.024-.038
the between-interviewer variance is highlyskewed. For example, only about 5 percent ofthe Census enumerator pairs produced highestimates of response variance for four or moreof six nonresponse items.
SUMMARY
In addition to the programs described in thisreport, three other approaches have beenemployed in efforts to assess the reliability and
with records have been made in a number ofspecial studies designed to assess the accuracy ofreporting chronic conditions, frequency ofhospitalizations, and frequency of visits todoctors.1 3‘18 The samples have been limited topersons whose names appear on designatedgroups of records, e.g., patrons of the HealthInsurance Plan of New York City and patients atdesignated hospitals in Detroit. The check startsfrom the records and goes back to a set ofinterviews.
acc;ra~y of the statistics Produced by the HIS. A second approach is comparison of theThe “first is
record checks.
32
estimates “of accuracy throughComparison of survey responses
statistics ofsources and
th; ‘HIS with statis~ics from otherexamination of the internal consis-
tency and reasonableness of the HIS statistics.This is a continuing activity. The re;~ts, how-ever, have not been published.
Third, experimental studies have beendesigned, in effect, to measure the difference inaccuracy between the HIS interviews as con-ducted and other alternative data collectiontechniques. The criterion of the more, the betterhas been explicit in this type of study. That is, aprocedure that gives higher estimates of morbid-ity, hospitalizations, etc., is regarded as havingproduced more accurate statistics than the onewith which it is being compared. Thus thedifferences between the estimates obtained bythe HIS procedure and the estimates obtainedby alternative procedures that give higher esti-mates are regarded as lower bound estimates ofthe biases of the HIS procedure.
None of the methods, singly or in combina-tion, has as yet produced an estimate of thetotal mean-square error of any HIS statistic. Thisis a task of formidable proportions that prob-
ably has not been accomplished for any statis-tical program. Such an estimate requires notonly the assessment of the accuracy of reportingby the respondent but also the assessment of theeffect on statistics of such factors as errors ofcoverage, nonresponse, recording, coding, andother processing errors.
The attainable goals of a program of measure-ment of the reliability and accuracy must, forthe foreseeable future, be regarded as rathermodest ones. The chief benefit to be hoped foris that of providing a basis for detecting andcorrecting shortcomings in the data-collectionand data-processing programs. The second goal isthe rather vague one of increasing the awarenessby the user of the Imitations of the statistics. Inthis way, informed judgments, rather than esti-mates, of the orders of magnitudes of totalmean-square errors can be made at least forsome of the HIS statistics. It is the purpose ofthe research on measurement of error toimprove the quality of these judgments.
REFERENCES
lNational Health Survey Act, Public Law 652, Chapter 510,
84th Congress, 2d Session, S. 3076.2Hansen, Morns H., Hurwitz, William N., and Bershad, Max
A.: Measurement errors in censuses and surveys. BulLInst.Internat.Statist. 38(2):359-74, 1961.
3Hansen, Morris H., Hurwitz, William N., and Pritzker, Leon:The estimation and interpretation of gross differences and thesimple response variance. Contributions to Statistics Presented toProfessor P. C. Mahakmobis on the Occasim of His 70thBirthday, pp. 111-36. Calcutta, Indm. Pergamon Press, 1965.
4U.S. Bureau of the Census: The current population surveyreinterview program, some notes and discussion. Tech. Paper No.6. Washington. U.S. Government Printing Office, 1963.
5U.S. Bureau of the Census: Evaluation and research program
of the U.S. censuses of population and housing, 1960: Accuracyof data on population characteristics as measured by reinter-views. Series ER60-No. 4. Washington. U.S. Government PrintingOffice, 1964.
6U.S. Bureau of the Census: Evaluation and research programof the U.S. censuses of population and housing, 1960: Accuracyof data on population characteristics as measured by CPS-censusmatch. Series ER60-No. 5. Washington. U.S. Government Print-ing Office, 1964.
7U.S. Bureau of the Census: Evaluation and research program
of the U.S. censuses of population and housing, 1960: Theemployer record check. Series ER60-No. 6. Washington. U.S.Government Printing Office, 1965.
‘U.S. Bureau of tbe Census: The cursent population surveyreinterview program, January 1961 through December 1966.Tech. Paper No. 19. Washington. U.S. Government PrintingOffice, 1968.
9B~m, B~b~: Recent res~rch in reinteMew procedures.
].Arn.Statist.A. 63(321):41-63, 1968.10Eckler, A. Ross, and Hurwitz, Wdliam N.: Response
variance and biases in censuses and sueys. Bull. Znst.Internat.
Stat&t. 36(2): 12-35, 1958.1 lHurley, R., Jabme, T., and Larson, D.: Evaluation studks
of the 1959 census of agriculture. Proc.Soc.Statist. Sec. Am.Statist. A. Paper presented at 122nd Annual Meeting, 1962, pp.91-103.
12U.S. Bureau of the Census: Evrduation and research
program of the U.S. censuses of population and housing, 1960:Effects of interviewers and crew leaders. Series ER60-No. 7.Washington. U.S. Government Printing Office, 1968.
13National Center for Hesdth Statistics: Reporting of hospi-talization in the Health Interview Swey. Vital and Health
Statktics. PHS Pub. No. 1000-Senes 2-No. 6. Public HealthService. Washington. U.S. Government Priiting Office, July “1965.
14Natio~ Center for Health Statistics: Health interview
responses compared with medical records. Vital and Health
Statistics. PHS Pub. No. 1000-Senes 2-No. 7. Public HealthService. Washington. U.S. Government Printing Office, July1965.
33
15Natiomd Center for Health Statistics: Interview data onchronic conditions compared with information derived frommedical records. Vital and Health Statistics. PHS Pub. No.1000-Series 2-No. 23. Public Health Service. Washington. U.S.Government Printing Office, May 1967.
16National Center for Health Statistics: Reporting healthevents in household interviews: Effects of reinforcement, ques-tion length, and reinterviews. Vital and Health Statistics. Series
2-No. 45. DHEW Pub. No. (HSM) 72-1028. Washington. U.S.Government Printing Office, Mar. 1972.
17Nationaf Center for Health Statistics: Reporting healthevents in household interviews: Effects of an extensive question-naire and a diary procedure. Vital and Health Statistics. Series2-No. 49. DHEW Pub. No. (HSM) 72-1049. Washington. U.S.Government Printing Office, Apr. 1972.
18Natio~ cent= for He~t,h Statistics: Effect of some
experimental interviewing techniques on reporting in the HealthInterview Survey. Vital and Health Statkt;cs. PHS Pub. No.1000-Series 2-No. 41. Public Health Service. Washkgton. U.S.Government Printing Office, May 1971.
34
APPEND~X IFORMULA FOR COMPUTING ERROR RATE
Error rate =(A+ B+ C+... +G)XIOO E is missed conditions from table II
TC+T’+TH multiplied by 4,F is missed hospitalizations mukiplied
whereby 3,
G is diagnostic errors (inconsistencies or
A is omissions from table I of question-other omissions) multiplied by 2,
TC is total conditions,naire,
B is omissions from table II of question-TA is total accidents, andTH is total hospitalizations.
naire.C is omissions from table A of question-
naire,D is missed conditions from questions The weights assigned to the types of error
6-12 multiplied by 4, reflect the seriousness of the errors.
35
APPENDIX II
TIME AND COST MODEL FOR HIS INTERVIEWING
GENERAL
Interviewer assignments in the HIS are classi-fied as either resident or nonresident assign-ments. Nonresident assignments require theinterviewer’s staying away from home one ormore nights; resident assignments do not. Forconvenience, separate models were developedfor resident and nonresident assignments.
MODELS FOR HIS ASSIGNMENTS
The time T required for a resident HISas~ignment, listing and/or interviewing, isexpressed as
T = ntl +(XISl +S2-h2)d1T1
+ 2A2d2r2 +S1d3r3 + S2t2 “
For nonresident assignments,
T= ntl +(XISl +S2-2)d1r1
+ S1d3r3 + S2t2 + 2d2r2
where
T is total time in minutes for an inter-view assignment;
n is number of interviewed households;t~ is time per completed interview, in-
cluding interview waiting, homework,telephone, and time for non-interviews;
t~ is time per segment for listing;i; is average number of visits per inter-
view segment;A2 is number of days on which travel is
required;S1 is number of interview segments;
S2dld2
d3
r~
72
73
is number of list segments;is average distance between segments;is average distance from home to asegment;is average distance traveled within asegment (including alI visits);is travel speed between segments(minutes per mile);is travel speed from home to seg-ments; andis travel speed within segments.
Values of the parameters in both models areidentical. Some of these values depend on theparticular assignment: n, S1 and S2~~Othervalues are estimated from accumulated data andare assumed to be constants in the model: Al,d3, rl, r2, r3, tl, t2. These estimates areprepared separately for five subuniverses that are
defined by degree of urbanization and popula-tion density. The remaining values, except forX2, are functions of the home address of aparticuku interviewer and the location of thePSU’S where she works. The number of days onwhich travel is required A2 applies to residentassignments only and is a function of otherterms in the equation.
To use the preceding equations for eachinterview assignment becomes very cumbersomeand time consuming. Some simplification isneeded so that the clerical computations can behandled routinely. When the expression for A2 issubstituted in the resident model and terms arecollected, the equation reduces to
T = (CA1)S1 + (CA2)7Z+ (C43)S2
where CA is the computation allowance, so that
36
CA ~ = ?tIdlrl +d3r3 +2d2r2d3r3 _ dl Y1 d3r3
280 280
2d2r2t1 _ dlrl tlCA2 = —+tl
280 280
2d2r2 t2 dl rl tlCA3 = -—+d1r1+t2.
280 280
The nonresident model reduces to
T = (CA1)S1 + (CA2)n + (CA3)S2 + CA4
where
CA1 = Xldlrl + d3r3
CA2 = tl
CA4 = 2d2r2 - 2d1rl .
If the subuniverse parameters Al, d3, T1, 72,and r3 are known (they are actually estimatedfrom data), the computation allowances (CA)are functions of dl and d2 only. Since aninterviewer may treat a PSU assignment asresident one time and nonresident another, bothsets of computation allowances are computedfor each interviewer. The dl and d2 values areflexible and can be changed when circumstanceswarrant, such as the interviewer moves or gets anew assignment, a replacement interviewer ishired, etc.
Regional offices compute the computationallowances and keep a cumulative record (appen-dix VIII) of assignments for each interviewer.
8
37
APPENDIX Ill
SOME THEORY OF MEASUREMENT ERRORSC
SOME DEFINITIONS
The term “survey “ is used to refer either tocomplete censuses or sample surveys. Inconsidering measurement errors we shall regard asurvey as being conceptually repeatable, that is,repetitions relate to the same point in time sothat carrying through the operation once doesnot influence results obtained through repeti-tions. The particular data obtained in a surveyare the result of one trial. This concept providesthe basis for defining variance and bias due toresponse, processing, or other sources of meas-urement errors. Such a postulate can reasonablyapproximate actual conditions for a single sur-vey regarded as a sample of one from such a setof surveys, even though in practice independentrepetitions of a survey may be impracticable orimpossible.
THE DESIRED MEASURE OR TRUE VALUE
We conceive of some desired measure or goalto be estimated from a survey. For simplicity,the assumption is made that the desired or truevalue to be measured is represented as a propor-tion of the population having a specified charac-teristic. Although ordinarily there will be manysuch values to be estimated from a survey, onewill be considered. Thus, it is assumed that thepopulation consists of N persons, each of whomcan be regarded as having the value of 1 if theperson has one or more chronic conditions (orhas some specified characteristic) or as havingthe value of O if the person does not have one ormore chronic conditions (or does not have thespecified characteristic). The desired or true
cThe discussion in this appendix is based on material inreferences 2 and 4.
proportion of persons having the characteristic issaid to be estimated, even for a complete surveyof the population under consideration, becauseonly observations or responses, which are sub-ject to errors, can be recorded.
THE GENERAL CONDITIONS THAT MAYAFFECT THE RESULTS OF A SURVEY
Measurement errors have many differentcauses and depend on the general conditionsunder which a survey is taken. Some of thesegeneral conditions may be beyond the :ontrol orspecification of the survey designer as, forexample, the general political, economic, andsocial situation at the time of the survey.Uncontrolled conditions also include manytemporary chance situations appearing at thetime a response is obtained. Some conditionscan be controlled to influence the quality ofsurvey results in the sense that various aspects ofthe conduct of the survey are specified. Thesespecifications are typically made in the effort toinsure adequate quality and include question-
naire design and survey procedures, personnelqualifications, pay system and rates, training,operating methods, inspection, and controls inthe survey. Such conditions, which may be onlypartially subject to the sponsor’s control, areusually indicated in the form of fixed rulesunder which the survey is to be taken,, Othercontrollable conditions that may be varied bydesign, or may be regarded as varying betweenthe conceived repetitions of the survey, are theparticular choice of interviewers and other per-sonnel chosen to do various aspects of the work,the specific assignments, and other similarvariable factors.
Actually, a survey sponsor is unable to specifyall of the factors, controlled or uncontrolled,that may affect the survey results. He canintroduce certain chance factors explicitly or
38
implicitly, he can impose certain specificationsor conditions, but he must accept the effects ofother uncontrolled factors.
AN ESTIMATE FROM A SURVEY(OR TRIAL) TAKEN UNDER A SET
OF GENERAL CONDITIONS
For simplicity, it is assumed that the survey iseither a complete census or a sample in which allunits have been given an equal probability ofselection, but without, at this time, any otherrestriction on the sample design. In accordancewith the definitions in the previous sections, thisparticular survey is regarded as one trial, i.e., onesurvey from among the possible repetitions ofthe survey under the same general conditions.An observation on a person or other unit in thesurvey has the value 1 if the unit is assigned to aparticular class under consideration or the valueO otherwise. A repetition of the survey on thesame or different units would constitute asecond trial. The generaI conditions, both con-trolled and uncontrolled, under which the sam-ple has been taken will have an effect on theobservations made in a trial.
In repetitions of the survey it is assumed thatthe response actually observed for any individualin the survey can be regarded as having beendrawn by a random process from the possibleanswers he might have given under the samegeneral conditions. (In practice a survey cannotbe repeated independently under the samegeneral conditions because respondents havebeen exposed to the original survey and becauseof other reasons. However, the initial survey canbe properIy regarded as a sample of one from aset of independent replications.) Thus, we aredealing with a random variable xjt, whose valueis 1, if element j is classified as having somecharacteristic on trial t of a survey (x maydenote the class “one or more chronic condi-tions”) or O, ifj is not so classified.
The estimate obtained from a survey, i.e., atrial, is the proportion classified as having thespecified characteristic in triaI t of a survey of nelements:
For example, in measuring the proportion ofpersons with one or more chronic conditions,x = 1 in a particukr survey (or trial) if theperson is classified as having one or morechronic conditions; otherwise x = O. Then
Pt ‘+’Lxjt
j
is the estimated proportion of persons with oneor more chronic conditions.
THE MEAN SQUARE ERROR OF ANESTIMATE FROM A SURVEY
(OR TRIAL)
Continuing the illustration, pt is the surveyestimate of the proportion of the populationwith one or more chronic conditions and ~ isthe true proportion in the population. While wegenerally cannot determine in practice the truevalue U. for any person, we can postulate that
ithe goa of the survey is the true value of theproportion of the population h~ving one ormore chronic conditions. Thus, U, the desiredtrue proportion, is estimated by the statisticsactually obtained in the survey where
Forpt, the mean square error (MSE) is
MSE@t) = E@t - ~z
(1)
(2)
where the expected value is taken over all trials.The mean square error can be divided into its
two main components:
MSE@t) = E@t - P)2 + (P- D)2 (3)
where P is the average of the estimates P ~ takenover all trials and over all possible samples. Thefiist term in equation 3 is the total variance ofpt and the second term is the square of the biasof pt. In practice, we are not able to measure thebias, P - ~, but sometimes we can define andestimate useful approximations to it. For exam-ple, a superior procedure or measurement may
39
be identified as a standard. If s represents thevalue obtained from such a superior measure-ment, then (pt - s) may be used as an approxi-mate estimate of bias.
The total variance of pt can be divided intothe response variance and the sampling variance:
MSE(pt ) = Response variance+ sampling variance
+ interaction + square of bias,
Expressing the response variance of Pt interms of response deviations where
djt = (Xjt - ~j) is the deviation of the responserecorded for person j on trial t from the averagevalue of the responses for person j over all trials,the response variance can be expressed as
dSij = ~ 1 + Pdjtdkt(rt - 1)1
?2[
2 the simple response variance, is thewhere Od,basic trial-to-trial variability in response averagedover all persons. The correlation term pdj~k~reflects the effect of correlated errors intro-duced into the survey process by interviewers,supervisors, coders, and by persons engaged inother operations.
If the intraclass correlation among responsedeviations is zero, the total response variance ofpt is l/n o~. On the other hand if the product(n - l)p is large, the total response variance maybe large even if the simple response variance isrelatively small. Thus, this model of errors insurveys permits the partitioning of the MSE (pt )into a set of components. These various compo-nents may be estimated by means of specialsurveys and experiments.
GROSS AND NET DIFFERENCES
In comparing the case-by-case results of twosets of measurements, the total number ofdifferences affecting the tabulated figure for anygiven class of a population is equal to thenumber of cases included in that class in the firsttrial but not in the second trial, plus the numberof those included in that class in the second trialbut not in the first trial. This sum may betermed the gross difference for the population inquestion.
The netthe Riven
difference of the tabulated figure forclass is the difference between the
total-for the class obtained in the first and thesecond trials. Usually the gross difference willinclude differences in both directions, partly orsubstantially offsetting, and the net difference isthe nonoffsetting part of the gross difference.
For example, suppose that the survey identi-fies each person as having or not having one ormore chronic conditions, and that a total of npersons have been sampled with equal probabilityand included in both a first and second trial.Table I shows that a of the individuals wereclassified as having one or more chronic condi-tions in both the first and second trials, a + c
were classified as having one or more chronicconditions in the first trial, and a + b in thesecond trial. The gross difference in the classifi-cation is b + c and the net difference is(a+c)-(a+b)=c-b.
Now let Xjt represent the result recorded for aparticular person in the first trial and Xjt J theobservation recorded for that same person” in thesecond trial. Furthermore, ~jt is assigned thevahe 1 if the person is recorded as having aparticular characteristic, and O otherwise, andsimilarly for xjt t. Then the response differencefor a particular person in the two surveys isrepresented as e. = Xjt - xjt t.
The sum o~ the values of ~j over the nobservations is the net difference between thetwo results.
Table 1. General representation of results of two sets of measure-
ments on identical persons
Results of
second trial
k=F5Y=Q,
Xjt, = I la Ibl a+b
40
and
n
n n
is the net difference rate.Similarly, b + c is the gross difference and is
2. This follows since e2 = 1 when-equal to~ ej
ever the response obtained in the first andsecond trials are different, that is, (O - 1)2 = 1.The gross difference rate is
~_(b+c).— .
n
GROSS AND NET DIFFERENCES ASEVIDENCE OF RESPONSE VARIANCE
AND BIAS
The estimated variance of the individualresponse difference is
.2 ~(ej-~)2 b+~ (.-b)*=e n-l ‘—-n-1 n(n-1)
where
n n
Often Z* is smalI enough that $ will be verynearly equal to g, and it is then convenient touse the gross difference rate g as the measure ofthe variance of the response differences. In anyevent, g is the mean squared difference for theoriginal and reinterview survey results and pro-vides a useful measure of the consistency orreliabilityy of the measurement process. It can bean exceedingly useful measure of reliability ofresponse with a weI1-designed evaluation studyor reintemiew survey.
If the individual response differences wereuncorrelated from one unit to another, theestimated standard error of the net differencerate would be
Js:_= —.
‘e n
In practice, the individual response differ-ences will not be independent from one unit toanother but will tend to be positively correlated.Under these circumstances, ~ gives a lowerbound for SF. Given certain conditions, anoverestimate of Sz can also be obtained. Theseconditions, would be met if, for example, asurvey is repeated over time or over differentareas or population groups and if the reinterviewsurvey is conducted on different units in each ofthese repetitions of the survey. This is thesituation for the HIS, which is taken each week.The HIS reinterview survey is taken on a distinctset of HIS households each week and sum-marized quarterly. From the group of quarterlyrepetitions an overestimate can be obtained ofthe standard error of the average net difference.(It will be an overestimate of the standard errorof the difference obtained by repetitions of thetwo surveys taken for different samples but withthe same personnel.) Thus, from the HIS reinter-view survey, results of both net differences andnet difference rates are obtained for each mquarters. If ZU is the net difference rate forquarter u, and nu is the number of persons inthe reinterview sample in that quarter, then forthe m quarters involved, the av>rageence rate can be expressed as
m
~ . z%%n
where n = Enu is the total numberviews over the m quarters. Then
net differ-
of reinter-
will be an overstatement of Sz, the estimatedstandard error of the average difference rate.
41
Thus an overestimate and an underestimate ofSz can be obtained. If these are not toodifferent, they yield a measure of the standarderror of the average net difference between theoriginal and reinterview survey results.
INDEX OF INCONSISTENCY
An index of the reliability of measurement;called the index of inconsistency, can be con-structed using the gross difference rate. Theindex of inconsistency is the ratio of the simpleresponse variance, estimated by g/2, to themaximum value it could take on, estimated bythe binomiaI variance p(l - p).
In terms of the table, let p ~ = (a + c)/n; thatis, P is the proportion, based on the originalsurvey, of the population in the specified classand P2 = (a + b )/n is the proportion based onthe reinterview survey. Then,
f=i!?
Pl(wh)+fm-pd”
The estimated maximum value for the grossdifference rate between the survey and reinter-view is pl (1 - pl) +p2(l - p2). This maximumvalue is obtained on the assumption that thesurvey and reinterview were conducted inde-pendently or that the results are positively
correlated to the extentconducted independently.
that they were notA second assumption
is that the reinterview is a repetition of thesurvey process and the expected value obtainedin the survey. Under these assumptions,
91(1 ‘Pi) +Pz(l - Pz) is very nearly equal to2P ( 1 -p) where p is the average proportion inthe original survey and reinterview having thespecified characteristic.
The index of inconsistency lies between O and1 if the assumptions given above hold. However,the estimator can be greater th~ 1.
A simple interpretation of 1 follows, Assumethat a sample of n elements is drawn with equalprobability and with replacement. Also, assumethat the between-element covariance of responsedeviations is zero; that is, that the quality ofresponse of one person is independent of thequality of response for any other person. Thenthe total variance defined in the first term ofequation 3 of the statistic pt reduces to the sumof the simple response vzuiance and the singlerandom sampling variance. The simple responsevariance is equal to or less than P( 1- p).
As the measurement of the specified charac-teristic becomes less reliable, but remains un-biased, the simple response variance increasesand the sampling variance decreases; the totalvariance remains constant. A high index ofinconsistency is associated with a high level ofresponse error.
42
APPENDIX IV
LEAST-SQUARES SOLUTION
The variance model used in the study is
(1)
In this model, the interviewer variance ~ isassumed to be dependent only on the number ofinterviewers, which remained constant over thestudy period. On the other hand, the samplingvariance term V~/n is a function of sample size.In equation 1, n is the number of quarters ofdata used to compute the individual estimates oftotal variance.
The data used are sets of estimates ~ for~=1 , . ...8. Each V: is an average of theestimates based on n quarters. The @n areestimates of Vf(n) for n = 1, . ...8. The modelgiven above represents the functional relation-ship existing between the variables HT, ~, andV; as n varies. The problem, then, is to estimatethe parameters of this function so that theestimates of ~ and V’ can be identifiedseparately.
There are several ways of estimating theseparameters. The ;al is to estimate @~ so thatthe estimates of # are as close as possible to
T(n)-2the observed values Vn. The method used wasthat of least squares. The criterion for theleast-squares estimate is to make the sum of thesqu~es of the difference between_the estimateof V; ~ ) and the observed values ~ as SmalI as
Ipossib e, that is, to minimize the value of @,where
These differences can be seen graphically asfollows:
I I I I I I I I
1234567 8
MEASURE OF SAMPLE SIZE (NUMBER OF QUARTERS)
As required by the model, a hyperbolic func-tion. rather than a strakht line. was fitted to the~ata~ No boundaries w~re placed on the value of~; it can become negative as n + 00.
Using the method of least squares, the partialderivatives of equation 1 are taken
(3)
If
then
$$=2W+~-+(’)=0 ‘5)
43
and the two partial derivatives @E and fi can besimdified to
Equation 5 becomes
Equation 6 becomes
Since
and
vi =~ 527_ (2-718)2.
8
(7) = 1.655 ‘@- 0.562 ~~: (9)
(8)
and
= 0.125 ~ ~n -0.340 ti; . (lo)
By substituting in the value for ~~ ,
These estimates of ~ and @E for o~e quarterwere used to compute the values of V; for oneto eight quarters. The values shown in tables 23and 24 are estimates for a sample covering fourquarters.
44
APPENDIX V
HIS OBSERVATION REPORT (NHS-HIS-406)
FORM t11$406 uS. DEPARTMENT OF COMMERCE 1. Regional Office 2. Mu(a.7.491 BUREAU OF THE CENSUS
HEALTH INTERVIEW SURVEY 3. Name of interviewer ~Code
OBSERVATION REPORT I4. Date observed 5. ~r:: observed - 6. Date of last observation
I ToI
7. Type of observation
= Systematic n Initial - First assignment m Other - .Specify=
m Reinterview rej ection n Initial - Second assignment
8. Reference uotesfor special attention -e.g., retiterviewresu1&, ~ Smple Errornotes from last observation, etc. I
Type A Productionrate rate
; B-
I B-
9. Segment coverage
Tall of1
Interviewed Type A T{:CB S#nnm:
house .Idsor callback
observed andtype B TA NTA
Item IExplain each “No” answer below Yes No lN/A Yes w lN/A Yes No lN/A
I Did interviewer correctly -
A Use maps, locate segments, locate sample addresses?
B Check area segment boundaries?
c Canvass area segments and look for concealed units?
D Fill area segment listing sheets?
E Determine “year built” when required?
F Fill Cols. 8, 9, and 13 of B segment address lists?
G Fill extension sheets in B segment? —10. GENERAL PERFORMANCE Explain below
E;ec$2c%& Needs im- fJnsat?s-
tfotapplicable
Evaluate each point for !he entire day’s observationpr.avemcnt factoiy
L Asking probe questions when needed and only when needed
2. Neutral probing
3. Allowine respondent reasonable time for answeriniz auestions
4. Maintaining a business-like attitude and rapport with respondent
5. Listening carefully to respondent
6. Accurately recording respondent’s answers and completing allrequired entries on questionnaire (Evdute this item on the basisof respondent’s answers and your edit of the questionnaires~fter the interview.)
7. Accepting suggestions and criticisms
8. Applying housing unit definition
9. Listing and interviewing within special dwelling places
10. Planning itineray
ASK INTER VIEWER DURING THE DAY:
Is *hare ❑y particular pmi of the proc.dur. y.. feel unsure about or would like to hove covered by further tmining? IRemarka
45
-
ast name
erson No. 1 2 3 4 5 6 7 8Handled
correctly:Xf’’No,”
ame explain
,ge
ROBE Person Ques-AGES tion Yes No
IONDl- Person ConditionION
Yes No Person Condition Yes No
AGE(S) I
I_JNone
I
I
IIOSPI.‘AL
Person Page Yes No
‘AGE(S)1
_J None2 I
3
fK;:R Person Yes No SljPpLE. Person Yes NoMENT
~ None
‘ERSON Person Yes No‘AGE(S) m None
‘1
2 HOUSE- Ques-::KJ; tion
Yes No
3(Es year
“f4
bul t andcoverageitems)
5
6
OMMENTS (including edit)
46
RESULTS OF OBSERVATION
Dverall evaluation on all phases of work
o Excellent n Satisfactory n Needs improvement n Unsatisfactory
Comments on general performance
Recommendations for next observation
3BSER VER: Note any area ofthe questionnaire or interviewer’s instructions which ia your view require modificationm clarification.
)bserver’s signature Date
,To be COWAM {fobswvmr mcemrrtwdk Wlwr{o#* probrMen# w Offrer aAIa }~~ & ‘- , .=
2omments of the Regional Director
legional Director’s signature Date
47
APPENDIX VI
HIS RECONCILIATION FORM (N HS-HIS-R-lX-T)
A. RECi)NCILIATiON SECTION FOR BEO DAYS, B. RECONCILIATION SECTION FOR CONDITIONS ANb HOSP ITALI ZATIONS REPORTED IN ON E INTERVIEW
RESTRICTED ACTIVITY DAYS, ANO TIME LOST We arc interested In finding out more about conditions (hospitalizations) which were reported at one time but not atOAYS IN PAST TWO WEEKS Ccr.dit ion D.te .1 ho.pitalizatlo,’ Condition
The original qucstionnoire showed , . . and I have . . .
Dale ,1 ho.ptt.liz.lio.
which is the correct information?
Original R.conclliatlm Explmn.tlcm ! Ex+mti.am
.%. Bed days Bed dmp❑ None o Nor.. I
5.. Cut dews day. Cut down dr,ymD None n None
Sf. WO,k daya Work d.ysm N... Non.
59. School days Soh.aol d.y.n No.. m NOW n Sustained D Deleted t ❑ Sustained fJ Deleted
C. RECONCILIATION SECTION FOR DIFFERENCES WITHIN MATCHED CONDITIONS - The origins I qutsti$mnai re showed . . . and 1 hove . . . . which is the correct i
1 2 3 4!.la:ch Rcinterviowaondition No. 1. Person No. Malcb Rekmavlmv condition No, 1. Pa,ac.a No. MmLch R=hterview condition No., 1. Person No. M.tch R.intmvimv aondilion No. 1. Pera.ar, No.
I
Original R.concl Ii.tlc.n Origin.al I R.em.ili.N.n Original R..oncili.tion Origiml I2.
R.ceneiliotlm
2. 2.D Ye.
2.D No l_JYe. a N. n Yes D No OYOS ON. n Ye. i_J No d Yes n No D Ye. a No m Ye. l_J No
30. 2-3. 30, 3..$
I I I3b. 3b. 2b. 3b.
I3.. 3c. %. 3..
I3d. 3d. 24 3d.
3*. 3*. 2., I 3*,
Oriqinal R*co.ci lioNon Original R.cc.ncl NoNon Otlginal ec.a. cillatim Oriolnol R*conclliotion
90. 0 Ye. ON. O Y.. ON. 90. a Ye. a No DY., ~No 90. ❑ Yes lg No O:YC. m No 9.s. ❑ Ye, a No a Yes m No
9b. a Yes D No D Y,. a No 9b. D Yes I_J No n Ye. n No 9b. a Yea O No I_JYe* a No 9b. OYC. UN. n Y,. a No
10. Days I D.y. I 10. Day. I n.Y. I 10. Day. I D.T= 10. Days Days
I!u222” !‘y- ‘N” !20”‘yes ‘we !oye”‘N” 120”““B ‘N” I‘yes‘N” Im”“e’ ‘N” I““” ‘No
25b., D Ye. 0 No O Yes O No 25b. a Ye. a NO ❑ Yes ON. 25b, O Yes a No a Y.S ON.
25..
El
Cured
8
Cmod 2s..
N
C.rcd
B
Cured 25..
la
Cured
El
CaredUnder c.anlrol Under control Under control Cnder con,rol
Other - SPOCNYF
Under c.ntr.l Undmr 00.uo1Dther - SPaCllY~ Other - s.uoclfy~ O,ha - s#ec1fy7 Dther - S@fy7 Other - SL.eclfyj?
1. I I I I I25d. hlonth(.): Year(s) M.anihls) i Yom(s) 25d, Mcmlh(d ; Ycnr(.4 Month(s) : Yew(.) 25d. Month(s); Yc.r(s) Mmlh(s) ; Y..,(,) 25d, Mo,:h(s) : Yem(s) Month(e) I Ye.r(8)
L, I -!,,0.. ..s..,,. ”., x.’, (..,.,>,
48
BUT NOT IN BOTH INTERVIEWSmother. Con you think of any ●xplanation for . . . not hovinq been reported }n the (ari~inol interview/our interview today)?
C.ndiliom D.!. .1 b.spit.li..ticm Camditi.. Dmte .1 ho.pit.Nz.LIem Condltic.m Due of ho.piuNz.ti.m
E.xp18s.tI.. S,xpl.n.tion Explu.tien
> n Su.tslmed m Deleted n s“a.ined m Deleted . i_J %stained u Deleted
nfonnatlon?
s 6 7 8M.tch ?lelnterview ccmditlon No. 1. P.m.. No. Mmtch Rcintavlew condition No. 1. Person No. Mntch Fleintetvlew ca.dith N.. I. Pc-o. No. Malcb Fleknewi.w c.mditi.. NO. L Person N..
Orlgln.1 Rmcortclllmtio. Odgi”d R.c..cili.tion Original I Re..nciIiaNon Oflgln.1
2.
R.cottci[ia*ic.n
UY.S ONO a Yes ❑ No 2“0 Y.* ~No O Yes ON. ‘U Ye* UN. o Yes ON. ‘ore. n No ~ Yes ❑ No
30. 3.. 30. 35
3b. 2b. 3b. 3b.
3c. 3.. 3.. 3C.
t ,13d, 3d, 3d. I 3d. i.
3.. I 3*’ I 3“” I ‘“” IOrldnd R.conclllmtlon Oric.1..l Reconciliafie. Odqi”d R...ncil lotion Original R..onciliotjon
9.. a Ye. DNo ~YG. ON. 9.. a Ye. ~ No a Yes m No 9.. 0 Ye. o No a Yes D No 9.. n Yes a No u Y.. a No
9b. D Y.. mNo ‘a Ye. D No 9b. 0 Ye. u N. m Yes a NO 9b. mYe. ON. ‘I_J Yes a N. ?b. i_JYu i_JNo a Ye. a No
10. D.p Day. 10. Dsys n.p 10. D.p Dv. 10. D.,. D.y.
I I I 1 u I I I 1 I
I I —— I I I I !25d. Month(s); Year(.) Month(.) j Ycu(el 25d. Month(.); Yc.is) M.mh(.) i Year(s) 25d. Momth(,) ; Year(.) Month(.) i Y..r(.) &!. M,mb(s) ; Ye,r(d Month(.) , Y-d.;
8i-i1
u, . ...!...
49
SUMMARY REPORT
APPENDIX
OF NHS-HIS RE
Vll
NTERVIEW (N HS-HIS-R-401)
FORM t’4ii5-H15-i?-401 U.S. DEPARTMENT OF COMMERC 1. Interviewer’s ‘me / Code I Telephone No.[s-5.66)
2, Regional OfficeSUREAU OF THE CENSUS 1 I
1 I I3. Reintervi ewer’s name I Code I Job title I 4 Psu i
SUMMARY REPORT OFI I n Program Supervisor .
NHS-HIS REINTERVIEW 5. Reinterview datem Altemaien Senior l.terviewer 6. Sample
m Od-ter
Section I - COVERAGE CHECK OF HOUSING AND OTHER UNITS Section II
HOUSEHOLD COMPOSITION CHECK .
Reinterview segmentPart A - Area Segments Part B P art c Number of persons
Number of units B Segment Check Number Segment Household Before
Number ofListed Extension sheet of wrong No. reinter- Added Del etedserial No. “iew
sample unitsbefore
house-before
entriesNumber Type Added
holds (1) (2)
rein ter-(3)
rein ter- Deleted(4)
inter-(5)
view view f Correct Incorrect viewed
(1) (2) (3) (4) (5) (6) (7) (8) (9)
rotal thisreinterviewPrevious
I
,,
:umulative total
4ew:umulative total .,, ,,
f Exclude units in special dwelling places in NTA segments and in ‘“large” specialdwelling places.
If the most recent listing for an NTA segment was performed by another person, enter namein ““Explanation of Differences” and prepare separate form NHS-HI%R-40 I reportingcolumns (4), (5), and (6) data for the segment.
explanation of differences in sections I and II
‘Give reference to section and segment, numbers of added or deleted units, and to segmentmd serial numbers of added or deleted persons. Describe type of error If column (8) is:hecked. Explain changes in classification of “year built” here OISO.)
II I I I
I
Totol this reinterview I--u
Previous cumulative total Iu
50
aecnon Ill - LUN IEN I LtltLK I
Part A - Personal Characteristics Part B - Characteristics Within Condi- Part C - Number of Conditionstions and Hospitalizations rnd Hospitalizations
Segment HouseholdDlff eren t Same Different Same
respondentDifferent
respondentSame
respondentNo.
respondent Is=rial No. Nu$fber
respondent respondmt 1
Dlffer- Nuo~ber Differ- Number of Di f f=r. Number o Accept- Accept-
*Q* =“==* .:=;: *f :;f&- Differ- Differ-ences ability ability
persons persons ‘“==s base 2ences 3 base 2
ences 3
(1) (2) (3) (4) (5) (6) (7) (8) (9) (lo) (11) ( 12) (13) (14)
Total this interview
ROViOus cumulotiva*tal
New cumulative total / / t1 Includes adults responding for children under 19. * Definitions
a (@ check conditions ?md hospitalizations Plus sustained conditions zmd hospitalizations I (I/)checks: Each deck represents one conditionplus number of conditions and hospitalizations added and sustained to reconciliation form. or hospitalization reported on both interviews. I
3 Number of original conditions and hospitalizations deleted from reconciliation form plus Sustained co”ditio”s or hospitalization are ~o=enumber of conditions and hospitalizations added a“d sustained to remnciliation form. which are reported only cm o“e interview ~d
retained after reco”ci Iiatio”. I
Section IV - ACTION TAKEN
Note: This section must be filled if the interviewer has been reiected in a“y of the preceding =ections. Explain action taken\or planned to retrain interviewer if her work has bee” rejected (R). (If more space is required, use odditio”cd shee+.) id
ection VPERFORMANCE
Decision
section A - AcceptR - Reject
2
aI-Al-cII
II-A
II-B
Ii-c] IUScor.w.oc
51
APPENDIX VI
PRODUCTION GUIDE FOR
I
NHS (11-102C)
p#ll. 102C U.S. DEPARTMENT OF COMMERCE 1. REG1ONAL OFFICE 2. SAMPLEBUREAU OF THW cENSUS
B-
PRODUCTION GUIOE FOR NHS 3. INTERVIEWER 4. INTERVIEWER CODE
INTERVIEW HOUSEHOLDSSEGMENTS INTERVIEWED
LIST SEGMENTS SUPPLEMENT THIS ASSIGNMENT CUMULATIVE
Lye: wEE K PsuCA
NUMBER c: NUMBER y NUMBER =: 4 ALLOWED ALLOWED PAYROLL AM;::::: PAYfiOL1.PRODUG
NVMBER MINUTES MINUTES MINUTES MINUTES TlON
(.) (b) (d (d) (e) (0 (s) (h) (i) o) cd (1) (m) (n)
RATIO
(0) (P)
1
2
3
4
5
6
7
8
9
10
11
12
13
14 I I
15REMARKS
‘7
I I
CHIEF, FIELD DIVIS!OH COPY:USCOMM.OC 21760 P-68
52
INSTRUCTIONS FOR COMPLETING FORM 11-102C
FORM 11-1OZC (10-7-63)
GENERAL
Maintain a Form 11-102C for each interviewer on athree month (NHS Sample) basis. At the end of the
three month period send the yellow copy to Chief,Field Division, retain the original in your files andstart a new 11-102C for each interviewer. One lineof the 11-102C should be completed for each inter-view and/or listing assi~ment.
Columns (a)-(p)
Enter the week in column(a) and the PSUin column (b).
Post the computation allowances to columns (d),(f), (h), and (i), using Production Standards Memo-randum No. 5 (Formerly GAM No. 70) and yourknowledge (from payroll records) of whether theassignment was overnight or non-overnight. Enter
the workload associated with each alIowance incolumns (c), (e), and (g).
Enter the number of current supplements (if any)compieted in co Iumn (j), and the allowance per unit,as given in Operations Memorandums, in column (k).
Multiply each allowance by its workIoad and addthe results to obtain the totaI allowance. (Be sureto add in CA 4.) Enter this total in column (1).
Enter the payroll minutes in column (m).
Revise the cumulative production ratio in columns(n), (o), and (p).
USC OMhl-DC 21759 P.6s
53
* u. s. OCXWNMENT p~G OFFICE :1973 5E,.2121A.8
f
Series 1.
Series 2.
Series 3.
Series 4.
Se7ies 10.
Series 11,
Series 12.
Series 13.
Series 14.
Series 20.
S6nies 21.
S~”es 22.
*
VITAL AND HEALTH STATISTICS PUBLICATION SERIES
Originally Public Health Service Publication No. 1000
Programs and collection procedures. —Reports which describe the general programs of the NationalCenter for Health Statistics and its offices and divisions. data collection methods used, definitions,and other material necessary for under standing the data.
Data evaluation and methods research. —Studies of new statistical methodology including: experi-mental tests of new survey methods, studies of vital statistics collection methods, new analyticaltechniques, objective evaluations of reliability of collected data, contributions to statistical theory.
Artalvtical studies .—Reports presenting analytical or interpretive studies based on vital and healthstatistics, carrying the analysis further than the expository types of reports in the other series.
Documents and committee reports. —Final reports of major committees concerned with vital andhealth statistics, and documents such as recommended model vitai registration laws and revisedbirth and death certificates.
Data jkom. the Health Interview .%rrvev. —Statistics on illness, accidental injuries, disability, useof hospital, medical, dental, and other services, and other health-related topics, based on datacollected in a continuing national household interview survey.
Data Jrom the Health Exmnination Survey. —Data from direct examination, testing, and measure-ment of national samples of the civilian, noninstitutional population provide the basis for two typesof reports: (1) estimates of ,the, medically defined prevalence of specific diseases in the UnitedStates and the distributions of the population with respect to physical, physiologi~l, and psycho-logical characteristics; and (2) analysis of relationships among the various measurements withoutreference to an explicit finite universe of persons.
D@Z from the Institutional Population Surveys —Statistics relating to the health characteristics ofpersons in institutions, and their medical, nursing, and personal care received, based on nationalsamples of establishments providing these services and samples of the residents or patients.
Data from the Hospital Discharge Survey. —Statistics relating to disci:;rged patients in short-stayhospitals, based on a sample of patient records in a national sample ~f hospitals.
Data on health resources: manpower and facilities. —Statistics on the numbers, geographic distri-bution, and characteristics of health resources including physicians, dentists, nurses, other healthoccupations, hospitals, nursing homes, and outpatient facilities.
Data on mortality .—Various statistics on mortality other than as included in regular annual ormontruy reports —special analyses by cause of death, age, and other demographic variables, alsogeographic and time series analyses.
Data on rtatality, mawiage, and divorce. —Various statistics on natality, marriage, and divorceother than as included in regular annual or monthly reports-special analyses by demographicvariables, also geographic and time series analyses, studies of fertility.
Data from the National Natdity and Mcrrtality Surveys. — Statistics on characteristics of birthsand deaths not available from the vital records, based on sample surveys stemrrdng from theserecords, including such topics as mortality by soci~onomic class, hospital experience in thelast year of life, medical care during pregnancy, health insurance coverage, etc.
Fbr a list of titles of reports published in these series, write to: Off ice of InformationNational Center for Health Stat isticsPublic Health Service, HSMHARockville, Md. 20852
/
