DATA EVALUATION AND METHODS RESEARCH Series 2Number 44

QualityControlina NationalHealthExaminationSurvey

DHEW Publication Nci. (HRA) 74-1023

U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFAREPublic Health Service

Health Resources AdministrationNational Center for Health Statistics

Rockville, Maryland

Vhl and Health Statistics–Series 2, No. 44Reprinted as DHEW Publication No. (HRA) 74-1023

August 1973

Library of Congress Cutolog Card Number ?0-608550

CONTENTSPage

Introduction . . . . . . . . .The Health Examination Survey .Field Operations . . . . . . .

Quality Control in Data Collection .

Means of Reducing Nonresponse . .Advance Publicity . . . . . .Interviewing Techniques . . . .Examination Policies . . . . .Evaluation of Nonresponse . . .

Control of Missing Data . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Means of Reducing Measurement Process ErrorPlanning, . . . . . . . . . . . . .Standardization of Test Environment . . .Standardization of Testing Process . . . .

Use of Mechanical Devices . . . . . . .

Selection and Training of Examiners . .Reduction of Subject Errors . . . . . .

Processing Errors . . . . . . . . . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Surveillance and Evaluation of Residual Measurement Process ErrorMonitoring Systems . . . . . . . . . . . . . . . . .

Biases and Controls in Replicate Measurements . . . . . . .Selection of Replicate Examinees . . . . . . . . . . . .Additional Replicate Data . . . . . . . . . . . . . . .Evaluation of Residual Measurement Error . . . . . . . .

References . . . . . . . . . . . . . . . . . . . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

,

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

112

3

5556

79

10

101112121313

14

141415161718

21

.,.UI

I SYMBOLS

Data not available ----------------------- ---

Category not applicable ------------------- . . .

Quantity zero ---------------------------- -

Quantity more than O.but less than 0.05----- ().0

Figure does not meet standards ofreliabilityor precision ------------------ *

QUALITY CONTROL IN A NATIONAL HbW..”fF!EXAMINATION SURVEY

Wesley L. Schaible, Division of Health Examination Statistics

INTRODUCTION

The Health Examination Survey

The Health Examination Survey is one ofthe major survey programs conducted by theNational Center for Health Statistics under theauthority of the National Health Survey Act of1956. This act provided “(l) for a continuingsurvey and special studies to secure on anon-compulsory basis accurate and current sta-tistical information on the amount, distribution,and effects of illness and disability in the Unit ec!lStates and the services received for or because ofsuch conditions; and (2) for studying methodsand survey techniques for securing such statis-tical information, with a view toward theircontinuing improvement.”

To obtain information about the health ofthe U.S. population the National Health Surveyis divided into three survey programs. 1 One ofthese, the Health Interview Survey, is primarilyconcerned with the impact and social dimen-sions of morbidity. It collects data by contin-uously sampling and interviewing the civilian,noninstitutional population of the UnitedStates. A second, the Health Resources Program,provides statistics on the health of the institu-tional population, the utilization of medicalfacilities, and health manpower. The third pro-gram of the National Health Survey is the HealthExamination Survey (HES). z In the HealthExamination Survey primary emphasis is on thecollection of data by direct examinations andtests on a probability sample from the civilian,noninstitutional population of the UnitedStates. Such examinations and tests can yieldstandardized information about diagnosed con-ditions, including those which persons may failto report or may be incapable of reporting in asurvey based upon individual interviews. They

can also reveal previously undiagnosed, unat-tended, and nonmani fested conditions.

The overall plan of the Iiealth ExalilinationSurvey is to conduct successive, separate sur-veys, each with specific objectives. Each of thesesurveys has been referred to as a “cycle.” Thusfar in HES the objective of each cycle has beento obtain data on specific health characteristicsof a certain age segment of the U.S. population.

Collection of sample data in Cycle I beganin October 1959, and was completed in Decem-ber 1962? The target population consisted of allcivilian, noninstitutionalized adults in the UnitedStates aged 18-79 years. The probability sample,which was based on households, identified 7,710sample persons .of whom 6,672 ~.vm-cexamined.The 2-hour examination focused p:tilicularly oncertain cardiovascular diseases, m-thritis andrheumatism, and diabetes. Varicw other exami -

nation data were collected, includi~l~ nleasure-ments of visual and auditory acuity ‘m-id bloodpressures; electrocardiograms; findings of medi-cal and dental examinations; blood analyses;hand, foot, and chest X-rays: and numerousbody measurements. Additional health historydata were gathered by interviewers and self-administered questionnaires.

The data collection phase of C~clc il lICI+UIin July 1963 and ~vas completed in December1965.4 This cycle was concerned ~rith nc)l”llll~ti-tutionalized children 6-11 years age: and f~om ~

total of 7,417 sample children, 7,119 ivereexamined. The 3-hour examination ~vas desi,qnedto obtain measures related to heulthy grmvthand development. The types of data cdlecLecl

included visual and auditory acuit~’, blood vrm -sures, electrocardiograms and spirograrlis, medi-cal and dental examinations, hand-wrist andchest .X-rays, psychological tests, and numerousbody measurements. Additional data ~,iwi-eob-tained by means of several questionnaires inclL!d-

1

ing one on household composition, two con-cerned with health history and habits, and one,completed by school personnel, on the child’sacademic and social achievement. Also, a copyof the birth certificate of each consentingsample child was requested from the appropriateState registrar to verify date of birth and providesupplemental information.

Data collection in the Cycle III samplebegan in March 1966 and was completed inMarch 1970.5 Of 7,514 sample youths identified6,768 were examined. The segment of concernin this cycle was the noninstitutionalized youthsaged 12 through 17 years. As in Cycle II, a3-hour examination focused on factors andconditions related to healthy growth and devel-opment. A physician performed a medical exam-ination designed with special attention given toitems relating to adolescent growth and develop-ment. A psychologist recorded the youth’sperformance on various tests to study thegrowth of certain aspects of thinking, socializa-tion, and motor coordination. Other parts of theexamination included a dental examination;tests of visual and auditory acuity, color vision,respiratory function, and grip strength; exercisetolerance on a treadmill, an electrocardiogram;an X-ray of the chest and one of the hand andwrist; and weight, height, skinfold thickness, andother body measurements. A blood sample wastaken for hemoglobin and hematocrit, serolog-ical tests, and extensive blood typing. The serumchemistries determined were total cholesterol,uric acid, and protein-bound iodine. In additionto data collected in the various examinations,further information was obtained from sevenquestionnaires: a household interview question-naire which provided household compositionand demographic information, a marital historyof the parents, a medical history of the youth bythe parent, a health habits and history of theyouth completed in the home by the youth, ahealth behavior questionnaire completed by theyouth at the examination center, a questionnaireadministered by the nurse to obtain menstrualinformation from female examinees, and a ques-tionnaire completed by the school which theexaminee attended. Ako, a copy of each consent-ing sample youth’s birth certificate was re-quested from the appropriate State registrar. As

in Cycle II the certificate was used ‘to verify dateof birth and obtain supplemental information.

Field Operations

The examinations were performed inmobile examination cent ers which were trans-ported throughout the country to the sampleareas. In Cycle I, two such centers were operatedsimultaneously in a “leap-frog” pattern alongthe scheduled route of 42 sample locations. Twocenters were again employed in Cycle II, visiting40 sample locations. In Cycle III, one center wasused to return to the same 40 locations as inCycle H. By using the same locations, and thusthe same households, in addition to the primaryobjective of obtaining the desired data foryouths aged 12 through 17, longitudinal datawere obtained for approximate y 35 percent ofthose youths who were examined in Cycle II.

In Cycle I, examinations were scheduledindividually at half-hour intervals with all ex-aminees going through the same sequence oftests. In Cycles 11 and III it was felt that theyouths would be more at ease if a number ofthem were present in the center at the sametime. After considering alternative schedulingarrangements, it was decided to bring six youthsto the center at the beginning of each morningand afternoon. The examination was performedaccording to a schedule designed to eliminatesituations in which one test might have anundesirable effect on a subsequent test. Efficientutilization of facilities and staff time, a secondconsideration in scheduling, led to differentialsequencing of examinees.

For discussion purposes the HES field staffmay be divided into three interrelated butdistinct groups. The first group consists ofBureau of Census interviewers and their super-visors, who are not permanent members of theHES staff but are assigned to HES from theCensus District in which the examination cent$ris located. These interviewers make the initialcontact with the households and obtain basicinformation pertinent to HES operations.

The second group is composed of the FieldOperations Managers, Administrative Assistants,and Health Examination Representatives, all ofwhom work in or out of an office located near

2

the site of the examination center. These mem-bers of the HES field staff are responsible foradditional interviewing in households whichhave eligible youths, checking for the presenceand consistency of questionnaire information,and arranging for transportation and variousother services. They are also responsible for themany other administrative functions necessaryfor an effective operation. The Health Exami-nation Representatives serve a particularly im-portant and delicate role in obtaining the coop-eration so necessary in surveys of this type.

The third group is the examination centerstaff, which thus far has consisted of physicians,dentists, psychologists, nurses, technicians, andclerical assistants.

The sample design, which was similarthroughout the three cycles, is that of a multi-stage, stratified probability sample of looseclusters of persons in land-based segments. Thesuccessive elements dealt with in the process ofsampling are primary sampling unit (count y orgroup of contiguous counties), census enumera-tion district, segment (a cluster of households),household, eligible person, and finally, sampleperson. The basic design is essentially self-weighting, although operational efficiencies re-quire some modification of sampling rates.However, in all situations the probability ofinclusion of every sample person is known. ForCycle I, the variables of stratification weregeographic location and population density. ForCycles 11 and HI the 1960 Census data wereavailable, and rate of population change between1950 and 1960 was added as a third variable ofstratification. This extremely abbreviated treat-ment of the subject may be supplemented bythe more complete discussions given in publica-tions more concerned with this aspect of thesurvey. 3-6

QUALITY CONTROLIN DATA COLLECTION

Two sources of error may enter into datacollection activities—— sampling error, that errorwhich occurs because data were gathered from asample rather than from the entire population ofconcern; and nonsampling error, a somewhat

loosely defined collection oferror.” If a sample is chosen

“other sources ofin an appropriate

manner, the sampling error can be estimated, afeature which is of concern in the design of allsample surveys.

Nonsampling errors, generally neglected inthe statistical literature until recently, are nowcommonly considered in the planning, conduct,and evaluation of surveys. Increasing the samplesize, a method frequently used to reduce sampl-ing errors, is not effective in the reduction ofnonsampling errors, which instead of diminish-ing as the sample size is increased may remainconstant or perhaps become larger. Therefore, inlarge samples such as those used in HES,nonsampling errors are of primary concern andit is on the identification, evaluation, andcontrol of nonsampling error that quality con-trol in data collection is centered. QualitycontroI not only implies a concern with keepingoutput (usually repetitive) within certain levelsof quality, but also the diverse types of activitywhich in general promote the quality of theproduct. In HES, the products are nationalestimates of various U.S. population characteris-tics such as those briefly described in theintroductory section, and the prescribed level ofquality is that of the highest attainable accuracyand precision within the usual limitations dic-tated by acceptable procedures and reasonablecosts.

Any attempt at producing a relativelycompIete list of types or sources of nonsamplingerrors would be a lengthy task. However, manytypes of nonsampling errors which arise in datacollection are included in the following broadand occasionally overlapping categories:

Conceptual errors–e.g., errors due to survey

design, definitions, or speci-fications.

Nonresponse errors-e.g., errors due to non-coverage, lack of respond-ent cooperation or recall,collecting agent omissions,illegible entries, lostrecords.

Measurement process errors–e.g., errors due tolack of environmental con-trol, poorly calibrated meas-uring instruments, badly

3

worded questionnaires, im-proper examiner or subjectinfluence in the perform-ance of measurements.

Processing and analysis errors–e.g., errors incoding, punching, computa-tion, presentation, interpre-tation.

The purpose of this report is to presentgeneral procedures which are important in HESefforts to identify, control, and evaluate nonre-sponse and measurement process errors includ-ing related conceptual errors; a brief section onHES processing procedures is also presented.

Reduction of nonresponse, including miss-ing data, is a primary concern of the qualitycontrol program. Various publicity practices andinterviewing techniques are employed in theeffort to reduce nonresponse. A second majorconcern of the total quality control effort isthose errors associated with the measurementprocedures and instruments. Detailed proceduresrelating to the data collection process are fol-lowed to help insure that HES data are of highquality. Aside from initial procedures designedto reduce errors, constant surveillance and evalu-ation is necessary, especially in a lengthy surveysuch as HES. Closely related to this aspect is thenecessity of providing, when possible, data ofsuch a nature that objective and quantitativestatements of residual errors and uncertaintiesassociated with reported values can be includedin reports of survey findings. The final majorresponsibility of the quality control program isin the processing of data where various verifica-tions, edits, and consistency checks aid in thediscovery and reduction of errors.

Aside from classification of types of non-sampling errors, it is particularly convenient anduseful to divide nonsampling errors into variableand systematic errors. Errors of a variable natureare usually due to a combination of known andunknown factors which singly are usually insig-nificant. Their combined effect on obtainedmeasurements and especially on estimates suchas means are lessened by the fact that they are,by their nature, as likely to overstate as tounderstate the true measurement, and so there isa tendency for errors to “cancel out. ” Sys-

tematic errors or biases, on the other hand, aredeviations from the true measurement which arethe result of circumstances which, if known andunderstood, would be expected to produce somepredictable direction and magnitude of differ-ence between the true measurement or estimate”and its obtained value. For example, if theillumination for a visual acuity test is less thanthat required for good results, a bias will resultin the data, since the examinees’ responses willtend to be consistently different (in the direc-tion of poorer visual acuity) from that whichwould have been obtained with proper ilh-un.ina-tion.

Investigations of nonsampling error oftenemploy models containing bias and variableerror components. More complex models mayalso consider biases which operate for only aportion of the data collection activities and alsobias and variable errors identified by major orsimilar sources. Although models separating var-iable and systematic errors are often intuitivelyappealing as well as theoretical y necessary, theiractual application is not lacking in problems,both theoretical and practical. For example,there can be many different solutions to theseemingly basic problem of deciding when anerror which is basically constant but not strictlyso throughout data collection, should be treatedas a bias or as a variable error. Further discussionalong these lines is presented with the evaluationof residual measurement errors where a simplemodeI is utilized.

Much time and effort in HES is devoted tothe reduction of nonsampling error and to themany problems involved in the evaluation ofresidual errors. All HES personnel involved withthe collection of data are also actively involvedin quality control efforts, and the initiation andexecution of quality control procedures are aresult of a concerted effort involving manymembers of the HES staff as well as numerousother consultants. Many general procedures pre-sented here are common to all cycles butmodifications of specific techniques are oftennecessary due to the population segment beingsampled during a particular cycle. More specificprocedures relating to particular data areas areincluded in reports of findings.

MEANS OF REDUCING NONRESPONSE

One type of nonsampling error that occursin perhaps all surveys, especially where subjectparticipation is voluntary, is that due to nonre-sponse. The amount of bias created by nonre-sponse generally, but not necessarily, varies withthe amount of nonresponse. The usual course ofaction to reduce nonresponse bias, therefore, isto obtain as great a response as possible. Fromthe outset HES has recognized the problem ofnonresponse and has experienced success in itscontrol and minimization. The procedures pres-ently used are a result of experience andknowledge obtained from pretests, previouscycles,7 methodological studies,8~9 and othersurveys.

Advance Publicity

If those persons chosen as a part of thesample fully understand the Health ExaminationSurvey and its purpose, they are much morelikely to give their complete cooperation. InHES, the informational effort is directed towardpersons who might be expected to fall in thesample and toward organizations and agencieswhose support for the survey is not onlydesirable in itself but might ako influence theresponse of sample persons.

In Cycle I, HES medical officers madepersonal visits to State and local health depart-ments, medical and dental organizations andsocieties, and in Cycle II, to State educationofficials as well. During these visits the medicalofficers explained the purposes of the survey,the specizdized nature of the examination, andthe confidentiality of the data collected. Theyalso pointed out that no results of the examina-tion would be disclosed to the examine, butthat he would be encouraged to sign anauthorization permitting a report of the findingsof his examination to be sent to his physician ordentist. In Cycle HI, which returned to the samesampling locations as Cycle II, an explanatoryletter was sent to these officials in lieu of apersonal visit.

Approximately 6 weeks before examinat-ions began in each area, a member of the HES

field staff made personal contact with health,school, and other concerned officials. In CyclesII and III the cooperation of school officials wasparticularly valuable, since youths participatingoften had to miss a half-day of school on theday of examination. Also schools were requestedto complete a questionnaire concerning socialand academic achievement of the youths.

Two or 3 weeks before examinations begana second letter was sent to these officials, as aremirider and to give the exact dates andlocation of the operations. At this time theCensus Bureau mailed to each household to becontacted a postcard stating that the CensusBureau would be visiting within a few days inconnection with a survey being conducted forthe U.S. Public Health Service. Shortly before’the commencement of interviewing by theBureau of the Census personnel, the locaLnewspapers were provided with a news release.Although this publicity reached a much largersegment of the population than HES attemptedto examine, a local newspaper article was often ameans of communicating with the desired seg-ment of the population. Newspaper articles werevaluable also to the Health Examination Repre-sentatives during the second household interview,providing them with a printed document from alocal source which could be used to impartinforrnation and establish the authenticityy of theHES program during a rather extensive explana-tion of the program.

A previous publication evaluating theeffect of the publicity on sample persons inCycle I gives examination rates of those reachedthrough various types of publicity. These ratesindicate that in general publicity had a positiveeffect on response. publicity was also associatedwith success in obtaining and scheduling anappointment at the time of initial interviewwith the sample person.

Interviewing Techniques

The personal interview, a technique used tocollect data, was also an effective means ofdisseminating information to sample personsabout the Health Examination Survey. The firstinterview conducted by Bureau of the Censusinterviewers, had as the primary purpose the

obtaining of household composition and demo-graphic data. During this preliminary interview,the Census interviewers also recorded any per-tinent information they felt would be useful tothe Health Examination Representatives(HER’s) who make followup interviews. InCycle I, the Census interviewer also told thesample person about the HES program andconcluded the interview by offering the oppor-tunity to have an examination. This offer wasmade only to the sample persons and not to aproxy respondent since a methodological studyghad shown that proxy respondents were lesswilling to commit others to an examination thanthemselves. If the offer was not accepted, allattempts to persuade the respondent were de-layed for the HER’s interview.

In CycIes H and 111, the Census interviewerwas not a real source of information concerningthe survey, deferring most respondent questiohsfor the HER’s followup. During this secondinterview, the HER’s presented a rather detailedexplanation of the HES program and extendedthe offer of an exarnination. Even though thecontent of the interview was carefully planned,the HER’s were allowed some latitude in thepresentation to aid in establishing proper rap-port. The HER’s were instructed that each casewas to be treated individually and that with fewexceptions, cooperation could be achieved ifthere was sufficient insight into the real mattersof influence. The general approach was that eachcase should be conscientiously pursued in aprofessional manner, both directly by the HER’sand indirectly through other potential influencesuntil an examination was achieved or until therewas, without a doubt, no chance of achieving anexamination. Exceptions to these instructionswere allowed in the case of persons who weremanifestly unable to come to the examinationcenter or where further pursuit would createproblems in public or professional relations.Within this ‘ approach, the HER’s were givenconsiderable latitude and independence in deal-ing with each case so long as the efforts andapproaches were straightforward and factual. Insome uncooperative cases, where a change ininterviewer personality might have been benefi-cial, a different interviewer was assigned. Inother cases, cooperation was sought through

another member of the household or some otherinfluential intermediary.

Various “selling points” might also befactors in the high response rate achieved byHES. In some cases the HER might emphasizethe fact that the sample person had the chanceto receive an expensive medical examinationabsolutely free.

Also, all sample persons were advised thatfindings are treated in a confidential manner.The Census postcard apprising the household ofthe survey carried a statement ofconfidentiality. During the initial interview,confidentiality was again assured. Also, theforms used for data collection carriedstatements of confidentiality.

Examination Policies

Several policies related to the examinationitself were important in the reduction of nonre-sponse. Every effort was made to arrange asuitable time for the examination, includingnight or weekend appointments for those per-sons who found the normal weekday appoint-ments inconvenient.

In addition, HES provided transporta-tion to and from the examination center. Thiswas not only a service for those who did nothave readily available transportation but it wasalso a valuable technique in obtaining examina-tions from persons who might have failed tokeep their appointments without some furtherstimulus. In Cycle II, the HER who had con-ducted the interview and obtained the consentfor examination was responsible for transportingthe children. In Cycle III, with an older agegroup and fewer HERs, local transportationcompanies provided this service, umder contractwith HES. An adult escort employed by HESaccompanied the youths. Where HES transporta-tion was refused, a mileage allowance was madeavailable to those who provided their owntransportation.

The examinations were designed to includea minimal number of tests that might bedistasteful and no tests which were potentiallyharmful to the exarninees. If, despite this generalpractice, a particular test was completely unac-ceptable to a specific examinee, it could be

6

omitted if this action assured his cooperation inthe remainder of the examination. Also ofimportant e was the effort made by the examin-ing staff to answer questions and explain pro-cedures and equipment to the examinees. Thesurvey was in an area long enough for thisatmosphere and news about the examination tobe relayed by examinees to their friends whomight also be part of the HES sample. There-fore, a sample person who initially refused theexamination might reconsider and consent afterlistening to a peer who has found the experienceinformative and enjoyable. Finally, there was animmediate followup and rescheduling of thosewho cancelled or failed to keep an appointment.In some cases those who failed to keep anappointment due to fear or various apprehen-sions could be contacted and repersuaded intime to be examined only a few minutes laterthan originally scheduled.

Evaluation of Nonresponse

For the purpose of discussion nonrespondentscan be divided into two groups: those personswho should have been, but were not, identifiedas sample persons and those identified as samplepersons but who were not examined. In HES thefirst group was composed almost entirely ofthose who were absent from the sample house-hold during the entire examination period andfor whom no information could be obtainedfrom family, neighbors, or other knowledgeablesources. The information defining whether aperson residing in one of these sample house-holds was indeed a sample person was thereforeunavailable. But if usual rates of occurrence ofsample persons per sample household wereapplied to these households, the resultant ex-pected loss of sample persons would be small. InCycle I this group comprised 1,6 percent of thetotal sample and in Cycles II and III, approxi-mately 0.10 percent. The greater percentage inCycle I is largely explained by the age segmentbeing sampled. Practically every household willcontain an adult but not necessarily a memberof the younger and more restricted age segmentssampled in Cycles 11and 111.In these two cycles,information “was obtained from neighborsothers familiar with the age composition

orof

these sample households, further reducing thisclassification of nonrespondents,

The second group of nonrespondents, thoseidentified but not examined, is larger andworthy of more dkcussion. This group wascomposed of those who died or moved awaybetween the first Census contact and the exami-nation, those manifestly unable to be examineddue to illness or severe physical disability, andthose who refused to participate. The lattermade up the great majority of the non responseexperienced by HES.

In Cycle I, of 7,710 sample persons 6,672were examined giving a response rate of 87percent. In Cycle H, of 7,417 sample children7,119 were examined, a response rate of 96percent. In Cycle III 90 percent of the 7,514sample youths were examined. Although theserates are good for a survey of this type, as longas there is nonresponse, the final estimates aresubject to a potential nonresponse bias. If thenonrespondents differ from the respondentswith respect to a particular characteristic, thenusual estimating procedures will produce abiased estimate for that characteristic. However,a small nonresponse rate is usually consideredacceptabl~ first, it is generally unavoidable andsecond, the characteristics of the nonrespond-ents would have to be substantially different to -produce even a small bias in the estimates of thecharacteristics being measured. In practice, thebiasing effect of nonresponse is never ,fullyknown although insight into potential effects isoften gained by the consideration of knowncharacteristics associated with the nonrespond-ents. This is especially true if there is anassociation between the known characteristics ofnonrespondents and the information of interestto the survey. Basic data on nonrespondents,such as age, race, sex, and urban-rural status, areoften known to survey personnel. Thus, inCycles II and III, for example, since height andweight (unknown for, nonrespondents) weregenerally associated with age (known) morecould usually be inferred concerning the biasingeffect of the nonrespondents on the estimate ofheight and weight than if no association werepresent. In HES rather detailed basic informa-tion was obtained on practically all samplepersons by means of questionnaires administered

7

‘before :,.ny mention of the examination. Com-parisons cm therefore bc made on a number ofVa.risk!..:: tie{: !::c~l tl~(~~e persons in the sample,,,)~<,., .( !:-:::~t”.(~ mid. weI-e examined and those-Ji(-,,2 .:.’.,~r;: ‘ -+ ‘This approach has been:1:]~~:.,’artll)ltJ..used in analyzing factors related to response inthe Cyclc I sa~mple.7

Another approach used to evaluate nonre-sponse is to investigate examination rates byvarious age, sex, and other demographic vari-ables. In cycle 1, the examination rate washighest for the youngest persons and diminishedwith increasing age. In the age group 18-24years, 90.2 percent of the sample was examined,whereas in the age group 75-79 years, theexamination rate was 74.3 percent. In Cycle II,the examination rates by single years of ageranged only fr9m 95.7 to 96.2 percent. By sex,the examination rate in Cycle I was 88.3 percentfor men and 85.0 percent for women. In CycleH, the rate vms 96.5 percent for males and 95.5percent fo, females. Looking at populationdensity, in Cycle I the examination rate of fivepopulation density classes ranged for 77.7 to92.0 percent, and in Cycle H eight classes rangedfrom 93.1. f:o 98.6 percent. In both cycles therewas a &.stinct relation between response andpopulation density, with rural areas having thehighest response and highly urban areas thelowest. TIlcse examples show that althoughoverall reqmnse rates may be good, individualcells in var~o}.~sgroupings may have a somewhatlow response. The fact that the nonresponse biasmay be la.rgm in these lower response cellsshould be kqt in mind when interpreting thedata. iVlore coi~~plete accounts considering theseand other variables have been published pre-viously.2J

Although the above methods of analysis do~ive insight into the problem, a quantitativeevaluation of nonresponse bias necessitates anestimate of the characteristic of interest for thenonrcsponclents. IrI practice, the value of thecbaracteri.stlc of interest for the nonrespondentsis never known, and therefore, the nonresponsebias cannot be accurately computed. But manyingenious tcc.h.niques have been devised to helpdeal with both general and specific aspects ofthe nonrespome problem. In surveys whereefforts to win cooperation are not particularly

intense, a technique sometimes employed is tosubsample the nonrespondents and through in-creased persuasion obtain enough response inthe subsample to make estimates for all nonre-spondents. With these estimates adjustment fornonresponse bias can be made. Several problemsare inherent in this approach, the most obviousbeing that the subsample of nonrespondentsmost probably will contain those who will notrespond even to the increased efforts, so that themethod used to adjust for the nonresponse biaswould itself be subject to a nonresponse bias. Insurveys such as HES, where intensive persuasionefforts are made on all subjects, those who arenot cooperative after these efforts are ~’hardcore” nonrespondents; and any attempt toachieve even limited cooperation, much less toobtain a satisfactory response from a subsample,would be futile.

In Cycle I information about nonre-sponclents obtained from existing records provedvaluable in evaluating the impact of nonre-sponse. During the household interview eachsamp~e person was asked to give the name andaddress of his personal physician and to indicatehow long it had been since he had last seen him.In each household the respondent was asked tosign a form authorizing his physician to releasemedical information to the National HealthSurvey. If a nonexamined person had signedsuch a medical release and given the name of apersonal physician whom he had seen in thepreceding 2 years, an inquiry ‘was sent to hisphysician. If the person had not/signed a release,the inquiry form would be sent to him~ with arequest that he forward it to his physician forcompletion. A similar inquiry form was sent toan examined person from the same place whowas of the same sex and, as nearly as possible,the same age. Although there were some prob-lems in obtaining usable medical information forthe nonexamined, the data collected were com-parable to that for examinees. The study con-cluded that it was improbable that the nonre-sponse introduced a serious bias in the findingsof the survey.3

Whether or not point estimates of charac-teristics of interest are made for nonrespond-ents, the evaluation of possible limits is oftenvaluable in subject areas where reasonable limits

8

canand

often be assumedexperience in the.

samrde mean is the

with the aid of knowledgesubject matter area. If theestimate of concern, a

qu~tity equal to that which would have keenobtained with 100 percent response is Pr Yr +Pn, Yn,, where P represents the proportion ofresponse (Pr) or nonresponse (P. ~) and ~ theestimate of the sample mean for respondents(~r,) and nonrespondents (~nr). With the abovenotation the nonresponse bias can be expressedas Pnr (Y, – ~nr,). When the response is ex-tremely high as in Cycle II and Pnr is on theorder of .04 or even if the category of interest isof relatively high nonresponse and Pn~ is on theorder of .06 (the highest nonresponse in allage-sex categories) the difference be~ween the—response and nonresponse values (Yr — Yn ~)would need to be quite sizable to produce morethan a negligible bias. However, in Cycle I in the75-79 years of age group, where the nonresponsewas .26, such a strong statement could not bemade. For estimation purposes the methodchosen to deal with that nonresponse which didoccur was imputing to nonrespondents thecharacteristics of “similar” respondents. Thiswas accomplished by multiplying, within classesof sample persons, the weights of respondentsby the reciprocal of the proportion responding.In all cycles classes were defined by age and sexwithin each geographic location. Since basicweights of sample persons within a locationwere generally the same this adjustment hadessentially the same effect on estimates of totalsand means as assuming that (~—~nr ) wasnegligible in each class, i.e., that thenonresponse bias was negligible in each class.

Control of Missing Data

In all operations where data are collected,missing data in the records of respondents are apotential problem. Missing data often createproblems during data processing, editing, andanalysis; but generally the responsible act occursduring the data collection process, and thereforemissing data are considered here as a type ofnonresponse rather than a problem related todata processing or analysis. As in any nonre-sponse problem, the most reliable means ofreducing the bias introduced by missing data is

the prevention of the nonresponse. There tiesome unavoidable losses of data: omission of atest when it might be detrimental to theexaminee, refusal of an examinee to participatein a particular test, or inability to obtain anacceptable performance of a procedure. Theexamining staff were expected to use discretionregarding these unavoidable losses. They alsowere responsible for preventing any avoidableloss of data. Much data in HES were recorded byhand on standardized forms, and illegible orinsufficient entries were essentially missing data.Emphasis was therefore put on standardized,accurate, complete, and legible entries in therecording of all data. Each examiner was respon-sible for reviewing each record as soon as he hadfinished making entries in it to be sure therewere no avoidable omissions or errors. If anomission was unavoidable, he entered an explan-atory note on the form. Additional personnelwere used as recorders in certain parts of theexamination to facilitate recording and to checkfor errors, omissions, and inconsistencies.

In addition immediate review of records inthe examination center while the examineeswere readily available was another valuabletechnique in the reduction of missing data. Afterthe test and the initial review, the examinerentered the time in and out, his initials andcomments on a control record and returne,d thefolder to the clerical assistant. The clericalassistant again reviewed all records and checkedto see that all tests had been performed andproperly recorded before the examinees left theexamination center. In some cases extra efforthad to be exerted to produce a record before theexaminees left the examination center. Forexample, all X-ray films were developed beforethe examinees left the center and inadequatefilms were replaced. To prevent loss of datafrom equipment-malfunction back-up equip-ment was kept in the examination center forcertain tests.

Review of questionnaires, especially thosewhich the respondents were responsible forcompleting, was productive in the discovery andreduction of missing data. As in the collection ofdata by examination, it was advantageous toreview questionnaires in the field where missingor inconsistent data could be readily corrected

9

by personal contact with the respondent or aresponsible member of the sample household.All HES questionnaires were reviewed in thefield, some quite extensively, some less so. Theonly exception was those Cycle II and IIIquestionnaires which were left with schoolsduring the summer and were not returned untilthe survey had left the area. The majority ofschool questionnaires, however, were returnedand reviewed while the survey was in the areaand missing information could readily be ob-tained. Edits were performed during processingon these as well as all HES data, but theadvantage of the field review in reducing missingdata was lost in this particular case. Breakage ofblood vials during shipment to the laboratoriesor as the result of accidents in the laboratoriesthemselves was another example of a source ofmissing data where the time of occurrenceprohibited remedial action by the field staff.

Preventive procedures greatly reduced theproblem of missing data in HES. The usualprocedure for treating that which did exist wasto allow the analyst who was familiar with thesubject area to determine an appropriate methodof imputation. On those parts of the examina-tion where the estimates to be made wereconsidered sufficiently critical, imputations ofmissing data were made. Where some but not allof the elements of a particular part of theexamination were completed for an individual,the missing elements were usually imputed bymatching those results which had been obtainedon this part of the examination with those forother examinees of the same age-sex-race groupwhose examinations were complete and thenrandomly selecting within. this frame the valuesfor missing elements. When an entire data areaof the examination was missing imputation wasmade by using values of an examinee withcomplete results from the same age-sex-racegroup. Whenever the extent and probable impactof missing data was serious enough to warrantconcern or when imputation techniques were ofspecial interest, the extent and methods oftreatment were included in reports of findings:

MEANS OF REDUCINGMEASUREMENT PROCESS ERROR

.4 second type of nonsampling error whichdeserves consideration equal to if not greaterthan that devoted to nonresponse is measure-ment process error. Measurement process error isused here to denote that error which occurs inthe determination and performance of the meas-urement procedures and includes errors involv-ing questions of validity and of reliability. It isnot used to encompass recording and transcrib-ing errors which are somewhat different and willbe considered in another section of this report.Hereafter measurement process error will bereferred to simply as measurement error.

It is simple to conceptualize a measurementcomposed of some “true” value plus somemeasurement error. But in the collection andanalysis of field data, the exact values of thesecomponents for a particular measurement can-not be obtained. Therefore the traditional esti-mates of central tendency and sampling errorwill be subject to some degree of bias dependingupon the nature of the measurement errors. Thelack of specific values of measurement errorsassociated with a particular measurement doesnot preclude reasonable estimates of the overallmeasurement error, although knowledge of thevalues could be useful. With these estimlates inmind, judgments can be made about the desira-bility of one process or procedure over anotherin the effort to reduce measurement errors. InHES the most direct attack on the problem ofmeasurement error was to adopt all feasibleprecautions and procedures to minimize meas-urement errors in the collection process. Theseefforts may be divided into three major areas:planning, standardization of the testing environ-ment, and standardization of the testing proc-esses.

Planning

The first and certainly one of the mostimportant steps in the conduct of a survey isdefining precisely the information to be col-

10

Iected and deciding the best way to collect thisinformation. Ideally, how the data are collectedshould be taken into account in deciding whatdata are to be collected. If methods for obtain-ing certain information are unreliable and it isnot feasible to develop a satisfactory method,then the resources and effort which would beused to collect the unreliable data might bebetter used to obtain other information whichmay be collected by more reliable methods.Advisors, both from within HES and from othersources, are actively involved in the determina-tion of what data are to be collected and themethods which should be used to collect thesedata.

On the basis of advisor recommendations,methodological studies, general knowledge ofthe problems involved, and expedience gained inprevious cycles, tentative plans for a cycle aredeveloped. The development of the final designfrom these tentative plans entails an intensiveseries of pretests, evaluations, and resultantmodifications. For example, in the preparationfor Cycle III the proposed design, which wassimilar to that used in Cycle II, was pretestedthree times. The first pretest was in Brooklyn,New York, on 93 youths representative of theage spread and roughly equal by sex. This verypreliminary pretest was conducted in conjunc-tion with the nineteenth location of Cycle II.The week prior to the pretest was spent inorient ation and on-the-site training for all mem-bers of the field staff. A flexible examinationschedule was maintained during the pretest toallow evaluation of various scheduling patterns,such as sex separations and varied numbers ofexaminees at one time. Exit interviews wereconducted with all examinees to obtain theirreactions and suggestions for improvement.

After much evaluation, additional planning,and modification involving facilities, procedures,equipment, sequencing, and allotment of timesfor various’ parts of the examination, a secondpretest was held in Detroit, Michigan. Like theNew York pretest it was performed in conjunc-tion with a Cycle 11 location. Again, the exami-nation schedule was purposely light to allowtime for training and continuing evaluation.After summary evaluations and a few newmodifications, the largest and final pretest was

conducted inpretest youths

Wihnington, Delaware. Of 188163 were examined. By this time,

most of the data collection procedures were inthe last stages of refinement and a number ofconsultants contracted by HES visited the exam-ination center to help with additional training ofexamining personnel and to plan final detailsconcerning procedures and equipment. The re-sults of this pretest were carefully consideredand, after a few minor modifications and revi-sions, preparations were made to begin examina-tion of the Cycle III sample. At the first samplelocation, the initial week of examinations wasdevoted to examining nonsample youths. These“dry runs,” as they are termed, provided a finaltraining period and on-the-spot testing of equip-ment and procedures.

In all HES cycles, after the test procedureshave been pretested and are ready for the startof a cycle, detailed written instructions arecompiled. Written instructions are essential tothe examining staff for training, reference, andreview. They are especially valuable in lengthysurveys to prevent minor changes and drifting inexaminer techniques and procedures. In addi-tion, written instructions serve as an essentialreference for those who analyze the data.

Standardization of Test Environment

Many measurements will vary depending onthe environmental influences present at the timeof the meastuement. Temperature, humidity,noise level, light intensity, and visual distractionsare a few of the many environmental factorswhich could directly influence the results ofvarious measurements taken in the Health Exam-ination Survey. The problem of standardizationof environment in HES is greatly reduced byperforming the examination in mobile trailerswhich are transport ed from location to location.

Upon arrival at a sample location thetrailers are parked parallel to one another andare connected by enclosed passageways to formthe examination center. Within the center thereis a sound-proof room in which the hearing testis conducted. The temperature is regulated andin the portion of the examination center wherethe exercise tolerance test is performed, thehumidity is also kept within certain limits. Light

11

intensity for visual testing is controlled. Inaddition, the arrangement of rooms within thecenter is carefully planned to take account ofthe possible effect of a particular area on anadjacent area, another fact or in the standard-ization of the test environment.

Little environmental control is possible inthe data gathered by home administered ques-tionnaires other than that resulting from theiradministration in the home and at a time whenthe respondent is not distracted by work,visitors, or unusual situations. But this is of lessconcern, since most HES data are gathered inthe controlled environment of the mobile exami-nation center, and the type of data gathered byquestionnaire are subject to little environmentalinfluence compared with those gathered in theexamination center.

Standardization of Testing Processes

Standardization of testing (measurement)processes is, for the most part, achieved by theuse of mechanical devices, appropriate opera-tional procedures, care in selection and trainingof examiners, and procedures designed to reducethe impact of subject errors. The magnitude ofsystematic error or bias associated with a testingprocess is generally difficult to ascertainalthough in some cases the direction of the biasis known and in certain instances the magnitudecan be estimated. Aside from the bias associatedwith testing processes, the variable errors arealso of concern. These variable errors are gen-erally more easily quantified in the resultantdata than biases for the primary reason that it isdifficult to discover biases. In the examinationsgiven by HES, the process variation associatedwith the test procedures was usually potentiallygreater and therefore of more concern than thatassociated with environmental factors. Many ofthe procedures employed to identify and reduceprocess variation were also important in theidentification and reduction of certain biases.

Use of mechanical devices

A well constructed measuring device is lesssubject to large variable and systematic errorsthan a human examiner; therefore, one method

of reducing errors in a testing process k to usemechanical devices. Devices which produce“hard documents,” in such forms as printouts,tracings, photographs, and magnetic tapes, areparticularly valuable since additional procedureswhich normally would allow human errors areperformed mechanically.

Although the use of mechanical deviceshelps reduce errors in a measurement process, itmust be recognized that these devices themselvesare subject to variation and must be calibratedregularly. Calibrations may be performed onlyonce at each location or as frequently as beforeand after each subject. The HES Field StaffProcedure Manual contains instructions for cali-brations to be performed by the examining staffas well as testing procedures and other pertinentinformation. In some cases the resources forcalibration of the more sophisticated instru-ments were not available at the examinationcenter, and either a technician experienced withthe instrument had to come to the examinationcenter or the machine had to be sent away forcalibration. Various arrangements and systemsof back-up equipment were used when instru-ments were away from the examination center.

Since mechanical devices are under directexaminer control or supervision, human errormay readily enter into the measurement processdepending upon the degree of examiner involve-ment. The degree of involvement varies fromsituations in which there is active involvement,as when a physician or nurse takes a bloodpressure, to those in which the examiner israther passively involved, as in the turning of aswitch on an instrument. In HES the attempt toreduce examiner errors was approached in sev-eral ways. Whenever possible the measurementprocedures (aside from the use of mechanicaldevices)” were highly standardized and theamount of subjective judgment necessary wassmall. In measurements where the degree ofhuman involvement and judgment was large, asystem of checks, when practicable, was in-cluded in the measurement procedure by using asecond person as a recorder-observer. For ex-ample, the recorder for the dental examinationserved a necessary function by relieving thedentist of recording findings throughout theexamination, but in addition, she was respon-

12

sible for double-checking the forms for incom-plete and inconsistent findings. In taking bodymeasurements the more active participation ofthe recorder-observer demanded that he havetraining in body measurement procedures equalto that of the examiner, and the techniciansresponsible for taking body measurements inter-change in the roles of examiner and recorder-observer. Repetition of measurements wasanother method used to reduce the impact oftesting process errors, including examiner errors.For instance, in obtaining blood hematocrit, atechnician routinely performed two deter-minations on each subject, and in addition asecond technician repeated readings on a samplebasis.

retraining in an entire area. An invaluable policyin the standardization of procedures and in thetraining and retraining of personnel is the use oftime before and during the survey for theexamination of nonsarnple persons. At the be-ginning of each location, the day prior to thestart of regular examinations was set aside forthe examination of a few volunteer nonsamplepersons. The light schedule allowed ample timefor reviewing procedures, clarifying any ques-tions regarding drift in technique, and assuringthat all equipment was properly prepared for theexamination of sample persons on the following “day.

Reduction of subject errors

Selection and training of examiners

Those examiners who were conscientiousand had personalities suitable for the type ofwork were less likely to introduce as great adegree of measurement error as those who were

.not so dispositioned. Obviously, proper basictraining was another prerequisite to becoming amember of the examining staff, but aside fromprevious experience and knowledge an initialtraining period and frequent retraining in thespecific techniques of HES were necessary foreach examiner to become properly skilled. Thepermanent HES advisory staff supervised thistraining. When necessary, cons~tants augmented

thk training both in the field and in their ownfacilities.

The length of time necessary to completeextensive surveys such as those conducted byHES creates problems of drift in techniques thatare not as likely to occur when data are gatheredin a shorter period of time. The practice ofproviding all members of the examining staffwith dettiled written instructions covering testprocedures helps to achieve consistency in meas-urement techniques throughout a cycle. Theforms that the field staff used were quitestructured, and most data were recorded asnumbers. Retraining is also important for theachievement of consistency in measurementtechniques in HES. Time spent retraining mightrange from a few minutes for reviewing a singleitem with a fellow examiner to several days’

Subject error is one of the most difficulttypes of measurement error to evaluate orcontrol. Subject cooperation and concentrationare extremely important when a mental responseis being elicited. In HES this type of cooperationwas perhaps most important in the psychologicaltesting. Subject cooperation and concentrationwere also often important when active physicalinvolvement was required as in the spirometrywhere data on maximum expiatory flow rateswere gathered, If a maximum response was to bemeasured, the subject had to give a maximumeffort. To this end, the technicians routinelygave vocal encouragement before and during thesubject’s exhalation.

Procedures designed to reduce subject errormay be quite similar to or even coincide withthose designed to reduce examiner error. Subjecterrors as well as examiner errors deserve correc-tive procedures in the taking of body measure-ments. That the subject is not standing or sittingin the proper position will tend to introduceerrors in many body measurements. This is moreof a problem than immediately apparent, sincethe technician in making some of the measure-ments, is not able to observe whether the subjecthas deviated from the correct position. In HESthis particular source of subject emor wascontrolled by using a second technician as arecorder who was also responsible for seeing thatthe subject was in the proper position for allmeasurements.

13

Use of repeated measurements is effectivein reducing subject error as well as other testingprocess- errors. An example is found in thedetermination of threshold hearing levels, wherethe threshold recorded was the lowest decibelreading at which responses were obtained in atleast 50 percent of the trials using a minimum ofthree trials.

PROCESSING ERRORS

In HES the specific methods by which datawere recorded vary: some data were recordeddirectly on Wgnetic tape for immediate com-puter use; other data were recorded in the formof hard documents such as X-rays, height phot-ographs, or weight printouts; but the majority ofHES data thus far have been recorded bychecking boxes and making written entries inwords or numbers in appropriate spaces ofstandard forms. Of these three methods the firstis obviously superior for purposes of reducinghuman errors in recording, coding, and punch-ing. But there were many areas of data collec-tion in HES where present technology and highcosts prohibited the use of this method. In theongoing work in HES, efforts are being made toreduce the proportion of data recorded on formsand increase the proportion recorded on mag--netic tape or as hard documents.

The processing oi’ data is an activity inwhich errors which have occurred previously buthave not been discovered and corrected can bedetected. But it is also an activity in which newerrors can easily occur. The objective of the HESdata processing program is to detect as manyprevious errors as reasonably possible whileproviding for tight control on process errors.Although extensive clerical edits to detect missi-ng and in some cases inconsistent data areperformed in the field, the actual processing ofdata can be considered to start after the dataarrives at headquarters from the field at the endof each location. At this point practically allexamination and questionnaire data are sub-jected to clerical edits of some nature. Heightsand weights recorded on the forms are allverified by comparison with the original photo-graphs and printed records. Approximately a

5-percent sample of all examination forms arereviewed on a routine basis as a part of theposting and evaluation of replicate data. TheCensus household questionnaires are reviewedagain for erroneous exclusion or inclusion ofthose who should or should not be samplesubjects as well as for other errors. Edits areperformed on samples of other questionnaires.Except for a few special cases all forms andquestionnaires are microfilmed before being sentfor coding and punching; if records are later lostin shipment or processing, the data are pre-served. Microfilm also provides a convenient,compact record of individual forms for imme-diate reference. After microfilming, the recordsare sent directly for coding, punching, andediting, according to detailed specifications andinstructions prepared in HES. In the design offorms consideration was given to the reductionof clerical coding so that extensive coding is notrequired. To reduce coding and punching errorsall coding and punching work is verified. Editsare then performed and printouts returned toHES where discrepancies and disallowed valuesare checked against the microfilmed records andrectified. The data are then forwarded to theDivision of Data Processing, NCHS, for transferto tape and more extensive edits and consistencychecks. Except for the problem of imputationfor missing values referred to earlier, the data arethen considered ready for analysis.

SURVEILLANCE AND EVALUATIONOF RESIDUAL MEASUREMENT

PROCESS ERROR

Monitoring Systems

Despite efforts to reduce measurementprocess errors, residual errors of a magnitudelarge enough to warrant concern occur withsome regularity. There is, therefore, a real andurgent need to have a system whereby theseresidual errors can be monitored. As statedpreviously, the concept of quality control isbased on the desire to obtain end products of acertain quality. Therefore, one of the mainpurposes of a monitoring system would be to

14

indicate whether or not the measurements pro-duced by a certain measurement process at-tained the desired quality. A second majorpurpose would be to make possible quantitativesummary descriptions of residual measurementerrors to aid in the interpretation of survey data.

In the Health Examination Survey severaltypes of monitoring systems were used. One ofthe more systematic of these was the review offorms and questionnaires. Records review is acommon practice in efforts to monitor andcontrol errors although the effectiveness of thismethod is generally limited to certain types oferrors, specifically, missing, inconsistent, andimpossible values. In addition, this method canbe used to detect errors in interpretation ofrecords, such as X-rays, electrocardiograms, andspirograms where readings of hard documentscan be checked for correctness of interpretationagainst the document itself. Various errors can,of course, have an influence on the process ofobtaining the hard document; and, therefore,procedures designed to control errors are asimportant in the creation of a hard document asin any other area of collection. In HES, aspreviously described, extensive review is per-formed in the examination center before theexaminees depart and during data processing.

Perhaps the most direct monitoring systemused in the Health Examination Survey was theobservation of the measurement process as itwas being applied to an examinee. Medical,dental, and psychological advisors from HES andother advisors and consultants regularly visitedthe examination center to observe examinationprocedures and retrain examiners if necessary. Agood example of how routine observation wasused as a monitoring system can be found in thetaking of body measurements. The one exam-iner, in addition to acting as a recorder andaiding in the positioning of the examinee, wasalso responsible for observing and correcting anyerrors in measmement technique.

The most extensive system of monitoringused in the Health Examination Survey in Cycle111was the collection and evaluation of replicatedata. Replicate measurements are useful for avariety of reasons which include use as a meansof increasing precision of estimates of indkidualmeasurements, as a training technique, and as a

monitoring system which includes the objectiveof final evaluation of measurement errors. Theseobjectives are not incompatible, and replicatedata collected primarily for one of these objec-tives often indirectly, if not directly, accomplishone or both of the remaining two. For thisreason replicate data are most often collectedwith a combination of these objectives in mind.The single most important source of replicatedata in Cycle III was the replicate examinationswhere approximately 5 percent of the reguku-exarninees were returned to the examinationcenter for a second complete examination ex-cept for drawing blood and taking X-rays. Othersources of ‘replicate data are discussed later inthis report.

Biases and Controls in Replicate Measurements

A major source of uncertainty in estimatesderived from replicate measurements is failure tomake the replicate measurement under the sameconditions and in the same manner as theoriginal measurement. This uncertainty is dif-ficult to evaluate and most evaluations arerestricted to subjective statements concerningthe direction and/or size of the bias and theneed for concern in the analysis of data. Severalpolicies regarding Cycle III replicate examina-tions were valuable in the attempt to obtainreplicate measurements taken under the sameconditions and in the same manner.

Replicate examinations were not con-ducted during a specific time set aside exclu-sively for them, but whenever possible wereinterspersed among the regular examinations. Anoriginal examination was given priority over areplicate exami nat ion in that no replicate exami-nation was scheduled if it occupied time neededfor a regular examination. In practice there wasoften space to interject replicate examinations inthe schedule without interfering with reguku-examinations. However, this priority plus thefact that replicates were drawn from thoseexamined had the effect of increasing thelikelihood that a replicate examination would hescheduled toward the end of the examinationperiod. Nevertheless, the attempt to space thereplicate examinations in the schedule was avaluable policy in that the interspacing of

15

replicate and original examinations created anatmosphere more conducive to the replicateexamination’s being conducted in essentially the~ame manner as the originaL

The examiners had been informed of thepurpose and importance of the replicate exami-nation program. It was emphasized that theyshould not vary their procedures on a replicateexamination or in any way try to collect“better” data than they normally would. There-after, the conduct of a replicate examinationwas not given any greater emphasis than anyother instruction since overemphasizing“sameness” might have created more bias than itwould have eliminated.

During the original examination neither theexaminer nor the examinee knew whether or notthe exarninee would be returned for a replicateexamination. During the replicate examinationexaminers were not specifically informed that anexarninee was a replicate although no attemptwas made to conceal this fact since in anexamination as lengthy as that given in HES theexarninee would undoubtedly be rememberedby several if not all examiners. Even though anexaminee might be remembered it was ex-tremely unlikely that an examiner would re-member a specific measurement after a timelapse of 2 or 3 weeks. Some bias might beintroduced by the examiner’s knowledge of thereplicate status of an examinee, but it wouldseem that generally this bias might be quitesmall when compared to the measurement errorand in some cases to the biases associated withthe knowledge and familiarity gained by theexaminee during the original examination. Ex-aminee bias can be important especially inmeasurements where a response is elicited orwhen due to the time lapse, the true value of themeasurement has changed. The effect of learningis certainly a confounding factor in areas such aspsychological testing and to a lesser extent inmeasurements such as determination of hearinglevels, where familiarity with the testing devices,procedures, and personnel may well influencethe results. Since the time lapse was usually 2 or3 weeks, some appreciable changes might “occurin certain measurements such as weight.However, for most of the data collected theactual change can only be very small and thiseffect may usually be neglected.

In Cycle III replicate data were c)btained onapproximately 70 percent of those selected forreplicate examinations. One explanation for thislow rate is that the persuasion and follow-upefforts were not as intensive as for regularexarninees. This is a partial result of givingpriority to regular examinees if interviewer orexamination time was limited. There also seemsto be an increased objection to returning for asecond examination, as demonstrated in themost frequent reasons for refusal: “One time isenough” and “I can’t miss school again. ”

Selection of Replicate Examinees

The selection of Cycle III examinees forreplicate examinations was random within cer-tain restrictions imposed by practical considera-tions. One of the restrictions was that replicateswere selected only from those examined duringthe first week and a half of the approximately3?4 weeks of examinations. This time period waschosen to facilitate the interspersing of replicateexaminations with originals in the examiningschedule without interfering tith the time al-lotted for original examinations and withoutscheduling additional time to accommodate rep-licates. In a voluntary survey it is obviouslyimpractical to follow a scientific random processin scheduling subjects, so those scheduled duringthe first week and a half are not, in the strictsense, a random sample of all those scheduled.But evidence that replicates might be considered“represent ative” is found in the fact that youthsof certain ages, locations, incomes, etc., are notroutinely more likely to be scheduled during anyparticular segment of the examination schedule.However the availability and desires of thesubjects do influence the composition of thereplicate sample. For instance, an exarnineewhose participation in an original examinationwas achieved only after repeated contacts bysurvey personnel might be excluded from areplicate examination since it is unlikely that hewould have received an original examinationduring the first week and a half. The schedule oflocations considering time of year, sequencing ofexaminations, relation to other ev(ents whichmight make subjects more or less available, andother related aspects give no obvious discrimi;natory factor. After examining these and other

16

relatively minor considerations the;e appears tobe no reason to believe that the subjectsscheduled and examined during the first part ofa stand differ from those scheduled and ex-amined during the latter portion of a stand withrespect to the data gathered.

Another restriction on complete ran-domness in the selection of examinees forreplicate examinations was the exclusion ofthose examinees who were “geo~aphically in-convenient” to the examination center. “Geo-graphically inconvenient” was arbitrarily definedas a distance of 30 miles or greater; although ifconditions dictated, exceptions were sometimesallowed. A primary consideration in choosing asite for the examination center was the cen-trality of the location in relation to the samplesegments (a segment is a cluster of households).Since segments were drawn with probabilityproportional to population, most segments werein relatively populated areas; and so the examin-ation center was also in or adjacent to arelatively populated area. Therefore, the subjectsdeleted by this 30-mile restriction usually re-sided in relatively less populated areas; so thisrestriction may create a bias in the replicate dataif, in fact, characteristics and errors of concerndiffered by population density. Even if differ-ences did exist, the total effect of this restraintwas not great since it excluded only approxi-mate y 10 percent of the eligible examinees.There were other minor restrictions of medicaland operational nature imposed on the completerandomness of the replicate sample, but theywere not readily associated with large differ-ences. Also they deleted at most only 1-2percen~ of the eligible examinees and for thesereasons are of small consequence.

Since the purpose of replicate examinationsis to give information about errors, the matter ofconcern between those excluded and thoseeligible for selection is not the possible differ-ences in the values of measurements but thepossible differences in the errors associated withthe measurements as shown by the discrepancybetween two measurements on the same subject.For example, measurements may vary markedlyby some demographic classification, but this isnot so relevant as the question of whether or notthe errors vary by this classification. It shouldalso be noted that although subjects did influ-

ence measurement errors, the environment, pro-cedures, and examiners were also highly influ-ential. The consideration of these additionalinfluences causes a completely random selectionof subjects to be of somewhat less concern.

Additional Replicate Data

Although the full scale replicate examina-tions were the single most important source ofreplicate data in Cycle HI, other replicatemeasurements were performed for the purposesof monitoring, training, and obtaining moreprecise estimates of individual characteristics. Asmentioned previously, blood was not drawn andX-rays were not taken during the full scalereplicate examination. However, replicate datawere obtained for all blood work and X-rays.Each day for 15 consecutive days at eachlocation an additional tube of blood was drawnfrom two subjects. The pairs of tubes, whichwere sent to a Iaborat ory in the same shipmentbut under different identifying numbers, weresystematically allotted to the laboratories per-forming Cycle III work for HES. In addition, forcertain tests the laboratories routinely per-formed their own replicate determinations onevery blood sample. Whenever differences largerthan predetermined tolerances occurred theanalysis was repeated Replicate X-ray interpre-tations were performed by evaluators on asample basis.

The body measurements were performedby four examiners and therefore, this replicatedata was then divided into two types, inter- andintraexaminer. Interexaminer replicates werecollected at the beginning of each locationwhere the four technicians each performed a setof measurements on one of the two dry runexaminees selected for replicates. The resultswere compared with predetermined tolerancesand if indicated, meastwements were repeated.However, the original measurements were pre-served for anal yt ical purposes. Technician pair-ing was rotated on a systematic basis at eachlocation. In these replicate data, the replicationwas performed the same morning or afternoonas the original, whereas with full scale replicatesthe period between examinations was usually aweek to 10 days. During the course of each

17

location, every technician repeated his ownmeasurements on one regular examinee. Theduplicate measurements were not performedimmediately after the original measurements,but usually at the end of the half-day period.Again results were compared to a set of toler-ances and differences were resolved by thesupervisory technician. Replicate body measure-ment data were also gathered during selectedretraining periods.

In the dental area, the general plan con-cerning the collection of replicate data was thattwo dental advisors alternated periodic replicatedata gathering sessions with the examiningdentists. In the course of a typical session,replicate measurements were gathered on ap-proximately 35 sample youths during a periodof 4 or 5 days. The duplicate examinationimmediately followed the original and discrep-ancies were discussed and resolved while thesubject was in the chair. In addition to thereplicates between advisors and examiningdentists, the two advisors also performed peri-odic replicates on one another under fieldconditions.

In the medical area, there also was a certainamount of replicate data collection, primarilyfor training purposes. During the first severaldays of each location, the dry run, and severalexamination days, both the examining physicianand medical advisor evaluated each examinee,comparing and discussing findings of the examin-ation.

Evaluation of Residual Measurement Error

Measurement error, as stated previously,includes those nonsampling errors associatedwith the determination and performance of themeasurement procedures. Since a reported valuewhose measurement error is entirely unknown isof questionable value, an evaluation of measure-ment error in the reports of findings of samplesurveys such as HES is highly desirable.

Many problems still exist in evaluation ofmeasurement errors in sample surveys. Researchis underway, but at present both rigorous theoryand detailed operational protocol are in shortsupply. In HES specific methods are in experi-mental stages. This section contains a very brief

introduction to the effects of measurementerrors and problems of evaluation, and to someof the methods by which HES evaluates residualmeasurement error.

Evaluation is facilitated by the fact thatmeasurement errors may be classified as variableor systematic (bias). The following simple modelincorporates this classification. It separates thek~~ obtained measurement on the j~’t individual(x~~) into three components: 1) the “true”measurement (Xj)> 2) a systematic measurementbias (b), and 3) a variable measurement error(ej~); that is X\k = xj + b + ejk. The “true”measurement (wluch is unobservable) would bethe value obtained by the application of theperfect measurement process. This processwould measure precisely the characteristic in-tended and contain no biases, conceptual orother. In addition, repeated measurements on asingle subject would not vary unless the charac-teristic being measured actually did changeduring the period of data collection. The biascomponent represents those errors which aresystematic and remain constant throughout theentire collection process. The variable measure-ment error component represents those errorswhich may vary from measurement to measure-ment and from day to day and are influenced byuncontrollable factors. It also includes errorswhich can be controlled but whose magnitudedoes not justify the effort or expense. If, as isoften possible, these variable errors can beconsidered random with one composite’ variableerror associated with each obtained value, thetask of evaluating the impact of bias and variableerrors is made considerably easier.

Assuming the above model, statements canbe made about the effect of measurement errorson estimates computed from sample data, specif-ically estimates of standard error and of centraltendency such as the mean, median, and mode.Biases as represented in this model alter tradi-tional estimates of central tendency by aquantity equal to the net bias. Because of thisinfluence, the standard error which is notaffected by a bias, alone is an unsatisfactorymeasure of the accuracy of estimates such assample means when the bias is large compared tothe standard error. In such cases, a measure ofaccuracy which contains a bias component is the

18

mean square error which is simply the sum ofthe squared bias and the variance of the esti-mate. But estimation of this measure necessi-tates a point estimate of the bias and since thebias component is squared does not indicate thedirection of the bias. In practice, point estimatesof bias can rarely be made with reasonablecertainty and the direction of the bias isimportant in the interpretation of results. There-fore the general approach in HES is to presentestimates of standard error and to discusssuspected biases giving, where possible, an evalu-ation of the magnitude and direction.

Because of the very large sample, the effectof variable errors on HES estimates of centraltendency is of less concern than it might be inother surveys. Variable errors can convenientlyand appropriately be assumed to have mean zeroin many measurement processes. It is likely,however, that the mean variable error obtainedin samples will not be zero, in which case anestimate of the population mean would be inerror. With a large sample, the mean variableerror is less likely to deviate from zero by morethan a small quantity, To evaluate this deviation,HES will use replicate data to obtain estimatesof the error variance.

Variable measurement errors can also influ-ence estimates of standard error. To estimatestandard error, HES uses a technique calledbalanced half-sample pseudoreplication. 1°?llThehalf-sample estimate of standard error is calcu-lated in a manner similar to that of thetraditional estimate. The half-sample method,however, requires deviations of a sampIe (se-lected acco~ding to a particular scheme) ofhalf-sample statistics from the total samplestatistic rather than deviations of individualvalues from the total sample statistic. Eachhalf-sample statistic is obtained from the indi-vidual values in one of two primary samplingunits. This estimate may be written as:

rL 1%

where ~ is the total sample estimate ofparameter F

~i is an estimate of ~ utilizing data from 1of 2 Psu’s

L is the number of half-sample replicates.

The most common approach in problemsof estimation is to assume that the obtainedvalues are nearly correct and to use them inestimating procedures as if they were “true”values. Also only one measurement is usuallymade on each uniG i.e., k =1, so that forsimplicity the k subscript will be dropped. Inthis case, using the notation of the above model,the half-sample replication estimate of standarderror used in reports of findings would be

But if it is allowed that obtained measurementsare not correct but in fact contain possiblebiases and variable errors, then substituting forx’ the following expression is obtained:

L L

It can be seen that, as in usual estimates of

standard error, a bias which operates as the biascomponent in the above model wiIl have noinfluence on half-sample estimates of standarderror. In practice, biases are more complex anddifferent biases may be operating in differentsituations and time periods, in which case theywould affect estimates of standard error. Con-sidering the relative magnitude of biases of thistype which are known to exist in HES it seemsthat generally biases should have only a smalleffect on HES estimates of standard error.

Variable errors can effect estimates ofstandard error, especially when data are pre-

19

sented, as HES’ usually are, by classificationssuch as age, sex, and race. If the estimates ofstandard error within each classification are notto be affected by variable errors, then thehalf-sample as well as the total sample must havemean error zero within each classification. Dueto the relatively small numbers in some cate-gories this is unlikely in HES data and the errorvariance will generally contribute to the estimateof standard error. If no assumptions are madeabout the sample mean values of variable errors,any change in the half-sample estimate ofstandard error due to variable errors depends onthe error variance term and the covariance termin formula (1) above. As is seen in this expres-sion if there is zero or a positive covariancebetween mean half-sample errors and meanhalf-sample “true” values, then the estimate willbe increased. If there is a negative covariance,the estimate of standard error is increased if thevariance of mean half-sample errors is greaterthan twice the covariance, unchanged if the twoquantities are equal, and decreased if this var-iance is less than twice the covariance. In generalvariable errors might be expected to slightlyincrease half-sample estimates of standard errorsin HES, due partially to the error variance andperhaps partially to the covariance. It wouldseem that the extent of the inflation due to theerror variance is small since in most of HES datathe variable errors are believed to be small inrelation to the magnitude of the “true” measure-ments. Also preliminary indications from, rep-licate data imply that at least in those measure-ments investigated this component is not ofgreat concern. More definite general statementsabout the effect of measurement errors aredifficult to make since in the wide range of datacollect ed by HJ3S the size, variabilityy, and natureof errors are somewhat diversified. In reports offindings, attempts are made to apprise the” readerof the impact of errors on the findings.

Many ingenious methods of evaluatingbiases have been developed by researchers tomeet the particular demands of the situations athand, Some of the more traditional methodsused in HES will be briefly mentioned here.

Systematic errors can produce considerablebias in estimates of certain parameters andunfortunately are in general difficult to detect.

When one and only one examiner is involved hthe measurement process, the question of sys-tematic errors can perhaps be addressed only bycomparing the results with similar stuldies. Inmany cases this is not an entirely satisfactorysolution since often it is difficult to find @diesthat are comparable in both the populationsmeasured and in the techniques and environ-ments used. Replicate data, aside horn providinga means of evaluating variable errors, can also beused to evaluate certain types of systematicerrors. For example, if in addition to the singleregular examiner an “expert” takes mea-

surements on a sub sample using the samemeasurement process, then the systematic errorassociated with the examiner as compared to the“expert” can be evaluated. In HES a scheme ofthis type was most extensively used in thle dentalarea. If two regular examiners apply a measure-ment process with different results, comparison,although still useful, is less revealing since itis usually difficult to determine which exam-iner was in error without referring back to an“expert.” If a larger number of examiners isused, comparisons bet ween examiners becomemore meaningful since, in general, with approxi-mate y equal training and expertise, an indi-vidual examiner who disagrees with th~e groupis in error.

In Cycles I and 11 the practice of comparingdata collected by different examiners was ex-tended to the comparing of data collected bytwo examination teams in two different exami-nation centers. The use of two teams and centersgreatly reduced the time needed to collect thesurvey data and also allowed comparison ofresults which could give indications of thepresence or absence of biases due to an examin-ing tearn or facility. In addition, at a fewlocations in Cycle II full scale replicate datawere collected, enabling comparisons of datagathered on the same subjects by the two teamsin the two different examination centers.

As mentioned earlier, there are many ques-tions to be answered in the evaluation ofresidual measurement errors and it is hoped thatmore substantial material can be presented at alater date. Also more specific information can befound in reports of findings.

20

REFERENCES

lNational Center for Health Statistics: Origin, programand operations of the U.S. National Health Survey. Vital andHealth Statistics. PHS Pub. No. 1000-Series l-No. 1. PublicHealth Service. Washington. U.S. Government Printing Office,Aug. 1963.

‘National Cent er for Health Statistics: Plan and initialprogram of the Health Examination Survey. Vital and HealthStatistics. PHS Pub. No. 1000-Series l-No. 4. Public HealthService. Washington. U.S. Government Printing Office, July1965.

3National Center for Health Statistics: Cycle I of theHealth Examination Survey, sample and response. Vital andHealth Statistics, PHS Pub. No. 1000-Series 1 l-No. 1. PublicHealth Service. Washington. U.S. Government Printing Office,Apr. 1964.

4National Center for Health Statistics: Plan, operation,and response restits of a program of children’s examinations.Vital and Health Statistics. PHS Pub. No. 1000-Series l-No. 5.Public Health Service. Washington. U.S. Government PrintingOffice, Oct. 1967.

5National Center for Health Statistics: Plan and operationof a Health Examination Survey of U.S. youths 12-17 years ofage. Vital and Health Statistics. PHS Pub. No. 1000-Seriesl-No. 8. Public Health Sen’ice. Washington. U.S. GovernmentPrinting Office, Sept. 1969.

6National Center for Health Statistics: Sample design

and estimation procedures for a National Health Examina-

tion Survey of Children. Vital and Health Statistics. Series2-No. 43. DHEW pub. No. (HSM) 72-1005. Washington.U.S. Government Printing Office, Aug. 1971.

21

“National Center for Health Statistics: Factors reIated toresponse in a Health Examination Survey. Vital and HealthStatistics. PHS Pub. No. 1000-Series 2-No. 36. Public HealthService. Washington. U.S. Government Printing Office, Mar..1968.

8National center for Health Statistics: Cooperaticm inHealth Examination Surveys. Vital and Health Statistics. PHSPub. No. 1000-Series 2-No. 9. Public Health Service. Washing-ton. U.S. Government Printing Office, July 1965.

9U.S. National Health Survey: Attitudes toward coopera-tion in a Health Examination Survey. Health Statistics. PHSPub. No. 584.-D6. Public Health Service. Washington. U.S.Government Printing Office, July 1961. (Out of print)

1 ‘National Center for Health Statistics: Replication, anapproach to the analysis of data from complex surveys. Vitaland Health Statistics. PHS Pub. No. 1000-Senes 2-No. 14.Public Health Service. Washington. U.S. Government PrintingOffice, Apr. 1966.

11Simmons, W. R. and Baird, J. T., Jr.: Pseudo-Replica-tion in the NCHS Health Examination Survey. Paper presentedat the Social Statistics Section proceedings of the AmericanStatistical Association, 1968.

000

22

* U. S.GOVERNMENT PRINTINGOFFICE :1973,543-S79/21

———

VITAL AND HEALTH STATISTICS PUBLICATION SERIES

Series 1. Programs and collection p~oceduves. —Reports which describe tile general programs of the NationalCenter for Health Statistics and its offices and divi~ions, data m)llec[ion methods used, definitions,and other material necessary for understanding the data.

Sevies 2. Data evaluation and methods vesearch. — Studies of new statisti LA methodology including: experi-mental tests of new survey methods, studies of vital statistics collection methods, new analyticaltechniques, objective evaluations of reliability of collected data, contributions to statistical theory.

Sevies 3. Analytical studies .—Reports presenting analytical or interpretive studies based on vital and healthstatistics, carrying the analysis further than the expository types of reports in the oth~i- series.

SeYies 4. Documents and committee repoyts. — Final reports of major corn mittees concerned with vital and

health statistics, and documents such as recommended model vital registration laws and revisedbirth and death certificates.

Series 10. Data from the Health Inte?@ew Suvvet’. -Statistics on illness, accidental injwries, disability, use

of hospital, medical, dental, and other services, and other health-related topics, based on data

collected in a continuing national household interview survey.

Se~ies 11. Data from the Health Examination .%?’vey. -ilata from direct cxarnination, testing, and measurem-

ent of national samples of the civilian, noninstitutional population provide the basis for LWOtypes

of reports: (1) estimates of the medically defined prevalence of specific diseases in the UnitedStates and the distributions of the population with respect to physical, physiological, and psycho-

logical characteristics; and (2) analysis of relationships among the various measurements without

reference to an explicit finite universe of p(’rsons.

Series 12. Data from the Institutioml Population Surveys —Stxistics reiating to the health characteristics ofpersons in institutions, and their medical, nursing, and personal care received, based on nationalsamples of establishments providing these services and samples of the residents or patients.

Series 13. Data fvom the Hospital Discluwge Swvey. —Statistics relating to di.: IL.r~ed patients in short-stayhospitals, based on a sample of patient records in a national sample of hospitals.

Series 14. Data on health ~esources: manpozuev and facilities .—Statistics on the numbers, geogr ~phic distri -

bution, and characteristics of health resources including physicians, denti scs, nurse~, other healthoccupations, hospitals, nursing homes, and outpatient facilities.

.%?ries 20. Data on mortal ity. -Various statistics on mortality other than as included in reguk on:-wal ormonthly reports —special analyses by cause of death, age, and other demographic i~mi~bles, also

geographic and time series analyses.

Series 21. Data on natality, marriage, and divorce. —L’ariou~ statistics [m n~ttility, marriage, and divorce

other than as included in regular annual m- monthly reports -+ljecial tinalyses by demographicvariables, also geographic and time series analyses, studies of fertilirf.

Series 22. Data from the National Natality and Mortality swwys.- Statistics on chaiactcris,.ics of birthsand deaths not available from the vital records, based on sample surveys stemming from theserecords, including such topics as mortality by socioeconomic class, hospital experience in the

last year of life, medical care during pregnancv, health insur.ince coverage, etc.