Archs oral Biol. Vol.1 I, pp.867-875, 1966. Pergamon Press Ltd. Printed in Gt. Britain.

EVALUATION OF EXAMINER VARIABILITY AND THE USE OF RADIOGRAPHS IN DETERMINING THE EFFICACY OF

COMMUNITY FLUORIDATION

H. S. HOROWITZ

Disease Control Branch, Division of Dental Health, Public Health Service, U.S. Department of Health, Education, and Welfare, Washington, D.C. 20201, U.S.A.

and J. K. PETERSON

North Dakota State Department of Health, Bismarck, North Dakota 58501, U.S.A.

Summary-DMF tooth and surface data of II- and 12-year-old children living in a fluoridated and a low fluoride community have been obtained by two examiners, working independently. One examiner (A) used only visual findings and the other examiner(B) used visual and visual-plus-radiographic findings in making an evaluation of fluoridation. A slight but consistent difference in the level of caries-detection was demonstrated in the visual findings of the two examiners; examiner A tended to record more carious surfaces than examiner B. Very similar percentage differences were shown, however, in the caries prevalence of the children in the two communities by each examiner’s sets of data.

Approximately 25 per cent more carious teeth and one-third more carious surfaces were diagnosed in each community when examiner B’s radiographic findings were added to his clinical findings, but the addition of radiographic findings did not appreciably alter the relative difference in caries prevalence in the two communities based on his visual findings alone.

The dental benefits of community fluoridation are apparent in this report. An average of the results obtained by the two examiners showed that children in the fluori- dated community had 40 per cent fewer DMF teeth and 50 per cent fewer DMF surfaces than their counterparts in the low fluoride community. According to examiner A’s visual findings, 18 per cent of the 12-year-olds in Fargo had zero DMF surfaces, whereas only 1 per cent of children that age in Moorhead were free of carious surfaces.

The protective effect of fluoride was found to be strikingly more apparent on approximal surfaces than on occlusal or buccal-lingual surfaces.

NUMEROUS surveys have been conducted to determine the prevalence of dental caries in given populations and to assess the efficacy of community water fluoridation. One of the problems in conducting such surveys and in evaluating their results when two or more examiners are used is the variability in diagnostic judgment that may exist between the examiners. Even when all examinations are made by a single examiner, it is difficult to compare results of one survey with others because of possible variations in the employed standards of diagnosis. Variation in diagnostic level can be consider- able (DEATHERAGE, WILSON and LEDGERWOOD, 1939; BERGGREN and WELANDER, 1960; GREEN and WEISENSTEIN, 1960; WEISENSTEIN, PETERSON and NORRIS, 1961: NIELSON, SCOLA and OSTROM, 1962). The present report focuses on this problem by

867

868 H. S. H~IWWIIL ANU J. K. PLI.ERSON

evaluating the results of two examiners employed in a study comparing DMF data obtained in an optimally fluoridated and a low-fluoride community.

This report also includes data on visual and visual-plus-radiographic findings. Radiographs have been used by numerous investigators to supplement the findings of clinical examinations for dental caries. A resume of studies on the value of radio- graphs used in clinical investigations appears in a report of a conference held in 1955 under the auspices of the American Dental Association (1955). All the studies cited therein show that radiographs reveal carious lesions not found by clinical ex- amination. Depending on the methodology used and the types of surfaces considered, however, the number of lesions discovered only by radiographic diagnosis varies from an additional 5 per cent to more than 100 per cent.

3%~ data reported herein were obtained from base line dental examinations made by two examiners in a study being conducted in the adjacent communities of Fargo, North Dakota and Moorhead, Minnesota, to test the effectiveness of a carics- preventive prophylaxis paste applied in a fluoridated and a low-fluoride community. Fargo has been fluoridating its water since June 1952 and has maintained an average level of fluoride close to 1.2 ppm. Moorhead is separated from Fargo by the Red River and, for many years, has had a low level of natural fluoride in its water supply, varying from 0.2 to O-4 ppm.

The 1 I- and IZyear-old children reported on have been life-long residents or have lived away from their respective communities for less than a year. Base line examin- ations were carried out in November and December 1964. Thus, all of the 1 l-year-old children and approximately half of the 12-year-olds in Fargo were born subsequent to the initiation of fluoridation.

All the children in the study were examined independently by both examiners, each of whom has had considerable experience conducting dental examinations for epidemiological and field investigations. The examiners did not attempt to standardize criteria for diagnosis or calibrate examining techniques. Seated in portable dental chairs under good artificial light, the children were examined with single-ended, sickle-shaped explorers (No. 23 Tempryte) and plane mouth mirrors. In addition, examiner B used a Burton diagnostic mouth light. The length of examination averaged approximately 2 min per subject for examiner A and a few seconds less for examiner B.

Two posterior and three anterior bitewing radiographs were taken of each child, and these were subsequently read by examiner B only. Any carious approximal sur- faces diagnosed on the radiographs by examiner B that were not present on the re- cording of his visual examination results were added to his record forms with a different coloured pencil. This method of recording allowed a comparison of the visual examinations of the two examiners and, in addition, a comparison of examiner B’s visual and visual-plus-radiographic findings.

EXAMINER VARIABILITY, USE OF X-RAYS IN FLUORIDATION SURVEYS 869

RESULTS

Results of the visual examinations by examiners A and B and the visual-plus- radiographic findings of examiner B are shown in Table 1. A comparison of the two visual examinations according to age and community reveals that examiner A detected, on average, 0.16 more DMF teeth and 0.49 more DMF surfaces per child than examiner B.

Examiner B diagnosed an average of approximately 25 per cent more DMF teeth and about one-third more DMF surfaces when his radiographic diagnoses were added to his visual findings.

The differences in DMF prevalence between Fargo (fluoridated) and Moorhead

(low-fluoride) found by each examiner are shown in Table 1 as perceniuge less in

TABLE 1. DMF TOOTH AND SURFACE RATES OF 1 l- AND 12~YEAR-OLD CHILDREN IN FARGO AND MOORHEAD ACCORDINQ TO VISUAL AND VISUAL-PLUS-RADIOGRAPHIC EXAMINATIONS

Age and community

Examiner A Examiner B Number

examined Percentage Percentage Visual Percentage Visual less in Visual less in plus less in

Fargo Fargo X-ray Fargo

DMF teeth Age11

Moorhead Fargo

Age 12 Moorhead Fargo

DMF surfaces Age 11

Moorhead Fargo

Age 12 Moorhead Fargo

145 444 221 2.67 40

3:: 5.37 3.47 35

145 7.61 7.21 938 221 3.84 50 3.37 53 4.33 54

9.5 9.36 315 5.25 44

4.38 5.30 2.36 46 2.94 45

5.22 6.72 3.35 36 4.17 38

8.61 4.93 43

12.19 6.45 47

Fargo. Based on examiner A’s visual findings, 1 l-year-old children in Fargo had 40 per cent fewer DMF teeth and 50 per cent fewer DMF surfaces than children the same age in Moorhead. The corresponding reductions based on examiner B’s visual findings are very similar, 46 per cent and 53 per cent, respectively. For the 12-year-old children, a comparison of the percentage differences in Table 1 based on visual findings of examiner A and examiner B shows a variation of only 1 percentage point for both DMF teeth and DMF surfaces. The reductions in caries prevalence found in Fargo do not change materially for either age group when the visual-plus-radiographic findings of examiner B are compared with his visual findings alone.

The findings of both examiners clearly show substantially lower DMF rates in Fargo than in Moorhead and thus demonstrate the efficacy of community fluoridation. Somewhat greater percentage differences are recorded when data on DMF surfaces are compared rather than DMF teeth,

870 H. S. HOROWITZ AND J. K. PETERSON

The benefits of fluoridation appear to be more pronounced for the II-year-old children in Fargo than for the lZyear-old group. When the results according to examiner and examining technique are averaged, children age 11 in Fargo had 44 per cent fewer DMF teeth and 52 per cent fewer DMF surfaces than their counterparts in Moorhead. For the 12-year-olds, the corresponding average per cent differences were 36 and 45, respectively.

Table 2 presents a breakdown of average DMF surface rates according to type of surface. There are minimal differences in the visual findings of examiner A and examiner B for occlusal and buccal-lingual surfaces. In regard to the approximal

TABLE 2. DMF SURFACE RATES ACCORDING TO TYPE OF SURFACE FOR ll- AND I&YEAR-OLD CHILDREN COMBINED IN FARGO AND M~~RHEAD

Type of surface

Examiner A

Visual Percentage

less in Fargo

33

72

38

44

Occlusal Moorhead Fargo

Approximal Moorhead Fargo

Buccal-lingual Moorhead Fargo

Total Moorhead Fargo

4.09 2.76

2.10 0.59

2.11 1.31

8.30 466

Examiner B __~.

Percentage Visual Percentage Visual

4.14 2.63

1.63 0.52

2.00 1.14

7.77 4.29

less in Fargo

36

68

43

45

plus X-ray

4.14 2.63

4.35 1.81

2.00 1.14

10.49 5.58

less in Fargo

36

58

43

47

surfaces, examiner A diagnosed 29 per cent more as carious in Moorhead than examiner B and 13 per cent more in Fargo. The addition of radiographic findings to examiner B’s visual findings resulted in DMF surface rates for approximal surfaces approximately 3 times as great as his rates based on visual examinations alone. In- cluding his radiographic findings, examiner B diagnosed more than twice as many approximal lesions in Moorhead children as examiner A and, in Fargo, approximately three times as many.

Substantial differences are shown in Table 2 on the relative effects of fluoridation according to type of surface. Even though the percentage differences in caries pre- valence in Fargo compared with Moorhead vary slightly according to examiner and examining technique, all three sets of data demonstrate considerably greater benefits to approximal surfaces than to occlusal or buccal-lingual surfaces. The reductions on approximal surfaces range from 58 per cent based on examiner B’s visual-plus-radio- graphic results to 72 per cent based on examiner A’s visual findings. The reductions for buccal-lingual surfaces are slightly greater than for occlusal surfaces.

EXAMINER VARIABILITY, USE OF X-RAYS IN FLUORIDATION SURVEYS 871

An additional analysis of the surface data, presented in Table 3, shows the com- parative prevalence of carious surfaces by type of surface in the Fargo and Moorhead children according to examiner and examination technique. A comparison of the

TABLE 3. PERCENTAGE DISTRIBUTION OF CARIES PREVALENCE IN FARGO AND M~~RHEAD FOR AGE GROUPS 11 AND 12 COMBINED ACCORDING TO TYPE OF SURFACE

-- __

Examiner A Examiner B Type of surface Visual Visual Visual lus X-ray

Moorhead Fargo Moorhead Fargo Moorhead Fargo __._____

Occlusal 49 59 53 61 39 47

Approximal 25 13 21 12 42 32

Bueeal-lingual 26 28 26 21 19 21

All surfaces 100 100 100 100 100 loo

visual findings of examiner A and examiner B indicates that the percentage distribu- tions by surface type are consistent. There is, however, a pronounced change in the distribution of carious surfaces in both communities when radiographic findings of examiner B are added to his visual results. In Moorhead, approximal surfaces accounted for 21 per cent of total caries prevalence as determined by examiner B’s visual examinations alone. This proportion was doubled (42 per cent) when his radiographic findings were added. The proportionate change on approximal surfaces was even greater in Fargo, increasing from 12 per cent of the total caries rate when only visual findings were considered to 32 per cent for visual-plus-radiographic findings.

Table 3 offers additional evidence that approximal surfaces of teeth are protected to a greater extent by fluoridation than are other surfaces. Regardless of which examiners’ data are used or whether or not radiographic findings are included, the percentage of carious approximal surfaces to total carious surfaces is lower in the fluoridated com- munity than in the low-fluoride community.

Cumulative percentage distributions of children age 12 in Fargo and Moorhead classified according to their number of DMF surfaces are shown in Table 4. This Table shows how the addition of examiner B’s radiographic findings appreciably alters his data on the cumulative distribution. The observable change in the cumulative distribution by the use of radiographic findings in general is more pronounced in the non-fluoridated community than in the fluoridated community. For example. according to examiner B’s visual findings in Fargo, 69 per cent of the lZyear-olds had six or less DMF surfaces whereas, when the radiographic diagnoses were added to his clinical findings, 61 per cent of the children still fell into this category. In comparison, 45 per cent of the lZyear-olds in Moorhead had six or less DMF surfaces on the basis of examiner B’s visual results. However, his visual-plus-radio- graphic results showed only 29 per cent in this category. Almost all the other categories of cumulative DMF surface prevalences show a similar relation between the two communities. Because the cumulative percentage distribution of DMF surfaces among 12-year-olds was altered to a greater extent in Moorhead than in Fargo, there

872 H. S. HOROWITZ ANT) J. K. PETERSOW

‘TABLF 4. CUMULATIVE PERCENTAGE DISTRIBUTIONS OF CHILDREN AGE 12 IN M~~RHEAD

AND FARGO ACCORDING TO NUMBER OF DMF SURFACES

Number ot

DMF surfaces

0 I or less 2 or less 3 or less 4 or less 5 or less 6 or less 7 or less 8 or less 9 or less

IO or less I1 or less 12 or less 13 or less I4 or less IS or less

Moorhead Fargo ~.~__._.

Examiner A Examiner R Examiner A Examiner R ___~~__ -. ..__-- _-.___-..

Visual Visual Visual Visual plus Visual Visual plus

X-ray X-ray

I 3 1 18 17 12 4 7 5 29 32 23 x 12 6 37 40 34

15 IS 7 44 46 41 26 30 13 51 54 48 36 36 20 60 60 52 41 4.5 29 67 69 61 46 50 34 71 74 65 50 61 42 76 XI 71 56 65 46 83 85 75 62 69 s2 x7 88 80 71 73 59 90 91 x3 13 76 64 92 93 87 80 XI 64 95 96 90 82 90 71 95 96 91 x5 90 71 96 96 93

-.

is an indication that there was substantially more carious activity (sub-visually) on the approximal surfaces of permanent teeth in the low-fluoride community than in the optimally fluoridated community.

The data in Table 4 further emphasize the value of community fluoridation for dental caries prevention. Using examiner A’s findings, 18 per cent of the children age 12 in Fargo had zero (0) DMF surfaces; only 1 per cent of that age group in Moorhead were free of caries. According to the visual-plus-radiographic results of examiner B, almost one-half of the Fargo children had less than 5 DMF surfaces, while only 13 per cent in Moorhead had such findings.

DISCUSSION

Except for occlusal surfaces in Moorhead, examiner A consistently detected, on average, more DMF surfaces by visual examination than did examiner B (Table 2). The disparity is especially apparent on approximal surfaces. This is understandable since examiner B knew that radiographs would be available to supplement his visual findings and thus he frequently skipped over questionable approximal areas to defer final diagnostic judgment to his examination of radiographs. On the other hand, examiner A knew that he would not refer to the radiographs and, therefore, inspected the approximal surfaces more carefully on the visual examination. Even though the examiners’ levels of clinical diagnosis were different, a comparison of relative caries prevalence on approximal surfaces in the two communities by each examiner’s visual data reveals very similar percent differences.

EXAMINER VARIABILITY, USE OF X-RAYS M FLUORIDATION SURVEYS 873

Radiographs added about one-fourth more DMF teeth and one-third more DMF surfaces to the visual findings of examiner B (Table 1). The proportion of the total caries prevalence contributed by examination of the radiographs may be unduly large because of the factors mentioned in the preceding paragraph. However, even when examiner B’s radiographic diagnoses were included, the per cent differences in caries prevalence of the two communities were about the same as those based only on his visual examinations. These data support the position that radiographs are not needed in surveys of comparative caries prevalence or in many epidemiological studies. ENGLANDER, DE PALMA and KESEL (1962) made the same conclusion in a study com- paring the dental health status of 1% and 19-year-old young adults in Aurora, Illinois (about 1.2 ppm of natural fluoride in water) and in Rockford, Illinois (about O-1 ppm). The authors stated that “roentgenograms raised the clinical scores of DMFT and DMFS, but they did not alter the relative differences between the groups to any important extent, and similar conclusions with and without roentgenograms could have been reached regarding the effect of fluoride.”

The beneficial effects of fluoridation are well demonstrated by these data. Sub- stantial reductions are shown in the caries experience of Fargo children compared with those in Moorhead, regardless of examination technique. There are two factors that may account for the magnitude of the reduction being slightly less than that usually expected for this age group. The first possibility is that Moorhead children derive some benefits by occasionally consuming Fargo water. This is likely because the communities are adjacent. Secondly, some consideration must be given to the fact that the natural fluoride content of the Moorhead water supply ranges up to 04 ppm, and thus may afford some cariostatic benefit to the city’s children.

The somewhat greater benefits observed in the 1 l-year-old children as compared with IZyear-olds in Fargo may be related to the fact that some of the 12-year-olds had not had lifetime exposure to the fluoridated water. Because the birth dates of the IZyear-old children occurred throughout the year 1952, it is probable, considering the date of examination, that about one-half of these children were born before the institution of fluoridation in June 1952.

The finding in this study that approximal surfaces receive a high degree of pro- tection from fluoridation is consistent with the results reported by AST et al. (1956) after 10 years of fluoride experience in Newburgh, New York. They concluded that “ingested water-borne fluorides at the optimum concentration affords (sic) selective protection for the approximal surfaces of posterior teeth in comparison with occlusal surfaces”. Their report stressed the value of fluoridation in preventing lesions that arc most difficult to detect and to correct. In an extensive epidemiological study, DEAN,

ARNOLD and ELVOVE (1942) found 19 times as much carious activity on approximal surfaces of the four upper incisors of children exposed to less than 0.5 ppm of fluoride as was observed in children living where the water supplies contained O-6-2.6 ppm of fluoride.

Other studies have shown a marked degree of protection for the smooth and gingival areas of buccal and lingual surfaces. BACKER DIRKS (1963) found the greatest effect of fluoridation to be on smooth buccal surfaces. Twelve-year-old children in a

874 H. S. HOROWITZ AND J. K. PETERSON

fluoridated city had 72 per cent fewer cavities on these surfaces then their counter- parts in a non-fluoride area. in a study by LUDWIG and PEARCE (1963), relative to the Hastings Fluoridation Project in New Zealand, it was reported that the reductions for approximal surfaces were greater than those obtained on occlusal surfaces, but the reduction for gingival surfaces exceeded those for both the occlusal and the approximal surfaces. For the examinations in Fargo-Moorhead, no distinction was made between pit and fissure areas and smooth areas of buccal-lingual surfaces and thus no com- parisons could be made with the data of BACKER DIRKS (1963) or LUDWIG and PEARCE (1963).

Resume-L’indice DMF et des resultats de surfaces d’enfants de I1 et 12 ans, habitant darts des villes dont I’eau est respectivement riche et pauvre en fluor, ont tte determines par deux examinateurs travaillant independamment. Le premier examinateur (A) n’a utilist? que des rtsultats visuels et le second examinateur (B) a utilise, en plus des rtsultats visuels, des examens radiographiques pour tester l’efficacite de la fluoruration. Une difference leg&e, mais nette, dans la frequence des caries detect&es est notee dans les resultats visuels des deux examinateurs: l’examinateur (A) avait tendance a enregistrer plus de surfaces carieuses que I’examinateur (B). Des differences tres similaires de pourcentage sont not& cependant dans la frtquence de caries des enfants des deux cites, en se basant sur les renseignements fournis par chaque examinateur.

Environ 25 pour cent plus de dents cariees et un tiers de plus de surfaces carieuses ont Cte diagnostiques dans chaque ville, lorsque les rtsultats radiographiques de l’examinateur B sont ajoutes % ses resultats cliniques. Cependant I’adjonction des resultats radiographiques ne change pas de facon appreciable la difference relative en frequence carieuse dans les deux villes, en se basant uniquement sur les resultats visuels.

L’action dentaire benefique dc la fluoruration urbaine est apparente dans cc rapport. Une moyenne des resultats, obtenus par les deux examinateurs, montre que les enfants de la ville, appliquant la fluoruration, ont un indice DMF et un indice dc surfaces DMF respectivement de 40 et 50 pour cent inferieur que les enfants vivant dans la ville dont l’eau est pauvre en fluor. D’aprts les resultats visuels de I’examinateur (A), 18 pour cent des enfants de 12 ans a Fargo ont un indice DMF de surfaces de zero, alors que seulement I pour cent des enfants de Moorhead, de mtme age, ne presentent pas de surfaces carieuses.

L’effet protecteur de fluor est nettcment plus apparent sur des surfaces proximalcs que sur des surfaces occlusales ou vestibulo-linguales.

Zusammenfassung-Von 2 unabhPngig voneinander arbeitenden Untersuchern wurden DMF-Zahn und -0berflachen Werte von 11 und 12 Jahre alten Kindern gewonnen, die in einer mit fluoridiertem Trinkwasser versorgten Gemeinde und in einer anderen mit geringem Fluoridgehalt lebten. Der eine Untersucher (A) verwendete ausschliesslich visuelle Befunde, der andere (B) benutzte bei der Untersuchung visuelle und visuell- plus-Rontgenbefunde. Ein geringer, jedoch anhaltender Unterschied bei der Kariesregis- trierung wurde bei den visuellen Beobachtungen der beiden Untersucher festgestellt ; Der Untersucher A neigte dazu, mehr kariose Oberfllchen als Untersucher B aufzuzeich- nen. Sehr Lhnliche prozentuale Unterschiede ergaben sich jedoch aus den Ergebnissen beider Untersucher hinsichtlich der Kariespraevalenz der Kinder in beiden Gemeinden.

Annlhernd 25 % mehr kariose ZBhne und I /3 mehr karibse Oberfllchen wurden in jedcr Gemeinde diagnostiziert, wenn die rontgenologischcn Befunde des Untersuchers B den klinischen Beobachtungen hinzugefiigt wurden; diese Addition der Rontgenbe- funde veranderte jedoch die relativen Unterschiede der Kariespraevalenz in beiden Gemeinden, die allein aus den klinischen Befunden hervorgingen, nicht nachhaltig.

Aus diesem Bericht gehen die Vorztige der allgemeincn Fluoridierung fur die Zahngesundheit eindeutig hervor. Ein Durchschnitt der von beiden Untersuchern

EXAMINER VARIABILITY, USE OF X-RAYS IN FLUORIDATION SURVEYS

erzielten Resultate llsst erkennen, dass Kinder in her fluoridierten Gemeinde 40% weniger DMF-Zlhne und 50% weniger DMF-Oberflgchen als ihre Altersgenossen in der Gemeinde mit wenig Fluorid besassen, Entsprechend den visuellen Befunden des Untersuchers A besassen 18 % der 12 Jahre alten Kinder in Fargo Null DMF-Ober- fl%chen, und nur 1% der Kinder der gleichen Altersgruppe in Moorhead wiesen keine kariiisen Oberfllchen auf.

Die protektive Wirkung des Fluorids trat an den proximalen OberflPchen deutlicher als an den okklusalen oder bukkal-lingualen Fllchen hervor.

875

REFERENCES

America1 Dental Association. 1955. Clinical testing of dental caries preventives. Report of a con- ference to develop uniform standards and procedures in clinical studies of dental caries. Appendix c, 51-53.

AST, D. B., SMITH, D. J., WACHS, B. and CANTWELL, K. T. 1956. Newburgh-Kingston caries-fluorine studv. XIV. Combined clinical and roentgenoaraohic dental findings after ten Years of fluoride expe-rience. J. Amer. dent. Ass. dent. Cosmos 527 31^4-325.

BACKER DIRKS, 0. 1963. The assessment of fluoridation as a preventive measure in relation to dental caries. Br. dent. J. 114,211-216.

BERGGREN, H. and WELANDER, E. 1960. The unreliability of caries recording methods. Acta odonr. stand. 18,409-420.

DEAN, H. T., ARNOLD, F. A., JR. and ELVOVE, E. 1942. Domestic water and dental caries. V. Additional studies of the relation of fluoride domestic waters to dental caries experience in 4,425 white children, aged 12 to 14years, of 13 citiesin4states.Publ. Hlth Rep. 57,1155-1179.

DEATHERAGE, C. F., WILSON, L. A. and LEDGERWOOD, R. 1939. Variability of routine diagnosis of dental defect in children of school age. J. Amer. dent. Ass. dent. Cosmos 26,1739-1748.

ENGLANDER, H. R., DE PALMA, R. and KESEL, R. G. 1962. The Aurora-Rockford, Ill., Study. I. Effects of water having naturally occurring fluoride on dental health of young adults. J. Amer. dent. Ass. dent. Cosmos 65,614-621.

GREEN, G. and WEISENSTEIN, P. R. 1960. Caries diagnosis and experimental caries conference. Ohio State University Research Foundation, Columbus, Ohio.

LUDWIG, T. G. and PEARCE, E. I. F. 1963. The Hastings fluoridation project. IV. Dental effects between 1954 and 1963. N. Z. dent. J. 59,298-301.

NIELSEN, A. G., SCOLA, F. P. and OSTROM, C. A. 1962. Dental examination reliability, memorandum report no. 62-2. U.S. Naval Medical Research Laboratory.

WEISENSTEIN, P. R., PETERSON, J. K. and NORRIS, P. E. 1961. A comparison of two types of clinical examinations: the effect on observed caries prevalence and increments. J. dent. Res. 40,492-496.