Assessment of S-Night Street Enumeration in the 1990 Census

by

Elizabeth Martin

U. S. Bureau of the Census

Washington, D.C. 20233

February 3, 1992

Assessment of S-Night Street Enumeration in the 1990 Census

Abstract

On March 20-21, 1990, the Census Bureau conducted "Shelter and Street-

Night" to count components of the homeless population in emergency shelters

and street locations. Observers were placed in a sample of street sites in 5

cities to report on the census enumeration process. Comparison of observer

reports with census returns indicates that street enumeration was not carried

out in a comparable, standardized way in the 5 cities. The main operational

problems were enumerator failure to enumerate sites and selectivity in

approaching people within sites. Variability in how the operation was carried

out reduces the comparability of street counts from place to place.

Introduction

On the night of March 20-21, 1990, the Bureau of the Census conducted a

"Shelter and Street-Night" (S-Night) operation to count selected components of

the homeless population in preidentified emergency shelters and open locations

in the streets and other places not intended for habitation (see Taeuber and

Siegel, 1991, for a description of S-Night). The procedures involved

enumerating all people at preidentified emergency shelters for homeless

persons, subsidized hotels and motels and temporary shelters, from 6 p.m. to

midnight, March 20. (For preliminary results of an assessment of the shelter

list completeness, see Schwede and Salo, 1991.) Street enumeration was

implemented from 2 to 4 a.m., March 21. For the street phase, enumerators

were to interview all people visible and awake, who were not in uniform or

engaged in money-making activities, in preidentified nighttime street sites

and all-night places of commerce. Sleeping persons were not to be wakened for

an interview, but they were to be counted and their age, race, and sex

estimated by observation. The sites--city parks, areas under bridges, bus and

3

train stations, hospital emergency rooms, and other locations where homeless

people were thought to stay at night--had been identified prior to the census

by local governmental units, police, groups working with homeless persons, and

Census Bureau district office personnel. From 4 to 8 a.m., enumerators were

to count people emerging from pre-identified abandoned buildings where

homeless persons were thought to stay.

The goals of the assessment were to assess how well enumeration

procedures were implemented and followed by enumerators at street sites, and

to identify external factors that influenced the street enumeration.1

For the assessment, researchers in 5 cities placed teams of 60 in-place-

observers (120 in New York) at a sample of street sites which had been

designated for S-Night enumeration. The cities (Chicago, Los Angeles, New

Orleans, New York and Phoenix) were chosen purposively to represent different

regions and weather conditions, and to include the 2 cities believed to have

the largest homeless populations (New York and Los Angeles). Based on

guidelines provided by the Census Bureau, observers were trained in census

enumeration procedures and how to conduct themselves on site. The observers

were instructed to stay in the open to enable census enumerators to see and

enumerate them. They were to observe whether enumerators came to the sites,

and if so, when they arrived, how long they stayed, and how they conducted the

enumeration. Observers also were to report whether they were interviewed or

believed they were counted by observation, and to describe environmental

conditions affecting the census count. Observers recorded their observations

on questionnaires which were filled out immediately after the street phase was

finished. Completed questionnaires were collected by Census Bureau staff the

next morning for transmittal to Washington, D.C., for coding and keying. Each

observer also filled out an Individual Census Report form "as you believe the

enumerator filled it out for you." District office personnel matched these

dummy forms against the census forms to remove census forms for observers who

4

were enumerated.

The purpose of this paper is to assess how well the street enumeration

was carried out, and consider the implications of the findings for the quality

of the data. S-Night street enumeration was implemented by the Census Bureau

for the first time in the 1990 Census. Likewise, the method of the

assessment, which relies upon reports of unobtrusive observers, is a new one

which has never been tried before by the Census Bureau. Therefore, both

sources of data must be carefully and critically examined in order to assess

the S-Night street enumeration, and the quality of counts resulting from it.

Site Selection and Matching

District offices in the 5 cities were purposively chosen in advance to

represent areas within each city where large numbers of homeless people were

expected to be found. In all cities but New York, the study area covered one

district office; the New York study area covered four. In all cities but New

Orleans, the sample area represents only part of a city. The results cannot2

be generalized beyond the specific district office areas covered.

Researchers were given standard instructions for selecting a systematic

random sample of all pre-identified commerce and street sites in their study

area, using S-Night Enumeration Records prepared by the local Census Bureau

district office to make enumerator assignments to the sites. There were a few

differences among study areas in how the sites were identified by the Census

Bureau and sampled by the researchers. In New York, the City had conducted

two extensive canvassing efforts to identify street sites for the S-Night

operation. The availability of information from the canvass permitted the

researcher to stratify sites and sample only those where 6 or more occupants

were expected. The New Orleans District Office identified large, four-by-four

block areas as sites, and the researchers sampled locations within these areas

to determine where to station observers. In Los Angeles, the researchers

5

randomly sampled sites from the list provided by the Census Bureau, then added

9 sites which were not on the list, based on their information indicating that

the sites included homeless occupants. The 9 extra sites were not part of the

sample and are excluded from the results reported below.

In order to ensure that census and observer results refer to the same

sites, Census Bureau staff matched geographic information from the official

census lists of street and commerce sites in the 5 study areas against

geographic descriptions of sites from researchers' lists and observer

questionnaires. Since a key assessment variable is whether or not census

enumerators were observed at the site, it was essential to determine that

observers were stationed at the sites the Census Bureau intended to enumerate.

The census listings included census geocode information, site

descriptors and names, and addresses. The researchers' site lists and

observer questionnaires included addresses and site descriptions. In many

cases, it was difficult to determine the match status of sites, and additional

sources were consulted, including the Census Bureau's computerized master

address list or Address Control File, S-Night Enumeration Records used to

assign enumerators to sites, census and city maps, and phone calls to city

officials, police, and the researchers. Census sources include information

about sites which were deleted at some stage in the census process as well as

those which were included in the final census count. Ultimately, 16 out of

156 observer sites could not be matched to the census. Three of these were

sites where observers went to wrong addresses (2 sites in Phoenix and one in

Los Angeles). Thirteen observer sites, all in Chicago, could not be matched

because they could not be identified in any census source. For that study

area, site descriptions in the census sources were often vague, and frequently

identified only the block within which a site was located, without specific

addresses or site descriptions needed for matching. However, one of the 13

sites had specific address information, and observers saw census enumerators

6

there, yet the site could not be located in census sources.

Other difficulties with matching occurred in New York, where the Census

Bureau subdivided several very large sites into multiple sites, with separate

geocodes and population counts reported for each. The researcher also

stationed observers at different locations within each of these areas.

Matching in these sites is somewhat arbitrary, because not enough geographic

detail is given to pinpoint exact locations for either the census or the

observers. In New Orleans and in Chicago, the researchers subdivided some

large sites into multiple sites, which the Census Bureau treated as single

sites. The observer reports can be readily matched to the census, but the

definition of what the "site" covers (and the total number of sites) is at

variance for the census versus the observers.

The results reported below are based on the 140 observer sites which

constitute the original sample exclusive of the 16 nonmatched sites. They

correspond to 140 sites as defined by the Census Bureau. Analysis is based3

on results for census-defined sites, to which enumerators were assigned and

for which official census counts were tabulated.

Official Census Results for Matched Sites

Official census counts were returned for 130 of 140 sites, and 1,803

people were counted at the 130 sites, as shown in Table 1. Ten sites, all in

Los Angeles or Phoenix, were eliminated at some stage in the census process

and final counts were not processed through the official census count. Five

of these sites had nonzero population counts recorded on the master address

list, which implies they were enumerated and the results were processed before

being deleted. The total population count for the deleted sites was 49.

Although sites with positive counts should not have been deleted, they appear

to have been eliminated during local review, or other closeout or cleanup

activities occurring at the time (Jackson, 1991). The effect was to reduce by

7

22 percent the total census population count for the Los Angeles sample sites;

the effect in Phoenix was negligible. The 10 deleted sites are excluded from

the rest of the analysis.

Table 2 presents census outcomes for the 130 matched, processed sites

according to whether census enumerators were seen by any observers stationed

at the sites. The results show that census outcomes and observer reports are

largely consistent. Sites where enumerators were observed were more likely to

return positive census counts than sites where enumerators were not observed

(66 versus 36 percent). Conversely, enumerators were seen in 77 percent (55

of 71) sites with positive counts, compared to 49 percent (28 of 57) sites

with zero counts. However, two aspects of these results require examination.

First, observers reported seeing no census enumerators at 35 percent of sites.

If observer reports are accurate, this would imply a failure to enumerate a

substantial number of street sites. Second, at 36 percent of street sites

where no census enumerators were observed, positive census counts nevertheless

were returned. This is a high rate of discrepancy between observer reports

and census results.

Ten of the 16 anomalies occurred in one district office in the New York

study area. When results for that office (in south Manhattan) are excluded ,4

census and observer results are more consistent. Enumerators were observed in

almost 90 percent of sites which had positive census counts. Census counts of

0 were returned in 82 percent of sites where no enumerators were observed.

The reasons for inconsistencies between observer reports and census

results, and for their concentration in one district office, require

examination. There may have been a high rate of failure to observe

enumerators, fabrication of results by census enumerators or by observers,

difficulty finding or identifying sites, or some other error or failure to

follow procedures on the part of observers or the Census Bureau.

8

Factors Affecting Consistency of Census and Observer Results

Timing. To ensure that census enumerators would be observed if they came to

the site, observers were instructed to arrive on site at 1:45 a.m. and remain

there until 4:15 a.m. Late arrival or early departure could reduce an

observer's chances of observing census enumerators, especially since

enumerators typically stayed at each site only 15 minutes on average,

according to observer reports. However, timing of observer arrival appears

to have had little if any overall effect on the likelihood of seeing census

enumerators. Census enumerators were seen at 67 percent of the sites where

observers arrived before 2 a.m. (N=105 sites), compared to 53 percent of sites

where observers arrived at 2 (N=15) and 50 percent of sites where they arrived

after 2 (N=10). These differences are not significant (X =1.9, df=2). 2

The timing issue is complicated by the fact that a number of census

enumerations occurred outside the 2-4 a.m. time frame. Early census

enumeration was reported by observers at 10 sites (5 of these were in New

York), and late enumeration was observed at one site. We have little

information on enumeration outside the scheduled 2-4 a.m. (observers could

observe it only if they themselves were present early or late). However,

early (or late) census arrivals could account for a few discrepancies between

observer reports and census results. To the extent that census enumerators or

observers did not adhere to the scheduled time of street enumeration, the

likely effect is underreporting of enumerators' presence at street sites.

Curbstoning. Fabrication of results by enumerators or by observers also was

examined as a possible explanation for anomalous results in south Manhattan.

Enumerator assignment records were checked to see if a suspicious number of

anomalous sites were enumerated by the same team, or had no enumerator names

recorded on the form. The anomalous sites were scattered among census

enumerator teams. Likewise, examination of the extensive write-in answers and

detailed descriptions which appear on most observer questionnaires did not

9

suggest curbstoning by observers. Although curbstoning by either group cannot

be ruled out, there is no evidence to suggest it happened.

Possible difficulty of identifying census enumerators. Some census

enumerators were supplied with and wore very visible vests labelled "census

taker," or carried large Census Bureau shoulder bags. Those who did were more

readily identifiable as enumerators. At a few sites, observers report seeing

people they took to be enumerators drive by in cars, sometimes stopping at the

site and sometimes not. (Enumeration by car was not standard procedure.)

Enumerators who enumerated from their cars may have been unnoticed by

observers, or observers may have incorrectly identified people driving by as

enumerators. Confusion about people in cars could lead to either overreports

or underreports of enumerators' presence at the sites.

Problems identifying sites and other impediments to observation. As discussed

above, in many instances geographic information about sites was vague or poor.

Sites where geographic information does not match have been eliminated.

Nevertheless, observers as well as Census Bureau personnel reported that some

site descriptions were ambiguous or inconsistent as to the area covered, and

its boundaries. Over 90 percent of the census enumerators in the study areas

reported having problems finding the S-Night places to which they were

assigned (Barrett, 1991), with little variation among areas.

Subtle failures of observation could occur in sites which were large,

dark, or contained visual barriers or passageways which made it difficult to

detect the presence of census enumerators. Pertinent to this point are 7

sites where observers disagreed among themselves on whether enumerators came

to the site. These were large and complex sites, such as subway stations,

parks, and a bus station, where an observer's vantage point might have

determined whether he or she noticed enumerators at the site.

The size and complexity of the sites, ambiguous site information, and

some problems with timing appear to account for most of the south Manhattan

10

sites for which census results were reported yet no enumerators were observed.

Census Bureau staff members visited and photographed all 10 anomalous site

locations (see Schwede, 1991). They found that site information on the

assignment records often was vague or defined a large area, and sometimes was

internally inconsistent (e.g., a building name and address did not

correspond). Thus, enumerators might have gone one place and observers

another, even though both went to the "site" as defined by the site

description. (These sites were described in sufficient detail, and

consistently enough in observer and census sources, that the sites were deemed

matches in the matching operation.) At several sites the evidence indicates

that census enumerators arrived early, and/or observers arrived late, left

early, or left the site unobserved for a period of time. It appears probable

that census enumerators enumerated these 10 sites (or perhaps areas close by),

but that observers and enumerators were present at slightly different places

or times, resulting in inconsistent observer reports and census results.

We did not investigate the other 6 anomalous sites, but it is possible

to contrast the effects of alternative assumptions about whether they were

enumerated or not. In column (1) of Table 3, we assume the anomalous sites

were not enumerated, and in column (2) we assume they were. In column (2), we

in effect take either an observer report or a positive census count as

evidence that census enumerators visited the site. In New York, we believe

that the results in column (2) are more accurate, and that about 90 percent of

the sites were visited by enumerators. In New Orleans and Los Angeles study

areas, it makes little difference what we assume: in New Orleans, we have

evidence that all sites were visited by enumerators, and in Los Angeles, about

half. In the Phoenix study area, evidence suggests that between 57 and 67

percent of sites were visited by enumerators, and in the Chicago study area,

between a third and a half. Thus, no matter what we assume about the

anomalous sites, evidence suggests that substantial numbers of sites may have

11

been missed in Chicago, Los Angeles, and Phoenix.

The quality of observer questionnaire data. Observer questionnaires were

filled out immediately after S-Night, generally at 5 or 6 a.m. Observers were

tired, and some questionnaires were incomplete or poorly filled out. A few

street observers appear to have been functionally illiterate or not fluent in

English, and for a number of questionnaires there are indications that

questions were misread or misunderstood. The data yield useful information

about the S-Night street enumeration, but the quality of the data is not high,

and it would not be valid to attempt to develop precise estimates of how many

people might have been missed in S-Night on the basis of them.

Comparison of Census and Observer Counts

Table 4 compares total census counts for the sites with the high and low

numbers of people the observers estimated to be at the sites between 2 and 4

a.m. The range of low estimates was calculated by selecting the lowest

observer's estimate of the low number people in each site and summing over

sites, and then by summing over the highest observer's estimate of the low

number of people in each site. For sites with one observer, his or her

estimate was included in both sums. The range of high estimates was

calculated similarly. (As an example, if one observer at a site reported 4

and 10 as the lowest and highest number of people, respectively, while another

reported 9 as both the lowest and highest number, then for that site the range

of low estimates is 4-9, and the range of high estimates is 9-10.) Separate

totals are given for sites according to whether census enumerators were

observed there or not. The observers' reports of numbers of people at the

sites should be treated with caution, since the summed estimates of their low

and high numbers cover considerable ranges. Possibly, some observers were

selective in whom they counted, as some enumerators appear to have been (see

below), which would introduce variability in the observer estimates.

12

Variability in observer estimates also partly reflects the way the sums were

formed as well as error in the data. In some sites, different observers,

although technically reporting on the same site, may have had different

vantage points and different areas they were reporting about.

In addition to high inter-observer variability, there are large

differences between the low and high number of people estimated by observers

to be present in the sites between 2 and 4 a.m. These differences imply

considerable mobility into and/or away from sites, and suggest the possibility

that mobile persons could be counted at multiple sites, or not at all, if they

left a site before enumeration or arrived afterwards. Differences between the

low and high numbers of people present also imply that the timing of

enumeration could influence the size of the count at any given site, depending

on whether enumerators arrived at the low point or peak of its occupancy. We

have no evidence to assess possible effects of mobility, however.

In all study areas except New York, census counts are within the range

of observer estimates of the low number of people present in sites where

enumerators were seen. This finding probably reflects enumerator selectivity,

as discussed below. In addition, observers reported their low and high

estimates over the entire 2 hour period; enumerators would not necessarily

have been present at the time when the greatest number of people was present.

New York produces census counts higher than the highest observer counts.

In some New York sites, lack of comparability between sites as defined by the

census and by observers implies that the two sets of counts refer to different

entities. An example is a large transportation terminal where the census

counted 653 people. Observers were stationed at specific areas within the

terminal, and their counts refer to those areas. A comparison of the high

observer count (100) for the site with the census count of 653 is misleading,

because the former refers to a particular part of the site and the latter to

the entire terminal. When observer counts are adjusted to sum across all

13

parts of the site, observer estimates are closer to census counts. Similarly

adjusted figures may be calculated to take account of similar situations in

New Orleans and Chicago, where observers' counts refer to subparts of larger

census sites and hence should be summed. 5

Observers reported substantial numbers of people in sites where no

enumerators were observed. Although there are official census returns for

some of these sites, comparison of observer estimates with census counts

suggests that considerable numbers of people may have been missed in sites

where enumerators were not seen in Los Angeles and Chicago study areas. (But

note the high variability in Los Angeles observers' estimates.)

To measure the consistency between the two sets of data, Table 5

presents correlations between census and observer counts of the number of

people present at the sites. The top panel shows that, for sites where

enumerators were seen, the highest observer's estimates of the low and high

numbers of people present in the site are both significantly correlated with

the census counts. None of these correlations is high, suggesting large

amounts of error and variability in these data.

The second panel in Table 5 presents results separately for the South

Manhattan District Office (where the 10 anomalous sites were) versus all other

areas. Outside of south Manhattan, there are no significant correlations

between observer and census counts in sites where no enumerators were

observed. This is expected, since only 6 such sites have positive census

counts. In south Manhattan, however, observer and census counts are

positively correlated for sites where no enumerators were seen. The positive

correlations imply a relationship between the two sets of data, and are

consistent with the conclusion that, in this district office, observer and

census counts are both genuine and refer to (roughly) the same sites.

Enumerator Behavior

14

Enumerators were instructed to enumerate everyone visible at the site,

except for people in uniform or engaged in money-making activities. They were

not to waken sleeping respondents to interview them, but rather to estimate

age, and record race and sex based on observation. If a person seemed

dangerous, or was mentally incapable of being interviewed, enumeration by

observation also was permitted.

Every observer should have been interviewed. Observers were instructed

to remain in sight and allow themselves to be interviewed by enumerators.

Table 6 presents the percent of observers who report they personally

were interviewed or thought they were counted by observation, in the matched,

processed sites. (Tables 6 and 7 exclude observer reports for the 16 sites

where no enumerators were seen but census counts were returned.) The

proportion of observers interviewed varies enormously, ranging from two-thirds

in New Orleans down to only 7 percent in the Chicago study area. An

additional 6 to 29 percent of observers in each study area believed they were

counted, or thought they might have been. Thus, Table 6 implies the percent

of observers who were certainly or probably enumerated in each study area is:

New Orleans 84%

New York 66%

Phoenix 55%

Los Angeles 39%

Chicago 25%

Several sources of error that influence observer reports have been

discussed. However, even granting that some observers might have been counted

who believed they were not, these rates are very low in some study areas and

show extreme variability among areas. In part, the variability occurs because

substantial numbers of sites apparently were not enumerated in Phoenix, Los

Angeles, and Chicago, where observers saw no enumerators, and census counts of

0 were returned, for 33, 48, and 46 percent of sites, respectively.

15

In addition, selective interviewing and enumeration by observation

contributed to variability in interview rates. Table 7 presents observers'

reports of whom census enumerators approached to interview. These reports are

of unknown reliability, since there are high rates of missing data, mostly

from observers who said they couldn't tell whom enumerators approached. The

proportion of observers who report that census enumerators approached everyone

visible in the street varies from 12 percent in New York to almost half in

Phoenix . Overall, 19 percent of observers report that enumerators approached6

only people who appeared homeless, with no statistically significant variation

among study areas. In all 5 study areas (but especially Chicago and New

York), observers commonly reported that enumerators neither approached

everyone, nor did they approach only homeless-appearing individuals. In some

areas (especially in New York; see Hopper, 1991a) census enumerators

apparently conducted the enumeration predominantly or entirely by observation,

regardless of whether the people in the site were awake and capable of being

interviewed. Enumeration by observation would not necessarily result in

counting errors, although data on age, race, and sex obtained this way are not

as accurate as data obtained by personal interview. Data on marital status

and Hispanic origin were not obtained for cases enumerated by observation.

These results suggest low, and variable, compliance with the standard S-

Night procedure of enumerating all visible persons. Observer reports indicate

no consistent pattern in whom enumerators approached to interview, either

within or across study areas. Enumerators who complied with the procedure of

enumerating all visible persons would obtain more complete counts of sites

than enumerators who enumerated only homeless-appearing people or who were

otherwise selective. Therefore, if observer reports are reliable, variations

in who was selected for enumeration imply that the completeness of the counts

varies among sites within study areas, and among study areas.

The lack of consistency in whom enumerators approached to interview may

16

reflect a weakness of S-Night training. However, the problem of enumerators

ignoring the instruction to enumerate all visible persons was documented in

previous tests (see e.g., Siegel, 1989), so the procedure was emphasized in

training in 1990. As noted, almost a fifth of observers report that

enumerators only approached people who appeared homeless. It has been

suggested that the publicity surrounding S-Night as a "count of the homeless"

seemed contradictory with a procedure of enumerating everyone, leading

enumerators to ignore the procedure and improvise their own ways of "counting

the homeless." Training may not improve compliance by enumerators who reject

the procedure because it appears inconsistent with their understanding of the

task. Enumerator selectivity may be an unavoidable weakness of the 1990 S-

Night procedures.

Limitations of the Assessment

The assessment provides only limited data about the adequacy of S-Night

street enumeration. The assessment was not designed to estimate how

completely the homeless population was counted in the 1990 Census. As yet, no

methods have been developed to accurately measure census coverage of this

population. In addition, it is not valid to generalize the results from the 8

district offices in the assessment to other places or the nation as a whole.

Thus, this assessment study cannot support conclusions about the rate of

census coverage of the homeless population in these cities or in the country,

nor can it support conclusions about how well or poorly S-Night street

enumeration was conducted in places not included in the assessment. In

addition, there is very little information to evaluate several important

aspects of the operation.

Adequacy of street site selection. Street sites to be enumerated were

compiled by district offices with assistance from cities and other agencies,

advocate groups, etc. The criteria used and the adequacy of the compilations

17

of street sites appear to vary from place to place. The assessment reports

note that a number of sites on the census lists appear to be daytime rather

than nighttime congregating sites, and raise other questions about the

adequacy of the sites for enumerating homeless people. However, the quality

and completeness of the list of street sites are unknown.

Who was counted. S-Night enumeration was intended to include homeless people

who otherwise would not have been counted in the census. However, it is

unknown how many of the people who were counted on S-Night had a usual home

elsewhere and were eligible for enumeration there.

Duplication with other operations. S-Night was conducted March 20-21, about 2

weeks before Census Day, April 1. It is unknown how many people who were

counted in the streets on S-Night also were counted as part of regular

household enumeration or as part of another census operation.

Summary and Conclusions

Consistency of observer reports and census results. Although they provide

useful information, observer reports about the census enumeration process are

fallible. Observational errors could occur in large sites with visual

obstructions, if observers did not locate the correct site, if census

enumeration was conducted unobtrusively or by car, if people who were not

enumerators were misidentified as such, if census enumeration was conducted

outside of the scheduled 2-4 a.m. period, or if observers arrived late, left

early, or left the site unobserved. The effect of most of these factors would

be observer underreports of the presence of enumerators. In general (with the

exception of south Manhattan), observer results are sufficiently consistent

with census results that the effect of these factors on observer reports

appears not to have been too great. In south Manhattan, followup

investigation indicates that the complexity and size of the sites combined

with ambiguities in site descriptions and timing problems resulted in some

18

discrepancies between observer and census results, probably because observers

and enumerators were not at precisely the same locations at the same times.

The observational method used in the assessment yielded valuable information

about the street enumeration process. Several of the operational problems

uncovered were not anticipated, and we might not have identified them at all,

or realized their extent, without the information provided by the observers.

However, it is clear that this method is absolutely dependent on accurate and

consistent information about site locations, if reliance is to be placed on

observer reports about what happened (or what failed to happen) in a street

site scheduled for enumeration. In large and complex sites, the method is

vulnerable to observational and definitional difficulties which can affect the

reliability of observer reports (see Hopper, 1991b, on this point).

Adherence to procedures. Observer reports suggest that enumerators in the 5

study areas did not consistently follow standard procedures for conducting

street enumeration. The most serious problem is indicated by the evidence

suggesting that enumerators may have missed half the street sites in Chicago

and Los Angeles study areas, and a third in Phoenix. The effects on the count

would depend on the numbers of people in missed sites who should have been

enumerated. Adverse effects are indicated in Chicago and Los Angeles, where

substantial numbers of people were reported by observers in sites which may

not to have been enumerated. The number of people who should have been

enumerated is unknown, due to inter-observer variability and variability in

the numbers of people present in the sites between 2 and 4 a.m. However, only

in Chicago does the number of people in missed sites appear large relative to

the total census count for all sample sites. Observer reports indicate that

the street count in the Chicago study area might have been doubled, or more,

had the missed sites been enumerated.

In sites which were enumerated, observer reports indicate that

enumerators often did not conduct interviews even when it was possible to do

19

so. Enumeration by observation appears to have been common, especially in New

York. However, there is no evidence that people were missed because of it,

and census counts are high relative to observer estimates in New York.

Many enumerators in all 5 study areas appear to have interviewed

selectively; 30 to 61 percent of observers reported that enumerators did not

approach everyone visible on the street meeting S-Night criteria. This does

not necessarily mean people were missed: in sites enumerated by observation,

enumerators may have approached no one but counted everyone who was eligible

to be counted. This may have happened in New York, where enumeration by

observation apparently did not result in missed people or lower counts

(although it may have affected data quality.)

However, almost 20 percent of observers report that enumerators only

approached people who appeared homeless, with no significant variation among

study areas. Enumerator selectivity clearly influences the numbers of people

counted, and is potentially a large source of variability in the size of the

street counts from site to site and city to city.

Variability among cities. Based on observer reports, street enumeration was

carried out very differently in the 5 study areas in the assessment study.

The various departures from standard enumeration procedures appear to have had

effects on the S-Night street counts ranging from slight to quite large.

Missed sites in the Los Angeles and Chicago study areas led to substantial

numbers of missed people. For 4 study areas, census counts are within the

range of low numbers of people observed in the sites where enumerators were

seen. Enumerator selectivity may have contributed to lower counts. (However,

comparison of observer and census counts cannot support this conclusion in any

definitive way since observers' estimates refer to the entire 2-4 a.m. period,

not to the time enumerators were present.) On the other hand, census counts

in the New York study area exceed observer estimates. Early enumeration,

which was reported more commonly in New York than anywhere else, may have

20

yielded higher census counts there than would have been obtained if all sites

were enumerated between 2 and 4 a.m. However, we have no systematic evidence

on the extent of early enumeration in New York or elsewhere so cannot assess

its possible effect on the counts.

Limitations of S-Night data from street enumeration. As emphasized above, the

observer data cannot support estimates of coverage of the homeless population.

Despite all the caveats noted above, this assessment can support several

conclusions about the limitations of the S-Night street data. It appears

clear that street enumeration was not carried out in a comparable,

standardized way in the district offices represented in the assessment.

Substantial departures from standard procedure appear to have occurred to

varying degrees in all 5 study areas, and the variations in how S-Night was

carried out affected the counts obtained. Most departures from S-Night

procedures (e.g., missed sites, enumerator selectivity) would result in

undercounts, although some departures from procedure (e.g., early enumeration)

could produce overcounts, relative to the standard procedure. Variations in

how S-Night was carried out imply that street counts are not comparable from

place to place, and should not be used to make comparisons of the absolute or

relative size of the homeless population in different places.

21

References

Barrett, D. 1991. Preliminary results of the S-Night enumerator debriefing

questionnaires for the S-Night "assessment" cities. STSD 1990 REX

Memorandum Series #P-7. Census Bureau, May 2, 1991.

Cousineau, M. and T. W. Ward. 1991. An Evaluation of the S-Night Street

Enumeration of the Homeless in Los Angeles. Final report for Joint

Statistical Agreement (JSA) 90-17 between the Census Bureau and the Los

Angeles Homeless Health Project, Los Angeles, CA. April 9, 1991.

Edin, K. 1990. Assessment of S-Night 1990. Final report for JSA 90-16

between the Census Bureau and North Park College, Chicago, IL. May

1990.

Hopper, K. 1991a. Monitoring and Evaluating the 1990 S-Night Count in New

York City. Final report for JSA 90-18 between the Census Bureau and

Nathan Kline Institute, Orangeburg, NY. May 1991.

Hopper, K. 1991b. Repeat Enumeration, Structured Interview, and Brief

Ethnographic Studies of the Street Census Project. Final report for JSA

90-19 between the Census Bureau and Nathan Kline Inst., Orangeburg, NY.

May 1991.

Jackson, A. 1991. S-Night group quarter units. Census Bureau, April 16,

1991.

Schwede, L. and M. Salo. 1991. The shelter component of S-Night.

Proceedings of the 1991 Annual Research Conference. Washington, DC:

Census Bureau.

Schwede, L. 1991. Investigation of reasons for anomalies in DO 2201 street

sites. Census Bureau, November 25, 1991.

Siegel, P. 1989. Observation of 1989 S-Night test, Baltimore. Census Bureau,

July 7, 1989.

Stark, L. 1991. An Assessment of the 1990 Enumeration of Selected Components

22

of the Homeless Population in Phoenix. Final report for JSA 90-15

between the Census Bureau and Community Housing Partnership, Phoenix,

AZ. May 7, 1991.

Taeuber, C. M., and P. Siegel. 1991. Counting the nation's homeless

population in the 1990 Census. In C. M. Taeuber (ed.) Enumerating

Homeless Persons: Methods and Data Needs. Washington, D.C.: Census

Bureau.

Wright, J. D. and J. A. Devine. 1990. Toward Census 2000: Some Lessons

Learned from the 1990 Shelter and Street Night (S-Night) Enumeration of

the Homeless. Final report for JSA 90-12 between the Census Bureau and

Tulane University, New Orleans, LA.

TABLE 1

Total Population Counted at Matched, Census-Defined Sites, by Study Area

New Orleans New York Phoenix Los Angeles Chicago Total

Census count 109 1,318 135 218 23 1,803

Number of sites (18) (54) (21) (23) (14) (130)

Site deleted 2 47 49from the census (4) (6) (10)

Matched sites 18 54 25 29 14 140

The determination that enumerators were observed was based on observers'a

answers to the question, "What was the total number of enumerators you saw at

your site from 2 to 4 a.m.?" Observer questionnaires also were coded to

record any explicit statement that enumerators did, or did not, come to the

site, and to record whether there were conflicting or ambiguous statements

about enumerator presence. This information was used when data on number of

enumerators was missing. If any of the observers at a site saw enumerators,

or if observers reported that enumerators came before or after the 2-4 a.m.

enumeration period, or were near a site but not in it, the site was counted as

one where enumerators were observed.

Two sites where enumerator presence was not ascertained are excludedb

from tables 2-4, for a total of 128 sites.

TABLE 2

Census Outcomes for Sites Where Observers

Did and Did Not Observe Enumeratorsa

Enumerators Notobserved observed

Census Outcome N % N %

Positive official count for site 55 66% 16 36%

Zero official count for site 28 34% 29 64%

Total matched, processed sites 83 100% 45 100%b

TABLE 3

Alternative Estimates of the Proportion of Sites

Visited by Enumerators, by Study Area

(1) (2)Percent of sites where Percent of sites whereenumerators were seen enumerators were seen OR + census counts

TotalStudy Area % of total N of sites % of total N of sites Sites

New Orleans 100% 18 100% 18 18

New York 72% 38 91% 48 53

Phoenix 57% 12 67% 14 21

Los Angeles 48% 11 52% 12 23

Chicago 31% 4 54% 7 13

Total 65% 83 77% 99 128

TABLE 4

Official Census Counts,

Compared with Observer Reports of Low and High Numbers of People at Sitesa

Enumerators were seen Enumerators were not seen Census Observer N of Census Observer N ofCount Low High sites Count Low High sites

New Orleans 109 34-123 78-248 18 -- -- -- 0

New York 1240 256-441 455-732 38 69 68-102 124-160 15

Phoenix 104 90-144 122-170 12 31 13-19 21-45 9

Los Angeles 217 139-258 171-337 11 1 32-212 67-238 12

Chicago 11 9-23 32-43 4 12 33-37 104-109 9

_______________

Low and high observer reports are based on responses to the question, "Thea

following questions refer to the total number of different people at your site

eligible to be enumerated by the census, that is, all persons except those who

were in uniform and those involved in money-making activities, other than

panhandling. If you do not know the exact number, please fill in your best

estimate in the "Approximate Number" column.

Exact Approx.

Number Number

a) If the number of people in the site changed:

1. What was the lowest number there between _____ ______

2 and 4 a.m.?

2. What was the highest number there between _____ ______

2 and 4 a.m.?"

For calculation of ranges of low and high observer estimates, see text.

Preference was given for exact rather than approximate numbers when both were

given; responses are eliminated for a few observers who gave low numbers

greater than their high numbers. If a high number was missing, the low number

was substituted; and if a low number was missing, the high number was used.

Figures given are totals across sites, with no adjustment for missing observer

data (N=2 sites).

TABLE 5

Correlations between Census Counts and

Observer Estimates of Numbers of People in Sites

Sites where enumerators were seen

Low number of people in site

Lowest observer estimate .06

Highest observer estimate .35**

High number of people in site

Lowest observer estimate .19

Highest observer estimate .42**

Number of sites 83

Sites where enumerators were not seen

South Manhattan Other offices Total

Low number in site

Lowest observer estimate .66* -.02 .23

Highest observer estimate .22 -.07 -.03

High number in site

Lowest observer estimate .80 -.09 .05**

Highest observer estimate .47 -.05 -.00

Number of sites 12 33 45

____________

p<.01 (one-tailed test)*

p<.001 " **

Results based on answers to the questions, "Were you interviewed by ana

enumerator?" (Yes, No), and "Do you think you were counted by an enumerator

without being interviewed?" (Yes, Maybe, No). Results are based on reports of

all observers at matched, processed sites, excluding 16 sites where no

enumerators were seen but census counts were returned. Two cases with missing

data excluded. Percents may not sum to 100 due to rounding error.

TABLE 6

Percent of Observers Who Report Being Interviewed or Counted,

by Study Areaa

New New Los Orleans York Phoenix Angeles Chicago

Interviewed 67% 37% 44% 33% 7%

Not interviewed:

Counted 10 17 8 2 0

Maybe counted 7 12 3 4 18

Not counted 10 20 10 13 25

Did not see 5 14 36 48 50enumerators

Total 100 100 100 100 100

N of observers 58 104 39 46 28

Results based on questions, "Did the enumerators approach: Everyonea

visible on the street (except those in uniform or those engaged in money-

making activities other than panhandling?"; "...Only those who appeared

homeless?" (Yes, No, Couldn't tell). Results include reports of all observers

at matched, processed sites who saw enumerators. Percents may not sum to 100

due to rounding error.

TABLE 7

Observer Reports of Who Enumerators Approached, by Study Areaa

Enumerators New New Los approached-- Orleans York Phoenix Angeles Chicago

Everyone visible 35% 12% 48% 14% 23%on the street

Only those who 15 17 22 32 15appeared homeless

Neither everyone, 15 42 13 23 46nor homeless-appearing only

Couldn't tell 35 28 17 32 15who approached;missing data

Total 100 100 100 100 100

N of observers 52 88 23 22 13

1.This paper reports the results of research undertaken by Census Bureau

staff. The views expressed are the author's and do not necessarily reflect

those of the Census Bureau. The S-Night Assessment project was managed by

Pamela Campanelli and Matt Salo, who, assisted by Laurel Schwede, did an

admirable job of planning and implementing the project and designing all

procedures within a very short time period. Rita Williamson and Diane Barrett

conducted the matching, and Annetta Clark and Judy Dawson gave special

assistance. Nancy Bates, Jenny Hess, Brian Jackson, and Elaine Fansler

provided coding and clerical assistance. The assessment method was originally

proposed by Kim Hopper, and data were collected and assessment reports

prepared under the direction of Kim Hopper (New York), James Wright and Joel

Devine (New Orleans), Kathryn Edin (Chicago), Michael Cousineau (Los Angeles),

and Louisa Stark (Phoenix). Thanks to Paul Siegel, Florence Abramson, Annetta

Clark, Diane Barrett, Robert Fay, Robert Groves, Laurel Schwede, Paula

Schneider, Nampeo McKenney, Laurie Moyer, Nancy Mathiowetz, and Robert Tortora

for useful comments on earlier drafts.

2.The study areas covered Manhattan south of 110th St. on the westside and

96th St. on the eastside; part of central Chicago (including the loop);

central Los Angeles (including Skid Row area), most of Phoenix excluding the

westernmost portion, and Orleans Parish.

3.The correspondence between numbers of observer and census-defined sites is:

Observer Sites Census-defined sites

New Orleans 29 18

New York 41 54

Phoenix 25 25

Los Angeles 29 29

Chicago 16 14

Notes

TOTAL 140 140

4.Numbers comparable to Table 2 for the South Manhattan District Office are:

Enumerators observed Not observed

+ census count 12 10

0 census count 1 2

5.Summing figures across observers at subparts of sites to adjust for more

inclusive census site definitions yields the following revised observer

estimates for sites where enumerators were seen:

Observer low Observer high

New Orleans 81-155 178-303

New York 564-911 751-1160

Chicago 13-24 47-55

6.The difference among study areas in the proportion who say everyone was

approached is significant (X =18.4, df=4, p<.01; calculation does not take2

account of clustering in the data).