Like – But Oh, How Different: The effect of different questionnaire formats in the 2005 Census of Agriculture Content Test.

Jaki S. McCarthyNational Agricultural Statistics Service,

United States Department of Agriculture

Census of Agriculture conducted every 5 years

• Includes all known operations with agriculture potential of any type (~3mil)

• Information collected on acreage, production, sales and expenses, production practices, and demographics

• Primarily mail out/mail back form, with follow up mailings, limited CATI and interview follow up

• Reporting is mandatory

Prior to Census, Census Content Test is Conducted• Test used to finalize forms, test data

collection and processing procedures• Small scale mail out/mail back with follow up reminder

• Follow up cognitive interviews conducted with subset of respondents

• Reporting was not

mandatory

2005 Census Content Test (CCT)

• Two forms tested (long and short)

• In 2007 Census, short form will be targeted to specific types of operations

• For CCT, split sample of all farms used for analysis

Both CCT Forms collected the same information

• Differences in:– Format– Layout– Number of pages

Why use a short form?

• May increase response rates

• May be faster and/or easier to complete

• Costs less to print and mail

Why use a long form?• More specific examples and instructions

can be included

• Explicit screening questions and routing instructions can be incorporated

• Common items are pre-printed

• Flexibility with layout can

allow related items to be

placed together

Differences Between Forms

LONG FORM• 24 pages• Acres operated by land type

collected immediately after acres operated by ownership

• Separate sections for different commodities; Most common items pre-printed, others listed

• Sales collected separately from production in list

• Government programs listed

SHORT FORM• 12 pages• Acres operated by land type

collected several pages after acres operated by ownership

• All crops and livestock collected in open write-in tables; Only a few items included as examples

• Value of sales collected with production in open tables

• Government program info collected in abbrev. tables

• Qs on production contracts and irrigated pasture omitted

Short Form, Field Crops and Hay Table

Long Form, Field Crop Section

Short and Long Forms

• Split sample used to send short and long forms to all types of operations

• Response rate for each form tracked • Content was the same, so comparisons of length were

not confounded with content• Data compared across forms• Information from cognitive

interviews compared across forms

Less pages ≠ Higher response

Form Total Mailed Receipts Response Rate % (Weighted*)

Short 12808 6870 53.24 (52.73)

Long 14860 8200 54.68

*short form response weighted for comparability to long form subgroup sizes

Long Form: page 2

Short Form: page 2

Long Form: Facing on Page 3

Short Form: 3 pages later on page 5

If you want people to report the same number twice,

don’t ask them 3 pages apart

Form % Total Land 1 missing (n)

% Total Land 2 missing (n)

% Total land 1 <> Total land 2 (n)

Short 6.3%(364)

14.6%(841)

39.61%(2180)

Long 5.1%(358)

11.4%(790)

26.54%(1779)

Problems reporting Total Acres

Quantity Unit preprintedCalculated yields = 1

Form Corn Soy

Short 11% 13%

Long 2% 2%

If you want bushels, ask for bushels

Calculated yields = 1

Form Corn Soy

Short 11% 13%

Long 2% 2%Quantity Unit must be written by respondent

Long Form

Short Form

If you want something reported, ask for it explicitly

Total reporting horses in prior census

Horses reported in prior census and in CCT

% (n)

Horses reported in prior census and 0 in CCT

% (n)

Short 1108 32.1%

(356)

67.9%

(752)

Long 1409 75.0%

(1056)

25.0%

(353)

Other problems

• Short Form: Fewer gov. payments reported• Short Form: More manual processing required

(extra sheets, uncodable responses, more than one number entered in a cell)

• Short Form: Slightly more omitted agriculture reported in cognitive interviews (9.9% vs 7.4%)

• Long Form: Self reported completion times slightly longer (44.46 vs 47.87 minutes)

Changes based on these findings

• Added additional types of operations to exclude from short form (e.g. horses, specialties, etc.)

• Use of more long form question formats on short form (e.g. government payments)

• Use of additional edit checks on short form

Conclusions• Decreasing # of pages does not increase

response rate

• More data problems with the short form, particularly underreporting

• Forms may collect the same

data, but their performance

is NOT equivalent—

Like, but different

Plans for use of Short Form

• Short Form will be used ONLY with limited set of operations (e.g. operations known to have 4 or less commodities; operations without specialty commodities; operations not previously reporting horses, goats, sheep, bees, etc)

• With over 3 million mailing

addresses, cost savings for

lighter mailing can be substantial

YES, it was the mountain Echo, Solitary, clear, profound, Answering to the shouting Cuckoo, Giving to her sound for sound!

Unsolicited reply To a babbling wanderer sent; Like her ordinary cry,

Like--but oh, how different! --W. Wordsworth