Experimental Research - An attempt by the researcher to maintain control over all factors that may affect the result of an experiment. In doing this, the researcher attempts to determine or predict what may occur.

Experimental Design - A blueprint of the procedure that enables the researcher to test his hypothesis by reaching valid conclusions about relationships between independent and dependent variables. It refers to the conceptual framework within which the experiment is conducted

Steps involved in conducting an experimental study

Identify and define the problem.

Formulate hypotheses and deduce their consequences.

Construct an experimental design that represents all the elements, conditions, and relations of the consequences.

Conduct the experiment.

Compile raw data and reduce to usable form.

Apply an appropriate test of significance.

1. Select sample of subjects.

2. Group or pair subjects.

3. Identify and control non experimental factors.

4. Select or construct, and validate instruments to measure outcomes.

5. Conduct pilot study.

6. Determine place, time, and duration of the experiment.

Essentials of Experimental Research

Manipulation of an independent variable.

An attempt is made to hold all other variables except the dependent variable constant - control.

Effect is observed of the manipulation of the independent variable on the dependent variable - observation.

Methods of Experimental Control

Physical Control

1. Gives all subjects equal exposure to the independent variable.

2. Controls non experimental variables that affect the dependent variable.

Selective Control

- Manipulate indirectly by selecting in or out variables that cannot be controlled.

Statistical Control

- Variables not conducive to physical or selective manipulation may be controlled by statistical techniques.

What is Scaling?

•Scaling is assigning numbers to indicants of the properties of objects

Types of Response Scales

• Rating Scales• Ranking Scales• Categorization

Types of Rating Scales

• Simple category• Multiple choice,• single response• Multiple choice• multiple response• Likert scale• Semantic differential

• Numerical• Multiple rating• Fixed sum• Stapel• Graphic rating

Category Scale

The category scale uses multiple items to elicit a single response. This uses the nominal scale. Example: where in northern California do you reside? North Bay, South Bay, East Bay, Peninsula, and other.

Likert Scale

• This is an interval scale and the differences in the responses between any two points on the scale remain the same.• The responses over a number of items tapping a particular concept or variable are then summated for every respondent.• Example: it is designed to examine how strongly subjects agree or disagree with statements on a 5-point scale for the work.

Semantic Differential Scale

It is used to assess respondent’s attitudes toward a particular brand, advertisement, object, or individual. This is treated as an interval scale. Several bipolar attributes are identified at the extremes of the scale such as Good-Bad, Strong-Weak, and Hot-Cold. Example: responsive-unresponsive, beautiful-ugly, and courageous-timid.

Numerical Scale

• It is similar to the semantic differential scale.• The 5-point or 7-point scale are provided.•Example: 7-point, extremely pleased-extremely displeased.

Itemized Rating Scale

A 5-point or 7-point scale with anchors, as needed, is provided for each item. When a neutral point is provided, it is a balanced rating scale, and when it is not, it is an unbalanced rating scale. Research indicates that a 5-point scale is just as good as any, and that an increase from 5 to 7 or 9 points on a rating scale does not improve the reliability of the ratings.

Example: 5 points, balanced rating scale with a neutral point. Example: 4 points, unbalanced rating scale without a neutral point.

Itemized Rating Scale (cont.)

Fixed or Constant Sum Scale

The respondents are here asked to distribute a given number of points across various items. This is in the nature of an ordinal scale. Example: In choosing a toilet soap such as fragrance, color, shape, size, and texture of lather.

Stapel Scale

This scale simultaneously measures both the direction and intensity of the attitude toward the items under study. This is does not have an absolute zero point, this is an interval scale. Example: supervisor’s abilities such as adopting modern technology, product innovation, and interpersonal skill, from +3 to –3.

Graphic Rating Scale

A graphical representation helps the respondents to indicate on this scale their answers to a particular question by placing a mark at the appropriate point on the line. This is an ordinal scale, but might appear to make it look like an interval scale.It is applied as faces scale such as feeling about their jobs.Example: rate your supervisor, excellent (10) – all right (5) – very bad (1).

Survey is process of collection of data by the researcher through oral, written or electronic medium by asking questions to the people concerned to the problem in some way.

Purposes of Surveys

Attitudes– Goals– Endorsements– Preferences– StylesBehaviors– Consumer– Education– Health/Safety– Voting

Opinions– Beliefs– Efficacies– Evaluations– Judgments– Satisfaction

Types of Surveys

Written– Provides respondents opportunities to reflect– Allows participants to compose extended responsesOral– Permits coverage of many topic areas quickly– Uses simple answers (i.e., yes/no)Electronic– Allows efficient administration– Permits effective data management– Provides access to large samples– Permits administration of large surveys with multiple sections

Key Questions To Consider When Asking Questions

Is this item understandable to the participant?

Is this item something the participant can answer?

Is this item something the participant will be willing to answer honestly?

How important is this item compared to others I have to avoid respondent burden?

Areas to Consider in Designing Your Survey

What type of survey methods are out there and what can they do for me?

What are the things to consider in maximizing the quality of data with respect to questions I ask?

How do I consider using existing scales and putting it all together?

Modes of Survey Delivery

Survey Method Options

IAQ SAQ

Including Diary Methods, Mail Surveys

CASI/CAPI (and A-CASI) CATI T-ACASI Internet

Interviewer-Administered Questionnaire (IAQ)

Most flexible and responsive to respondent re: comprehension of question, selection of response options.

Able to handle skip patterns that could confuse the respondent in a self-administered questionnaire.

Human element, rapport & responsiveness also can introduce problems in lack of standardization of delivery, self-presentation.

Self-Administered Questionnaire (SAQ)

Allows for greater privacy than face-to-face interviewing

Tends to be associated with higher rates of reporting of sensitive behaviors

Literacy requirements for respondent Cannot be adjusted to be responsive

to needs of individual participant Cannot handle complex “skip

patterns” Low-cost survey delivery option

Diary Methods Provides optimal potential for recording of

accurate rates of target behaviors Recall Exploration of variations over time

Allows for question design to examine co-occurrence of key behaviors (e.g., sexual behavior and drug use)

Burden of consistent completion of information means that data may be missing in possibly non-random pattern, or completed at a later time than designed.

NOTE: Can be useful to design for Web-based implementation.

Mail Surveys Many of considerations of SAQ are

applicable -- with format even more crucial. Many respondents report liking the

flexibility of completing surveys at their own pace.

Need for clear and easy format to maintain respondent interest and ease of completion.

Critical to have contact information and clear procedures to follow up on respondents to ensure adequate response rates.

Computer-Assisted Self-Interview /Personal Interview (CASI/CAPI)

Survey is delivered via computer (can be laptops for maximal flexibility), with respondent keying in responses.

Usual mode is “Audio-CASI” or A-CASI, which involves respondent listening to audio recording of survey as they see items on-screen.

For special PCs with touch screens, can avoid keyboard and use fingers to select choice by touching on-screen.

CASI/A-CASI Advantages Privacy Branching in complex questionnaires Automated consistency checks Automated range checks Automated adaptation of question

wording based upon prior responses Audio-CASI allows for standardized

verbal delivery of all questions and does not require respondent literacy

Computer-Assisted Telephone Interview (CATI)

List samples of “live” residential phone lines are called to screen households and determine eligibility of members.

Once consent is obtained, interviewer uses a computer to guide them through asking survey questions, providing prompts, entering data, etc. as in A-CASI.

Interviewer also has some flexibility in dealing with respondent comprehension/literacy level.

CATI Advantages Opportunity to access representative

sample (costly with eligibility requirements that are less common).

Some advantages of live interviewer, but also anonymity of phone call.

Has capacities of computer-assisted technology with respect to branching, adaptation of wording, consistency/range checks, etc.

Telephone Audio Computer-Assisted Self-Interview (T-ACASI)

First a telephone interviewer contacts a household, screens for an eligible respondent, and secures consent.

Phone call then transferred to automated system where computer-controlled, pre-recorded questions are read aloud.

Respondent provides answers by pressing keys on a touch-tone phone.

At end, respondent can be returned to live person to close out interview.

Internet Surveys Access to large sample at relatively low cost in

relatively short time frame. May provide access to difficult-to-reach

populations. As with other computer-assisted methods, may

result in higher rates of reporting of sensitive information, and provides consistency and validity checks.

Rates of internet access are growing (if still disparities exist in different populations).

Participation rates of target population hard to gauge

Dropout rates Direct data entry into database facilitates

cleaning and analysis.

Summary: Survey Delivery Considerations

Mode of survey delivery clearly influence: Respondent comfort and likelihood of self-

disclosure (due to level of privacy) Respondent ease of navigating through the

survey if there are any skip patterns Time and effort expended per respondent The audience you can reach with your

survey Quality of data, data entry and analysis

While important, how you actually ask the questions is just as critical -- our next focus.

Survey Construction

Developing the Survey: Question Organization

Topics should be generally organized from least to most intimate

Initial questions are crucial -- respondent must get “hooked” by at least some questions which are felt to be personally salient

Initial questions must also require low effort (cognitive demands) to answer

Thus, if possible, avoid demographic questions to start (especially income!)

Developing the Survey: Question Organization

A survey -- no matter how delivered -- is a type of conversation.

Topics should follow an orderly sequence, with common questions grouped together.

This flow allows transitions to more intimate matters without jarring or acute awareness.

Transitional phrases can help in shifting set. All this also provides respondent with notion

that there is a rationale to questions asked.

Question Ordering Be careful with filter questions: if not

worded carefully, respondents will learn that a “no” response means avoiding subsequent questions.

Be careful about summary attitudinal questions: more specific queries to begin will influence summary responses by forcing respondent to think about their overall beliefs more carefully.

Question Ordering Having similar response choices

reduces cognitive burden on respondent; grouping items with such options together is thus useful. It can also be useful to have the same number of options to reduce complexity (e.g., always using a four-point or five-point scale).

However, arrange types of questions to provide some variety and reduce response set.

Avoid asking unnecessary questions!!

Question Wording/Construction

Comprehension/Interpretation

Not just question of literacy, but influenced by cognitive burden and attention required

Wording choices Length and complexity of survey items Ordering of clauses Consistency of phrasing Avoid “not’s” or items whose meaning

will be lost by inattention to a single word Phrasing also influences response

distribution along a Likert-type agree-disagree scale

Cultural Considerations Individuals can come up with very

different understandings of survey items, based upon their personal experiences.

The meanings that respondents bring to the survey experience can be strongly influenced by culture, acculturation and other background variables -- not just education and economic status.

These differences are highlighted in the vast literature that exists on different response patterns of different ethnic groups to the same standardized scales.

Key Considerations

Examine all survey items with eye for their appropriateness for target population.

Be aware of pre-existing measures so as to not have to reinvent the wheel.

When using pre-existing scales, review in same way for clarity & ease of response.

Get feedback on item wording and construction from as many colleagues as possible.

Pre-test survey, using methods first described by Cannell et al. (1989).

Cognitive Burden & Recall Demand

Recall accuracy is influenced by a variety of factors, including: salience/vividness of behavior, time frame, complexity of response information

necessary (ever, counts, close-ended categories, etc.),

frequency and consistency of behavior, effort respondent is willing to exert, cueing techniques and prompts

Time Frame - Sexual/Drug Use Behaviors

3 month time frame generally considered the outside range of accurate reporting; some suggest limiting to 1 month.

Longer time frame may be possible if asking if a behavior ever happened.

One important timeframe consideration is dealing with infrequent behaviors -- what do you want to capture with respect to either behavior events or possible behavior change (if longitudinal study)?

Sensitivity Item sensitivity not consistent across

populations (e.g., sexual behavior among MSM vs. general population, drug use behavior among heavy drug abusers/IDUs, discussing sexuality among different ethnic groups).

Rationale for sensitive questions helps response rates.

Asking if “ever” occurred first, prior to asking about (for example) the last month, may be less charged for respondent.

Importance of neutral wording of question and (if IAQ) careful training of interviewer

In Closing ... Selecting your methods of data collection must

be carefully considered based upon target population, study questions and design, and costs.

No matter what the method of data collection, your instrument is critical.

Survey design is a combination of art and science, with the “art” strongly reliant upon interviewing skills.

The science of survey design currently has a wealth of research literature to examine and guide you.