Psychological Research Methods

Jamie DeCoster

Department of PsychologyUniversity of Alabama

348 Gordon Palmer HallBox 870348

Tuscaloosa, AL 35487-0348

Phone: (205) 348-4431Fax: (205) 348-8648

May 9, 2001

These notes were prepared with the support of a grant from the Dutch Science Foundation. I would liketo thank Reint Jan Renes, Mirjam Tazelaar, and Jan-Willem van Prooijen for comments made on earlierversions of these notes. If you wish to cite the contents of this document, the APA reference for them wouldbe

DeCoster, J. (2001). Psychological Research Methods. Retrieved <month, day, and year you downloadedthis file> from http://www.stat-help.com/notes.html

For future versions of these notes or help with data analysis visithttp://www.stat-help.com

ALL RIGHTS TO THIS DOCUMENT ARE RESERVED.

Contents

1 Developing a research question 1

2 Reviewing the Literature 4

3 Designing your Study 7

4 Validity 13

5 Preparing Study Materials 20

6 Conducting Research 24

7 Data analysis 29

8 Writing a Research Paper 32

i

Chapter 1

Developing a research question

The first thing that researchers must do when conducting research is to determine the research question thatthey want to investigate.

1.1 Sources of potential questions

• There are many ways to come up with a research question. Possibly the most common source ofresearch questions involves examining a theory. Here we are defining a “theory” as a set of relatedprinciples that are designed to explain a broad set of real-life events. Not all researchers try to developtheories, and not all research is theory-driven. However, many do use theories to guide the researchquestions they choose to investigate.

Research questions can be derived either from attempts to promote or to dispute a theory. If you wishto promote a theory the first thing you can do is question whether the predictions of the theory aretrue. If you are working from a pre-existing theory, however, this was usually done by the originalresearchers. In this case you can question whether the principles of the theory can be applied to new,previously unconsidered phenomena. You could also develop questions about why the theory works. Ifthe theory is described at a certain level (such as reading comprehension) you might think about whatlower-level process (such as perception and word recognition) might be driving the theory.

To dispute a theory you might question the validity of existing findings that support the theory. Ifyou believe that a study suffered from methodological or statistical errors you might question what acorrect version of the study might find. If the existing support appears valid you could also questionwhether there are factors that might limit where the theory applies. You can consider whether thetheory is only successful in explaining phenomena under particular circumstances or with particularpopulations. Finally, you might question whether there are external factors that must be present orabsent for the theory to accurately predict behavior.

• A research question sometimes suggests itself while you are reading about or listening to researchthat someone else has conducted. When you are reading articles you should write any thoughts youhave at the time in the margins (assuming you aren’t using a library copy!). These ideas can beexcellent sources of research questions. Similarly, you should also consider writing down ideas that youhave whenever you hear someone giving a research presentation. People often take notes on what thespeakers say but fail to write down ideas of their own that are provoked by the talk.

While it is certainly possible to wait until you finish an article or listen to the complete talk beforerecording your ideas, this is generally not as effective as writing them out at the time you have them.First off, when you wait you run the risk of forgetting your ideas. This is quite possible since beforeyou finish you will be reading or hearing about many other ideas on the same topic which mightinterfere with your memory. Second, often when people have flashes of insight their ideas are not fullyformulated in linguistic terms. You will be better able to find the right way to express your questionif you try to do so immediately than if you have to try to work from your memory.

1

• You can get ideas for research questions from arguments and discussions with your teachers, students,and colleagues. This can be a more fruitful source than either articles or presentations since as adiscussant you must actively think about the topic in order to come up with reasonable contributionsto the conversation. Your partner can also be valuable in helping you evaluate how important orinteresting your question might be.

Just like with articles and presentations, you are probably best off if you write down potential researchquestions as soon as you think about them, even if this means disrupting the flow of the conversation.While this may seem to be rude, you will be surprised at how many people feel complimented whenyou indicate that the discussion was meaningful enough for you to want to write down some notes.

• The final major source of research questions is through observation of things around you in everydaylife. You might see or hear about an unusual phenomenon that you think might be worth investigating.Or else you may simply want to learn more about a common aspect of life. When generating researchquestions about life events you can use the five “W’s” of journalism: Who is involved in the event?What occurs during the event? Where does the event occur? When does the event occur? Why doesthe event occur?

1.2 Evaluating a research question

An important thing to keep in mind is that not all research questions are equal. Some topics by their verynature will never lead to important or interesting research. Here are some things you can ask to help youevaluate a potential research question.

• To what extent has the question already been examined? There is very little to be gained by investigatinga question whose answer is already known. After deciding on a preliminary research question you shouldsearch the literature to see what studies have already examined the question. Studies that investigateminor variations of a well-documented phenomenon are typically less valuable than those that helpopen up the way to new avenues of research

• Is the question of social or scientific importance? You need to consider exactly what will be gainedby finding the answer to your research question. Think about both the applications of the research toeveryday life as well as the implications it might have for future research on similar topics. While astudy does not necessarily need to have direct real-world applications to be meaningful, their presencedoes increase the potential value of the research. Similarly, a study can be atheoretical and relativelyindependent of the rest of the literature and still be important. However, research that speaks tocontroversial theoretical issues in the field will likely have a larger overall impact than research thatdoes not.

• Is the question appropriate to the researcher? Just because you develop a novel research question withsocial and scientific importance doesn’t mean that you should feel compelled to investigate it. Notall research questions are right for all researchers. First and foremost you must decide whether youactually have an interest in investigating the question. The work of researching a topic will be muchmore enjoyable if you can maintain your interest in it throughout the project. Additionally, people areable to write much better and much more convincingly about a topic when it is personally importantto them.

You should also consider whether you have sufficient background knowledge to be able to design a studyto examine the question. If you do not, you should either commit yourself to gaining this knowledgeor else choose a new topic. Unless you know the literature you will not be able to judge whether yourquestion has already been answered. You will be unfamiliar with the standard research methods usedin the field, possibly requiring you to spend a long time developing your methodology before you seeinterpretable results. Finally, you will find writing your introduction and discussion sections to be verydifficult since you don’t really know the research to which you should relate to your study.

Finally, you should decide if the question is suitable for the general type of project that you had inmind. Questions that are appropriate for a full-length journal article could never be investigated in

2

time for a class assignment. Similarly, simple questions that can be answered with a single study areoften too limited in scope to be accepted by journal editors. You should give serious thought to theamount of effort that would be needed to sufficiently answer the question. If it requires more than youare able to give then you should either refine the scope of the question or choose a new topic.

3

Chapter 2

Reviewing the Literature

Once you have a basic idea of what it is you want to investigate you should see what other researchers havedone on related topics. This will make it easier for you to decide on the specific details of your own study.You will also need this information when writing the Introduction and Discussion sections of your paper.

2.1 Searching the literature

• Computerized Indices. A number of databases are available on CD-ROM or over the internet. Thesewill allow you to use keywords, and Boolean combinations of keywords, to locate articles relevant toyour analysis.

◦ Selecting the keywords for your search is very important. First, you should determine the basicstructure of what you want in your search (such as PRIMING and IMPRESSION FORMATION).You should then determine the synonyms that would be used for these terms in the database. Ifthe database assigns subject words to each study, you should make sure that your list of synonymsincludes all the relevant subject words. Many databases publish a thesaurus to make this easier.For PsycInfo the name of this thesaurus is “Thesaurus of Psychological Index Terms” (APA,2001).

◦ Most databases support the use of wildcards, which you should use liberally. To locate researchon priming in PsycInfo we might use the search term PRIM*, which would find studies that usethe terms PRIMING, PRIMES, PRIMED, and other words beginning with PRIM.

◦ You should then enter your search into the database. Each construct will be represented by a listof synonyms connected by ORs. The constructs themselves will be connected by ANDs or NOTs.In the example above we might try: (PRIM* or IMPLICIT MEMORY) and (IMPRESSION orJUDGMENT).

◦ If your initial search produces a large number of irrelevant studies related to a single topic, youmight try to keep them out of further searches by introducing a NOT term to your search.

◦ Keep in mind that different indices use different terms, so you may have to define your searchdifferently when working with different databases.

◦ Whenever you conduct a computerized search you should record the name of the database, theyears covered by the database at the time of the search, and the search terms you used. This canhelp prevent redundant searches if you decide to look through the literature again at a later pointin time.

◦ The databases most commonly used by psychologists are:

1. PsycLit/PsycInfo (PsycLit is the CD-ROM version, and is less complete)2. ERIC (Educational Resources Information Center)3. Dissertation Abstracts Online4. ABI/Inform (a worldwide business management and finance database)

4

5. Sociological Abstracts (sociology literature)6. MEDLINE (biomedical literature including health care, clinical psychology, gerontology, etc.)7. Mental Health Abstracts

There are also a number of databases available within more specialized research areas.

• Descendant search. If you can locate an important study that was performed at an early date you canuse the SSCI (Social Science Citation Index) or SCI (Science Citation Index) to locate later articlesthat cite them in their references. This is a very nice complement to the standard computerized search,and can now be performed fairly easily since both indices are available on CD-ROM.

• Ancestor search. You should check the references of papers you already have to see if they contain anyrelevant studies that you are unaware of.

• Research registers. Research registers are actively maintained lists of studies centered around a commontheme. Currently there are very few research registers available for psychological research, but thismay change with the spread of technology.

• Reference lists of review articles. Review articles are often a fruitful place to look for relevant studies.

• Hand search of important journals. If there is a journal that is particularly relevant to your topic thenyou might go back and read through the table of contents of that journal for the years that there wasactive research on your topic. You might make use of Current Contents, a journal containing the tablesof contents of other journals.

• Programs from professional meetings. This is a particularly good way to locate unpublished articles,since papers presented at conferences are typically subject to a less restrictive review (and are there-fore less biased towards significant findings) than journal articles. Probably the two most importantconferences in psychology are the annual meetings of the American Psychological Association (APA)and the American Psychological Society (APS).

• Letters to active researchers. If there is a particular person whose research is highly relevant you maydirectly ask them for recommended reading. When trying to locate people you may want to make useof:

◦ Academic department offices/Department web pages

◦ Alumni offices (to track down the authors of dissertations)

◦ Internet search engines (such as http://www.switchboard.com or http://people.yahoo.com)

◦ APA, APS membership guides

2.2 Summarizing articles

• It can often be useful to compose short summaries of important articles when you need to write aliterature review. This allow you to develop your interpretation of the article soon after you read itinstead of weeks or months later when you write your review. Summaries are also useful when youwant to go back and refresh your knowledge of the literature at a later point in time.

• Ideally a summary should include the most important aspects of an article in a form that can be read ina relatively short period of time. Summaries that are too short may miss important points and providean inaccurate view of the article. Summaries that are too long will include unimportant informationmaking it more difficult to extract the main points of the article when later reviewing the summary.

• An important step is to determine what articles you should summarize. This will depend in part onthe reason you are conducting the literature review. If you are writing an actual review paper then youwill likely want to summarize each article you wish to include in the paper. The purpose of the reviewin this case is to provide a comprehensive description of studies in the literature. To help ensure thatyour review is unbiased you should equivalently examine all the relevant articles.

5

When you are reviewing the literature as part of the introduction to an empirical article, on the otherhand, you will typically only want to summarize a selection of the relevant studies. The purpose ofthe review in this case is to provide a context in which your study appears important and justified.Your review is not intended to be comprehensive. Instead it is designed to illustrate the reason thatyou chose your topic of research. You are therefore justified in concentrating your efforts on articlesyou see as more important.

• A well-written summary of an empirical article should take up less than one single-spaced typewrittenpage. Summaries of review articles may be somewhat longer depending on the size of the literatureand the way that it was reviewed.

• There are five aspects of an empirical article that should always be documented in a summary

1. APA reference for the article. This should appear at the top and should be visually separatefrom the rest of the summary. You can accomplish this by putting a box around the reference, orsimply by including blank lines between the reference and the remainder of the summary.

2. Research question examined by the article. You should state the main topic the article investigates.You should also include a short description of any theories that are important to the studies. Yougenerally will not want to record arguments for the importance of the research question.

3. Abstract descriptions of studies within the article. You want to provide a basic idea of the exper-iments and surveys conducted as part of the article. Provide specific details only to the extentthat they are important to understanding the results of the study.

4. Major findings. Report the important results of each study in the article. You will generally notrecord the specific statistics. Instead describe the overall pattern of results.

5. Conclusions. You should describe both the specific implications drawn from the results of eachstudy as well as any general conclusions made from the article as a whole. Be sure to report theimplications the article may have for theories in the literature.

• There isn’t a specific procedure for summarizing the contents of a review article since their organizationsvary greatly. However it is probably best to order the contents of your summary in the same way asthe original article.

• In addition to summarizing the contents of an article you may also want to write down how the articlerelates to your own research project.

• You should try to use complete sentences when writing your summaries. First, the additional thoughtrequired to write the information down in a comprehensible form may give you some greater insightsinto the article. Second, you will make writing your literature review easier since you will be able todraw more directly from the contents of your summaries.

6

Chapter 3

Designing your Study

3.1 Generating research hypotheses

• Once you have found a satisfactory research question the next step in conducting research is to comeup with one or more hypotheses that will help answer the question. A hypothesis is simply an unproventheoretical prediction that answers some aspect of your research question.

• A given hypothesis does not necessarily need to provide a definitive answer to the question you wantto investigate. It simply needs to explore and hopefully answer some aspect of your question. A singleresearch question can be the source of many different hypotheses.

• A hypothesis should be falsifiable. That is, there should be some form of evidence that would indicatethat the hypothesis is incorrect. If a hypothesis is able to accommodate any pattern of data thenfinding evidence for it won’t really tell us anything about the phenomenon of interest.

• Hypotheses typically involve statements about the relationships between two or more variables. Bydefinition, a variable is some feature that can be categorized or measured with a number. A variabletakes on different values to reflect the specific characteristics of a given observation. Variables can referto features of the environment, such as the presence or absence of noise, or they can refer to featuresof an individual, such as the time it takes someone to respond to a question.

3.2 Relationships between variables

• There are two different types of relationships that can exist between a single pair of variables. Ahypothesis can propose a causal relationship indicating that changes in one variable are directly re-sponsible for changes in a second variable. For example, researchers have proposed that attractivesources of information are more persuasive than unattractive sources. This hypothesis would statethat if you keep everything else constant, just changing the attractiveness of a person presenting anargument will change the likelihood that a listener will accept the argument.

A hypothesis might also propose a correlational relationship, simply stating that you can predict onevariable from another without making any claims about causality. For example, researchers haveproposed that how much a child reads is associated with their intelligence. This hypothesis makesno claims about what causes the relationship, but does propose that knowing the value of one of thevariables will allow accurate predictions of the other.

• When discussing a causal relationship researchers call the variable causing changes the predictor vari-able. The variable being affected by the change is called the response variable. These terms are alsooccasionally applied to variables with a correlational relationship if there is a justification for sayingthat one particular variable is being used to predict the second. Some people refer to predictor variablesas independent variables and response variables as dependent variables.

7

• If you have at least three variables you have some additional options for your hypotheses. You couldpropose that one variable mediates a causal relationship between two other variables. This would meanthat a predictor variable causes changes in a response variable, but only through changes in a thirdmediator variable.

The idea here is that the predictor variable has a causal relationship to the mediator which in turn hasa causal relationship to the response variable. It is also possible to have partial mediation where thepredictor has a direct influence on the response variable in addition to the mediated relationship.

For more information about how to demonstrate a mediated relationship see Baron and Kenny (1986).

• A hypothesis could also propose that one variable moderates the relationship between two other vari-ables. This means that the exact relationship between two variables depends on the value of a thirdvariable. This is exactly the same thing as having an interaction between predictors in regression orANOVA. Both correlational and causal relationships can be moderated by third variables.

The idea here would be that you can successfully predict the response only if you have the values ofboth predictors. Knowing the value of either predictor variable itself is not sufficient to predict theresponse because the relationship between the first predictor and the response depends on the value ofthe second predictor, and the relationship between the second predictor and the response depends onthe value of the first predictor.

3.3 Evaluating hypotheses

Here are some questions that you can ask to help you evaluate potential hypotheses.

• Is the hypothesis novel? You generally do not want to test the same hypothesis as another researcherunless you have reason to suspect errors in their study. Exact replications are not greatly valued bythe scientific community.

• How general is the hypothesis? Hypotheses that are framed in more abstract terms can be appliedto a wider variety of situations, increasing their potential impact.

• Does the hypothesis concern important variables? People pay greater attention to research thatthey feel can have a personal influence on them. Consider how often people talk about the variables inthe hypothesis in everyday life (social importance) and the extent to which other researchers use thosevariables in their own studies (scientific importance).

• Does the hypothesis propose anything that is unusual or unexpected? Others will have moreinterest in findings that defy common sense or provide a new way of thinking about a phenomenon.

• How likely is it that the hypothesis is correct? There is always some element of uncertaintysurrounding a hypothesis. Tests of incorrect hypotheses provide much less interesting information thantests of hypotheses that turn out to be correct.

However, there is usually a direct tradeoff between how unexpected a hypothesis is and its likelihood ofbeing correct. The more unusual (and therefore the more interesting) your hypothesis is the less likelyit will work out the way that you want. How much risk a hypothesis should have is a matter of personalchoice. High-risk research tends to provide more interesting findings, but low-risk research is generallymore likely to provide publishable findings. In the long term it is probably best for researchers topursue a mixture of high-risk and low-risk projects.

3.4 Specifying variables for a study

• After proposing a hypothesis researchers typically must provide some type of evidence to support theirclaims. While at times this may be strictly through a rhetorical argument, more commonly researchersconduct a study to provide empirical evidence for their idea.

8

• The first step in designing a study is to specify what variables you will record and how their valueswill be obtained. This information is called the design of your study. The design of your study willalways be strongly tied to the hypotheses that you want to examine.

• You will want to include the following types of variables in your design.

◦ Variables that are explicitly discussed in your hypotheses.

◦ Variables that you believe could potentially influence the relationships proposed in your hypotheses(such as moderators or mediators).

◦ Variables that you believe may strongly impact on the response measures you collect, even if thisinfluence is completely independent of the relationships proposed in your hypotheses. These arecalled covariates or blocking variables. By recording these variables you can statistically removetheir influence from your analyses providing you with clearer results.

◦ Variables that you believe may be related to those in your hypotheses but for which you don’tpredict any specific relationships. These are called exploratory variables.

• Once you have decided on the variables you will include you must then determine what type of valuesthe variable will record. Each variable can have either categorical values, used to represent groupdistinctions or numeric values, used to represent features that vary in measurable quantities.

It is possible to use a categorical variable to hold information about something that is really numeric.For example, you could use a categorical variable to code a person’s age. Values of the variable wouldrepresent age groups rather than a person’s specific age.

• You additionally need to decide the way you will obtain the values for each variable. While there willbe many differences in the specific procedures you use to collect each variable, there is one distinctionthat is particularly important when determining your study design. Specifically, a variable can eitherbe manipulated, where their values are forced by the experimenter, or measured, where their values aresimply observed and recorded during the study.

3.5 Testing causal hypotheses

• Merely observing that there is a relationship between a pair of variables X and Y cannot be takenas evidence that changes in X cause changes Y . There are two other types of relationships that canproduce this same result.

1. Changes in Y cause changes in X.

2. Changes in both X and Y are caused by a third variable Z.

• To infer a causal relationship between a predictor variable and a response variable you must show thefollowing.

1. There is a relationship between the predictor and the response.

2. Changes in the predictor precede changes in the response.

3. There are no other possible causes for changes in the response other than changes in predictor.

• If you wish to test a causal hypothesis your predictor variables can be either manipulated or measured.However it can be very difficult to rule out alternative explanations for changes in your response variableif your predictor variable is measured. There is always the possibility that your predictor variable doesnot actually cause changes in your response variable but instead is simply correlated with the actualfactor responsible for the changes.

• The easiest way to demonstrate that a predictor variable causes changes in a response variable is toperform a study that manipulates the predictor variable, measures the response variable, and keepsall other variables constant. If you can show a relationship between the manipulated predictor and

9

the response variable then you know that there is a causal relationship. You manually changed thepredictor so you know that its changes preceded changes in the response. You also know that thereare no alternative explanations because you held all of the other factors constant. All three of therequirements to infer causality have therefore been met.

• Any study that manipulates at least one variable is called an experiment. A study that only measuresvariables is called a survey. Experiments are primarily used to demonstrate causal relationships whilesurveys are primarily used to demonstrate correlational relationships.

• The predictor variables in an experiment (except for measured numeric variables) are also sometimesreferred to as the factors. The values that a factor may take are often called its levels. The particularcombination of factor levels that a participant is in is called their condition. Measured numeric predictorvariables aren’t generally considered factors because they don’t divide your participants into groups.Rather each subject has their own specific measured value.

• The response measures in an experiment must always be measured and should be collected in everycondition of the study.

• If you decide that you want to manipulate a variable in an experiment there are two basic ways todo it. You can choose to manipulate your variable within subjects or you can manipulate it betweensubjects.

◦ In a within-subject manipulation each participant experiences every value of your manipulatedvariable. Here you will observe the same group of people for each level of your manipulation.

◦ In a between-subject manipulation each participant experiences only a single value of your manipu-lated variable. Here you will observe different groups of people for each level of your manipulation.

There are alternatives to strict within- or between-subject designs, such that each participant experi-ences more than one level of your variable, but not all of them. However, these designs are best avoidedbecause they can be extremely difficult to analyze. When designing an experiment you should try tomanipulate each factor in either a within-subject or a between-subject fashion. There are no problemswith including both within-subjects and between-subjects manipulations in the same experiment, aslong as each variable is either one or the other.

• Next you need to specify exactly what conditions you want to include in the experiment. If you onlyhave a single factor then you should have a condition for each level of the factor. If you want to usemore than one factor in an experiment the best way to determine your conditions is with a factorialdesign. In a factorial design you have a condition for each possible combination of the factors. Forexample, assume that you wanted to conduct an experiment examining the influence of both externalreward (high or low) and the length of a task (short or long) on task performance. To create a factorialcombination you would want to measure task performance under the conditions of high reward/shorttask, high reward/long task, low reward/short task, and low reward/long task.

There are other ways you can select conditions from a combination of factors but under most circum-stances a factorial design is best. It includes the most information about your variables and guaranteesthat you will be able to distinguish the influence of one of your variables from all the others. It alsoallows you test for any interactions between your factors.

• An important requirement to infer causality from the results of an experiment is that all variables notmeasured or manipulated in the experiment be held constant. If there are other variables that take ondifferent values during the experiment then you can’t be sure that the observed changes in the responsevariable are actually caused by your predictor variables.

One of the major sources of influential external variables are differences between the types of par-ticipants in your between-subjects conditions. If there are differences between the people in yourbetween-subject groups (other than differences directly related to the between-subject factor) thenthis might produce an alternative explanation for your findings. This would prevent you from beingable to conclude that there is a causal relationship between your predictor and response variables.

There are two ways that you can try to make your groups similar.

10

1. You can try to equate the participants in your between-subjects conditions. To do this you needto identify the external variables that you think would be most likely to affect your measuredvariables. You then try to assign them to your groups so that each has approximately the samedistribution of each external variable you want to control. This procedure is sometimes calledmatching your between-subjects conditions.

2. You can randomly assign participants to each of your between-subjects conditions. On averagethe influences of external factors within each group should cancel themselves out. That is, therewill be positive and negative influences on your measurements within each group. As long as theseinfluences are truly random your inferential statistics will be able to take them into account.

Although it may be counterintuitive, it is much better to have randomly assigned groups than to havematched groups. You may be able to make matched groups more similar than randomly assignedgroups. However, the variability of matched groups is more likely to contain systematic differencesbecause you are manually controlling the assignment. It is much better to have random than systematicdifferences because tests of statistical inference can fully account for the influence of random differences.Systematic differences, on the other hand, go undetected by statistics and can potentially introduce analternative explanation for your results. Even though you can chose a whole list of variables to controlfor, there is always the possibility that there is something that you failed to consider when creatingyour matched groups. Random assignment, on the other hand, balances all potential influences, eventhose of which you may be unaware.

3.6 Specifying the design of a study

• Information about what variables are recorded in your study and how their values are obtained iscalled the design of the study. You typically report this information in the “Design” section of a paper(which is a subsection of the “Method” section).

• Before attempting to write your design section you need to know the following information.

◦ The name of each variable examined in your study.

◦ Whether each variable is categorical or numeric.

◦ Whether you consider each variable to be a predictor, a response, or neither.

◦ Whether each variable is manipulated or measured.

◦ For experiments, whether each factor varies between subjects or within subjects.

◦ For experiments, how you chose the conditions in the study. Most commonly this will be throughthe use of a factorial design.

◦ How you assigned subjects to the different levels of any between-subjects manipulations.

• Once you have all this information you are ready to describe your design. If you conducted a survey(and therefore have no manipulated variables) you can write your design section using the followingprocedure.

1. Start with “The survey measured”.

2. List the information about each variable in the study.

◦ If the variable is categorical, write the name of the variable followed by its possible values inparentheses.

◦ If the variable is numeric, write the name of the variable followed by the unit of measurementin parentheses. If the variable is measured using a scale you should report the name of thescale (if it is well-known) or simply say “reported on a scale”. Sometimes people decide notto report the units of measurement, especially if there are many variables that were measuredusing relatively unknown scales.

11

Here is an example of how you might report the design of a survey. “The survey measured eachparticipant’s occupation (unemployed, student, laborer, or professional), age (in years), weight (in kg),mood (reported on a scale), and depression level (measured using Beck’s Depression Inventory).”

People writing journal articles often omit the design section for surveys since the same information istypically presented in the procedure section.

• If you conduct an experiment (with one or more manipulated variables) you can write your designsection using the following procedure.

1. Start with “The experiment was conducted using a”.

2. State how you chose the conditions of your study.

◦ If you have only one manipulated variable and no other categorical predictors you write “Ngroup”, where N is the number of levels in your manipulated variable, followed by the nameof the manipulated variable in parentheses. You then write “between-subjects design” or“within-subjects design,” depending on how manipulated the variable. For example, “a 4group (time pressure) between-subjects design.”

◦ If you have two or more factors in a factorial combination you list out the number of levelspossessed by each factor followed by the factor name in parentheses, putting “x”s between thedifferent factors. After listing all of the factors you put “between-subjects factorial design”if all the factors varied between subjects, “within-subjects factorial design” if all the factorsvaried within subjects, or “mixed factorial design” if you have both within- and between-subjects factors. For example, “a 2 (sex) x 2 (task) x 3 (reward condition) between-subjectsfactorial design” or “a 3 (diet) x 2 (health status) mixed factorial design.”

◦ If you have used some other method of combining your factors you must fully describe thathere.

3. Report how participants were assigned to the levels of between-subjects factors. If you matchedon any variables you would report that here. For example “Participants were randomly assignedto the levels of all between-subjects manipulations” or “Between-subject groups were created bymatching participants on hair color, eye color, and fashion sense.”

4. Next you have a sentence for each factor describing the different levels that it can have (unlessthey are obvious, like for gender). For example, “the two levels of task indicated whether theparticipant performed an intellective task or a decision-making task.”

5. If your experiment includes any numeric predictor variables you should list their names followedby the units in which they were measured in parentheses. For example, “The experiment alsomeasured each participant’s intelligence (using a standard IQ scale).”

6. Finally you report each of your response variables.

◦ If a variable is categorical you state the name of the variable followed by the values it canhave in parentheses (unless the values are obvious).

◦ If the variable is numeric you state the name of the variable followed by the units in whichit was measured. If the variable is measured using a scale you should report the name of thescale (if it is well-known) or simply say “reported on a scale”. Sometimes people decide notto report the units of measurement, especially if there are many variables that were measuredusing relatively unknown scales.

For example, “The response variables of interest were the length of their response (in words) andthe content of their response (past oriented or future oriented).”

12

Chapter 4

Validity

4.1 Overview

• When designing a study you must always keep the validity of your study in the back of your mind.Validity concerns whether the conclusions that you want to draw from your study are appropriate ornot. Aspects of your study that might potentially make your study invalid are called threats to validity.Issues of validity are of concern to both experiments and surveys.

• Validity is specifically tied to the conclusions being drawn from a study. For any given study there area set of valid conclusions and a set of invalid conclusions that may be drawn. The important questionis whether the design of the study allows the conclusions stated by the researcher. Therefore, beforedesigning a study you should explicitly decide what conclusions you wish to make so you can spotpotential threats to the validity of your study.

• Drawing on Cook and Campbell’s (1979) categorization we are going to consider four types of validity:construct validity, internal validity, statistical conclusion validity, and external validity. However,these “types” just provide a useful way to think about the different ways that a study could be valid orinvalid. You might discover a threat to validity that doesn’t fit well into the category structure. Theimportant thing is to consider whether the method of the study actually allows the stated conclusionsto be drawn.

4.2 Construct validity

• Construct validity concerns the extent to which the variables in your experiment accurately and ex-clusively represent the corresponding theoretical constructs.

• When assessing the construct validity of a variable you should consider both convergent validity anddivergent validity. A variable with convergent validity shows relationships with other variables thatshould be related to the theoretical construct of interest. A variable with divergent validity has norelationships with other variables that are theoretically unrelated to the construct of interest.

• Manipulations lack construct validity either when they fail to alter the theoretical construct of interestor when outside variables are inadvertently changed along with the construct of interest.

• Measurements lack construct validity either when changes in the construct of interest fail to causechanges in the measure or when changes in outside variables can cause changes in the measure whenthe construct of interest is held constant.

• Here are some questions that you can ask about a study to assess its construct validity.

◦ Do the manipulations applied in the study actually change what they are supposed to?

◦ Is anything influenced by manipulations in the study other than the construct of interest?

13

◦ Do the measures in the study accurately reflect what they are designed to measure?

◦ Can different responses on measures be caused by anything else other than changes in the constructof interest?

• Below are descriptions of some specific threats to construct validity.

◦ Mono-method bias. Sometimes a study (or an entire field of research) consistently manipulatesa variable in a single fashion, or uses a single way to measure a variable. Any single methodis necessarily going to contain idiosyncratic elements that are not truly part of the theoreticalconstruct of interest. When you use a single method to examine a variable you cannot separatethe theoretical construct of interest from irrelevant characteristics of the method. It is thereforebest to demonstrate relationships using multiple ways of manipulating or measuring the constructsof interest.

◦ Hypothesis guessing. People have a natural tendency to try to figure out what is going on aroundthem. When placed in an study they will try to make some guesses as to what it is about. Theiractions will sometimes be influenced by these beliefs, causing them to behave in a way that isunnatural. Sometimes this simply contributes random noise to your data, but other times (suchas when people are able to accurately guess your hypotheses) this can systematically bias yourresults. It is therefore important to keep your hypotheses hidden from your participants, and totry to reduce hypothesis-guessing tendencies in general. One way of accomplishing the latter isto provide reasons (though not necessarily accurate ones) for each task that participants perform.People are less likely to try to come up with their own explanations if you give one to them first.It can also be useful to ask participants afterwards what they thought the study was about. Ifpeople seem to be able to regularly guess the purpose of your study then you know you may needto worry about hypothesis-guessing biases.

◦ Administrator expectations. Studies have shown that the beliefs of research administrators canhave profound influences on the data they collect, even when they are making supposedly objectivemeasurements (Rosenthal, 1964). The responses then reflect the behavior of the researchersinstead of the behavior of the participants. The easiest way to prevent this problem is to minimizethe role of the administrator in the study. Try to use standardized methods of manipulating andmeasuring your variables. If your study requires that the administrator be more deeply involvedthen you should try to use administrators who are unaware of the research hypotheses.

◦ Evaluation apprehension. People are generally concerned whenever they are being evaluated bysomeone they perceive to be an authority. They will therefore do their best to present themselvesin a way that makes them appear competent and psychologically healthy. You must considerwhether your manipulations are inducing a desire in your participants to be perceived favorablyin addition to (or instead of) causing changes in the construct of interest. You should also examineyour response items to make sure that they are not simply reflecting participants’ attempts tocreate favorable impressions.

◦ Placebo/Hawthorne effects. The mere presence of an experimental environment can sometimescause changes in participants. It is well known, for example, that the mere expectation thatthey are receiving medical care can sometimes cause improvement in people’s health, even if theyactually receive no medical treatment. This can obscure the effect of any manipulations you wishto perform. When designing your study you should include “control groups” of people who areplaced in similar circumstances. This way the effect of the experimental environment is removedfrom any statistical comparisons you make.

◦ Order effects. Sometimes one part of your study can influence the way that people perform inanother part. For example, if you have people perform a difficult arithmetic task it might makethem feel dejected, causing them to put less effort into their responses on later tasks. You shouldtherefore try to put important tasks, as well as tasks that might be sensitive to order effects, earlyin your study.

14

4.3 Internal validity

• Internal validity concerns the plausibility of alternative causal explanations for the results of a study.If a study is not attempting to demonstrate any causal relationships then internal validity is not aconcern.

• The first thing that you need to consider when a study proposes a causal relationship is whether thedirection of causation is correct. This will not be an issue if the predictor is manipulated but may bedifficult to demonstrate when both the predictor and the response variable are measured.

• Even when you manipulate your predictor variable you must be concerned with potential confoundingvariables. A confounding variable is an outside variable (one not examined in your experiment) thatcovaries with your predictors and could potentially influence your response variable. If a predictorcovaries with a confounding variable then you cannot conclude that the predictor causes the response.The confound always provides a possible alternative explanation for your results.

• While people assessing internal validity are typically interested in finding confounding variables thatmight have produced the observed relationships between variables, it is also possible for a confoundingvariable to obscure a relationship between two variables. For example, people who know that they arein a more difficult level of a factor may try to work harder than those in an easier level. This confoundwould reduce the difference between these groups.

• Most threats to internal validity can be avoided if you randomly assign participants to the differentbetween-subject conditions in your study. On average every outside variable should be evenly dis-tributed throughout the different conditions of your study, removing most potential threats to validity.To properly randomize the assignment of participants to conditions you must choose a method thatmakes each participant equally likely to be in each level of your between-subject variable.

• It is important that you use the same recruitment procedures for participants in every condition of anexperiment. If the different levels of a between-subject manipulation have different requirements thenparticipants recruited for the experiment should all meet the requirements of every condition. Similarly,if different conditions take different amounts of time then all participants should be recruited expectingthat they may be in the longest condition. People willing to be in a longer experiment may be differentfrom people who are not.

• It is better if you assign the level of your between-subject conditions on a participant-by-participantbasis than to entire groups of participants. For example, it is a bad idea to for you to assign the firstlevel of a variable to the first 10 participants and the second level to the next 10 participants. Thisconfounds your manipulation with the order of recruitment, and possibly with the time of day thestudy was conducted. Sometimes you must run people in groups, such as when the manipulation is afilm clip seen by participants. If you have multiple participants in each session of your experiment itmay not be practical to try to show them each a movie individually. In this case you should randomlydetermine which groups are exposed to each level of your between-subjects variable. Having multiplesessions for each level (so the conditions are at least somewhat dispersed) can also reduce the potentialfor a confound.

• There are some threats that neither matching nor randomization will be able to prevent. Specifically,if a confound is in some way created by your manipulation then randomization will not help. Forexample, a manipulation that withholds rewards from some participants may inspire resentment, apotential alternative explanation for your results. Additionally, if the influence of an outside factorwould be different for different levels of a predictor (i.e., the confound interacts with the predictorvariable) its influence will also not be removed by randomization. For example, consider a study tryingto determine whether taking school trips to a museum influences school performance. One potentialconfound is the SES of the student, since wealthier students might be more likely to go to museums ontheir own. This confound also may interact with the condition, because wealthy students who don’tgo on school trips may go on their own while wealthy students who do go during school may not go on

15

their own. The wealthy students are acting differently depending on what condition they are, so evenif your two groups are equally wealthy there will still be differences in the behaviors of the groups.

If you cannot remove the potential influence of a confounding variable through randomization thenyou will need to include extra measurements or control groups in your design to test for the influenceof the potential confound on your response variables.

• Here are some questions that you can ask about a study to assess its internal validity.

◦ Is the direction of causality clearly established?

◦ Could the observed relationships have been created by unaccounted outside influences?

◦ Could some outside influence be obscuring the true relationships between the variables?

◦ Is the assignment of participants to between-subjects conditions properly randomized?

• Below are descriptions of some specific threats to internal validity.

◦ History. If your study takes place over an extended period of time there is a possibility that someunusual event might take place during the course of your study that might affect your results. Inthis case you cannot be sure whether you would have obtained the same results in the absence ofthe event.

◦ Testing. Participants who are asked to fill out a questionnaire or perform a test more than asingle time may use their past performance to guide their behavior on later tests. They mayremember specific responses and just recall them instead of re-generating their response. If youare specifically investigating memory then this is not a problem, but most often you want thethe tests to reflect changes caused by some other variable. In this case their experience acts asa confound. To get around this you can design parallel versions of the tests and use a differentversion at each test session.

◦ Maturation. If your study takes place over an extended period you need to take into account thatpeople grow and change over time. It is possible that any differences that you see over the courseof treatment are due to natural development instead of your manipulation. This is especiallyimportant to consider when using younger participants.

◦ Regression toward the mean. If you administer a test to a sample at one point in time youwill naturally observe that some people will score high and some will score low. Any form ofmeasurement, however, naturally involves some amount of random error. So if you were toadminister the same test again you would observe some differences in the scores, even if therewere no actual change. You would also see that those who scored high on the first test will tendto score lower on the second, while those who scored low on the first test will tend to score higheron the second. This is because if a person received a high score, they are more likely to have hadrandom factors working to their benefit. In a sense, they are toward the high end of the possiblevalues they might obtain on the test. If they took the test again they would be more likely tohave a lower score. Similarly, people with low scores are more likely to have had random factorsworking against them, so later tests are likely to provide higher results.This can be a problem if you put people into different groups based on a pretest score and thenyou want to observe the effect of a manipulation with a posttest. Even without the influence ofany other factors, regression toward the mean will cause the high group to have lower scores andthe low group to have higher scores at posttest. To minimize the influence of this threat youshould choose measures with as little random error as possible.

◦ Selection. This is a threat when you have different types of people in different levels of a between-subjects manipulation. In this case you can’t be certain whether the effects you observe are causedby the manipulation or the individual differences. Random assignment to conditions can usuallyremove threats of selection.

◦ Mortality. If you conduct a study that occurs for a long period of time it is common to have acertain percentage of your participants drop out before the study is completed. Sometimes thisthis dropout creates differences between the people in your different conditions. For example, you

16

might have one treatment that is more difficult than another. In this case you might see the lazypeople drop out of the difficult treatment but not out of the easy treatment. If this happens thenthe type of people in your groups will be confounded with your experimental manipulation.

◦ Diffusion of treatments. If participants in different conditions of your study receive different typesof training you need to consider whether the information received by members of one group mightbe transmitted to members of another group. If this happens you will see an artificial reductionof the difference between your conditions. This is particularly a problem when your training takesplace over a long period of time and you expect that your participants will have opportunities tointeract with each other.

◦ External equalization. This threat appears when your study involves providing some sort of benefitto some groups while withholding them from other groups. In this case you will sometimes seeother organizations trying to provide compensatory benefits to those in your underprivilegedcondition. For example, you might want to compare the performance of students in a studyprogram to those not in the program. The school may feel guilty about denying the potentialbenefits to some of its students and so may start providing extra study services of its own.

◦ Compensatory rivalry. If participants find out that they are in a disadvantaged or more difficultcondition of your study they will sometimes put forward additional effort to compensate for yourprocedures. You usually want to investigate the direct influence of the procedures rather than theeffects of participants’ reactance so their extra motivation acts as a confounding variable.

◦ Resentful demoralization. This is sort of the reverse of compensatory rivalry. Sometimes whenparticipants know that they are in a disadvantaged condition of your study they lose motivationto perform well at your tasks. This is especially likely if they perceive the task as exceptionallydifficult. Just like compensatory rivalry, resentful demoralization will act as a confounding variablein your design.

4.4 Statistical conclusion validity

• Statistical conclusion validity concerns whether the reported relationships in a study are accurate.

• A study’s statistical conclusion validity is determined by characteristics of both the variables andthe statistics. Your variables should be consistently manipulated and accurately measured. Poormeasurement or manipulation of your variables can prevent you from observing relationships. Yourstatistical analyses should be appropriate for the data and performed correctly. Improper statisticalanalyses can both prevent you from observing true relationships and can also cause you to concludethat there are relationships between unrelated variables.

• Here are some questions that you can ask about a study to assess its statistical conclusion validity.

◦ Were the measurements and manipulations reliable?

◦ Were the manipulations strong enough to produce observable effects?

◦ Do the data meet the assumptions of the statistical analyses?

◦ Do the statistical tests have sufficient power?

• Below are descriptions of some specific threats to statistical conclusion validity.

◦ Random influences in the study. The most common inferential statistics all involve some compar-ison of the amount of variability in the response measure that can be explained by the predictorvariables to the random variability in the measure. Anything that would increase the random vari-ability in the measure will reduce your ability to detect significant relationships. You can reducethe influence of random factors by making your procedures consistent across your participants.Sometimes there are random factors that you cannot control but which you can measure, suchas characteristics of specific participants. If you record this information then you can remove itsinfluence by including a blocking variable or a covariate in your analyses.

17

◦ Unreliable measurements. Any measurement will include some amount of error which will addrandom variability to your results. The extent to which an instrument will produce the samevalue on repeated measurements is referred to as its reliability. It is difficult to find significantrelationships using variables measured with unreliable instruments. If a study fails to find aneffect using unreliable measurements you cannot be sure whether the effect truly doesn’t exist orwhether the study simply lacked the ability to detect it.A study that successfully detects a significant relationship using an unreliable instrument, however,should not be called into question. A lack of reliability always works against attempts to findsignificance. If anything, observing a significant relationship using an unreliable measure meansthat the relationship must be particularly strong.

◦ Low power. The ability of your statistical analysis to detect a relationship is known as thepower of that analysis. Studies that have very small sample sizes generally have low power.Additionally, not all statistical procedures are equally good at detecting relationships betweenyour variables. Certain statistical procedures, such as the chi-square test for independence andmany nonparametric statistics, are known for being very poor at detecting relationships. Youshould be skeptical about any conclusions based on the failure of a low-power test to find asignificant relationship.Low power does not ever call significant findings into question. Just like low reliability, low poweris something that must be overcome to successfully detect a relationship. It cannot cause you tofind a relationship where one doesn’t exist.

◦ Weak manipulations. If a study fails to find a significant effect of a manipulation you must considerthe extent to which the procedures actually change the theoretical variable of interest. It mightbe possible that a stronger manipulation of the same variable would have had an observable effecton the response measure.

◦ Ceiling and floor effects. If you use a measurement that has minimum or maximum boundariesit can be difficult for you to detect differences when the average score is near those boundaries.Response differences become compressed near the ends of the scale, reducing variation. Thereisn’t much “room to move” so factors that would normally create more extreme responses willnot have their normal influence. One way to deal with this is to modify to response measure sothat it has broader endpoints.

◦ Violated test assumptions. All statistical tests have certain assumptions that they make about thedata that must be satisfied for the tests to produce accurate results. Some violations will causeyour tests to report significant relationships when there really is no relationship, while otherviolations will prevent a test from detecting the presence of an existing relationship. If the yourdata violates the assumptions of a test then you cannot draw any conclusions from the results ofthat test.

◦ Fishing. Standard inferential statistics are usually interpreted in terms of a “p-value” that indi-cates the probability of observing a relationship as strong as you did given that no relationshipactually existed. These p-values, however, are calculated on the assumption that only a singletest is being performed. The more test statistics that you calculate to analyze your study themore likely that you will observe a p-value below a specific value. To correct for this you shouldlower the critical value that you use when you perform a large number of statistical tests.

• One note should be made in particular about studies that accept the null hypothesis. It is very difficultto provide convincing evidence that there is not a relationship between two variables. You really can’ttell whether you failed to find a relationship because your statistical test didn’t have enough power orbecause there isn’t a relationship there to be found. Usually the best that you can do is demonstratethat one relationship is significantly less strong than some other comparative relationship.

4.5 External validity

• External validity concerns the ability to apply inferences drawn from a study to other populations,environments, and times.

18

• The more reliant a study’s results are on the specific features present in the original study the lessexternal validity is possessed by the study.

• To assess external validity you must first determine the real-life circumstance to which the researcherwants to generalize their conclusions. You then consider what characteristics of the study were im-portant for obtaining the results related to that conclusion. If these characteristics are not likely tobe present in the real-life circumstance then you have reason to question the external validity of theresults.

• External validity is not equally important for all types of studies. It is very important for studies whoseaim is to describe the relationships between variables in a particular target population. However, it ismuch less important for studies whose goal is to test some aspect of a theory. For example, a theorymight predict that a particular pair of variables should be related. This relationship should be presentin any particular sample your might draw. If you conducted a study and found no relationship thiswould be evidence against the theory even if the study had low external validity.

Even studies testing theories can benefit from having external validity. Externally valid tests of atheory provide more general (and thus stronger) evidence for or against the theory. For example,contradictory findings with low external validity may be dismissed by supporters of the theory asunimportant exceptions. Contradictory findings with high external validity, on the other hand, aremuch more difficult to ignore.

• A study can provide conclusions that may be validly generalized to one population but which do notvalidly apply to another. For example, a study providing information about shopping preferences mayprovide valid conclusions for the community in which the study was conducted but might be invalidfor understanding the preferences of a different community in another part of the country.

• Not all differences between samples and target populations create external validity problems. In orderfor a difference to be a threat to external validity there must be a reason to suspect that the differencewould influence the results of the study.

• When conducting your own study the key to establishing external validity is to make the importantcharacteristics of your research environment match those of the real-world situation to which you wishto generalize your findings.

• Here are some questions that you can ask about a study to assess its external validity.

◦ Is the purpose of a study to test a theory or to provide a description of a target population?

◦ What differences are there between the people, time, and setting of the study and the targetsituation? Could these differences conceivably have influenced the results?

• Below are descriptions of some specific threats to external validity.

◦ Unrepresentitive sample. If the type of people that you select for your study are affected by yourtreatment differently than would the average person that makes up your target population thenthe results of your study cannot be properly generalized to the population of interest. This is whyit is best to draw a sample that is truly representative of the target population.

◦ Unrepresentative setting. Similarly, if the environment in which you conduct your study is verydifferent from that to which you would like to generalize your results then your conclusions maynot be valid. It is usually more important to consider what is present in your study environmentthen what is absent. You do not necessarily need to replicate all of the different things thatyou feel would be present in real life. However, you should be fairly certain that the results youobtained were not caused by some peculiarity of your study environment.

19

Chapter 5

Preparing Study Materials

5.1 Study timeline

• The first thing you should do when trying to implement a study is to create a timeline detailingeverything that happens during your study. This will be relatively short for a survey but can becomequite complex for an experiment.

• When writing your timeline take the point of view the person administering the study. Your timelineshould be detailed enough so that another person could actually use it to conduct the study, assumingthey had access to your materials.

• Describe everything that the research administer would have to do from the time participants enterto the time they leave the study. If the procedure differs for participants in different conditions youshould describe what would happen in each.

• It is usually easiest to follow a timeline if it is presented as a hierarchical outline. That way you canquickly find the details about a specific part of the study.

• A timeline is useful at three different points of the research process.

1. When you are building your study materials a timeline forces you to think through every part ofthe study. It can be used to organize the generation of study materials, helping to make sure thatnothing is forgotten. Constructing a timeline will also make you more likely to notice threats tothe validity of your study.

2. When you are actually conducting the study the timeline can be used as a guide so that thecorrect things are done at the correct times. If things become hectic while running the study theadministrator can always turn to the timeline to make sure that all the appropriate things aredone.

3. When you are writing the methods section of an article the timeline is an excellent resource tohave. Even if you are able to memorize the correct procedure when conducting the study youmay forget details by the time you actually try to write about the research.

5.2 Verbal instructions and interactions

• You must next write the instructions for your study that you will give to participants. Every part ofyour study that requires participants to do something new should have its own set of instructions.

• Whether you plan to give the instructions orally or in a written format you should have them explicitlywritten down. You want to be sure that each participant gets exactly the same instructions. Whenproviding oral instructions the best way to do this is to have your research administrators read theinstructions from a script. You should try to have scripts for every standard interaction between theadministrator and the participants.

20

• If the task is particularly difficult or there are important details that participants need to know toperform it correctly you are better off providing the instructions orally. Participants will often skimwritten instructions or skip them entirely.

• Make your instructions as clear as possible. Use simple language to explain what participants shoulddo. Provide examples if this will help participants understand the task.

• Always provide participants with an opportunity to ask the administrator questions if they don’tunderstand something about what they are supposed to do. Participants that misunderstand the taskwill provide meaningless data, making it more difficult for you to obtain significant results. It is agood idea to end your instructions by asking if participants understand and by saying that they shouldcontact the administrator if they ever have any questions.

If you find out after the study is finished that a participant misunderstood the task you should makea note so that you can later exclude their data from analysis if necessary.

5.3 Materials for your manipulations

• If you are conducting an experiment you need to create the materials associated with your manipula-tions.

• You need to create a randomization scheme for any between-subjects manipulations that must beapplied by the experimenter. This scheme is a table where each row represents a participant and eachcolumn represents one of your between-subject variables. The particular condition each participantshould be in can be determined simply by reading the cells in the table. Sometimes people do therandomization while running the experiment but it is usually easier if you do this beforehand andrecord it on a randomization scheme.

There are several different ways to obtain random numbers to help you set up your scheme. Youcan use Microsoft Excel to generate columns of random numbers if you first install the Data AnalysisToolpak. This is part of the standard distribution of Excel (so you don’t need to buy it separately)but is not installed by default. To install it select Tools → Add-Ins and then check the box next to“Analysis Toolpak.” Afterwards you can generate random numbers using the Tools → Data Analysis→ Random Number Generation option.

Probably the best web resource for generating random numbers is the Research Randomizer at

http://www.randomizer.org

This online tool can generate lists of random numbers using many different options that you can specify.

• You will also need an experiment record. This is a sheet where you can list the details about eachparticipant in your experiment. It is usually designed as a table where each row holds a participantand each column holds a feature that you want to record. When you run a participant through thestudy you record the information on each feature in the appropriate cell of the sheet.

You will want to have columns representing the participant number, the date and time the participantwas run, and the values of any between-subject manipulations that were assigned by the experimenter.The experiment record may also code other details such as the sex of participants or the number of thecubicle they used. You usually do not use the experiment record for information on actual responsevariables. Its purpose is to note procedural details in case you find evidence that there were unusualinfluences on the results.

Your experiment record should provide some space (possibly on the back of the sheet) for the adminis-trator to note anything unusual that happens during a session. This will allow you to take these eventsinto account when analyzing the data.

• Finally you must create any stimuli that are to be used as part of the experiment. The specifics ofwhat you will need will vary greatly from study to study. You should construct them carefully andmake sure that you have the stimuli you need for every possible condition.

21

5.4 Materials for your measurements

• You need to prepare all of the measuring instruments (whether physical or written) used in your study.

• If you have physical instruments you must ensure that they are functioning properly. You should alsosee that those who will be operating the instrument have been properly trained in how to set up anduse the device.

• If you have observers code the behavior of participants they should have detailed instructions providedin a coding scheme. Observers should have extensive practice using a coding scheme before they recordthe data of real participants.

• If you are collecting data with a questionnaire you must determine exactly what questions you willask, what types of responses participants will provide, and in what order they will see the questions.Items that are answered in more or less the same way will generally be presented together to reduce theamount of instructions that you need to give. You should put the most sensitive and most importantitems early in the questionnaire so that their responses are less likely to be influenced by the wayparticipants respond to other items.

• When using an established scale (or coding scheme) you should use it without modifications, even ifyou think that your changes could improve the quality of the responses. Established scales have usuallybeen designed to have particular psychometric properties. If you introduce changes to a scale you canno longer be certain that the new version will retain all of the old properties. You should always reportany changes you make to established scales when reporting the methodology of your research.

• If you need to write any additional questions you should try to make them simple, clear, and unbiased.You want to make your items as simple and straightforward as possible so that your respondents areable to fill out the scale quickly and easily. Complicated questions can lead to misunderstandingsand annoy respondents, reducing the likelihood that you obtain good data using the scale. Unclearquestions can cause respondents to interpret your items in different ways, reducing the likelihood thatthey will answer in ways that are related to the theoretical construct of interest. An item should bedesigned such that every respondent interprets it in the same way. If the language of your questionsmakes some responses seem more expected or desired than others you will be unlikely to get a truemeasurement of respondents’ characteristics. Instead they will alter their answers in a way that makesthem look better according to the language of the question. This will alter the actual meaning of theitem, moving it away from the theoretical construct of interest.

5.5 Post-study survey

• You should design a post-study survey to see how participants perceive your study.

• The point of a post study survey is to provide you with insight into each participant’s personal ex-perience while going through your study. If participants provide abnormal data the responses in thesurvey can sometimes provide an explanation for their behavior.

• There are three questions you should always ask in your post-study survey.

◦ “Were the instructions complete and understandable? Was there ever a time when you wereuncertain about what you were supposed to do?”

◦ “Did anything unusual happen while you were going through the study?”

◦ “What did you think this research was about?”

• If you can think of any additional questions that could provide you with insight on factors that mightinfluence participants’ performances your should include them in the survey.

22

5.6 Ethics materials

• You must generate the materials required to satisfy APA ethical guidelines and local human subjectscommittees.

• Many universities and local communities have created Human Subjects Committees that you must getto approve your study before you are allowed to conduct it within their jurisdictions. APA guidelinesrequire that researchers receive approvals from any such organization before conducting research. Thesecommittees typically require that you send in an application indicating the purpose of your researchand detailed information about what participants’ experiences will be in the study. Your applicationwill then be reviewed and you will either receive approval for your study or the committee will requestspecific changes to your procedures.

• APA guidelines require that you obtain the written consent of everyone who participates in your studyunless it meets their listed criteria for disposing with informed consent. For this you will need to createan Informed Consent Form to have signed by each participant. This form should provide a descriptionof what participants will be doing in the study, how long it will take, the extent to which they willbe personally associated with their responses (i.e., anonymity procedures), and a list of who will haveaccess to the data. It should describe the potential benefits of participation to the participant and tosociety. Participants must be warned if there are any expected discomforts associated with the study.The description of the study should concentrate on what a participant’s experience will be while thestudy is conducted. Somewhere in the document you should report the name and contact informationof the primary investigator.

APA guidelines require that the consent form states that participants may quit the study withoutpenalty at any time and that they have the right to know the outcome of the research. There shouldbe a statement in the document saying that by signing the participants give their informed consent tobe part of the study. At the bottom you should have a space where the participants will sign and datethe form. When running the study you should keep a signed form for your records and give a blankone for the participant to keep.

• You should also design a debriefing form to give participants once they finish the study. This formtells participants exactly what you were researching and the purpose of each part of the study. Thedebriefing should reveal the truth behind any deceptions in the study and should report any hiddenmeasures that you took. Like the informed consent, the debriefing form should provide the primaryinvestigator’s name and contact information.

Providing a debriefing lets participants understand their importance to the scientific process and pro-vides them with knowledge about your research. This helps keep them happy and will foster goodwillbetween researchers and the public. If your firm or university gets a bad reputation regarding thetreatment of participants you may find it very difficult to perform research.

23

Chapter 6

Conducting Research

6.1 General guidelines

• Psychological research should adhere to the published APA ethics code. All APA journals (and mostnon-APA journals as well) require authors to state upon submission that the research was conductedaccording to APA guidelines. The full APA ethics code can be found online at

http://www.apa.org/ethics/code.html

Below are some of the main points relating to conducting research.

◦ Research must not violate any applicable laws.

◦ Research must be approved by any applicable institutional and community human subjects com-mittees.

◦ Researchers must obtain the informed consent of participants prior to collecting data from themexcept in the case of anonymous surveys and naturalistic observations.

◦ All inducements for participation must be provided and all commitments made to participantsduring a study must be kept.

◦ Researchers must not offer excessive inducements when they might tend to coerce participation.

◦ Deception should not be used in research unless it is justified by the potential knowledge to begained by the study and alternative methods are unavailable.

◦ The researcher must provide participants with information about the purpose of the study oncethey conclude their involvement. At this time the researcher should correct any misconceptionsheld by participants resulting from participation in the study.

• Research administrators should always act professionally when conducting a study. This improvesthe credibility and authority of the administrator and will cause participants to take the study moreseriously. A professionally presented study can make participants more likely to volunteer, encouragethem to listen to or read the instructions more carefully, and cause them to put more effort to thetasks they perform.

Here are some things that research administrators should do to appear more professional.

◦ Dress as if they were working in a professional job.

◦ Be prepared and organized. They should try to have all the materials set up and accessible beforeinteracting with participants.

◦ Be familiar with study procedures. Administrators who are unsure about what they are doingwill reduce a participant’s confidence in the study.

◦ Treat participants professionally. Administrators should not express anger or annoyance withparticipants nor should they be flirtatious or overly friendly. The best attitude would probably

24

be “unobtrusively pleasant.” A study administrator should only be noticed by participants whenthey specifically need to interact.If you should ever troublesome participants the best thing for you to do is to fully compensatethem for the time they were in the study and then dismiss them. Any sort of confrontation willreflect badly on the study and could affect the performances of other participants that are present.

◦ Be knowledgeable about the research. If participants ask question about study procedures theadministrator should be able to respond quickly and confidently. Administrators should also haveenough knowledge to answer any theoretical questions that are likely to be asked by participantsduring debriefing.

◦ Be attentive. It is generally best that administrators not engage in outside activities (such asreading a book or listening to the radio) while they are in the presence of participants. Similarlytwo administrators should not have personal discussions in front of participants while conductinga study.

• You should make your study as pleasant as possible for your participants. Discomfort and fatigueare often sources of confounding variables in a study. They can also reduce the reliability of yourmanipulations and measurements.

Below are some things you can do to make your study more pleasant.

◦ Make your instructions clear. Participants can get frustrated if they have to spend much timetrying to figure out what they are supposed to do.

◦ Make your materials easy to see and read. Avoid unusual fonts when presenting text and makeimages large enough so they can be seen without straining.

◦ Make your study interesting. You can greatly increase participants’ motivation in your study ifyou can somehow make the task more fun or intriguing.

◦ Try to maintain a comfortable working environment. Conduct your study in a location that iscomfortably warm and relatively free from outside distractions such as noise.

◦ Keep your study sessions short. As the length of a study grows the likelihood of participantsbecoming bored or fatigued increases. If you have the option between two equivalent measuresyou are usually better of choosing the shorter one. If possible try to keep your study sessionsunder one hour.

◦ Provide breaks during long or repetitive tasks. Even short breaks can help reduce fatigue whenparticipants are performing the same actions over and over. If you have a session that lasts formuch longer than an hour you should also provide at least one opportunity for participants to usethe bathroom.

◦ Design your study so that it is easy for participants to perform. The interface used by participantsshould be as simple as possible. If you are conducting a survey try to use a standard format foryour questions. If you are conducting an experiment try to minimize the amount of physical workparticipants need to do in their tasks.

◦ Maintain a pleasant attitude throughout the session. This is especially important if you have asignificant amount of interaction with the participants.

◦ Take all questions from participants seriously. If a participant comes to you with a question neversend them away without being completely sure that you solved their problem. Participants areunlikely to ask a question a second time and will become frustrated if they must perform a taskthey don’t understand.

• There are two basic formats you can use when conducting research. If your entire study could beplaced on paper (including all manipulations and measurements) then you are in a position to usea quesionnaire. If your study requires a heavily controlled environment, complicated procedures orapparatus then you must instead perform a laboratory study. The main difference between the twois the portability of the materials. Questionnaires can be easily moved and administered in differentlocations while laboratory studies must usually be conducted in a single location. For more details on

25

administering a questionnaire see section 6.2. For more details on conducting a lab study see section6.3.

• It may require multiple sessions for you to run all of the participants you want. At the end of eachsession you should store any completed forms in a safe place. The more you move your forms aroundthe greater chance that you might lose or damage them.

6.2 Administering a questionnaire

• When administering a questionnaire the first thing you need to do is to decide exactly how and whereyou will give the questionnaire. You have a large number of options because a questionnaire can becompleted almost anywhere.

◦ You can advertise and have participants come to a specific location, just like you would fora laboratory study. The advantages of this method are that you can run a large number ofparticipants at a time, the common setting reduces random sources of variance, and you can uselong questionnaires since you know that anyone recruited for the study will have the appropriateamount of time available. The disadvantages are that you will almost always need to compensateparticipants to get them to show up, and you are susceptible to selection artifacts since peoplewho are willing to travel to be in your study may be different from those who are not.

◦ You can go to a public location with clipboards and ask people passing by to fill out your ques-tionnaire. The main advantage of this method are that you can often recruit participants free ofcharge. A disadvantage is that this method can only be successfully used with relatively shortquestionnaires.

◦ You can travel to people’s homes and conduct interviews with participants. The advantages ofthis method are a relatively high response rate and an increase in the quality of the data. Therewill generally be a reduced chance of coding errors since the administrator will be recordingthe responses. The administrator also has the opportunity to provide additional instructions orexamples for items that confuse participants. Some disadvantages of this method are that youcan only run one participant at a time and the presence of the administrator may influence theresponses provided by participants. You may also have difficulty obtaining a truly random samplesince certain types of people spend more time at home than others.

◦ You can conduct phone interviews with participants. The advantages and disadvantages of phoneinterviews are similar to those of live interviews. Some differences are that the use of the telephonerequires less effort on the part of the administrator and provides participants with a greater senseof anonymity. However it makes it very difficult if not impossible to provide any pictorial orgraphic information as part of the questionnaire.

◦ You can mail your questionnaire to potential respondents and provide a self-addressed stampedenvelope for them to return their response. The main advantage of this is that it offers the greatestanonymity of all the methods. The disadvantages are that you will generally have a low responserate and it will take a long time before you collect all of your data.

◦ You can send your questionnaire to people over email. The advantages of this are the exceptionallylow cost (no photocopying) and the relative ease with which you can send your questionnaire tolarge numbers of people. The disadvantages are that you can only contact a limited community(people who use email) and that you will generally have a low response rate.

◦ You can post your questionnaire on the World Wide Web. The advantage of this method is thatyou can collect huge amounts of data in a relatively short period of time with a minimum of cost.The main disadvantage is that you have no control over and limited knowledge about the type ofparticipants you receive and the circumstances they are in when they fill out the questionnaire.

• Once you have decided on the method you will use you must recruit your participants. You want yoursample of participants to be representative of your target population on all important dimensions.If you have a specific target population (such as members of a particular profession) you may need

26

to obtain lists of population members from which you can draw your samples. For more generalpopulations you may decide to perform your recruitment at the time you are actually administeringthe questionnaire.

• You must then make copies of your questionnaire and bring them to the site at which you will conductyour study. You should also bring any supplemental materials, such as pencils, erasers, or paper, thatwill be needed to complete the study. You shouldn’t count on participants having anything they needwith them, even if you tell them beforehand.

• At this point you will actually administer the questionnaire, using the guidelines presented in section6.1. Follow the timeline you created when generating your study materials. Be sure to use informedconsent and debriefing forms if they are required by your study.

6.3 Conducting a laboratory study

• If you decide to conduct a lab study the first thing you will need to do is to recruit participants.This will generally need to be done in advance since you must arrange for participants to be at thelaboratory at the time you are running the study. When recruiting you will need to tell people thegeneral nature of what they will be doing, how much time the study will take, the requirements forparticipation, the compensation they would receive, and where and when the study will be held. It isa good idea to provide potential participants with reminder slips so they don’t forget where and whenthey should be at the laboratory.

• When it comes time to conduct the study you should always go to the laboratory early. This will allowyou to make the appropriate preparations and be ready before participants arrive.

• You should greet participants as they arrive in the laboratory. Have some empty chairs available sothat participants can sit down in case you are not ready to start the study when they arrive. If youmust have them wait you should give them an approximation of how long it will be before the studywill begin.

• At this point you will actually conduct the study, using the guidelines presented in section 6.1. Followthe timeline you created when generating your study materials. Be sure to use informed consent anddebriefing forms if they are required by your study.

6.4 Pretesting your study

• You will usually want to perform a trial run of your study before you attempt to conduct it using realparticipants. This is referred to as performing a pretest. The goal of a pretest is not to obtain accurateor reliable data, rather it is to determine the effectiveness of your manipulations and measurementsand to expose any potentially difficult aspects of the study’s procedure.

• In a pretest you are specifically not concerned using your results to test your hypotheses. In fact,you will often add in diagnostic procedures that you know will affect your results and would normallyprovide alternative explanations for your findings. Additionally, you do not need to be particularly con-cerned about the representativeness of your sample since you will not be drawing theoretical inferencesfrom your results.

• To determine if your manipulations are accomplishing what you intend them to you can performmanipulation checks. A manipulation check is simply a measurement of the variable that you aretrying to manipulate. For example, if you have people watch either a happy or a sad movie to try tomanipulate mood, you can perform a manipulation check simply by asking people what their mood is.You often don’t include manipulation checks in the regular study procedure because the check mightmake people aware of the manipulation and cause them to try to correct for it.

27

• If you measure a variable using a scale with multiple questions you can calculate the reliability of thescale to determine the precision of its measurements. Reliability tells you the extent to which peopletend to respond the same way to all of the items in the scale. Scales with low reliabilities aren’tmeasuring a single, consistent construct and therefore will provide poor results when used in statisticalanalyses.

• It is fairly common to have a short five-minute interview with pretest participants after they finishyour study. You should ask them the type of questions that you would ordinarily put into a post-studysurvey. Actually conducting a live interview is more work but is better (at least during the pretest)because it will allow you to ask follow-up questions about anything participants might say.

28

Chapter 7

Data analysis

7.1 Creating a data set

• You will almost certainly want to perform your statistical analyses using computer software. Thisrequires you to create a file on the computer representing the data set. You will usually want eachrow of this data set to represent a participant and each column of the data set to represent a variable.This means that any particular cell of the data set holds the value of the variable in that column forthe participant in that row.

• If you collected your data using surveys or other written measures you will need to type the data intothe computer. The three most common options are to use either a word processor (like WordPerfector Microsoft Word), a spreadsheet (like Lotus 123 or Microsoft Excel), or a database (like Paradox orMicrosoft Access).

The best option is probably to use a spreadsheet program. Spreadsheets are generally much easier towork with for data entry than either of the other two options. Additionally, most statistical programshave procedures that allow you to directly translate spreadsheet files to data files. If you use a wordprocessor you will probably have to save the data as a text file and then go through a possibly lengthydata import procedure. If you use a database you will likely have save the data as a spreadsheet fileanyway for it to be read by your statistical software.

• Perhaps the best strategy to remove data entry errors is to have the questionnaires entered into thecomputer twice, in two different files. You can then locate mistakes by having the computer checkfor differences between the two files. Data that has been entered incorrectly in one file should showup as mismatching the other file (since it is highly unlikely that the two files would contain the sametypographical error on the same item). Most modern word processors have a “Document Compare”function which allows you to locate differences between two files. This is referred to as double entry.It is easy to get almost complete accuracy using this method.

You can either have two different people type in the data or you can have the same person type itin twice - it actually makes little difference. If you use a spreadsheet or database for data entry youshould probably convert the file to a text document and then use the document compare from a wordprocessing program. You can then use the results to make corrections to the original spreadsheets ordatabases. The reason you should do this is because the compare functions available in word processorsare usually much easier to use than the compare functions of spreadsheets or databases.

• Sometimes the initial form of your data set will not be what you need for a particular statistical analysis.In this case you can use a statistical program to transform your data set into the new form you desire.Specific details about how to transform your data set can be found in our Notes on Transforming andRestructuring Data, freely available on the Stat-Help.com website.

29

7.2 Statistical analysis

• A detailed discussion of how to analyze your data is beyond the scope of these notes. This section willinstead provide information on what procedures you should use to examine the types of relationshipsthat are most commonly proposed in hypotheses. It will be left to the reader to learn the specifics ofhow to perform each procedure.

• The following procedures examine the relationship between a single pair of variables. The one youshould use depends on the types of variables you want to examine.

◦ Both correlations and simple linear regression can be used to examine the relationship betweentwo numeric variables.

◦ T-tests or ANOVA (analysis of variance) is used to examine the relationship between a numericvariable and a categorical variable. You use a t-test if your categorical variable has only twolevels, otherwise you must use ANOVA.

◦ A chi-square test is used to examine the relationship between two categorical variables.

• The following procedures examine the relationship between a single numeric response variable and twoor more predictor variables. The one you should use again depends on the types of predictor variablesyou want to use.

◦ Multiple regression relates a numeric response variable to two or more numeric predictor variables.

◦ ANOVA (analysis of variance) can relate a numeric response variable to two or more categoricalvariables.

◦ If you want to relate a numeric response variable to a combination of numeric and categoricalvariables then you will need to use some form of the general linear model. Both multiple regressionand ANOVA are actually specific applications of the general linear model.

• If you want examine the relationship between a single categorical response measure and two or morepredictor variables (either numeric or categorical) you will need to use logistic regression.

• If you want to examine the relationship between two or more response variables and a set of predictorvariables you will need to use multivariate analysis.

7.3 Documenting your study

• You should always document a study as soon as you finish analyzing the data. It should be documentedwhether or not you found evidence for your hypotheses and whether or not you intend to publish yourresearch.

• There are several different reasons why you should document your study.

◦ You will need a description of the methods and results if you wish to publish the research.

◦ You may want to have a record of what you did as a reference when conducting future studies.

◦ You or another researcher may want to go back at a later point in time and re-analyze your data.

Notice that the last two justifications apply even if you didn’t obtain publishable results. Sometimesresearchers will just store the data from an unsuccessful study without any documentation. Howeverit is quite possible that you might want to run a variant of the study at a later point and could use awriteup so that you can decide on appropriate changes. It is also possible that a study with unexpectedresults may become understandable later on when new theories become available.

• To properly document a study you need to include information about the exact procedures that wereused and the results that were obtained. Your documentation should include the following items.

◦ A complete writeup of the method section as if for publication (see section 8.3).

30

◦ A complete writeup of the results section as if for publication (see section 8.4).

◦ A copy of the study timeline.

◦ A copy of the data (usually on disk) generated during the study.

◦ An index to the data. For each data file you should provide a general description of what is inthe file, the location and name of the file, and explanations of each variable contained in the dataset.

◦ Copies of any programs you used for statistical analysis.

31

Chapter 8

Writing a Research Paper

8.1 General comments about writing

• The official guide to writing psychology papers (both for classes and for journals) is the PublicationManual of the American Psychological Association (APA, 1994). This book contains information aboutwriting style, punctuation, formatting a document, and references. The website

http://www.uwsp.edu/psych/apa4b.htm

contains a nice summary of APA style, including examples of formatting and references (Plonsky,2000).

• The way that you write about your work can be just as important as the actual content. It is wellworth the effort to make your papers as clear and as easy to read as possible. Well-written papers havemany advantages over poorly written papers.

◦ Their concepts are easier to understand.

◦ The importance of the research can be more readily seen.

◦ People are more likely to finish reading the article.

◦ The research is more likely to be remembered.

◦ Readers are more likely to select articles authored by people known to be good writers.

◦ Well-written arguments are more persuasive.

◦ People perceive good writers as more knowledgeable and more competent.

• Here are some things you should do when writing.

◦ Write in complete sentences using appropriate grammar. Other people will have a very difficulttime understanding what you want to say if you write in incomplete sentences. Improper grammartents to attract attention to itself so that you reader is distracted from the content of your writing.

◦ Be sure the vocabulary is used correctly. If you are not completely sure of the definition of aword you should either look it up or avoid using the word in your writing. In addition to beinga nice guide to writing papers, Sternberg (1993) provides definitions of words that are commonlymisused by psychological authors.

◦ Write clearly. Your sentences should each read smoothly and plainly express its point. Alwaysread through your document after you are done writing. Try to take the perspective of a minimallyknowledgeable reader and consider whether each sentence is understandable. If you have anydoubts you should rewrite the sentence to make it clearer.It can be difficult for authors to notice unclear writing in their own papers. Often times the writerunconsciously uses their own background knowledge to make unclear sentences understandable.For this reason it can be helpful to have another person read your paper and point out unclear

32

sections. If you can’t find someone else to go over your paper you can also try reading the paperout loud. This makes it more difficult to unconsciously clarify your writing.

◦ Write simply. Avoid the temptation to use intellectual jargon or overly technical language. Read-ers will not be impressed by your knowledge of vocabulary. Rather they will usually becomeannoyed at the increase in effort required to read your paper. When you have a choice between asimple and a complicated way to say something choose the simpler option.

◦ Write for your reader. The nature of your intended audience should influence your writing inseveral ways. First, you should choose a vocabulary that would be understood by the least-informed member of your intended audience. If you are writing a paper to be read by medicaldoctors you can use more complicated medical terminology than if you were writing about thesame topic for patients. Second, you should provide technical details to the extent that they arerequired by your audience. You can describe a well-known memory task to a group of cognitivepsychologists using much less detail than would be required to communicate the same thing to amore general audience. Finally, you should focus your discussion on the aspects of your researchthat would be of greatest interest and concern to your intended audience. An article written fora popular magazine should focus on general findings and implications while one written for ascientific journal should provide more detail about methodology and statistical analysis.

◦ Avoid excess language. This incorporates two basic points. First, if a word or phrase can beremoved without changing the meaning of a sentence then those words should be dropped. Twomajor sources of extra words are introductory clauses (“Indeed, the major finding was that theexperimental group exceeded the control”) and unnecessary qualifiers (“Participants performeda rather uninteresting task”). The two example sentences can just as easily be written as “Themajor finding was that the experimental group exceeded the control” and “Participants performedan uninteresting task.”Second, you should try to avoid redundant sentences when possible. Redundant writing is notalways bad - it can emphasize important points or provide additional opportunities to understanda difficult concept. However, redundancy also encourages knowledgeable readers to skim yourdocument, possibly causing them to miss important points. You are usually better off to makeyour first presentation of an idea clear and interesting than to reiterate your point later in thepaper.

◦ Prefer active to passive voice. Sentences written in active voice contain action verbs (“Attentiondecreased during the experiment”) while those written in passive voice contain linking verbs(“A decrease in attention was observed during the experiment”). A passive-voice sentence canusually be transformed to active voice with some slight rewording. Documents written usingactive voice are easier to read and more interesting than those using passive voice.

◦ Include transitions and summary statements. Often times you will have distinct parts within asingle section of your document. You should include sentences and paragraphs to cleanly concludeeach part and to provide a link to the following topic. Summaries not only help you emphasizeyour main points but also notify your reader that a topic change will soon occur. Transitions helpyour reader move from one part to another without losing track of the underlying theme of yourpaper. Documents without appropriate transitions and summaries appear choppy and disjointed.

◦ Avoid sexist language. Do not exclusively use male pronouns when you want to refer to a genericindividual who could be of either sex. You can use “one” or “he or she,” but unfortunately both ofthese terms are awkward and can bog down your writing. Probably the best option is to rephraseyour sentence so that it refers to a plural so that you can use the pronoun “they.” For example,instead of saying “Each participant wrote down his first name” you could say “Participants eachwrote down their first name.”

8.2 Writing the introduction section

• The main purpose of the introduction is to tell your reader what it is that you studied and to convincethem that it is important.

33

• The first paragraph typically provides an introduction to your general topic, often with real-worldexamples to demonstrate the prevalence of the phenomenon of interest. It should convey the importanceof the research area and motivate people to read the rest of the paper. Try to avoid using citations inthe first paragraph unless your study is concerned with issues tied to a specific study or methodology.

• Following the first paragraph you should provide a review of the literature relevant to your research.The purpose of this review is to provide a context in which your study and its importance can beunderstood. You therefore do not need to give a comprehensive summary. Instead focus on articlesthat are directly relevant to your study.

• After reviewing the prior research you then must link it to your current research. You need to makea convincing argument that your study is necessary, valuable, and interesting. Specifically you shouldpoint out the gap in the literature that your study is designed to fill and convince readers that muchcan be gained by filling it.

• Finally you present the general idea behind the study you conducted. You should state the researchquestion your study investigates and what information the study is designed to provide. Report thehypotheses that your study was designed to test. You may also provide an overview of the study,although specific details should be left to the method section.

• Be sure to put effort into relating the literature review to your own research. Explicitly state anylinks that you wish readers to make. You cannot assume that people will appreciate the importanceof something unless you specifically point it out to them.

• Keep in mind that you are not simply reporting past research in the introduction. You are trying tosell the idea that your own study is important. From beginning to end the introduction should be aconvincing argument that the research you propose is interesting and necessary.

8.3 Writing the method section

• The purpose of the method section is to tell your reader exactly how you conducted the study. Youshould include enough detail so that someone reading your method would be able to conduct a repli-cation of the study.

• The method section is usually divided into several subsections, each describing a different aspect ofyour study. The exact order of these subsections is not fixed. You should choose an order thatmakes your study easiest to describe. Your method additionally does not need to have all of thesubsections. Sometimes a particular subsection may not apply to your study, and sometimes you maydecide to combine two subsections to make presentation easier. Include whatever subsections you feelare necessary to fully describe your study.

• Below are the most common subsections in a method section.

◦ Participants. This subsection should list the total number of participants in the study and providea general description of their characteristics. You should also report how they were recruited andwhat compensation they received for participation. If you decided to exclude any participantsfrom your statistical analyses you would state that here and provide an explanation why theywere dropped.

◦ Apparatus. Here you should describe any physical equipment that you used when conducting yourstudy. You should provide the make and model of any important items. You should also describehow the apparatus was used in your study. If your equipment was put together in an unfamiliarway you should provide a photograph or a schematic of the setup you used.The use of computers in your study is not usually described in this section. If your study wasconducted using a computer program you will typically provide a short note in the proceduresubsection describing the type of computer you used (i.e., PC or Macintosh) and the programminglanguage in which the study was written. The use of computers and software for statistical analysisis not usually reported unless the analyses are particularly unusual.

34

◦ Design. The design subsection should describe all of the variables that were measured or manip-ulated in the study. It should present the conditions you used in your study and how participantswere assigned to the conditions. The specifics of how to write up your design are provided insection 3.6.

◦ Procedure. Here you should describe what your participants did from the time they entered to thetime they left your study. You will find that your study timeline will be very useful when writingthis section. You should also include details on any coding schemes you used.When describing your procedures you should stay at a conceptual level. For example, instead oflisting out all of the questions you asked in a survey you would simply state what theoretical vari-ables were measured by the questions. If you have several questions composing a scale measuringa single concept then you may also choose to report the reliability of the scale. When describinga manipulation you would want to provide its purpose and general format of any stimuli ratherthan a list of the specific stimuli used in the study. If you wish to include detailed informationabout any new scales or stimuli you develop you should put them in an appendix rather than inthe method section.

8.4 Writing the results section

• The results section is where you report your data analyses. It should provide both summary statisticsdescribing the data as well as any inferential statistics you calculated to test your hypotheses.

• You usually start this section with any statistics you calculated to verify the validity of your study.This would include manipulation checks and tests of order effects. If you have any scales for which youdid not provide reliability measurements in the method section you would also provide that here. Ingeneral you want to remove any concerns that readers may have about your methodology before youpresent your major findings. If you do obtain a finding that would question your study’s validity youshould honestly report the implications it has for the inferences you wish to make.

• You then want to present the main results of your study. If you are testing causal hypotheses you willlikely want to organize your presentation around the different response variables. For each responsevariable you should start with a table containing the mean and standard deviation of the responsein each cell of the experimental design. If your design is very complicated you might instead providea graph, though tables are generally preferred. You should then follow this with the results of theinferential tests you performed (such as t-tests, ANOVA, and regression) relating the response variableto the predictor variables.

If you are testing correlational hypotheses then the organization is more up to your own discretion.You can present the means and standard deviations of your variables if you like. The main part of yourresults section will be dedicated to reporting the results of inferential tests you performed relating thevariables to each other.

• While descriptive statistics (such as means and standard deviations) are often displayed in tables orgraphs, inferential statistics are presented as sentences within the body of your paper. When reportingthe results of a commonly used inferential test you need to provide the name of the test, a statementas to whether the test was significant or not, the value of the test statistic, the degrees of freedomassociated with the test, and finally the p-value of the test. The last three of these are usually separatedfrom the rest of the text by parentheses or commas. For example, you might say “A within-subjectst-test indicated that there was a significant difference between the experimental and the control group(t[15]=4.3, p < .01)” or “An ANOVA revealed that the interaction between prime type and delay wasnot statistically significant, F[2, 39]=1.4, p > .05.” If you use an unusual statistic you should provideadditional information about interpreting the statistic as well as a reference where more advanceddetails can be found.

• When reporting p-values people conventionally report that the p was less than a particular value ifthe test was significant while they report that it was greater than a particular value if the test was not

35

significant. In psychology we typically consider p-values less than .05 to be significant. Some peoplecall p-values between .05 and .10 “marginally significant.”

• APA format requires that test statistics named using normal letters (such as t, F, and p) be presentedunderlined in the standard font. Test statistics named using Greek letters (such as χ2 and λ), on theother hand, should be presented in the standard font without underlining.

• After you report your inferential statistics you can then provide the results of any secondary analysesyou performed to further explore a significant finding. For example, after reporting a significant maineffect in ANOVA you might follow this with statistical comparisons of the individual cell means.

• If you report the results of an inferential test that has implications for one of your hypotheses you shouldfollow it with a single sentence describing the implication. For example, “This result was consistentwith our expectations.” Save any detailed comments for the discussion section.

8.5 Writing the discussion section

• In the discussion section you provide an interpretation of the results of your study. You should startwith a description of how well the data support the hypotheses you proposed in the introduction. Youshould talk about the implications for each hypothesis individually. These implications will fall intoone of four categories.

1. The findings provide support for the hypothesis. In this case the relationships proposed by thehypothesis were observed in your data. Additionally, your results provide no evidence of anyalternative explanations for proposed causal relationships. If your hypothesis proposes multiplerelationships it is possible to find partial support for the hypothesis if some of the proposedrelationships are observed while others are not.

2. The findings provide support against the hypothesis. In this case you observed relationships inyour data that are inconsistent with the hypothesis.

3. The findings fail to support the hypothesis but do not provide evidence against it. This happenswhen you fail to observe relationships proposed by the hypothesis but do not observe any con-tradictory relationships. Inferential statistics do not allow us to draw strong conclusions whenwe obtain nonsignificant results, so observing lack of evidence must be treated differently fromobserving specific evidence against a hypothesis.To draw any conclusions from a null effect you will need to provide convincing evidence that yourstudy was powerful enough so that you would have observed a significant relationship if one hadbeen present. One way to do this is to show that your study was able to detect other relationshipsthat should be about the same strength as the those predicted by the hypothesis. You may alsodirectly calculate the power of the study, which is defined as the probability that you would obtainnonsignificant results given that the proposed relationship actually does exist.

4. The findings cannot be interpreted. It is possible that some aspect of your study prevents youfrom drawing any conclusions regarding the hypothesis. For example, you might have detecteda confounding variable that provides an alternative explanation for a proposed causal relation-ship. Another possibility is that you find evidence that your manipulations or measurements lackconstruct validity. This would mean that the relationships you observe in your data do not trulyreflect the relationships between the theoretical variables you wish to examine.

• You should next state what conclusions you wish to draw from your study. Specifically, you shouldstate whether the study as a whole provides support for or against any theories relevant to the research.If the results failed to support your own hypothesis you might wish to propose alternative explanationsfor the results.

• You may also discuss any practical implications your study has for either research or everyday life. Youshould consider whether the results of your study might help explain other findings in the literature. Ifthere are any direct implications of your results for real-life behavior you should also make them clear.

36

• Finally you should describe specific directions that you think future research on the topic should take.Section 8.6 discusses the different ways you may choose to follow up a study.

8.6 Planning future studies

• The final stage of the research process is a return to the very beginning: Generating new researchhypotheses. However, at this point you are in a different circumstance than when you originally started.You have conducted, analyzed, and drawn inferences about a new study which should hopefully presentnew opportunities for research.

• If your study supported your research hypotheses then you want to think about ways to further verifyor extend your findings.

◦ You could perform a conceptual replication of your study. Since hypotheses are usually writtenin abstract terms it should be possible for you to test the same hypothesis using different manip-ulations or measurements. For example, you might have originally conducted a study where youmanipulated how much participants were paid to show that motivation influences performance ona task. To follow this up you might conduct a second study where you manipulate the perceivedimportance of the task (another way of varying motivation) and see if this has similar effects onperformance.

◦ You might try to locate mediators or moderators of the relationships you discovered. Once youestablish the presence of a relationship between two concepts you can go deeper and try to addother variables to the equation. For example, you might have originally conducted a study showingthat skill level expectations influence task performance. To follow this up you could examinewhether individual differences in optimism and pessimism might moderate this relationship.

◦ You can attempt to show that the observed relationships can be observed in different populationsand different environments. A single study is necessarily limited in terms of where and on whomit was conducted. Demonstrating that the findings can be replicated in either more general ormore specialized circumstances can be valuable.

• If your study failed to support your hypotheses then you should plan studies to examine and correctthe reasons for the failure.

◦ You could run the first study again but with increased power. You might consider strengtheningthe manipulations (such as by using more extreme stimuli) or improving the sensitivity of yourmeasurements (such as by increasing the number of items in a scale).

◦ You also might try to test the same hypotheses but using different manipulations or measurements.The difficulties you had in detecting relationships could be tied to the specific procedures in thestudy so changing them might give you the results your desire.

◦ You could also attempt to locate a moderating variable that may be suppressing the relationshipyou wish to find. For example, you might have conducted a study that failed to find a hypothesizedrelationship between current events and overall mood. This might be because there are manypeople who do not pay attention to current events, but that for those that do the nature of theevents would affect their mood. You might therefore perform a follow-up study using the samebasic method but with additional measures to determine how often participants read newspapersor watch the news on television.

• Finally you should also consider any relationships that you observed involving one or more exploratoryvariables (those you measured but were not specifically related to your hypotheses). You might try todevelop hypothetical explanations for the relationships that can be tested with a study.

37

References

American Psychological Association (1994)(4th edition). Publication manual of the American Psycho-logical Association. Washington, DC: American Psychological Association.

American Psychological Association (2001)(9th edition). Thesaurus of Psychological Index Terms. Wash-ington, DC: American Psychological Association.

Baron, R. M., & Kenny, D. A. (1986) The moderator-mediator variable distinction in social psychologicalresearch: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology,51, 1173-1182.

Cook, T. D., & Campbell, D. T. (1979). Quasi-Experimentation Design & Analysis Issues for FieldSettings. Boston, MA: Houghton Mifflin Company.

Plonsky, M. (2000). Psychology with style: A hypertext writing guide (Version 3.1). Retrieved from theweb March 20, 2001. http://www.uwsp.edu/psych/apa4b.htm

Rosenthal, R. (1964). Experimenter outcome-orientation and the results of the psychological experiment.Psychological Bulletin, 61, 405-412.

Sternberg, R. J. (1993). The Psychologist’s Companion: A Guide to Scientific Writing for Students andResearchers. Cambridge, UK: Cambridge University Press.

38