![Page 1: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/1.jpg)
Reliability and Validity of Researcher-Made Surveys
![Page 2: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/2.jpg)
Reliability
![Page 3: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/3.jpg)
Reliability
“Authors should provide
reliability coefficients of
the scores for the data
they analyze even when
the focus of their
research is not
psychometric.”
![Page 4: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/4.jpg)
Reliability
“Authors should provide
reliability coefficients of
the scores for the data
they analyze even when
the focus of their
research is not
psychometric.”
Reliability is the amount
of random fluctuation in
individual scores.
![Page 5: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/5.jpg)
Reliability
“Authors should provide
reliability coefficients of
the scores for the data
they analyze even when
the focus of their
research is not
psychometric.”
Reliability is the amount
of random fluctuation in
individual scores.
“In practice, score
reliability is a matter of
degree, because all
scores include some
random fluctuation.”
![Page 6: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/6.jpg)
Reliability
“It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual.”
Reliability is the amount
of random fluctuation in
individual scores.
“In practice, score
reliability is a matter of
degree, because all
scores include some
random fluctuation.”
![Page 7: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/7.jpg)
Reliability
“It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual.”
“Poor score reliability
may compromise …
score ‘validity’.”
“In practice, score
reliability is a matter of
degree, because all
scores include some
random fluctuation.”
![Page 8: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/8.jpg)
Reliability
“It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual.”
“Poor score reliability
may compromise …
score ‘validity’.”
“Poor score reliability
may compromise the
ability of a study to yield
noteworthy effects”
![Page 9: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/9.jpg)
Reliability
“Poor score reliability
may compromise …
score ‘validity’.”
“Poor score reliability
may compromise the
ability of a study to yield
noteworthy effects”
![Page 10: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/10.jpg)
Reliability
“Scores can’t both measure nothing and measure something.”
“Poor score reliability
may compromise …
score ‘validity’.”
“Poor score reliability
may compromise the
ability of a study to yield
noteworthy effects”
![Page 11: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/11.jpg)
Reliability
“Scores can’t both measure nothing and measure something.”
“Perfectly unreliable scores are perfectly random and cannot yield… significant results.”
“Poor score reliability
may compromise …
score ‘validity’.”
“Poor score reliability
may compromise the
ability of a study to yield
noteworthy effects”
![Page 12: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/12.jpg)
Reporting Reliability
“Scores can’t both measure nothing and measure something.”
“Perfectly unreliable scores are perfectly random and cannot yield… significant results.”
“Poor score reliability
may compromise …
score ‘validity’.”
“Poor score reliability
may compromise the
ability of a study to yield
noteworthy effects”
![Page 13: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/13.jpg)
Reporting Reliability
![Page 14: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/14.jpg)
Reporting Reliability
“Reporting reliability coefficients for one’s own data is the exception rather than the norm...Too few reliability estimates for analyzed data are provided in both journals…and doctoral dissertations.”
![Page 15: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/15.jpg)
Reporting Reliability
“Reporting reliability coefficients for Reporting reliability coefficients for one’s own data is the exception one’s own data is the exception rather than the norm...rather than the norm...Too few Too few reliability estimatesreliability estimates for analyzed for analyzed data data are provided in both journals… both journals…and and doctoral dissertations.”
![Page 16: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/16.jpg)
Reporting Reliability
“The most commonly used (reliability) estimate is Cronbach’s (1951) coefficient alpha ().”
![Page 17: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/17.jpg)
“The most commonly used (reliability) estimate is Cronbach’s (1951) coefficient alpha ().”
number of items (sum of item variances)
number of items-1 test variance
Reporting Reliability
1-
![Page 18: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/18.jpg)
“The most commonly used (reliability) estimate is Cronbach’s (1951) coefficient alpha ().”
“Item score covariance plays an important role in both the numerator and the
denominator of the estimate.”
Reporting Reliability
![Page 19: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/19.jpg)
“The most commonly used (reliability) estimate is Cronbach’s (1951) coefficient alpha ().”
“The intercorrelations of the items are the essential source of this
kind of reliability.”
Reporting Reliability
![Page 20: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/20.jpg)
Validity of Researcher-Made Surveys
![Page 21: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/21.jpg)
Evidence of Validity
![Page 22: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/22.jpg)
Evidence of Validity
• Patterns of Association
![Page 23: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/23.jpg)
Evidence of Validity
• Patterns of Association
• Comparing Results from Different Versions of the Same Question
![Page 24: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/24.jpg)
Evidence of Validity
• Patterns of Association
• Comparing Results from Different Versions of the Same Question
• Comparing Responses to Data from Other Sources
![Page 25: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/25.jpg)
Evidence of Validity
• Patterns of Association
• Comparing Results from Different Versions of the Same Question
• Comparing Responses to Data from Other Sources
• Asking the Same Question Twice and Comparing Results
![Page 26: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/26.jpg)
• Patterns of Association
• Comparing Results from Different Versions of the Same Question
• Comparing Responses to Data from Other Sources
• Asking the Same Question Twice and Comparing Results
Evidence of Validity
Reliability
Reliability
![Page 27: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/27.jpg)
• Patterns of Association
Evidence of Validity
![Page 28: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/28.jpg)
• Patterns of Association• Scores from different measures believed to measure
similar things should correlate. Scores from different measures believed not to measure similar things should not correlate.
• Responses to items believed to represent the same dimensions or factors should correlate.
Evidence of Validity
![Page 29: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/29.jpg)
• Comparing Responses to Data from Other Sources
Evidence of Validity
![Page 30: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/30.jpg)
• Comparing Responses to Data from Other Sources
• Compare to records. Compare to physical testing. Compare to population estimates.
Evidence of Validity
![Page 31: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/31.jpg)
Face Validity of Survey Questions
![Page 32: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/32.jpg)
Face Validity of Survey Questions
• Have a reason for every question you ask.
• Keep questions simple.
• Keep questions precise.
• Avoid leading questions.
• Foresee social desirability.
• Response options should be mutually exclusive and exhaustive.
• Provide temporal frame of reference.
• Use Likert format correctly.
![Page 33: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/33.jpg)
Pilot Testing
![Page 34: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/34.jpg)
Pilot Testing
• Sample size >15
• Discuss questions with respondents to find confusion or ambiguity.
• Pretest sample should resemble study sample.
• Examine variance among respondents.
• Refine answer options.
• Time how long it takes.
![Page 35: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/35.jpg)
Group Assignment
![Page 36: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/36.jpg)
Produce a Valid and Reliable Attitude or Psychological Scale in 90 Minutes
![Page 37: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/37.jpg)
Produce a Valid and Reliable Attitude or Psychological Scale in 90 Minutes
Write a 7 to 10 item scale. 30 minutes
Pilot test your items. 15 minutes Use another group. If necessary, revise your scale based on your pilot testing.
Administer the revised scale to 10 minutes at least 6 people. Collect data.
Enter data on SPSS. 20 minutes Compute coefficient alpha. Revise your scale based on reliability data.
Report. 15 minutes
![Page 38: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/38.jpg)
Entering Survey Data on SPSSItem1 Item2 Item3
Survey1 4 1 5
Survey2 5 1 4
Survey3 4 2 3
Item 1 “I like salt.”
Strongly Disagree Disagree Neutral Agree Strongly Agree
1 2 3 4 5
![Page 39: Reliability and Validity of Researcher-Made Surveys](https://reader033.vdocuments.net/reader033/viewer/2022060106/5518c53255034638098b4b0c/html5/thumbnails/39.jpg)
Analyzing for Reliability in SPSS1. Enter survey data.2. Choose menu options:Analyze
ScaleReliability Analysis
3. Choose and move variables to Items box.4. Click Statistics…Click on: Item
ScaleScale if Item Deleted
5. Read output. Identify items which have an “Alpha if Item Deleted” larger than the scale’s Alpha. Remove those items.
6. Re-run analysis with remaining items until satisfied.