1. The Good, the Bad, and the Ugly:How to design an effective surveyWorkshop26.02.2013Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J

2. The Good, the Bad, and the Ugly:How to design an effective surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 3. Workshop overview10.00-10.15 Introduction to survey design10.15-11.15 How to design an effective surveyWriting good questions11.15-11.30 Morning tea11.30-12.30 Tips and tricksHow to maximise response rateGetting the most out of your data12.30-1.00Q&A and CloseCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 4. IntroductionSo you want to create a survey... What is the purpose of the survey? Needs assessment Impact assessment Knowledge assessment Client satisfaction / program feedback Program evaluation What survey mode will you use? Telephone, online, paper, interviewCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 5. When to use a survey To collect information from a large number ofpeople or records To collect this information relatively inexpensively To say something about the population from whichthe sample was drawn When you do not need very in-depth information When you want the data collection procedure tobe as standardised as possibleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 6. Some advantages of surveys Anonymity Asking sensitive questions Standardised data to monitor trends Quick to administer Simple data analysis (%, frequencies) Data can be presented as graphs/charts Multiple distribution channels (online,paper, social networking sites)Curtin University is a trademark of Curtin University of TechnologyHow to design an effective surveyCRICOS Provider Code 00301J 7. Some limitations of surveys Limited explanations for the data Comments open to a range ofinterpretations Dont know why questions were skippedor why survey was partially completed No opportunity to explain questions thatpeople dont understand Cant guarantee the survey was evenreceived or completed by the right person Basic literacy levels neededCurtin University is a trademark of Curtin University of TechnologyHow to design an effective surveyCRICOS Provider Code 00301J 8. What makes a good survey?Have you ever created a survey and feltdisappointed with the data you collected?Have you ever been asked to answer a survey andfelt irritated with the questions asked? A good survey considers both your needs for dataand the capabilities and experience of therespondent. Collects complete, accurate, and reliable data.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 9. What is a bad survey? Too wordy Language is hard to understand Bad timing - e.g. respondents tired at the end of a session No Other response category Questions that seem irrelevant to the topic Repetitive Boring e.g. same structure throughout, no images Does not consider the capabilities of the respondentsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 10. Is a long survey a bad survey?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 11. HOW TO DESIGN AN EFFECTIVE SURVEY IN 3 EASY STEPS!Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 12. Step 1.Establish your survey goals! Ask yourself: Why am I creating this survey? What do I hope to learn or accomplish? How do I plan on using the data that I collect? How will the data influence my decisions? Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 13. Step 2.Create a dummy tableTo ensure all objectives are being addressed and allquestions address objectives. Consider data analysisbefore, during and after survey development. Areas of inquiryQuestions DemographicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 14. A dummy table Ensures you can interpret data Ensures each question will collect theinformation you need Ensures you leave nothing of importance out Ensures you dont collect more informationthan you needCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 15. Example: Health Care Experience of theLGBTI* Community in WAActivity Find questions related to: Area of inquiry Questions LGBTI health issues Demographics and lifestyle Current experience of healthcare services How and where to reorient current health services *LGBTI lesbian, gay, bisexual, transgender, intersexCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 16. Answers Area of inquiry Questions LGBTI health issues 10,11,24 Demographics and lifestyle1,2,3,4,5,6,7,8,9,15,16 Current experience of healthcare12,13,14,20,21,22,23 services How and where to reorient current 17,18,19,25 health servicesCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 17. Step 3.Write good questionsTry to use a range of question types: Open-ended questions To explore issues Open to a variety of responses (comment boxes) Closed-ended questions Dichotomous Likert scale Guttman scale Multiple choice (one response, multiple responses) Rank orderCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 18. Example - DichotomousCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 19. Example Likert ScaleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 20. Example - Guttman Scale Agreement with any one item implies agreement with the lower-orderitems. E.g. are you willing to permit _______ to... - live in your country? - live in your community? - live next door to you? - marry your child?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 21. Example Multiple Choice (one response)Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 22. Example Multiple Choice(multi-response)Evaluation of Health Promotion to design an effective survey HowCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J382 23. Example Rank Order Rank the following methods of receiving a positive STIdiagnosis in order of preference from highest to lowest(1=your most preferred method; 5=your least preferredmethod) SMS______ Phone call ______ In person ______ Email ______ Letter ______Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 24. Developing a Client SatisfactionSurveyExample: The Wheatbelt Resilience Program Included a series of community workshops A survey was developed to find out how useful theworkshops were in terms of increasing awareness of theWheatbelt Resilience Program; understanding of suicideand suicidal behaviours; and opportunities to help with thisissue in the community Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 25. Can you identifyany ways in whichthis feedback sheetcould be improved?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 26. The questionnairewas adapted tocapture morespecific informationand to also gaugesatisfaction with theworkshop. Thepre-questionnairewas on a separatedocument to thepost-workshopevaluation.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 27. Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 28. WRITING GOOD QUESTIONSCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 29. What do you want to know?What do you need to know? Dont waste the opportunity to collect good qualitydata by asking vague questions Be economical with the number of questions butmake sure you collect all the information required Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 30. Activity: Work in small groupsRe-write these questions and include closedended responses: (a) How often do you participate in physicalactivities on average? (b) When were you last physically active?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 31. What to avoid Double-barreled questions Biased/leading questions Questions with a double negative Loaded questionsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 32. Example: double-barreled questionPlease indicate how much you agree ordisagree with each of the following statementsabout our service:I feel welcomed by staff and other youth at thecentre Example: leading questionNegotiating condom use with a sexualpartner is difficult. Did this session helpyou feel more confident to negotiatecondom use successfully?Curtin University is a trademark of Curtin University of TechnologyHow to design an effective surveyCRICOS Provider Code 00301J 33. Example: double negativeHow do you feel about the following statement?We should not reduce investment in sexual healthservices.Strongly Agree AgreeDisagree Strongly DisagreeExample: Loaded questions Which of the following services most urgently requires improvement?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 34. What to avoid Questions which could create a response set i.e. the tendency for a respondent to answer later questions the same way they answered earlier questions Technical language/jargon Use simple, common words Emotive language Do you approve of the governments oppressive immigration policy?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 35. What to avoid Using and, or, but in the same question Long, complex, multi-part questions Questions which contain a negative and a positiveIf you were in a foreign country and did not speak the language but knewsomeone who did would youa. Let the person that spoke the language speak for you Huh?!b. Try to learn the language on your ownc. Not try to learn the language or let the person that you knew that spokethe language speak for you.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 36. What to avoidResponse categories that are not exhaustiveWhat is your current age? 10 to 2021 to 30 Im 41!31 to 40Response categories that are not mutually exclusiveWhat is your current age?10 or less10 to 20Im 30!20 to 3030 or greaterCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 37. Avoid fuzzy words in the question Those words which are open to differentinterpretation by the respondents. Have you ever had any major injuries during the past year? Avoid words like: important, major, better, regular,often, occasional (subjective interpretation). Best to give specific time & detailsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 38. Irrelevant questions Questions which are irrelevant to the surveyobjectives E.g. do you really need to know someones income? Questions which are irrelevant to the respondent e.g. Dont ask nurses to compare the effectiveness of different patient management techniques if they have no experience of the alternatives.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 39. Presuming knowledge Dont assume people either know or haveappropriate experience to answer thequestion.What isbullying? Have you been bullied this year?What is a How many standard drinks have you drunk in thestandarddrink? last week? How many cigarettes have you smoked in theI dont last week?smoke! Use filter questions or provide moreinformation to avoid these problemsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 40. Presuming knowledge Hypothetical or predictive questions Would you like to try marijuana? Avoid - many people might like to try anything once. Generally avoid: Would you like to... What would you do if..?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 41. Personal questions Be sensitive! May need to preface personal questions to giverespondent permission to respond truthfully Consider the structure of response choices people dont like to consider themselves at extreme ends of a scaleCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 42. Activity: Design a brief survey toevaluate this workshop Workshop objectives Increase participants knowledge and skills in effective survey design Assess participants satisfaction with the SiREN training workshop Areas of inquiryQuestions DemographicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 43. MORNINGTEACurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 44. TIPS AND TRICKSCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 45. Critically assess your surveyquestions Is the respondents understanding of responsecategories likely to be influenced by more thanwords? The vaguer the question and answer categories,the greater potential for error / poor data Is the survey information being collected in morethan one mode? E.g. phone, interview, email Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 46. Critically assess your surveyquestions Is changing a question acceptable to the surveysponsor? Is the question necessary? Will it improve the validity of your study if you doask more than one question?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 47. Critically assess your surveyquestions How easy is it to get an accurate answer? Can people accurately recall and report pastbehaviours? Will the respondent be willing to reveal therequested information? Will the respondent feel threatened or defensiveby the question? Will the respondent feel motivated to answer eachquestion?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 48. Tips and tricks Insert dummy questions These may be irrelevant to the research to avoid respondentsguessing desirable answers Reverse the wording of some survey items This technique can avoid creating response setsI always enjoy going to workI dread going to workCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 49. Tips and tricks The order of questions may influence responses Put interesting questions first Sensitive questions towards end Demographic information location of these questions depends on importance Similar themes together Put instructions for questions before the questionnot at the beginning of the surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 50. Tips and tricks Be specificExample: How can we improve our service?This is too vague and may elicit single responses about manydifferent aspects of the service without much guidance to help youdecide which ones are important.Instead, consider asking respondents to rank from 1-5 the quality ofdifferent aspects of the service What claims will you use the data to support? Be honest and consider if this is realistic or credible. This technique helps you to clarify the wording of questionsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 51. HOW TO MAXIMISE RESPONSE RATECurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 52. Aim to get a 65% response rate That is: 65% of your sample complete andreturn a questionnaire Non-respondents may bias your responses Need to consider response rates when selectingsample size Need to consider attritionCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 53. How to maximise response rate Involve your target group in design if possible Pre-test! Ask questions that people can and will answer Uplift, motivate and encourage the respondents togive information Avoid bad survey questions Allow sufficient time to complete surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 54. How to increase responses Plan multiple contacts Pre-notification (tell people you are planning to survey) Reminders for non-responders Consider incentives Individual incentives more effective than prize draws e.g. incentive given for getting others to complete the survey Dependent on resources Obtain ethics approval for your surveyCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 55. A note about survey ethicsConsiderations may include: Information and consent forms Confidentiality and anonymity Samples which include minors How to ask questions about potentially illegal activities Length of survey / burden on respondents Format of survey - e.g. acceptability, culturally appropriatePlease contact siren@curtin.edu.au for more advice about survey ethicsCurtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 56. GETTING THE MOST OUT OF YOUR SURVEY DATACurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 57. Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 58. Planning and evaluation cycleNeedsassessmentSurvey data Use differentRe-planningProgram types of survey planningdata as part of a continuousSurvey data programSurvey data planning and evaluation cycleEvaluation ImplementationSurvey dataCurtin University is a trademark of Curtin University of TechnologyHow to design an effective surveyCRICOS Provider Code 00301J 59. How will you use the data? Use survey data to inform the program planning cycle Adopt a mindset of continuous program improvement After implementing the survey, critically evaluate thequality and completeness of the data collected Were any questions ambiguous? Were all the questions necessary? Did any questions appear to cause confusion? Ask yourself what Did you get some unexpected responses? will you change? Were the data useful? What decisions could the data inform? Are there still gaps in the data?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 60. Getting the most out of your data Dont just file the survey data away until the next activityreport for your funding provider is due!Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 61. Health promotion short course example survey data How confident do you Pre course Post course11 11 feel to undertake a needs assessment to plan a 3 52 health promotion 0 program?NotConfidentVeryNot Confident Very confident confident confident Confident How confident do you feel to Pre coursePost course 109 contribute to the design and 7 implementation of an 33 evaluation of a health 0 promotion initiative?NotConfident VeryNot Confident VeryconfidentConfident confident confident Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 62. Health promotion short course example survey data Pre coursePost course How confident do you feel to13 12 work collaboratively with your community and with 4 3 other organisations? 0 0Not Confident VeryNot Confident Veryconfident Confident confident confident What could you conclude from these data? What decisions about the course would you make as a result of these data? Are these decisions valid? What other data might you want to collect? Curtin University is a trademark of Curtin University of Technology How to design an effective survey CRICOS Provider Code 00301J 63. Any questions?Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 64. FOLLOW UP SESSIONCurtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J 65. Follow up session SURVEY CLINIC Tuesday 12th March2013, 10am 12 noonCurtin University (Room TBA) In two weeks time! Aim to put what you have learned into practice Either:1.Critically review a survey you have used previously OR2.Design a new survey using what you have learned. Bring your surveys and any questions to the survey clinicCurtin University is a trademark of Curtin University of TechnologyHow to design an effective surveyCRICOS Provider Code 00301J 66. Workshop evaluationPlease draw aroundyour hand (or bothhands) and providesome feedback ontodays workshop.Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J 67. THANKS FOR COMING!Please dont forget to complete the workshop evaluation! Thank you Curtin University is a trademark of Curtin University of Technology How to design an effective surveyCRICOS Provider Code 00301J