welcome to the cirtl network’s teaching-as-research capstone seminar
DESCRIPTION
Survey, Data Collection, Instruments and Questionnaires. Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar. Dr. Mary Elizabeth Besterfield - Sacre. Session begins at 3 -4 ET/2-3 CT/1-2 MT/12-1 PT When you join the room please run the Audio Setup - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/1.jpg)
www.cirtl.net
Welcome to the CIRTL Network’sTeaching-as-Research Capstone Seminar
SURVEY, DATA COLLECTION, INSTRUMENTS AND QUESTIONNAIRES
Director, EERC
Associate Professor and Fulton C. Noss Faculty Fellow
Department of Industrial Engineering
University of Pittsburgh
Session begins at 3-4 ET/2-3 CT/1-2 MT/12-1 PT
• When you join the room please run the Audio Setup
Wizard: Tools Menu->Audio->Audio Setup Wizard
• While we wait for the session to begin, please
feel free to test your microphone and webcam
• If you are experiencing problems and/or have
questions, please type into the chat window
Dr. Mary Elizabeth Besterfield-Sacre
![Page 2: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/2.jpg)
www.cirtl.net
Ways to Interact during theTAR Working Session
• Turn on/off your microphone:
• Raise your hand if you have a question or comment
• Turn on/off your video:
• Use the chat window to add comments, ask questions, or request help
![Page 3: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/3.jpg)
Survey, Data Collection Instruments,
QuestionnairesMary Besterfield –Sacre
Director, EERCAssociate Professor and Fulton C. Noss Faculty
FellowDepartment of Industrial Engineering
University of Pittsburgh
![Page 4: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/4.jpg)
Once Upon A Time…• I was a grad student…• My research focus– Industrial statistics– Quality assurance
• Finished one NSF grant in healthcare and put on another
• Engineering education– Improving the freshman engineering
curriculum – integrated curriculum– I pouted…
![Page 5: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/5.jpg)
Integration of my skills to solve an engineering education problem
• The effort needed a means to determine if change had occurred in students– Exam questions– Projects– Attitudes and perceptions of engineering– Retention in engineering
• I set up my first evaluation plan…– Objectives, strategies, measures of the
objectives
![Page 6: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/6.jpg)
Thought it would take a weekend…
• First researched the area of student attitudes and students leaving the STEM field– Seymour and Hewitt – Talking About Leaving– Others in the STEM areas– Search for other instruments
• None existed• Got a great survey book - Dillman• Developed an initial survey
– Some closed ended responses based on the research, but two critical open-ended responses• Things that enticed students about engineering• Things that turned students off from engineering
• Piloted the 1st survey on Soph, Jr and Sr
Content Validity
![Page 7: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/7.jpg)
Thought it would take a weekend…
• Initial data analysis– Reliability analysis – Crombach’s alpha– Exploratory factor analysis – some hypotheses yes;
others no– Grounded theory approach to open-ended questions –
created new items• Revised the questionnaire
– Conducted verbal protocols or cognitive interviews• Revised the questionnaire – again• Obtained my first IRB Human Subjects
clearance
![Page 8: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/8.jpg)
Thought it would take a weekend…
• Launched it on Freshman engineering class– “Old” programming
• Launched it on incoming Freshman engineering class– Incoming pre attitudes about engineering– Again, at the end of their Freshman year • “New” programming
![Page 9: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/9.jpg)
Thought it would take a weekend…
• Data analysis– Confirmatory factor analysis– Establishment of weights based on the
factor analysis– Item reliability analysis - again
![Page 10: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/10.jpg)
![Page 11: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/11.jpg)
What did we learn?• Differences between the two cohorts of freshmen
– Besterfield-Sacre, ME, Atman, CJ, and Shuman, LJ. “Engineering Student Attitudes Assessment.” Journal of Engineering Education 87, no 2 (April 1998): 133-141.
• Prediction of freshmen who will leave engineering– Besterfield-Sacre, M.E., C.J. Atman and L.J. Shuman. “Characteristics of
Freshman Engineering Students: Models for Determining Student Attrition and Success in Engineering.” The Journal of Engineering Education 86, no 2 (April 1997): 139-149.
• My first grant– Besterfield-Sacre, M.E., M. Moreno, L.J. Shuman, and C.J. Atman.
“Gender and Ethnicity Differences in Freshmen Engineering Student Attitudes: A Cross-Institutional Study.” Journal of Engineering Education 90, no 4 (October 2001): 477 - 489.
Criterion-related Validity - Concurrent Validity
Criterion-related Validity – Predictive Validity
Criterion-related Validity - Concurrent Validity
![Page 12: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/12.jpg)
Since then…• People still ask for it!– I get about 5-8 schools ask per year
• A group at ASU redid the work a few years back to ‘update’ the instrument
• Construct Validity?– Demonstrates an association between the
survey and the prediction of a theoretical trait
![Page 13: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/13.jpg)
Your questions, issues, challenges?
![Page 14: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/14.jpg)
www.cirtl.net
THANK YOU FOR JOINING TODAY’S SESSION
What did you think of today’s session?
Please take a minute right now to fill out a 1-question survey on today’s experience:
We very much appreciate your feedback.
![Page 15: Welcome to the CIRTL Network’s Teaching-as-Research Capstone Seminar](https://reader035.vdocuments.net/reader035/viewer/2022062323/5681665e550346895dd9e3b8/html5/thumbnails/15.jpg)
www.cirtl.net
UPCOMING TAR CAPSTONE SESSION
Working Session: Qualitative and Quantitative Data Management and Analysis
March 17, 20143-4 ET/2-3 CT/1-2 MT/12-1 PT
Facilitated by:Mark Urban-Lurain,
Associate Professor, Center for Engineering Education Research, Michigan State University
To sign up to hear about these and other CIRTL events, email [email protected].