![Page 1: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/1.jpg)
Helping Program Assessment Using Automated Text Processing
Anne Gilman
SoTL Brown Bag
25 Sept 2013
![Page 2: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/2.jpg)
9/25/13
Finding questions we hadn't thought to ask...
..in what students say.
![Page 3: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/3.jpg)
Assesment Tools
The Law of the Instrument:
“Give a small boy a hammer, and he will find that everything he encounters needs pounding.”
-Abraham Kaplan
![Page 4: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/4.jpg)
Assesment Tools
Are there tools that can help us ..to articulate goals
we take for granted? ..to point in new
directions?
![Page 5: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/5.jpg)
Cool Hand LIWC
![Page 6: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/6.jpg)
Data
LiM participant essays 57 from 2007-2012
Holistic ratings 23 Director’s Cut Totals:
9 Excellent 11 Good 3 Just barely makin’ it
![Page 7: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/7.jpg)
9/25/13
LiM AssignmentIn a paper of at least two pages, discuss what the Language in
Motion experience has meant for YOU. Be as specific as possible and provide examples where relevant.
Consider the following:A. Why did you choose to participate in Language in Motion? What
personal goals, both short and long term, did you have for this experience? Were they met? If so, how well? If not, why not?
B. In what areas have you improved your skills and/or knowledge?
Consider what you learned about the following:
Your second language, if any, The topic(s) of your presentations, Your home culture, Your second culture, American education, Presentation skills: Organization, ability to give clear directions,
adjustment of content to audience (age and knowledge level), etc. Yourself: Communication or cross-cultural communication abilities, time
management, self-confidence, flexibility, leadership, etc.
C. Was there something you wanted to learn through this
experience that you didn’t? If so, what was it and why did you not learn it? Is there something we should do differently?
![Page 8: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/8.jpg)
9/25/13
What did LIWC say?
![Page 9: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/9.jpg)
What did liwc say?
![Page 10: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/10.jpg)
Next Steps:
Analyze by paragraph Consider value of enjoyment Look under the hood
9/2
5/1
3
![Page 11: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/11.jpg)
Can we look under the hood?
![Page 12: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/12.jpg)
Topic sorting c. 1974
![Page 13: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/13.jpg)
Self-organizing maps c. [guess!]
![Page 14: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/14.jpg)
Are these more familiar?
Hoping the XPS viewer works in our room!
9/2
5/1
3
![Page 15: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/15.jpg)
9/25/13
Sample LiM Wordles
![Page 16: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/16.jpg)
9/25/13
What's a trigram?
Demo
![Page 17: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/17.jpg)
Results: Cluster-to-Category Match9
/25
/13
Rating Cl-1 Cl-2 Cl-3
Adequate 0 2 1
Good 6 2 3
Excellent 6 1 1
Better: Talking less about
your own country zzt, zto, top
TBD: Less: tha More: ent, eth
![Page 18: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/18.jpg)
9/25/13
Next Steps
Get more ratings Add more trigrams Compare to LiM, ILAC goals Consider POEs
![Page 19: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/19.jpg)
9/25/13
Many Many Thanks
LiM Director: Dr. Deb Roney Gilman Lab Assistants
LIWC project: Helen Hu, Aric Koestler, Olivia Moody, Sungouk Park, Seth Weil
Trigram project: Tori Rehr, Tori Buser Dr. Kim Roth The Juniata Department of Psychology The SoTL Leadership & the Lakso Fund
![Page 20: Helping Program Assessment Using Automated Text Processing](https://reader036.vdocuments.net/reader036/viewer/2022062423/568143da550346895db066d6/html5/thumbnails/20.jpg)
Gory Details > aggregate(limtrunc[-1], by=list(cluster=fittrunc.km$cluster), mean) cluster age all and ati den 1 1 0.002255296 0.002815929 0.007260874 0.006236264 0.002169403 2 2 0.004112582 0.002038445 0.006445021 0.007287239 0.003207497 3 3 0.001498536 0.002247922 0.007557460 0.006659985 0.004155337 ent ere ers ese est 1 0.007423606 0.003938757 0.002885631 0.003315418 0.003273650 2 0.010172528 0.003758447 0.001438849 0.005238279 0.003749762 3 0.010626493 0.004926549 0.002321635 0.004856809 0.003323920 eth eve for. hat her 1 0.003958753 0.002694240 0.003256768 0.005716957 0.003714643 2 0.002908776 0.002441420 0.002277172 0.004412824 0.002856234 3 0.003351811 0.002440801 0.003763736 0.002332821 0.003069785 ing ion zto zzt ver 1 0.008014828 0.005659334 0.001344568 0.001344568 0.003019872 2 0.008223715 0.009158820 0.001807482 0.001807482 0.001872784 3 0.005631146 0.006514531 0.004178863 0.004178863 0.003330204 ter tha the thi tio 1 0.002892176 0.005949131 0.01666473 0.003739603 0.005432872 2 0.002505902 0.004485792 0.01535570 0.003083640 0.008802554 3 0.002784424 0.003212007 0.01513814 0.004036677 0.006142843 top tth nts ons opi 1 0.002193457 0.003513021 0.002240106 0.002399789 0.001670029 2 0.002822616 0.002456230 0.002990221 0.002544836 0.002204109 3 0.005067310 0.002195498 0.003912161 0.002463628 0.004443421 ore pia pre rea res 1 0.002220215 0.001284905 0.003154103 0.003108067 0.004400721 2 0.001488311 0.001586731 0.005186367 0.002198381 0.005775334 3 0.002816267 0.004117230 0.004174962 0.003289315 0.005404926 sth 1 0.002374196 2 0.002569780 3 0.002009747