educational standard correlation services for digital libraries
DESCRIPTION
Educational Standard Correlation Services for Digital Libraries. Holly Devaul Jonathan Ostwald John Weatherley Digital Learning Sciences UCAR. Anne Diekema Jennifer Bailey CNLP Syracuse University. Presentation Overview. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/1.jpg)
Educational Standard Correlation Services for Digital Libraries
Holly DevaulJonathan OstwaldJohn Weatherley
Digital Learning SciencesUCAR
Anne DiekemaJennifer Bailey
CNLP Syracuse University
![Page 2: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/2.jpg)
Presentation Overview
We will discuss a suite of educational standards assignment and alignment services that support cataloging and discovery of educational standards information in the NSDL – First half - how these services work, scenarios for
use, and expected performance levels based on recent evaluations
– Second half - more technical in nature; aimed at software engineers and web developers focusing on how the APIs work
![Page 3: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/3.jpg)
CASAA- Computer Assisted Standards Assignment and Alignment
NSF NSDL Service project with two goals Support the assignment of standards to
learning resources– Content Assignment Tool (CAT)
Provide a cross-walk between the different state and national standards – Standards Alignment Tool (SAT)
![Page 4: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/4.jpg)
Motivation
No Child Left Behind Act (NCLB) Teachers must document how their teaching is
aligned to standards States required to develop assessments that
evaluate student achievement relative to standards
Teachers want access to standards information for accountability reporting
Opportunity for digital libraries to provide this service
![Page 5: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/5.jpg)
Content Assignment Tool (CAT)
Natural Language Processing technology
assesses resource content, suggests relevant
educational content standards
Semi-automatic approach– CAT suggests relevant standards to the user– User makes final determination
System learns from vetted assignments Web tool plus REST API for easy integration with
other applications
![Page 6: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/6.jpg)
Web-based access to CAThttp://cat.nsdl.org:8280/casaa
User name- guest
Password- p
![Page 7: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/7.jpg)
Integration with theDigital Collection System (DCS)
Accesses resource content via URL
Utilizes data from catalog record -url, grade level
-description, keywords (optional)
Is configured to query only NSES
![Page 8: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/8.jpg)
Suggestions provided in context of cataloging
![Page 9: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/9.jpg)
Visual cues help track suggested and human selected standards – human makes final decision on what to assign
Sta
nd
ard
s
Selected Suggestion
Selected but not suggested
Suggestion
![Page 10: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/10.jpg)
CAT Evaluation
Compared human-assigned NSES standards to CAT suggested standards
Research questions:– Does the CAT Service return standards that a
human would choose for assignment? – Do the standards human catalogers have
chosen occur near the top of the ranked list of CAT Service suggestions?
– Does the inclusion of metadata improve standards assignment via the CAT Service?
![Page 11: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/11.jpg)
National Science Education Standards (NSES)
Hierarchical in structure, classified by grade band and content area and can be grouped as follows:– inquiry - (nature of science; Standard A)– subject - (life, physical, Earth and space
science; Standards B, C and D)– applied - (science and technology, personal
and social perspectives, and the history and
nature of science; Standards E, F, and G).
![Page 12: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/12.jpg)
Does the CAT Service return standards that a human would choose for assignment?
CAT Service suggestions include 28-51% of the standards humans assigned
This offers a time savings in the initial assignment of the standards by presenting the cataloger a smaller subset of standards to choose from, that includes a substantial number of the preferred standards.
![Page 13: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/13.jpg)
Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions?
Analysis was conducted to assess the placement of the human-assigned standards in the ranked list of suggestions from CAT
If metadata included, 83% of the standards suggested at rank 1 were also chosen by a human; then drops off quickly
Subject standards performed better than applied or inquiry
We speculate that the language used in the resource corresponds better with subject standards than with applied or inquiry which require more subjective interpretation
![Page 14: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/14.jpg)
Does the inclusion of metadata improve standards assignment via the CAT Service?
YES- if no metadata included, 60% of the standards at rank 1 were also chosen by a human
In addition, including metadata expands the utility of this tool to suggest standards for non-text-based resources that have descriptive metadata associated with them
See technical paper for full details of this analysis
![Page 15: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/15.jpg)
Standards Alignment Tool (SAT)
Dynamic automatic correlation of standards Combination of various algorithms to find
equivalent (correlated) standards Allows great flexibility for indexing and
search Takes standard text or ASN standard PURL REST API for easy integration with other
applications/systems
![Page 16: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/16.jpg)
Web-based access to SAThttp://sat.nsdl.org:8380/casaa-sat/
User name- guest
Password- p
![Page 17: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/17.jpg)
SAT-DLESE Discovery integration
Testbed collection with NSES standards cataloged at 4th level
Select state of interest to view corresponding state standards side-by-side
Proof-of-concept for technical integration; alignments not yet vetted
![Page 18: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/18.jpg)
http://sat.dlsciences.org
![Page 19: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/19.jpg)
State standards are displayed side-by-side national standards
![Page 20: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/20.jpg)
SAT Evaluation
Looked at NSES to state mapping for five states (OH, MA, MN, CO, NY)
NSES standards distributed proportionately across inquiry, subject and applied subgroups and across grade level (n=49)
Queried SAT for state alignment to NSES Evaluated first 5 suggestions for each
standard (strong, moderate, weak correlation) or none at all
![Page 21: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/21.jpg)
Preliminary analysis
Issues– Lack of equivalent state standards – Differing granularity of scope of content
A mechanism to better discriminate no match and multiple matches is needed
For this alignment (NSES-state) human mediation is still necessary (as with CAT)
Other alignment pathways (state-to-state) need to be evaluated to get the full picture
Additional machine learning may improve results
![Page 22: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/22.jpg)
Part Two – The Technical Bits
Aimed at software engineers and web developers who wish to utilize the API’s and integrate CAT and SAT functionalities with existing applications
![Page 23: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/23.jpg)
CAT API in DCS: Topics
CAT / DCS Interaction The Educational Standards Hierarchy API call: suggestedStandards Constraining the search for suggestions Other uses of API in DCS RPC vs REST Services
![Page 24: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/24.jpg)
DCS Communicates with CAT via Web Service
![Page 25: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/25.jpg)
Selectable / Suggested Standards are leaf nodes
Leaf
Standards Hierarchy
![Page 26: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/26.jpg)
suggestStandards request
CAT search for suggestions guided by info supplied to the API
Returns StandardsList suggested by the CAT Service
CATDCS
Urls, Constraints
StandardsList
•ASN Identifier•Benchmark
![Page 27: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/27.jpg)
Constraining Suggestions
author – (NSES) topic – (Science) maxResults – (10) startGrade endGrade keywords
– Description– (weighted) keywords
![Page 28: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/28.jpg)
RESTRepresentational State Transfer
![Page 29: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/29.jpg)
CAT Service Response
![Page 30: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/30.jpg)
CAT API Requests
suggestStandards – Suggests aligned standards for a specific lesson plan
saveStandards – Save standards to a specific assignment
getStandard – Retrieve the details for a specified standard
getStandards – Retrieve the details for a specified list of standards
![Page 31: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/31.jpg)
CAT API Requests (continued…)
removeStandards – Remove standards from a specific assignment
getAssignments – Retrieve the standards for a specific assignment
getAllAssignments – Retrieve all assignments (including the standards for each assignment) for a specified user
getAuthors – Retrieve all assignments (including the standards for each assignment) for a specified user
getTopics – Retrieve a list of all topics available in CAT
![Page 32: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/32.jpg)
SAT Service: Topics
DLESE use case: Display state standards that are aligned to NSES
API call: suggestedStandards Displaying the suggested standards in the
DLESE interface SAT API overview Development tips
![Page 33: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/33.jpg)
DLESE Request for State Standards
User selects a state in the DLESE UI DLESE provides ASN identifiers for NSES
standards– suggestStandards request accepts ASN identifiers
or text SAT returns a list of suggested standards as
XML– Response contains text suitable for display in the UI
DLESE UI displays the standards to the user
![Page 34: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/34.jpg)
DLESE Communicates with SAT via Web Service
SuggestStandardsREST Request
Standardslist returned as XML
SATService
DLESELibrary
Application
• ASN Idor
• Text
![Page 35: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/35.jpg)
NSES Standards Supplied by DLESE, State Standards Returned by SAT
NSES standards supplied by DLESE as ASN Ids
State standards returned by SAT service as text
![Page 36: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/36.jpg)
suggestStandards request
One or multiple standards can be supplied in the request
Multiple standards handled one of two ways:– Combine the text of all the standards into a single
query and return a single result set - or -
– A query for each standard and return as many result sets as there are standards
DLESE used the latter– DLESE has multiple standards cataloged for each resource– Display best match for each standard
Supply grade range to refine results
![Page 37: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/37.jpg)
SAT suggestStandards Response
![Page 38: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/38.jpg)
DLESE Display
State standards displayed side-by-side with NSES standards
When a state is selected by the user, request is made asynchronously using AJAX– Provides time for SAT to respond and gives the
user feedback with a spinner graphic
Selected state is remembered throughout the user’s session
![Page 39: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/39.jpg)
SAT API Requests
suggestStandards – Suggests aligned standards for specific standard
saveStandards – Save standards to a specific assignment
removeStandards – Remove standards from a specific assignment
getAssignments – Retrieve the standards for a specific assignment
getAllAssignments - Retrieve all assignments (including the standards for each assignment) for a specified user
![Page 40: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/40.jpg)
SAT API Requests (continued…)
getAuthors - Retrieve a list of all authors (states) available in SAT
getTopics - Retrieve a list of all topics available in SAT
getStandard - Retrieve the details for a specified standard
getStandards - Retrieve the details for a specified list of standards
![Page 41: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/41.jpg)
Development Tips
Use AJAX to retrieve and display standards at point of request
Retrieve and cache standards alignments ahead of time if needed
Use JSP/PHP to retrieve and display the results in a Web setting
![Page 42: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/42.jpg)
REST Service URLs
CAT http://cat.nsdl.org:8280/casaa/service.do
SAThttp://sat.nsdl.org:8380/casaa-sat/service.do
![Page 43: Educational Standard Correlation Services for Digital Libraries](https://reader036.vdocuments.net/reader036/viewer/2022062409/568148ad550346895db5c09a/html5/thumbnails/43.jpg)
Contact us for more information
Anne Diekema [email protected] Holly Devaul [email protected] Jonathan Ostwald [email protected] John Weatherley [email protected]