portfolios for lifelong competence assessment
DESCRIPTION
Portfolios for Lifelong Competence Assessment. Assessment of Competence - More than One Approach. Traditional (primarily external scrutiny) Emerging (more internal, self-directed) Research (educationally-driven). Assessment of Competence - The Traditional Approach . - PowerPoint PPT PresentationTRANSCRIPT
1
Portfolios for Lifelong Competence Assessment
2
Assessment of Competence - More than
One ApproachTraditional (primarily external scrutiny)Emerging (more internal, self-directed)Research (educationally-driven)
3
Tests standardized where possible to maximize reliability (e.g. simulations - MCQ/ virtual, SP)Narrowly focused on knowledge and reasoningHigh stakes, summativeMay focus on individuals (“assessment”) or programs (“evaluation”)All about minimal competencePreferred approach of regulators (viz. societal protection)
Assessment of Competence - The
Traditional Approach
4
Assessment of Competence - The
Emerging Approach Tests tailored to the actual practice and learning needs of the individualAssessment beyond knowledge (e.g. includes skills and behaviors)Low stakes, formative, abundant feedbackUsually focuses on the individual All about driving continuous improvementImpelled by innate professionalism (internal locus of accountability), not just external scrutiny
5
Assessment of Competence - The Research Approach
Seeking better pedagogies, best practicesBroadening base of assessment, shaping learningFinding effective balance between traditional and emerging approachesFocus may be on individual, but more often programmatic Seeking balance between baseline competence and continuing improvementPrimarily the realm of professional educatorsDriven by need to define effective educational approaches
6
Availability of Data Sources and
Methodologies
For the Traditional approach - GoodFor Emerging & Research approaches - Poor
7
Needed: Access to Many New Data Sets - Some
ExamplesWork learning experience - scope of clinical practice and changes over time Clinical performance - processes of care, outcomes, managementBehaviors and skillsAchievement of competency milestonesLongitudinal research data set
8
Comes the Portfolio aka eFolio - Definition
An mechanism to collect electronic data sets in near real time (i.e. A Process)A particular collection of data and data sets specific to purpose - “Assessment of competence” (i.e. A Thing)Both a Process and a Thing (i.e. A “Proing”)
9
The Good News: Bad News Debate Healthcare professionals leave a massive contrail of electronic data in their daily work
BUT relevant data are very fragmented, scattered across a galaxy of repositories
HIT now allows the potential for sourcing and compilation of data in near real time
BUT the relevant repositories have scant tradition nor appetite for data-sharing
10
The eFolio Connector Vision
Data Commons A collection of Portfolios and Repositories that
can provide data for many different applications
eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by
the learner
Data Sourcing
11
Data Commons A collection of Portfolios and Repositories that
can provide data for many different applications
eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by
the learner
eFolio Connector: Initial Data Sets Planned
eFolio ConnectorSuite of apps compiling data reports for a
variety of viewers authorized by the learner identified
Data Commons Bridging Many Portfolios and Repositories, and providing data for many different applications
Data Sourcing
12
Data Commons A collection of Portfolios and Repositories that
can provide data for many different applications
eFolio ConnectorSuite of apps compiling data reports for the learner, and for other viewers authorized by
the learner
eFolio ConnectorSuite of apps compiling data reports for a
variety of viewers authorized by the learner identified
Some Other Important Issues
Data Sourcing “VirtualRepository”
Reporting out to authorized
viewers specifications
Trusted Agent Hub
13
The “Virtual Repository”: Affiliated RepositoriesSign standard data-sharing agreements (Commons likely to become separate entity) Generically willing to share data, but decide on case by case basis Follow data custodian best practices Expose data in XML, comply with technical specifications for each data setMay consume data, with authorization of the learner identifiedMight charge fee to Apps for data provided
14
More on Trusted Agent HubProvides a backbone of connectivity
across Data Commons infrastructureIncludes array of System Tools (Apps) for
ID matchingDe-identificationAuthenticationBusiness rulesPermissionsTransaction tracking and auditing
Channels of DistributionViewers (authorization by learner or de-identification for identified
data)
Data Commons
Apps and Data Compilations
More on Report-Out to Viewers
Report Out
16
Compiling Data Sets: How Sausage Really Gets
MadeClearly define purpose (e.g. Understanding my practice)Devise necessary schema (e.g. to codify narrative)Do use cases to define content, context, use of data setFind reliable sources of data requiredExecute agreement with repositories to share data Build technical specification to facilitate interoperability Source data from repositories using web servicesPay any agreed-upon fees to repositories providing data
Data Commons
eFolio
Examples of New Data Reports In Development
Data Sourcing
18
What If?There is no current source for data required?
Add needed data to existing repository(ies)Build a new repository, affiliate with Commons
Permanent access to data required, but repository(ies) unwilling or unable to maintain
Arrange with repository to provide archivingBuild central archiving function outside Commons
Learner says that data in a report is wrong?
Build central process for reconciliation of data
19
What if? -2Legal discovery is attempted on one of the data sets generated?
Compiled data sets are expunged by policy immediately after report-out (cf. Mandalas) and no data values are stored in the system
A request is received for identified data from someone other than the relevant learner
No date are moved, compiled or reported out without authorization of the learner
20
Evolution of the eFolio Connector - Feasibility
Work2007 Invitational Conference (sponsored by AAMC, ACGME, FSMB and NBME)2008 Pilot Organizational Meeting; User advisory group established2010 Feasibility Pilot - built ANSI-conformant international technical specification for “Educational Trajectory” data set
21
Evolution of the eFolio Connector: Pre-Implementation
2011At least two more specifications (likely “Assessment of Milestones” and “My Practice”)Marketing, pricing, and business modelPrototype featuring 3-4 reports for feedback
22
ConclusionsThe eFolio Connector can source data from a diverse galaxy of repositories affiliated with the Data Commons and provide potentially useful data reports that
bridge the continuum provide feedback on competence and inform lifelong learning
Reports are available in near real time to learner and any other viewers authorized