assessing an integrated course, content, library and eportfolio system nick laudato barbara frey dan...
TRANSCRIPT
Assessing an Integrated Course, Content, Library and
ePortfolio SystemNick LaudatoBarbara FreyDan Wilson
Richard P. OravetzCenter for Instructional Development
& Distance Education
University of PittsburghCopyright © 2005 Nicholas C. Laudato, Barbara Frey, and the University of Pittsburgh. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the authors.
Agenda Background
University of Pittsburgh ePortfolio Committee Bb Content System Assessment Project
Approach Findings Observations Conclusions
Pittsburgh
GreensburgJohnstown
Bradford
Titusville
University of Pittsburgh
Founded in 1787Five campuses26 Academic CentersGraduate,
undergraduate, professional, and continuing education programs
30,030 Fall-term FTE 4,533 full- and part-time
faculty members
Blackboard at Pitt Pitt was an early
adopter (CourseInfo V 1.0)
Training required for a faculty control panel
Adopted by all schools and campuses
20-40% annual increase in adoption
Reaches 75% of student population
216
839
1,197
1,630
2,329
3,219
3,804
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
AY1999
AY2000
AY2001
AY2002
AY2003
AY2004
AY2005
Course Sections using Blackboard
Background: ePortfolio Committee
Initiated in December 2003 Goals:
Identify needs and interests Establish criteria for an e-portfolio
system Discuss policy and procedure issues and
make recommendations to the Provost Examine major software options to
assess their capabilities and potential viability
What are ePortfolios?
An electronic portfolio is a collection of multimedia objects and artifacts that provides a record of, and reflection on, activities and achievements, and can be accessed differentially for multiple purposes and reviewers
- University Committee on Electronic Portfolios
Motivators for the Project Recognized need to
Gain real experience with ePortfolios Generate real sample ePortfolios
Interest in extensions to Blackboard Improve the management of course
content Multiple sections of the same course Sharing materials across multiple instructors
Provide an eReserves capability integrated with Blackboard courses
The Blackboard Content System
Content Management - sharing and reusing files across sections, courses, and departments
ePortfolios - organize, document and share information for assessment, advising, career planning, and academic growth
eReserves - incorporate library resources directly into online course materials
Virtual hard drive - network file storage space easily accessible from any computer at any time
BbCS Assessment Project Goals
Evaluate the Blackboard Content System (BbCS) as a tool for Enabling faculty to manage their own
course content Supporting multiple sections of a course Supporting school, departmental, and
program libraries of course materials Supporting e-Reserves Metadata schemes and search
capabilities
… BbCS Assessment Project Goals
Evaluate ePortfolio for use by Students Staff/faculty for career and
professional development Assess the Bb Content System
Storage and processor requirements User support requirements
BbCS Assessment Project Approach
Participants Faculty Librarians Instructional Designers and Technologists Students
“Kick the Tires” Assessment Measures
Interviews and Surveys Student Group Project: RFP Evaluation
BbCS Assessment Project Timeline
Preliminary planning in March 2005 Planned to address summer 6-week sessions
Acquired access to the system on April 11, 2005 pitt.blackboard.com
Summer Session began May 9, 2005 Included 12-week sessions
Originally targeted to end midterm (June 30, 2005)
Extended through the end of the summer term (end of August)
Project Participants Courses
Management Information Systems, Laudato and DeSantis
Information Systems Analysis and Design, Zhou CNS Clinical Practicum I, Henker Retrieving Information, Knapp Individual Research I, Tomer and Alman
ePortfolios Students in Career Services and in MIS course Faculty and staff within CIDDE
Findings1. Managing Course Content2. Support for Multiple Sections3. Libraries of Course Materials4. eReserves5. Metadata and Search6. Student ePortfolios7. Professional Development Portfolios8. Support Requirements
Personal, Course, Institutional,
and Library ContentMyContent
InstitutionContentLibraryContent
Course SectionCourseContent
What Users Said … Library and Information Science
Faculty “Tested WebDAV on both PCs and
Macs with positive results.” “Faculty will have to work smarter to
reap the benefits of the content system. An object oriented repository will have a direct impact on changing the way we do business.”
What Users Said … Nursing Faculty
“With WebDAV, I can get to my content from anywhere in the world.”
“Nursing courses build on one another, so being able to see what previous instructors have taught is valuable.”
Information Science Faculty “Using the Content System is not
worth the effort if you are only teaching a course one time.”
What Students Said … “The content system is easy to access …” “… good way to gather and manage
scattered data in order to create academic, personal, career documents.”
“Most people are familiar with the Windows environment as well as web browsers. Being web-based, Blackboard is an excellent choice to provide these things.”
“I believe this is a revolution …”
What Students Said …
VS S N D VD NA
6. the WYSIWYG editor. 0% 0
10%
1
30% 3
30%
3
30%
3
0% 0
7. the file storage features of the Blackboard Content System.
30%
3
60%
6
10% 1
0% 0
0% 0
0% 0
8. WebDAV. 10%
1
50%
5
30% 3
0% 0
0% 0
10%
1
9. the metadata component of the Blackboard Content System.
10%
1
40%
4
50% 5
0% 0
0% 0
0% 0
Please rate your overall satisfaction with ...
Bb Course with Content System
MyContent
CourseContent
InstitutionContent
LibraryContent
Metadata
Course Side Content Side
Course Archive?
Pre-App Pack 3 Interface to BbCS
Content System Content
Note: This is actually an App Pack 3 screen shot
2: Support for Multiple Sections
Without the Content System Managing two
sections of the same course requires two section maintenance efforts
Faculty merge rosters to avoid this work
What is Content to the BbCS?
In Bb, a course folder or learning unit contains: Text Images Multimedia
Content Interactive Content Assignments Quizzes Surveys
What We NeedCourse Side
We must be able to embed a complete learning object (course folder or learning unit) into the content system
Content System Side
3: Libraries of Course Materials
Tested capabilities with mock school and departmental libraries and a sample library of images for use in teaching languages
Faculty speculated that “Core courses by different instructors can
share an area and offer students similar experiences and knowledge bases”
“It is important that courses build on the foundation of previously-created courses”
Unable to evaluate with real case
eReserves for a Course Section
Library Content
eReserves
Course Section 1
Course Section 2
Course Section 3
Electronic Journal 1
Electronic Journal 2
Electronic Journal 3
5: Metadata and Search Bb provides four metadata schemas:
general, IMS, Dublin Core, and custom Uploaded an image collection and
populated it with custom, IMS, and Dublin Core values
Limited experience with this function High cost of creating the metadata
along with the single-use nature of this assessment project did not justify the effort
What Users Said … Information Science Faculty
“Cannot simultaneously search across multiple metadata schemes”
“Can neither import nor export metadata”
“Data entry is tedious and error-prone” Library and Information Science Faculty
“Metadata capabilities are good, but faculty will not use it unless it can be imported.”
6: Student ePortfolios Students in a graduate MIS class
Assigned two portfolios and directed to use the content system
Competency file Job search tool
Assigned a group project to evaluate the Bb Content System against ePortfolio functional and technical requirements
Student workers in Career Service office
What Students Said About ePortfolios…
“… it is like we have a goal and direction we can follow.”
“… would help incoming students save and organize academic documents ... it seems like it will be able to deliver these things.”
“… allows for a cohesive view of academic achievement that can be maintained throughout the academic career and beyond.”
What Students Said …
VS S N D VD NA
1. the e-Portfolio component of the Blackboard Content System.
0%0
90%
9
10% 1
0% 0
0% 0
0% 0
2. the e-Portfolio creation wizard.
20%
2
50%
5
10% 1
10%
1
0% 0
10% 1
3. the ability to download an e-Portfolio.
30%
3
40%
4
30% 3
0% 0
0% 0
0% 0
4. the ability to share an e-Portfolio with internal users.
40%
4
60%
6
0% 0
0% 0
0% 0
0% 0
5. the ability to share an e-Portfolio with external users.
20%
2
60%
6
10% 1
0% 0
0% 0
10% 1
Please rate your overall satisfaction with ...
The MIS Group Assignment Two independent groups assigned a
project to assess the ePortfolio “… each group will be asked to assess
the ePortfolio component of the Blackboard Content System.”
Given the requirements portion of the final report of the ePortfolio Committee 59 functional and technical
requirements
Degree the Requirements Were Met
Requirement CategoryRequirement Achievement
Percentage
Access 100%
Other features 100%
Review and Feedback 91.7%
Content 89.3%
Usability 87.5%
Reporting and Output 86.1%
Technical Requirements 79.2%
Training and Support 71.4%
Overall Achievement 86.4%
Results of Requirements Analysis
Critically Important
4
42 12, 56 7, 18, 43, 51, 55, 57, 58
1, 8, 9, 19, 20, 21, 22, 23, 25, 26, 28, 29, 30, 31, 34, 39, 40, 48, 53, 54
Moderately Important
3 17, 43 11, 36, 44 16, 50, 52 2, 3, 5, 6, 10, 13, 14,
15, 24, 27, 32, 35, 37, 38, 41, 45, 46, 47
Somewhat Important
2 4 59 33
Not Important at
All1
1 2 3 4
Not Achieved
At All
Somewhat Achieved
Moderately Achieved
Fully Achieved
7: Professional Development Portfolios
Nursing Faculty Member “I love the ePortfolio for globally
collaborating with colleagues to share articles, files and clinical websites.”
Instructional Designer “Users take responsibility for building
the matrix as a professional development tool that demonstrates mastery of a list of criteria. The WYSIWYG editor needs work, but the matrix concept is valuable.”
8: Support Requirements Adopted some Bb tutorials and created
Camtasia recordings For Course
Content system complexity mandates some faculty training
For ePortfolios MIS students received a 10 minute
orientation Career services students received an
hour orientation and telephone support
… Support Requirements Processor and memory requirements
Unable to measure Storage requirements
Default quotas much too low for faculty Increasing defaults in the system does
not impact existing users (get it right up front)
Count on at least 100 MB per account, with some up to 1 GB
Plan for ongoing training and support
Observations The content system must be even
further integrated with the course side Must be able to embed course objects,
such as folders containing quizzes, into the content system
App Pack 3 upgrade was a move in the right direction
Course archive must address content areas
WYSIWYG editor is not WYSIWYG
… Observations Student response to ePortfolio
exceeded our expectations Had not considered it “best of breed”
Bb templates are NOT portfolio templates but rather are Web page templates Our building block allowed us to create
ePortfolio template functionality ePortfolios need additional controls and
capabilities to handle reflections and institutional components
… Observations Metadata functionality needs to be
improved Easier more effective data entry Import and export functions Ability to simultaneously search across
metadata schemes (against related attributes)
What Students Said …
SA A N D SD NA
1. An e-Portfolio system should be available on an ongoing basis to support your academic work.
40% 4
50% 5
0% 0
10% 1
0% 0
0% 0
2. The e-Portfolio component of Blackboard should be available on an ongoing basis to support your academic work.
30% 3
50% 5
20% 2
0% 0
0% 0
0% 0
3. The Blackboard Content System should be available on an ongoing basis to support your academic work.
50% 5
30% 3
20% 2
0% 0
0% 0
0% 0
4. Creating an e-Portfolio is a valuable academic activity.
40% 4
50% 5
10% 1
0% 0
0% 0
0% 0
5. The e-Portfolio component of Blackboard contributed to your learning.
30% 3
50% 5
20% 2
0% 0
0% 0
0% 0
Please rate your level of agreement with the following statements.
Conclusions The Blackboard content system is still
new and evolving, but worth the effort The assessment team is
recommending that the University acquire the content system