September 30, 2010Recommendation – Villanova University’s LMS Replacement

Over the last two years, a faculty pilot group, UNIT staff and student volunteers have actively researched and tested the various LMS applications. A total cost of ownership model was developed, surveys were completed, collaborative efforts with other schools, and testing of the vendor/product options were all part of the evaluation/due diligence process used to make a good decision for not only the University as a whole but for the individual faculty and students with varied teaching and learning needs. It is significant to note that for Villanova, neither cost nor the core feature sets stood out as the biggest differentiators between the competing LMS products, but the extra features that add enhancements to our current platform that will provide more diverse tools and capabilities in the learning environment.

During the process there was discussion about the possibility of Villanova moving towards an open-source solution for the next LMS product. After review by the faculty, it was determined that the current release of the more widely used open-source product (Sakai) does not provide the broad spectrum of features required by the faculty as a whole. The next release from the Sakai community is expected in early 2012. This release appears to meet many of the functional requirements cited by the faculty. This could potentially be a longer range strategy to be considered by the University.

In the final discussion, the main factor considered by faculty was the added functionality to the environment - new innovations without losing the features of the current LMS. Support for mobile devices, more flexibility with testing, easier to use interface, etc., were some of the discussion points that lead to the final recommendation. While all the evaluated products did retain most of the core functionality required, the Blackboard Learn release is the only application that provided enhancements and also supported the integration of the many third party products that are currently accessed through WebCT and used across campus. Thus the Blackboard Learn system, with its familiar look and feel and significant improvements, is the recommendation from the faculty as the outcome of this research process.

The attached report will be submitted to the Governance Committees for final approval.

1

The Next Step: Villanova’s Learning Management SystemPrepared by UNIT’s Instructional Technologies

This paper summarizes Villanova University’s search for its next Learning Management System (LMS). It describes the needs assessment process that the University undertook to gather requirements, the survey process for comments and feedback and other associated documents that resulted, and the process by which an LMS was chosen. In the final phase, three Learning management Systems were considered as possible alternatives: Blackboard and Desire2Learn, both leading proprietary systems, as well as Sakai, open source initiative. This paper summarizes all of the activity that occurred during this process and the implications of the decision for Villanova.

Introduction

During the academic year 2008-2009, Instructional Technologies began the process to determine what LMS should replace the current WebCT system. For a brief history, the University first contracted with WebCT in 1998. The company has proven to be a good partner with its continual focus on teaching and learning. The University benefited from having an extensible learning platform and predicable small increases in licensing costs throughout the years. In October of 2005, Blackboard purchased the WebCT company in an attempt to merge the best-of-breed in learning management systems. Once again this relationship has proven to be beneficial to the University. Unit has been able to continue to provide a very cost-effective learning management system by leveraging our relationship with the vendor and using our current infrastructure to deliver an enterprise learning system with a wide variety of 3rd party teaching tools for faculty and student.

In 2008 we renewed our contract and agreed to test a learning adaptor for Blackboard. This agreement enabled us to have a three year freeze on our license cost while awaiting delivery of a next-generation product as promised by Blackboard. In 2011 our contract with Blackboard for the WebCT product ends. This end of contract has been coordinated to give the University sufficient time to make a thoughtful and diligent decision in the selection of our next LM system.

Over the years we have seen a tremendous extension of the WebCT platform from the delivery of thousands of journal articles which are part of the Library’s eReserve to departmental training of staff and the development of certificate programs. In reviewing usage statistics from year to year over 9300 students are enrolled in a least one WebCT course each semester; we see usage of faculty as high as 62% with an average of approximately 55% on a yearly basis. Many faculty deliver video and audio content through WebCT which enables them to have copyrighted material in a password protected environment thus keeping them in compliance with copyright law. Several faculty are now using an ePortfolio system which is also integrated with the WebCT environment. In addition to the collaborative, testing and tracking tools which come packaged as part of WebCT, faculty also routinely use “powerlinks” (3rd party applications integrated with WebCT). These include Respondus, a test generator and security testing package; SafeAssign – a plagiarism detection program; iTunes University and Wimba for synchronous collaboration. Other third party applications have also been added into the environment to address specific needs of individual faculty or groups of faculty, e.g., Scholar, etc.

As we move forward, several drivers will determine the direction the University will take in pursuit of a new learning management system. The primary one, as previously mentioned, is the software is at end-of-life. The company will no longer develop new tools for the software platform. Blackboard will also at the end of year 2012 no longer support the WebCT product. This means that any further updates for product fixes to the current tools, security patches, etc.., will not be released. As we look at advances in technology over the last few years, such as mobile computing, improvements in collaborative tools, drop and drag functionalities, etc. , unfortunately faculty and students will not have access to these tools in the learning environment until we move on to a new platform.

From a central computing perspective, many things have to be considered regarding the administrative and support requirements of a new system. Will we need additional staff in place to support a new environment? What are the integration requirements for 3 rd party applications? What will it take to integrate all the third-party applications with respect to financial and staff? In Appendix 1, you will see a list of functional requirements that have been aggregated from need assessments developed with input from students, faculty and UNIT’s Instructional Technologies group. What we have attempted to do is to get feedback from all stakeholders and determine which tools are the most used, the most effective and which tools must to be part of our next LMS. We consider gaps that exist and how we can address those with 3rd party applications.

Other drivers being considered: what is the health of the vendor in terms of financial stability-is the vendor or community here to stay or are they subject to the negative effects of the current economic climate? Is the platform scalable --- not only capable of meeting the educational needs of our classrooms but the broader needs of the University as a whole --- we have many departments across campus that make use of the LMS system to provide training for their staff, certificate programs to external users and as an environment where online collaboration can occur within departments and/or among groups. The LMS has to handle many of the University's functional requirements, from educational to institutional use. These requirements include integration with existing systems, Banner, LDAP, MyNova, etc., and functionality to meet staff workflow and governmental regulatory standards.

2

Is the company or organization innovative – are they looking at future technology trends and actively updating their application’s tool sets? This is an important aspect as we move forward with our next learning management system. As we've seen over the last couple of years some of the tools in the WebCT environment have become outdated since the purchase of the application by Blackboard. When we first acquired the current version, the toolset was varied and adaptable to the learning environment capable of meeting the needs of a broad spectrum of faculty users. With very limited development over the last few years, the toolset has become stagnant. It is important that the next company and product that we select for our new learning management system has the vision to not only put in place the current technologies but also continue to look at the direction of technology and incorporate tools into their suite to keep it current.

The financial impact in the selection of the next learning management system is also an extremely important consideration. A separate document provided a total of cost of ownership for the three packages being considered. The cost estimates were extended over a five-year period. In reviewing the document, a local installation of the two proprietary products is less expensive than the hosted option. However, the costs to purchase and maintain hardware necessary to support a local installation, the staff resources and the overall environment as a whole in terms of heating, air-conditioning, physical space, cabling, routers, switches, etc., - all play into the final total cost of ownership. All factors have to be considered in making a truly diligent decision in terms of our next learning management system. Although the financial impact in terms of budget allocation is a considerable one, it is not the primary focus or the primary factor in determining the direction of our next product. The functional use of the product in terms of meeting faculty and student needs in the learning environment is the primary factor in the selection process.

Process and Input

In 2008 academic year, Unit staff began reviewing the applications that existed in the LMS market. At that time the top five products were: Blackboard Learn, Sakai, Moodle, Angel and Desire2Learn. This represented three proprietary and two open source products. UNIT actively reviewed these products in terms of the toolsets inclusive at the time, the cost and the ability to make a change and sustain the application in our environment.

In conjunction with one of the classes in the Business School, UNIT reviewed the five products previously mentioned in structured process to isolate and weigh all the parameters within the applications and place a value on the usage of the tools, the cost, and the ability to integrate into the environment. This process was to quantify a forward direction in the selection process. In Appendix 3, attached to this document you can review some of the tables that were included in the final report. The final report was based on student input on the usability of the products and what made sense in terms of going forward with a replacement application. At the time, It was determined that to replace WebCT with another product would not be a cost-effective initiative. There would be no cost savings or enhancements in toolsets that would add value sufficient to shift another product. A decision was made to stay with WebCT for the interim until the next generation products became available. If cost were not a factor at the time, the student preference would have been for the Angel system. Subsequently Angel was bought by Blackboard. As we went through this process with the students, we were able to glean some valuable information in terms of student usage of a learning management system. The likes and dislikes of the student groups were tabulated as part of the Project. Student input was captured as to what they would like to see in any of the next generation products; what tools were considered helpful and what tools were absolutely necessary.

In the 2009/2010 academic year, UNIT has also reviewed additional applications which have either emerged as new possibilities in the LMS marketplace or have been bought and rebranded by other companies. These additional applications included CourseCruiser, Edvance360, Instructure, Web Study and Moodle for the second time. This was done in conjunction with faculty input and with UNIT’s knowledge of the toolsets that are currently used across campus.

In the ACIT meeting of October 2009, the selection process of the next learning management system was discussed. A recommendation from the committee was to convene a pilot group of faculty who could research and have input into the selection process. One of the faculty offered to present the request for faculty participants to the Faculty Senate. Other faculty present at the ACIT meeting were asked to take the request back to the individual colleges. A core group of faculty representative of all the colleges was convened with the task of reviewing and researching the top applications in the learning management marketplace in November 2009

At the onset the faculty committee consisted of 40 faculty members representing each college and staff from the Law school, UNIT and the Falvey Library. The Faculty Committee was asked to review, research and score the top three products in a pilot environment: Blackboard Learn Suite in a hosted environment; Sakai in a hosted and locally installed environments and Desire2Learn in a hosted environment. Meetings were scheduled to address the milestones of the project. Information was disseminated that included introductions to the applications being reviewed; training session schedules, webinars for each of the applications; a cost estimates for each of the applications and score sheets were provided for each of the application. A needs assessment survey also was made available for faculty to have input regarding desired tools and functions within the LMS application. The first application that was introduced to the faculty was Sakai. Arrangements were made with a Sakai reseller, RSmart, to set up a hosted environment for faculty testers for Villanova. We also installed a Sakai instance on Villanova servers. The purpose of this installation was to ascertain whether we as a department could do the installation, the administration and the future support of

3

an open-source product on campus. The review and research of Sakai ran from December 2009 and continued through the end of the summer session.

The next application for review began in February 2010. The application was Desired2Learn which was hosted by the vendor. The Faculty Committee was given access to use sandboxes on the vendor site. Additionally one faculty opted to teach a summer course with the Desire2Learn product (survey included in Appendix 3). The last application reviewed was Blackboard Learn 9.0 suite of software. Arrangements were made with the vendor to host a Villanova installation. Again, faculty were given IDs and passwords with the ability to set up a content, investigate the toolset and were provided a score sheet to rate the application. In summary beginning the first of December through August 30th, faculty had access to research, review and score the top three applications.

Additionally relevant documents available in the public sector, as well as LMS vendor literature were made available and played a crucial role in the review process. This literature review, as well as having access to the vendor applications, allowed the University to gain knowledge of the benefits and drawbacks of each system. Public documentation in the form of surveys of technical requirements, faculty and student evaluations, as well as vendor responses to RFQs were all very informative,

As we have gone through this process, there have been additional opportunities for faculty to have input into the selection process or at a minimum to gain knowledgeable about the possibilities of the different alternatives. On April 22 during the Technology Expo, all vendors were invited to participate. The vendors had the opportunity to have a table with the ability to do a half an hour or hour presentation. Many faculty took advantage of this opportunity and spent time with the vendors discussing their products and having a demonstration of the user interface and software functionality. On May 13th during the Teaching and Learning Strategies Day hosted by VITAL and CIT, a presentation was made describing the learning management selection process and its implications for Villanova over the next year. More than 40 faculty attended the session and information was collected in regards to faculty input and concerns about the next learning management system at Villanova. On September 8th the entire day was devoted to vendor presentations by representatives from Blackboard, Desire2Learn and RSmart. All faculty were invited to the presentations via Campus Currents and multiple emails were sent to the faculty participants of pilot group. Twenty-two of faculty attended the session and were introduced to the vendor representatives and able to ask questions and have demonstrations of the individual tools of their learning management systems. Additionally, members of the Northeast eLearning Consortium were invited to provide some different perspectives.

During the second week of classes (Fall 2010 semester), a survey was generated and sent to all faculty in all the colleges requesting their input into the selection process. The survey consisted of a series of questions and ratings of the different tools in an attempt to get a broad indication of the functionality that faculty would like to see in the next platform. A representation of the feedback is included below in Figure I. This represents all the tools and features that have been indicated by the faculty as required in the next learning management system. Across-the-board all of the products we are currently reviewing have these features included. The products differentiate themselves by how well these features actually function within the toolset or if the feature is an add-on application. For example Sakai as an open-source solution is a series of modules that are integrated in one application. The modules are written by different people at different schools and thus sometimes have a different look and feel. This is particularly noticeable, for example, when an HTML editor is part of one tool and not part of another. As a functional example, in the case of a text posting for a Wiki, a faculty member or student would have to know some basic HTML in order to post formatted text.

4

Content MGT for documents & video assets

Email, Chat & IM

Security

Application Sharing

Content Delivery Online Testing

Students Tracking

Online Grade bookCourse Calendar

SIS Integration

Blogging

Threaded Discussion

Photos in Grade book

LDAP Integration

Mobile Component

RespondusePortfolios

Respondus

Plagiarism Detection

Drop & Drag Functionality

Send/Receive mail external to LMS

Grade Exchange

Respondus

Next LM System

MyNova Integration

Wikis

Respondus

Figure I – A Model for Proposed System

Appendix 4 and Appendix 5 include summaries of the score sheets and the tool survey sheets that faculty prepared as part of the review process. Appendix 4 is a tabulation of all the features and tools that faculty would like to see in the next learning management system inclusive in this table are the actual highest ratings to the lowest ratings of the designated tools. In Appendix 5 is a summary of the score sheets that faculty submitted after using the applications in a hosted environment.

Financial Costs and Implications

A separate document included in the summary is the total cost of ownership for the individual products. As can be seen the hosted solutions versus a local installation is significantly different for all products. We have attempted to include all aspects of hosting and self hosting to adequately reflect the money that would have to be spent for the next-generation product. What is not included in this analysis is the cost that would be absorbed by the University in terms of the physical requirements for server hosting on campus. A study was done last year with Facility Management and UnIT to ascertain what improvements would be needed to maintain the data center as it was in the Mendel facility. The cost for the physical equipment and renovations to the building to include improvements in environmentals such as heat and air conditioning were substantial enough to consider an option for off site hosting for Villanova’s major computing systems. A decision was made to move the Data Center in Spring of this year (2010) to an off-site location. To determine a complete and total cost of ownership right for a local installation of the future LMS product a cost factor of the off-site hosted environment for the LMS would need to be included. The addition of these costs added to the items already included in the TCO makes the opportunity for a vendor hosted solution appear to be cost-effective and fiscally responsible.

Conclusion

The process used to select the next learning management system at Villanova has been laborious for both faculty and staff but necessary in order to make a good decision for not only the University as a whole but for the individual faculty and students with varied teaching and learning needs. It is significant to note that for Villanova, neither cost nor the core feature sets stood out as the biggest differentiators

5

between the competing LMS products, but the extra features that added value for small groups with specific needs, e.g., wikis, mobile computing, selective release capability, etc., seemed to drive the decision. There are two areas that potentially could provide cost savings to the University in the future. First – ePortfolios are becoming more widely used on campus as individual departments and programs are realizing their value. Blackboard is the only product with a built-in portfolio that would meet the current needs across campus. We are currently using a 3rd party application (Taskstream) – our portfolio initiatives over time could potentially be transitioned to the Blackboard platform thus saving the annual licensing cost on the Taskstream application. The second area where cost savings is a potential would be the consolidation of licenses. Currently the Law School carries a separate Blackboard license. Future upgrades to the next generation product (Learn 9.2) would allow for one license for the campus with separate administrative control for each entity.

During the process there was discussion about the possibility of Villanova moving towards an open-source solution for the next LMS product. After review by the faculty, it was determined that the current release of Sakai does not provide the broad spectrum of features required by the faculty as a whole. As indicated by the Gartner targeted research, Appendix 6, open source solutions for higher education are expected to have mainstream adoption within the next two years. The next release from the Sakai community is expected in early 2012. This release appears to meet many of the functional requirements cited by the faculty. This could potentially be a longer range strategy to be considered by the University.

In the final discussion, the main factor considered by faculty was the added functionality to the environment - new innovations without losing the features of the current LMS. Support for mobile devices, more flexibility with testing, easier to use interface, etc., were some of the discussion points that lead to the final recommendation. While all the evaluated products did retain most of the core functionality required, the Blackboard Learn release is the only application that provided enhancements and also supported the integration of the many third party products that are currently accessed through WebCT. Thus the Blackboard Learn system, with its familiar look and feel and significant improvements, is the recommendation from the faculty as the outcome of this research process.

6

Appendix 1: Functional Requirements

7

Course Content Ability to post various types of content

Reporting/Access to updated roster

Require the ability to access detailed downloadable reports of student progress, grades, attendance, and enrollment based on individual or group criteria

Security System must be secure with SSl encryption and password protection for copyright compliance

Integrated Gradebook Instructors have the ability to record and release grades and provide feedback privately to students.

Assignment Submission An integrated assignment submission area gives instructors more control over when and how assignments and content is released.

Discussion Board, Wikis, Blogs Require intuitive interface with multiple forum capability; posts must be gradable items with html editor

Easily Edit Course Elements Require the ability to make edits to course structure without disrupting student progress.

Integration support System must be integrated w/LDAP/MyNova & Banner as well as 3rd party apps currently in use

Automated Testing and Scoring The testing tool must be able to create assessments that use the following types of questions: Multiple Choice, Calculated Answer, and Short Answer/Essay. Questions can be imported from existing test banks or can be both built with the tool. An equation editor must be available. The testing tool can support timed test submission and completion. The automated scoring can score multiple choice and short answers type questions with optional immediate feedback.

Student Tracking Software must track both student-centered information (first access date, most recent access date to all parts of course, e.g., readings and contributions, etc) and content-centered information (number of accesses to each page of content, average time spent on each page of content, etc).

Ease of Use The interface must intuitive; tools are easy to find and use.

CalendarAbility to provide a calendar to list course events, requirements and assignments

Notifications Notification or Announcement capability

Academic Integrity

Must have a core feature to detect plagiarized submissions or capable of integration with 3rd party app; must also integrate with Lockdown browser or other feature to deter cheating

Synchronous collaboration toolsAbility to collaborate with students through chat , whiteboards or 3rd party software based integrated conferencing tool

Assigned Relative Weights to criteria

Organized Criteria by topic/function;

Created parent and sub criteria

Segmented criteria into Technical and Business Hierarchies

Aggregated Similar Criteria, resulting in 75 Criteria

Began with 105 Criteria

Appendix 3: Business Decision Criteria LMS Alternative EvaluationBDM Consulting Group

The BDM Consulting Group an organization comprised of students from the Business Decision Making (DIT) class of Spring 2008 at the Villanova Business School evaluated potential alternatives to the WebCT Learning Management System. To evaluate alternatives, Analytical Hierarchy Process (AHP) and Decision Lens software was used.

After performing the evaluation of potential LMS alternatives it was the opinion of the BDM Consulting Group that Villanova University should continue to license the WebCT (Blackboard) product for the duration of their contract. This recommendation stems from the results calculated using the Decision Lens Suite and based on the technical, functional and cost comparisons provided by UNIT

A matrix of 105 criteria was considered during the evaluation. These criteria formed the decision making model that lead to a LMS recommendation.

A complete copy of the report prepared by BMD Consulting ( Sean Coady, Marie Lokey, Duane Lacsamana, Matthew Didden, Scott Kosman, Seth Strohl, Brian Cotroneo) is available.

8

Appendix 4: Survey on importance of Tools in next LMS

ToolAverage

Score ToolAverage

Score Tool

AverageScore

Course content posting 4.66

Synchronous collaboration 3.11 Announcements 3.5

Discussion board 2.85Assignments/

Drop box 3.72 Calendar 3.08

Blogs 2.18 Quizzing 3.66 ePortfolio 2.2

Wikis 2.2 Grade book 3.66Plagiarismdetection 3.46

Whiteboard 2.71 Rubrics 2.71

9

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5Course content posting4.66

Discussion board2.85

Blogs2.18

Wikis2.2

Whiteboard2.71

Synchronous collaboration3.11

Assignments/Drop box3.72

Quizzing3.66 Gradebook

3.66

Rubrics2.71

Announcements3.5

Calendar3.08

ePortfolio2.2

Plagerism detection3.46

Average score out of 5; 114 respondents

Appendix 5: Average Scores of Products by Pilot Group

ToolSakai

Avg score out of 5Blackboard

Avg score out of 5Desire2learn

Avg score out of 5

Discussions 3.2 4.75 4

Assignments/Drop box 3.6 4.5 3.85

Group tools for presentations/share files

3 4.5 3.75

Chat (in course) 3.5 4.75 3.87

Streaming media (w/out HTML) 2.5 5 4

Whiteboard 3.1 4.3 3.5

Assign students to groups (instructor)

4 4.75 4.4

Self tests 3.6 4.75 4.6

Chat (system wide) 2.3 3.6 2.3

Assign students to groups (random) 3.4 4.65 4.13

Course Builder (guide for instructors)

3.1 4.15 4.65

Templates for standard pages 3.2 3.9 4.15

Quiz tool 3.85 4.6 4.27

Selective release 3.5 4.5 4.2

Context-sensitive Help 3 4.8 3.8

Student tracking 4.2 4.8 4.5

Automatically graded assessment questions

3.9 4.9 4.6

Intuitive interface 3.1 4.5 4.25

Contemporary look and feel 3.2 4.7 4.1

N= 14 8 10

Total Average 3.33 4.55 4.05

10

Appendix 6: Gartner Targeted Research Published in July 2010

11