a study of the effectiveness of the cengage mindtap application

This study was conducted by MarketingWorks and SEG Measurement, independent educational research firms. This research was supported by a grant from Cengage Learning.

January 2013

A Study of the Effectiveness of the Cengage MindTap Application

Improving Student Knowledge and Skills With Digital Learning Tools:

ABSTRACT

The purpose of this study was to explore the impact of MindTap’s use on student learning and to gather student and teacher perceptions for further refinement of the product. This study investigated the following question: Do students in classes that use MindTap show gains in course knowledge and skills? The study also explored the results by gender and ethnicity. The study used a mixed-method, qualitative and quantitative research design to evaluate this question. First, using a pre-post, treatment-group-only design, this study evaluated student growth in knowledge and skills. Second, a qualitative study of instructor and student perceptions of MindTap effectiveness using surveys and interviews was conducted. The findings indicate that students in classes using MindTap substantially increased their course knowledge and skills. An effect size of .84 was found for students in Nutrition classes, and an effect size of .77 was found for students in

Macroeconomic classes.

1



Table of Contents

Executive Summary ............................................................................................................................................. 2

Overview ........................................................................................................................................................... 2

Findings ............................................................................................................................................................. 2

Conclusion ........................................................................................................................................................ 4

Chapter 1: Introduction ...................................................................................................................................... 5

Student Sample .............................................................................................................................................. 10

Chapter2: Quantitative Study ........................................................................................................................... 13

Study Design .................................................................................................................................................. 13

Description of the Pretest and Posttest ..................................................................................................... 13

Description of the Treatment ..................................................................................................................... 14

Findings ........................................................................................................................................................... 15

Descriptive Statistics ..................................................................................................................................... 15

Evaluating Growth in Language Skills ....................................................................................................... 15

Chapter 3 Qualitative Study ............................................................................................................................ 16

Feedback from Instructors .......................................................................................................................... 16

Feedback from Students .............................................................................................................................. 23

Chapter 4: Conclusion ....................................................................................................................................... 30

2



Executive Summary

Overview

In 2012, Cengage Learning introduced a new online course management and instructional support

tool, MindTap. In the fall of 2012, MarketingWorks and SEG Measurement, two independent

research organizations, conducted a semester-long study of students and instructors in classes using

MindTap as a preliminary evaluation of the effectiveness of the product.

Instructors and students in classes using MindTap completed an introductory survey, a bi-weekly

survey, and a concluding survey to ascertain their perceptions of the efficacy of MindTap, how it

was used, areas of strength and weakness, and recommendations for improvement. A treatment-

group-only design was used to gather quantitative evidence of effectiveness. Students in classes

using MindTap took a pretest of course content knowledge and skills at the beginning of the

semester and a posttest at the end of the semester. The amount of growth between pre- and

posttest was analyzed statistically as an indicator of the effectiveness of MindTap.

Findings

Students in classes using MindTap showed substantial growth from pretest to posttest in both

Macroeconomics and Nutrition (see Figure 1). Students in Nutrition classes using MindTap showed

significant gains in course content knowledge and skills (t=10.21;df=90;p<.01; Effect size=.84).

Students in Macroeconomics classes using MindTap showed significant gains in course content

knowledge and skills (t=5.89;df=58;p<.01; Effect size=.77).

3



At the conclusion of the semester, the average rating of instructors’ overall experience with

MindTap was 2.9 out of 4, or a grade of B. In addition, “adoption potential,” meaning the

willingness to recommend MindTap to other colleagues, was rated 3.1 out of 4, with 77% saying

there was an “excellent” or “above average” possibility that they would recommend MindTap to

their colleagues. All but one of the instructors said that the program contributed to the

improvement of student learning, and 85% said that students were either very or somewhat engaged

with their coursework, largely as a result of their use of MindTap.

4



Students rated their overall experience with MindTap as 3.3 out of 4, a solid B+ rating. In addition,

most found the coursework to be very engaging, saying that they were more engaged in the course

because of their use of MindTap. Finally, about two-thirds of students said that they thought they

had learned more than they would have otherwise because of their use of MindTap. On average,

students rated their agreement with the statement “I would recommend the use of MindTap for this

course to my friends” as 4.6 out of 5.

Conclusion

The results of both the qualitative and quantitative study suggest that MindTap is an effective tool

for improving student learning. Both students and instructors reported positive perceptions of the

MindTap product, and students in both the Nutrition and Macroeconomics classes using MindTap

substantially increased their course content knowledge and skills over the course of a semester.

Cengage plans to confirm these preliminary positive results in larger studies to be conducted during

the 2013-2014 school year.

5



Chapter 1: Introduction

In 2012, Cengage Learning introduced a new digital course management and instructional support

tool, MindTap, an online, personalized learning experience built on Cengage Learning’s content.

MindTap combines student learning tools - readings, multimedia, activities, and assessments - into a

Learning Path that guides students through their course. Instructors can personalize the experience

by customizing Cengage Learning content and learning tools, including the ability to add their own

content via apps that integrate into the MindTap framework with Learning Management Systems.

To gather preliminary evidence of the effectiveness of the MindTap program, MarketingWorks and

SEG Measurement, two independent research organizations, conducted a semester-long study in the

fall of 2012 of students and instructors in classes using MindTap. The study was designed to collect

information for use in refining the MindTap application and to conduct a preliminary evaluation of

the effectiveness of MindTap. The primary research question for this evaluation was: Do students in

classrooms using MindTap show significant gains in course content knowledge and skills?

All instructors and a sample of students in classes using MindTap completed an introductory survey,

a bi-weekly survey, and a concluding survey to ascertain their perceptions of the efficacy of

MindTap, how it was used, areas of strength and weakness, and recommendations for improvement.

Both instructors and students were interviewed mid-way through the semester to further determine

their perceptions of the MindTap product.

A pre-post, treatment-group-only design was used to gather quantitative evidence of effectiveness

from a subset of the classes. Students in these classes using MindTap were administered a pretest of

course content knowledge and skills at the beginning of the semester and a posttest of knowledge

and skills at the end of the semester. The amount of growth between pre- and posttest was analyzed

as a preliminary indicator of MindTap’s effectiveness.

6



Instructor Sample

A total of 13 instructors teaching a variety of courses at 13 different institutions of higher education

provided feedback about their experiences using the MindTap product throughout the fall, 2012

semester. All instructors used one of the following eight Cengage Learning courses:

A People & A Nation, Brief Ninth Edition: Beth Norton, et. al. (2)

Business: Pride/Hughes/Kappor, 10th Edition (1)

Business Communication: Process and Product, M. E. Guffey (1)

Management: Richard L. Daft (1)

Medical Terminology For Health Professions: Ann Ehrlich/Carol L. Schroeder, 6th Edition (2)

Nutrition: Concepts and Controversies: Frances Sizer (2)

Principles of Economics: N. Gregory Mankiw, 6th Edition (2)

Principles of Macroeconomics: N. Gregory Mankiw, 6th Edition (2)

Instructors and students in three course areas originally participated in the study of the efficacy of

MindTap: Macroeconomics, Nutrition, and Medical Terminology. Since fewer than 10 Medical

Terminology students participated in the full study, they were eliminated from this portion of the

study but remained in the qualitative feedback portion.

To summarize, all 13 institutions in the research were non-profit schools, with fourth being two-year

schools and 8 being 4-year. Over half of the schools were in urban environments, with the rest

divided between suburban and rural locations. A list of participating institutions appears in the

Appendix.

Instructors held a wide range of titles, from adjunct instructor to department chair. A majority of

instructors were full-time; were 41 or older; were white; held a masters as their highest degree; and

had four or more years of experience teaching overall and teaching this specific course. Just over

half were female. All said they were very or somewhat experienced with the use of technology.

All but three instructors were using MindTap for the first time when the research was conducted,

with the majority using the associated textbook for three years or less. All but one instructor

7



selected it on their own. They were using MindTap on a desktop (6), laptop (3), or with Blackboard

(3). The majority had used similar products in the past, including three who had used Aplia, another

Cengage Learning product. Video activities and assessments were the digital tools most widely used

with MindTap

Demographic breakdown of the instructors and institutions is as follows:

Number of instructors

Type of Institution

2-year non-profit 4

2-year for-profit 0

4-year non-profit 8

4-year for-profit 0

Instructor Title

Adjunct Instructor/Professor 3

Instructor 2

Assistant Professor 2

Associate Professor 3

Professor 1

Other 1 Dept Chair, 1 Sr Academic Staff

Full-time/Part-time Status

Full-time 10

Part-time 3

Highest Degree Attained

Bachelors 0

Masters 8

Other professional degree beyond BA 1

Ed.D/PhD 3

Other 1 All but dissertation

Teaching Experience

First year 2

1-3 years 2

4-10 years 4

More than 10 years 5

8



Years Teaching This Course

First year 2

1-3 years 2

4-10 years 7

More than 10 years 2

Years Using Textbook

First semester 6

1-3 years 3

4-6 years 3

More than 6 years 1

Time Using MindTap

First semester 10

1 semester 0

2 semesters 3

Use of Similar Products in Past

Yes Cengage Emerge

SAM 2010

My Lab/Mastering

Aplia 3

Connect (McGraw Hill)

MyHistoryLab (Pearson)

Cengage Brain

No 5

Choice of MindTap

Instructor chose it 12

Department required it 1

Location of School

Urban 7

Suburban 3

Rural 3

Digital Tools Used

Lecture capture system 1

Student response devices 1

Social networking 2

Google apps for education 0

9



Video activities or assessment 6

Web collaborative tools 0

Other Blackboard (2)

Tweet

Blog

Moodle (2)

Podcasts/movies

ANGEL

None 1

Experience with Technology

Very experienced 8

Somewhat experienced 5

Not very experienced 0

Not at all experienced 0

Gender

Male 6

Female 7

Age

21-30 1

31-40 3

41-50 2

More than 50 7

Ethnicity

White/Caucasian 12

Black 0

Hispanic 1

Asian 0

Platform Using MindTap

Blackboard 3

Desktop 6

Laptop 3

Smartphone 0

Other 1

10



Student Sample

Approximately 260 students participated in the study of the effectiveness of the Cengage Learning’s

MindTap. Table 1 shows the number of students in each gender and ethnic category. This sample

provides a 99% level of confidence that the margin of error is within 93% of the results that would

have been obtained had all US college students been studied. For MacroEconomics about three-

fifths of the participants were male and two-fifths were female. Nearly three-fifths of the

Macroeconomics students were Caucasian, about one-fifth were classified as Asian, and one-tenth

were Hispanic; the remaining Macroeconomics students were classified as other, with no students

reporting they were African American.

For Nutrition, half the students were male and half were female. One-third of the students

participating in Nutrition classes were Hispanic, one-quarter were Caucasian, just under one-fifth

were Asian or classified as other, and the remaining 6% were African American.

Table 1

Profile of Student Participants

In some instances, teachers did not provide complete background information for a student or a

student did not take one of the tests included in the analyses. Where data was missing, the student’s

results were eliminated from those analyses.

Variable

Macroeconomics Number of Students

Nutrition Number of Students

Gender

Male 49 (59%) 86 (50%)

Female 34 (41%) 86 (50%)

TOTAL 83 172

Ethnicity

Caucasian 47 (57%) 44 (26%)

African American 0 (0%) 11 (6%)

Hispanic 10 (12%) 57 (33%)

Asian/Pacific Islander 17 (20%) 27 (16%)

Other 5 (6%) 26 (15%)

Not Reported 4 (5%) 7 (4%)

TOTAL 83 172

11



A total of 26 of the 260 students were selected to provide feedback during the course of the study,

demographic information was collected direct from the students. This sample provides a 99% level

of confidence that the margin of error is within 90% of the results that would have been obtained

had all 260 students been sampled. In general, these students were under 30 (almost half are 21 or

less), spoke English as their native language, were enrolled full-time at their institutions, obtained

high school GPA’s that were B or better, expected to complete a Bachelor’s, or Master’s degree, and

were somewhat experienced with technology. Most had prior experience with digital tools in their

classes. Few were using any digital tools for studying, however, with social networking being the

standout as a choice of 23% of the students.

Demographic breakdown of the 23 students is as follows:

Age

0-20 46%

21-30 39

31-40 11

41-50 4

51 or above 0

English as Native Language

Yes 89%

No. of Courses Taken in the

Semester

1-3 11%

4-6 79

7 or more 11

GPA in High School

A or A+ 14%

A- 29

B or B+ 50

B- 4

C or C+ 4

Less than C- 0

Disabilities/Medical Conditions

12



Learning disability (e.g., dyslexia) 4%

Psychological disorder 4

Highest Degree Intended to Earn

Associates 7%

Bachelor’s 32

Master’s 43

Professional (e.g., MD, PhD, JD) 18

Technology Experience

Very experienced 32%

Somewhat experienced 68

Not very experienced 0

Not at all experienced

Prior Digital Experience in Classes

Prior to MindTap

Yes 71%

No 21

Not sure 7

Technology Tools Used This Year

in School for Studying/Learning

Required Optional

E-portfolio 3% 1%

Student response system (clickers) 9 4

Lecture capture system 1 4

Social networking 14 23

Video activities or assessment 16 17

Web collaboration tools 8 10

Social document sharing 19 16

Group messaging for study groups 16 16

Note taking or Web clipping service 5 9

Other 9 1

13



Chapter2: Quantitative Study

Study Design

The growth in students’ course content knowledge and skills was evaluated by using a treatment

only, pre-post test design. Students in classes using MindTap were given a pretest at the beginning

of the semester and a posttest at the end of the semester. The difference in the pretest and posttest

results was analyzed statistically as an indicator of the effectiveness of MindTap instruction.

Description of the Pretest and Posttest

Student learning was operationalized as the gains in course content knowledge and skills between

the pre- and posttest. Student knowledge and skills in the study was measured using a 30-item

pretest administered at the beginning of the semester in August or September and a 30-item posttest

administered at the end of the semester in December. The pretest and the posttest contained 15

questions in common and 15 unique questions.

An item analysis revealed that all items were technically sound. No items had p values below .30 or

above .95, and no items had point biserial correlations (discrimination coefficients) below .20. This

shows that the questions were neither too hard nor too easy and effectively differentiated between

those students who had the knowledge and skills and those who did not.

To help ensure the validity of the measures used, both the Macroeconomics and Nutrition tests were

designed to measure the content taught in the typical Macroeconomics and Nutrition courses

reflected in the textbooks used in the study. For each content area, the major content and skill

objectives were identified to correspond to the chapter headings in the book. Each content/skill

objective was “assigned” 1-3 test items based on the amount and importance of the content and

skills associated with the objective. The items were written to generate content parallel pre- and

posttests for the study. The reliability of the Macroeconomics pre- and posttest was .82(KR 20).

The reliability of the Nutrition pre- and posttest was .84 (KR-20). Reliability of .80 or above is

generally considered acceptable.

14



Description of the Treatment

The Treatment in this study was classroom use of MindTap. Teachers reported on average using

MindTap 4.17 hours per week. Students received between10 and16 weeks of instruction.

Data Collection

Instructors participating in the study were provided with the pretests, answer sheets, and

administration manuals in August 2012. The instructors then administered the pretest according to

the administration instructions provided. At the end of the semester, in December 2012, the

instructors administered the posttest.

The completed test booklets and answer sheets were returned to SEG Measurement for processing.

The answer sheets were analyzed and entered into a database. All data were reviewed and checked

for accuracy before scoring and analysis. The pre and posttest student records were merged to

create a single record for each student containing both pre and posttest data. The pretest and

posttest results were compared as a basis for evaluating growth.

Qualitative and quantitative feedback about MindTap was obtained from all instructors through

PubCentral’s CAP tool on a biweekly basis, starting with feedback for the two weeks prior to

September 30, 2012 and ending with feedback for the two weeks prior to December 14, 2012. In

addition, a concluding survey was administered at the end of the semester, typically during the week

of December 17. And finally, a telephone interview was conducted about mid-way through the

semester with all instructors.

In addition, 26 students from the 13 classes provided feedback about their experiences with

MindTap throughout the research study. Like instructors, qualitative and quantitative feedback

about MindTap was obtained from all students through PubCentral’s CAP tool on a biweekly basis.

In addition, a concluding survey was administered at the end of the semester, and a telephone

interview was conducted about mid-way through the semester.

15



Findings

Descriptive Statistics

The first step in analyzing the results of the study was to examine the average scores (mean)

achieved by students on the pre and posttests as well as the variation (standard deviation) in those

scores. We calculated the mean and standard deviation for both the pre and posttests (see table 2).

Table 2

Descriptive Statistics

MindTap Pre and Posttest Data

Test/Language Skill Area N Pretest

Mean

Pretest SD Pretest N Posttest

Mean

Posttest

SD

Posttest

Macroeconomics 59 14.25 3.16 59 17.31 3.96

Nutrition 91 14.47 4.07 91 18.30 4.54

Evaluating Growth in Language Skills

Student course content knowledge and skills at the beginning of the semester were compared to

their knowledge and skills at the end of the semester using t tests. The t test is a commonly

accepted statistic for comparing growth over time and evaluating whether or not those differences

are due to chance or reflect actual growth. The magnitude of the difference is expressed as the

effect size, a commonly accepted measure of the degree of difference. (Effect size was calculated as

the difference between the pre and posttest score means, divided by the overall standard deviation.)

Only students for whom matched pretest and posttest results were available were included in the

analysis. Students who left the class during this period or who joined the class during this period

were not included in the growth comparisons.

Statistically significant growth was found within classes using MindTap for both Nutrition and

Macroeconomics. Students in classes that used MindTap showed substantial growth from pretest to

posttest in both course areas. Students in Nutrition classes using MindTap showed significant gains

16



in course content knowledge and skills (t=10.21;df=90;p<.01; Effect size=.84). Students in

Macroeconomics classes using MindTap showed significant gains in course content knowledge and

skills (t=5.89;df=58;p<.01; Effect size=.77). This means that the students using MindTap showed

gains in course knowledge and skills greater than what would be expected by chance.

Chapter 3 Qualitative Study

Feedback from Instructors

Qualitative and quantitative feedback about MindTap was obtained from all instructors through

PubCentral’s CAP tool on a biweekly basis, starting with feedback for the two weeks prior to

September 30, 2012 and ending with feedback for the two weeks prior to December 14, 2012. In

addition, a concluding survey was administered at the end of the semester, typically during the week

of December 17. And finally, a telephone interview was conducted about mid-way through the

semester with all instructors.

At the conclusion of the semester, the average rating of instructors’ overall experience with

MindTap was 2.9 out of 4, representing a grade of B. Typical comments from participants included

the following:

The information included in MindTap is excellent. Great work!

For my three overall objectives - - cost to students, better engagement, and ease of adapting to our requirements

- - MindTap met and exceeded expectations.

All but one student was able to adapt to ebooks without going back to the textbook. It seems the technology

will be accepted by the students more as time goes by.

A key strength of MindTap is that the variety of exercises appeals to students with different learning styles.

The assignments are well rounded in that they measure comprehension of basic facts as well as concepts, and

the primary source assignments help students develop critical thinking skills.

17



MindTap allowed me to start students very late in the semester, completing the course with only a one-week

extension. Without MindTap, I would not have been able to assist them.

Negative comments typically came from a minority of instructors and usually related to the way

particular elements of the program functioned. For example:

MindTap was very difficult to get to work right in the first half of the semester, although this did get better as

time went on, perhaps as students became more familiar with the program’s quirks and how to work around

them.

There were some aspects of my ability to manage the course that were frustrating. For example, when only

one student takes a quiz, I couldn’t make further changes to the ground rules (e.g., how many attempts a

student could make).

The substance of the activities and quizzes fell below my expectations. For some activities, there’s not much

substance. And the quizzes included strangely worded questions. There’s a lot of potential, but significant

obstacles that need to be addressed in terms of technical issues as well as intellectual content.

“Adoption potential,” (meaning the willingness to recommend MindTap to other colleagues) was

rated 3.1 out of 4, with 10 out of 13 instructors (77%) saying there was an “excellent” or “above

average” possibility that they would recommend MindTap to their colleagues. Strengths of the

system included:

Ability to turn it over to students and get them to self-engage and go beyond listening to a lecture. Its basic

organization and ease of adaptability to specific instructors’ needs make it superior.

Having all of MindTap instead of the textbook and MindTap helps students to make better use of the

program.

18



Progress tracking and ability to add material as supplements that are gradable, use of video uploading

Applications that are embedded in the text

Help given students who need motivation in preparing for tests and reinforcing the lessons they should be

learning from the lectures and readings

The “live” links that allow students to move vertically as well as horizontally through the assigned chapters,

the layout of the chapter listing learning outcomes and objectives, and the links in each chapter allowing

students to navigate to a specific section of a chapter easily

The functionality and tools in the ebook

Suggestions for improving the product included the following:

More diversity in videos (to include people of color) and/or sharing database links for web sites with videos

that can be loaded into MindTap.1

More functionality in the progress tracking and kaltura/Google apps2

More professor control with regard to quizzes (e.g., allowing students to see answers once the quiz is

completed, changing questions, changing the number of chances students have to complete the quizzes, and

integrating with Blackboard and MoodleRooms)3

Accessibility for hearing and visually impaired students

1 Since the study was completed, the ability to add web links has been added to MindTap.

2 Similarly, MindTap 26 made significant improvements in the progress app.

33 Integration with Blackboard Learn 9.1 and above was added after the study and incudes grade book exchange.

19



More flexibility with assignments, more customization, tools to better manage individual students, and

improvements in the progress app

Flexibility in how students can view the ebook pages (a significant minority didn’t like having to use the etext

in a vertical position on the “blackboard” and wanted it traditionally laid out.)4

On average, instructors felt that MindTap met the needs of students with a wide range of capabilities

(average rating 2.9 out of 4). In addition, most instructors felt that MindTap made teaching easier

and better (average rating of 2.5 out of 4), although three complained that it made more work for

them because students were asking a lot of questions about the program.

All but two of the 13 instructors (85%) said that students were either very engaged or somewhat

engaged with their coursework, and all but three (77%) said that students’ engagement with the

coursework was impacted by their use of MindTap.

Some relevant comments follow:

There was more in-class participation, better attendance, and more discussion.

The tool pushes them to stay with reading, and this helped with out-of-class structure and personal discipline.

Students are engaged with the product so they can learn the information. I explained to them that it’s like

learning an international language.

I’d say about two-thirds of the students were very engaged.

The students who were most engaged in the class also thought MindTap had promising capabilities, although

I agree that it’s not altogether necessary for an introductory course in psychology.

4 The MindTap 2.6 release will allow students with iPads to change the orientation of the text in the MindTap Reader.

20



Students are accustomed to going through many classes without having to open their textbooks. With

MindTap, that’s impossible as it requires students to actually read the book closely. It requires more work

from them. From an instructor’s point of view, this is excellent. However, when students complained about

MindTap I perceived that this is what they were really chafing against. The problem wasn’t with

MindTap; they were irritated that they had to actually read and think about the book.

There are some students who loved the course, others who I have not seen since the first day!

All but one of the instructors (92%) said that MindTap contributed to the improvement of student

learning outcomes. (31% said it greatly contributed, 54% said it contributed somewhat, and 8% said

it contributed slightly). Specific comments include the following:

Great ebook with pre-lecture and post-lecture parts, meaning that students have a notion of the materials

before the lecture without reading the whole chapter. That’s a major difference compared to other ebooks.

Students that used the ebook loved the interface.

I think it kept students focused on the material throughout the semester rather than simply before a test.

The great thing about MindTap is that it accommodates a great variety of learning styles so the students can

find a combination of resources that best help them learn.

As with any assignments, students varied in their dedication to attend to the work. Students who seriously

approached the assignments had higher scores on the unit tests and had a stronger commitment of chapter

content.

In spite of time management and other challenges students faced in these business writing classes, anecdotal

evidence suggests that students’ business writing seems better than it has in past semesters.

21



In addition, all but two instructors (85%) said that MindTap helped students to get through the

course and focus on areas where they needed the most help - - either extremely well (38%) or

somewhat well (46%). Specific comments include the following:

The structure of MindTap is such that it keeps students on task. I like the learning path because it focuses

students’ attention on what we are covering in class that week.

I think MindTap helps students understand the material to the degree that it gets them to think about the

material from more than one perspective..

I believe the quizzes helped students a great deal. The “check your understanding” activities were very

helpful.

The learning outcomes and outline of each chapter before the post-lecture quiz and reading are great ways to

get students focused on what’s important.

Average rating of satisfaction with the instruction they received for setup of their MindTap course

was 2.8 out of 4. Some instructors felt the resources and training sessions were “excellent” while

others felt the setup was “extremely complicated” and even “stressful.” One mentioned not

receiving a welcome package with instructions as they had expected from prior experience, and a

couple of instructors mentioned that they learned the program “on the fly” and discovered features

that had not been explained on the WebEx training they received.

To evaluate their own experience with MindTap, instructors were asked to rate their agreement with

the following statements on a 5-point scale, with 1 meaning complete disagreement and 5 meaning

complete agreement. Average ratings for the 13 instructors were as follows:

MindTap allowed me to engage

more with my students.

3.5

MindTap helped me track my 3.2

22



students’ progress more easily.

MindTap helped to improve my

teaching of the course.

3.6

MindTap’s content worked well

with my textbook in this course.

3.9

The primary issue raised by several was about the difficulty of tracking students’ progress. They

complained that no average is calculated for each student to give them an idea of their progress and

that instructors can view only percentages and not numbers that would allow grade calculations.

One instructor commented that the Aplia grade book was much more functional than the one in

MindTap and did a far superior job of tracking the progress of students.

Content assets named as most and least value to the instructors were as follows:

Most Value Least Value

Assessments 1

Game play 1

Notes 1

Read aloud 1

Flash cards 2

Videos 2 2 - -lack of diversity

Pre and post-tests 1

Prelecture part of ebook 1

Glossary 1

Ebook functions 2

Quizzes 3 2

Kaltura 1

Audio chapter overview 1

Links to primary sources 1

Zoomable maps 1

Learning outcomes and objectives 1

23



ConnectYard 3

Homework post-lecture 1

Activities that merely test reading

comprehension

1

More advanced bells and whistles

(e.g., ability to use outside sources)

1

Progress app’s lack of functionality 1

Dictionary and glossary 1

Average ratings on a 4-point scale (where A=4 and F=0) of specific MindTap features from all of

the bi-weekly surveys completed throughout the semester were as follows:

Overall structure 3.3 out of 4

Overall experience 3.0

Cengage support 3.7

Readings 3.6

Quizzes and homework 3.1

Feedback from Students

Qualitative and quantitative feedback about MindTap was also obtained from a sample of students

through PubCentral’s CAP tool on a biweekly basis, starting with feedback for the two weeks prior

to September 30, 2012 and ending with feedback for the two weeks prior to December 14, 2012. In

addition, a concluding survey was administered at the end of the semester, and a telephone interview

was conducted about mid-way through the semester.

Twenty-six students completed all of the biweekly surveys, the mid-semester interview, and the

concluding survey. As a group, they used MindTap about once a week (38%), several times a week

(50%), or almost every day (8%); and they typically spent 2-3 hours a week on homework and

studying outside of class (65%).

24



Half of the students reported that they experienced problems using MindTap, although many noted

the problems receded as the semester progressed. The following issues were noted:

Freezing pages when using the read-aloud feature or taking quizzes

Java script not refreshing when doing assignments

Slow loading, freezing, or not loading at all without exiting out of the program completely

Responses to quizzes not always recorded

Getting results from quizzes

Speed of the system

Insufficient help in the tutorial about how to access the apps

Inability to use the backspace button without the page reloading

Opening assignments

Class book access that was disabled

On average, students rated their overall experience using MindTap as 3.3 out of 4 (a B+ average

rating). Comments from students include the following:

I thoroughly liked the program. The information is relevant and easily accessible.

A great alternative to the book. In fact, I only used the book twice during the semester and that was due to

not having access to MindTap on my iPad.

I love this program. The information is useful, relevant to what we are learning, and it provided a good

overview.

Much better than the other ones out there.

I enjoyed using MindTap. It was extremely useful for my course. It was very helpful as a study tool because

it kept my work very organized and there are a lot of useful tools to use.

MindTap was always awesome. I was always able to use it effectively and I’m overall very pleased with my

experience. I hope other classes I take in the future will utilize MindTap.

25



MindTap was extremely useful for me. Working with a computer program is much more attractive for people

in our generation than to do practice problems from a textbook. Also, it is a way for me as a visual and

kinesthetic learner to interact with the material. See the material more than once, through lectures, text

readings, and then MindTap helped me learn better - - repetition of the concepts. I only rated it a B because

of the technical problems and because the pre-lecture quizzes don’t allow you to see which questions you got

wrong.

I feel that MindTap really improved my learning of the materials with strengths being the text, media

resources, galleries, and quizzes. All of these were very useful to the learning process. Weaknesses of the

system include: ineffective tutorial, not being able to utilize the text and galleries in full screen(some of the

images appeared very blurry and were hard to read the small print), and at times the system ran very slow but

got better after mentioning it in the reviews. All in all, I would say that the positives outweigh the negatives.

Technically, MindTap functions well once learned.

My over all experience was a good experience. It is an extremely easy to use program that I would

recommend to anyone wanting to take an online history course.

A lot better than I expected!

By and large, students found the coursework to be either very engaging (58%) or somewhat

engaging (35%), with 54% saying that they were more engaged in the course because of their use of

MindTap. As a sign of their engagement, 19% said they discussed topics, ideas, or concepts with

their instructors or other faculty members outside of class frequently (almost every week), and an

additional 31% said they did so sometimes (several times during the course).

Comments from students about their engagement and learning included the following:

I realized, after I had stopped using MindTap, how convenient it was to be able to work on an assignment

and check the answers immediately to see how well I was understanding the material.

26



My only suggestion to be more engaged would be to include more quizzes that can be accessed (not just for

grades or request of the instructor), and to place any apps that would benefit writing papers on the tools bar.

One suggestion would be a thesaurus next to the dictionary to make it conveniently located while I am

reviewing learning concepts and writing papers at the same time.

The media made me want to learn more about what I was studying.

I was very engaged in my coursework because MindTap was a simple, helpful website that provided me with

my homework and quizzes.

I was very engaged because 1) thanks to MindTap, I was engaged in the material in more than one way --

rather than only using the book or the notes to study the material, I was engaged with the material online,

and 2) thanks to MindTap, I spent more time with the coursework overall, resulting in my being more

engaged.

My interest in the course was definitely increased as MindTap helped me understand what was going on.

Even when I fell behind on the readings and lectures, I knew I could catch up through MindTap and use it

as a checkpoint for whether I was grasping important concepts.

I think this resource makes learning much more engaging than a typical book. The tutorials, flashcards and

videos were excellent.

Interaction with other students and the instructor was a must and was enhanced through MindTap.

I was much more engaged than I'd otherwise be because homework and quizzes that I take in lecture are not

as helpful as the ones on the website. I can refer to the textbook at any time and write notes.

I was more interested and engaged in the material thanks to MindTap because MindTap was not only a

comprehensive resource, it was

27



an enjoyable one. I looked forward to using it and that made me more interested and engaged.

MindTap kept be interested in the course!

In addition, almost two-thirds of the students (65%) said that they thought they had learned more

than they would have otherwise because of their use of MindTap. Comments included:

MindTap greatly improved what I learned over the duration of this course and I do feel that I learned more

than I otherwise would have. The apps, media gallery, and text make it easy to learn and provide an

alternative to just reading a paper text. This increases the interest in the topics.

Because I spent more time with the material, in a number of different ways, I learned more thanks to

MindTap.

Content assets names as most and least value to students were the following, with Quizzes being

cited most frequently as of most value and Connect Yard as of least value:

Most Value Least Value

Read-aloud text 2 2

Flashcards 7 1

Highlighter 3 2

Note cards 4 1

Search tool 4

Post-lecture readings 1

Quizzes 11 1

Homework assignments 4

Dictionary/Glossary 4 3

Explanations of correct 1

28



answers

Tutorials 1

Videos 1 1

Diet Analysis Plus 1

Online textbook 1

Tests 2

Media 1

Small book in the side tab 1

Enlarged print 1 1

Chapter reviews 2

Connect Yard 4

No printable features 1

Objectives 1

Google Docs 2

Readings 1

To evaluate their own experience with MindTap, students were asked to rate their agreement with

the following on a 5-point scale, with 1 meaning complete disagreement and 5 meaning complete

agreement. Average ratings for the 26 students responding were as follows:

Average Ratings

I would recommend the uses of MindTap for this course to my

friends.

4.6

MindTap helped me better understand the expectations of my

instructor.

4.1

MindTap helped me complete assignments on time. 4.0

MindTap helped me better prepare for tests. 4.3

MindTap was valuable in helping me learn new concepts. 4.5

Using MindTap allowed me to better track my progress in this 3.8

29



course.

Using MindTap allowed me to engage more with my instructor. 3.5

Using MindTap allowed me to engage more with my fellow

students.

3.3

I used the textbook less because of MindTap. 3.6

The MindTap site was easy to navigate. 4.4

The content of MindTap was relevant to me. 4.7

The content of MindTap was relevant to my course. 4.8

Ratings overall were high and in all cases above the median point of 3.0 on the 5-point scale. The

highest ratings were given to MindTap’s relevancy to the students’ course (4.8) relevancy to them

personally (4.7), its value in helping students learn new concepts (4.5), its ease of navigation (4.4),

and its help in preparing students for tests (4.3). Most importantly in terms of overall acceptance, all

but two agreed completely (17 students) or somewhat (7 students) that they would recommend the

use of MindTap for this course to their friends.

There was less agreement about MindTap’s ability to engage students with their fellow students (3.3)

or with their instructors (3.5).

In addition to technical issues as detailed above, other specific recommendations to improve

MindTap include the following:

Allow students to see which questions they answered incorrectly and to view the answers to

the practice tests and quizzes

Add a thesaurus next to the dictionary to make it more conveniently located

Include a spell checker along with the dictionary if a student doesn’t know how to spell a

word being looked up

Make MindTap available in a textbook format

Add games or more interactive media to make it more engaging

30



Allow instructors to customize the quizzes to make them more aligned to the portions of the

work actually covered in class

Add mobile apps

Allow images to be zoomed in

Make the illustrations and charts more visible

Chapter 4: Conclusion

SEG Measurement and MarketingWorks conducted research during the Fall 2012 semester to

determine instructor and student perceptions of Cengage’s MindTap digital instructional tool and to

examine its effectiveness. A qualitative study collected survey and interview data from instructors

and students using MindTap to determine their perceptions of the product. A quantitative study of

MindTap users evaluated the extent to which students in Nutrition and Macroeconomics classes

increased their content knowledge and skills. Growth in knowledge and skills was evaluated using a

pre- post/treatment-group-only design.

The results of the quantitative study found that students in classes that used MindTap showed

substantial growth in knowledge and skills. Nutrition students grew over four-fifths of a standard

deviation (Effect size=.84) and Macroeconomics students grew over three-quarters of a standard

deviation (Effect size=.77). These gains are considered to be quite large by researchers and are

meaningfully greater than are typically seen across a college semester, suggesting that MindTap may

be a useful tool for improving student learning.

The results of the qualitative study were generally positive as well, with over three-fourths of

instructors and 92% of students surveyed indicating a strong or good possibility of recommending

MindTap to their colleagues and friends.

The quantitative study results are preliminary. The positive findings of this study need to be

confirmed with larger samples of students and with more robust research employing a treatment-

31



control group design. Cengage Learning plans to complete such studies during the 2013-2014

school year to confirm the positive results found in this study.

a study of the effectiveness of the cengage mindtap application

Documents