an analysis of student performance in online vs. face-to...

Perspectives on Economic Education Research, 2018, 11(1) 1-18

Journal homepage: cobhomepages.cob.isu.edu/peer/

An Analysis of Student Performance in Online vs. Face-to-face Principles of Economics Controlling for Selection Bias

Kathleen Aranoa1, Samuel Schreyerb, Dosse Toulaboec aSchool of Business, Indiana University Southeast, United States, [email protected] bDepartment of Economics, Finance, and Accounting, Fort Hays State University, United States cDepartment of Economics, Finance, and Accounting, Fort Hays State University, United States

Abstract

With the proliferation of online courses, including Principles of Economics, researchers have examined the success of students in these courses in comparison to traditional face-to-face (F2F) delivery. The literature reveals mixed outcomes, often a result of study design and methodology. This study examines students in online and F2F Principles of Micro and Macro in a more contained setting, controlling for self-selection bias, instructor and course characteristics, and student characteristics including study habit and attitudes. We find evidence of students performing better in F2F vs. online in Micro, but no significant difference in Macro, after controlling for self-selection. Indeed, there was no significant self-selection bias in Micro but there was for Macro. To allow us to investigate potential differences between the two delivery platforms for below-average and above-average student performance, we also employ quantile regression. The quantile regression results suggest that top-performing students benefit the most from F2F instruction. Key Words: online vs. face-to-face, selection bias, student performance JEL Codes: A22

1 Corresponding author.

cobhomepages.cob.isu.edu/peer/

mailto:[email protected]

Arano, Schreyer, Toulaboe/Perspectives on Economic Education Research, 2018, 11(1) 1-18

2

1. Introduction

Online education is widespread and rapidly growing. The information revolution and advancements in instructional technology have led to a proliferation of courses or entire academic programs being offered online. The Online Learning Consortium’s 2015 Online Report Card – Tracking Online Education in the United States reports a year-to-year 3.9% increase in the number of distance education students using available 2014 data from the U.S. Department of Education’s Integrated Education Data System (IPEDS). More than 5.8 million students (28.4% of all enrolled students) are taking at least one distance education course and more than 2.8 million students are taking their courses exclusively online. The largest proportion of distance education students are from public institutions—72.7% of all undergraduate and 38.7% of all graduate-level distance students. Indeed, these figures seem to indicate that the relevant question today is not whether the use of online education is at an inflection point, but instead how the merits of online education compare with those of face-to-face (F2F) classroom teaching.

Distance learning presents some distinguishing and appealing features that make it a learning method of choice for some students. From the student perspective, online courses offer greater flexibility and convenience. It is true that, and as Lundberg et al. (2008) put it, whatever the reason, if online education attracts students who otherwise would not have taken a specific course or attended higher education, this is of importance from a policy perspective as it has a positive effect on the accumulation of human capital. However, it seems appropriate to also say that online learning is not for everyone. For instance, students who are less motivated may have particular trouble learning in an online environment. According to Navarro (2000), a significant percentage of instructors in his survey believed that older students often seem more motivated and self-directed, and perform better in online courses. Using an endogenous switching model, Coates et al. (2004) found that students who select into the online classes perform better than they would in a F2F class.

These empirical results from the literature—which suggest that there is either no significant difference between the two modes of delivery or that the online delivery has a statistically positive impact on student academic performance relative to F2F delivery—have been criticized by other research on the ground that the data, underlining assumptions, and the methodology adopted are flawed. The purpose of this study is to further refine the methodology, by not only accounting for the possibility of self-selection bias and a controlled environment, but to also investigate the possibility that taking a course online vs F2F may affect a high-performing student differently than a low-performing student. The next section reviews the debate surrounding distance vs. F2F education. Sections three and four discuss the data and the empirical methodologies used in this study. Section five reports the results of the analysis and is followed by a brief conclusion.

2. The debate: Literature review

The literature provides evidence that, in general, online students are students with greater outside responsibilities (work, childcare, etc.), students that feel more at ease with the computer, live farther from campus, and students that are older and nontraditional (Dutton et al., 2002; Bengiamin et al., 1998; Wallace and Mutooni, 1997). Research indicates that the impact


3

and benefits of online education derives from many factors, including student ability, student motivation and self-direction, course level, course types, gender, and age (Ray and Grimes, 1992, Barlett and Feiner, 1992; Borg and Stranahan, 2002; Brown and Liedholm, 2002). It is important that we differentiate between the flexibility and other positive features that online education affords students and the impact that online education has on learning. As Oblinger and Hawkins (2006) correctly put it, to get an answer as to whether technology makes a difference, we need to ask: "Difference in what?" Can online education offer courses that are at least as effective (in terms of student academic performance) as the traditional courses, thus preparing students for futures careers?

Proponents of distance learning claim that, if properly designed, the learning outcomes compare favorably with those in conventional classroom settings, confirming the view that online education is a highly effective way to learn (Boulet and Boudreault, 1998; Dutton et al., 2001; Liu et al., 1998; Navarro and Shoemaker, 2000; and Smeaton and Keogh, 1998). According to Navarro (2000), the cyber environment can create and nurture more student-to-student and faculty-to-student interactions through different means (e.g., chat rooms, teleconferencing, e-mail, and bulletin board discussions) that may give students a greater chance for interaction compared to a large lecture hall generally dominated by a handful of vocal students. Using experiments, Agarwal and Day (1998) and Primont and Summary (1999) find that traditional teaching methods enhanced with internet technologies have a positive impact on student retention and the learning of economic concepts.

Neuhauser (2002) analyzed student performance of 62 management students and finds no significant difference in the mean value of test scores for online vs. F2F students. In the same vein, Conrad (1997) and Talley (2000) showed that although internet use increases enjoyment, there are no statistically significant gains observed in student performance. These studies are in line with the seemingly popular book “No Significance Phenomenon” by Russell (1999). Russell’s review of over 350 studies on course delivery led him to conclude that “No matter how it is produced, how it is delivered, whether or not it is interactive, low-tech, or high-tech, students learn equally well with each technology and learn as well as their on-campus, F2F counterparts”.

Course types and subject matters are factors that may influence the success of students in an online course. We suspect online students to perform as well as F2F students in courses or subject matters that involve knowledge of basic concepts and definitional items as compared to more sophisticated courses or more complex materials. Brown and Liedholm (2002) shows that no significant difference exists between online and campus student performance for definitional and recognition exam questions. They find, however, that the difference between the two forms of delivery becomes significant for questions requiring more difficult subject matters, questions requiring the deepest understanding, and exam questions that tapped the students' ability to apply basic concepts in more sophisticated ways.

Critics of Brown and Liedholm (2002) posit that although their model controls for factors such as students’ gender, pre-knowledge in mathematics, and their high-school grades, it failed to control for other factors such as students’ outside responsibilities (work, childcare, etc.), motivation, self-direction, age, and self-selection (Lundberg et al., 2008). Dennis Coates et al. (2004) addresses some of these problems by controlling for students’ outside responsibilities, age, and by using an endogenous switching regression model to correct for self-selection. They claim that failure to account for self-selection of students into online or F2F courses biases


4

toward zero the differential in performance scores between online and F2F students. Stephenson et al. (2005) shows that, based on test performance, distance and traditional classroom instructions are not perfect substitutes for each other and that all but the very gifted students in distance courses do not learn as much as similar students in traditional classrooms. After controlling for other factors, their switching regression results show that the amount of learning in the online environment is substantially less than that of the traditional format. Crouse (2002) also shows that traditional students generally obtain higher grades on tests, and have a higher opinion of course instruction than distance students, and that distance education is not an equivalent educational alternative to traditional classroom instruction. According to Harrington (1999), traditional-learning students outperform online students and only online students with high GPA earn outcome similar to traditional students. Comparing two sections of beginning algebra (one taught online and one onsite), Weems (2002) found that although there was not a significant difference between exam averages for the two platforms, there was a significant decrease in performance by the online students across the exams.

For economics courses in particular, studies have pointed to the issue of self-selection bias (since students are not randomly assigned to either F2F class or to online) and the importance of holding other factors constant, i.e., a controlled environment (Howsen and Lile, 2008). However, even with attempts to account for these issues, results with respect to student performance are still mixed. For principles of macroeconomics class, Howsen and Lile (2008) found that online delivery results in lower average test scores by about a letter grade, ceteris paribus. On the other hand, Bosworth and Bowles (2009) presented results that online students generally perform better in an economics class that combines an introduction to economics principles and a survey of macroeconomics. Given the mixed results of the success of students between the two delivery modalities, Switzer and Rebeck (2015) investigated the determinants of success between the two and find that in online courses, differences in study habits may play a significant factor in determining student success. In particular, they find that students who score high on concentration and study attitudes are at an advantage in online courses, after controlling for student, instructor and course characteristics.

The empirical models we estimate account for self-selection and a controlled environment including study habits and attitudes of students. In addition, we examine how the type of delivery, either online or F2F, impact students across different points of the performance distribution by using quantile regression. This methodology sheds insight on an important question that has largely gone unaddressed in the literature: is the differential effect from an online vs F2F course the same for low-performing students as it is for high-performing students? Previous studies have focused attention on the conditional mean estimates, yet there is no a priori reason to believe these estimated relationships hold true for students not represented by the conditional mean. Indeed, Asarta and Schmidt (2017) argue that comparing the class-wide performance means between traditional (i.e., F2F) and blended (i.e., flipped and flexible instructional modes) may not give an accurate indication of differences (or lack thereof). The results of their study revealed no significant difference between the two delivery modes when the analysis was restricted to class-wide means. However, by considering differences in prior academic performance means based on low, middle, and top grade point averages, a more precise set of findings was uncovered. Performance was significantly better in the traditional


5

version of the course at low grade averages, while the reverse was true at high grade point averages, and no significant difference was detected at the middle zone of grade point averages.

3. Data

We designed the study to maximize the ability to disentangle the impact of the course delivery platform from other potential confounding factors. The data were gathered over two semesters (full academic year) from Principles of Microeconomics and Principles of Macroeconomics classes at a four year public university in the Midwest. Both online and F2F classes were taught by the same instructors, utilized the same textbooks and online homework management system (MyEconLab; MEL), covered the same chapters, and used the same course grade weights for exams and homework. The online sections were provided lecture videos from a textbook publisher which were carefully selected and mapped by the instructors to correspond to the textbook chapters. This serves as the online counterpart to the in-class lectures. Online students were also given the opportunity to email or call instructors for questions, the counterpart for office hours to the F2F classes. The set of questions for the exams (all multiple choice) were not identical between the online and F2F classes, as this was not possible given the asynchronous set up of the online sections. However, there was a common exam pool for both platforms carefully selected by the instructors to match specific learning objectives per chapter. Thus, in terms of level of difficulty and specific concepts covered on the tests and desired learning outcomes, the exams between the two platforms are comparable. The online sections had deadlines to take exams which were close to the exam dates for the F2F sections but could technically take the exams any time before the deadlines.2 Both platforms utilized Blackboard to post all available course materials. In addition to data related to student performance, we also administered a survey to gather more information related to demographic and other student academic performance-related variables. The objective of this study is to test whether there is a difference in academic performance by modality within each course, rather than to compare academic performance between courses. Nonetheless, examining results from two similar but still distinct courses may provide some useful insights. A descriptive summary of variables along with definitions utilized in the empirical models is presented in Table 1. We use the average of the three exams (typically each exam covers 3-4 chapters) to measure academic performance.3

2 Very few online students take exams earlier than the deadlines. 3 We used average test scores to homogenize the sample further given that the exams taken between the two modalities are not identical. The models were also estimated using individual exam scores and results were comparable.


6

Table 1: Descriptive Statistics by Delivery Platform (mean values)

Variable Definition Principles of Micro Principles of Macro

Face to Face Online Face to Face Online

EXAM_A Average of 3 exam scores 0.68 0.61 0.66 0.63 MAJOR =1 if Business major; 0=otherwise 0.50 0.68 0.55 0.65 ACT Composite ACT score 22.87 23.95 23.41 23.45 AGE Age 20.50 28.90 20.88 28.62

CR_HRS Credit hours taking in current semester

14.17 11.34 13.64 10.56

FR =1 if freshman; =0 otherwise 0.22 0.07 0.09 0.12 SO =1 if sophomore; =0 otherwise 0.31 0.22 0.43 0.19 JR =1 if junior; =0 otherwise 0.29 0.44 0.27 0.36 MS =1 if single; =0 otherwise 0.98 0.37 0.96 0.47 RES =1 if in-state resident; =0 otherwise 0.77 0.63 0.80 0.68 CHLD =1 if have children; =0 otherwise 0.03 0.51 0.04 0.40

EMP_HRS Average number of work hrs. per week

12.31 34.98 14.26 35.98

ECFI_F =1 if the class is the student’s first economics course; =0 otherwise

0.59 0.53 0.47 0.51

GPA_C Current cumulative GPA (4-point scale)

3.13 2.99 3.12 2.97

MATH =1 if taken College Algebra; =0 otherwise

0.74 0.76 0.80 0.77

STUDY_TST Average hrs. spent studying for a test 4.15 9.16 6.11 8.20 HW_MD Median score for all homework 0.77 0.77 0.79 0.81

MEL_D1

=1 if agreed/strongly agreed to statement “MEL homework helped me understand the course better”; =0 otherwise

0.63 0.54 0.81 0.53

MEL_D2 =1 if agreed/strongly agreed to statement “MEL homework helped with tests”; =0 otherwise

0.59 0.47 0.66 0.38

READ_TXT Average hrs. per week reading text 2.18 5.03 2.57 4.59

GRADE_EX =1 if expected grade is A or B; =0 otherwise

0.76 0.82 0.77 0.81

ST_D1 =1 if agreed/strongly agreed to statement “Class lectures help me understand better”; =0 otherwise

0.67 0.51 0.74 0.47

ST_D2 =1 if agreed/strongly agreed to statement “I learn better on my own”; =0 otherwise

0.32 0.36 0.32 0.51

ST_D3 =1 if agreed/strongly agreed to statement “Online courses are more convenient”; =0 otherwise

0.63 0.94 0.66 0.97

ONLINE =1 if taken an online class before; =0 otherwise

0.49 0.85 0.55 0.91

FATHER_E =1 if father has a college degree or higher; =0 otherwise

0.38 0.28 0.43 0.30

MOTHER_E =1 if mother has a college degree or higher; =0 otherwise

0.40 0.32 0.50 0.38


7

There were a total of 333 students in the Micro classes, 265 F2F and 68 online, and a total of 268 students in the Macro classes, 191 F2F and 77 online. The distribution of exam scores is slightly skewed to the left. The mean exam average for online sections is lower by 7% for Micro and by 3% for Macro relative to their on-campus counterparts. F2F students are about 20 years of age (about 9 years younger than online students) and tended to be freshmen and sophomores while online students tended to be juniors. The reported ACT scores are almost identical for both cohorts although the cumulative GPA is slightly higher for F2F students at 3.11. A majority of online students had jobs (90%) and worked an average of 35 hours per week compared to 62% of F2F students with jobs who worked an average of about 13 hours per week. On the other hand, F2F students tended to be enrolled in three additional credit hours than online students. In terms of course-related activities, online students spent about twice as many hours reading the text per week and about 4 additional hours studying for the exams compared to F2F students. Thirty-two percent of F2F students for both Micro and Macro, agree to the statement “I learn better on my own” while 51% and 36% of online students agree to the same statement for Macro and Micro, respectively. About 82% of online students expected a course grade of B or higher compared to 77% for F2F students.

4. Empirical models 4.1. Heckman selection model The first set of models estimated is the two-step model using the Heckman procedure where we estimate: (1) a probit model for choice of the delivery platform (online vs. F2F), the selection equation; and then (2) a model for class performance (EXAM_A), the main equation. The dependent variable in the selection equation (TYPE=1 if online, considered the “treatment”; =0 if F2F) is an explanatory variable in the main equation which also includes the correction for selection bias, λ (the inverse Mills ratio). Since students could potentially choose to enroll either online or F2F, estimating the student performance by OLS without the correction factor would lead to biased and inconsistent estimates and tend to overstate the effect of the online class, i.e., the “treatment effect” (Green, 2000). Our two-step model is estimated for both the Micro and Macro class following the specification:

𝑇𝑌𝑃𝐸 = 𝛽0 + 𝛽1𝑀𝑆 + 𝛽2𝐴𝐺𝐸 + 𝛽3𝐶𝐻𝐿𝐷 + 𝛽4𝑅𝐸𝑆 + 𝛽5𝐸𝑀𝑃𝐻𝑅𝑆 + 𝛽6𝑆𝑇𝐷1 + 𝛽7𝑆𝑇𝐷2

+ 𝛽8𝑆𝑇𝐷3 + 𝛽9𝑂𝑁𝐿𝐼𝑁𝐸 + 𝜀

𝐸𝑋𝐴𝑀𝐴 = 𝛽0 + 𝛽1𝑇𝑌𝑃𝐸 + 𝛽2𝑀𝐴𝐽𝑂𝑅 + 𝛽3𝐹𝑅 + 𝛽4𝑆𝑂 + 𝛽5𝐽𝑅 + 𝛽6𝐴𝐶𝑇 + 𝛽6𝐶𝑅𝐻𝑅𝑆

+ 𝛽7𝐺𝑃𝐴𝐶 + 𝛽8𝑀𝐴𝑇𝐻 + 𝛽9𝐺𝑅𝐴𝐷𝐸𝐸𝑋 + 𝛽10𝑆𝑇𝑈𝐷𝑌𝑇𝑆𝑇 + 𝛽11𝑅𝐸𝐴𝐷𝑇𝑋𝑇

+ 𝛽12𝐻𝑊𝑀𝐷 + 𝛽13𝑀𝐸𝐿𝐷1 + 𝛽14𝑀𝐸𝐿𝐷2 + 𝛽15𝐹𝐴𝑇𝐻𝐸𝑅𝐸 + 𝛽16𝑀𝑂𝑇𝐻𝐸𝑅𝐸

+ 𝛽17𝐸𝐶𝐹𝐼𝐹 + 𝛽18𝜆 + 𝜀

(1)

(2)


8

The variables AGE, CHLD and EMP_HRS are included in the selection equation to capture potential preference for online courses of older employed students due to a higher opportunity cost of going to school. On the other hand, older students may prefer F2F classes due to apprehension about returning to school (Howsen and Lile, 2008). The set of dummy variables ST_D1 to ST_D3 capture overall attitude of students to perceived benefits of online education as discussed in the literature in section 2. In general, we expect that students who agree to the statement “I learn better on my own” (ST_D2) and “online courses are more convenient” (ST_D3) and who disagree to the statement “class lectures help me understand better” (reference category for ST_D1) are more likely to choose the online platform. Students who have taken online courses previously (ONLINE) are more likely to choose online if they have had a positive experience (Russel, 1999). The main equation is shown in (2) and the primary variable of interest is TYPE. If there is a difference in student performance between online and F2F, the parameter estimate for TYPE will be statistically significant. It remains an empirical question whether this is positive or negative. ACT scores are used to capture innate ability while GPA_C can capture work effort and overall student success in school: both variables are hypothesized to have a positive effect on EXAM_A. The value of performing better in school is likely higher for students with parents who are college educated (FATHER_E and MOTHER_E). FR, SO, and JR capture student classification and are hypothesized to have negative coefficients (the reference group is seniors) if we assume older students have more college experience and thus perform better. Business majors (MAJOR) are hypothesized to have more desire to perform better as principles of Micro and Macro are required courses and are part of the business core courses. Students who have taken a prior economics course (ECFI_F) and college algebra (MATH) are postulated to perform better. Higher scores from the homework (HW_MD) should likewise lead to better performance along with average hours spent studying for tests (STUDY_TST) and reading the text (READ_TXT). The two-step specification is superior to OLS estimation of performance if the selection bias corrector (λ) is significant in equation (2).

4.2. Quantile regression model

We next employ quantile regression (QR) analysis which, to our knowledge, has not been employed in previous studies in this particular line of literature. Perhaps the main advantage of QR is that it provides insight how covariates are related to the regressand along the entire conditional distribution, rather than limiting attention to the conditional mean. In the context of this paper, QR allows us the ability to consider how online vs F2F classes impact low-performing students and high-performing students rather than simply considering the impact on the average-performing student. Additional benefits of the QR methodology include its robustness to outliers. Extreme observations can lead to a loss in efficiency and biased estimates in a least squares regression since the corresponding residuals will be large, and squaring these residuals gives these extreme observations greater weight. A well-known estimation technique robust to extreme observations is Least Absolute Deviations (LAD) regression which minimizes the absolute deviations from the predicted dependent variable. LAD, however, is a special case of QR. For brevity we refer the reader to Koenker and Hallock (2001) and Koenker (2005) for a detailed description of QR and its many applications.


9

For our QR model, we examine student performance by estimating equation (2) at the 20th, 40th, 60th, and 80th quantiles. The estimates at the 20th quantile correspond to relatively low-performing students, whereas estimates for the 80th quantile correspond to relatively high-performing students. The estimated coefficients are expected to have the same signs as described in section 4.1. Of particular interest is if, and how, the estimated coefficient for TYPE changes across the quantiles as this indicates the effect of online vs F2F varies by student performance. Although Asarta and Schmidt (2017) differs from this study in methodology and scope, Asarta and Schmidt did find that students with lower grade point averages performed better in traditional courses, whereas students with higher GPAs performed better in blended courses. Given this, the results in this paper are loosely consistent with Asarta and Schmidt (2017) if the estimated coefficient on TYPE is negative at low quantiles and positive at high quantiles. 5. Results Table 2 provides estimation results for the two-step models. The table presents the results for Micro and Macro together for brevity, but caution is warranted in directly comparing results between these two courses. Both are introductory economics courses but they are two distinct courses. The second column reports the marginal effects of the probit selection equation, while estimates for the performance equation are in column 3. Our previous hypotheses regarding the selection equation are mostly verified. Older students are significantly more likely to choose online delivery, and each additional hour of employment per week increases the likelihood of choosing online by 0.5% and 0.7%, for Micro and Macro, respectively. On average, single students in Micro and Macro are 15% and 10% less likely to choose online platform relative to married students. These results tend to support the hypothesis of higher opportunity cost of attending traditional school for older, employed and married students. The results likewise confirm the perceived benefits of online delivery. Students in Micro and Macro who agree “class lectures help me understand better” are 7% and nearly 9% less likely to choose an online course, respectively. Students in Micro and Macro who agree “online courses are more convenient” are 6.5% and 20% more likely to choose an online course, respectively.

In the main equation, ACT is a significant and positive predictor of student performance for both courses. Students who do better with homework also perform better in exams. Students who expected a grade of A or B in the class at the beginning of the semester also scored higher on the tests, perhaps indicating students’ accurate gauge of their abilities and thus a realistic expectation. In contrast to our expectations, students who have previously taken College Algebra (MATH) actually perform worse for both courses. Introductory level economics uses elementary math skills (e.g., slope of a line and area or a triangle) which are studied in College Algebra. However, the MATH variable in our model is a simple dummy variable of whether or not a student has taken College Algebra, and it may not capture the actual math skills acquired by a student. An alternative measure—such as the grade in College Algebra or whether or not the student passed this course on first take—may better capture mastery of elementary math skills that are utilized in an introductory level economics course.


10

Table 2: Two-Step Model Parameter Estimates (standard errors in parentheses)

Variable Selection Equation

(TYPE) Variable

Main Equation (EXAM_A)

MICRO MACRO MICRO MACRO

MS -0.147*

(0.087) -0.104 (0.094)

TYPE -0.135***

(0.030) -0.020 (0.037)

AGE 0.015***

(0.005)

0.019*** (0.006)

MAJOR -0.003 (0.012)

-0.021 (0.021)

CHLD 0.118 (0.090)

0.075 (0.086)

FR -0.054** (0.023)

-0.016 (0.045)

RES -0.156*** (0.051)

-0.143** (0.058)

SO -0.013 (0.022)

0.019 (0.030)

EMP_HRS 0.005***

(0.0009) 0.007*** (0.001)

JR -0.158 (0.021)

0.018 (0.027)

ST_D1 -0.071* (0.038)

-0.088* (0.050)

ACT 0.011*** (0.002)

0.015***

(0.003)

ST_D2 -0.047 (0.032)

0.067 (0.048)

CR_HRS -0.005* (0.003)

-0.005 (0.004)

ST_D3 0.065*

(0.037) 0.198*** (0.066)

GPA_C 0.085*** (0.013)

0.029**

(0.022)

ONLINE 0.073* (0.038)

0.106* (0.058)

MATH -0.03* (0.016)

-0.10*** (0.037)

GRADE_EX 0.044*** (0.016)

0.061** (0.026)

STUDY_TST 0.001 (0.001)

0.003 (0.003)

READ_TXT -0.002 (0.003)

-0.001 (0.005)

HW_MD 0.389*** (0.056)

0.322*** (0.065)

MEL_D1 0.030*

(0.016) -0.057* (0.031)

MEL_D2 -0.006 (0.015)

0.085*** (0.024)

FATHER_E 0.037**

(0.015) 0.003 (0.021)

MOTHER_E -0.043*** (0.013)

0.027 (0.022)

ECFI_F -0.032** (0.013)

0.012 (0.019)

λ (lambda) 0.030 (0.0183)

-0.050*

(0.037)

*Significant at the 10% level; **Significant at the 5% level; ***Significant at the 1% level


11

The results for the main variable of interest, TYPE, are noteworthy. For Micro, the coefficient is negative and relatively large. Ceteris paribus, online students perform on average 13.5% lower than F2F students on exams, which is more than a letter grade lower. The magnitude of the effect is comparable to results from Stephenson, et al. (2005) and Howsen and Lile (2008). On the other hand, we find no significant difference between online and F2F students’ average exam performance for Macro students. In addition, there is no significant evidence of self-selection bias in Micro (i.e., λ is not significant) while there is for Macro. It is important to note that although students had the option to choose between online vs. F2F, the university in consideration has a nontrivial proportion of its students who reside in locations such that it is impossible for them to be an on-campus student; on the other hand, a proportion of students do reside in close proximity to the university and are able to choose to enroll in online or F2F classes. Thus, to some extent, the choice of delivery platform may not be completely self-selected, and the results for Micro may be a reflection of this. Additionally, the university does not have an explicit requirement that Micro be taken before Macro (or that the two courses be taken simultaneously), but students are advised to take Micro first and strongly discouraged to take the courses simultaneously. Most students default to taking Micro first because of its lower course number (201 vs. 202). Given this de facto sequencing, it is possible that students “learn” while taking Micro and are thus more mindful and selective of the course delivery choice when they take Macro.4

The models were also estimated using OLS without accounting for the Heckman selectivity correction and the results are available in Appendix 1. The parameter estimates for TYPE are comparable and of the same signs, albeit now marginally significant for Macro (in the Heckman selection equation, TYPE was not significant for Macro). The remaining parameter estimates are likewise comparable. These results warrant the use of the Heckman procedure. Since there is evidence of marginal significance of self-selection for Macro, correcting for it indicates that the perceived difference in student performance between online vs F2F disappears. Even without significant self-selection for Micro, there is value added to using the Heckman procedure as it allows for a more precise and explicit investigation of the decision to choose a course delivery method.

We next estimate our QR model, partly as a robustness check and partly because this methodology can shed insight into how online vs. F2F instruction (TYPE) affects students performing above and below the conditional mean. We estimate equation (2) separately for Micro and Macro courses using the 20th, 40th, 60th, and 80th quantiles. The results for the QR estimates are presented in Table 3.

4 Previous studies on the effect of Micro/Macro sequencing on student learning have not reached a consensus. For instance, Fizel and Johnson (1986) find that student learning is greatest when Micro is taken before Macro. Lopus and Maxwell (1995) obtain results which lead them to conclude Macro should be taught before Micro. Terry and Galchus (2003) find evidence that student learning is greatest when Micro and Macro are taken concurrently.


12

Table 3: Quantile Regression (standard errors in parentheses)

Variable MICRO

(EXAM_A)

MACRO

(EXAM_A)

q20 q40 q60 q80 q20 q40 q60 q80

TYPE -0.0727**

(0.0323)

-0.0827***

(0.0295)

-0.0813***

(0.0299)

-0.105***

(0.0306)

-0.0219

(0.0413)

-0.0459

(0.0492)

-0.0727

(0.0477)

-0.102*

(0.0537)

MAJOR -0.0086

(0.0169)

-0.00386

(0.0138)

-0.0104

(0.0162)

0.0137

(0.0177)

-0.0119

(0.0349)

-0.0273

(0.0380)

-0.0226

(0.0382)

0.0113

(0.0337)

FR -0.0360

(0.0410)

-0.0575

(0.0374)

-0.0654*

(0.0342)

-0.0397

(0.0367)

-0.0573

(0.0908)

0.0262

(0.0889)

-0.0854

(0.0928)

-0.101

(0.0869)

SO -0.0298

(0.0426)

-0.0347

(0.0383)

-0.0235

(0.0410)

-0.0155

(0.0410)

0.0150

(0.0407)

0.0468

(0.0487)

0.0351

(0.0502)

0.004

(0.056)

JR -0.0133

(0.0388)

-0.0261

(0.0368)

-0.0292

(0.0326)

-0.0334

(0.0370)

0.0141

(0.0403)

0.0516

(0.0508)

0.0141

(0.0507)

0.0071

(0.0480)

ACT 0.0062**

(0.0031)

0.0074**

(0.0032)

0.0042

(0.0038)

0.0079**

(0.0033)

0.0128**

(0.0053)

0.0118*

(0.0061)

0.0137**

(0.0061)

0.023***

(0.0061)

CR_HRS 0.0002

(0.0044)

-0.0037

(0.0029)

-0.0039

(0.0037)

-0.0022

(0.0042)

0.0035

(0.0064)

-0.0026

(0.0061)

-0.0092

(0.0058)

-0.0095

(0.0064)

GPA_C 0.106***

(0.0239)

0.0981***

(0.0211)

0.0968***

(0.0212)

0.0754***

(0.0238)

0.0235

(0.0513)

0.0869*

(0.0448)

0.0192

(0.0437)

0.015

(0.0373)

MATH -0.0143

(0.0236)

-0.0287

(0.0246)

-0.0196

(0.0271)

-0.0102

(0.0272)

-0.0233

(0.0642)

-0.0514

(0.0813)

-0.101

(0.086)

-0.176**

(0.0814)

GRADE_EX 0.0802***

(0.0280)

0.0451**

(0.0222)

0.0410

(0.0266)

0.0253

(0.0326)

0.0276

(0.0457)

0.0591

(0.0427)

0.0824**

(0.0400)

0.0284

(0.0415)

STUDY_TST 0.0007

(0.0033)

0.0006

(0.0027)

-0.0012

(0.0025)

0.0005

(0.0028)

0.0033

(0.0042)

0.0009

(0.0049)

0.0014

(0.0042)

0.0031

(0.0042)

READ_TXT -0.0071

(0.0047)

-0.0045

(0.0048)

-0.0052

(0.0058)

0.0004

(0.0084)

-0.0009

(0.0084)

-0.0015

(0.0092)

-0.0043

(0.0096)

0.0012

(0.0115)

HW_MD 0.426***

(0.0957)

0.509***

(0.0857)

0.510***

(0.109)

0.393***

(0.138)

0.719***

(0.189)

0.469***

(0.164)

0.497***

(0.156)

0.306*

(0.176)

MEL_D1 0.0583**

(0.0265)

0.0482*

(0.0272)

0.0306

(0.0226)

0.0073

(0.0270)

-0.0021

(0.0605)

-0.0134

(0.0602)

-0.0880

(0.0567)

-0.0897

(0.0558)

MEL_D2 0.0153

(0.0190)

-0.0030

(0.0252)

-0.0059

(0.0252)

0.0071

(0.0248)

0.0618

(0.0441)

0.0470

(0.0432)

0.0981**

(0.0447)

0.0618

(0.0378)

FATHER_E 0.0419**

(0.0180)

0.0411*

(0.0218)

0.0345

(0.0258)

0.0316

(0.0269)

-0.0291

(0.0349)

-0.0022

(0.0313)

0.0087

(0.0325)

0.012

(0.0267)

MOTHER_E -0.0502***

(0.0184)

-0.0576***

(0.0201)

-0.0376

(0.0253)

-0.0326**

(0.0260)

0.0425

(0.0355)

0.0179

(0.0376)

-0.0089

(0.0357)

-0.0133

(0.0305)

ECFI_F -0.0284

(0.0178)

-0.0285**

(0.0143)

-0.0284

(0.0175)

-0.0204

(0.0213)

0.0396

(0.0344)

0.0237

(0.0298)

0.0032

(0.0252)

0.0346

(0.0313)



13

Somewhat similar to the results from the main equation of the Two-Step Heckman, ACT impacts EXAM_A in a nearly 1:1 manner such that an additional point on the ACT exam results in an additional percent point on EXAM_A. The point estimates vary slightly across the quantiles for both courses, yet this variation is not statistically significant. Also similar to the results from the Two-Step Heckman is that GPA_C is a significant positive predictor of student performance in Micro, and the magnitude of this relationship is such that an additional point in a student’s cumulative GPA (using a 4-point scale) increases EXAM_A by about 8-11 percentage points depending on the quantile considered. Note that the point estimate from the Two-Step Heckman model was 0.085. A particularly interesting result is the general lack of significance that having had taken college algebra before Micro (MATH) has on academic performance. Moreover, MATH has a counterintuitive negative, significant, and large impact on student performance for high-performing students in Macro (i.e., the 80th quantile). The differential impact of F2F vs. online instruction on student performance is the primary focus of this paper. The estimated QR coefficients for TYPE reveal that the decision to take an online course over a F2F course imparts no statistical benefit in student performance regardless of the quantile being considered. In the case of Micro, the estimated coefficients across the quantiles are all negative and significant, indicating that taking an online course results in lower exam performance by about 7-11 percentage points.5 For Macro, the decision to take this course online or F2F is statistically insignificant for all but the 80 th quantile. This general lack of significance is similar to the result obtained from the Heckman Two Step model. However, top-performing students in Macro (i.e., at the 80th quantile) are estimated to score about 10 percentage points worse on exams if taking the course online rather than F2F. As a whole, we find no evidence that online courses improve student performance relative to F2F. Indeed, the results here provide some evidence that the decision to take an online course becomes more costly for the best-performing students. Our findings are not consistent with Asarta and Schmidt (2017), although as discussed earlier the scope and methodology is not directly comparable. 6. Conclusions

Our empirical estimates indicate no significant self-selection in Micro but a marginal self-selection bias in Macro. The more random choice of course delivery for Micro manifests itself by showing a significant difference in student performance in favor of the F2F. These results highlight the importance of accounting for self-selection. Given the way students decide on which course to take first, i.e., most likely Micro, it would seem students’ experiences from taking Micro help them to make more thoughtful choices when taking Macro, course delivery mode included. Indeed, in Micro, performance on average are lower for students for which the class is their first economics class compared to students who have taken other economics courses (either in high school or college). It should also be noted that Micro and Macro are distinctly different courses, despite their belonging to a common discipline and having a similar course design and structure, thus direct comparability of results cannot be necessarily inferred. Indeed, the results illustrate

5 An examination of how these coefficients change across the quantile gives the appearance that low-performing students gain the least by taking Micro F2F and high-performing students gain the most. However, caution is warranted as interquartile range estimate resulted in no statistical difference.


14

some disparities on the influence of the predictors in student outcomes between the two courses. The results further suggest the issue of self-selection is course-specific.

Our findings from QR are broadly similar to those obtained from the Heckman Two Step Model. Results from the QR model also reveal that the difference between online vs F2F on student performance is partly a function of the performance level of the student in question. The findings suggest that the value added from F2F instruction for a student’s comprehension of principles of economics, as measured by exam performance, primarily extends to the best performing students. Precisely why top-performing students would benefit most from F2F instruction, whereas poor-performing students receive less to no benefit from the same instruction is a provocative question and one in which we can only speculate on in this paper. There are several aspects of F2F instruction that differ from online instruction by varying degrees. F2F instruction typically allows for more spontaneous interaction during the teaching/learning process. Additionally, F2F instruction tends to require a student have strong note-taking skills and listening skills since the instruction is often in the form of a one-time college lecture (as opposed to, for example, a video that can be replayed). It is reasonable to think that better-performing students generally have higher skills in note-talking, listening, etc. Thus, the important question for researchers as well as school advisors and administrators might not be so much whether online instruction is as effective as F2F instruction, but instead whether students are matching themselves with teaching platforms best suited to their learning styles. Indeed, Switzer and Rebeck (2015) find significant variation in characteristics across students between the two modalities—students who work full time, commute from further away, who self-describe as less able to push through difficult material and more likely to procrastinate, are more likely to choose online delivery. This may indicate a potential mismatch in choice of course delivery platform (reflecting a “want”) vs the ideal given the student’s learning style (reflecting a “need”). We find similar results with respect to employed students but our results with respect to student attitude and study habits seem to indicate that our students are self-selecting correctly to some degree.


15

References Agarwal R. and A. E. Day. 1998. The Impact of the Internet on Economic Education. Journal of Economic Education 29 (2): 99-110. Asarta C.J. and J.R. Schmidt. 2017. Comparing Student Performance in Blended and Traditional Courses: Does Prior Academic Achievement Matter? Internet and Higher Education 32: 29-38. Bartlett, R. L. and S. F. Feiner. 1992. Balancing the Economics Curriculum: Content, Method, and Pedagogy’. American Economic Review 82 (2): 559-564. Bengiamin, N., A. Johnson, M. Zidon, D. Moen and D. Ludlow. 1998. The Development of an Undergraduate Distance Learning Engineering Degree for Industry – A University/Industry Collaboration. Journal of Engineering Education 87(3): 277–282.

Borg, M. O. and H. A. Stranahan. (2002). The Effect of Gender and Race on Student Performance in Principles of Economics: The Importance of Personality Type. Applied Economics 34(5): 589-598.

Bosworth, D. S. and T. J. Bowles. 2009. Online Enrollment and Student Achievement: A Treatment Effects Model. Perspectives on Economic Education Research 5(1): 15-31. Boulet, M. and S. Boudreault. 1998. Using Technology to Deliver Distance Education in Computer Science. Journal of Engineering Education 87(4): 433-436. Brown, B. W. and C. E. Liedholm. 2002. Can Web Courses Replace the Classroom in Principles of Microeconomics? The American Economic Review 92(2): 444-449. Coates, D., B. R. Humphreys, J. Kane, and M. A. Vachris. 2004. “No Significant Distance” Between face-to-Face and Online Instruction: Evidence from Principles of Economics. Economics of Education Review 23(6): 533–546. Conrad, C. A. 1997. Computers and Pedagogy: Lessons from Other Disciplines. Paper presented at the Allied Social Sciences Association Meetings, New Orleans LA. Dutton, J., M. Dutton, and J. Perry. 2001. Do Online Students Perform as Well as Lecture Students? Journal of Engineering Education 90(1): 131-136. Dutton, J., M. Dutton, and J. Perry. 2002. How do Online Students Differ from Lecture Students? JALN 6(1). Fizel, J.L. and J.D. Johnson. 1986. The Effect of Macro/Micro Course Sequencing on Learning and Attitudes in Principles of Economics. The Journal of Economic Education 17(2): 87-98.


16

Harrington, D. 1999. Teaching Statistics: A Comparison of Traditional Classroom and Programmed Instruction/Distance Learning Approaches. Journal of Social Work Education 35: 343-352. Howsen, R. and S. Lile. 2008. A Comparison of Course Delivery Methods: An Exercise in Experimental Economics. Journal of Economics and Finance Education 7(1): 21-28. Koenker, R. 2005. Quantile Regression No. 38. Cambridge University Press. Koenker, R. and K. Hallock. 2001. Quantile Regression. Journal of Economic Perspectives, 15(4): 143-156. Lundberg, J., D. Castillo-Merino, and M. Dahmani. 2008. Do Online Students Perform Better than Face-to-face Students? Reflections and a Short Review of some Empirical Findings. Monograph: The Economics of E-learning, Revista de Universidad y Sociedad del Conocimiento 5(1). Liu, X., R. MacMillan, and V. Timmons. 1998. Assessing the Impact of Computer Integration on Students. Journal of Research on Computing in Education 31(2): 189-203. Lopus, J.S. and N.L. Maxwell. 1995. Teaching Tools: Should We Teach Microeconomic Principles Before Macroeconomic Principles? Economic Inquiry 33(2): 336-350. Navarro, P. 2000. Economics in the Cyber classroom. Journal of Economic Perspectives 14(2): 119-132. Navarro, P. and J. Shoemaker. 2000. Policy Issues in the Teaching of Economics in Cyberspace: Research Design, Course Design, and Research Results. Contemporary Economic Policy 18(3): 359-366. Neuhauser, C. 2002. Learning Style and Effectiveness of Online and Face-to-Face Instruction. American Journal of Distance Education 16(2): 99-113.

Oblinger, D. G., and B. L. Hawkins. 2006. The Myth about No Significant Difference: Using Technology Produces No Significant Difference. EDUCAUSE Review 41(6): 14–15. Primont, D. and R. M. Summary. 1999. An Analysis of Student Attitude and Performance in Web-Enhanced Economics Courses. Manuscript, Southeast Missouri State University. Ray, M. A. and P.W. Grimes, P. W. 1992. Performance and Attitudinal Effects of a Computer Laboratory in the Principles of Economics Course. Social Science Computer Review 10(1): 42-58. Russell, T. L. 1999. The No Significant Difference Phenomenon, North Carolina State University, Raleigh, NC.


17

Smeaton, A. F. and G. Keogh. 1998. An Analysis of the Use of Virtual Delivery of Undergraduate Lectures. Computers and Education 32(1): 83–94. Stephenson, K., A.McGuirk, T. Zeh, and D. Reaves. 2005. Comparisons of the Educational Value of Distance Delivered versus Traditional Classroom Instruction in Introductory Agricultural Economics. Review of Agricultural Economics 27(4): 602-620. Switzer, D. and K. Rebeck. 2015. Determinants of Success in Economics Principles: Online vs. Face-to-Face Courses. Perspectives on Economic Education Research 32(1): 86-99. Talley, D. 2000. Technology and Teaching: Learning in a High-Tech Environment. Presented at the Midwest Economics Association, Chicago, IL. Terry, A. and K. Galchus. 2003. Does Macro/Micro Course Sequencing Affect Student Performance in Principles of Economics Courses? Journal of Economics and Finance Education 2(2): 30-37. Wallace, D. and P. Mutooni. 1997. A Comparative Evaluation of World Wide Web-Based and Classroom Teaching. Journal of Engineering Education 86(3): 211–219. Weems, G. H. (2002). Comparison of Beginning Algebra Taught Onsite versus Online. Journal of Developmental Education 26(1): 10-12.


18

Appendix: OLS Regressions of Student Performance (standard errors in parentheses)

Variable MICRO MACRO

TYPE -0.102*** (0.022)

-0.059* (0.031)

MAJOR -0.005 (0.012)

-0.006 (0.022)

FR -0.053** (0.023)

-0.078* (0.047)

SO -0.012 (0.022)

-0.002 (0.032)

JR -0.017 (0.021)

0.007 (0.029)

ACT 0.009*** (0.002)

0.017*** (0.003)

CR_HRS -0.004 (0.003)

-0.004 (0.004)

GPA_C 0.088*** (0.014)

0.023 (0.025)

MATH -0.029* (0.017)

-0.118*** (0.039)

GRADE_EX 0.049*** (0.016)

0.044 (0.028)

STUDY_TST 0.001 (0.002)

0.002 (0.003)

READ_TXT -0.003 (0.003)

-0.006 (0.005)

HW_MD 0.383*** (0.059)

0.341*** (0.071)

MEL_D1 0.032* (0.017)

-0.056 (0.033)

MEL_D2 -0.003 (0.016)

0.067** (0.027)

FATHER_E 0.038** (0.016)

0.005 (0.023)

MOTHER_E -0.042*** (0.014)

0.017 (0.024)

ECFI_F -0.036*** (0.013)

0.008 (0.021)


an analysis of student performance in online vs. face-to...

Documents