online program effectiveness

On-line Program Effectiveness 1

Running head: EVALUATING ON-LINE PROGRAM EFFECTIVENESS

Evaluating program effectiveness relative to the immediacy of technologies of an

Internet-based system

Research Prospectus: OM8022 – Survey of Research Methods

Winter Term 2011

Capella University

bross2011


Introduction

“Web-based instruction (WBI) is becoming a favored training option in industry,

government, and higher education, Sitzmann, et al, 2006, pg. 623.” Further, it is

through web-based instruction that has served as the new teaching and learning

environment for colleges and universities to move from the brick-and-mortar

instructional delivery system. “Electronically offered programs both support and extend

the roles of educational institutions. Increasingly they are integral to academic

organization, with growing implications for institutional infrastructure, COC, 2011, pg.1.”

As with any program infrastructure there is a need to have a method of measuring

effectiveness not only by courses satisfaction levels from a student or instructor

perspective but the degree of the program’s effectiveness relative to the rapid changes

in technology. In a case study commissioned by the National Education Association

and Blackboard, Inc., research was conducted and an overriding question was posed,

Phipps and Merisotis (2000) “are the current benchmarks appropriate and necessary to

ensure quality Internet-based distance education, pg 8.” This query although written

eleven years ago and with technology ever evolving as well as the life of most

technologies decreases rapidly from the time of introduction to the next version or

edition of the software, equipment or the like is still a necessary consideration. For

example, Blackboard is the leader in Internet-based course management software. Its

technologies are aligned with the Apple line of computer equipment which includes,

iPad and iPod. In 2010, Steve Jobs, the CEO of Apple, Inc. introduced the Apple iPad,

which was to be very consumer-friendly and portable. Almost a year later, the second


version of the Apple iPad is introduced. How does a change in technology impact the

level of program effectiveness for organizations and the infrastructure? Moreover, at

what point does the organization move from an old system, and old being in terms of

technology years, to a newer one? Further, does this transition increase the programs

overall effectiveness? The same is for the case with the Blackboard 7 system to the

current Blackboard 9. Therefore the overriding research question that will be

investigated in this study is what is the correlation between program effectiveness and

the immediacy of technology? In order to sufficiently address the research question, a

qualitative approach which will include surveys of faculty perceptions of instruction and

technology used and perceived learning of the students; students’ satisfaction of

instruction, technologies employed; and educational program reports, technology

reports and surveys of a sample of six member institutions of the eight accrediting

bodies in the United States. The questions will be derived from the benchmarks that

have been identified by stakeholders to determining an effective on-line program as the

conceptual framework. Evaluating program effectiveness is significant because, as

stated initially by Phipps and Merisotis (2000) “are the current benchmarks appropriate

and necessary to ensure quality Internet-based distance education, pg 8.” The same

concerns of eleven years ago is the same today. Further, “whether as a promise or as

a threat, understanding this new form of educational engagement is imperative in

preserving and improving the higher education enterprise, Thompson, 2004, pg. 64.


Problem/Opportunity Statement

There exists a pedagogical term that refers to web-based courses, “electronic

pedagogy or online pedagogy,”( Minkler, 2008, pg. 5). In the electronic pedagogy,

instructors learn to not only deliver instruction using the technology but based on the

learning style, the instructor must be able to vary the subjects of delivery based on the

learning styles.

It is important to recognize that the perception of most organizations relative to

online courses and programs that this is the answer to financial issues and providing a

product in an innovative way. However, teaching the virtual generation, Prosperpio &

Gioia (2007) indicates that “Technological and social changes in the wider environment

can have major implications for teaching and learning pedagogies.” Therefore, this

study will examine four areas in order to determine effectiveness but more importantly

determine what are the correlating benchmarks of technology immediacy and :

instruction in an on-line environment; learning styles; management expectations; and

the technologies connecting the learning and instruction.

Studies have been conducted to determine if learning and instruction in the

traditional context is the same for a web-based/online environment. These comparisons

have used some of the commonly used learning style models in order for instructors to

develop a plan for teaching. Instructors however, have for the most part focused on

teaching and their perception of how students should learn has not changed from the

context of traditional learning environment to the online environment. This study will


focus on the effectiveness of programs based on the instruction and learning in an

online environment. Courses have traditionally been deemed effective in accomplishing

the learning outcomes based on the student’s performance. These learning outcomes

have been measured using traditional face-to-face models of instruction and learning.

There is a difference between technology in a traditional course and a true on-line

environment. This study based on the criteria and areas of focus in the benchmarks,

Phipps and Merisotis (2005), will identify which benchmarks for evaluating on-line

program effectiveness are correlated with the immediacy of technology.

Purpose Statement

The purpose of this study is determine if the level of effectiveness in a program is

dependent upon the technologies used or is it solely based on the commonly used

benchmarks. “With the rapid development of technology, online instruction has

emerged as an alternative mode of teaching and learning and a substantial supplement

to traditional teaching,” (Tallent-Runnels et al, 2006, pg. 93). With this emergence of

this new technology, online courses in the United States have increased by 33% each

year, since 2002, (Tallent-Runnels et al, 2006). Rapid increases in the technology

available and the number of courses offered in this new environment have not occurred

at the same rate for the evaluation of such courses or programs. Because of this

evolution in learning and technology, the process for assessing its effectiveness is as

varied as the courses and programs offered. Evaluation of courses and programs need


to reflect benchmarks not solely based on responses by a research participant, but

show the different levels of effectiveness relative to technology.

Conceptual / Theoretical Framework

The conceptual framework, Figure 1 that is used in this study is the Best

Practices (SAC et al, 2005) and the benchmarks for evaluating an on-line program. The

Best Practices identifies five areas of focus: institutional context and commitment,

curriculum and instruction; faculty support; student support and evaluation and

assessment. Making the parallel to the benchmarks, the diagram shows based on

specific criteria, certain actions/decisions are made. Further the levels of importance of

each benchmark are connected to one of the Best Practices that are identified. As such

these areas are identified and described as critical components for a quality program to

exist. Further, these areas include the benchmarks from the various stakeholders. Each

focus has specific questions which are criteria to be met through the program/course

offerings. Based on the responses and degree for which these questions are given

response indicates the level meeting the standard. The four areas are institutional

context and commitment which is where this process begins. In order to carry out the

institutional context and commitment to online courses and programs, a curriculum and

instruction plan is developed. It is through this plan that the type of instruction and

learning expected is derived.


As a result, the faculty is provided support and training in order to teach in the

environment. This is where the adoption of an online pedagogy occurs. This on-line

pedagogy is a plan which is the overlay of the traditional program with the on-line

program. The student is provided with support in terms of technology to demonstrate

understanding of concepts and skills learned in the environment. The evaluation and

assessment component is constantly reviewed. “Both the assessment of student

achievement and evaluation of the overall program take on added importance as new

techniques evolve, (SACS, 2005, pg. 12).” The results of such inquiries are used to

guide curriculum design and delivery, pedagogy, and educational processes, and may

affect future policy and budgets decisions. Therefore, showing whether technology

immediacy influences the level of effectiveness of a program is important because a

number of determinations are made based on the outcome. The research pursued as a

result of this conceptual framework will on those indicators that are evaluated to

determine a program’s effectiveness and the influence from technology that affects the

level of program effectiveness in an on-line environment.

Management / Research Questions

The management dilemma that this study is to address is the effectiveness of a

program influenced by the immediacy of technology and if it is influence to a degree,

how different are the outcomes based on the technology used by the various

stakeholders. From this dilemma the research questions that arise are based on the

technologies, what indicators affect the level of effectiveness of a program? How does


technology immediacy influence the effectiveness of the program from the stakeholders’

perspective? 1. What are the common threads of the responses given by the

stakeholders to determining the indicators of a quality on-line program? Are the

influences of the indicators similar to each of the stakeholders? What factors influence

the stakeholders to identifying an indicator of quality? Finally, based on the indicator

rankings, are they significant to provide the criteria for quantifying or qualifying as a

measure of effectiveness relative to the technologies?

Literature Review / Current Understanding of the Phenomena

In order to evaluate the effectiveness of a program based on technologies, more

specifically its immediacy, the components which are critical to the evaluation process

must be identified and evaluated. It is from this evaluation of courses that reverts back

to the evaluation of programs which in turn reverts back to the organization that

determines whether or not the program is effective. The first part of this process is to

define the components to be evaluated. Proserpio & Gioia (2007) identifies the learners

of the current generation in an exploratory research article discusses instructors of the

virtual generation (V-Gen) to become accountable and incorporate the technologies that

have come to past in this technological revolution. It further questions whether or not

technology-influenced instruction which would lead to a better understanding of

management and organizational phenomena for the virtual-generation student. The

article attempts to provide answers to this and other questions relative to the question of

keeping current with the technologies and updating teaching methodology through


philosophical supposition. The significance of this report resides in the

acknowledgement of the new generation of learners, virtual, and how the teacher

transitioned initially from the TV (television) generation to now the virtual generation.

This paradigm shift is more a lateral shift rather than a vertical one. Whereas with the

TV generation instruction was about taking the tactile activities and making them visual

activities; now the shift is to make it more interactive. Teaching the virtual generation,

Prosperpio & Gioia (2007) indicates that “Technological and social changes in the wider

environment can have major implications for teaching and learning pedagogies, pg. 72.”

Minkler (2008) dissertation focused on the connection of various teaching styles

and connecting the learning in an on-line environment. Using as its conceptual

framework, Grasha (1996) Integrated Teaching and Learning Styles Model was used.

Such a model has not previously been used in studies regarding on-line environments.

This mix-methods research design, conducted telephone interviews with 12 survey

respondents and completed web-based surveys by 113 faculty of on-line programs at

41 community colleges in nine states of United States regarding perceptions and

expectations related to the on-line learning. It was concluded through this study placed

more emphasis on the teaching rather than the learning. Further the researcher

acknowledges that the model used was effective for this study but it fell short on

meeting the parameters of how teaching and learning is handled in an online

environment.

On-line courses and programs have “Although the current headlong rush by

some to move this little-understood but potentially transformative educational


phenomenon into the mainstream can lead to impatient demands for evaluation

information to justify projects, such impatience must be resisted. Only thoughtful and

focused examination of online courses and programs will reveal the ways in which

individuals and institutions can maximize the promise and remove the threat of online

education, Thompson, 2004, pg. 65.”

Technologies designed for instruction and learning continue to focus on a

course-by-course method. The literature of the research conducted relative to the

instruction and learning in an on-line environment and its management is varied.

Further the evaluation of effective strategies and techniques for on-line instruction and

learning is scarce. The field is relatively new and guides to the types of inquiries and

theories related to the inquiry is limited or not directly addressed.

Methodological Approach & Research Design

In considering the research design of this study, the sequential explanatory

approach would be employed because it takes the ethnographic surveys to be used in

this study and determines if there is a correlation between the survey responses made

by the identified stakeholders and the further investigation responses done with

interviews and/or focus groups. Further, because the research is to determine if there is

a correlation using quantitative data analysis of relationships between two variables

using Pearson correlation. Qualitative data analysis is to search for important meanings,

patterns and themes in what the research has heard and seen, pg 236.” Research

questions for this study would be an override mix-method questions followed by the


sub-questions of a quantitative and qualitative nature as the sequential explanatory

approach would suggest. Using the mixed-methods approach data is collected from

regarding students and instructors. It is through this analysis that the learning,

instruction and technologies identified will be connected to the expected learning

outcomes and the measures used to determine program/course effectiveness. In order

to determine the significance of technology immediacy influence on the effectiveness of

an on-line program key variables that would be the focus of this study and examination

are the faculty instruction evaluation instruments, (administrative and student survey),

students interviews and focus group session. The study would not only seek to explain

the influence but understand the behaviors and other nuances surrounding the different

outcomes when looking at the ever changing technology.

Sources of Information & Measurement Plan

The sources whereby the information for this study will be measured based on

the institution’s formula for gauging whether or not the program is effective. These

instruments used for evaluation of the courses and/or programs would be best

determined by the organization and a ruling on what aspects would be considered in

this process will determined.

Data Collection Plan

The data collection plan that will be used for this study will consist of

strategic and institutional effectiveness plans of 3 private and 3 public institutions that


are similar in size and composition. Also, a survey sent to faculty, professional listservs

and organization, administrators, technology professionals, as well as students of the

institutions selected for the study. Faculty would be considered those at the By looking

at the various institutions of higher learning similarities in terms of teaching, learning

and technologies will provide an opportunity to relate information obtained to the level of

effectiveness achieved. After surveys have been distributed, based on the analysis of

the data will determine the types of questions that will be generated in the focus groups.

All participants will receive the Informed Consent Form. After the data analysis the raw

information will be destroyed. The surveys, given that they will be administered for the

most part on-line will be deleted. These measures will be taken in order to ensure

confidentiality of the participants.

Data Analysis Plan

The data analysis plan will follow two stages. The first stage will be spent coding and

analyzing the date. In this analysis, the types of responses would be coded based on

specific types of answers. Then a statistical analysis conducted to determine the

number of occurrences of the answers. After the statistical analysis, then based on the

responses, the answers and feedback received from the focus groups will be analyzed.

Then a correlation between the answers given on the survey, benchmarks, and focus

groups will be conducted to determine if in fact the outcomes would be the same

despite the changes in technology.


Significance / Potential Contributions

Evaluating program effectiveness is significant because, as stated initially by

Phipps and Merisotis (2000) “are the current benchmarks appropriate and necessary to

ensure quality Internet-based distance education, pg 8.” The same concerns of eleven

years ago are the same today. Further, “whether as a promise or as a threat,

understanding this new form of educational engagement is imperative in preserving and

improving the higher education enterprise, Thompson, 2004, pg. 64. It is also important

that stakeholders know that program effectiveness is greater than items that meet

accreditation standards. It is important that in the evaluation of program effectiveness it

is because of the program and is not solely contingent upon the technologies of the day.

Ethical Considerations

The unique ethical considerations of conducting research in this study would

require that the environment particularly in the data collection phase where the research

is conducted would not show any propensity to appear bias in the report of its finding

based on the financial or any other external support provided. Therefore, the surveys

will be done on-line. This will significantly decrease the knowledge of the participants

with the research.

Personal Reflections and Insight

As a researcher working through this process and understanding the work with

the incorporation of every-changing technology, I have found such research to be very


important and it more about speed, efficiency and analysis-enabled for a program and

the immediacy of the technology. Further, there are a number of models used to

determine quality and trying to provide a uniform set of criteria for evaluation is an

arduous task. Research that is over five years old can provide an opportunity to adding

new knowledge to the existing one. There are a number of assumptions made when

determining the quality of on-line programs. Some research has determine that

technology specialists are stakeholders and rightly so considering institutions have a

formula of investment where in previous years it was a foregone conclusions that

whatever is needed to make it happen was allocated. This is due to economic

constraints.


References

Biktimirov, E., Klassen, K., (January/February, 2008). Relationship between use of

online support materials and student performance in an introductory finance

course, Journal of Education for Business, 153-158.

Byrne, R. (2002). Web-based learning versus traditional management development

methods. Singapore Management Review, 24, 2, 59. Retrieved from Business

Source Complete database.

Commission on Colleges, Southern Association of Colleges and Schools. (2005). Best

practices for electronically offered degree and certificate programs. Retrieved

from http://www.sacscoc.org/pdf/081705/commadap.pdf

Committee on Science, Engineering and Public Policy, National Academy of Sciences,

National Academy of Engineering and Institute of Medicine (1995). On being a

scientist: Responsible conduct in research, Washington, DC: National Academy

Press.

Edwards, C., & Fritz, J. H. (1997). Evaluation of three educational online delivery

approaches. (ERIC Document Reproduction Service No. ED 430-516)


Hay, A., Hodgkinson, M., Peltier, J. W., & Drago, W. A. (2004). Interaction and virtual

learning. Strategic Change, 13, 4, 193-204. doi:10.1002/jsc.679

Hirner, L. J. (2008). Quality indicators for evaluating distance education programs at

community colleges, (Ph.D., University of Missouri-Columbia)., 452. Retrieved

from http://proquest.umi.com.library.capella.edu/pqdweb

Minkler, S. (2008). Connecting teaching styles and student learning styles in community

college online courses. (Ed.D., University of Hartford)., 255. Retrieved

from http://proquest.umi.com.library.capella.edu/pqdweb

Phipps, R., & Merisotis, J., (2000). Quality on the line: Benchmarks for success in

Internet-Based distance education (ERIC Document Reproduction Service No.

ED44-407)

Proserpio, L., & Gioia, D. (2007). Teaching the virtual generation. Academy of

Management Learning & Education, 6, 1, 69-80. Retrieved from Business Source

Complete database.


Shrag, B., (2005). Teaching research ethics: Can web-based Instruction satisfy

appropriate pedagogical objectives?, Science and Engineering Ethics, 11, 347-

366

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher. R., (2006). The comparative

effectiveness of web-based and classroom instruction: A meta-analysis,

Personnel Psychology, 59, 623-664.

Swanson, R.A., Holton, III, E. F. (2005). Research in organizations: Foundations and

methods of inquiry. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Tashakkori, A. & Creswell, J.W. (2007). Editorial: Exploring the nature of research

questions in mixed methods research, Journal of Mixed Methods Research, 1, 3,

207 – 211.

online program effectiveness

Documents