the use of a scoring rubric in an online research methodology course

International Journal of Education

ISSN 1948-5476

2012, Vol. 4, No. 2

www.macrothink.org/ije 1

The Use of a Scoring Rubric in an Online Research

Methodology Course

Kayla Bartlett*, Jacqui Floyd*, Shanda Davis*, Greg Haas*, Kathy Cox*,

Anthony J. Onwuegbuzie** & Rebecca K. Frels1*

*Department of Educational Leadership and Counseling, Sam Houston State University

Box 2119, Huntsville, Texas 77341-2119, USA

E-mail: [email protected]

1*Lamar University, Beaumont, Texas, USA

Received: November 19, 2011 Accepted: January 16, 2012 Published: April 4, 2012

doi:10.5296/ije.v4i2.1604 URL: http://dx.doi.org/10.5296/ije.v4i2.1604

Abstract

Research methodology courses can be the most difficult courses in master’s-level programs

representing the social, behavioral, and health sciences because, in these courses, students

typically are expected to learn to think critically and contextually about social and/or

academic problems in addition to learning new terminology and methodological concepts not

previously part of each specific discipline. Further, the challenges of online learning might

increase due to the nature of research methodology courses and the new concepts taught.

Thus, as students and instructors of an online research methodology course, we describe the

use of a scoring rubric as a performance assessment and provide our student research

proposal project as an exemplar of effectively developing research knowledge, skills, and

dispositions for use in future online learning of research methodologies.

Keywords: research course, performance assessment, online, research methods, rubric


ISSN 1948-5476

2012, Vol. 4, No. 2


Research methodology courses often represent required courses in graduate-level

programs—particularly, master’s-level programs—among virtually all disciplines in the

social, behavioral, and health science fields and beyond. In these courses, graduate students

typically are expected to learn to think critically and contextually about ways that research

impacts social problems (Vandiver & Walsh, 2010). As noted by Onwuegbuzie, Slate,

Paterson, Watson, and Schwartz (2000):

Their experiences in these courses might ―cement their attitudes towards the field of

research, and thus determine whether they become consumers of research in the future

(Onwuegbuzie, DaRos, & Ryan, 1997; Ravid & Leon, 1995). However, although the goal

of many educational research instructors is to produce students with research consumer skills

(i.e., the ability to read, to interpret, to synthesize, and to utilize research) and research

production skills (i.e., the ability to design and to implement original research studies) (Ravid

& Leon, 1995), many students who have completed research methodology courses report

being inadequately prepared either to understand or to conduct research (Fleming, 1988;

Green & Kvidahl, 1990; Rackliffe, 1988). Moreover, many students typically experience

lower levels of performance in these courses than in their other graduate-level classes.

These students often view educational research courses as merely a hurdle, which they must

overcome in order to obtain their degrees… (p. 53)

Further, many students experience a unique anxiety in such courses (Kracker, 2002;

Murtonen, 2005; Onwuegbuzie, 1997a, 1997b; Papanastasiou & Zembylas, 2008).

According to the Sloan-C (2009) national survey of online learning, the number of

students taking online courses in higher education has grown at a higher rate than has the

number receiving face-to-face instruction, with approximately 30% of the total higher

education student population enrolled in at least one online course. The combination of

teaching a research methodology course and teaching online, thus, can be problematic. In fact,

pedagogical strategies differ tremendously between research methodology courses taught

online and research methodology courses taught face-to-face. Indeed, as surmised by

Ivankova and Stick (2007), online courses create the need for instructors of research

methodology courses to change their pedagogical roles from being teachers to serving as

facilitators of cognitive development, learning, educational growth, and achievement. Yet,

regardless of the instructional format, two components are critical in order for students to be

successful in research methodology courses: autonomy and self-directed learning (Girod &

Wofcikiewicz, 2009; Onwuegbuzie, Frels, Leech, & Collins, 2011). As such, the purpose of

this article is to provide support for instructors of online research courses through the use of a

scoring rubric both to guide and to assess the comprehensive project that is assigned in many

research methodology courses—the research proposal. Specifically, as students who wrote

the research proposal and instructors who taught the course, we collaboratively introduce the

components of the Scoring Rubric (Onwuegbuzie, 2011) and provide our exemplar of the

final product written in a master’s-level research methodology course.

Conceptual Framework

Hutchinson (1995) explained that performance assessments can utilize flexible time


ISSN 1948-5476

2012, Vol. 4, No. 2


frames, open-ended formats, and cooperative and collaborative learning techniques. Further,

as noted by Elliot (1995), students’ levels of performance can be improved when the

performance assessment comprises the following components: (a) selecting assessment tasks

that are aligned clearly and are connected to the material being taught; (b) specifying clearly

the scoring criteria for the assessment task to students prior to attempting the task; (c)

providing students with explicit statements of standards and/or various models of acceptable

performance before they attempt a task; (d) encouraging students to undertake

self-assessments of their performances; and (e) interpreting students’ performances by

comparing them to those of other students, as well as to standards that are developmentally

appropriate. Therefore, the use of a rubric within the framework of performance assessment

for teaching and learning research methods promotes active learning, which, in turn, and as

explained by Cobb (1993), increases students’ sense of responsibility. The remainder of this

article is presented using the five components of performance assessment (Elliot, 1995) as

they pertained to our course: (a) Assessment Tasks: The Research Proposal; (b) The Criteria:

The Scoring Rubric; (c) Standards for Performance: Instructor Feedback; and (d)

Self-Assessments: An Exemplar of the Research Proposal; (e) Interpreting Performance: A

Conclusion of the Process. The student exemplar of the final product is presented in

Appendix A and the Scoring Rubric: Part I and Part II is presented in Appendix B. Finally, in

Appendix C, we present an example of instructor feedback using the Scoring Rubric Part I

and Part II.

Assessment Tasks: The Research Proposal

In a recent study by Frels, Onwuegbuzie, Leech, and Collins (2011), whereby they

interviewed 12 leading methodologists who also taught research methodology courses, the

primary project of a research proposal was assigned most often as a final project. As noted by

Fink (2003), a final project is important for the application of skills in student learning. Thus,

as instructors of the research course, we incorporated the research proposal as a type of

performance assessment, which as an assignment that is clearly aligned with course

objectives and, as noted by Elliot (1995), connects students intimately to the material being

taught.

After moving through the phases of foundational learning of research philosophy,

research traditions (e.g., quantitative, qualitative, mixed methods), research sampling,

research design, and various research data collection methods and various mediating

assignments (e.g., conducting a survey, interpreting basic statistics), student groups are

instructed to select a topic and to create the research prospectus. The research prospectus is

the first draft of the research proposal, whereby students are required to provide sufficient

information about the statement of the problem, list of potential topics and/or headings for the

literature review section, theoretical framework/conceptual framework, rationale of the study,

statement of the purpose, research questions, hypotheses (if relevant), educational

significance, population/context, sampling frame, selection criterion, participants, instruments,

procedures, and analyses. After receiving feedback on the prospectus assignment, student

groups embark on the final assignment of the research proposal. The syllabus of the course

presents this assignment as follows:


ISSN 1948-5476

2012, Vol. 4, No. 2


Research Proposal (150 points/300 points)

Each cooperative learning group will submit a written research proposal. Each research

proposal is worth 150 points. The goal is to allow students to practice formulating research

questions, conducting reviews of the literature, selecting the research design and sampling

design, identifying and/or designing the instrument(s), developing the procedures, selecting

appropriate analyses, and identifying potential limitations of the study. That is, the research

proposal should contain all the major elements of a traditional research proposal. Simply

put, the goal is to allow students the opportunity cooperatively to practice the

decision-making skills required for planning a research study with feedback from the

instructor. To this end, the research proposal must be original in its entirety. It is expected

that, upon completion of the proposal, students will be very familiar with the research process.

As such, the research proposal will play a major role in demystifying the research process.

The main body (i.e., not including reference pages and appendices) must not exceed 20 pages

using 12-point font, 1-inch margins all around, and double spaces throughout. That is, the

reference list page should not begin any later than on page 21. Manuscripts in which this

20-page limit is exceeded will not be graded and will be assigned a score of zero. Please note

that one point will be deducted for every missing, incomplete, or inconsistent reference.

Detailed feedback will be given, utilizing a scoring rubric. For each individual, his/her group

score will be weighted by the participation score, such that if he/she receives 100% of the

participation points available, his/her individual score will be exactly equivalent to the group

score. If the student receives 90% of the participation points available, his/her individual

score will be worth 90% of the group score, and so on. This research proposal should be

completely written in APA style (sixth edition). Thus, you should check, if you are not

certain, to verify all the parts that must be included in a research proposal. Your submission

must include all of these parts. Missing parts and APA errors will result in substantial

reductions in the grade you receive.

The Criteria: The Scoring Rubric

Before beginning the research proposal, each group decides on the research tradition:

quantitative, qualitative, or mixed methods. For this assignment, according to which approach

is selected, students are given a checklist that is specific to the components needed for the

selected approach. For the purpose of this article and because we as students selected to

design a mixed methods research study, we will discuss this checklist, hereafter referred to as

the Scoring Rubric, which was specific to mixed methods research. The utility of discussing

this rubric is that virtually all of the items that appear in both the Scoring Rubric specific to

quantitative research and the Scoring Rubric specific to qualitative research appear in the

Scoring Rubric specific to mixed methods research. The Scoring Rubric comprises two

sections: (a) content (Part I), the rubric for instructors for feedback pertaining to the essential

components of mixed research; and (b) style (Part II), the rubric for feedback pertaining to

adherence to American Psychological Association (APA, 2010) guidelines.

Part I of the Scoring Rubric pertains to 10 essential sections of a research report: (a) title,

(b) method, (c) instruments, (d) procedure, (e) analysis, (f) legitimation, (g) results, (h)


ISSN 1948-5476

2012, Vol. 4, No. 2


discussion, (i) reference list, and (j) appendix. Appendix A presents the Rubric Part I and the

items included in the 10 essential sections. It should be noted that as instructors of the course,

we continually revise and update the Scoring Rubric with new concepts, theories, procedures,

and language as they emerge.

Part II of the Scoring Rubric considers the writing proficiency of student groups in the

presentation of concepts. Further, Part II of the Rubric is presented in the same organizational

format as Part I. Similar to Part I, it allows the instructor to provide additional points for a

student who has provided detailed references. Thus, Part II of the Checklist facilitates the

technical application of writing style important for students to succeed as self-directed

learners. Appendix A also presents Part II of the Rubric (i.e., adherence to APA [2010] style

guidelines), directly after Part I of the Rubric.

Standards for Performance: Instructor Feedback

As described by Frels, Sharma, Onwuegbuzie, Leech, and Stark (2010), the act of

providing detailed feedback on the part of instructors is central for instilling a sense of

direction and autonomy for student learning. It is the feedback of the research proposal that

instills confidence and allows students to reflect on their levels of performance for greater

responsibility in future actions (Butler, 2004). As instructors of the research methodology

course, we utilize the Review tab of Microsoft Word—specifically the comment and tracking

function—to provide details for students pertaining to both Part I and Part II of the Scoring

Rubric. Appendix B presents some examples of feedback on the exemplar assignment

presented in this article.

In addition, even though students progress to other courses in a master’s program, by

attending and affirming the feedback, students increase awareness of research components

and increase writing skills important for success in future courses and careers. Kolb (1984)

described this process as a reflective and meaning-making endeavor and specified that

attending (spending time with the event) and affirming (accepting and valuing the feedback,

even if the first response might be to disregard it) represented positive and developmental

components of learning.

Self-Assessments: An Exemplar of the Research Proposal

Elliot (1995) explained that the performance assessment includes the means for students

to undertake self-assessments of their levels of performance. Uniquely, the Scoring Rubric is

the guide for constructing the research proposal and by working in groups, students begin the

phase of self-assessing not only this major project, but also the overall learning of content to

be applied in the project. As students, we selected the topic of school culture in exemplary

Title 1 schools (i.e., U. S. federal programs that provide money to schools with high numbers

or high percentages of students from low-socioeconomic backgrounds) and, as noted

previously, we used a mixed methods research approach and the aforementioned specific

rubric to guide our writing. After feedback, and through a type of self-assessment of our final

product, we hope that our exemplar can serve as a model for future instructors and students

alike to address the component in performance identified by Elliot (1995) for a standard


ISSN 1948-5476

2012, Vol. 4, No. 2


and/or model of acceptable performance. The completed research proposal assignment is

presented in Appendix A.

Interpreting Performance: A Conclusion of the Process

Our exemplar of the research proposal presented in Appendix A presents the many

details important to the research process and important to include when writing a research

proposal. As noted by Elliot (1995), it is important to assess and to interpret performance

with respect to other students, as well as with regard to standards that are appropriate.

Therefore, the use of the Scoring Rubric as our guide through the mode of group work,

specifically—evaluating each other’s contribution to the group project—facilitated

interpretation of both the process and performance. The social processes that are connected

to the acquisition of knowledge (Piaget, 1954; Vygotsky, 1978) were a component of the

performance assessment and the use of the Scoring Rubric to stimulate our thought processes,

questions to each other, and conversations. The interaction of writing and feedback (socially

and contextually) among students and between instructors and students after using the

Scoring Rubric is vital to the process for interpreting the level of performance in the research

methodology course. In closing, and as a final testimony to this interaction, we present our

collaborative outcomes of experiences in our online research methodology course. The

independent skills required to succeed not only with online learning specifically but also with

the online learning of research methodology, can be facilitated through the explicit guidelines

contained in instruments such as the Scoring Rubric, which might provide a type of launching

pad for self-directed learning. Further, we hope that by providing both the Scoring Rubric and

an example of our research proposal, we can contribute to the future strategies used by online

research methodology instructors to include performance assessment and an effective scoring

rubric to guide student learning.

References

American Psychological Association. (2010). Publication manual of the American

Psychological Association (6th ed.). Washington, DC: Author.

Butler, S. (2004). Question: When is a comment not worth the paper it’s written on? Answer:

When it is accompanied by a level, grade or mark! Teaching History, 115, 37-41.

Cobb, G. (1993) Reconsidering statistics education: A National Science Foundation

Conference. Journal of Statistics Education, 1(1). Retrieved by e-mail at:

[email protected].

Elliott, S. N. (1995) Creating meaningful performance assessments. ERIC Digest E531

(ERIC Document Reproduction Service No. ED381985).

Fink, L. D. (2003). Creating significant learning experiences in college classrooms. San

Francisco, CA: Jossey-Bass.

Frels, R. K., Onwuegbuzie, A. J., Leech, N. L., & Collins, K. M. T. (2012). Pedagogical

mailto:[email protected]


ISSN 1948-5476

2012, Vol. 4, No. 2


strategies used by selected leading mixed methodologists in mixed research courses.

Manuscript submitted for publication.

Frels, R. K., Sharma, B., Onwuegbuzie, A. J., Leach, N. L., & Stark, M. (2011). The Use of a

Checklist and Qualitative Notebooks for an Interactive Process Teaching and Learning

Qualitative Research. The Journal of Effective Teaching, 11(1), 60-77.

Girod, M., & Wofcikiewicz, S. (2009). Comparing distance vs. campus-based delivery of

research methods courses. Educational Research Quarterly 33(2), 47-61.

Hutchinson, N. L. (1995). Performance assessments of career development. ERIC Digest

(ERIC Document Reproduction Service No. ED414518).

Ivankova, N., & Stick, S. (2007). Online higher education programs: New challenges, new

frontiers, new learning, and new scholarship. In D. Wright, & M. Miller (Eds.), Training

higher education policy makers and leaders: A graduate program perspective (pp.

77–97). Charlotte, NC: Information Age Publishing/Greenwood Press.

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and

development. Englewood Cliffs, NJ: Prentice Hall.

Kracker, J. (2002). Research anxiety and students’ perceptions of research: An experiment,

Part I: Effect of teaching Kuhlthau’s ISP model. Journal of American Society of Science

Technology, 53, 282-294. http://dx.doi.org/10.1002/asi.10040

Murtonen, M. (2005). University students’ research orientations: Do negative attitudes exist

toward quantitative methods? Scandinavian Journal of Educational Research 49:

263–80. http://dx.doi.org/10.1080/00313830500109568

Onwuegbuzie, A. J. (1997a). The teacher as researcher: The relationship between research

anxiety and learning style in a research methodology course, College Student Journal,

31, 496-506. http://dx.doi.org/10.2466/pr0.1997.80.2.496

Onwuegbuzie, A. J. (1997b). Writing a research proposal: The role of library anxiety,

statistics anxiety, and composition anxiety. Library and Information Science Research,

19, 5-33. http://dx.doi.org/10.1016/S0740-8188(97)90003-7

Onwuegbuzie, A. J. (2011). The scoring rubric. Unpublished manuscript. Sam Houston State

University, Huntsville, TX.

Onwuegbuzie, A. J., Frels, R. K., Leech, N. L., & Collins, K. M. T. (2011). A mixed research

study of pedagogical approaches and student learning in doctoral-level mixed research

courses. International Journal of Multiple Research Approaches, 5, 169-202.

Onwuegbuzie, A. J., Slate, J., Paterson, F., Watson, M., & Schwartz, R. (2000). Factors

associated with achievement in educational research courses. Research in the Schools,

7(1), 53-65.

Papanastasiou, E. C., & Zembylas, M. (2008). Anxiety in undergraduate research methods


ISSN 1948-5476

2012, Vol. 4, No. 2


courses: Its nature and implications. International Journal of Research & Method in

Education, 31, 155-167. http://dx.doi.org/10.1080/17437270802124616

Piaget, J. (1954). The construction of reality in the child. New York, NY: Basic Books.

Sloan-C (2009). Learning on demand: Online education in the United States. Retrieved from

http://sloanconsortium.org/publications/survey/learning_on_demand_sr2010

Vandiver, D. M., & Walsh, J. A. (2010). Assessing autonomous learning in research methods

courses: Implementing the student-driven research project. Active Learning in Higher

Education, 11(1), 31-42. http://dx.doi.org/10.1177/1469787409355877

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.

Cambridge, MA: Harvard University Press.

Appendix A

An Exemplar of the Research Proposal:

School Culture in Exemplary Title I Schools

Kayla Bartlett, Jacqui Floyd, Shanda Davis, Greg Haas, and Kathy Cox

Sam Houston State University

Most people believe that all schools are the same from the outside looking in (Ayers, 1995).

The problem with that frame of mind is that schools are schools, not airplane factories or

dairy farms; schools look like schools, and schools are run as schools, but with a little closer

look, schools are extraordinarily different. The closer you get, the more different schools look.

Each school has a distinctive feel.

This feel is so persistent and potent that people experience it immediately when walking into

a school. Being accustomed to this experience there is worry about a school that lacks a

particular feel. Somehow it has not been inhabited. It does not have a history; and it does not

have a personality. People are not yet experiencing its culture.

Once there is a feel to the school's culture, however, it leaves a strong impression on

everyone. The combination of the structural design, the symbols, the artifacts, the look and

behavior of the people, the conversations had, and the treatment received—all this and much

more—creates a memorable impression.

One definition of school culture is the deep pattern of values, beliefs, and traditions that have

been formed over the course of its history (Deal & Peterson, 1990). A school culture is

formed by its school community, which includes teachers, administrators, board members,

business leaders, and the students themselves (Johnson & Christensen, 2010). Every member

of a school community is either part of the cultural mission of the school, an obstacle to that

mission, or part of a subculture with a different mission (Daresh, 2007). Our group believes

that a positive school culture, from leadership downwards, has a tremendous impact on the

performance of Title I schools. In this research we hope to identify what parts of a positive

culture impacts the schools the most, and how much leadership plays a part in developing that


ISSN 1948-5476

2012, Vol. 4, No. 2


culture (Sergiovanni, 1984). For ideas to research, we considered our own experiences as

teachers (Johnson & Christensen, 2010).

Review of the Related Literature

School culture includes the norms, values, and cultural artifacts shared by members of a

school, which affects the school’s overall functioning (Engels, Hotton, Devos, Bouckenooghe,

& Aelterman, 2008). Lunenburg and Ornstein (2008) further explained culture to include

the history of the organization; myths and stories that explain the organization; cultural norms;

heroes of the organization; and traditions, rituals, and ceremonies of the organization. The

information discovered about characteristics of effective schools, schools containing a

positive culture, leadership in exemplary schools, and effects of positive school culture on

student achievement will be shared.

Effective schools and schools with a positive culture. Effective schools involve strong

leadership, a climate of expectation, an orderly atmosphere, and effective communication

(Kelley, Thornton, & Daugherty, 2005). Relationships, including student to student, teacher

to student, teacher to family, administrator to staff, and school to community, are important in

creating good school culture (Keiser & Schulte, 2009). Healthy schools set high standards for

academics and possess appropriate leadership and collegiality (Macneil, Prater, & Busch,

2009). Furthermore, according to Macneil et al. (2009), these effective schools contain

several factors, such as goal focus, communication, optimal power equalization, resource

utilization, cohesiveness, morale, innovativeness, autonomy, adaptation, and problem solving

adequacy. More qualities of healthy schools include teachers who: feel good about their

students, have confidence in their students’ abilities, and possess a trustworthy relationship

between administration and staff (Henderson et al., 2005). High commitment and

performance also appear to be characteristics of schools that contain a healthy school culture

(Engels et al., 2008). Moreover, teacher collaboration is prevalent in a successful school

(Johnson, Snyder, Anderson, & Johnson, 1996). Schools that possess shared views of

fairness, justice, respect, cooperation, and compassion contain a positive sense of community

(Keiser & Schulte, 2009). According to Bryan and Henry (2008), Title I elementary schools

must utilize the strengths found in children, their families, and communities.

School-family-community partnerships help in establishing strong environments that students

need to be successful (Bryan & Henry, 2008).

Leadership. The link between effective school cultures and leadership is supported by

educational research findings (Macneil et al., 2009). In fact, there is a strong association

between effective principals and school cultures that support learning (Engels et al., 2008).

Southworth and Du Quesnay (2005) surmised that school leadership plays an important role

in the growth of schools. Overall, school culture, and the relationships that shape it, is

strongly guided by the principal (Macneil et al., 2009). A principal who builds a culture

promoting and encouraging learning helps students be successful. Viewing their school’s

environment in a holistic way allows leaders better to shape the values, beliefs, and attitudes

crucial to creating an ideal learning environment (Macneil et al., 2009). The principal should

act as an entrepreneur, with a vision, who is able to inspire and to motivate their staff (Engels


ISSN 1948-5476

2012, Vol. 4, No. 2


et al., 2008). According to Howley, Woodrum, Burgess, and Rhodes (2009), a school

operates best when leaders plan, implement policies, and initiate practices responsive to the

culture of the community surrounding the school. School leadership is most effective when it

embraces cultural norms and values of the community. In productive schools, answers to

learning problems take place when administrators facilitate problem solving and express

instructional standards (Johnson et al., 1996).

Student learning. School culture has an effect on student learning (Engels et al., 2008). In

particular, Henderson et al. (2005) concluded that effective leadership increases student

achievement. Teacher affiliation, resource support, and academic emphasis also strongly

correlate with student achievement. Schools that hold their students to high expectations and

maintain orderly environments obtain higher student achievement on standardized tests

(Henderson et al., 2005). In addition, a strong leader, the climate of the school, and the

attitudes of faculty and staff can directly affect student achievement (Kelley et al., 2005).

According to Kelley et al. (2005),

School climate might be one of the most important ingredients of a successful instructional

program. Without a climate that creates a harmonious and well functioning school, a high

degree of academic achievement is difficult, if not downright impossible, to obtain (p. 19).

Conceptual/Theoretical Framework

The framework for this research proposal will be designed with the intentions of being able to

apply our conclusions to help in the future development of Title I schools. Lester (2005)

states that ―a research framework is a basic structure of the ideas (i.e., abstractions and

relationship) that serve as the basis for a phenomenon that is being investigated‖ (p. 458).

Theoretical framework is based on formal theory, whereas conceptual frameworks are based

on a wide array of sources. With this in mind, we have developed strong ideas that have built

our framework. The first is that attitudes by staff and administration in schools have a strong

influence on children learning. Second, the implementation of Title I funding from the No

Child Left Behind (NCLB) Act has had a positive influence on improving economically

disadvantaged students education. Also, the Texas Assessment of Knowledge and Skills

(TAKS) test scoring system is an adequate measurement of educational progress within

Texas Public Schools. We believe that all these components work together to shape

successful Title I exemplary elementary schools.

Purpose of Study

More information is needed on school effectiveness with different populations, settings, and

sites—specifically, successful Title I campuses. Title I schools are very unique and

encompass unique students with an abundance of different needs. Much more information is

needed on how to help students and staff be successful at these types of schools. Although

research has been conducted on characteristics of effective schools, there is a lack of research

focusing on successful Title I schools specifically; and additional research is needed

regarding these schools. We believe that this study will contribute to the knowledge base of

what is known about effective Title I schools. Furthermore, we believe the findings will help


ISSN 1948-5476

2012, Vol. 4, No. 2


generate new ideas for other schools to implement. Finally, we hope the findings of this

research will have an impact on the school at which we teach and other similar schools. The

purpose of this study will be to investigate the characteristics of the school culture in 10

exemplary-rated Title I elementary campuses.

Research Question(s) (Grand Tour Questions/Sub-Questions)

The following research questions will be addressed: What are the strategies that are being

taken by teachers, administration, and staff within exemplary elementary schools to achieve

and to maintain exemplary status? How has/does parental involvement impact the academic

achievement of students in exemplary Title I elementary schools? What are the distinguishing

components in the climate and culture of exemplary Title I elementary schools? What is the

leadership style utilized to lead the campus to achieve exemplary status in Title I elementary

schools?

Significance of the Study

The research will target teachers, administrators, and other stakeholders involved in the

academic achievement of students. The researchers will identify the characteristics of

exemplary Title I elementary schools and the attitudes and behaviors of its students, teachers,

and staff. It is hoped that the results of this study will identify effective concepts that are

currently in place, and also enable the teachers, administrators, and other stakeholders to

develop new programs, strategies, interventions, and so forth that will yield positive

outcomes.

Method

Participants

The participants in this study will include administrators, teachers, and paraprofessionals of

socioeconomically disadvantaged (Title I) elementary campuses that have achieved and/or

maintained exemplary status. The sample members will be selected using convenience

sampling. According to Onwuegbuzie and Collins (2007), a convenience sample consists of

individuals who are available and willing to participate in a study. Purposeful random

sampling and triangulation will be utilized to enable the collection of credible data from the

selected respondents and to develop an understanding of the attitudes and behaviors that

contribute to the different school’s exemplary status. Johnson and Christensen (2010) defined

triangulation as involving looking for convergence of findings from various methods while

studying the same phenomenon.

Population context/size/population characteristics/selection-eligibility characteristics.

The 10 elementary schools we will study have to meet two major requirements. The

elementary campuses must be considered Title I eligible and recognized as Exemplary each

of the previous 3 years. Title I schools are campuses that receive federal funding because of a

high percentage of low-income students. The Elementary and Secondary Education Act

(ESEA) was enacted in 1965 by the Johnson Administration in an effort to give all children

an equal environment to learn. This title’s purpose is to ensure that every child has a fair and

https://docs.google.com/document/d/1HV-Pv-6xOlCcnr_t3cboCNvJJ56sJRk6Qof5seVe-14/edit?hl=en&authkey=CLenhpcM#_msocom_1

https://docs.google.com/document/d/1HV-Pv-6xOlCcnr_t3cboCNvJJ56sJRk6Qof5seVe-14/edit?hl=en&authkey=CLenhpcM#_msocom_1


ISSN 1948-5476

2012, Vol. 4, No. 2


equal chance to obtain a quality education and be proficient on state achievement standards

and academic assessments (U.S. Department of Education, n.d.b). This act is better known

today as NCLB. Forty percent of the students in eligible schools are considered low-income.

Examples of how the funds are used include providing additional funds for extra instruction

in reading and mathematics, and special programs such as preschool, after-school, and

summer school (U.S Department of Education, n.d.a). Funds also are allocated to help

involve parents in the education of their children, and to provide teachers and school staff

with substantial opportunities for quality professional development. Funds are used in an

overall attempt to close the achievement gap between low-and high-performance children

(U.S Department of Education, n.d.b).

The second requirement is that the school has reached exemplary status all of the previous 3

years, as determined by the Texas Education Agency (TEA). There are three standards by

which a campus may be labeled: Exemplary, Recognized, and Academically Acceptable. All

three of these labels are determined by TAKS tests scores. This test has five large areas that

are measured: Reading, Writing, Social Studies, Mathematics, and Science. Also, this test is

administered to students from third grade through 11th grade. Testing for different subjects

are staggered in years. For example, writing is tested in Grades 4 and 7, whereas social

studies is graded in Grades 8, 10, and 11. To determine a school’s standards, the number of

students who score high enough to pass the test is divided by the total number of students.

For a school to be Exemplary, at least 90% of students must pass all subject areas. For a

school to be included in our research, they must officially meet these TEA requirements

(TEA, 2010).

Sample size. Twenty-five participants from each of the 10 exemplary Title I elementary

schools in the study will be interviewed. Johnson and Christensen (2010) mentioned that an

intramethod mixing approach allows for quantitative as well as qualitative data to be obtained

through a single method. The sample size for the intramethod questionnaire will depend on

how many individuals respond.

Sample characteristics. It will be valuable to administer questionnaires to teachers,

administrators, counselors, and any other teaching assistants who are involved in the

day-to-day educational process on the campus. Johnson and Christensen (2010) explained

that researchers are not limited to face-to-face interviews, because telephone interviews are

acceptable forms of data collection.

Instruments

All 25 faculty and staff members at 10 exemplary Title I schools will be interviewed. The

interview will consist of six open-ended questions. The interview questions are as follows:

1. In your own words, what role does culture play in your Title I school?

2. Describe the atmosphere at your school.

3. How are parents involved in their child’s education?

4. How does the administration interact with the faculty and staff?


ISSN 1948-5476

2012, Vol. 4, No. 2


5. Which policies and procedures at your school help your students be successful?

6. What are the attitudes of faculty and staff at your school toward meeting an exemplary

standard?

Probing question will include: Can you think of some examples? Could you tell me more

about your thinking on that? Each participant will be interviewed only once, using a formal

interview guide approach. According to Johnson and Christensen (2010), this approach to

interviewing involves specific questions being asked in any order; and the interviewer has

freedom to rearrange the wording of questions. The interviews will be semi-structured,

allowing the researchers to talk conversationally with the participants. Members of the

research team will be responsible for interviewing at each school. The site used to interview

will be a convenient and comfortable location within the school, such as the school library.

At each school, the research team will interview three administrators (the principal, one

assistant principal, and one counselor), 15 classroom teachers, and seven paraprofessionals.

The interviews will be scheduled during each interviewee’s conference time and will last

between 20 and 30 minutes. Furthermore, a member checking or participant feedback

approach will be implemented. This method consists of the researchers discussing

conclusions made from the interviews with the actual participants to achieve accuracy

(Johnson & Christensen, 2010). This method will take place through a follow-up interview

with participants.

Rationale for selection of instrument. By interviewing participants, we will be able to

obtain a better grasp of the school culture through the participants’ own words. The

interviews will allow us to view better the participants’ attitudes, thoughts, and beliefs about

their school. In addition to the qualitative interviews, the research team will implement an

intramethod questionnaire. Johnson and Christensen (2010) mentioned that an intramethod

mixing approach allows for quantitative as well as qualitative data to be obtained through a

single method. It is cost efficient and will provide data in various formats. The purpose of the

questionnaire will be to obtain quantitative and qualitative data via a single tool. It will

contain questions about the culture of the school, concerning topics such as leadership,

relationships, parent involvement, curriculum, instruction, and community involvement. The

questionnaire will consist of items created by the researchers.

Format of items. The questionnaire will consist of eight Likert-format items and 10

open-ended questions about school culture. The Likert-format items will ask participants to

indicate each response from 1 to 4, with 1 representing never occurs, 2 representing

sometimes occurs, 3 representing frequently occurs, and 4 representing always occurs. The

goal will be to determine the frequency of certain occurrences at the school. The 10

open-ended questions will involve questions concerning leadership, relationships, parent

involvement, curriculum, instruction, and community involvement at the school. The eight

Likert-format items are:

1. Teachers and staff tell stories of celebrations that support the school’s values.

2. The administration treats faculty and staff as equals.


ISSN 1948-5476

2012, Vol. 4, No. 2


3. The school supports and appreciates new ideas by members of the school.

4. The school staff is encouraged to make instructional decisions rather than waiting for

administrators to tell them what to do.

5. Teachers and staff discuss instructional strategies.

6. Teacher planning time is used to plan as teams rather than individually.

7. Parents are involved in their child’s school experience.

8. The community plays an active role in the school.

Examples of qualitative questions include:

1. Describe the interaction among faculty and staff outside of school

2. Describe any unwritten traditions present at your school.

3. What are some ways the community is involved at your school?

4. What are some ways parents are involved at your school?

Administration, scoring/tabulating, and interpretation. The questionnaire will be

administered through the use of Survey Monkey, a data collection tool. The link to the

questionnaire will be sent out to all faculty and staff at the school by email. The email will

explain the purpose of the study; and the faculty and staff will be encouraged to participate in

filling out the questionnaire. The responses for each of the Likert-format items will be

tabulated. Each response will be assigned a point value. A ranking of never occurs will be

assigned 1 point, sometimes occurs will be assigned 2 points, frequently occurs will be

assigned 3 points, and always occurs will be assigned 4 points. For an eight Likert-item scale,

scores will range from a low of 8 (i.e., 8 x 1) to a high of 32 (i.e., 8 x 4), with high scores

representing positive attitudes and interactions at the school. These scores then can be used to

compare frequencies of occurrences at the campus. The open-ended responses will be

tabulated as well.

Score reliability reported by instrument developers. Test-retest reliability will be obtained

by administering the questionnaire to a test group of participants. One month later, the same

questionnaire will be administered to the same group of participants; and the responses will

be compared (Trochim, 2006).

Score reliability that will be computed in present study. The research team will conduct a

follow-up study among the school, administering the same questionnaire several months later,

to view the results obtained. The results of the questionnaire will be compared to those of the

original study. Internal consistency score reliability will be calculated by using Cronbach’s

alpha. Scores for Cronbach’s alpha usually range between 0 and 1. The closer Cronbach’s

alpha is to 1, the greater the internal consistency of the items (Gilem & Gilem, 2003).

Content- and criterion-related validity reported by instrument developers. A panel of

experts on school culture will review the questionnaire and suggest items that can be


ISSN 1948-5476

2012, Vol. 4, No. 2


improved. The panel will let the research team know if the items will provide a good way of

discovering information about the culture of the schools. Results of the questionnaire will be

compared with characteristics from the Correlates of Effective Schools research by Lawrence

Lezotte (Lezotte, 1991). These correlates consist of a safe and orderly environment, a climate

of high expectations, instructional leadership, a clear and focused mission, the opportunity to

learn and student time on task, frequent monitoring of student progress, and home-school

relations (Lezotte, 1991). This comparison will help assure criterion-related validity by

relating the data gathered in this study to another school effectiveness measure that has

already been demonstrated to be valid.

Construct-related validity reported by instrument developers. Multiple operationalism,

which is the use of different measures of a construct, will be employed (Johnson &

Christensen, 2010). The construct will be measured using a variety of ways, such as with

interviews, quantitative questions, and qualitative questions. According to Johnson and

Christensen (2010), the probability of acquiring a more complete as well as accurate

representation of the construct increases as more measures of the same construct are

implemented.

Development and validation procedures for any new instrument. After discussing the

data that we plan to collect, consulting previously conducted studies, and researching

components of a school culture, the research team will develop the items on the questionnaire.

Peer review, the sharing of thoughts and interpretations with one’s peers, will be used.

Participant feedback, whereby responses received are actually verified with the participants

themselves, will be utilized as well (Johnson & Christensen, 2010). The survey also will be

given to a test group of participants beforehand, to check for the understanding of the items

included and the layout of the questionnaire.

Procedures

Ethical nature of data collection. All participants will be informed of the purpose, nature,

and instruments used in the study through an informed consent form. This form will allow the

participant to agree to be a part of a study after being informed of its purpose, procedures,

risks, benefits, and confidentiality (Johnson & Christensen, 2010). Additionally, the

participants will be assured that all information shared or observed through the research will

remain anonymous and confidential. This proposal will be submitted to the Institutional

Review Board (IRB) for approval of this study as well. Data collectors will be trained

concerning what specific characteristics and qualities to look for while interviewing.

Furthermore, they will be trained concerning probing question techniques and how to watch

for non-verbal responses while interviewing participants.

Timeline for data collection. The intramethod questionnaire will be administered at the

beginning of the study. The faculty will be given 1 week to respond to the survey at each

school. After this, a reminder email will be sent out, encouraging those who have not

completed the survey to do so. After 1 additional week, the survey will be closed. Concerning

interviews, the research team will plan to conduct approximately five interviews a day at each

school, over a time period of approximately 1 week.


ISSN 1948-5476

2012, Vol. 4, No. 2


Research paradigm. Our research paradigm will include pragmatism, because a mixed

research approach will be carried out (Johnson & Onwuegbuzie, 2004). Johnson and

Christensen (2010) mentioned that a mixed research approach consists of both quantitative

and qualitative components. Pragmatism involves the view of what works in practice and

encourages social justice. Pragmatists believe in implementing the research design that will

best allow the researchers to answer their research questions (Johnson & Christensen, 2010).

Our qualitative-dominant research design will allow us better to gather the thoughts and

happenings at 10 Title I exemplary schools.

Research design. The research design will consist of a qualitative-dominant concurrent

method. This method includes the qualitative approach being given more weight than the

quantitative, and each approach being conducted concurrently (Johnson & Christensen, 2010).

This mixed research design will help discover more about the school culture at the 10 Title I

campuses, through the viewpoint of the faculty and staff who work there. The researchers

will be certain to remain neutral while interviewing participants. Furthermore, results of

similar studies will be researched and compared to the discoveries of this study.

Limitations

Threats to legitimation. First, according to Onwuegbuzie and Leech (2007), internal

credibility includes conclusions made within a setting or group, such as the truth value,

applicability, consistency, neutrality, dependability, and/or credibility of explanations.

Threats to internal credibility for our study include descriptive validity, reactivity, and

researcher bias. Descriptive validity includes the researchers accurately documenting each

participant’s words or views. This will be minimized by member checking, the process of

systematically receiving feedback about conclusions made from the study group

(Onwuegbuzie & Leech, 2007). Furthermore, participants might be uncomfortable sharing

truthful information with a researcher they do not know during the interview. This might be

due to reactivity, changes that take place in participants when they know they are being

observed (Johnson & Christensen, 2010). During interviews, participants also might exhibit

frontstage behavior. This consists of participants acting differently than normal, displaying

behaviors they want the researcher to see when they know they are being observed by the

interviewer (Johnson & Christensen, 2010). This threat will be minimized by the researchers

clearly explaining the purpose of the study and reminding the participants that their responses

will be completely confidential and anonymous. To continue, researcher bias, involving

personal assumptions of the researchers, might be evident to the participants and ultimately

influence their attitudes, behaviors, or experiences (Onwuegbuzie & Leech, 2007).

Onwuegbuzie and Leech (2007) mention that research bias can be minimized by using

unobtrusive measures, explaining the intentions of the researcher, carrying out interviews in a

neutral place, and keeping a clear focus on the research questions.

On a different note, external credibility involves the ability to generalize findings of a study

to various populations including different people, settings, and times (Onwuegbuzie & Leech,

2007). Threats to external credibility include interpretative validity, researcher bias, reactivity,

and order bias. Onwuegbuzie and Leech (2007) explained that interpretative validity is the


ISSN 1948-5476

2012, Vol. 4, No. 2


accuracy that researchers have in correctly displaying the perspective of the participants and

the meanings of their words and actions. This should be reduced by member checking, as

explained before. Researcher bias, mentioned previously, also is an external threat, because

the researchers’ bias might hinder the data interpretation, making it ungeneralizable. Peer

debriefing, the process of obtaining an external view of the research process, will be carried

out to decrease bias (Onwuegbuzie & Leech, 2007). Reactivity, also mentioned previously,

threatens the generalizability of the findings, because it will not be known if the same results

will be obtained if this threat did not take place. Finally, order bias, the order in which

components are carried out, might make a difference to the findings. Also, more variables

might be related to school culture than what the questions asked cover. This threat will be

minimized by triangulation, the process of implementing various methods and sources to

obtain data (Onwuegbuzie & Leech, 2007).

Concerning mixed methods legitimation, inside-outside legitimation might cause differences

in the research. According to Onwuegbuzie and Johnson (2006), this type of legitimation

involves the researcher accurately portraying inside and outside views. The correct

representation of the inside view will be checked by implementing participant review. This

involves participants assessing the researcher’s interpretations (Onwuegbuzie & Johnson,

2006). The outside view will be checked by using peer review, the process of having an

outside researcher not involved in the actual research, examine interpretations and

conclusions (Onwuegbuzie & Johnson, 2006). Furthermore, conversion legitimation might

play a role in research results. Onwuegbuzie and Johnson (2006) state that this type of

legitimation involves the degree to which quantizing (i.e., converting quantitative data into

qualitative data; Johnson & Christensen, 2010) and/or qualitizing (i.e., converting qualitative

data into quantitative data; Johnson & Christensen, 2010) data provides interpretable data and

large inference quality. Lastly, multiple validities legitimation involves the degree all

research studies are employed and the research can be viewed high on the multiple validities

(Onwuegbuzie & Johnson, 2006). In other words, all of the validities in the quantitative and

qualitative components of the researched are addressed and achieved. This will be minimized

by the researchers previously identifying the validities in each of the quantitative and

qualitative components as well as methods to decrease them.

Analysis

Method of analysis. Concerning qualitative data collected, which will include interview

responses and the open-ended responses gathered through the intramethod questionnaire, a

constant comparison analysis will be implemented. Leech and Onwuegbuzie (2008) stated

that a constant comparison analysis will allow the researchers to create theories or a set of

themes from data gathered. This approach will involve coding each qualitative response. The

data will be broken down into segments and given a code that describes each segment. Next,

the codes will be grouped into similar categories and themes will be formed (Leech &

Onwuegbuzie, 2008).

Regarding the quantitative data obtained through the eight Likert-format items on the

questionnaire, the mean, median, mode, and range will be calculated on responses for each


ISSN 1948-5476

2012, Vol. 4, No. 2


item. These figures will be used to support the findings of the qualitative portion of the

research. Furthermore, the numerical data collected also will be qualitized. This involves

converting quantitative data into qualitative data (Johnson & Christensen, 2010). The

researchers will describe the findings of each question in words.

Each of the six researchers will be trained the exact same way in coding, and will participate

in coding each of the interviews. Findings will be compared to assure intercoder reliability.

This reliability involves consistency among various coders (Johnson & Christensen, 2010). A

posteriori codes (i.e., codes based on the data) will be implemented, and the data analysis

will be exploratory in nature. This bottom-up or theory-generation approach will allow the

researchers fully to compare their findings.

References

Ayers, W. (Ed.). (1995). To become a teacher: Making a difference in children’s lives. New

York, NY: Teacher College Press.

Bryan, J., & Henry, L. (2008). Strengths-based partnerships: A school-family-community

partnership approach to empowering students. Professional School Counseling, 12,

149-155. http://dx.doi.org/10.5330/PSC.n.2010-12.149

Daresh, J. C. (2007). Supervision as proactive leadership. (4th ed.) Prospect Heights, IL:

Waveland Press.

Deal, T. E., & Peterson, K. D. (1990). The principal’s role in shaping school culture.

Washington DC: U.S. Department of Education.

Engels, N., Hotton, G., Devos, G., Bouckenooghe, D., & Aelterman, A. (2008). Principals in

schools with a positive school culture. Educational Studies, 34, 159-174.

http://dx.doi.org/10.1080/03055690701811263

Gilem, J. A. & Gilem, R. R. (2003, October). Calculating, interpreting, and reporting

Cronbach’s alpha reliability coefficient for Likert-type scales. Paper presented at the

Midwest Research to Practice Conference in Adult, Continuing, and Community

Education. Retrieved from

https://scholarworks.iupui.edu/bitstream/handle/1805/344/Gliem+&+Gliem.pdf?sequenc

e=1

Henderson, C. L., Buehler, A. E., Stein, W. L., Dalton, J. E., Robinson, T. R., & Anfara, V. A.

(2005). Organizational health and student achievement in Tennessee middle level

schools. NASSP Bulletin, 89(644), 54-73.

http://dx.doi.org/10.1177/019263650508964404

Howley, A., Woodrum, A., Burgess, L., & Rhodes, M. (2009). Planning for culturally

responsive leadership: Insights from a study of principals of exemplary schools.

Educational Planning, 18(2), 12-24.


ISSN 1948-5476

2012, Vol. 4, No. 2


Johnson, R. B., & Christensen, L. B. (2010). Educational research: Quantitative, qualitative,

and mixed approaches (4th ed.). Thousand Oaks, CA: Sage.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm

whose time has come. Educational Researcher, 33(7), 14-26.

http://dx.doi.org/10.3102/0013189X033007014

Johnson, W. L., Snyder, K. J., Anderson, R. H., & Johnson, A. M. (1996). School work

culture and productivity. The Journal of Experimental Education, 64, 139-156.

http://dx.doi.org/10.1080/00220973.1996.9943800

Keiser, K. A., & Schulte, L. E. (2009). Seeking the sense of community: A comparison of

two elementary schools’ ethical climates. The School Community Journal, 19(2), 45-56.

Kelley, R.C., Thornton, B., & Daugherty, R. (2005). Relationships between measures of

leadership and school climate. Education, 126(1), 17-23.

Leech, N. L., & Onwuegbuzie, A. J. (2008). Qualitative data analysis: A compendium of

techniques and a framework for selection for school psychology research and beyond.

School Psychology Quarterly, 23, 587-604.

http://dx.doi.org/10.1037/1045-3830.23.4.587

Lester, F. (2005). On the theoretical, conceptual, and philosophical foundations for research

in mathematics education. ZDM, 37, 457-467. http://dx.doi.org/10.1007/BF02655854

Lezotte, L. W. (1991). Correlates of effective schools: The first and second generation.

Okemos, MI: Effective Schools Products, Ltd.

Lunenburg, F. C., & Ornstein, A. C. (2008). Educational administration: Concepts and

practices. Belmont, CA: Wadsworth Cengage Learning.

Macneil, A. J., Prater, D. L., & Busch, S. (2009). The effects of school culture and climate on

student achievement. International Journal of Leadership in Education, 12(1), 73-84.

http://dx.doi.org/10.1080/13603120701576241

Onwuegbuzie, A. J., & Collins, K. M. T. (2007). A typology of mixed methods sampling

designs in social science research. The Qualitative Report, 12, 281-316. Retrieved from

http://www.nova.edu/ssss/QR/QR12-2/onwuegbuzie2.pdf

Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research

in the Schools, 13(1), 48-63.

Onwuegbuzie, A. J., & Leech, N. L. (2007). Validity and qualitative research: An oxymoron?

Quality & Quantity: International Journal of Methodology, 41, 233-249.

http://dx.doi.org/10.1007/s11135-006-9000-3

Sergiovanni, T. J. (1984). Leadership and excellence in schooling. Educational Leadership,

41(5), 4-13.

Southworth, G., & Du Quesnay, H. (2005). School leadership and system leadership. The



















ISSN 1948-5476

2012, Vol. 4, No. 2


Educational Forum, 69, 212-220. http://dx.doi.org/10.1080/00131720508984685

Texas Education Agency. (2010). Accountability manual. Retrieved from the Texas

Education Agency website:http://ritter.tea.state.tx.us/perfreport/account/2010/manual/

Trochim, W. M. (2006). Research methods knowledge base: Types of reliability. Retrieved

from http://www.socialresearchmethods.net/kb/reltypes.php

U.S. Department of Education. (n.d.a). Improving basic programs operated by local

educational agencies. Retrieved from the US Department of Education website:

http://www2.ed.gov/programs/titleiparta/index.html

U.S. Department of Education. (n.d.b). Title I- Improving the academic achievement of the

disadvantaged. Retrieved from the US Department of Education website:

http://www2.ed.gov/policy/elsec/leg/esea02/pg1.html#sec1001

http://ritter.tea.state.tx.us/perfreport/account/2010/manual/

















http://www.socialresearchmethods.net/kb/reltypes.php


































ISSN 1948-5476

2012, Vol. 4, No. 2


Appendix B

Part I and Part II Mixed Methods Research Proposal Scoring Rubric

PART 1: SCORING CHECKLIST FOR CONTENT IN MIXED RESEARCH

REPORTS

Name:

Semester: Date:

DIRECTIONS:

For each of the following statements, indicate the extent to which you agree or disagree to the

corresponding ratings of students, according to the scale below. (Note: Any statements

which are not applicable will automatically receive a Astrongly agree@ rating.)

1=strongly disagree 2=disagree 3=neutral 4=agree 5=strongly agree

CONTENT

Title:

1. The title makes clear the population of interest. 1 2 3 4 5

2. The title makes clear the primary independent variable(s). 1 2 3 4 5

3. The title makes clear the dependent variable(s). 1 2 3 4 5

4. The title indicates the specific relationship between the major variables. 1 2 3 4 5

5. Vague, ambiguous, and emotional-laden terms have been avoided. 1 2 3 4 5

Introduction/Literature Review:

6. An explicit statement of the problem is present. 1 2 3 4 5

7. The statement of the problem is stated in the opening paragraph. 1 2 3 4 5

8. Adequate background information is given on the problem presented. 1 2 3 4 5

9. All background information given on the problem is relevant. 1 2 3 4 5

10. The statement of the problem leads smoothly to the next paragraph. 1 2 3 4 5

11. Quotations are avoided, and are only used when paraphrasing would lead

to a loss of meaning or representation.

1 2 3 4 5

12. Every statement of fact is supported by one or more citations. 1 2 3 4 5

13. All findings from previous research are supported by one or more

citations.

1 2 3 4 5

14. The literature review is comprehensive (i.e., the review contains at least

20 citations that are complete and exactly consistent with the citations

presented in the reference list).

1 2 3 4 5

15. All references cited are relevant to the problem under investigation. 1 2 3 4 5

16. All or almost all of the sources are primary. 1 2 3 4 5

17. Most of the references are current. 1 2 3 4 5

18. The review explicitly relates previous studies to the problem. 1 2 3 4 5

19. The literature review section contains an appropriate number of 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


subheadings.

20. Each subheading is reflective of the text contained within it. 1 2 3 4 5

21. At least some of the references have been critically analyzed. 1 2 3 4 5

22. The references have been compared and contrasted adequately. 1 2 3 4 5

23. The review logically flows in such a way that the references least related

to the problem are discussed first and the most related references are

discussed last.

1 2 3 4 5

24. Clear connections are made between the present study and the previous

research.

1 2 3 4 5

25. All conceptual/theoretical terms are directly/operationally defined. 1 2 3 4 5

26. A clear conceptual/theoretical framework is presented. 1 2 3 4 5

27. The conceptual/theoretical framework is associated with one or more

appropriate citations.

1 2 3 4 5

28. The gap in the literature is clearly identified. 1 2 3 4 5

29. The review concludes with a brief summary. 1 2 3 4 5

30. The summary is an adequate representation of material that was

presented previously.

1 2 3 4 5

31. The summary does not contain new information that should have been

introduced earlier.

1 2 3 4 5

32. A clear rationale for the study is provided. 1 2 3 4 5

33. There is an explicit purpose statement. 1 2 3 4 5

34. The purpose statement flows logically from the rationale. 1 2 3 4 5

35. The purpose statement makes clear the population of interest. 1 2 3 4 5

36. The purpose statement makes clear the primary dependent variable. 1 2 3 4 5

37. The purpose statement indicates the specific relationship between the

major cases/variables.

1 2 3 4 5

38. The purpose statement is consistent with the title. 1 2 3 4 5

39. The research problem is researchable. 1 2 3 4 5

40. One or more explicit research questions are presented. 1 2 3 4 5

41. The research questions follow the purpose statement. 1 2 3 4 5

42. Each research question makes clear the population of interest. 1 2 3 4 5

43. Each research question makes clear the primary independent variable. 1 2 3 4 5

44. Each research question makes clear the primary dependent variable. 1 2 3 4 5

45. Each hypothesis is stated clearly and concisely. 1 2 3 4 5

46. In each research hypothesis, the expected relationship or difference is

clear.

1 2 3 4 5

47. Each hypothesis makes clear the population of interest. 1 2 3 4 5

48. Each hypothesis makes clear the primary independent variable. 1 2 3 4 5

49. Each hypothesis makes clear the primary dependent variable. 1 2 3 4 5

50. Each hypothesis logically flows from the theoretical framework. 1 2 3 4 5

51. Each hypothesis is testable. 1 2 3 4 5

52. The educational significance is stated explicitly. 1 2 3 4 5

53. This section is well-integrated (i.e., flows well). 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2



Method

Participants for Quantitative Sample/Phase

55. The (approximate) population size is provided. 1 2 3 4 5

56. The major characteristics of the population are described adequately. 1 2 3 4 5

57. The selection-eligibility criteria are described adequately. 1 2 3 4 5

58. If a sample was selected, at least one of the 24 sampling schemes

identified by Onwuegbuzie and Collins (2007) is specified accurately.

1 2 3 4 5

59. If a sample was selected, the sampling scheme is described

clearly and accurately.

1 2 3 4 5

60. An adequate rationale is provided for choice of sampling scheme. 1 2 3 4 5

61. The (approximate) sample size and group sizes are provided. 1 2 3 4 5

62. The sample size and group sizes are adequate for the research design. 1 2 3 4 5

63. The sample size is consistent with the type of generalization suggested

by the title, purpose statement, and research question.

64. The major characteristics of the sample are described adequately. 1 2 3 4 5

65. Evidence of ethical considerations is provided adequately. 1 2 3 4 5



Participants for Qualitative Sample/Phase

68. The setting is described adequately. 1 2 3 4 5

69. The (approximate) population size is provided. 1 2 3 4 5

70. The major characteristics of the population are described adequately. 1 2 3 4 5

71. The selection-eligibility criteria are described adequately. 1 2 3 4 5

72. If a sample will be selected, at least of the 24 sampling schemes

identified by Onwuegbuzie and Collins (2007) is specified accurately.

1 2 3 4 5

73. If a sample will be selected, the sampling scheme is described clearly

and accurately.

1 2 3 4 5

74. An adequate rationale is provided for choice of sampling scheme. 1 2 3 4 5

75. Evidence is presented that the sample of words/observations (i.e., sample

space) will represent the population of words/observations (i.e., truth space)

for the underlying context.

1 2 3 4 5

76. The (approximate) sample size and group sizes are provided. 1 2 3 4 5

77. An adequate rationale is provided for the sample size/group sizes. 1 2 3 4 5

78. The sample size and group sizes are adequate for the research design. 1 2 3 4 5

79. The sample size is consistent with the type of generalization suggested

by the title, purpose statement, and research question.

1 2 3 4 5

80. The major characteristics of the sample are described adequately. 1 2 3 4 5

81. The relationship of the researcher to the participants are fully described

(e.g., participant observer, non-participant observer, collaborator).

1 2 3 4 5

82. The role of the researcher is described clearly (e.g., neutral, 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


collaborative, objective).

83. Evidence of ethical considerations is provided adequately. 1 2 3 4 5

84. This section of the proposal is well-integrated (i.e., flows well). 1 2 3 4 5


Mixed Sampling Design

86. The mixed sampling design is provided using Onwuegbuzie and

Collins’s (2007) framework.

1 2 3 4 5

87. The time orientation (i.e., sequential vs. concurrent) for selecting the

quantitative and qualitative samples is specified clearly.

1 2 3 4 5

88. The time orientation (i.e., sequential vs. concurrent) for selecting the

quantitative and qualitative samples is described clearly.

1 2 3 4 5

89. The relationship of the quantitative and qualitative samples (i.e.,

identical, parallel, nested, multilevel) is specified clearly.

1 2 3 4 5

90. The relationship of the quantitative and qualitative samples (i.e.,

identical, parallel, nested, multilevel) is described clearly

1 2 3 4 5

91. This section of the report is well-integrated (i.e., flows well). 1 2 3 4 5


Instruments for Quantitative Phase

93. An adequate rationale is given for the selection of each instrument. 1 2 3 4 5

94. Each instrument is described adequately in terms of purpose and content. 1 2 3 4 5

95. The developers of all instruments are specified clearly. 1 2 3 4 5

96. The format of the items is specified clearly and accurately. 1 2 3 4 5

97. The administration, scoring or tabulating, and interpretation. procedures

are fully described.

1 2 3 4 5

98. Citations are presented for any information provided pertaining to the

development of all instruments (e.g., standardization/norming techniques).

1 2 3 4 5

99. Citations are provided for all statements of facts and research findings

pertaining to the characteristics of the instruments.

1 2 3 4 5

100. Each instrument appears to be appropriate for measuring the intended

variables.

1 2 3 4 5

101. Evidence is given that each instrument was appropriate for the sample. 1 2 3 4 5

102. Each instrument appears to be appropriate for the sample under study. 1 2 3 4 5

103. Adequate information is provided which indicates that administrators,

observers, or interviewers were well trained.

1 2 3 4 5

104. Adequate information is provided which indicates that there was no

administrator, observer, or interviewer effect.

1 2 3 4 5

105. Instrument score reliability presented by the developer is described

adequately in terms of type of coefficients.

1 2 3 4 5

106. Instrument score reliability presented by the developer is described

adequately in terms of size of coefficients.

1 2 3 4 5

107. If appropriate, subtest score reliabilities presented by the developer are 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


provided adequately.

108. If no information on score reliability was available in the literature, this

is specified clearly.

1 2 3 4 5

109. Instrument score reliability that was computed by the researcher is

described adequately in terms of type of coefficients.

1 2 3 4 5


described adequately in terms of size of coefficients.

1 2 3 4 5


described adequately in terms of the purpose of coefficients.

1 2 3 4 5

112. All instruments used appear that they yielded scores that were

sufficiently reliable.

1 2 3 4 5

113. Citations are provided for all reliability coefficients presented. 1 2 3 4 5

114. Instrument score validity is discussed and coefficients given if

appropriate.

1 2 3 4 5

115. Each instrument is described in terms of content-related validity. 1 2 3 4 5

116. Each instrument is described in terms of criterion-related validity. 1 2 3 4 5

117. Each instrument is described in terms of construct-related validity. 1 2 3 4 5

118. If no information on content-, criterion-, and/or construct-related

validity was available in the literature, this is specified clearly.

1 2 3 4 5

119. All instruments used appear that they yielded scores that were

sufficiently valid for the study.

1 2 3 4 5

120. Citations are provided for all validity coefficients presented. 1 2 3 4 5

121. If an instrument was designed specifically for the study, the procedures

involved in its development are described adequately.

1 2 3 4 5

122. If an instrument was designed specifically for the study, the procedures

involved in its score validation are described adequately.

1 2 3 4 5

123. If an instrument was designed specifically for the study, the

administration, scoring or tabulating, and interpretation procedures are fully

described.

1 2 3 4 5



Instruments for Qualitative Phase

126. An adequate rationale is given for the selection of each instrument. 1 2 3 4 5

127. Each instrument is described adequately in terms of purpose and

content.

1 2 3 4 5

128. The developers of all instruments are specified clearly. 1 2 3 4 5

129. The format of the items (e.g., open-ended) is specified clearly and

accurately.

1 2 3 4 5

130. Citations are provided for all statements of facts and research findings

pertaining to the characteristics of the instruments.

1 2 3 4 5

131. Each instrument appears to be appropriate for measuring the intended

phenomenon/variables.

1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


132. Evidence is given that each instrument is appropriate for the sample. 1 2 3 4 5

133. Each instrument appears to be appropriate for the sample under study. 1 2 3 4 5

134. Adequate information is provided which indicates that administrators,

observers, or interviewers are/will be well trained.

1 2 3 4 5

135. If an instrument was/will be designed specifically for the study, the

procedures involved in its development are described adequately.

1 2 3 4 5

136. Samples of interview central questions, issue subquestions, and topical

subquestions are provided.

1 2 3 4 5

137. Interview questions/subquestions are described as basic descriptive,

follow-up, experience/example, simple clarification, structural/paradigmatic,

and comparison/contrast (Janesick, 2004).

1 2 3 4 5

138. Type of interview is specified (e.g., unstructured, partially structured,

semi-structured, structured, totally structured).

1 2 3 4 5

139. All interviewers are identified clearly. 1 2 3 4 5

140. Location of interview is specified (i.e., interviewee’s workplace). 1 2 3 4 5

141. Length of interview is specified (i.e., short vs. long). 1 2 3 4 5

142. Estimated number of interviews is specified (e.g., single vs. multiple). 1 2 3 4 5

143. Format of interviews is specified (e.g., formal vs. informal). 1 2 3 4 5

144. Samples of focus group questions are provided. 1 2 3 4 5

145. Samples of probes to questions are provided. 1 2 3 4 5

146. All observational data collection techniques are described adequately. 1 2 3 4 5

147. All observation protocols used (e.g., Flanders, 1965) are described

adequately.

1 2 3 4 5

148. The type of observations is specified using Fontana and Frey’s (2005)

categorization (i.e., kinesic, proxemic, chronemic, paralinguistic) .

1 2 3 4 5

149. Observer is classified as peripheral member, active member, or

complete member (Adler & Adler, 1987).

1 2 3 4 5

150. Observation is classified as descriptive observation, focus observation,

or selective observation (Werner & Schoepfle, 1987).

1 2 3 4 5

151. All documents that will be analyzed are identified clearly. 1 2 3 4 5



Procedure for the Quantitative Phase

154. All data collecting procedures are clearly described. 1 2 3 4 5

155. The study was conducted for an appropriate length of time for the

predicted outcomes to be observed.

1 2 3 4 5

156. The training of data collectors is clearly described and adequate. 1 2 3 4 5

157. It appears that the data collection procedure was conducted in a

consistent manner.

1 2 3 4 5

158. The ethical nature of data collection method is discussed adequately. 1 2 3 4 5

159. Procedures are described in sufficient detail to permit replication. 1 2 3 4 5

160. Citations are provided for any procedural information delineated which 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


are directly or indirectly based on previous research.

161. If a pilot study was conducted, its purpose is described adequately, as

well as its impact on the subsequent study.

1 2 3 4 5

162. The procedures provide sufficient control for internal validity. 1 2 3 4 5

163. The procedures provide sufficient control for external validity. 1 2 3 4 5

164. The research paradigm used is clear. 1 2 3 4 5

165. The research design is stated clearly and accurately. 1 2 3 4 5

166. Adequate justification is provided for the research design used. 1 2 3 4 5

167. Citations are provided when describing the research design used. 1 2 3 4 5

168. The design appears to be appropriate for answering the research

question and/or testing the hypothesis.

1 2 3 4 5

169. If treatment groups were compared, it is clear whether participants were

randomly assigned to groups.

1 2 3 4 5

170. If treatment groups were compared, the number of participants in each

group was stated.

1 2 3 4 5

171. If treatment groups were compared, the number of participants per

group used appears to be adequate, or else an appropriate rationale is

provided as to why the group sizes were smaller than recommended.

1 2 3 4 5

172. If groups were compared, it is clear whether participants were blinded

as to what treatment group they were assigned.

1 2 3 4 5

173. If groups were compared, it is clear whether the individual measuring

the outcome variable(s) was blinded to the treatment group to which the

participants were assigned.

1 2 3 4 5

174. If groups were compared and participants were aware of their group

assignment, it is clear whether this knowledge affected their responses.

1 2 3 4 5

175. If groups were compared and the individual measuring the outcome

variable was not blinded, it is clear that the measurements were not biased by

this.

1 2 3 4 5

176. If groups were compared, the conditions of all experimental groups are

described clearly and completely.

1 2 3 4 5

177. If groups were compared, the conditions of all control groups are

described clearly and completely.

1 2 3 4 5

178. If groups were compared, participants in all groups received the exact

same experimental procedures and measurements, except for the treatment

intervention.

1 2 3 4 5

179. If groups were compared, it is clear that there was strict adherence to

the protocol in all groups.

1 2 3 4 5

180. If groups were compared, any participant attrition is described clearly. 1 2 3 4 5

181. If groups were compared, it is clear whether any unexpected outcomes

ensued.




ISSN 1948-5476

2012, Vol. 4, No. 2


Procedure for the Qualitative Phase

184. All data collecting procedures are clearly described. 1 2 3 4 5

185. The study will be conducted for an appropriate length of time for the

predicted outcomes to be observed.

1 2 3 4 5

187. The ethical nature of data collection method is discussed adequately. 1 2 3 4 5

186. The training of data collectors is clearly described and adequate. 1 2 3 4 5

188. The research paradigm used is clear. 1 2 3 4 5

189. The philosophical correlates of the research paradigm are specified

clearly (e.g., hermeneutics, post-positivist, post-structuralist, post-modernist,

constructivist, feminist, idealist).

1 2 3 4 5

190. Citations are provided for selected correlates of research paradigm. 1 2 3 4 5

191. The assumption(s) of research paradigm are provided. 1 2 3 4 5

192. The research design is stated clearly and accurately. 1 2 3 4 5

193. If a case study design is used, the type of case study (i.e., instrumental,

intrinsic, collective/multiple; Stake, 2005; Yin, 2004) is identified and

described clearly.

1 2 3 4 5

194. If a phenomenological design is used, the type of phenomenology (i.e.,

reflective/transcendental, dialogical, empirical, existential, hermeneutic,

social, psychological; Creswell, 2007) is identified and described clearly.

1 2 3 4 5

195. Adequate justification is provided for the research design used. 1 2 3 4 5

196. Information is provided about how the research design may evolve as

the process unfolds.

1 2 3 4 5

197. Citations are provided when describing the research design used. 1 2 3 4 5

198. The design appears to be appropriate for answering the research

question and/or testing the hypothesis.

1 2 3 4 5

199. Evidence is provided that the researcher will not be the sole voice of the

participant.

1 2 3 4 5

200. Evidence is provided that any interviewers will not influence the

content of the interviewee’s/focus group’s description in such a way that

these descriptions do not truly affect the actual experience.

1 2 3 4 5

201. Evidence is provided that all transcripts will be accurate. 1 2 3 4 5

202. The form of all collected data is adequately described (e.g., field notes,

audio tapes, videotapes, referential material).

1 2 3 4 5

203. Evidence is provided that rich data will be collected. 1 2 3 4 5

204. Evidence is provided that the data sources are well chosen. 1 2 3 4 5

205. Evidence is provided that the number of data sources is adequate. 1 2 3 4 5

206. Evidence is provided that data will be collected in a systematic manner. 1 2 3 4 5

207. It is described adequately how entry into the research context will be

obtained.

1 2 3 4 5

208. No important verification procedures are omitted. 1 2 3 4 5




ISSN 1948-5476

2012, Vol. 4, No. 2


Procedure for the Mixed Research Phase

211. The rationale for conducting a mixed research study is stated clearly

(i.e., triangulation vs. complementarity vs. development vs. initiation vs.

expansion; cf. Greene, Caracelli, & Graham, 1989, which is discussed in the

Johnson & Christensen, 2010 text).

1 2 3 4 5

212. The rationale for conducting a mixed research study is described clearly

(i.e., triangulation vs. complementarity vs. development vs. initiation vs.

expansion; cf. Greene, Caracelli, & Graham, 1989, which is discussed in the

Johnson & Christensen, 2010 text).

1 2 3 4 5

213. The type of mixed research design is identified and labeled (cf. Johnson

& Christensen, 2010).

1 2 3 4 5

214. The type of mixed research design is described adequately (cf. Johnson

& Christensen, 2010).

1 2 3 4 5

215. It is specified clearly whether a partially mixed or fully mixed research

design is used (cf. Johnson & Christensen, 2010).

1 2 3 4 5

216. It is specified clearly whether the mixed research design is sequential or

concurrent (cf. Johnson & Christensen, 2010).

1 2 3 4 5

217. It is specified whether the quantitative or qualitative phase is dominant

or whether there is approximately equal emphasis (cf. Johnson &

Christensen, 2010).

1 2 3 4 5



Limitations

220. At least two possible threats to internal validity are discussed

adequately.

1 2 3 4 5

221. At least two possible threats to external validity are discussed

adequately.

1 2 3 4 5

222. Each threat to internal validity discussed is labeled appropriately. 1 2 3 4 5

223. Each threat to external validity discussed is labeled appropriately. 1 2 3 4 5

224. Citations are provided when referring to threats to validity. 1 2 3 4 5

225. Discussion as to how to minimize each threat to internal validity is

adequate.

1 2 3 4 5

226. Discussion as to how to minimize each threat to external validity is

adequate.

1 2 3 4 5

227. All important threats to internal validity are discussed. 1 2 3 4 5

228. All important threats to external validity are discussed. 1 2 3 4 5

229. The discussion of threats to Verification/Trustworthiness/legitimation/

Authenticity/Credibility/Transferability/Dependability/Confirmability of data

is adequately undertaken using a framework (e.g., Creswell, 2007; Creswell

& Miller, 1987; Guba & Lincoln, 1989; Lather, 1991, 1993; LeCompte &

Goetz, 1982; Lincoln & Guba, 1985; Miles & Huberman, 1994).

1 2 3 4 5

230. Each legitimation threat discussed is labeled appropriately. 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


231. Citations are provided when referring to threats to legitimation. 1 2 3 4 5

232. All important threats to legitimation are discussed. 1 2 3 4 5

233. At least one verification procedure is described in detail e.g.,

prolonged engagement, persistent observation, triangulation,

contextualization of observations, method of constant comparison, checking

for representativeness of sources of data, checking for researcher effects,

weighing the evidence, examining extreme cases, checking for spurious

relations, examining rival explanations, looking for negative evidence,

obtaining feedback from informants, leaving an audit trail, thick description,

assessing structural relationships, use of referential material, theoretical

sampling; Onwuegbuzie & Leech, 2007).

1 2 3 4 5

234. Citations are provided when referring to each verification procedure. 1 2 3 4 5



Analysis of Quantitative Data

237. An adequate description is provided as to the analysis intended to

address the research question(s) and/or test the hypotheses.

1 2 3 4 5

238. All dependent variables are specified clearly. 1 2 3 4 5

239. The scale of measurement pertaining to each dependent variable is

presented clearly and accurately.

1 2 3 4 5

240. The dependent variables presented are consistent with the research

questions/hypotheses.

1 2 3 4 5

241. All independent variables are specified clearly. 1 2 3 4 5

242. The scale of measurement pertaining to each independent variable is

presented clearly and accurately.

1 2 3 4 5

243. The independent variables presented are consistent with the research

questions/hypotheses.

1 2 3 4 5

244. The method of analysis is appropriate for testing the research

hypothesis.

1 2 3 4 5

245. The method of analysis is for testing each research hypothesis is

presented.

1 2 3 4 5

246. The method of analysis is for testing each research hypothesis is

consistent with the scales of measurements presented.

1 2 3 4 5

247. The significance level of the statistical tests is delineated, and

appropriate control is made for Type 1 and Type II error.

1 2 3 4 5

248. All assumptions (e.g., linearity, independence, normality, homogeneity

of variance) associated with the inferential test are described adequately and

accurately.

1 2 3 4 5

249. Statistical power is discussed adequately. 1 2 3 4 5

250. Effect size(s) are discussed. 1 2 3 4 5

251. The type of effect size(s) (e.g., Cohen’s (1988) d; R2, ω2) is delineated. 1 2 3 4 5

252. All statistical software are specified (e.g., SPSS, SAS, SYSTAT). 1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2


253. Citations are provided for all statistical software identified. 1 2 3 4 5



Analysis of Qualitative Data

256. It is specified adequately how the emergent categories will be analyzed

(e.g., method of constant comparison).

1 2 3 4 5

257. It is clear whether exploratory or confirmatory techniques (or both) will

be used.

1 2 3 4 5

258. All qualitative software are specified (e.g., QDA Miner, NVIVO,

Atlas-ti, Ethnograph).

1 2 3 4 5

259. Citations are provided for all statistical software identified. 1 2 3 4 5



Mixed Analysis

262. The number of data types that will be analyzed is described clearly 1 2 3 4 5

263. The number of data analysis types that will be used is described clearly 1 2 3 4 5

264. The time sequence of the mixed analysis is described clearly (i.e.,

sequential vs. concurrent).

1 2 3 4 5

265. It is specified whether data were qualitized and/or quantitized. 1 2 3 4 5

266. If a sequential mixed analysis was used, the order of the mixed analysis

(e.g., quantitative analysis followed by qualitative analysis or qualitative

analysis followed by quantitative analysis)

1 2 3 4 5

267. This section of the report is well-integrated (i.e., flows well). 1 2 3 4 5


Reference List

269. All citations provided in the text are contained in the reference list. 1 2 3 4 5

270. All citations provided in the reference list are contained in the text. 1 2 3 4 5

271. The names of all authors provided in the text are consistent with the

names presented in the reference list.

1 2 3 4 5

272. All authors are presented in strict adherence to APA guidelines. 1 2 3 4 5

273. All titles are written accurately and in strict adherence to APA

guidelines.

1 2 3 4 5

274. All publication dates in the reference list are consistent with those in the

text and are written in strict adherence to APA guidelines.

1 2 3 4 5

275. All sources are written accurately and in strict adherence to APA

guidelines.

1 2 3 4 5

276. Every aspect of the reference list strictly adheres to APA guidelines. 1 2 3 4 5

Appendix


ISSN 1948-5476

2012, Vol. 4, No. 2


277. The appendix section contains samples of any researcher-made

instruments.

1 2 3 4 5

278. All researcher-made instruments appear to be appropriate for the study. 1 2 3 4 5

279. The appendix section contains an appropriate number of informed

consent forms.

1 2 3 4 5

280. Each informed consent form is written appropriately for the intended

reader.

1 2 3 4 5

281. Each informed consent form contains all important information. 1 2 3 4 5

282. The information provided in each informed consent form is consistent

with the information provided in the methods section.

1 2 3 4 5

___________________________________________________________________________

Number of Occurrences:

___________________________________________________________________________

Number of points assigned:

TOTAL SCORE FOR RESEARCH REPORT OUT OF 1410:

PERCENTAGE SCORE FOR RESEARCH REPORT:

RUBRIC EQUIVALENT SCORE OUT OF 60:


ISSN 1948-5476

2012, Vol. 4, No. 2


PART II: MIXED REPORTS SCORING CHECKLIST FOR QUALITY OF WRITING

AND ADHERENCE TO APA STYLE

MECHANICS

Name:

Semester: Date:

DIRECTIONS:

For each of the following statements, indicate the extent to which you agree or disagree,

according to the scale below. (Note: Any statements which are not applicable will

automatically receive a ―strongly agree‖ rating.)

1=strongly disagree 2=disagree 3=neutral 4=agree 5=strongly agree

Title Page

1. The title page contains all essential components. 1 2 3 4 5

2. The page header adheres strictly to APA guidelines. 1 2 3 4 5

3. The page header text is of high quality (e.g., grammar, punctuation). 1 2 3 4 5

4. The running head adheres strictly to APA guidelines. 1 2 3 4 5

5. The running head text is of high quality (e.g., grammar, punctuation). 1 2 3 4 5

6. The title adheres strictly to APA guidelines. 1 2 3 4 5

7. The title text is of high quality (e.g., grammar, punctuation). 1 2 3 4 5

Introduction/Literature Review

1. This section contains all the salient information. 1 2 3 4 5

2. No inappropriate information is presented in this section of the report

(including repetitive information).

1 2 3 4 5

3. This section is informative. 1 2 3 4 5

4. This section is entirely accurate. 1 2 3 4 5

5. This section does not contain any contradictions. 1 2 3 4 5

6. This section is comprehensive. 1 2 3 4 5

7. This section is written in strict adherence to APA guidelines (including

margins).

1 2 3 4 5

8. This section is clearly written throughout. 1 2 3 4 5

9. The writing in this section is of high quality (e.g., grammar, punctuation). 1 2 3 4 5

Method

Participants




1 2 3 4 5


ISSN 1948-5476

2012, Vol. 4, No. 2







margins).

1 2 3 4 5



Instruments




1 2 3 4 5






margins).

1 2 3 4 5



Procedure




1 2 3 4 5






margins).

1 2 3 4 5


9. The writing in this section is of high quality(e.g., grammar, punctuation). 1 2 3 4 5

Limitations




1 2 3 4 5






ISSN 1948-5476

2012, Vol. 4, No. 2



margins).

1 2 3 4 5



Analysis




1 2 3 4 5






margins).

1 2 3 4 5



___________________________________________________________________________

Number of Occurrences:

___________________________________________________________________________

Number of points assigned:

Total Score out of 305_________________

Percentage Score:

Rubric Equivalent Score out of 40: _________________


ISSN 1948-5476

2012, Vol. 4, No. 2


Score Summary Sheet

RUBRIC EQUIVALENT SCORE FOR CONTENT OUT OF 60:___________

RUBRIC EQUIVALENT SCORE FOR MECHANICS OUT OF 40: _________

BONUS POINTS FOR INTRODUCTION/LITERATURE REVIEW

(1 = Material presented in this section is slightly more than was required)

(2 = Material presented in this section is more than was required)

(3 = Material presented in this section is substantially more than was required): _________

BONUS POINTS FOR CONTENT IN PARTICIPANTS SECTION

(1= Material presented in this section is slightly more than was required)

(2= Material presented in this section is more than was required)


BONUS POINTS FOR CONTENT IN INSTRUMENTS SECTION




BONUS POINTS FOR CONTENT IN PROCEDURE SECTION




BONUS POINTS FOR CONTENT IN LIMITATIONS SECTION




BONUS POINTS FOR CONTENT IN ANALYSIS SECTION




PENALTY POINTS FOR MECHANICS IN RESEARCH REPORT


ISSN 1948-5476

2012, Vol. 4, No. 2


Number of missing, incomplete, or inconsistent references: ____________

TOTAL NUMBER OF POINTS:__

Comments:


ISSN 1948-5476

2012, Vol. 4, No. 2


Appendix C

Instructor Feedback


ISSN 1948-5476

2012, Vol. 4, No. 2



ISSN 1948-5476

2012, Vol. 4, No. 2


Copyright Disclaimer

Copyright reserved by the author(s).

This article is an open-access article distributed under the terms and conditions of the

Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

the use of a scoring rubric in an online research methodology course

Documents