3 Standards for Sound AppliedLinguistics Research

The qual–quant not only provides an interactive continuum useful for characterizing applied linguistics research but can also serve as a basis for understanding the standards that researchers use in judging the soundness of such research. This discussion will again focus on primary research, leaving secondary research out because it involves a separate set of issues. Because I consistently started with the qualitative-exploratory end of the continuum in the previous section, out of fairness, I will start with the quantitative- experimental end in this section.

3.1 The quantitative-experimental end ofthe continuum

Researchers at the quantitative-experimental end of the continuum value the concepts of reliability, replicability, validity, and generalizability.

Generally speaking, those are the standards quantitative researchers hold up in judging the soundness of their research.

3.1.1 Reliability The standard of reliability in quantitative research requires

researchers to demonstrate both (1) the reliability of the instruments used in their studies and (2) the reliability of the results of their studies.

Reliability of instruments is concerned with the degree to which the results of a questionnaire, test, or other measuring instrument are consistent. Addressing this issue typically means answering the question: To what degree would the results be the same if the instrument were administered repeatedly?

Similarly, the reliability of the results of a study is concerned with the degree to which the results would be likely to reappear if the study were replicated under the same conditions.

3.1.2 Replicability The standard of replicability in quantitative research requires

researchers to provide enough information about a study to allow other researchers to replicate or repeat the study exactly as it was originally conducted.

The replicability of a quantitative study can be improved by thorough and complete descriptions of: (1) the participants in the study and how they were selected,

(2) the instruments used in the study as well as arguments for their reliability and validity, and

(3) the procedures followed in collecting the data, scoring or coding the instruments, and analyzing the results (see Brown, 1988, ch. 5 for a more complete discussion of what should be included to make a study adequately replicable).

3.1.3 Validity The standard of validity in quantitative research requires

researchers to demonstrate both internal and external validity. Internal validity is the degree to which the results of a study can be

accurately interpreted as meaning what they appear to mean. External validity is the degree to which the results of a study are

contrived or artificial, or put another way, the degree to which the results apply to the outside world. Listed below are a number of issues identified by Campbell and Stanley (1963) as potential threats to the internal and external validity of a study:

The internal and external validity of a quantitative study can be improved by consciously guarding against all of these threats when planning and conducting research (for more on controlling these potential threats, see Campbell & Stanley, 1963; Brown, 1988, 1997; or Hatch & Lazaraton, 1991).

3.1.4 Generalizability As mentioned above, the standard of generalizability in

quantitative research requires researchers to show the degree to which the results of a study can justifiably be generalized, or applied, to a larger population or to other similar groups.

A study can be extremely well designed, controlled, and internally valid, but lack external validity. The problem is that controls, when introduced, may make the study artificial and thus limit the external validity.

3.2 The qualitative-exploratory end of the continuum

Researchers at the qualitative-exploratory end of the continuum value the concepts of dependability, confirmability, credibility, and transferability.

Generally speaking, those are the standards qualitative researchers hold up in judging the soundness of their research.

3.2.1 Dependability The standard of dependability in qualitative research requires that

researchers account for (1) any shifting conditions directly related to the people and things they are studying and

(2) any modifications they have made in the design of their study as it has progressed.

The purpose of such accounting is to help researchers and their readers get a more exact understanding of the context.

Dependability is roughly analogous to the concept of reliability (described above) in quantitative studies. The dependability of a qualitative study can be improved by using such techniques as stepwise replications, overlapping methods, and/or inquiry audits (for more on these concepts, see Davis, 1992, 1995; or Brown, 2001).

3.2.2 Confirmability The standard of confirmability in qualitative research requires that

researchers fully reveal the data they are basing their interpretations on, or at least make those data available.

The point is that, whether or not anybody actually takes the researchers up on it, they should make their data available so that other researchers could examine them and confirm, reject, or modify the original interpretations.

The confirmability of a qualitative study is approximately analogous to the concept of replicability (described above) in quantitative studies.

Confirmability can be improved by using audit trails (as described briefly in Davis, 1992, 1995; Brown, 2001; or in more detail in Lincoln & Guba, 1985).

3.2.3 Credibility The standard of credibility in qualitative research requires researchers

to show that they maximized the accuracy of their definitions and their characterizations of the people or things under investigation – especially as the various participants in the study judged those interpretations.

Credibility is more or less analogous to the concept of internal validity (described above) in quantitative studies.

The credibility of qualitative studies can be enhanced by using such techniques as member checking, negative case analysis, peer debriefing, persistent observations, prolonged engagement, referential analysis, and/or triangulation (for more on these techniques, see Davis, 1992, 1995; or Brown, 2001).

3.2.4 Transferability

The standard of transferability in qualitative research requires researchers to describe the research design, context, and conditions so well that the readers can decide for themselves if the interpretations apply to another context with which they are familiar.

Transferability in qualitative research is approximately analogous to the concept of generalizability (described above) in quantitative studies.

Transferability can be enhanced by using thick description (for more on this notion, see Davis, 1992, 1995; Lazaraton, 1995; or Brown, 2001).

3.3 How interactive are the standards ofsound research?

Figure 19.6 is my attempt to represent the interactive relationships among the various standards continua. Recall that earlier, I defined interactive as all possible combinations of the points along the continua.

In other words, for each of the standards continua in Figure 19.6, a study may need to focus entirely on the standard at the qualitative-exploratory end of the continuum or the standard at the quantitative-experimental end.

In other cases, researchers may need to balance their concern for standards at both ends of the continuum to varying degrees.

3.4 Ethical considerations General social sciences research ethics have been discussed from

many points of view (for an overview of this work, see Kimmel, 1988). Periodically over the years various national and international

organizations have even attempted to provide guidelines for their memberships (e.g., the American Psychological Association, which has provided various sets of guidelines for the ethical conduct of research: 1953, 1982, 1994).

Kimmel (1988) discussed some of the sorts of ethical problems that arise in social sciences research:

Instead of dwelling on the potential problems that unethical behavior can cause in research, I would prefer to simply delineate some of the steps that can be taken to avoid ethical pitfalls in applied linguistics research anywhere along the qual–quant continuum. Some of the most important ethical and professional responsibilities fall into three categories (adapted from Brown, 1997): participant issues, analysis responsibilities, and concerns for the audience of a study:

Since ethics is an area where all research methods and techniques come together and tend to agree, I will end here. However, I would like to mention one further set of considerations that is seldom listed in discussions of research ethics.

In my view, all researchers in applied linguistics have two overriding ethical responsibilities: (1) to continue reading, learning, and growing as researchers in order to better serve the field, and

(2) to design research that is effective and fits well into the particular institutional contexts involved by selecting those characteristics and standards along the qual–quant continuum that will best interact to provide systematic and principled answers to the many important questions that remain to be answered in applied linguistics.

Thanks!