QUANTITATIVE INQUIRY

Hwa Chong InstituteSemester I, 2010

Advance Organizer

• Introductions• Unit structure and outline• Textbook and reading materials• Stages of conducting research:

1. Conceptualisation: The What and Why Issues in Education Research

1.1. General Orientation to Education Research

The Purposes of Education Research

• Why read education research? Notions of evidence-based practice

• Why do education research? Adding to the stock of knowledge in the discipline

OECD Definition of Research• Creative work undertaken on a systematic basis in

order to increase the stock of knowledge, including knowledge of humanity, culture and society, and the use of this stock of knowledge to devise new applications.

• Any activity classified as research and experimental development is characterised by originality; it should have investigation as a primary objective and should have the potential to produce results that are sufficiently general for humanity's stock of knowledge (theoretical and/or practical) to be recognisably increased.

• This can include both empirical and non-empirical scholarly work (added summary).

1.2. Developing Interests into Questions

What is a “good” research question?

• From where do you source your questions?– Previous literature– Theory– Empirical pilot data

• FINER framework:– Feasible– Interesting– Novel– Ethical– Relevant

Class Exercise: Identify Problems with these Questions

• Do females lack directional sense?• University lecturers’ attitudes towards student

evaluations of teaching • Do teachers use ICTs more in the classroom now

than before?• Management of children with behavioural

problems in classrooms• What school problems are faced by children

diagnosed with AD/HD?

Types of Research Questions• Empirical and non-empirical• “Quantitative” and “qualitative” questions• Some key concepts:

– Ontology – concerned with articulating the nature and structure of the world

– Epistemology – concerned with the nature of human knowledge– Methodology – concerned with principles and procedures of

inquiry– Paradigm (interpretive framework) – net of epistemological,

ontological, and methodological premises that guide research actions

Quantitative Research• Ontological– Existence of objective, absolute truths– Focus on operational definitions and rational

explanations• Epistemological– Researcher (knower) and object of study (known)

independent – focus on objectivity– Assumption that inquiry can approximate objectivity

(i.e., be value-free); disagreements between observers due to errors and/or observer biases

• Methodological– Replicability as a means for testing truth– Focus on generalization– Criteria – notions of internal/external validity

Qualitative Research• Ontological– Reality is local and specific– Constructions cannot be absolutely true or correct (but

can be less sophisticated/informed)– Reality actively constructed rather than discovered

• Epistemological– Researcher and object of study inherently dependent– Inquiry inherently value-bound– Multiple interpretations can be equally valid

• Methodological– Focus on induction– Relative lack of emphasis on generality – use purposive

samples– Criteria – trustworthiness, credibility, transferability,

confirmability

Positioning in “the” qualitative/quantitative debate

• The purist– Qualitative and quantitative methods are incompatible

(grounded in different ontologic and epistemologic assumptions)

– Advocate mono-method studies• The situationalist

– Both approaches have merit for answering different types of research question

– Advocate mono-method studies but accept the two approaches as complementary

• The pragmatist– Dichotomy is false; many associations with each paradigm

erroneous (e.g., experiments must be quantitative)– Advocate mixed-method approaches

Class Exercise: Classifying Research Questions

• What is the effect of cooperative learning methods on mathematics achievement in sixth graders?

• What are the perspectives of primary level teachers on using suspension as a behaviour management strategy?

• What is the relationship between severe behaviour problems and academic performance?

• How do English teachers deal with students with disabilities in the secondary level classroom?

Individual Exercise: Formulate a small set of possible questions related to an

area of interest to you and note whether these lend themselves to a qualitative or quantitative approach

1.2. Presenting a Rationale

Main Elements

• Establish the importance of the question and place in a meaningful context.

• Review what has already been done through a review of the literature

• Present a thesis/argument and demonstrate how the questions derive from this background

Conducting a Systematic Review of the Literature (Example from Boakes, 2009)

• 2.2. Literature Review Method• To ensure that the background for the current research programme was based on

a systematic review of previous AP research, an explicit set of procedures was used in first locating this literature. Initially, all studies which made reference to investigating or using AP were collated. Inclusion/exclusion criteria were then applied systematically to determine which of these should be considered to inform the hypotheses proposed.

• 2.2.1. Search Procedures• As indicated, the initial literature pool included all studies that had used or

investigated AP effects. The only exclusion criterion applied at this initial point was that the articles had to be available in English. To generate the initial pool, systematic searches were conducted using the following databases: PsycINFO, Science Direct, ProQuest 5000 International, and Google Scholar. Once these studies were gathered, the references in each were scanned for other articles that might be relevant. Previous literature reviews were also consulted for any additional studies.

• 2.2.2. Inclusion and Exclusion Criteria• A vast array of different procedures has been used in studies that have purported

to examine AP effects. After generating the initial study pool, the studies were sorted and their key methodological characteristics tabulated. Inclusion/exclusion criteria were then generated based on the following categories.

• Inclusion Criteria. Various study characteristics were deemed to be potentially important moderators of the AP effects obtained, but not to invalidate studies as investigations of these effects. These were:

• Format of Prime-Target Pairs: Three main types of stimuli have been used in previous studies as primes and targets: pictures, facial expressions, and words (e.g., nouns, adjectives, and verbs). All combinations of these were included.

• Attention to the Prime: Studies varied considerably in terms of the instructions issued to participants regarding the prime. In some cases, participants were asked to attend to the prime. In others, they were told to ignore the prime or were given no instructions relating to the prime. All such studies were included in the review.

• Exclusion Criteria. Clearly, as the goal of this review was to explore the effects obtained in “basic” (i.e., not applied) AP studies, any studies which applied AP in the investigation of other processes (e.g., racial stereotyping, depression, anxiety) were excluded.

Multiple Purposes of a Rationalefrom http://www.meaning.ca/archives/archive/art_how_to_write_P_Wong.htm

• Ensures that you are not "reinventing the wheel". • Gives credits to those who have laid the groundwork for your

research. • Demonstrates your knowledge of the research problem. • Demonstrates your understanding of the theoretical and research

issues related to your research question. • Shows your ability to critically evaluate relevant literature

information. • Indicates your ability to integrate and synthesize the existing

literature.• Provides new theoretical insights or develops a new model as the

conceptual framework for your research. • Convinces your reader that your proposed research will make a

significant and substantial contribution to the literature (i.e., resolving an important theoretical issue or filling a major gap in the literature).

Common Problems with Rationales from http://www.meaning.ca/archives/archive/art_how_to_write_P_Wong.htm

• Lacking organization and structure • Lacking focus, unity and coherence • Being repetitive and verbose• Failing to cite influential papers • Failing to keep up with recent developments • Failing to critically evaluate cited papers • Citing irrelevant or trivial references • Depending too much on secondary sources

Small Group Exercise: Discussion of Brief Rationales

2. Research Design: The How Issue in Education Research

Quantitative Approaches

2.1. Variables in Quantitative Research

• Variable – a phenomenon that is changeable or subject to variation

• Independent variable – the predictor or presumed effect variable

• Dependent variable – the outcome variable• Types of variables – categorical, ordinal,

interval, ratio

Class Exercise: Identifying and Classifying Variables

• A. A study indicated that wall colour may reduce the time taken by seventh graders to complete

• simple arithmetic tasks.• Grade level: ______________• Time taken: _____________• Wall colour: ________________• B. A study indicated that attitudes toward using ICTs in the classroom differed

across primary and• secondary teachers.• Attitudes toward using ICTs: ______________• Phase of schooling (i.e., primary/secondary): ____________________• C. A study indicated that amongst male teachers, attitudes toward quality

assurance differed with• time spent in the system.• Sex of teacher: ________________• Attitudes toward quality assurance: ____________• Time spent in system: _________________

2.2. Sampling in Quantitative Research

Probability-based sampling– Aimed to ensure that individuals have the same chance of being

included in the sample– Simple random sampling procedures (e.g., random number

generation)– Stratified random sampling (e.g., identify subpopulations of

interest)

• Non-probability-based sampling– Convenience sampling (grabbing anyone willing)– Snowball sampling (ask respondents to nominate others who

might be willing to participate)

2.3. Quantitative Data Collection: Measurement Issues

• Common sources of quantitative data:– Archival records– Direct observation– Questionnaires, inventories, scales, tests

• The notion of operational definitions– Conceptual – abstractions that facilitate

understanding– Operational – statement of definition in

measurable terms

Common domains of measurement in Quantitative

Research

• The cognitive domain• The affective domain• The psychomotor domain

Common approaches to measurement

• Cognitive domain:– Selected response– Short answers– Cloze tests– Essays

• Affective domain:– Questionnaires– Interviews

• Psychomotor:– Performance-based assessment– Structured observation

Technical Adequacy

the consistency of assessment: reliability the appropriateness of interpretations of

assessment: validity Bias in assessment interrelationships between reliability and

validity other indexes of test characteristics

The Purpose of the Standards

• to promote the sound and ethical use of tests• to provide assessment professionals with• guidelines for the evaluation, development, and• use of testing instruments• to provide a frame of reference for addressing

relevant issues

Stability of Assessments: Reliability

• Classical test theory and the true score model:– Observed score = true score + random error

• Common Sources of Measurement Error– inconsistencies across testing occasions– inconsistencies across forms of the tests– inconsistencies between raters– inconsistencies in sampling the content domain

• Key Concepts– reliability as a characteristic of test scores– major approaches to establishing reliability and

links to specific error sources– statistical basis for reliability analyses: the

reliability coefficient and the standard error of measurement

Common Approaches to Establishing Test Reliability

• The Test-Retest Method• Reactivity and practicality

• Alternate Forms Methods• Practicality and expense

• Internal Consistency Methods• Limited information on reliability

Some Current Issues in Evaluating Reliability

• limitations of the traditional approach– factors affecting reliability coefficients– obtaining independent estimates of error

sources and interactions between error sources

– Generalisability (G-)Theory

Purposes of Assessments: Validity

Key Concepts– validity as a property of test scores– validity as a matter of degree– specificity to particular uses or interpretations

Traditional Approaches to Validation– historical developments in conceptions of

validity– the three-part framework: criterion, content,

and construct validation

Content-Related Validity The Standards Definition:– the extent to which the sample of items, tasks, or

questions on a test are representative of some defined universe or domain of content

Approaches to Establishing Content-Related Validity– domain sampling, relevance, clarity– tables of specifications: content domains and structure– logical analysis of test content– examining test content and format– assessing suitability for the given purposes

Criterion-Related Evidence

The Standards Definition:– the extent to which scores are systematically

related to one or more outcome criteria Approaches to Establishing Criterion-Related Validity– predictive validity– concurrent validity– assessing criterion-related evidence according to

decision purposes

Construct-Related Evidence The Standards Definition:– extent to which test measures the identified underlying

characteristic of interest

• Approaches to Establishing Construct-Related Validity– explicating construct meaning– convergent and divergent evidence– deriving and testing predictions about test performance

from the underlying theory

Small Group Exercise: Constructing an Operational Definition of Bullying

Defining bullying

• Conceptual definition: Tattum and Tattum (1992) - "Bullying is the wilful, conscious desire to hurt another and put him/her under stress".

• A. Provide an operational definition of bullying based on the conceptual definition provided.

• B. Evaluate the following behaviours according to your operational definition:

• Hitting someone for no reason• Gossiping about another person• Coercing another person to do something they don’t want to

do• Taking the possessions of another person• Telling another person what to do• Refusing to speak with another person because of their

personal characteristics

• C. Evaluate the following scenarios according to your operational definition:• Jack is a slight boy in the 8th grade who seems to have “fallen out” with a

dominant group of other boys in his year. At least once per week, a group of five other boys play a prank on him, which can include stealing and hiding his locker key, to more physical pranks such as removing his trousers in front of his peers.

• Michael is a popular 11th grader with his peers. He is seen as a leader amongst members of his peer group. There is one boy in his year, however, who has taken a disliking to him. This boy frequently attempts to tease him over a number of minor issues (e.g., clothing, and his popularity with others). Michael takes no notice of this, and appears to “laugh it off”, because the boy who is engaging in these behaviours is himself unpopular with his peers.

• Sandra is in the 10th grade. She is socially isolated, generally spending her recess and lunchtimes alone. When she attempts to engage with other girls from her year group in joint activities in class, they refuse to speak with her. She is never invited to social events outside school.

The Revised Standards Framework

• has more background material• greater number of standards• reflects changes in federal law and measurement trends

affecting validity, etc.• addresses professional and technical issues of test

development and use• Move towards sources of validity evidence rather than

distinct types of validity• Notion of consequences (Messick) explicitly taken into

account

The 1999 Revision

Validity• Focus on the obligations of the test developer to

users,examinees, and other testing practitioners on user obligations to examinees

• Developers owe users– Enough information to make judgments about the

appropriateness of their interpretation of test scores for their intended use(s) population(s) for which test is appropriate constructs tested

– uses/interpretations NOT intended or ecommended– content descriptions, domains, criticality– qualifications of experts/judges/raters– rating/scoring procedures– population/situation variables involved in validation

• Developers owe examinees– reasonable assurance that tests will not be used

improperly– use of content/constructs appropriate to recommended

and/or intended use(s)– adequate warning to users against uses NOT

recommended or intended– accuracy in criterion validation, when performed– investigation of unintended/unexpected outcomes (e.g.,

DIF)

Bias In Assessment

Bias and error in testing• Formal definition of bias

Differences in the extent to which the test taker has had the opportunity to know and become familiar with the specific subject matter or specific processes required by the test item

Some Common Sources of Test Bias– test content and characteristics (e.g.,

cultural, gender, language)– test takers (e.g., anxiety, achievement

motivation, self-esteem, personality characteristics, practice)– test environment (e.g, examiner

characteristics)– test usage (e.g., overinterpretation,

improper standardisation, selectivity)

Characteristics of “culture-loaded” and “culture-reduced” tests

• Culture-Loaded: Comprised of objects, situation, beliefs, or lore peculiar to one particular culture or subculture (e.g., reliance on verbal content, recall of past-learned experience, reading required, speed tests)

• Culture-Reduced: Comprised of objects,, information, or symbols that are universal, i.e., members of any of the cultures or subcultures being tested would have equal familiarity and experience with the content and demands of the test items (e.g., performance-oriented, purely pictorial, oral response, power tests, non-verbal content, solving novel problems)

Examples of Bias: Intelligence Testing

Weschler’s View of Intelligence

• Viewed intelligence as a global, aggregate entity, rather than a specific ability:

“....the capacity of the individual to act purposefully, to think rationally, and to deal effectively with his or her environment” (1944, p.3).

Class Exercise: Critiquing the WISC III subtests

Similarities Subtest

• Task: Child must explain the similarity between pairs of common objects or concepts

• ExamplesQ1. In what way are MILK and WATER alike? How are they

the same?• 1 Point (Pass)

– Both are liquids/beverages/fluids– You drink both; something you drink– Quench Thirst– Nutritious; Good for you

• 0 Points (Fail)

– Substances – Both are food– Both are wet– You could spill them– Both things you pour– Both in a jug

Q18. In what way are SALT and WATER alike?• 2 Points

– Any response indicating that both are necessary for life or both are chemical compounds, e.g., People use them to live; Things we need in our bodies; Basic things for our bodies; Both chemical compounds

• 1 Point

– Compounds (q)– Both have chemicals in them (q)– Natural substances– Both made of minerals– Both mixtures of different elements (q)– Ingredients– Used to cook with (q)– Edible (q)– Put them both in food– Body perspires them– Both found in the ocean

• 0 Points– Both (substances, resources, have elements) (q)– Minerals; Chemicals– Organic Compounds– Salt dissolves in water– Eat and drink them

Vocabulary Subtest

• Task: Child must explain orally the meaning of words read aloud by the examiner.

• ExamplesQ1. What is a CLOCK?• 2 Points

– A timepiece; it tells you the time– To time something– You can learn time with it– It has hands that move around on a dial

• 1 Point

– A watch– It is for time (q)– It goes tick-tock (q) – It’s got little numbers going around it (q)– It has (hands, a dial, numbers) (q)– It tells you when it’s morning, afternoon, and night– Tell you when to go home and when to go to school

• 0 Points– An alarm clock; Wakes you up (q)– A toy– Hangs on a wall (q)– It makes a sound (q)

Comprehension• Task: Child responds orally to a series of questions that require

solutions to commonplace problems or understanding of social rules and concepts

• Example– Q6. What is the thing to do if a boy (girl) much smaller than yourself starts to

fight with you?• 2 Points

– Don’t (fight, beat her up, hurt him, hit back)– Don’t hit him, find out what’s the matter– Tell them you don’t want to fight– Try to talk her out of it; Calm him down; Talk to her sensibly– Leave him alone– (Walk, Turn, Go) away– Nothing (q) - I’d do nothing, not fight with her

• 1 Point– Tell him to (stop, quit, settle down, leave you alone)– Tell him you don’t want to hurt him (q)– Tell her (not to fight, forget it, go away)– Avoid the fight (q)– Ask (the teacher, my father,) to stop her

• 0 Points– Don’t do anything; Nothing (q)– Say, “I’m telling on you”– (Punch, Hit, Fight, Beat) him– Tell (the teacher, my father) (q)– Hit her lightly; Shove him away; Hold him down– Tell her to leave me alone or else– I’d just let her fight (q)– Call for help

Small Group Exercise: Constructs and Operationalisation

Look at three “common” terms that we use to characterize people: (1) Self-confidence; (2) Personality; (3) Intelligence.

Think about the first steps in how you would go about designing an assessment for these characteristics in adults.

Suggested Steps in Tackling the Task:• Define the construct.• Decide whether this is a uni- or multi-dimensional construct.• Propose three exemplar questions/tasks to assess each dimension.• Identify any factors (e.g., situational) that may moderate (or bias)

responses to/performance on the tasks.