Research Methods - An Introduction

Gabriela Avram

What’s the Difference Between “Method” and “Methodology”?

Method: p Techniques for

gathering evidence p The various ways of

proceeding in gathering information

Methodology: p The underlying

theory and analysis of how research does or should proceed, often influenced by discipline

(Sandra Harding)

Epistemology, Methodology, and Method

"a research method is a technique for (or way of proceeding in) gathering evidence" while "methodology is a theory and analysis of how research does or should proceed" and "an epistemology is a theory of knowledge"

Sandra Harding, “Is There a Feminist Method?”

Epistemologyp Epistemology (theory of knowledge) is the branch of

philosophy concerned with the nature and scope (limitations) of knowledge; addresses the questions: ■ "What is knowledge?" ■ "How is knowledge acquired?" ■ "What do people know?" ■ "How do we know what we know?"

p Internalism vs externalism p A priori/a posteriori p Empiricism; Rationalism; Constructivism

An Overview of Empirical Research Methods

Descriptive (Qualitative) p Ethnography p Case Study p Survey/Sampling p Focus Groups p Discourse/Text Analysis p Quantitative Description p Prediction/Classification

Experimental (Quantitative) p True Experiment p Quasi-Experiment p Meta-Analysis

From Lauer and Asher, Composition Research: Empirical Designs and MacNealy, Empirical Research in Writing

Research methods in Software Engineering

p Controlled experiment ■ Wohlin et al. 2000

p Survey ■ Pfleger and Kitchenham 2001

p Case study ■ Yin 1994

p Ethnography p Data collection: Interviews, observation, documentation, ... p Design science / constructive research

■ Hevner et al. 2004, Kasanen et al. 1993 p Action / participatory research

■ Avison et al. 1999, Stringer 1999

A different view –Research methods in IxD

Three ways of studying design (Jacob Buur, Mads Clausen Institute) ■ Ethnographic research - Observing actual practice ■ Action research - Experiments in practice - ■ Research through design - Experiments with artefacts

■ Critical design/ Speculative design

Assessing Methodsp Research Question(s) is/are key p Methods must answer the research question(s) p Methodology guides application p Epistemology guides analysis p All must include “rigor”

Formulating a research questionp The research question serves two purposes:

■ It determines where and what kind of research the writer will be looking for.

■ It identifies the specific objectives the study or paper will address.

p Qualitative or quantitative?

Example of Research Question

p “What are the practices of cultural heritage professionals in small museums in preparing, creating and adapting museum activities, and how do digital technologies mediate those practices? “ (Laura Maye, PhD thesis, 2016)

Experimental Research: True Experiment

+ Random sampling, or selection, of subjects (which are also stratified)

+ Introduction of a treatment + Use of a control group for comparing subjects who

don’t receive treatment with those who do - Adherence to scientific method (seen as positive,

too) - Must have both internal and external validity - Treatment and control might seem artificial

Experimental Research: Quasi-Experiment

+ Similar to Experiment, except that the subjects are not randomized. Intact groups are often used (for example, students in a classroom).

+ To draw more fully on the power of the experimental method, a pretest may be employed.

+ Employ treatment, control, and scientific method - Act of control and treatment makes situation artificial - Small subject pools

Positive Aspects of Experimental Research

p Tests the validity of generalizations p Seen as rigorous p Identifies a cause-and-effect relationship p Seen as more objective, less subjective p Can be predictive

Problems with Experimental Researchp Generalizations need to be qualified according to

limitation of research methods employed p Controlled settings don’t mirror actual conditions;

unnatural p Difficult to isolate a single variable p Doesn’t allow for self-reflection (built-in)

Survey Research+ An efficient means of gathering large amounts of

data + Can be anonymous and inexpensive - Feedback often incomplete - Wording of instrument can bias feedback - Details often left out

Survey Research+ Important issues: + Data requirements + Data generation method + Sampling frame + Sampling technique + Response rate and non-responses + Sample size + Accuracy range and confidence level

Case Studies+ Focus is on individual or small group + Able to conduct a comprehensive analysis from a

comparison of cases + Allows for identification of variables or phenomenon

to be studied - Multiple sources and methods - Time consuming - Depth rather than breadth - Not necessarily representative or generalizable

Types of case studiesp Exploratory p Descriptive p Explanatory

Approach to time: p Historical p Short-term p Longitudinal

Ethnographies+ Observational field work done in the actual

context being studied + Focus on how individuals interrelate in their own

environment (and the influence of this environment)

- Difficult to interpret/analyze - Time consuming/expensive - Can influence subject behavior

Focus Groups+ Aid in understanding audience, group, users + Small group interaction more than individual

response + Helps identify and fill gaps in current knowledge re:

perceptions, attitudes, feelings, etc. - Does not give statistics - Marketing tools seen as “suspect” - Analysis subjective

Discourse/Text Analysis+ Examines actual discourse produced for a

particular purpose (job, school) + Helps in understanding of context, production,

audience, and text + Schedule for analysis not demanding - Labor intensive - Categories often fluid, making analysis difficult

Quantitative Descriptive Studies+ Isolate systematically the most important variables

(often from case studies) and to quantify and interrelate them (often via survey or questionnaire)

+ Possible to collect large amounts of data + Not as disruptive + Biases not as likely - Data restricted to information available

Positive Aspects of Descriptive/Qualitative Research

p Naturalistic; allows for subjects to interact with environment

p Can use statistical analysis p Seeks to further develop theory (not to influence

action); Prescientific p Coding schemes often arise from interplay

between data and researcher’s knowledge of theory

Problems with Descriptive/Qualitative Research

p Impossible to overlay structure p Impossible to impose control p Subject pool often limited, not representative p Seen as more “subjective,” less rigorous p Beneficial only in terms of initial investigation to

form hypothesis

“Bovine design”

p Ann Light, Alison Powell, and Irina Shklovski. 2017. Design for Existential Crisis in the Anthropocene Age. In Proceedings of the 8th International Conference on Communities and Technologies (C&T '17). ACM, New York, NY, USA, 270-279. DOI: https://doi.org/10.1145/3083671.3083688