a scientific approach to educational research edd 12

7/31/2019 A Scientific Approach to Educational Research EdD 12

1/52

A scientific approach to

educational research?


2/52

Whats this all about?

Understanding the assumptions behind the

scientific approach to educational research

Understanding the power and limitations of

experimental designs and of statistical tools Understanding the potential of mixed

methodologies and methods

Supporting critical reading and personalresearch decision making


3/52

Positivist Assumptionsafter Lincoln & Guba 1985

There is a single reality that can be revealed; thisreality will not be contested by fair minded

individuals

Part of this can be studied independently; the

whole is the sum of those parts

It is possible to study this objectively,

independently of the researcher.

Results will apply at other times; in other places Cause and effect can be distinguished

Inquiry is value-free


4/52

True of science?

Science is about building models not mapping reality.Different models are possible from the same data

Data has error so no picture of reality is error free(triangulate across forms of data, alternative studies etc)

0+0 = 100 Observing a system necessarily changes it

x.ph

Sometimes causes are complex and not open to

definition Values determine what science is done and what is

regarded as important


5/52

True of social science?

If its not really true of physical science, its

not likely to apply cleanly to social science

BUT ALSO

Scientific social research is dependent on

the logic of statistical significance

Meaning of statistical significance

Limitations of statistical significance

Issues of sampling


6/52

Dependence on the logic of

statistical significance An example

Group 1

10 people, average score=6

Group 2

10 people, average score=8

Is this difference between the groups important?

FIRST STEP: is it likely to occur just by chance?


7/52

Basic notion of significance

Assume no difference in outcome between the two groups.

This is our NULL HYPOTHESIS Ho

The groups are like random samples from the same population.

Would it be unusual for two random samples from one

population to have means as different as 6 and 8? If yes, we could assume that the two groups came from

different populations

Difference is statistically significant. Reject Ho

If no, we would say that the difference is probably just down tochance

Difference is not statistically significant. Accept Ho


8/52

Population

8 9 0 8 35 4 7 7 5

10 0 4 1 6

4 9 9 5 9

7 9 2 9 6

6 7 7 1 2

4 3 7 1 54 5 7 7 6

0 5 2 6 4

5 4 5 2 3

3 7 4 7 7

8 3 7 1 7

6 7 4 7 3

2 6 8 7 3

3 9 3 0 9

1 10 1 9 10


9/52

One random sample8 9 0 8 3

5 4 7 7 510 0 4 1 6

4 9 9 5 9

7 9 2 9 6

6 7 7 1 2

4 3 7 1 5

4 5 7 7 60 5 2 6 4

5 4 5 2 3

3 7 4 7 7

8 3 7 1 7

6 7 4 7 3

2 6 8 7 3

3 9 3 0 9

1 10 1 9 10

Mean of red group = 7.1


10/52

Two random groups

8 9 0 8 35 4 7 7 5

10 0 4 1 6

4 9 9 5 9

7 9 2 9 6

6 7 7 1 2

4 3 7 1 5

4 5 7 7 6

0 5 2 6 4

5 4 5 2 3

3 7 4 7 7

8 3 7 1 7

6 7 4 7 3

2 6 8 7 3

3 9 3 0 9

1 10 1 9 10

Mean of red group = 7.1; mean of blue group = 5.9

Difference = 1.2


11/52

Distribution difference of two means


12/52

Distribution difference of two means


13/52

Not just group differences

Is there a correlation between two

variables?

One group

10 people

IQ andattitude scores are linked


14/52

Correlation

IQ and attitude to school

attitude to school

20100

180

160

140

120

100

80

60

40


15/52

Correlation

Correlations

1.000 .564**

. .000

40 40

.564** 1.000

.000 .

40 40

Pearson Correlation

Sig. (2-tailed)

N

Pearson Correlation

Sig. (2-tailed)

N

IQ score

atti tude to s chool

IQ score

attitude to

school

Correlation is significant at the 0.01 level (2-tailed).**. IQ and attitude to school

attitude to school

20100

180

160

140

120

100

80

60

40


16/52

Statistically significant?

Is it unlikely that we would get this

correlation if we just chose 40 pairs of

numbers from a set of random numbers?

If it IS unlikely, then our results probably

arent just pairs of random numbers

there is a link between IQ and attitude

If is NOT unlikely, then our result could

well be nothing more than chance


17/52

The importance of significance

It may be significant but does it

matter?


18/52

The importance of significance

It may be significant but does it

matter?


19/52

Effect of sample size

Critical Values ofrfor Rejecting the NullHypothesis

(r= 0) at the .05 Level Given Sample Size nn r

3 .9975 .878

10 .632

20 .444

50 .276

100 .196

500 .0881,000 .062

5,000 .0278

10,000 .0196Note: Values are taken from Table 13 in Pearson and Hartley (1962).


20/52

What if? arguments

If we get a significant result with a sample size if

100 how small would the sample have to be for

this to become non-significant?

(If very small fairly confident in rejecting Ho; if 90 (say)not very confident in rejecting Ho)

If we get a non-significant result with a sample

size if 20 how big would the sample have to be

for this to become significant?(If 30, say, not much confidence in Ho; if very largefairly

confident in Ho)


21/52

Effect size

121

sdmmsizeEffect

Effect Size =1

- the mean for one group coincides

with boundary score for the top 16% of

the other group


22/52

Non-significant results


23/52

Sampling

Samples that allow statistical generalisation

random

systematic

stratified random

cluster multi-stage

Samples that dont allow statistical generalisation quota

convenience

snowball


24/52

Sampling

Samples that allow statistical generalisation

random

systematic

stratified random

cluster multi-stage

Samples that dont allow statistical generalisation quota

convenience

snowball


25/52

Is your sample representative?

Since its quite difficult to get

representative samples, the answer may

well be no.


26/52

The experiment

SMs gold standard for revealing cause

and effect relationships


27/52

The experiment

Logic of experimental methodology

Avoiding threats to validity

Problems

Sampling and assignment

Models of causation

Solutions

Quasi-experimental designs Mixed methods within an interpretive

paradigm


28/52

OXO

Is X the cause of any difference between

the first and second observation?


29/52

Issues of validity at the design

stage experimental designs

Internal threats to validity

History

Maturation Testing

Instrumentation

Selection Statistical regression

Mortality


30/52

External threats to validity

Interaction of selection bias and

treatment

Interaction between testing and

treatment

Reaction to being in an experiment


31/52

True experiment

R OeX

- Oc


32/52

Why are experimental designs so

useful?

They deal with treats to validity.

..??

ROeX

- Oc


33/52

So why bother with SM at all?

Can be very powerful in revealing complex

interactions and relationships


34/52


35/52

Multiple regression

Dependent variable English O-level

Independent variables

DAT scores (VR,NA,VN,SR,MR,CSA,SP,LU)

Attitudes ( A, B, .I)

Creativity (Flu,Flex,Orig)

Eng O level =

0.65LU + 0.35SP +0.66VR +2.64B +31.26

Adjusted R squared = 0.36

Standard error = 15.18

I f t h kli t


36/52

Infant checklist

1 59 42 70 72 58 59 49

2 48 56 85 72 40

3 43 83 58

4 40 79 60

5 64

643 61

Clea

rspeech

Give

smessage

Liste

nstostory

Organisedtale

Describesenv

Pick

supsongs

Can

usetools

Draw

sonjects

Matc

hpictures

Copysquare

Copycross

Settlesaway

Rece

ptivetosch

Integr

ates-p

eers

Plays-peers

Rela

t

es-

teach

er

Eagertolearn

Pers

istsattask

Stay

sontask

Chooses


37/52

Cluster results


38/52

So why bother with SM at all?

Can be very powerful in revealing complex

interactions and relationships

So how do we get the benefits in spite of the

problems?

through limiting the population

through non-inferential stats

through links with interpretive designs


39/52

Limiting the population


40/52

Non-inferential use of statistics
http://images.google.com/imgres?imgurl=http://www.student.oulu.fi/~sigma/galleria/wanhat/limbo.jpg&imgrefurl=http://www.student.oulu.fi/~sigma/galleria/wanhat/&h=382&w=390&sz=28&tbnid=9sPueCsXDM8J:&tbnh=117&tbnw=119&start=104&prev=/images?q=limbo&start=100&hl=en&lr=&rls=GGLD,GGLD:2004-44,GGLD:en&sa=N


41/52

Quasi experiments no R

O X OO O


42/52

Using quantitative methods within

an interpretive methodology


43/52

Mixing approaches?

Paradigms as watertight boxes simply a

choice to be made

Paradigms as coherent systems that serve

as the starting point for creative thinking


44/52

One example

Using interpretive methodologies to

explore possible threats to validity in a

quasi-experimental design

I t ti A ti


45/52

Interpretive Assumptionsafter Lincoln & Guba 1985

Realities are multiple, and are

individually constructed

The knower cannot be separated from

the known

We can only make statements that are

time and context bound

All entities are continually shaping each

other

Inquiry is inevitably value-bound


46/52

Creative thinking

Linking qualitative and quantitative data

qualitative work gives rich exemplification of generalisablerelationships established by statistical methods (Sci

Paradigm)

quantitative work establishes the generalisablity ofhypotheses which emerge from a qualitative enquiry (SciParadigm)

qualitative and quantitative work are used together(iteratively) to deepen the understanding of the particularcaseson which we have been working. (Interp. Paradigm)


47/52

Its

NOT the purpose of qualitative work simply to give rich

exemplification of generalisable relationshipsestablished by statistical methods to give a humanface to a statistical study.

NOT that quantitative work should be used to establishthe generalisablity of hypotheses which emerge from aqualitative enquiry - as if this is in some way anecessary step in order that the qualitative findings canbe taken seriously.

BUT qualitative and quantitative work are used together

(iteratively) to deepen the understanding of theparticular caseson which we have been working.


48/52

Another example

Using scientific and interpretive

methodologies iteratively to explore factors

influencing learning in FE classrooms


49/52

C1 site differences

Student Negotiation

3.43.23.02.82.62.42.22.0

S

haredControl

3.5

3.0

2.5

2.0

1.5

ESOL LS

WrkbdAst

ADMPA

Engneer

ITSkills

BTECHlth

AVCET&TVoc Path

SpprtMS

GNVQBus

Connect2

Pth4 Prn

ASPsych

ELDrama

CACHE

Shared Control >Student Negotiation

Student Negotiation >Shared Control

Explaining the High SC/Low SN


50/52


grouping

Support for Mature Students

Self assessment, negotiation based onassignments, individual learning plans

agreed and reviewed by tutor and student

Workbased assessment

Individual support from tutor (undergroundworking)



51/52


grouping

Workbased assessment

Different geographical placements

Support for Mature Students and IT skills

Same room

but

Different times

ESOL Same room, same times

but

Different languages

ISOLATION

Does isolation feature


52/52

Does isolation feature

elsewhere?

Student Negotiation

3.43.23.02.82.62.42.22.0

SharedControl

3.5

3.0

2.5

2.0

1.5

ESOL LS

WrkbdAst

ADMPA

Engneer

ITSkills

BTECHlth

AVCET&TVoc Path

SpprtMS

GNVQBus

Connect2

Pth4 Prn

ASPsych

ELDrama

CACHE

Shared Control >Student Negotiation

Student Negotiation >Shared Control

Isolation(broadly defined) appeared to be

a factor related to a site culture in which

there was low student negotiation.

Isolation was a feature of

the top left sites

In 4 of the bottom rightsites isolation was not at

all evident in this site

a scientific approach to educational research edd 12

Documents