Malcolm Hunt and Mike Harris

Evidence & Research Team, Becta

Becta’s Evidence Base on the Becta’s Evidence Base on the educational use of ICT educational use of ICT

– – an approach to evidence collection, analysis and dissemination

Presentation Overview

Becta’s Remit and the role of the E Base

The development & working of the

Evidence Base

- The process of data collection & analysis

- Key Outputs

Future Developments

Conclusions

Becta’s Remit and Evidence Work -1

To develop an evidence base to underpin the educational use of ICT, To draw together the evidence from national and international sources, projects and research literature, To consider how best to disseminate these findings in ways that influence practice To identify and exemplify models of effective practice. To offer advice that is both timely and evidence informed.

Becta’s Remit and Evidence Work -2

To develop Becta’s Awards Scheme as a primary source of information about the effective use of ICT in teaching, learning and management.To study trends in technology, and carry out in-depth case studies to evaluate new technologies, applications and software.To evaluate the take-up of technologies, and provide advice on the educational value and sustainability of new and emerging technologies.

Becta’s Research Strategy and recent Programme of Work

Managed Longitudinal Studies

Adding value to secondary sources

Analysing data

Awards, fieldwork, & online forum

Planned interventions & short

term evaluations

• Evidence Evidence BaseBase

• ICT Research Network

• Research Bursaries

• Becta Research website

• ‘What the Research Says’ series

The Evidence Base & Associated Outputs

EVIDENCE BASE

• Hardcopy files

• End Note database

• Key Theme priorities

Academic literature

Market Intelligence

Scanning Process

Bibliographies & Summary Tables

What the Research Says Series

Literature Reviews

Reports to DfES, ICT in Schools Division

The development & working of the Evidence Base

Developed from March 2001 onwards

Increasing demands on Directorate for evidence material

Directorate proposal 2001-2002: “…to develop a robust, sound evidential base on the use and impact of ICT and education.”

Background

To provide a searchable database of key evidence material for the purpose of supporting and enhancing the Directorate’s project work and responses to requests for information.

Aim

3,497 documentsArticles, reports, research papersOrganised under themesAbstracts stored on BRS database, now Endnote library

Documents stored in hardcopy files.

Current status

Particular response to conditionsNot conceived as a research-dedicated database

Not an analytical tool in itselfNeed for flexibility, changing needs and priorities

Responds to levels of evidence, and supports a range of outputs

Developing an ‘evidence culture’.

Key considerations

AssessmentAttainment /improving standardsAttitudes/behaviour (including motivation)Continuing Professional Development (CPD)Cost effectiveness/TCODigital Divide and equality issuesEffective pedagogy - barriers to effective useEffective pedagogy - classroom practiceEffective pedagogy - factors for effective use

Key evidence themes

Effective pedagogy in online learningHealth and SafetyHome/school/community linksICT support for schoolsInitial Teacher Training (ITT)Internet safetyManagement (+ MIS and data rich institutions)MLEs/VLEsNetwork technologies

Key evidence themes

Online communitiesPortable technologiesSEN/InclusionStatistics/targets/policyTeacher workloadsVideo conferencingWeb-based technologiesWhiteboardsWhole school ICT integration.

Key evidence themes

Key evidence themesWeb-basedtechnology

3%Statistics

15%

Teacherworkloads

1%

Videoconferencing

4%

Effectivepedagogy

19%

Health & safety1%

Home-school links5%

Internet safety6%

Management5%

MLEs/VLEs4%

Network technology

6%

Onlinecommunities

2%

Portables5%

Presentation technology

4%

Costeffectiveness

1%

Digitaldivide

6%

CPD3%

Attainment5%

Attitudes2%

Assessment4%

SEN1%

Scanning - magazines/journals/newsletters

TES

THES

ACITT newsletter (email)

Assoc. for learning technology newsletter/journal

Computer education

Connected (NGfL Scotland)

Education, communication and information (ECi)

Education and Information Technologies (ejournal)

Scanning - magazines/journals/newsletters

Innovations in education and training international (ejournal)

International Journal of Educational Telecommunications

NAACE newsletter (email)

NGfL Scotland newsletter (email)

Teaching ICT (ACITT)

BJET (ejournal/paper copy)

JCAL (ejournal)

Scanning - magazines/journals/newsletters

Computers & education (ejournal)

Educational technology research and development

Review of educational research

TOPIC

Teacher development

Educational technology

British educational research journal (ejournal)

British journal of special education (ejournal/paper)

Scanning - magazines/journals/newsletters

Becta Extra (ejournal)

Special

EC&T

JITTE

NFER news/NFER current awareness

Scanning - websitesAdam Smith Institute

Cambridge University School of Education

DfES research

European Schoolnet

Internet Watch Foundation

Learning Lab

MMU Institute of Education

Mirandanet

NFER/CERUK

Scanning - websites

OECD CERI

Safer internet

Scottish Council for Research in Education Specialist Colleges Trust

The Education Network (TEN)

University of Exeter TLRP

Becta

What’s new in elearning?

Online research resources

BEI

ERIC

Australian Education Index

Educational Research Abstracts online

PsycINFO

De Havilland (political information source)

LexisNexis

ZETOC (British Library electronic table of contents)

Plus web search engines.

Documents - filtering at each stage

Literature search 100-500 Scope of field

Wider bibliography 50 Key areas/themes, methodologies

Selected bibliography/ Robustness, how

summary table 10-15 representative

‘What the Research 10-15 Also availability, Says’ title accessibility.

Key outputs

Responses to enquiries (internal, DfES, researchers, media)

Bibliographies/summary tablesQuarterly research briefings ‘What the Research Says’ seriesResearch reports, literature reviews.

Literature search, summary table/bibliography, ‘What the Research Says’ = 17.5 hours or 2.5 days each per theme

Project total:

Scanning = 25 days

Literature searches = 30 days

Summary tables/bibls = 30 days

‘WTRS’ titles = 30 days.

Project effort

Levels of evidence and range of outputs

Not systematic reviews, instead:Counter ‘lack of evidence in ICT’Stimulate research into ICTShare key findings and themesDevelop an organisational ‘evidence culture’

Accessibility to practitioners

Quality

Prioritise refereed research, larger sample sizes, robust methodologies

Discuss initial findings with experts (colleagues, subject associations)

Supply lists of sources (eg. bibliographies/summary tables)

Debate issues such as synthesis and future research in forums (ICT Research Network, conferences)

Next phase of ICT research?

Quality

Malcolm Hunt and Mike Harris

Evidence & Research Team, Becta

Education Funders’ Forum Meeting, Education Funders’ Forum Meeting, 14 January, 200414 January, 2004

Becta’s Evidence Base on the Becta’s Evidence Base on the educational use of ICT educational use of ICT

– – an approach to evidence collection, analysis and dissemination

Post Presentation Slides and issues arising from questions/ discussion

Given the short time frame in which you operate surely rigour is compromised?

Type & nature of Evidence

What evidence do you need to:- Know something

- Believe something- Do something

What does this mean for:- A civil servant or policy maker working at a national level- A LEA official- A Head, ICT Co-ordinator or teacher working at the ‘chalkface’

The Amount of Evidence- One conclusive research study or a combination of evidence from various sources?- What amounts and types of analysis are acceptable for a given situation or audience?

Type & nature of the Research Questions

Need to move on from whether ICT has general educational benefit to how and why ICT works

Need to refine our research questions and consider subsets of ICT- By subject- By ICT application eg Email, use of spreadsheets

Focus more on the context of use

Maddux model of Maddux model of Stages in ICT researchStages in ICT research

STAGE 1: Computers produce general educational benefits

STAGE 2: Exposure to a specific application produces a specific educational gain

STAGE 3: Which and how learners interact with teaching variables when using specific aspects of ICT