|

faculty of arts applied linguisticsict in education

The EMBED Project: Integrating the CEFR in academic writing and speaking

Sake Jager, Applied Linguistics / ICT in EducationFaculty of ArtsUniversity of Groningen

(on behalf of the EMBED Project team)

1

|

faculty of arts applied linguisticsict in education

Overview

› Background› CEFR in Faculty of Arts› EMBED project› Future developments

2

|

faculty of arts applied linguisticsict in education

Language teaching in Faculty of Arts

1. Departments of Language and Culture:› Majors / Minors in language departments › Minors in inter-/multi-disciplinary programmes (e.g.

International Relations and International Organization, Communication and Information Sciences)

2. Language Centre:› Language teaching other faculties (e.g. Law, Economics

and Business, Medicine)› Staff training› External provision (commercial, individuals, companies)

3

|

faculty of arts applied linguisticsict in education

CEFR in Faculty of Arts› Proficiency outcomes at end of BA for all

language skills expressed according to CEFR since 2001

› Teaching innovation project to integrate CEFR at course level from 2006 to 2009

› Use of CEFR-related tools › Dialang, (European) Language Portfolio, WebCEF, CEFCult)

› Pedagogical objectives› Making students familiar with principles and levels of

framework› Making them aware of own levels of proficiency› Setting personal learning objectives

4

|

faculty of arts applied linguisticsict in education

EMBED Project: Using the CEFR for EAP

› Objectives: Standardization of assessment (consistency/transparency) Among teachers, across disciplines (International Relations and

Organizations, Law, English Language and Culture) Use of technology to support standardization

› Domain: Academic writing in English (essays/reports)

› Reference: Council of Europe Manual 2009

› Team: Kevin Haines, Petra Jansma, Wander Lowie, Estelle Meima, Nicole

Schmidt, Sake Jager

5

|

faculty of arts applied linguisticsict in education

Project procedures› Meetings project team

Screening of 120 Law essays for ‘flat’ samples, later samples other courses

Selection of c. 10 samples for assessment by panels Regular meetings to discuss outcomes panel assessment Publications and other dissemination activities

› Assessment panels Local panel: 5 experienced English teacher/assessors Local consultant External panel: assessors at universities from our network of partner

universities in Belgium (Leuven), Finland (Helsinki) & Italy (Padua)

6

|

faculty of arts applied linguisticsict in education

Assessments procedures› Familiarization (CEFTrain):

Introductory activities with scales Training area (tasks and samples)

› Standardization (Blackboard): Use of procedures described in A Manual Delivery through project Blackboard site

7

|

faculty of arts applied linguisticsict in education

8

|

faculty of arts applied linguisticsict in education

EMBED Project: Results› Overall agreement among panelists, but some discrepancies› Flat samples with same scores for Range, Coherence, Accuracy

and Argumentation are hard to find› Challenges for assessment:

Finding flat samples Multidimensional nature of proficiency Within-learner inconsistency (“instable system”) How to capture this variability in overall grade?

› Detailed discussion in following articles: Jansma, P.N., Jager, S., Lowie, W.M. & Haines K.B.J. (2011). De beoordeling van

Engelstalige essays: het ERK in het universitair schrijfvaardigheidsonderwijs. Levende Talen Tijdschrift, 1, 15-25.

Lowie, W.M., Haines, K.B.J. & Jansma, P.N. (2010). Embedding the CEFR in the academic domain: Assessment of language tasks. Procedia Social and Behavioral Sciences, 3, 152-161.

9

|

faculty of arts applied linguisticsict in education

EMBED Example (sample 6)

Has a sufficient range of language to be able to give clear descriptions, express viewpoints on most general topics, using some complex sentence forms to do so. Language lacks, however, expressiveness and idiomaticity and use of more complex forms is still stereotypic.

Has a good command of a broad range of language allowing him/her to select a formulation to express him/herself clearly in an appropriate style on a wide range of general, academic, professional or leisure topics without having to restrict what he/she wants to say. The flexibility in style and tone is somewhat limited.

B2

C1

WRITING; RANGE (– applied to sample #6)

|

faculty of arts applied linguisticsict in education

Sample 6, fragment

According to the Declaration of the human rights, has every person the right on these universal rights. This means that states should protect people from abuses of the human rights. However, in many states the government does not protect these rights; moreover, the government violates the them. Even though almost every state has signed the Universal Declaration. The discussion when there is a right to intervene is difficult; however, the United Nations (UN) should have the right to intervene in states which seriously abuses human rights, not only because the purpose of the UN is to protect the international security, but also because of the importance of the human rights.

|

faculty of arts applied linguisticsict in education

Sample 6: Range12

Sample 6 R C A Argument Overall

Internal assessors

x 5

B2.2 B2.2 B2.1 B2.2 B2

Local consultant

B2.2 B2.2 B2.1 B2.2 B2

External Finland 1

B2 B2 B1 B2.2 B2

ExternalFinland 2

B2 B2 B1 B2.2 B2

ExternalItaly

B2 B1 B1 B2.2 B2

External Belgium

B2 B2 B1 B2.1 B1

|

faculty of arts applied linguisticsict in education

Sample 6: Accuracy13

Sample 6 R C A Argument Overall

Internal assessors

x 5

B2.2 B2.2 B2.1 B2.2 B2

Local consultant

B2.2 B2.2 B2.1 B2.2 B2

External Finland 1

B2 B2 B1 B2.2 B2

ExternalFinland 2

B2 B2 B1 B2.2 B2

ExternalItaly

B2 B1 B1 B2.2 B2

External Belgium

B2 B2 B1 B2.1 B1

|

faculty of arts applied linguisticsict in education

Sample 6: Argument14

Sample 6 R C A Argument Overall

Internal assessors

x 5

B2.2 B2.2 B2.1 B2.2 B2

Local consultant

B2.2 B2.2 B2.1 B2.2 B2

External Finland 1

B2 B2 B1 B2.2 B2

ExternalFinland 2

B2 B2 B1 B2.2 B2

ExternalItaly

B2 B1 B1 B2.2 B2

External Belgium

B2 B2 B1 B2.1 B1

|

faculty of arts applied linguisticsict in education

EMBED Project: Results15

Sample 6 R C A Argument Overall

Internal assessors

x 5

B2.2 B2.2 B2.1 B2.2 B2

Local consultant

B2.2 B2.2 B2.1 B2.2 B2

External Finland 1

B2 B2 B1 B2.2 B2

ExternalFinland 2

B2 B2 B1 B2.2 B2

ExternalItaly

B2 B1 B1 B2.2 B2

External Belgium

B2 B2 B1 B2.1 B1

|

faculty of arts applied linguisticsict in education

Dissemination› Sharing and transparency are key aspects of

CEFR standardization› Dutch CEFR site www.erk.nl set up to facilitate

access to information on and samples of CEFR› EMBED results integrated (no HE samples on

this site previously) › ERK website (sample 6)

16

|

faculty of arts applied linguisticsict in education

Evaluation› EMBED useful for:

Bringing out the multidimensional nature and variability of language proficiency

Enhancing transparency of assessment Producing localized samples for standardization Bringing CEFR-based assessment to the

university level (LC and Acad lang depts, across disciplines)

Establishing role of technology in this context Sharing experiences with others

17

|

faculty of arts applied linguisticsict in education

Future directions› Application of EMBED procedures to oral production and

oral interaction tasks› Online collaborative tasks (Skype, videoconferencing) › WebCEF as assessment tool for panels› Blackboard Interactive Rubrics › Extension to other languages› Finding more colleagues for external validation› Incorporating student portfolios (Footprint)› Setting up new projects in this area

18

|

faculty of arts applied linguisticsict in education

Thank you for your attention

On behalf of the Embed Project team:Sake JagerPetra JansmaEstelle MeimaNicole SchmidtKevin HainesWander Lowie

Please feel free to contact us:Wander Lowie - w.m.lowie@rug.nl

19