assessing taiwanese learners’ italian proficiency: an ... file/doc/nati.pdf · finally changed...
TRANSCRIPT
1
Assessing Taiwanese Learners’ Italian Proficiency: an Explorative Study
Francesco Nati
Department of Italian Language and Culture
Fu Jen Catholic University, Taiwan
Abstract
The Department of Italian Language and Culture at Fu Jen Catholic University, created in
1996, is the only academic entity specifically devoted to the teaching and promotion of Italian
language and culture in Taiwan.
Growing relationships between Taiwan and the European Union in recent years and the
subsequent active promotion of the Common European Framework of Reference (CEFR)’s
guidelines, especially for what concerns Language Proficiency Tests, did have some impact in
Taiwan, especially since the adoption of the CEFR by the Ministry of Education in 2005. Locally
much has been said and discussed in relation to the CEFR’s levels and their suitability for the
Taiwanese EFL context, and different opinions exist as to whether it should be adopted for teaching
and testing; as discussion related to other European languages ensued, we believe that it’s
worthwhile to carry out an evaluation regarding Italian in Taiwan.
The purpose of this study is to carry out an explorative comparison between the B1 level of
the four officially recognized Italian Language Certifications (CILS, CELI, PLIDA and IT),
introducing their design characteristics and investigating their appropriateness in measuring our
students’ language ability. For this research, we decided to start from the analysis of passive skills
(listening and reading) because they can be easily graded in loco, while active skills are usually
graded in Italy by each testing center and thus require a more specific training.
Research methods for this study are mainly quantitative: statistical data regarding the results
of the four Italian language proficiency test will be presented (an experimental group of 8 university
students was selected, all sharing the same basic characteristics: around 20 years of age; 2 years and
a half of Italian learning; mother tongue Chinese speakers), alongside with an analysis of university
scores from this Department regarding the students’ performance in our Language courses;
moreover, students’ questionnaires will be presented regarding their perception of the four language
tests and their opinions on language courses taken within our Department in comparison with
official Italian Language Certifications.
The results of this study will shed further light on the possibility of developing an Italian
Language Proficiency Test for Taiwanese users, and at the same time will give us an insight into
our students’ language performance in comparison with international standards.
Keywords: Language assessment, Italian, Chinese learners, Taiwan
2
ASSESSING TAIWANESE LEARNERS’ ITALIAN PROFICIENCY: AN EXPLORATIVE STUDY
Francesco Nati
Fu Jen Catholic University
3
1. Introduction
The Department of Italian Language and Culture at Fu Jen Catholic University, created in
1996, is the only academic entity specifically devoted to the teaching and promotion of Italian
language and culture in Taiwan; every year we accept around 60 students (most of whom are
Chinese mother tongue speakers) who will obtain, hopefully within four years, a bachelor’s degree
in Italian language and culture. General linguistic goals for each year are largely set according to
the standards of the CEFR, with the first year expected to reach an A1-A2 level, the second year an
A2-B1 level and the third year a B1-B2 level; since the fourth year has almost no compulsory
language courses, it is very difficult to determine the linguistic level of the students at the time of
graduation. We are also an authorized center for CILS exams (one of the two main officially-
recognized Italian certifications - see next section for more information), held twice a year in our
University, with an average of 14 examinees registering at every session, a good proportion of
which are students from our Department1.
In this study we carried out an explorative comparison between the four officially
recognized Italian Language Certifications (CILS, CELI, PLIDA and IT), concentrating on passive
competences (reading and listening tasks), mainly because they can be easily graded in loco without
specific training. For our research, 8 volunteers sharing the same study background were found and
all four language tests’ Listening and Reading exercises were administered; subsequently, exam
sheets were graded and results were compared to the students’ scores within our Department.
Additionally, questionnaires were given to the testees in order to both do self-evaluation and to
provide opinions on the tests’ perceived difficulty.
The results of this research are meaningful not only in the perspective of better
understanding our students’ language ability and verifying if their proficiency will be up to our
expected standards (from another point of view, it may also tell us something about the
appropriateness of choosing one or another certification in order to measure their linguistic level),
1 Statistical data referred to the years 2001-2011, gathered from Tulli (2011).
4
but also in seeing if and how different testing styles for the same language level (in this case, B1)
may lead to different performances; we believe that the importance of this last point cannot be
overlooked because, in the 21st century, owning an official language certificate may sometimes
make the difference in accessing universities, finding jobs, calculating scores for public selections
etc., hence candidates could and should fully understand the differences between different tests
before registering in one of them.
2. An historical overview of the officially recognized Italian proficiency tests.
Language assessment in Italy is a rather new area of research that has witnessed remarkable
progress in the last 15-20 years; it has been pointed out2 that only in 1982, on the occasion of a
national convention regarding the status of the Italian language in the world, did Italian scholars and
government representatives recognize the importance of developing a clear policy for the promotion
of Italian in the world, and at the same time the necessity of establishing an officially recognized
Italian proficiency test, catching up with other European languages who were already starting to
take their first steps in this direction.
After some years of researching, based on different theoretical starting points, in the middle
of the nineties three academic entities came up with their own Italian language assessment models:
on one side was the newly founded University of Roma Tre, where the Department of Language
Sciences designed a proficiency test called “IT”; on the other side were the two Universities for
Foreigners of Siena and Perugia, which developed respectively the CILS (Certificato di Italiano
come Lingua Straniera - “Certificate of Italian as Foreign Language”) and the CELI (Certificato di
Lingua Italiana - “Certificate of Italian Language”) exams. The dissimilarity between their
approaches derived from these universities’ different history and background: while the Department
of Language Sciences at Roma Tre was more centered on pure and applied linguistics, the
Universities for Foreigners in Siena and Perugia enjoyed a much higher degree of
2 Vedovelli M. et al. (on print), p. 28-40 and Novello (2009), p. 99-128.
5
internationalization because of their conspicuous number of foreign students coming from every
part of the world, and thus had more opportunities to calibrate their tests according to the testees
actual performance and needs.
The last Italian test to appear on the market was developed by the Dante Alighieri Society,
an officially recognized non-profit organization founded in 1889 which aims at promoting Italian
language and culture around the world; their language certificate was conceived within their project
called “PLIDA” (Progetto Lingua Italiana Dante Alighieri - “Dante Alighieri Italian Language
Project”) and was ratified by the Italian Ministry of Foreign Affairs, starting examinations from
1999. Not being an academic institution, the Dante Alighieri Society created a scientific committee
which would supervise every aspect of the tests, including design, creation, validation, marking etc.
All Italian language tests did undergo some kind of modification since the middle of the
nineties, in a moment when the six CEFR levels were taking shape by means of the research carried
out within the Council of Europe: for example, the CILS, which had started as a four-level
certificate, was finally structured in six levels in 1998; the CELI also was gradually changed to six
levels after the middle of the nineties; the IT, which had originally started as a two-level certificate
(as we said before, the theoretical bases were different from the Universities for Foreigners), was
finally changed into a four-level structure which included A2, B1, B2 and C2 levels. The PLIDA,
being the newest certificate, followed the six CEFR levels from the beginning.
At the beginning of the new century, two years after the publication of the CEFR by the
Council of Europe in 2001, the Manual for Relating Language Examinations to the Common
European Framework of Reference for Languages was made available to the public; subsequently,
many language certificates around the world were linked to the 6 CEFR levels and in Italy this work
was mainly carried out in the Universities for Foreigners of Siena and Perugia, respectively
members of EALTA and ALTE3.
3 See Grego Bolli (2008) and Barni et al. (2010) for specific reports on linking the CELI and the CILS to the CEFR.
6
Presently, the two Italian language tests with the highest number of candidates are the CELI
and the CILS (summing up to more than 150,000 candidates until 20094); except for the IT, which
is held annually, all Italian certification exams are held twice a year.
3.1 Characteristics of the testees.
Eight of our students volunteered to take part in this project, all sharing the same basic study
background: Chinese mother tongue speakers, majoring in Italian, attending the third year of studies
(therefore, with approximately 2 and a half years of Italian study experience, that is a little more
than 5 semesters), university-aged (20-23), with university-level English proficiency5. The only
discriminating factor was that 5 of them had spent one semester as exchange students in Italy, while
the rest had always studied in Taiwan.
Before starting the tests, students were asked to fill in a questionnaire regarding their self-
perceived language ability (questions and results can be found in Appendix 1). For every self-
assessed ability, the first four questions correspond to descriptors for A2 level, while questions 5 to
8 are descriptors for B1 level, nevertheless students weren’t informed of this and were just asked to
indicate the extent to which they agreed with each statement.
Averagely, students tended to agree more with statements corresponding to the A2 level,
while some of the B1 descriptors yielded inconsistent answers: for example, in the listening section,
while all students agreed on being able to “understand phrases and expressions related to areas of
most immediate priority provided speech is clearly and slowly articulated”, they showed less
confidence in being able to “follow the main points of extended discussion […] provided speech is
clearly articulated in standard dialect” (3 students disagreeing). For what concerns reading, more
uncertainty prevailed for abilities related to the B1 level: for instance, while almost 6 students
strongly agreed with being able to “understand short, simple texts as long as they are on familiar
4 See Vedovelli, M. et al. (on print), p. 28-40.
5 Students weren’t asked to provide any proof of English language proficiency; anyway, since they were all Chinese
speakers who grew in Taiwan, it was assumed that they all had non-mother tongue English ability derived by their
compulsory education (about 12 years of study from elementary school to university).
7
matters of a concrete type which consist of high frequency vocabulary”, most of them didn’t feel
able to “identify the main conclusions in clearly signaled argumentative texts and […] recognize
significant points in straightforward newspaper articles on familiar subjects” (5 students
disagreeing).
Nevertheless, as we will see, the students’ self-perception of their own listening ability is
somewhat different from the skills required to perform well in language proficiency tests: even
though in class they are usually exposed to standard and controlled everyday conversation, most of
the listening tasks for the B1 level are adapted from radio programs and thus are much more
difficult to understand, actually requiring specific preparation in order to be fully accomplished.
3.2 Characteristics of the four tests.
We will now introduce the basic structure of the four tests’ Listening and Reading tasks for
B1 Level; the presentation will also include the specific texts and topics pertaining to the test
sessions that were chosen6.
Table 1 - Basic Characteristics - Listening
Test Tasks # Items # Total time Max score Passing score
CILS 3 23 30 min 20 11
PLIDA 2 22 20 min 30 18
IT 4 20 30 min 20 12 CELI 4 33 20 min 40 24
Table 2 - Basic Characteristics - Reading
Test Tasks # Items # Total time Max score Passing score
CILS 3 32 50 min 20 11
PLIDA 2 22 30 min 30 18
IT 5 21 60 min* 25 15
CELI 5 37 60 min* 40 24
* The actual time assigned is 2 hours in total for reading and writing.
6 Specifically: CILS June 2009, PLIDA May 2009, It 2011 and CELI June 2007. These tests, at the time, were all
available on the Internet as free examples for download.
8
As we can see from the tables, the PLIDA stands out immediately not only because of its
lower overall time, but also because of the generally low number of tasks: a low number of texts
will allow the testee to concentrate on less inputs, and at the same time a limited number of tasks
means more time to complete each item; on the other hand, the CELI has the highest number of
tasks and items, requiring much more concentration and efficiency on the students’ side, and we
may infer that it is the most anxiogenous among the four tests.
One interesting thing to notice is that the CILS is the only certification with a passing mark
of 55%, slightly below the passing mark of other proficiency exams7; as we will see in section 4.2,
2 of the students who reached the passing level on the CILS had barely obtained a 55%, which
would have been considered a failing score in another exam.
Table 3- Characteristics of the tasks - Listening
Test Type of tasks Length of texts Topics Type of text
CILS Dictation; multiple choice (1 out of 4); information retrieval
dictation: 60 words; listening: 2:00; 2:20
photography exhibition; plans for future studies; tourism (architecture exhibition)
radio news; conversation
PLIDA Multiple choice (1 out of 3); information retrieval
3:30; 3:00 literature; wine fair radio interview; radio news
IT Multiple choice (1 out of 3); multiple choice (1 out of 2); True or False; sentences completion
3:50; 5 sec x 6; 1:10; 1:10
advice for travelers; ecology (shopping bags); sales season
radio interview; radio programs (beginning and ending); radio news
CELI Multiple choice (1 out of 3), multiple-text; multiple choice (1 out of 3), multiple-text; sentences recognition; information retrieval
25-30 sec x 4; 30-35 sec x 4; 3:00; 2:00
daily life, shopping, health etc.; housing and social trends; perfumes, culture
radio news; radio interview; radio commercial
7 As pointed out in Novello (2009) at p. 137, passing scores for different language proficiency tests in Europe vary a lot,
from 50% in CIEP’s exams (CIEP stands for “Centre International d’Etudes Pédagogiques” or International Center for
Pedagogical Studies, responsible for the assessment of French) to 70% in the Dele (Diploma of Spanish as a Foreign
Language). Also, differently from all other tests in Europe, each section of the Italian language proficiency exams
(reading, listening, writing etc.) must be over the passing threshold in order to obtain a Certificate.
9
Table 4- Characteristics of the tasks - Reading
Test Type of tasks Length of texts Topics Type of text
CILS Multiple choice questions (1 out of 4); information retrieval; text reconstruction
400-450 words x 2; around 250 words for scrambled text
Young Italian writers; special rates for electricity bills; vacation on the Alps
Informative article; fiction/diary
PLIDA Information retrieval; multiple choice cloze (1 out of 4)
300 words, 230 words
Excerpt from a novel; elementary school regulations
fiction; instructions
IT Multiple choice questions (1 out of 3); word retrieval (synonyms); True or false; referents retrieval; linking sentences and texts
300 words x 2, 500 words
New train service; online shopping; literature
newspaper article; book review
CELI Multiple choice questions (1 out of 3); information retrieval; multiple choice gap-filling (1 out of 4); multiple choice cloze (1 out of 3); free gap-filling (pronouns)
660 words, 215 words, 60 words (5 sentences), 140 words, 60 words (5 sentences)
Daily life Newspaper article; magazine article
When looking at tables 3 and 4, it is evident that multiple choice and information retrieval8
exercises are usually present in all tests; anyhow, it’s not difficult to notice that every test is
somewhat different from the others. For listening, the CILS is the only test including a dictation
exercise, while the IT contains a completion exercise where the testees need to fill sentences with
information according to what they hear on the recording; the CELI is the only test with a sentence
recognition exercise, where the candidates can listen to a text only once and must choose which of
the written sentences are present in the recording. Topics all conform to the B1 level and text types
are mostly excerpts from radio programs; texts are simplified, professionally recorded and read in a
clear and slow manner, except for the IT test, whose recording quality was not as good as the others
and where only authentic recordings were used. Texts are always played twice (except for the
aforementioned sentence recognition task in the CELI exam).
8 Information retrieval exercises are similar to True or False, but the candidate is asked to read a list of sentences and
decide which of them contains information that is present in the article. The candidate may or may not be informed on
the number of sentences with correct information.
10
As regards reading assessment, multiple choice and information retrieval exercises are still
widely adopted; specific situations are as follows: the CILS is the only test with a text
reconstruction exercise, while the CELI is the only one to have gap-filling exercises; the IT also has
some specificity, including synonyms retrieval, referents9
retrieval and linking texts with
corresponding sentences. Topics all conform to the B1 level and text types are usually adapted
excerpts from newspaper articles or fiction.
Once again, when compared to the others tests, the PLIDA seems to be the easiest to deal
with, including only the most common types of exercises, a low number of texts and very common
topics related to daily life.
One other differentiating aspect among the tests is in how students are informed about the
way their exams will be scored: for example, both the CILS and the CELI’s scoring methods are
never explicitly stated within the exam sheet and even though negative scores are assigned for
wrong answers, this information is only provided when downloading sample tests with keys from
the Internet; also, total scores for each task aren’t written on the exam sheet, so the students aren’t
aware of which exercises are more “valuable” to the tester. On the other hand, the PLIDA exam
sheet provides the tasks’ total scores and grading method10
, while the IT is the simplest in that no
negative scores are given for wrong answers and total scores for each task are written at the
beginning of the exam paper.
4.1 Method of administration.
After selecting the students to be tested, who all participated voluntarily in the experiment,
they were informed that four test sessions would be organized, corresponding to the four officially
recognized Italian proficiency exams, and that they would be tested in their listening and reading
ability for the B1 level of the CEFR. Students weren’t informed about the structure of the tests
9 In this case, “referents” indicates those words which are referred to, within the text, by means of pronouns and other
elements that the testees have to recognize. 10
It even tells the students how many answers they should provide in the information retrieval exercise, warning them
that every exceeding answer will have a negative score; amusingly, one of our examinees failed to pass the PLIDA
reading test because of forgetting to carefully read the instructions, thus choosing an excessive number of answers.
11
themselves and were asked not to undergo specific training for any of the sessions; from the point
of view of the students’ previous knowledge, the only discriminating element could be that they
were previously tested on a CILS-like simulation for the A2 level in June 2011, but still we don’t
believe that this had a significant influence on their performance because language levels were
different and some of the students were studying in Italy at that time, so they didn’t take the
simulation test.
The four tests were taken between March 21st and March 30
th 2012, with a time span of 2-5
days between each session. At the beginning, students were briefly informed about the sequence of
the exercises and then given the exam sheets, where they had to write their examinee progressive
number (1 to 8); subsequently, they were first tested in listening (by means of an MP3 file played
through a desktop computer with professional speakers installed in the classroom) and then in
reading.
After each session, students were asked to fill in a questionnaire regarding their perceived
difficulty and validity of the test. Finally, exam papers were anonymously graded11
and
questionnaire results were calculated.
4.2 Analysis of post-test questionnaire results.
In this paragraph, we will analyze the results of questionnaires given to the students after
each session. Specific questions and detailed results can be found in Appendix 2.
The first question was about the intelligibility of written instructions within the tests, i.e.
what students were asked to do in the exam papers. Most tests scored positive opinions, with the
only exception of the IT, whose listening test created some confusion both because of the aforesaid
recording quality and also because the second exercise (see p. 8) had students listen to excerpts
from radio programs and choose if they were at the beginning or at the end of them12
.
11
CILS, CELI and PLIDA all provide exercise keys along with their test samples, while IT didn’t provide any. 12
During informal conversations after the test, the students admitted to choosing answers randomly in that exercise
because they hadn’t understood what they were required to do.
12
Regarding the level of familiarity towards the topics (questions 2), most answers fall in the
categories of little or no acquaintance, with only the PLIDA and the CELI scoring 3 completely
positive answers out of 8; among the tests, the most unfamiliar was the IT’s listening, with half of
the students stating that they weren’t acquainted at all with the subjects. Question 4, which is
slightly related to the second one, asked about the tests’ similarity with our language courses
content, and here answers generally show more positive results, with only the IT being the
exception. This shows that, even though we usually talk about topics related to all aspects of Italian
daily life, students still show some level of unfamiliarity towards them (see next session for
possible explanations for this).
Questions 3 and 5 were more related to the students’ perceived difficulty of the tests. Here,
it is evident the majority feels comfortable with the time allotted for completing all exercises, while
also most of the testees perceived a moderate-to-high level of difficulty for the CILS, the PLIDA
and the IT, the only exception being the CELI, where most students ranked it in the moderate-to-
low difficulty range13
.
Finally, the last question was investigating on the examinees’ perceived appropriateness of
the tests in measuring their ability, i.e. its “face validity”14
; the highest level of agreement was
reached with the CILS’s listening, the PLIDA’s listening and the IT’s reading exams while some
disagreeing answers (2 out of 8) were scored only for the IT’s listening (probably for the reasons
stated above) and the PLIDA’s reading.
13
When comparing this with the tests results in the next section, we’ll see that there was a great deal of
misunderstanding on the students’ side regarding the complexity of the CELI: indeed, even though its perceived
difficulty was rather low when compared to other tests, results show that the students’ average performance in the CELI
was inferior to all other tests. Our assumption, not yet verified, is that, even though the CELI’s listening and reading
texts seem to be simpler, their exercises are more difficult to complete, including misleading distractors in multiple
choice exercises, subtler meanings in the information retrieval sentences etc. 14
Much has been said over the concept of face validity: cf. Bachman (1990), Alderson et al (1995), Brown (2003) etc.
While face validity may not have scientific bases, as it is purely an opinion by non-experts, our interest was just to
understand if and how the students perceive the tests’ as efficient in measuring their language ability.
13
4.3 Analysis of the tests results.
Test results are shown in Table 3, while in Table 4 we can see a comparison with our
Department’s average scores (along with criteria adopted for their calculation).
Table 3 - Test results15
Testee # Listening Reading Listening
Average
Reading
Average
Proficiency tests
Average CILS PLIDA IT CELI CILS PLIDA IT CELI
1 73 60 60 59 37 80 56 52 63 56 59
2 60 80 55 22 43 73 68 45 54 57 56
3 25 67 30 41 46 53 80 36 41 54 47
4 58 80 40 46 42 67 72 50 56 58 57
5 80 40 50 54 47 73 68 82 56 68 62
6 58 73 30 39 51 80 64 52 50 62 56
7 63 60 40 37 30 67 52 50 50 50 50
8 60 80 55 63 66 80 80 48 65 68 66
Average 59 68 45 45 45 72 68 52 54 59 57
Naturally, the first thing we notice is that Italian proficiency test results for the B1 level are
averagely under 60% and, even when we consider the average scores separately for listening and
reading, we can see that they are also under this threshold. Among the four proficiency tests,
students’ performed best in the PLIDA, where all the examinees managed to obtain passing scores
in listening and reading, with only one exception (testee nr. 3, who didn’t pass the reading test); on
the other hand, the hardest exam was the CELI, where only student n. 8 passed the listening test and
a different one, n. 5, managed to pass the reading test. In the middle, the CILS and the IT showed
controversial results: in the former, the listening test yielded better results (with only one person
failing and two students just above the passing score of 55%), while in the latter there was a better
outcome in the reading test, with only two students failing to pass16
.
When calculating average scores for all of the four proficiency tests, only two students
obtained a passing percentage, with only one of them (n. 8) managing to reach a passing average
both for listening and reading.
15
The scores have all been percentaged for the sake of comparison; marks in bold characters are over the passing score. 16
Nevertheless, if we examine the structure of the IT exam, we can see that it includes more grammatical-oriented
exercises (for example: synonyms, referents retrieval), where our students usually perform better, and also one
True/False exercise where the chance of correct guessing was 50%.
14
Table 4 - Comparison between test results and Fu Jen grades17
Testee # 1st year average 2
nd year average
3rd
year - 1st
semester average
Fu Jen average (1st to
3rd
year)
Proficiency tests
average
1 90 83 85 87 59
2 84 72 78 79 56
3 78 71 21 65 47
4 79 78 83 79 57
5 83 83 81 83 62
6 74 74 75 74 56
7 79 77 81 78 50
8 82 81 77 81 66
If we compare these results with scores obtained in our Department, some interesting
considerations can be done. First of all, average Fu Jen scores are higher than proficiency exams;
secondly, some of the students who delivered an unsatisfactory performance in the proficiency tests
are actually ranked in our top 25%, confirming that the skills they are being trained to develop may
not correspond to what these tests are requiring them to do; thirdly, Fu Jen rankings and proficiency
tests’ rankings are also different: for example, the first three in the Fu Jen average are number 1, 5
and 8, while the first three in the other category are the same people but in a different order: 8, 5
and 1. This means that not only required skills vary, but also the importance attached to them is
different (for example, in proficiency tests grasping the general idea of a text may have more
importance if compared to what we require in our exams).
The reasons for this discrepancy are manifold, and we will just try to list some of them
based on our teaching experience in the last 12 years:
1. First of all, our Department aims at bringing students to a B1-B2 level by the end of their
third year, and this means that right in the middle of it, not all students may be able to fully reach a
B1 level, as we can see in their self-assessment questionnaire presented on page 6. Also, there is a
higher correlation between our courses’ content and our exams, as we usually make adjustments
17
We took into consideration only scores related to our compulsory language courses: Italian Grammar, Italian Reading
& Composition, Italian Conversation and Italian Pronunciation and Phonetics; whenever these scores had been obtained
from Italian universities during exchange programs, they were treated as Fu Jen scores.
15
according to the students’ needs and interests, while international proficiency tests are calibrated
with a general “foreign/international examinee” profile in mind;
2. No matter which ability we take into consideration, our students are generally unfamiliar
with the topics dealt with in the proficiency tests (see questionnaire analysis on page 11); also,
textbooks we use in class may or may not deal with these subjects, and even when they do, some of
the students find it hard to feel concerned or interested about them because of the different cultural
background;
3. In our Department, and generally in Taiwan and most of East Asia, language training
techniques often differ from Western teaching styles: in order to understand foreign language
content, students rely a lot on written texts, show a great deal of attention for grammatical rules,
have a tendency to look up every word on the dictionary, and are frequently unable to synthesize or
grasp the gist of a long language input. Unless we insist on a change of perspective and learning
style, it’s not hard to anticipate that they will mostly be at a disadvantage when compared to
Western students18
;
4. As observed by Bachman19
, among the elements that influence test scores, there is always
a percentage of random factors. In this case, it may stand to reason that students were influenced by
some psychological factors like: fatigue or stress derived by school assignments, anxiety and
insecurity (i.e. the inability to listen to prolonged recorded content or a tendency to “give up” when
not being able to understand etc.), but also being aware that they were participating in an
experiment not having any concrete repercussion on their school scores or their personal career,
which may have led to random guessing on their side.
5. Conclusions.
18
For a deeper understanding of Taiwanese students’ learning styles and motivation (particularly in relation learning
Italian), see Nati (2006). 19
Bachman (1990), p. 164.
16
At the beginning of this article, we asked ourselves if our students’ language proficiency
would be up to our expected standards, but also if and how different testing styles for the same
language level could lead to different performances.
We’re afraid that the answer to the first question will be more on the negative side: it is
undeniable that our students struggled with language certifications’ Listening and Reading for the
B1 level, and even without testing them on the other abilities (like writing, speaking etc.), we can be
fairly sure that most of them wouldn’t obtain their certification on their first attempt. Of course, the
students’ performance on a certification exam cannot be considered a complete and accurate
evaluation of their language proficiency in a real-life situation; nevertheless, as we previously stated,
professional certificates (not only for what concerns foreign languages, but also in many other fields,
for example in medicine, law, accounting, music, transports etc.) do play an important role in
society, and being able to obtain one of them is undoubtedly an important step in a university
student’s career, not to mention how this could improve self-confidence and motivation for study.
As to the second question, the experiment seemed to provide a positive answer: passing
results were uniformly different from one test to another, with the PLIDA standing out as the most
“accessible” and the CELI being the hardest. From the users’ perspective, this could just mean that
choosing an “easier” certification would allow them to obtain their certificate with less effort; from
our perspective as teachers, though, this could also mean that the PLIDA’s approach and
interpretation of the CEFR levels could be more suitable for our students. Indeed, this is confirmed
by Vedovelli, stating that those who study Italian in the world are extremely diversified not only in
their geographical distribution, but also in their motivation; thus, “the plurality of the certification
authorities represents a valid strategy to create broader opportunities for foreigners to get in contact
with our language and culture”20
. Because every test is designed for a different kind of user, the
most logical choice for the examinees is to choose those proficiency tests that can be more suitable
to measure their language ability.
20
Vedovelli (in print), p. 20-21.
17
In conclusion, even though Certification exams should not be the only basis for evaluating
our students’ Italian proficiency, they can provide an interesting perspective on what they are able
to do with their foreign language: on one hand, it is evident that linguistic ability can’t be
disconnected from the knowledge of the country where it is used, therefore students should be
reminded that the Italian language is strictly related to its culture even when it is studied abroad,
outside its original context; on the other hand, since most of our students almost never end up
relocating to Italy, it would be advisable to adopt, or maybe even develop locally, language
proficiency tests that do not attach excessive importance to practical aspects of Italian everyday life,
incorporating cultural elements in a more traditional sense.
6. Bibliography
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press
Alderson J.C., Clapham C. & Wall D. (1995). Language Test Construction and Evaluation.
Cambridge: Cambridge University Press
Bachman, L. (1990), Fundamental considerations in language testing, Oxford: Oxford
University Press
Bachman, L.F. and Palmer, A.S. (1996), Language testing in practice. Oxford: Oxford
University Press.
Barni M., Scaglioso A.M, Machetti S. (2010), “Linking the CILS examinations to the CEFR:
the A1 speaking test”. In W. Martyniuk (ed.), Aligning Tests with the CEFR. Reflections on using
the Council of Europe’s draft manual, Cambridge: Cambridge University Press, 159-176.
Brown, H. D. (2003), Language Assessment Principles and Classroom Practices, San
Francisco: Longman
Buck, G. (2001). Assessing listening. Cambridge: Cambridge University
Figueras, N. et al (2005), Relating Examinations to the Common European Framework: a
Manual, Language Testing 22 (3), 1–19
Grego Bolli, G. (2008), “Progetti europei: nuove prospettive sulla scia del Quadro Comune
Europeo di Riferimento”. In Ciliberti, A. (ed.) Un mondo di italiano. Perugia: Guerra Edizioni, 25-
48.
Nati, F. (2006). “Riflessioni sull’uso di metodi comunicativi per apprendenti di italiano a
Taiwan”. Article presented at “The 3rd Cross-Strait Conference on Foreign Language Teaching &
18
Learning”, Fu Jen Catholic University, Taipei, Taiwan, 2 November 2006. Can be downloaded
from: www.svd.fju.edu.tw/fl/lang_teaching/papers/ita_3.pdf
Novello Alberta (2009), Valutare una lingua straniera: le certificazioni europee, Venezia:
Cafoscarina
Tulli, A. (2011), The Certification of Italian as a Foreign Language (CILS) in Taiwan and
its distribution in other Asian countries, presentation held at the Second International Symposium
on European Languages in East Asia, Taipei, Taiwan, 5-6 November 2011
Vedovelli M., Barni M., Bagna C., Machetti S. (in print), Vademecum. Le certificazioni di
competenza linguistica in Italiano come lingua straniera, Roma, FAPI, retrieved on May 11th at:
http://www.aislo.it/ImagePub.aspx?id=54704
19
Appendix 1 - Self-evaluation questionnaire on Italian proficiency (original questions were in
Chinese)
Please read the statements and indicate the extent to which you agree with each of them (4 strongly agree 1
strongly disagree).
Listening ability
4 3 2 1
1. I can understand phrases and expressions related to areas of most immediate priority (e.g. very
basic personal and family information, shopping, local geography, employment) provided speech
is clearly and slowly articulated.
5 3 0 0
2. I can generally identify the topic of discussion around me that is conducted slowly and clearly. 5 3 0 0
3. I can catch the main point in short, clear, simple messages and announcement.
I can understand simple directions relating to how to get from X to Y, by foot or public transport.
4 3 1 0
4. I can understand and extract the essential information from short recorded passages dealing
with predictable everyday matters that are delivered slowly and clearly.
4 2 2 0
5. I can understand the main points of clear standard input on familiar matters. 5 2 1 0
6. I can generally follow the main points of extended discussion around me, provided speech is
clearly articulated in standard dialect.
1 4 3 0
7. I can understand simple technical information, such as operating instructions for everyday
equipment and I can follow detailed directions.
2 4 1 1
8. I can understand slow and clear radio news bulletins and recorded material, as long as it is
about familiar subjects or those related to my field of specialization.
3 4 1 0
Reading ability
4 3 2 1
1. I can understand short, simple texts as long as they are on familiar matters of a concrete type
which consist of high frequency vocabulary.
6 2 0 0
2. I can understand short simple personal letters and recognize basic types of standard routine
letters (enquiries, orders, letters of confirmation etc.) on familiar topics.
3 4 1 0
3. I can find specific, predictable information in materials such as advertisements, prospectuses,
menus, reference lists and timetables, and I can understand everyday signs and notices such as
directions, safety warnings etc.
4 1 2 1
4. I can identify specific information in simpler written material such as letters, brochures and
short newspaper articles describing events.
3 3 2 0
5. I can read straightforward factual texts on subjects related to my field and interest with a
satisfactory level of comprehension.
1 5 2 0
6. I can understand the description of events, feelings and wishes in personal letters. 3 4 1 0
7. I can scan longer texts in order to locate desired information and I can find and understand
relevant information in everyday material.
0 5 3 0
8. I can identify the main conclusions in clearly signaled argumentative texts and I can recognize
significant points in straightforward newspaper articles on familiar subjects.
0 3 5 0
20
Appendix 2 - Perceived validity of each proficiency test (original questions were in Chinese)
1. I think that the instructions in
the test are:
a. clear b. a little unclear c. not clear
2. I am… with the topics: a. acquainted b. a little acquainted c. not acquainted
3. I think that the time given for
answering is:
a. too much b. just enough c. not enough
4. This test is… what I learned in
class:
a. similar to b. a little different
from
c. very different from
5. The overall difficulty of this
test was:
a. high b. moderate c. low
6. This test can measure my real
reading/listening ability:
a. I agree b. I have no opinion c. I disagree
Questionnaire results:
Question
Test 1 2 3 4 5 6
a b c a b c a b c a b c a b c a b c
CILS - listening 7 0 1 2 6 0 0 7 1 4 4 0 2 6 0 4 4 0
CILS - reading 8 0 0 0 5 3 1 3 4 3 4 1 7 1 0 3 5 0
PLIDA - listening 8 0 0 3 5 0 0 8 0 5 3 0 0 7 1 4 4 0
PLIDA - reading 8 0 0 3 4 1 0 7 1 3 4 1 3 5 0 3 3 2
IT - listening 0 4 4 1 3 4 0 8 0 1 6 1 8 0 0 3 3 2
IT - reading 6 2 0 1 5 2 2 6 0 3 5 0 2 6 0 4 4 0
CELI - listening 6 2 0 3 5 0 0 7 1 3 5 0 1 5 2 2 6 0
CELI - reading 7 1 0 5 3 0 1 7 0 5 3 0 0 4 4 3 5 0