the use of facet theory for the identification of scholastic competences - roazzi et al. 2006

8/8/2019 The Use of Facet Theory For The Identification of Scholastic Competences - Roazzi et al. 2006

1/8

I S B N 80 - 8 6742 - 0 9 - 1

Facet Theory:Design, Analysis and Applications

Edited by

Wolfgang Bilsky an d Dov ElizurWestfalische Wilhelms-Universitat Ba r Ilan UniversityMiinster and the Lander Institute

Roma 2005


2/8

.................................................................................................................................reface 1Chapter 1. Psychological AssessmentSmallest Space Analysis of the W echsler Intelligence Scale for Children - V............................................................................ Cohen, C.A. Fiorello, F.H. Farley 3Validation of Rorschach Scoring System through Smallest Space Analysis............................................................................................................................. Cohen 1Chapter 2. FT and Evaluation ResearchFaceting Program Evaluation: A Systematic Approach......................................................................................................... Bilsky, D. Cairns 19Program Evaluation: A Facet-Theoretic Approach............................................................................................................................ Doring 27The Use of Facet Theory for the Identification of Scholastic CompetencesA. Roa zzi, B. C. de Souza, K Bilsky, M. B. B. D ias ..................................................9Chapter 3. Social Issues - Social Representation, Attitudes, StereotypesModeling Social Representations of European N ations and European Union:A Facet Theory A pproach ...................................................................S. DeRosa, M D'Ambrosio, E. Cohen 49Exploring the Structure of Social Representations of the Euro via Facet TheoryApproach and WSSAl Procedure .................................................................. Gioiosa, A.S. DeRosa, M. D'Ambrosio 57

, National Stereotypes of Slovenians and Italians by Slov ene Students ofPsychology .............................................KS. Rus, P. Meneghin, L. Horvat, M. Rus-Makovec 65The Underlying Dim ensionality of the Survey of Cultural Attitudes andBehaviorsS. M. Miller, A. Cohen, K.L. Miller ............................................................................ 75Questionnaire Design and Data Analysis using the Facet Approach: Examplesfrom the International Census on Attitudes toward Lang uages

.......................................................................................................................... Manabe 8I


3/8

The Use of Facet Theory for the Identification of Scholastic Competences 39

The Use of Facet Th eory for the Identification of ScholasticCompetencesA n to n io ~ o a z z i ' , r u n o C. d e ~ o u z a ' ,W o lfg a n g ~ i l s k ~ ~ ,

M ar ia d a G r a ~ a .B . ~ i a s ''~n iver sida de ederal de Pemambuco, Brazil'~es t fa l i scheWilhelms-Universitat Miinster, Germany

Abstract. The National Examination of Interm ediate Level Education(ENEM) in Brazil aims to evaluate five basic intellectual competences bymeans of the assessment of 21 abilities. The underlying idea was that thecompetences correspond to different combinations of abilities. However,such competences were originally defined with a great deal of overlap.Therefore, an SSA was done on actual data from the ENEM 2000 in orderto identify the basic organization of the abilities without overlapping. Anaxial structure with three distinct groups of abilities emerged from the re-sults, each corresponding to a clearly defined com petence. Performance in-dicators for these three competences were then computed, as well as theirassociation to soc io-demographic variables. In the end, imp lications of thefindings are discussed.

1. IntroductionThe National Examination of Intermediate Level Education (ENEM: Exame Na-cional do Ensino Mkdio; Brazil, 2005) was created by the Brazilian Ministry ofEducation aiming to evaluate five basic cornpetences: Mastery of language, Un-derstanding of phenomena, Facing of problem situations, Construction of argu-ments, and Elaboration o f proposals. For this purpose, 2 1 abilities were de fined,each one measured through three questions with increasing levels of difficulty,totalizing 63 items. The basic idea underlying the test is that different combina-tions of abilities correspond to each one of the competences being evaluated. Itoccurs, however, that the combinations defined for each competence presenta great amount of overlap , as each individual a bility is used for the e valuation ofthree to five distinct competences. This makes it difficult or even impossible toclearly differentiate between these compete nces.The purpose of this article is to make use of Facet Theory to better under-stand the identification of school cornpetences using the National Examinationof Intermediate Level Education (ENEM). This exam was established by the Na-tional Institute of Studies and Educational Researches - INEP, in 1998, to begiven to end-of-term students and those with equivalent education. The ENEMis done annually, with the main purpose of evaluating students' performance at


4/8

40 Antonlo Roazzr, Bruno C de Souza, Wolfgang B~lskyMana da Graqa B.B Dlas The Use of Facet Theory for the Ident~fic at~o nf Schola st~c ornpetences 41the end of basic education, to confront the development of fundamental compe-tences with what is considered necessary to fulfill their citizenship. The exam is

B[Ol. Ability 1 ]sponding to the end of basic education. [02. Ability 2 ]order to clearly d efine its assumptions and sketch its o perational characteristics. [ ........... ... ...... 1This model contemplates the definition of competences and of the students' [21.Abili ty21 ]general abilities during the cognitive development period at the end of basiceducation, considering it associated with the contents of fundamental and me- Cdium leveled teaching. Within this context, competences are structural modali-

of "knowing how to". Fig. 1. Mapping Sentence of the ENEM Competences ModelI More specifically, the Competences model developed to structure ENEMI aims to evaluate five basic competences: 1. Mastery of langu age; 2. Understand-I ing of phenomena; 3. Facing of problem situations; 4. Construction of argu- In Table I a matrix is shown indicating the number of similar abilitiesments; 5. Elaboration of proposals. Each competence would be related to a se- shared between the five competences and pointing o ut a quite high overlappingI ries of abilit~ es 21), each of them belng measured three times through objective for each comparison. In the end, these considerations are important in order toquestions evaluated as of high, average and low difficulty. Therefore, the ENE M &ow how the competences, by the way they were theoretically elaborated, aretest, from the cognitive perspective, results from an exam of competences and not independent of each other.1 abilities which shapes the contents translated into a series of itemslquestions Table 1. Numb er of abilities shared between the five Competences

nomena situations1

Level of Difficulty (3 levels).

1 the competences 3 and 4 by 14 and 15 abilities, respectively.Illi


5/8

42 Antonio Roazzi, Bruno C. de Souza, Wolfgang Bilsky, Maria da G raqa B.B. Dias

This exam already resulted in several data sets since 1998. Taking thisdata basis into consideration, it is possible to accomplish an empiric verificationof the theoretical model of competences and abilities in agreement with facettheory.In the present article, the model of Competences will be verified from thedata basis of the year 2000, composed of 352.487 participants. This correspondsto 23% of the end-of-term students from high school - f we consider the schoolcensus of 1999 (N = 1.535.943). From the results of these analyses, to be doneusing the Statsoft STATISTICA 6.0 statistical package, it will be possible toidentify the natural groupings of the 2 1 abilities without superposition or ambi-guities and so establish underlying structural properties of facet(s) that representthe universe of the information that compose ENEM problems. The establish-ment of these facet(s) and its relationships will make it possible to supply sup-port for future and probable problems elaboration and questions on EN EM tests.2. MethodThe sample of this study comprises the 352.487 participants of the ENEM 2000(59.1% girls and 40.9% boys). The frequency distribution for the year of birth ispresented below in Figure 2. More than half of the participants (64.8%) wereborn in 1983 or later, indicating a majo rity of individuals aged 17 or less.The frequency distribution for the monthly family income is presented inFigure 3, below. Based on the mid-points of the seven intervals of incomeconsidered in the form, and given a minimal wage value of R$ 15 1 OO at thetime, it is possible to estimate the average income o f the sample to be of roughlyR$ 2.487,24, which is equivalent to approximately US$ 1,100.00.About 42.7% of the participants went exclusively to 9Ih, loth and1 th in public schools, 49.6% exclusively in private schools, and 7.7% toboth public and private schools. Estimating again from the mid-points of theseven intervals of income considered in the form, the average income of thosewho studied in:

o Public schools alone was of roughly R$ 1.038,58;o Both in public and privates schools was o f roughly R$ 2.275,36;o Private schools alone was of roughly R$ 4.03 1,70.

Measuring income in the original 1-7 Likert scale used in the ENEM-form, allthe differences in income between the three types of schooling are statisticallysignificant (p


6/8

44 Anton10 Roazzl, Bruno C. de Souza, Wolfgang B ~lsk y,Mar la da Gny a B B. Dlas The Use o f Facet Theory for the Identrficatlon of S cholast ~c ornpetences 45

Considering the Language, Science and Geom etry elements as underlying3.1. SSA Classification of Abilities dimensions of the exam , it is possible to create indexes for them based on arith-

Spearman Rho). This particular metric was chosen due to the ordinal nature of

plot (Figure 4).Scatterplot 2D come, level of private schooling, no. of computers at home, access to com puters

Distance 1-Spearman Rho - Ward's Method - Alienation=.l2 and Stress=.lO at school, taking or not computer classes, night schooling), the following step-wise models were obtained (Table 2).Table 2. Logistic regression analysis of the performance on Science, Lan-guage and Geom etry items (50% right o r more vs. Less than 50%) as a func-tion of several variables.

Sens~ t~vlt y Spec~ficrty ch12 PNo of Computers at Home (0 39),

Science Income (0 21), Prrvate School~ng(0 18), Nlght Schoolrng (-0 37), 68% 64% 391 99 < 000001Computer Classes (-0 38)Income (0 37), No of Computersat Home (0 31), Prlvate SchoolingLanguage (0 I I), Computer Classes (-0 41), 94% 30% 386 79


7/8

46 Antonio Roazzi, Bruno C. de Souza, Wolfgang Bilsky, Maria da Graqa B.B. Dias

already binary), so as to establish a common scale, it is possible to do an SSA,associating the socio-cultural variables present in the official ENEM form withthe indexes created for each of three elements of the ability facet identified, so asto characterize the relationship between the three cognitive dimensions in ques-tion and also between them and the predictive social and cultural variables con- 'sidered in the logistic regressions mentioned above (see Figure 5, below).Ase

LANGUAGE

Computer at HomeIncomeSCIENCEGErn.ETRY

Computer Course

Computer at SchoolNight Schooling

Dimension 1Fig. 5. SSA of the socio-cultural variables and Language, Science and Geometry(Manhattan City-Block - Ward's Method - Stress = .07 and Alienation = .12 -2D Scalogram).A glance at the scalogram above clearly shows that:

o Science and Geometry are more closely associated to each other than toLanguage;o Science and Geometry are closely associated to income and to having ac-

cess to a computer at home, and, to a lesser degree, to private schoolingand having access to.computers at school;o Language is also positively associated to income and to having access toa computer at home and, to some extent, with age;o Science and Geometry show a distant association with the variables age,participation in computer courses and night schooling (negative correla-tions, as indicated by the coefficients from the logistic regressions sum-marized on Table 4);

I The Us e of Facet Theo ry for the Identification of Scholastic Competences 47Language seems to have a distant association with the variables privateschooling, having access to computers at school, participation in computercourses and night schooling (negative correlations, as indicated by thecoefficients from the logistic regressions summarized on Table 4).

The above results tend to further corroborate the notion that the three elementsof an ability-facet identified are indeed distinct dimensions each with their par-ticular behavior and associations.

/ 4. isc cuss ion and ConclusionThe SSA study of the multidimensional associations between the 21 abilities

1 measured by the ENEM 2000 showed the existence of an axial structure com-1 prised of three distinct elements (Language, Science and Geometry). A series ofir logistic regression analyses with indexes for each of the facets as dependent/ produced results indicating different dynamics and explanatory factorsfor each of them. Therefore, it is safe to assume that the relationship between the! 21 abilities can be expressed by a mapping sentence that classifies them into the! three distinct categories found.1 The three categories arising from the data analysis are in stark contrast tothe a priori five-fold classification suggested by the proponents of the exam. The.. categories that were actually found were mutually exclusive, while the fiveoriginal categories overlap in two thirds of the abilities or more. Also, while thefive-fold classification attempted to divide the abilities according to types ofreasoning, the actual three categories found classify them according to subjectmatter. The bulk of this evidence argues against the validity of the initial classi-fication.Income, private schooling, computers at home and access to computers atschool all showed a positive association with the performance on the exam. Age,participation in computer courses and night schooling all had a negative associa-tion.

ReferencesBrasil (2005). Exame Nacional do Ensino Mkdio - ENEM. Downloaded from

the Site of the Instituto Nacional de Estudos e Pesquisas Educacionais - TNEPhrrt,::i~~?~cv.ii~ep.ov.br:enei~~


8/8

) I I

I Appendix ITable A l . Abilities vs. Compctences (1 = evaluates, 0 does not evaluate)

I. Mastery of 11. Under- Ill. Facing IV. Con- V. Elabora-Abilities language standing of problem struction of tion of pro- Total. I- situations arguments posais01. Understanding of written descrip- I I I 0 0 3 - Ilons or lnstructlons02 lnterpretat~on f 1 I 1 0 0Cartesian plots03 A nalys~s nd ~nterpretat~onf stat~s- I 0 1 1 1 4 P h a n t ~ r .tlcal dlstr~butlons04. Association between the uses of 1 0 I I 0language in different fields of knowl- Idge05 lnterpretatlon of texts accord~ngo I 0 0 1 1sociocultural context06 Analvs~s f the funchons of the I 1 0 I 0 . . . r 1 _ - O A Avariations o f language in a text07. Characterization of energy conser- 0 1 1 0 I Attltuaes, arereocypesvationitransform ation and its uses08. Understand ing o f the impac ts o f 0 1 0 ' I I 3using material and energetic resources09. Understanding of the importance of 0 I 1 0 I 3water and its physical variations10. Using different time scales to de- 0 1 I 0 1 3scribe transformations in geographicspace11. Knowledge of the structures, pat- 1 1 0 0 1 3terns and processes of living organisms12. Knowledge of factors associated to I I I 0 I A-human development and qual~ty f life13 Understand~ng arth 's b~o-sys tem I 1 0 1 1 4 1and the impacts of human14. Application of planar and spatial 1 1 1 I 1 5geometry to real-life problems15. Knowledge of Probability and Sta- 0 I I 1 0 3tistics and their use to actual problems16. Analysis and interpretation of envi- 0 1 I 0 1 3ronmental problems and their solutions I17. Understanding how materials and 0 1 I 0 1 3energy are obtained and the implica-tions18. Understanding of variety and value 1 1 0 0 1 3of ethno-cultural assets19. Confrontation of the perspectives of 0 0 1 I 1 3history, science, art and commonsense

20. Understanding of socioeconomic 0 I 0 I 1 3processes and their contexts21. Understanding of history and geog- 0 I I I 0 3raphy alonn with their causal factors

Total I I 17 14 I I 15

the use of facet theory for the identification of scholastic competences - roazzi et al. 2006

Documents