preservice teachers' belief systems toward curricular outcomes for physical education

11
This article was downloaded by: [Northeastern University] On: 09 October 2014, At: 16:03 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Research Quarterly for Exercise and Sport Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/urqe20 Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education Pamela Hodges Kulinna a , Timothy Brusseau b , Matthew Ferry c & Donetta Cothran d a Division of Educational Leadership and Innovation , Arizona State University b Department of Kinesiology, Sport Studies, and Physical Education , State University of New York–College at Brockport c Department of Kinesiology, Health, and Sport Studies , Wayne State University d Department of Kinesiology , Indiana University Published online: 23 Jan 2013. To cite this article: Pamela Hodges Kulinna , Timothy Brusseau , Matthew Ferry & Donetta Cothran (2010) Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education, Research Quarterly for Exercise and Sport, 81:2, 189-198 To link to this article: http://dx.doi.org/10.1080/02701367.2010.10599666 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and- conditions

Upload: donetta

Post on 17-Feb-2017

216 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

This article was downloaded by: [Northeastern University]On: 09 October 2014, At: 16:03Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41Mortimer Street, London W1T 3JH, UK

Research Quarterly for Exercise and SportPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/urqe20

Preservice Teachers' Belief Systems Toward CurricularOutcomes for Physical EducationPamela Hodges Kulinna a , Timothy Brusseau b , Matthew Ferry c & Donetta Cothran da Division of Educational Leadership and Innovation , Arizona State Universityb Department of Kinesiology, Sport Studies, and Physical Education , State University of NewYork–College at Brockportc Department of Kinesiology, Health, and Sport Studies , Wayne State Universityd Department of Kinesiology , Indiana UniversityPublished online: 23 Jan 2013.

To cite this article: Pamela Hodges Kulinna , Timothy Brusseau , Matthew Ferry & Donetta Cothran (2010) Preservice Teachers' BeliefSystems Toward Curricular Outcomes for Physical Education, Research Quarterly for Exercise and Sport, 81:2, 189-198

To link to this article: http://dx.doi.org/10.1080/02701367.2010.10599666

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in thepublications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations orwarranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsedby Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified withprimary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings,demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectlyin connection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction,redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expresslyforbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Page 2: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

RQES: June 2010 189

Kulinna, Brusseau, Ferry, and Cothran

Key words: fitness, goals, physical activity

For traditional “core” subject matter areas, the text-books and standardized tests used by the school

district influence, or determine, class content. Although teachers in the core areas may have input into the course content, texts and tests often provide at least a strong outline of scope and sequence, as well as a detailed guide for teachers to follow. In contrast, teachers in the “special” content areas (e.g., art, music, physical educa-tion) often have much flexibility within national (e.g.,

National Association for Sport and Physical Education, 2004), state, and/or district standards to select activities and sequencing, because those classes rarely have stan-dardized texts or tests. In the absence of direct, external forces driving course content, these teachers often rely on their personal belief systems about the course, its goals, and the most appropriate content to achieve those goals when designing the curriculum.

This reliance on personal belief systems in education-al decision making is well documented in the literature (e.g., Lara-Cinisomo, Fuligni, Ritchie, Howes, & Karoly, 2008). A group of beliefs clustered around a situation or object becomes an attitude that is prone to action. When beliefs function to evaluate (or compare/judge) and call for action, they become values. Collectively, an individual’s beliefs, attitudes, and values form his or her belief system (Pajares, 1992). Teachers’ belief systems guide their behaviors and decisions (Pajares, 1992) and affect a myriad of teacher behaviors.

For example, teachers’ curricular beliefs influence their stated K–12 student learning goals and expectations for academic performance and behavior (Ennis, Ross, & Chen, 1992). Teaching beliefs are also deeply held

Preservice Teachers’ Belief Systems Toward Curricular Outcomes for Physical Education

Pamela Hodges Kulinna, Timothy Brusseau, Matthew Ferry, and Donetta Cothran

Submitted: October 30, 2007 Accepted: February 19, 2009 Pamela Hodges Kulinna is with the Division of Educational Leadership and Innovation at Arizona State University. Timothy Brusseau is with the Department of Kinesiology, Sport Studies, and Physical Education at The State Univer-sity of New York–College at Brockport. Matthew Ferry is with the Department of Kinesiology, Health, and Sport Studies at Wayne State University. Donetta Cothran is with the Depart-ment of Kinesiology at Indiana University.

This study was grounded in the belief systems and physical activity literature and investigated preservice teachers’ belief systems toward curricular outcomes for physical education programs. Preservice teachers (N = 486; men = 62%, women = 38%) from 18 U.S. colleges/universities shared their beliefs about curricular outcomes. Preservice teachers completed a previously validated belief systems instrument designed to measure the relative importance of four outcome goals for programs (physical activity/fitness, self-actualization, motor skill development, and social development). Internal consistency reliability for the instrument was .98. A confirmatory factor analysis demonstrated a good fit of the current sample to the hypothesized outcomes model. Multivariate analysis of variance results revealed a significant interaction in outcome preservice teachers’ priorities for year in school by region. The teachers’ views also differed on the important outcome goals for physical education. Two critical “tensions” are discussed: (a) the need to examine more fully the consistency of preservice teacher/teacher belief systems, and (b) implications for teacher education and professional development programming. It is important to heed prospective teachers’ voices and address their belief systems in teacher education programs.

Pedagogy

Research Quarterly for Exercise and Sport©2010 by the American Alliance for Health,Physical Education, Recreation and DanceVol. 81, No. 2, pp. 189–198

Kulinna.indd 189 5/4/2010 1:22:39 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 3: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

190 RQES: June 2010

Kulinna, Brusseau, Ferry, and Cothran

and resistant to change (Kagan, 1992). It is important to acknowledge this aspect of belief systems, as preservice teachers typically enter their programs with well formed pre-existing beliefs about educational practice.

Given that belief systems are often unexamined and difficult to change, it is critical that teacher education personnel know more about preservice teachers’ beliefs and how they may evolve over time. Previous studies with preservice teachers suggest that prior knowledge (Rovegno, 1992, 1993a), field experiences (Woods, Goc Karp, & Escamilla, 2000), and culturally based assump-tions regarding sport and physical education (Rovegno, 1993b) were influential filters as preservice teachers progressed through their teacher education program (Rovegno, 2003). This line of literature documents ex-periences conducive to preservice teachers restructuring their knowledge base and belief systems and having those changes translate to appropriate instructional practices. These experiences may include: (a) critiquing their own K–12 physical education experiences, (Rovegno 1993b), (b) teacher as researcher in early field experiences (Woods et al., 2000), (c) opportunities for reflection (Se-bren, 1995), and (d) learning to politically conceptualize their subject matter (Rovegno, 1993b).

Specific to physical education teachers’ curricular belief systems, Kulinna and Silverman (2000) assessed four important outcome priorities for teachers’ views of physical education. Using a Likert-type scale rather than the forced-choice format used in similar inquiries, the authors investigated the relative importance of outcome priorities to teachers. The four domain areas were: (a) physical activity leading to fitness, which concentrates on the importance of promoting physical activity par-ticipation leading to improved fitness and health; (b) self-actualization, with a focus on developing self-esteem, self-confidence, enjoyment, and self-efficacy for partici-pating in physical activities; (c) motor skill development that directs effort on acquiring prerequisite motor skills needed for successful participation in many activities and sports; and (d) social development centered on creating social skills and behaviors as well as an appreciation for and acceptance among K–12 students. Kulinna and Silver-man found that teachers reported physical activity leading to fitness as their top priority.

What is not known from the Kulinna and Silverman (2000) work, however, is how those belief systems devel-oped. For example, do preservice teachers begin with certain curricular belief systems and/or do they change over time? Early explorations into preservice teachers’ beliefs about the purposes of physical education found they viewed it as an area to learn to play and participate in sports, games, and fitness activities (e.g., Placek et al., 1995). Furthermore, those teachers held a custodial view (Lawson, 1988); that is, they were planning to preserve physical education as they had experienced it rather than

transform it. Placek et al. (1995) conducted a national study of U.S. preservice teachers’ (N = 476) belief systems about the purposes of physical education. Most reported K–12 students learning skills/activities as the top purpose. Similarly, in a review of preservice teachers’ belief systems O’Sullivan (2005) also indicated the dominant view was a skill-oriented perception of physical education.

Differences in Preservice Teachers’ Belief Systems

The term preservice teachers encompasses a broad spectrum of people, so belief variation within this group would be expected. Richardson (1996) wrote that enter-ing preservice teachers are not an undifferentiated group; as one example, she cited differences between traditional and nontraditional preservice teachers. Kile (1993) re-ported differences between traditional and nontraditional preservice teachers’ belief systems, with nontraditional preservice teachers (who were often older and had fami-lies) holding a more realistic view of teaching and learning complexities. Similarly, Morine-Dershimer (1989) found differences in planning and content between undergradu-ate seniors and graduate preservice teachers.

Differences in physical education preservice teach-ers’ belief systems were also noted in teachers across the years in education programs. Matanin and Collier (2003) reported on data collected across 5 years, suggesting that preservice teachers assimilated program messages into their belief systems about teaching physical education related to content, teaching effectiveness, and planning. Teaching behavior differences (i.e., skills in observing and interpreting instructional events) were also reported among novice preservice teachers, experienced preser-vice teachers, and expert teachers (Graham, French, & Woods, 1993).

In addition to potential program differences by year, we were interested in potential preservice teacher belief system differences by region. Grounded in the physical activity literature, adults in the western U.S. reported the highest prevalence of leisure-time physical activity, while the southern regions reported the lowest levels (Reis et al., 2004). U.S. high school students reported similar trends, with students from the west reporting the highest preva-lence of physical activity behavior and lowest sedentary be-havior, and the southern students reporting reverse trends (Springer, Hoelscher, & Kelder, 2006). It seems likely that these geographical differences might play out in different preservice teacher experiences and belief systems about physical activity. Work with inservice teachers’ curricular value orientations also suggested that context, which is at least somewhat related to geography, influenced teachers’ curricular priorities (Ennis & Chen, 1995).

Therefore, the purpose of this project was to exam-ine preservice teachers’ belief systems toward curricular outcomes for physical education programs. Additional

Kulinna.indd 190 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 4: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

RQES: June 2010 191

Kulinna, Brusseau, Ferry, and Cothran

research questions focused on whether there were differ-ences in their belief systems by year or U.S. region.

The results of this investigation should provide valu-able information on a number of fronts. Without know-ing preservice teachers’ belief systems, teacher education programs cannot be designed for maximum effectiveness. These programs may be designed to support and enhance a teacher’s ability to implement a program consistent with entry belief systems or may have specific interventions to encourage preservice teachers to change their belief systems. As an example, programs may attempt to cre-ate graduates with a strong commitment to and skills in implementing health-related physical activity curricula. Related to this, knowledge of the program’s effectiveness is needed; by tracking their preservice teachers’ belief systems throughout the program, teacher educators may evaluate program effectiveness. In essence, we sup-ported the call by O’Sullivan (2003, p. 288) for physical education scholars “to pay closer attention to what pre-service teachers actually know, can do and value about teaching…” However, little is known about preservice teachers’ beliefs about curricular issues and, specifically, about important physical education outcomes and how these might change over time. It is important to listen to prospective teachers and address their belief systems in teacher education programs in order to develop gradu-ates with sustainable teaching practices.

Method

Instrument

The instrument we used had been previously vali-dated for use with physical education teachers (Kulinna & Silverman, 1999); it contains 36 items, nine from each of four domains representing important outcomes for physi-cal education programs: (a) physical activity and fitness, (b) motor skill development, (c) self-actualization, and (d) social development. The instrument uses a Likert-type scale as the response format in which teachers selected the number that best represented their belief about each item, with 1 = very important to 5 = not important.

Previous Instrument Validation. The validation study for this instrument (see Kulinna & Silverman, 1999) included a content validity study with 28 experts and a reliability and validity study with 253 physical education teachers from 18 U.S. states. Results included a mean percent agreement among the experts on the 36 items’ classi-fications of 0.91. Cronbach’s alpha results also showed adequate levels of internal consistency reliability for the four subdomains and overall instrument, ranging from 0.81 to 0.89. The instrument demonstrated the psycho-metric properties of producing reliable and valid scores in a diverse teacher population.

Recruitment

We recruited preservice teachers from teacher edu-cation programs in four U.S. regions (west, northeast, midwest, and south; Grunbaum et al., 2004; Springer et al., 2006). We obtained University Human Subjects ap-proval. Faculty at those institutions asked all preservice teachers (who were identifiable by major) to participate. Preservice teachers provided informed consent electroni-cally or in writing prior to completing the belief systems instrument.

Phase 1. In Phase 1 of data collection, we contacted faculty from 13 physical education teacher education programs in all four U.S. regions via e-mail and telephone and asked if they would assist in this project; personnel from 10 institutions agreed to participate (77%). We sent paper surveys to nine of the participating schools; of the preservice teachers (N = 506) surveyed, 331 participated (65%). Personnel from the 10th institution asked for an electronic survey. An additional 45 preservice teach-ers completed the survey online (www.surveymonkey.com), which represented about 5% of the students in the program. Thus, a total of 376 preservice teachers participated in Phase 1.

Phase 2. In Phase 2, we randomly selected 16 physical education teacher education programs from the Direc-tory of Physical Education Teacher Education Programs (Ayers, Housner, & Kim; 2004), while controlling for re-gion using SPSS. From these, eight agreed to participate. Preservice teachers completed the instrument online using Survey Monkey, resulting in an additional 110 participants and a grand total of 486 preservice teachers from 18 programs.

Participants

Participants included men (n = 302) and women (n = 182), ages 17–52 years (M = 23 years, SD = 4.64; 2 did not report gender), from 18 U.S. colleges/universities repre-senting the four U.S. regions (see Table 1 for descriptive characteristics of participants by region).

Participants reported their ethnic background as Cau- casian (80.2%), Asian/Pacific Islander (6.4%), African American (5.8%), Hispanic (4.3%), American Indian/AK Native (1.2%), or other (2.1%). They also repre-sented all years in school including: (a) freshmen (9.3%), sophomores (7.6%), juniors (28.0%), seniors (41.6%), postbachelor (3.3%), and graduates seeking teacher cer-tification or additional endorsement (9.7%). Most of the preservice teachers were taking methods courses (64.2%), with others taking initial pedagogy courses (21.2%), student teaching (9.5%), or other (5.1%). Most were single, never married (82.7%), and few reported hav-ing children (17.5%; with 35 not reporting). Preservice teachers self-reported their G.P.A. on a 4.0 scale ranging

Kulinna.indd 191 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 5: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

192 RQES: June 2010

Kulinna, Brusseau, Ferry, and Cothran

from 2.7 to 4.0 (M = 3.83, SD = 14.02), 44 chose not to report their GPA.

Procedures and Data Analyses

The two major steps in data analysis were validation and inferential analysis. Validation included a pilot study and confirmatory factor analysis/internal consistency reliability. Inferential analysis included investigating dif-ferences in (a) preservice teachers’ belief system profiles and (b) across year in their program and region.

Current Instrument Validation. We conducted a pilot study to investigate its validity with preservice teachers. Pilot participants were a convenience sample of 54 pre-service teachers from one university (women = 37.0%, men = 61.1%; 1.9% did not report gender). All pilot participants volunteered and provided informed consent. They were asked if the instrument was clear, if they had any suggestions for changes, and if it was easy to complete in writing. All reported that the instrument was clear and easy to complete, and they did not suggest any changes. Pilot results showed a high level of internal consistency reliability for instrument items (α = .94).

We also conducted a confirmatory factor analysis (CFA) with the entire sample to confirm instrument ap-propriateness for use with preservice teachers. CFA was conducted using maximum likelihood structural equation modeling procedures (EQS: Bentler, 1995), with EQS software version 6.1, to determine whether the preservice teacher data fit the four measurement models.

Data Management. Data were reverse coded (e.g., 1 = very important became a 5 so that higher scores reflected higher values) and scored by creating measures for each domain (e.g., social development, motor skill develop-ment, self-actualization, and physical activity and fitness) as well as an overall belief systems score. The year catego-ries were reclassified into four groups to gain adequate cell sizes for comparison (i.e., freshman and sophomore preservice teachers were combined as was postbachelor and graduate preservice teachers). Because we were in-terested in preservice teachers’ belief systems rather than differences in their teacher education programs, we used preservice teachers as the unit of analysis. They were not truly nested; rather they were self-selected, as the self-se-lection process could have resulted in teachers attending different college/university programs.

Table 1. Physical education teacher education program descriptive characteristics by region

Program City University # participants Total preservice # full-time Community Curriculardemographics/ population population from each teachers in faculty model(s)region & school program programs

Northeast 1 151,176 5,000+ 45 375 33 U 1, 2, 3, 4 2 18,423 7,300 45 1000 16 S 2, 3, 5 3a 3,068 7,600 30 391 14 R 5, 6, 7South 4 486,411 27,000 30 60 3 U None 5a 27,410 36,014 20 64 2 U 1,6, 8 6a 28,340 10,321 29 86 4 R NoneMidwest 7 871,121 33,000 62 150 5 U 1, 6, 8, 10 8a 785,597 8,874 1 39 4 U None 9a 84,167 65,753 5 70 4 U None 10a 64,084 14,300 7 180 3 S NoneWest 11 49,807 19,753 5 10 2 R 6 12 22,352 11,636 31 82 3 R 4, 6, 8, 1, 3 13 89,046 12,981 67 200 5 S 2, 4, 6 14 25,688 13,000 34 100 2 R 4, 6 15a 86,268 16,415 13 67 4 R 1, 16a 58,213 14,526 13 82 2 S 11, 8 17 73,316 17,034 15 105 4 R 1, 2, 3, 5, 6, 7, 8 18 909,863 20,357 34 50 5 U None

Note. Curricular models: 1 = movement education, 2 = skill themes, 3 = outdoor/adventure education, 4 = tactical games ap-proach, 5 = teaching games for understanding, 6 = sport education, 7 = teaching personal and social responsibility, 8 = fitness education, 9 = lifetime physical education, 10 = dance, 11 = developmental; U = urban, S = suburban, R = rural.aRandom selection; community

Kulinna.indd 192 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 6: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

RQES: June 2010 193

Kulinna, Brusseau, Ferry, and Cothran

A Box-Cox test on the data set rejected the normal-ity assumption and indicated the best transformations were: (a) social development and motor skill develop-ment X3 (raising to the third power), and (b) physical activity and fitness and self-actualization X4 (raising to the fourth power). After transforming the data, residu-als were plotted to check for independence. A scatter plot of the residuals is shown in Figure 1, which suggests the transformed data met the independent assumption for inferential statistics. A profile analysis was run using repeated measures analysis of variance (RM-ANOVA) for the four outcomes goals to determine if the preservice teachers considered each goal to be equally important. The relative importance of the four domains (outcome goals) was also determined using multivariate analysis of variance (MANOVA) for “year in school” by “region” (4 x 4). Finally, descriptive statistics were also calculated for the belief system instrument scores.

Results

Results of the CFA demonstrated an adequate to good data fit for preservice teachers (N = 486) to the four-do-main model. The comparative fit index (CFI), considered

the fit index of choice (Bentler, 1990), demonstrated a good fit (CFI = .915), along with the chi-squared/degrees of freedom (χ2/df = 4.22). The root mean square error of approximation (RMSEA = .079 [.075, .062]) was slightly high in the “reasonable error of approximation” range, or adequate for research purposes (Browne & Cudeck, 1993; Steiger, 1989). Recently, some researchers argued that it may not be appropriate to specify an adequate fit range for the RMSEA and suggested that results from this measure be considered in conjunction with chi-square and other goodness-of-fit indexes (Chen, Curran, Bollen, Kirby, & Paxton, 2008). The internal consistency reliability for the overall instrument was α = .98.

The RM-ANOVA profile analysis was significant, F(3, 405) = 405.22, p < .0001, η2 = .68, suggesting that preservice teachers did not view the four outcome goals as equally important. They identified physical activity leading to fitness as the most important outcome goal for physical education followed by self-actualization, motor skill de-velopment, and social development, respectively. Table 2 reports the alpha values for the subscales, which show a high level of internal consistency for all the measures.

MANOVA analysis for Region x Year in School showed a significant interaction effect, Wilks’ lambda, F(36, 1605) = 2.29, p < .0001, η2 = .05, main effect for year Preservice Teachers’ Beliefs 1

123456789

1011121314151617181920212223242526272829303132333435363738394041424344

‚ ‚ ‚ A ‚ 2 ˆ A ‚ ‚ A AA AA A A ‚ AAA A AAA A A B A AA B ‚ AA A A A A AA A A A A A A B ‚ A A A A A ABABAB A A A A ‚ A A BB A B B A AA A ABA A A A A A 1 ˆ A A A A A ‚ B BB A A A AAA A A A AAA A AA AA A A A ‚ A A B A A AC A A AAA A A ‚ B A A AA B A A A A A A A A A A ‚ A A AA A A AAB A A A A B ‚ A BAA AA AA A A A B A AAAA AAA B AAA AB A A A ‚ BA A A A A AAA A B A A AA AAA

Res

idua

ls

0 ˆ A B C A AA A AA A A AA A A A A ‚ B AB A A A B A AA A C A A B AA BA A A A A ‚ A A A A AA A A A A B B A A ‚ AAAAA B AA A A A A A A A ‚ AA A A A AA AA A A AB A B AA A A AAAA A ‚ A AA A AA AA AB A A AA A A A A A ‚ AAA A B A AAA A A A A A A -1 ˆ AAAA A A AA AA AB A AA AAAB A ‚ A A A A A A AA A AAA ‚ A A A A A A A AA A ‚ A A B A A A ‚ A A A A ‚ A A A A ‚ A -2 ˆ A A A A A A A AA A ‚AAA A A A A AA A A A A ‚ ‚ A A ‚ ‚ ‚ -3 ˆ Šˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒ

60 120 180 240 300 360 420 480 540

Participant ID

Figure 1. Plot of the transformed data showing independent assumption was not violated.

Kulinna.indd 193 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 7: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

194 RQES: June 2010

Kulinna, Brusseau, Ferry, and Cothran

in school, Wilks’ lambda, F(12, 1,132) = 1.75, p = .05, η2 = .05, and main effect for region, Wilks’ lambda, F(12, 1,132) = 2.15, p = .01, η2 = .02.

Interactions for Region x Year in School were pres-ent for all four dependent variables, that is, physical activity and fitness, F(9, 431) = 2.45, p < .01, η2 = .05, self-actualization, F(9, 431) = 2.22, p = .02, η2 = .04, mo-tor skill development, F(9, 431) = 2.14, p = .03, η2 = .04,

and social development, F(9, 431) = 2.32, p < .02, η2 = .05. Tukey follow-up tests for the Region x Year in School interaction suggested that freshman and sophomore preservice teachers rated physical activity and fitness, self-actualization, and social development outcomes for physical education as having significantly lower priority than did seniors and postbachelor/graduates (p range: <.01–.02). Follow-up tests indicated there were no year in school motor skill development differences or specific trends in regional differences (see Table 3 for descriptive statistics by year in school and region).

Discussion

We investigated preservice teachers’ belief system profiles (i.e., whether they had the same relative outcome priorities for physical education). We also explored differ-ences in outcome priorities by geographical region and year in the teacher education program. Preservice teach-ers’ profiles differed, suggesting they did not view the four

Table 3. Descriptive statistics by level and region

Domain Year in School F & S Junior Senior Post/grad Total M SD M SD M SD M SD M SD

Physical activity & fitness East 37.36 8.62 38.7 9.43 38.62 9.17 43.50 2.38 38.31 8.83 n = 44 n = 14 n = 50 n = 4 n = 112 South 28.08 13.95 39.12 9.87 37.12 10.49 41.86 2.91 36.75 11.12 n = 12 n = 25 n = 24 n = 7 n = 68 Mid 41.40 3.36 39.47 5.22 39.35 5.67 41.87 2.69 40.33 4.69 n = 5 n =17 n = 26 n = 23 n = 71 West 42.56 3.24 38.72 7.75 39.90 5.70 40.69 3.32 39.81 6.19 n = 16 n = 69 n = 85 n = 26 n = 196Self-actualization East 35.27 6.93 37.57 8.08 37.64 8.50 41.00 3.37 36.82 7.77 South 28.42 12.59 38.76 8.66 37.38 9.07 41.29 3.15 36.70 9.91 Mid 39.40 3.21 38.29 4.80 38.50 5.79 39.82 2.83 38.94 4.56 West 41.12 3.40 37.68 7.30 39.00 5.47 40.65 3.20 38.92 5.91Motor skill development East 36.93 7.23 35.57 8.25 35.54 7.81 34.50 4.65 36.05 7.50 South 28.25 11.92 37.48 8.94 36.00 9.06 40.57 3.91 35.64 9.75 Mid 38.20 4.94 38.17 5.23 37.34 6.52 37.26 4.97 37.57 5.59 West 40.06 4.28 36.85 7.33 37.84 5.50 38.50 3.67 37.76 5.98Social development East 33.59 6.39 36.14 6.49 37.24 8.35 39.00 3.92 35.73 7.43 South 28.75 10.47 37.80 8.46 35.33 8.54 38.71 4.57 35.42 9.04 Mid 37.80 4.09 37.67 5.06 37.38 5.48 37.43 3.85 37.52 4.71 West 39.81 4.65 36.10 6.85 37.25 6.14 39.19 3.60 37.31 6.11

Note. F & S = freshman and sophomore preservice teachers; Post/grad = post bachelor and graduate preservice teachers; M = mean; SD = standard deviation; Mid = midwest; participant N are the same across all four domains.

Table 2. Descriptive statistics by outcome goal priorities

Domain M SD Range Alpha

Physical activity and fitness 39.09 7.73 10.00–45.00 .96Self-actualization 38.09 7.03 10.00–45.00 .94Motor skill development 37.02 7.12 10.00–45.00 .94Social development 36.68 6.87 10.00–45.00 .94

Note. M = mean; SD = standard deviation.

Kulinna.indd 194 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 8: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

RQES: June 2010 195

Kulinna, Brusseau, Ferry, and Cothran

goals as equally important. Although preservice teachers held strong belief systems for all four domains, they rated physical activity and fitness outcomes as a higher priority than the other areas (self-actualization, motor skill devel-opment, and social development).

These ratings are similar to the Kulinna and Silver-man (2000) investigation of a broad cross-section of practicing teachers’ belief systems. They also found that teachers highly rated all four outcome goals and held the same order of outcome priorities as in our study. Teachers reported differences by teaching level and years of experi-ence; secondary teachers placed a higher importance on physical activity leading to fitness, while elementary and middle school teachers placed more emphasis on mo-tor development and social development. Clearly, more longitudinal work is needed to establish how different curricular belief systems develop over time.

Preservice teachers often enter teacher education programs with views that physical education’s foundation lies in skill development (O’Sullivan, 2003). In the present study, preservice teachers did not report skill develop-ment as the most important outcome goal; instead, they reported that all the goals were important. The difference between the highest rated outcome goal (physical activ-ity and fitness) and the lowest (social development) was only 2.41 (out of 45.00). Studies of important outcomes from other countries have shown specific, high priorities for physical education programs. For example in Flan-ders, Belgium, preservice teachers reported that social responsibility and disciplinary mastery (traditional skill and physical fitness development) were their highest pri-orities (Behets & Vergauwen, 2004). These more focused curricular beliefs may be due, at least partially, to a formal district curriculum that interacted with the teachers’ belief system to influence the content and implementation of physical education lessons (Behets, 2001).

The fact that teachers in our study favorably rated physical activity leading to fitness is promising news for organizations that have targeted school physical educa-tion as a primary intervention site for the nation’s health challenges (e.g., Centers for Disease Control and Pre-vention, 1997). An overwhelming amount of evidence supports the positive contribution of physical activity to young people’s health (i.e., less adiposity, decreased risk for cardiovascular disease, metabolic syndrome, depres-sion, anxiety, diabetes, some forms of cancer, as well as increased skeletal health; Strong et al., 2005). Story (1999) reported that implementing changes in schools could lead to improved youth health. In her review of 13 school-based intervention studies (all but one included a physical education component), she concluded the interventions were effective for making positive health changes, including decreased body weight and/or body fat and increased aerobic capacity, muscular endurance, flexibility, knowledge, and self-efficacy.

Preservice Teachers’ Belief Systems Across Year

There were also similarities among our findings and other investigations related to preservice teacher beliefs. For example, we found some differences in preservice teachers’ belief systems at different points in their train-ing program, which parallels across-time investigations of preservice teachers. Specifically, MANOVA results for Region x Year in School showed that younger preservice teachers’ belief systems about the importance of various physical education outcome goals were lower than senior and graduate preservice teachers. This could possibly be due to preservice teachers’ addressing (changing) their K–12 belief systems during the first couple years of their teacher education programs. Two previous studies (i.e., Solmon & Ashy, 1995; Timken & van der Mars, 2009) also reported fluctuations in preservice teachers’ belief systems over time. Graham, Hohn, Werner, and Woods (1993) reported that preservice teachers’ conceptions (or belief systems) of teaching were more inconsistent than those of student teachers or cooperating teachers.

There are also intriguing differences between our findings and other investigations of preservice teachers’ belief systems. Placek et al. (1995) and O’Sullivan (2005) reported that preservice teachers rated skill development as the highest program outcome, whereas those in our study reported that outcome as the third most impor-tant. It is likely that some of these rating differences are methodological in nature (e.g., different instrumentation, the use of qualitative and quantitative techniques), but it is also interesting and important to understand how preservice teachers’ belief systems may change over time due to cultural, programmatic, or personal factors. Clearly additional research is needed in this area.

Preservice Teachers’ Belief Systems Across Region

Although there was a main effect for region and an interaction of region with year in school, there was no clear trend in regional differences (i.e., no significant follow-up test findings). This may be related to unbal-anced participant numbers by region. Curricular work with inservice teachers suggested that social context and culture (e.g., Banville, Desrosiers, & Genet-Violet, 2002; Chen, Liu, & Ennis, 1997) influenced teacher beliefs; thus, further work is needed in this area also.

Possible Tensions

Although results from the current study add to the literature related to preservice teacher belief systems, the real significance of these findings may be that they highlight critical “tensions” in preservice teacher/teacher belief systems, specifically: (a) the need to examine more fully the consistency of preservice teacher/teacher belief

Kulinna.indd 195 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 9: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

196 RQES: June 2010

Kulinna, Brusseau, Ferry, and Cothran

systems, and (b) implications for teacher education and professional development programming.

The first critical tension for future study is the need to examine more fully the consistency of preservice teacher/teacher belief systems. Although there were differences by year in teacher education program and region; in general, preservice teachers from across the nation believed relatively the same things at different points in their programs (i.e., outcome priorities of physi-cal activity leading to fitness, followed by self-actualization, motor skill development and social development). When viewed in conjunction with the Kulinna and Silverman (2000) study, we have a picture of a generally consistent belief pattern among preservice and inservice teachers. How did such a consistent pattern develop, and why?

One could argue that teacher education programs provide the guiding force for these health-related physi-cal education beliefs, but something else might be at work. The most likely influence would seem to be media messages and/or personal experience related to the negative changes in American population’s health (e.g., campaigns such as “Alliance for a Healthier Generation,” a partnership between the Clinton Foundation and the American Heart Association promoting healthier school environments; Healthier Generation, 2008). It is also widely known that for U.S. youth ages 6–19 years over-weight prevalence tripled over the past 25 years and the trend is continuing (Ogden, Flegal, Carroll, & Johnson, 2002; Ogden et al., 2006). Furthermore, U.S. youth have the highest overweight rates in the world (Wang & Dietz, 2002). Thus, youth health issues are important to the general public and teachers and should be addressed in teacher education programs.

Alternatively, one could argue the data suggest near equal valuing of other curricular outcomes rather than teachers focused on health-related outcomes. In essence, the question is whether statistical significance translates to a practical difference when all outcome items are so closely rated (less than a 5-point difference between areas by the senior year), with the small effect sizes suggest-ing little practical difference across year in the program and geographic region. Do teachers hold clear enough beliefs about what is most important to be able to build a philosophically consistent and effective curriculum? Rink (1993), speaking specifically about secondary education but offering a curricular message that applies to all pro-gramming, suggested we, as a profession, have failed to clearly articulate our program goals and are, therefore, severely handicapped when building quality programs, because the expected outcomes are not clear. Given the limited time and resources available to physical educators, teachers must often choose which curricular outcomes are most important (Ennis, 1994); one interpretation is that teachers are not able to make those difficult decisions. Additional research is needed in this area.

A second tension for consideration is what these find-ings mean for teacher education and professional develop-ment programming. Clearly more health-related physical education programming is needed at both levels if we are to provide teachers with the knowledge and experiences necessary to implement programs consistent with their outcome priorities. What might a university program look like that produces graduates capable of implementing highly active physical education programming? Different curricular models can be used to support active classes in quality physical education programs. For example, Hastie and Trost (2002) showed that Sport Education can meet the criteria of 50% of class time spent in physical activity. In addition, using the Sport, Physical Activity and Active Recreation for Kids curriculum has also produced recom-mended levels of physical activity participation (Kelder et al., 2003).

Limitations of this study include the nonrandom sampling of teacher education programs in Phase 1; however, the random selection in Phase 2 and recruit-ment of programs representing all regions of the country strengthened the generalizability of the findings. Another limitation of this study was the small MANOVA effect sizes for year in the program by geographical region, highlight-ing the issue that statistical significance does not always translate into practical significance.

Even with these limitations, however, the current findings provide important information about preservice teachers’ beliefs systems and how a variety of factors may influence those beliefs. It is critical to understand (a) pre-service teachers’ beliefs systems when they enter teacher education programs and during their program experi-ences and (b) how belief systems held during teacher education programs may affect beliefs and practices later in teachers’ careers. These questions and others suggest that more work needs to be done to better understand preservice teachers’ outcome priorities. The current study supports earlier work suggesting that preservice teachers’ belief systems may not be stable and that teacher education programs can make lasting changes in teach-ers’ belief systems and even behaviors, as Graber (1996) and Rovegno (1993b) found in their investigations of an effective teacher education program. This study also pro-vides a new understanding of preservice teachers’ current conceptualizations of physical education priorities.

References

Ayers, S. F., Housner, L. D., & Kim, H. Y. (2004). Directory of physi-cal education teacher education (PETE) programs. Morgantown, WV: Fitness Information Technologies.

Banville, D., Desrosiers, P., & Genet-Violet, Y. (2002). Compari-son of value orientations of Quebec and American teach-ers: A cultural difference? Teaching and Teacher Education, 18, 469–482.

Kulinna.indd 196 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 10: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

RQES: June 2010 197

Kulinna, Brusseau, Ferry, and Cothran

Behets, D. (2001). Value orientations of physical education pre-service and inservice teachers. Journal of Teaching in Physical & Education, 20, 144–154.

Behets, D., & Vergauwen, L. (2004). Value orientations of elementary and secondary teachers in Flanders. Research Quarterly for Exercise & Sport, 75, 156–164.

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.

Bentler, P. M. (1995). EQS structural equation program manual. Encino, CA: Multivariate Software, Inc.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.

Centers for Disease Control and Prevention. (1997). Guidelines for school and community programs to promote lifelong physical activity among young people. Morbidity and Mortal-ity Weekly Report, 46(RR-6), 1–36.

Chen, A., Liu, Z., & Ennis, C. D. (1997). Universality and unique-ness of teacher educational value orientations: A cross cultural comparison between the U.S. and China. Journal of Research and Development in Education, 30, 135–143.

Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistics in structural equation mod-els. Sociological Methods and Research, 36, 462–494.

Ennis, C. D. (1994). Urban secondary teachers’ value orienta-tions: Delineating curricular goals for social responsibility. Journal of Teaching in Physical Education, 13, 163–179.

Ennis, C. D., & Chen, A. (1995). Teachers’ value orientations in urban and rural school settings. Research Quarterly for Exercise and Sport, 66, 41–50.

Ennis, C. D., Ross, J., & Chen, A. (1992). The role of value orientations in curricular decision making: A rationale for teachers’ goals and expectations. Research Quarterly for Exercise and Sport, 63, 38–47.

Graber, K. C. (1996). Influencing students’ beliefs: The design of a “high impact’ teacher education program. Teaching and Teacher Education, 12, 451–466.

Graham, K. C., French, K. E., & Woods, A. M. (1993). Observ-ing and interpreting teaching-learning processes: Novice PETE students, experienced PETE students, and expert teacher educators. Journal of Teaching in Physical Education, 13, 46–61.

Graham, K. C., Hohn, R. C., Werner, P. H., & Woods, A. M. (1993). Prospective PETE students, PETE student teachers, and clinical model teachers in a university teacher education pro-gram, Journal of Teaching in Physical Education. 12, 161–179.

Grunbaum, J. A., Kann, L., Kinchen, S., Ross, J., Hawkins, J., Lowry, R., et al. (2004). Youth risk behavior surveillance—United States, 2003. Journal of School Health, 74, 307–324.

Hastie, P., & Trost S. G. (2002). Student physical activity levels during a season of sport education. Pediatric Exercise Science, 14, 64–74.

Healthier Generation. (1997). Healthier generation. Retrieved De-cember 1, 2008, from http://www.healthiergeneration.org.

Kagan, D. M. (1992). Implication of research on teacher belief. Educational Psychologist, 27, 65–90.

Kelder, S. H., Mitchell, P. D., McKenzie, T. L., Derby, C., Strik-miller, P. K., Luepker, R. V., et al. (2003). Health Education and Behavior, 30, 463–475.

Kile, R. (1993). Preconceptions of elementary and secondary preservice teachers. Unpublished doctoral dissertation, University of Arizona, Tucson.

Kulinna, P. H., & Silverman, S. (1999). The development and validation of scores on a measure of teachers’ attitudes toward teaching physical activity and fitness. Educational and Psychological Measurement, 59, 507–517.

Kulinna, P. H., & Silverman, S. (2000). Teachers’ attitudes to-ward teaching physical activity and fitness. Research Quarterly for Exercise and Sport, 71, 80–84.

Lara-Cinisomo, S., Fuligni, A. S., Ritchie, S., Howes, C., & Karoly, L. (2008). Getting ready for school: An examination of early childhood educators’ belief systems. Early Childhood Education Journal, 35, 343–349.

Lawson, H. A. (1988). Occupational socialization, cultural studies, and the physical education curriculum. Journal of Teaching in Physical Education, 7, 265–288.

Matanin, M., & Collier, C. (2003). Longitudinal analysis of pre-service teachers’ beliefs about teaching physical education. Journal of Teaching in Physical Education, 22, 153–168.

Morine-Dershimer, G. (1989). Preservice teachers’ conceptions of content and pedagogy: Measuring growth in reflective, pedagogical decision-making. Journal of Teacher Education, 40, 46–52.

National Association for Sport and Physical Education. (2004). Moving into the future: National standards for physical education (2nd ed.). Reston, VA: Author.

Ogden, C. L., Carroll, M. D., Curtin, L. R., McDowell, M. A., Ta-bak, C. J., & Flegal, K. M. (2006). Prevalence of overweight and obesity in the United States, 1999–2004. Journal of the American Medical Association, 295, 1549–1555.

Ogden, C. L., Flegal, K. M., Carroll, M. D., & Johnson, C. L. (2002). Prevalence and trends in overweight among U.S. children and adolescents, 1999–2000. Journal of the American Medical Association, 288, 1728–1732.

O’Sullivan, M. (2003). Learning to teach physical education. In S. Silverman & C. Ennis (Eds.), Student learning in physical education (2nd ed., pp. 275–294). Champaign, IL: Human Kinetics.

O’Sullivan, M. (2005). Beliefs of teachers and teacher candi-dates: Implications for teacher education. In F. Carreiro da Costa, M. Cloes, & M. Gonzalez Valeiro (Eds.), The art and science of teaching in physical education (pp. 149–164). Cruz Quebrada-Dafundo, Portugal: Faculdade de Motricidade Humana.

Pajares, M. F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of Educational Re-search, 62, 307–332.

Placek, J., Dodds, P., Doolittle, S., Portman, P., Ratliffe, T., & Pinkham, K. (1995). Teaching recruits’ physical education backgrounds and beliefs about purposes for their subject matter. Journal of Teaching in Physical Education, 14, 246–261.

Reis, J. P., Bowles, H. R., Ainsworth, B. E., Dubose, K. D., Smith, S., & Laditka, J. N. (2004). Nonoccupational physical activ-ity by degree of urbanization and U.S. geographic region. Medicine & Science in Sports & Exercise, 36, 2093–2098.

Richardson, V. (1996). Role of attitudes and beliefs in learning to teach. In J. Sikula, T. Buttery, & E. Guyton (Eds.), Hand-book of research on teacher education (2nd ed., pp. 102–119). New York: Macmillan.

Kulinna.indd 197 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014

Page 11: Preservice Teachers' Belief Systems Toward Curricular Outcomes for Physical Education

198 RQES: June 2010

Kulinna, Brusseau, Ferry, and Cothran

Rink, J. (1993). What’s so critical? In J. Rink (Ed.), Critical cross-roads (pp. 1–6). Reston, VA: National Association for Sport and Physical Education.

Rovegno, I. (1992). Learning to teach in a field-based methods course: The development of pedagogical content knowl-edge. Teaching and Teacher Education, 8(1), 69–82.

Rovegno, I. (1993a). The development of curricular knowledge: A case of problematic pedagogical content knowledge dur-ing advanced knowledge acquisition. Research Quarterly for Exercise and Sport, 64, 56–68.

Rovegno, I. (1993b). Content-knowledge acquisition during un-dergraduate cultural templates and learning through prac-tice. American Educational Research Journal, 30, 611–642.

Rovegno, I. (2003). Teachers’ knowledge construction. In S. Silverman & C. Ennis (Eds.), Student learning in physical edu-cation: Applying research to enhance instruction (pp. 295–310). Champaign, IL: Human Kinetics.

Sebren, A. (1995). Preservice teachers’ reflections and knowl-edge development in a field-based elementary physical education methods course. Journal of Teaching in Physical Education, 14, 262–283.

Solmon, M. A., & Ashy, M. H. (1995). Value orientations of pre-service teachers. Research Quarterly for Exercise and Sport, 66, 219–230.

Springer, A. E., Hoelscher, D. M., & Kelder, S. H. (2006). Prevalence of physical activity and sedentary behaviors in U.S. high school students by metropolitan status and geographic region. Journal of Physical Activity and Health, 3, 365–380.

Steiger, J. H. (1989). CaU.S.l modeling: A supplementary mod-ule for SYSTAT and SYGRAPH. Evanston, IL: SYSTAT.

Strong, W. B., Malina, R. M., Blimkie, C. J. R., Daniels, S. R., Dish-man, R. K, Gutin, B., et al. (2005). Evidence based physical activity for school-age youth. Pediatrics, 146, 732–737.

Story, M. (1999). School-based approaches for preventing and treating obesity. International Journal of Obesity, (Suppl. 2), S43–S51.

Timken, G. L., & van der Mars, H. (2009). The effects of case methods on pre-service physical education teachers’ value orientations. Physical Education and Sport Pedagogy, 14, 169–187.

Wang, G., & Dietz, W. H. (2002). Economic burden of obesity in youths aged 6–17 years: 1979–1999. Pediatrics, 109, 1–6.

Woods, M., Goc Karp, G., & Escamilla, E. (2000). Preservice teachers learning about students and the teaching-learn-ing process. Journal of Teaching in Physical Education, 20, 15–39.

Authors’ Note

Please address correspondence concerning this article to Pamela Hodges Kulinna, Arizona State University, 7271 E. Sonoran Arroyo Mall, 330Q Santa Catalina Hall, Mesa, AZ 85212.

E-mail: [email protected]

Kulinna.indd 198 5/4/2010 1:22:40 PM

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

16:

03 0

9 O

ctob

er 2

014