journal of technology education · journal of technology education vol. 27 no. 2, spring 2016-7-...

Journal of Technology Education Vol. 27 No. 2, Spring 2016

-1-

Contents 2 Editorial Articles 4 Efficacy of the Technological/Engineering Design Approach: Imposed

Cognitive Demands Within Design-Based Biotechnology Instruction John G. Wells

21 Positioning Technology and Engineering Education as a Key Force in STEM Education Greg Strimel & Michael E. Grubbs

37 Problem Decomposition and Recomposition in Engineering Design: a Comparison of Design Behavior Between Professional Engineers, Engineering Seniors, and Engineering Freshmen Ting Song, Kurt Becker, John Gero, Scott DeBerard, Oenardi Lawanto, & Edward Reeve

57 Identifying Indicators Related to Constructs for Engineering Design Outcome Cheryl A. Wilhelmsen & Raymond A. Dixon

78 Characterizing Design Cognition of High School Students: Initial Analyses Comparing those With and Without Pre-Engineering Experiences John Wells, Matthew Lammi, John Gero, Michael E. Grubbs, Marie Paretti, & Christopher Williams

Miscellany 92 Scope of the JTE

Editorial Review Process Manuscript Submission Guidelines Subscription Information JTE Co-sponsors and Membership Information JTE Editorial Review Board Electronic Access to the JTE


-2-

Editorial

Publishing Scholarly Work Entails Hard Work As I put together the final pieces of the spring edition of the

JTE, I sit in awe of the scholars we have in and around technology and engineering education whom I have been fortunate enough to work with over the past six years (six years of a five-year term, but who’s counting?). Moreover, the men and women who serve on the JTE Editorial Review Board should be commended for the work that they do to review potential manuscripts for publication and to offer the author or authors of manuscripts input on how to improve their work so that their manuscripts can be considered further for publication. Finally, to the outstanding Technical Editor (Amanda Fain) for whom I have the utmost appreciation; Amanda’s “spot on” editing skills go far beyond commas and grammar; in fact, the entire JTE community would be shocked by the precision with which Amanda edits. Publishing scholarly work entails hard work, and all of us do the best we can.

I wanted to share my appreciation for the hard work that everyone puts into JTE, but I also wanted to take this time to write about a growing concern of mine—let’s say, for lack of better words, my concern is complacency for doing what is right. Don’t scholars have a responsibility to conduct, write, and submit research-based manuscripts that are accurate, ethical, and professional? Of course they do, right! Well, this is not always the case, and the more manuscripts, reports, and book reviews I receive, the more I question our complacency. For example, I routinely get communications from potential authors (averaging about five per week) that simply state “publish my work” with a file attached, or I receive manuscripts that are so full of plagiarism that I cringe, and I hear of professionals in our field questioning the role of JTE as a flagship journal. What happened to writing a professional cover letter (by the way, to the professionals that do this on a routine basis, thank you) describing who you are, what the manuscript is about, how to reach you, and who your coauthors


-3-

are; submitting original work that is not duplicated or plagiarized; and submitting manuscripts for potential publication that relate to the scope of journal? I am asking for your help. If you are in a position to mentor professionals doing scholarly work, please take the time to do so. Publishing scholarly work that is “scholarly” entails hard work from all of us.

I hope you enjoy this edition of the JTE as much as I have enjoyed putting it together. I believe you will find the content of the articles to be thought provoking.

Chris Merrill Editor


-4-

Efficacy of the Technological/Engineering Design Approach: Imposed Cognitive Demands Within

Design-Based Biotechnology Instruction

John G. Wells

Abstract Though not empirically established as an efficacious pedagogy for

promoting higher order thinking skills, technological/engineering design-based learning in K–12 STEM education is increasingly embraced as a core instructional method for integrative STEM learning that promotes the development of student critical thinking skills (Honey, Pearson, & Schweingruber, 2014; Kolodner, 2002; NGSS Lead States, 2013). To demonstrate the efficacy of these practices for promoting student use of higher order thinking skills (schematic and strategic knowledge), a group of mixed-discipline (STEM) students enrolled in a 16-week Biotechnology by Design™ graduate course were immersed in a series of biotechnology design challenges developed to intentionally teach select content and practices of technology and engineering design concurrent with those of science and mathematics. A pre-experimental, one-group pretest–posttest design was used to assess student responses to the continuum of cognitive demands imposed by the biotechnology design challenges. Overall findings indicate strong connections between student gains in biotechnology content knowledge and practices and supports the conclusion that technological/engineering design-based learning strategies improve a student’s capacity for responding to all four levels of imposed cognitive demand (declarative, procedural, schematic, strategic), lead to deeper learning of both content and practices, and promote student development of schematic and strategic (higher order) thinking skills. Keywords: efficacy; imposed cognitive demands; higher order thinking skills; critical thinking skills; design-based biotechnology literacy; technological/engineering design-based learning.


-5-

The pedagogical intent that underpins technological/engineering (T/E) design-based learning (DBL) as an instructional approach is to (a) promote student understanding of the connections between disciplinary content and practices (schematic domain) and (b) foster the ability for making informed decisions (strategic domain) based on that understanding (National Assessment Governing Board [NAGB], 2008; NGSS Lead States, 2013). The integration of STEM content and practices employing T/E DBL approaches uniquely imposes cognitive demands that tomorrow’s problem solvers must be prepared to address. A student’s ability to respond to higher order cognitive demands provides the premise for bridging instructional strategies with the assessment of student performance expectations at basic, proficient, and advanced levels (Wells, 2010).

The research presented in this article was designed to address the need for demonstrating the efficacy of T/E design-based practices in promoting student use of higher order thinking skills (schematic and strategic knowledge). Specifically, this research was conducted to evidence the potential of Design-Based Biotechnology Learning (DBBL™) to improve a learner’s response capacity to the higher order cognitive demands imposed by these unique T/E DBL challenges.

Background Meeting the global challenges of the 21st century will require individuals

who possess the capacity for integrating both the content and the practices requisite of specialists in the science, technology, engineering, and mathematics (STEM) disciplines (National Academy of Sciences, National Academy of Engineering, & Institute of Medicine, 2007, 2010). In anticipation of these challenges, the educational system in the United States is aggressively promoting the use of integrative approaches to STEM education (Honey, Pearson, & Schweingruber, 2014; Wells, 2008, 2010). As a result, STEM education disciplines are modifying their current national standards to incorporate the content and practices of disciplines other than their own, most notably those of T/E design (Burke, 2014; International Technology Education Association [ITEA], 2007; NGSS Lead States, 2013; Singer, Nielsen, & Schweingruber, 2012). Pedagogically, the underlying intent behind the incorporation of T/E design as an instructional strategy within the STEM education disciplines is to promote higher order learning skills by enhancing student understanding of the connections between disciplinary content and practices (schematic domain) and fostering their ability to make informed decisions (strategic domain) based on that understanding (NAGB, 2008; NGSS Lead States, 2013). Higher order cognitive abilities such as these are based on how cognitive theorists have come to distinguish between various types of knowledge. Beginning as early as 1949, the British philosopher Ryle envisioned knowledge that has been acquired to be demonstrated as declarative (knowing


-6-

that) and procedural (knowing how), which was later supported empirically through Anderson’s (1983) research. Utilization of acquired knowledge has more recently come to be recognized (Alexander, Schallert, & Hare, 1991) as a different type of knowledge (conditional) with two distinct forms (conceptual and metacognitive). Given that conditional knowledge is a subtype of metacognition, they could be collapsed into a single type (Li & Shavelson, 2001) that is referred to as strategic knowledge (knowing when and where). Recognition of relationships between multiple concepts or facts is referred to as schematic knowledge (knowing why) and is a precursor to development of strategic knowledge. This theoretical understanding of knowledge types underpins and is explicitly conveyed in the expected and observed performance outcomes as described in both the National Assessment of Educational Progress (NAEP) Science 2009 (NAGB, 2008, p. 83) and the NAEP Technology and Engineering 2014 (WestEd, 2012, p. A-38–A-40).

Performance expectations are generated by crossing the content to be learned with the practices that demonstrate understanding of that content (Wells, 2010). This approach to assessment provides, by design, the structure needed to obviate the connections between the instructional strategies employed by the educator and the performance outcomes (content and practice) demonstrated by the student. In the NAEP 2009 science framework (NAGB, 2008) and now in the Next Generation Science Standards (NGSS Lead States, 2013), science performance expectations are inclusive of a student’s ability to employ T/E design in the pursuit of learning science content and practices. Theoretically, the basis for this inclusion is the belief that T/E DBL promotes and advances a student’s ability to respond to the cognitive demands associated with the ill-structured, ill-defined challenges that they will undoubtedly be confronted with when meeting the challenges, both local and global, of the 21st century—that is, the knowledge and skills that they will need (Bybee, 2010; National Research Council [NRC], 2010; Partnership for 21st Century Skills, 2009) to compete in a global society (Engineering Challenges; National Academy of Engineering, 2008). Specifically, its incorporation is predicated on the pedagogical basis that the very nature of T/E DBL requires students to utilize,1 and therefore demonstrate, their declarative (knowing that), procedural (knowing how), schematic (knowing why), and strategic (knowing when and where) cognitive abilities.

Accepting this pedagogical basis as valid, student responses to the full spectrum of cognitive demands would therefore provide a mechanism for assessing their knowledge gains along the continuum from declarative to strategic and thus some evidence for the efficacy of T/E DBL as an instructional

1The term utilize is intentional and is distinguished from the term apply, which is often the perspective taken in science education regarding the role of T/E DBL that views engineering as a tool “in the service of science” (Sneider, 2012).


-7-

strategy that achieves the goal of integrative STEM education (Wells, 2013). However, in spite of this embedded pedagogical basis in the recently published Next Generation Science Standards (NGSS Lead States, 2013), the content and practices of T/E design are not expressly targeted or assessed learning outcomes, nor is there sufficient empirical evidence that the pedagogical approach of T/E DBL actually does enhance the ability of students to respond to cognitive demands, specifically schematic and strategic (i.e., higher order thinking skills).

Empirical evidence resulting from prior research related to DBL concludes that T/E DBL is a better approach for teaching core science concepts and leads to higher gains in science knowledge achievement (Doppelt, Mehalik, Schunn, Silk, & Krysinski, 2008; Fortus, Dershimer, Krajcik, Marx, & Mamlok-Naaman, 2004; Mehalik, Doppelt, & Schunn, 2008). Though such research has provided evidence of knowledge gains, concurrent gains in higher order thinking skills associated with the development of desired 21st century problem solving abilities were not addressed. Herein lies the need and potential for documenting the extent to which T/E DBL, as practiced in the secondary technology education classroom can and does place such cognitive demands on the learner and in so doing results in more well developed higher order thinking abilities required in responding to them.

Rationale Although not empirically established as an efficacious pedagogy for

promoting higher order thinking skills, K–12 STEM education engineering, and specifically engineering design, is increasingly embraced as a core instructional method and teaching tool for integrative STEM learning (Kolodner, 2002; NGSS Lead States, 2013). Furthermore, few if any effectiveness trials have been conducted to present empirical evidence of T/E DBL as an effective integrative STEM instructional method (Honey, Pearson, & Schweingruber, 2014). Although widely accepted as a necessary precondition for effectiveness trials (Sloane, 2008), efficacy research in education is often not presented as a precursor to effectiveness research on new interventions. For T/E DBL, this begs the question: Why invest resources into implementing an intervention that has yet to be demonstrated efficacious? The research presented here is an efficacy study designed to establish, within an ideal setting, the viability of T/E DBL as a pedagogical approach that through imposed cognitive demands supports student development of critical thinking skills.

Purpose The purpose of this study was to evidence the potential of T/E DBL to

improve the capacity of students to respond to higher order cognitive demands imposed by select engineering design challenges. Specifically, the biotechnical engineering design challenges used in this research are drawn from the T/E


-8-

Design-Based Biotechnology Learning teaching guide (Wells, 2015). The research was guided by the following questions:

To what extent can T/E Design-Based Biotechnology Learning design challenges: 1. Facilitate student gains in biotechnology content knowledge

(declarative and procedural), 2. Enhance the ability of students to respond to embedded higher order

cognitive demands (schematic and strategic), and 3. Provide evidence for the validity of T/E DBL as an instructional

method? In the United States, the contextual basis for studying technology education

content is organized around three inclusive technological categories: physical, biological, and informational systems (ITEA, 1996, 2006). Biotechnology is a content area housed specifically within the context of biological systems. Within the Standards for Technological Literacy: Content for the Study of Technology (STL; ITEA, 2007) biotechnology is addressed directly in Standard 15 (pp. 149–157) and found naturally embedded across all five STL content standard categories as well (Wells & Kwon, 2009, p. 265). In the fields of both science and technology education the broadly accepted and employed operational definition of biotechnology is “‘any technique that uses living organisms, or parts of organisms, to make or modify products, improve plants or animals, or to develop microorganisms for specific purposes’ (OTA, 1988/1991, FCCSET, 1992/1993)” (ITEA, 2007, p. 149; e.g., Dunham, Wells, & White, 2002; FCCSET Committee on Life Sciences and Health, 1993; ITEA, 2007; Stotter, 2004; U.S. Congress, Office of Technological Assessment, 1984, 1988, 1991; Wells, 1994, 1999, 2012, 2015; Wells & Kwon, 2008, 2009). As operationally defined, educators from the classroom to preservice levels are provided with a set of explicit criteria for determining what is or is not recognized as biotechnology content or practices (Wells, 1995):

1. “Any technique”: This first criterion specifies the full spectrum of practices, from micro to macro, involved in biotechnical processes.

2. “That uses living organisms”: This criterion underscores the requirement that biotechnical processes must include living organisms (such as plants, microbes, fungi, and even macro scale organisms such as human beings).

3. “Or parts of organisms”: As an extension of the living organisms, this criterion further specifies that components within the organism or its cellular elements (e.g., organelles, enzymes, proteins, DNA) can be isolated and used independently.

4. “To make or modify products, improve plants or animals, or to develop microorganisms for specific purposes”: These final elements provide specificity for the range of potential biotechnical applications.


-9-

In technology education, biotechnology is one content area that naturally imposes cognitive demands when employing T/E DBL approaches to teach the content and practices of both science and technology. In the case of Design-Based Biotechnology Learning, which uses a T/E design approach to teach biotechnology content and practices (Wells, 1992, 1994), each cognitive demand is intentionally targeted within a series of authentic experiences that are integral to the design of instruction. The DBBL™ curriculum (Wells, 2015) uses biotechnology problem scenarios to present students with open-ended, T/E design-based biotechnology challenges that intentionally teach the content and practices of both science and technology concurrently.

Method Participants

This research involved a mixed-discipline (STEM) group of graduate students enrolled in a 16-week Biotechnology by Design™ (BBD™) graduate course. The course is designed to immerse participants in a series of biotechnology design challenges in the same manner as it would be delivered to secondary-level students while concurrently reflecting on the educator’s requirements for delivering such instruction. Of the 16 graduate students enrolled in the course, 75% (12) were female and 25% (4) were male, with the class composition representing each of the four primary (home) STEM disciplines (Table 1). All students were currently, or had been, practicing K–12 educators in their primary disciplines with classroom experience ranging from 2 to 30 years. The 50% of students representing the technology and engineering disciplines had prior experience in using the T/E DBL approach but only minimal formal biology content or practice preparation (i.e., high school only). None of the science or mathematics students had any prior experience using the T/E DBL approach.

Table 1 Student Disciplinary Demographics

Number of students representing primary disciplines Gender Science Technology Engineering Mathematics Female 5 2 3 2 Male 0 3 0 1

Specifically, the participants in this study, who were all licensed K–12

educators, were prepared to teach only one of the four STEM education subjects. Each participant possessed at least the minimal required level of classroom expertise (content and practice) to teach their respective disciplines, but they lacked that same level of expertise for teaching subjects other than their own. Therefore, from a preparation perspective, it is a fair assumption that the


-10-

participants represent a replicable subsample. However, it is recognized that any duplication of this study might well produce different results, and as an efficacy study, there is no intent to generalize beyond this population.

The Biotechnology by Design™ course was delivered simultaneously to both on-campus and distance students, 10 and 6 respectively, using a synchronous audio–video platform. All students received the same instruction, materials, and course supplies with no appreciable differences in engagement during the regularly scheduled 3-hour class sessions. The overarching instructional objective of the course was to intentionally teach select content and practices of T/E design concurrent with those of science and mathematics using the T/E DBL approach. Procedure

Graduate students in the course were engaged in a sequence of biotechnology design challenges. Problem Scenario 4A: Alternative Fuel Bioreactor (Bioreactor Scenario) was presented first, followed by Problem Scenario 4C: Microbial Fuel Cell (Fuel Cell Scenario), which is a more complex design challenge. The number of students in the course allowed for five design teams comprised of three to four on and off campus students. Multidisciplinary teams were purposefully assembled to include at least one student representing either technology or engineering and one representing science. Class composition allowed only two teams to include a student of mathematics. Different design teams of similar disciplinary composition were assembled for each of the two design challenges. Both problem scenarios provided the context, challenge, and constraints framing the challenge and asked student design teams to design, develop, and test a working prototype. The Bioreactor Scenario prototype calls for the design of a functioning bioreactor that harnesses S. cerevisiae (common yeast) immobilized in alginic beads to metabolize a dextrose substrate and produce ethanol and carbon dioxide byproducts. The Fuel Cell Scenario challenges students to design a functioning organic microbial fuel cell that exploits the electron production abilities of select benthic microorganisms to generate an electrical current sufficient enough to power a light emitting diode.

Teams were allotted 5 weeks to complete each problem scenario, at which point they would present their functional prototype, discuss performance results, and submit a detailed report documenting work performed in the form of a collaborative portfolio reflecting every phase of the T/E design process. As part of the course materials, students were provided with the PIRPOSAL blended pedagogy model (Wells, 2015) portfolio document used to detail the T/E design process and guide all students, both independently or collaboratively, in achieving plausible design solutions. Elements of the PIRPOSAL portfolio include Problem identification, Ideation, Research, Potential solutions, Optimization, Solution evaluation, Alterations, and Learned outcomes. The


-11-

PIRPOSAL portfolio document structured student engagement in a sequence of predetermined investigations designed to highlight relationships between key technological and biological variables critical to making informed biotechnical design decisions. In this way, students were guided in their exploration and exposure to prerequisite biological and technological content and practices unique to each problem scenario and necessary for achieving viable biotechnology solutions. For the Bioreactor Scenario, an immersive strategy was used in which students acquired both biology and technology content and practice following the steps of the design process. These steps were presented in the PIRPOSAL document as a mechanism for both teaching and guiding students through the technology design process. No direct (didactic) instruction of content or practice was provided. In contrast, because of the more complex concepts involved with organic generation of free electrons and their capture for use in an electric circuit, a small degree of didactic instruction was necessary for initiating the Fuel Cell Scenario. Weekly class discussions were used to assist in further clarifying technological and biological concepts, processes, and practices, but design teams worked independently to design and develop their final biotechnology prototyped solutions.

This study followed a preexperimental, one-group pretest–posttest design (Creswell, 2014). The full spectrum of research that was conducted utilized a battery of data collection instruments (Biotechnology Stages of Concern, Awareness, General Content Knowledge, ProbScen Knowledge, Terminology, and Literacy) intended to assess student variables on multiple levels. However, the purpose of this article is to present evidence of the cognitive demands inherent within T/E DBL and, therefore, focuses only on the pre–post changes in ProbScen Knowledge and corresponding assessment of students responses to the continuum of cognitive demands (i.e., cognitive gains) imposed by select design-based biotechnology problem scenarios. Instrumentation

Prior to introducing either problem scenario, students were asked to complete ProbScen Content and Practice Knowledge (CPK) questionnaires developed for each. CPK items had been developed to closely correspond with those included in the NAEP Science 2000–2011 twelfth grade sample questions (National Center for Educational Statistics, 2014) to ensure each assessed the specific biology and technology content and design-based practices intentionally targeted within the design of instruction. Every item was independently analyzed by an expert from engineering education and an expert from biological science education. The experts then met to discuss, arbitrate, and reach consensus on alignment of each with one of the four cognitive demands (declarative, procedural, schematic, and strategic) imposed by the design-based instructional approach. The same CPK questionnaires were administered again at the completion of each design challenge and following team presentations of


-12-

final biotechnology prototypes. All pre- and post-CPK questionnaires were web-based instruments administered during class.

Findings Data from the Bioreactor and Fuel Cell pre- and post-Knowledge

questionnaires were analyzed to assess student knowledge gains and their ability to respond to questions aligned with imposed cognitive demands along a continuum from declarative to strategic. Of the 17 items comprising the Bioreactor questionnaire, roughly 54% targeted declarative knowledge, 5% procedural, 23% schematic, and 18% strategic. Of the 17 items used for the Fuel Cell questionnaire, roughly 43% targeted declarative knowledge, 11% procedural, 11% schematic, and 35% strategic. Pretest–posttest data analyses for the Bioreactor Scenario are displayed in Table 2, and data analyses for the Fuel Cell Scenario are shown in Table 3. Table 2 Bioreactor Scenario: Pretest–Posttest Biotechnology Domain Knowledge

Domain M SD SEM df t p †ES

Declarative Pre 2.86 1.66 0.44 13 6.63 0.0001* 0.86 Post 7.71 2.20 0.59

Procedural Pre 0.00 0.00 0.00 13 4.16 0.0001* 2.19 Post 0.57 0.51 0.14

Schematic Pre 1.71 0.73 0.19 13 6.73 0.0001* 2.44 Post 3.64 0.84 0.23

Strategic Pre 1.43 0.94 0.31 13 5.38 0.0001* 2.31 Post 3.07 0.47 0.20

Combined Pre 27.64 10.59 2.93 13 11.15 0.0001* 3.61 Post 70.62 13.21 3.66

Note. n = 14

*p < .05, two-tailed, paired; †Effect Size


-13-

Table 3 Fuel Cell Scenario: Pretest–Posttest Biotechnology Domain Knowledge

Domain M SD SEM df t p †ES

Declarative Pre 8.69 1.74 0.44 15 2.61 0.0197* 0.68 Post 9.93 1.02 0.26

Procedural Pre 2.50 0.89 0.22 15 1.82 0.0891 0.77 Post 2.94 0.25 0.06

Schematic Pre 2.31 0.87 0.22 15 2.76 0.0145* 0.94 Post 2.88 0.34 0.09

Strategic Pre 7.50 2.83 0.71 15 2.30 0.0361* 0.61 Post 8.88 1.67 0.42

Combined Pre 74.88 16.19 4.05 15 3.16 0.0064* 0.91 Post 85.88 8.08 2.02

Note. n = 16

*p < .05, two-tailed, paired; †Effect Size

Data analysis for the Bioreactor Scenario shown in Table 1 indicates

significance for pretest/posttest differences (p < .05) individually across all four levels of cognitive demand, and also for the aggregate analysis (Combined). The practical strength of these mean differences for all analyses is substantiated by large effect sizes, ranging from .86 to 3.61. The same series of analyses performed for data from the Fuel Cell Scenario (Table 3) similarly indicate significance (p < .05) across all cognitive demands except for procedural, which was not found to be significant. The practical strength of the mean differences displayed in Table 3 is substantiated by large effect sizes, ranging from 0.68 to 0.94.

Discussion Design to Understand

As the signature pedagogy of technology education, technological design is privileged in the context of Integrative STEM Education (Wells, 2014) in which the teaching of discipline specific content and practice is intentional within the selected design-based instructional strategies. Such strategies are intent on positioning the students' achievement of understanding within the need-to-know learning context imposed by the challenge of designing a


-14-

functional prototype solution. Design-Based Biotechnology Learning is built upon this pedagogical premise in which the instructional goal is intent on having students design to understand when working toward a viable biotechnology solution. Problem Scenario Comparisons

Based on the combined (whole class) findings for the Bioreactor Scenario (Table 2), students clearly demonstrated a significantly better understanding of the technology, science, engineering, and mathematics content and practices following this immersive approach. These findings are supported by similar results from prior research into design-based learning (Calabrese-Barton, 1998; Doppelt, et al., 2008). However, due to the more complex nature of biology and technology content associated with the Fuel Cell Scenario, a more didactic approach was necessary for initiating this design challenge. As a result, students did receive some direct instruction in order to explain some of the more difficult to understand technology and biology concepts and processes. The remainder of the design challenge was guided by the same steps of the design process outlined in the PIRPOSAL document.

The comparison of combined results in Tables 2 and 3 (Bioreactor vs. Fuel Cell) reveal several interesting points. First, the average pretest combined scores for the Fuel Cell Scenario were significantly higher than those for the Bioreactor Scenario. This should be expected because students were provided direct instruction of content/practice prior to beginning the Fuel Cell Scenario. Furthermore, given that the biological elements were distinctly different in the second design challenge, the higher combined outcome scores would suggest that the T/E knowledge acquired in the first design challenged might well have been applied in the second challenge. Moreover, some content and practice covered in the completion of the Bioreactor Scenario was common and applicable to the design challenge in the Fuel Cell Scenario. The second interesting point was that strategic pretest and posttest scores were substantially higher for the Fuel Cell Scenario than for the Bioreactor Scenario. Recognizing the body of content and practice knowledge acquired in the completion of the Bioreactor Scenario, it is logical to consider that design-based decision-making (strategic) knowledge would be cumulative and therefore find application in the second biotechnology design challenge. Third, a significant difference was not observed in the pre–post procedural scores. This result provides some indication that the procedures repeatedly followed in the T/E design process were well engrained through completion of the problem scenarios.

Conclusions This research examined the cognitive demands encountered by graduate

students when engaged in developing T/E design solutions (functional biotechnical prototypes) to challenges presented in two select Design-Based


-15-

Biotechnology Learning problem scenarios. Specifically, the research was designed to investigate the extent to which these biotechnology scenarios facilitated student gains in biotechnology content knowledge (declarative and procedural), the extent to which these problem scenarios enhanced their ability to respond to embedded higher order cognitive demands (schematic and strategic), and whether these together with the pedagogical approaches provided sufficient evidence to validate the T/E DBL as an instructional method.

Overall findings indicate strong connections between student gains in biotechnology knowledge and the design-based biotechnology instructional strategies used to intentionally teach that content and practice, along with suggestions that the immersive approach is a viable strategy for facilitating those gains. As such, they support the conclusion that T/E DBL strategies improve a student’s capacity for responding to all four levels of imposed cognitive demand, lead to deeper learning of both content and practices, and promote student development of schematic and strategic (higher order) thinking skills. Although this research offers valuable empirical support for T/E DBL as a viable pedagogical approach, one must acknowledge the research limitations and consider the extent to which they affect the applicability of the conclusions reached.

Implications Demonstrating T/E DBL to be an efficacious instructional approach for

enhancing student use of higher order thinking skills carries with it at least one significant implication that is noteworthy for the T/E education profession. Specifically, findings indicate that T/E DBL is a viable approach for achieving cognitive learning goals (higher order thinking skills) similar to those espoused to be targeted in other core K–12 STEM subjects. For decades, scholars in technology education have repeatedly called for just such validation of practice (Foster, 1996; Hoepfl, 2002; Lewis, 1999; Zuga, 1994) and credibility among core K–12 subjects. Collectively, these calls are poignantly mirrored by Lewis (1999) in his statement that “To take its place squarely in school curricula, technology education must establish itself not only in its own right, but crucially in relation to other subjects” (p. 49). The implications of evidencing the inherent value of T/E pedagogical practices within the educational enterprise and establishing its legitimacy among other school subjects have significant potential for advancing the profession.

Recommendations Acknowledging that there are limitations to this research, caution must be

exercised regarding the extent to which conclusions can be drawn and implications made. To establish the broad validity necessary for acceptance of T/E DBL as a viable approach that supports student development of higher order (critical) thinking skills, many more efficacy studies are needed. In addition to


-16-

replicating the research presented here, similar efficacy studies should be conducted with different populations, larger populations, across geographic locations, and using various disciplinary team configurations. Further research is needed for improving the alignment of cognitive demands imposed by T/E DBL with content and practice assessment items. A replication efficacy study is currently underway with modifications addressing item construction and alignment with cognitive demands, alignment of cognitive demands with the PIRPOSAL blended pedagogy model (Wells, 2015) of integrative STEM education, as well as longitudinal assessment of knowledge retention. Efficacy studies of this type will be a necessary precondition and precursor to any effectiveness studies that might follow.

References Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard

University Press. Alexander, P. A., Schallert, D. L., & Hare, V. C. (1991). Coming to terms: How

researchers in learning and literacy talk about knowledge. Review of Educational Research, 61(3), 315–343. doi:10.3102/00346543061003315

Burke, B. N. (2014). The ITEEA 6E Learning byDesign™ model: Maximizing informed design and inquiry in the integrative STEM classroom. Technology and Engineering Teacher, 73(6), 14–19.

Bybee, R. W. (2010). A new challenge for science education leaders: Developing 21st-century workforce skills. In Rhoton, J. (Ed.), Science education leadership: Best practices for a new century, 33–49. Arlington, VA: NSTA Press.

Calabrese-Barton, A. (1998). Examining the social and scientific roles of invention in science education. Research in science education, 28(1), 133–151. doi:10.1007/BF02461647

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: Sage.

Doppelt, Y., Mehalik, M. M., Schunn, C. D., Silk, E., & Krysinski, D. (2008). Engagement and achievements: A case study of design-based learning in a science context. Journal of Technology Education, 19(2), 22–39. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v19n2/pdf/doppelt.pdf

Dunham, P., Wells, J., & White, K. (2002). Biotechnology education: A multiple instructional strategies approach to biotechnology education. Journal of Technology Education, 14(1), 65–81. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v14n1/pdf/dunham.pdf

FCCSET Committee on Life Sciences and Health. (1993). Biotechnology for the 21st century: Realizing the promise. Washington, DC: Federal Coordinating Council for Science, Engineering, and Technology.


-17-

Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., & Mamlok-Naaman, R. (2004). Design-based science and student learning. Journal of Research in Science Teaching, 41(10), 1081–1110. doi:10.1002/tea.20040

Foster, W. T. (1996). A research agenda for technology education. The Technology Teacher, 56(1), 31–33.

International Technology Education Association/International Technology and Engineering Educators Association. (2006). Technological literacy for all: A rationale and structure for the study of technology (2nd ed.). Reston, VA: Author.

International Technology Education Association/International Technology and Engineering Educators Association. (2007). Standards for technological literacy: Content for the study of technology (3rd ed.). Reston, VA: Author.

Hoepfl, M. (2002). A strategic framework for research in technology education. In H. Middleton, M. Pavlova, & D. Roebuck (Eds.), Learning in technology education: Challenges for the 21st century (Vol. 1, pp. 190–210). Brisbane, Australia: Centre for Technology Education Research, Griffith University.

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K–12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. doi:10.17226/18612

Kolodner, J. L. (2002). Facilitating the learning of design practices: Lessons learned from an inquiry into science education. Journal of Industrial Teacher Education, 39(3), 9–40. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v39n3/kolodner.html

Lewis, T. (1999). Research in technology education—Some areas of need. Journal of Technology Education, 10(2), 45–59. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v10n2/pdf/lewis.pdf

Li, M., & Shavelson, R. J. (2001, April). Examining the linkage between science achievement and assessment. Paper presented at the American Educational Research Association Annual Meeting, Seattle, WA.

Mehalik, M. M., Doppelt, Y., & Schunn, C. D. (2008). Middle-school science through design-based learning versus scripted inquiry: Better overall science concept learning and overall equity gap reduction. Journal of Engineering Education, 97(1), 71–85. doi:10.1002/j.2168-9830.2008.tb00955.x

National Academy of Engineering. (2008). Grand challenges for engineering. Washington, DC: National Academy of Sciences. Retrieved from http://www.engineeringchallenges.org/File.aspx?id=11574&v=ba24e2ed

National Academy of Sciences, National Academy of Engineering, & Institute of Medicine.. (2007). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Washington, DC: National Academies Press. doi:10.17226/11463

National Academy of Sciences, National Academy of Engineering, & Institute of Medicine. (2010). Rising above the gathering storm, revisited: Rapidly


-18-

approaching Category 5. Washington, DC: National Academies Press. doi:10.17226/12999

National Assessment Governing Board. (2008). Science framework for the 2009 National Assessment of Educational Progress. Washington, DC: Author. Retrieved from https://www.nagb.org/content/nagb/assets/documents/publications/frameworks/science/2009-science-framework.pdf

National Center for Educational Statistics. (2014). The NAEP questions tool. Retrieved from http://nces.ed.gov/nationsreportcard/nqt/

National Research Council. (2010). Exploring the intersection of science education and 21st century skills: A workshop summary. Washington, DC: National Academies Press. doi:10.17226/12771

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press. doi:10.17226/18290

U.S. Congress, Office of Technological Assessment. (1984). Commercial biotechnology: An international analysis. Washington, DC: Author. Retrieved from http://ota.fas.org/reports/8407.pdf

U.S. Congress, Office of Technological Assessment. (1988). New developments in biotechnology: U.S. investment in biotechnology—Special report. Washington, DC: U.S. Government Printing Office. Retrieved from http://ota.fas.org/reports/8840.pdf

U.S. Congress, Office of Technological Assessment. (1991). Biotechnology in a global economy. Washington, DC: U.S. Government Printing Office. Retrieved from http://ota.fas.org/reports/9110.pdf

Partnership for 21st Century Skills. (2009). Framework for 21st century learning. Retrieved from http://www.p21.org/storage/documents/21stcskillsmap_science.pdf

Ryle, G. (1949). The Concept of Mind. New York, NY: Barnes & Noble. Singer, S. R., Nielsen, N. R., & Schweingruber, H. A. (Eds.). (2012). Discipline-

based education research: Understanding and improving learning in undergraduate science and engineering. Washington, DC: National Academies Press. doi:10.17226/13362

Sneider, C. (2012, October). Workshop—Intersection of the three dimensions of science, engineering. Presentation at the Building Capacity for State Science Education (BCSSE) Conference, Indianapolis, IN.

Stotter, D. E. (2004). Assessment of the learning and attitude modification of technology education students who complete an instructional unit on agriculture and biotechnology (Doctoral dissertation). Available from ProQuest Dissertations and Theses Database. (UMI No. 3137129)

Wells, J. G. (1992). Establishment of a taxonometric structure for the study of biotechnology as a secondary school component of technology education (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg, VA.


-19-

Wells, J G., (1994). Establishing a taxonometric structure for the study of biotechnology in secondary school technology education. Journal of Technology Education. 6(1), 58–75. Retrieved from https://scholar.lib.vt.edu/ejournals/JTE/v6n1/pdf/wells.pdf

Wells, J. G. (1995). Defining biotechnology. The Technology Teacher. 54(7), 11–14.

Wells, J. G., (1999). Biotechnology content organizers. Journal of Industrial Teacher Education. 36(4), 70–76. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v36n4/wells.html

Wells, J. G. (2008, November). STEM education: The potential of technology education. Paper presented at the 95th Annual Mississippi Valley Technology Teacher Education Conference, St. Louis, MO. Retrieved from http://www.mississippivalley.org/wp-content/uploads/2015/12/Wells_2008_MississippiValleyConference_STEM-ED_TE-Potential.pdf

Wells, J. G. (2010). Research on teaching and learning in science education: Potentials in technology education. In P. A. Reed & J. E. LaPorte (Eds.), Research in technology education (pp. 192–217). 59th Yearbook of the Council on Technology Teacher Education. Reston, VA: Council on Technology Teacher Education.

Wells, J. (2012, April). Design-based biotechnology: Integrative STEM education. Paper presented at the NSTA 2012 National Conference on Science Education, Indianapolis, IN

Wells, J. G. (2013). Integrative STEM education at Virginia Tech: Graduate preparation for tomorrow’s leaders. Technology and Engineering Teacher, 72(5), 28–35.

Wells, J., (2014, March). Design based biotechnology: Evidencing cognitive demands. Paper presented to the Council of Technology and Engineering Teacher Education of the International Technology and Engineering Educators Conference, Orlando, FL.

Wells, J. G. (2015). Design based biotechnology literacy: Teaching guide (11th ed.). Blacksburg, VA: Technology Education Biotechnology Curriculum Project, Biosens.

Wells, J. G., & Kwon, H. (2008). Inclusion of biotechnology in US standards for technology literacy: Influence on South Korean technology education curriculum. In M. J. de Vries (Ed.), Proceedings of the 19th Annual Pupils Attitude Toward Technology Conference (pp. 315–334). Reston, VA: International Technology and Engineering Educators Association. Retrieved from http://www.iteea.org/File.aspx?id=86750&v=680f4f6a

Wells, J., G. & Kwon, H. (2009). Research trends and issues regarding biotechnology inclusion in technology education: A meta-analysis of relevant literature. Korean Journal of Technology Education, 9(1), 257–278.


-20-

WestEd. (2012).Technology and engineering literacy assessment and item specifications for the 2014 national assessment of educational progress (Pre-Publication Edition). San Francisco, CA: Author.

Zuga, K. F. (1994). Implementing technology education: A review and synthesis of the research literature. Columbus, OH: Center on Education and Training for Employment. Available from ERIC database. (ED372305)

About the Author

John G. Wells ([email protected]) is Associate Professor in the Integrative STEM Education Program at Virginia Polytechnic Institute and State University.


-21-

Positioning Technology and Engineering Education as a Key Force in STEM Education

Greg Strimel & Michael E. Grubbs

Abstract

As the presence of engineering content and practices increases in science education, the distinction between the two fields of science and technology education becomes even more vague than previously theorized. Furthermore, the addition of engineering to the title of the profession raises the question of the true aim of technology education. As a result, the technology and engineering education community must effectively communicate its role in an evolving STEM education landscape. During this time of change, it is important that we understand how the technology education profession has transitioned in the past while we figure out how to balance traditions and contemporary needs. The authors present three pathways that appear most salient in moving forward: (1) adhering to the fundamental goals of technology education, (2) collaborating with science education to potentially become a core discipline, or (3) revitalizing the field through a shift to engineering education. A final recommendation is made to energize the field by centering on becoming a true provider of K–12 engineering education. Keywords: technology and engineering education; science education; STEM; engineering.

The philosopher Eric Hoffer (1973) once reflected that “In a time of drastic change it is the learners who inherit the future. The learned usually find themselves equipped to live in a world that no longer exists” (p. 22). As Hoffer generalizes consequences of responding to mass movements of change, he illustrates the shortcomings of remaining stagnant and committed to previously held views. The efficacy of his quote for the field of technology and engineering education is the parallel it draws with STEM educational reform and responses to the Next Generation Science Standards (NGSS). Specifically, the infusion of engineering content and practices into science education further weakens the already vague distinction between the fields of science and technology and engineering education.

Although the International Technology Educators Association explicitly included engineering and design in the Standards for Technological Literacy 15 years ago, it is now the NGSS that is recognized and critiqued by organizations such as the American Society for Engineering Education (ASEE). Concerns have largely been directed towards science educators’ ability to appropriately


-22-

and effectively incorporate engineering content into science education (Buchanan, 2013; Hosni, 2013). However, the engineering communities have expressed support and recommendations for science educators’ use of engineering at the K–12 level. In turn, as the ASEE has acknowledged the rise of K–12 engineering education standards, they have endorsed approaches for adequately preparing and supporting “the educators who will teach engineering in K-12 classrooms, many of whom have no experience in engineering” (Engineering4Kids, 2015, para. 1). This has resulted in the creation of resources to assist K–12 teachers who wish to teach engineering. Although such documents are aimed at all teachers, it is the NGSS that is frequently cited, the ITEEA community or the Standards for Technological Literacy are only referenced minimally. Perhaps this displays the engineering communities’ confidence in technology and engineering educators’ ability to deliver engineering content, or rather, there exists little recognition of the technology school subject as a viable pathway for engineering.

There is no doubt that the architects of technology and engineering education are confronted with a daunting task of adequately preparing for an evolving landscape. The authors of this paper recognize the urgency of this challenge. Therefore the intent of this article is to promote discussion at a time when technology and engineering education is presented with multiple avenues in response to the adoption of engineering into science education. Although this article includes commentary on past responses of technology and engineering education to change, we hope that this article will evoke discussion that will lead to the selection of viable pathways for the future.

Change and Evolution Similar to the evolution and progress of technology over the past 100 years,

change has been synonymous with the field of technology and engineering education (Hill, 2006; Lewis, 2004, 2005; Sanders, 2001). Over time, changes in technology and engineering education, often related to the dominant industries of the time (Grubbs, 2014), affected the aim, objectives, curricula, and instructional practices of the school subject. Presently in the United States, educational initiatives in STEM, focus on transdisciplinary teaching and learning, the Next Generation Science Standards, the ASEE Standards for Preparation and Professional Development for Teachers of Engineering, and the National Assessment of Education Progress Technology and Engineering Literacy Assessment are but a few examples promoting a shift towards engineering (Strimel, 2014b). Much like the industrial arts profession shifted to instruction on how technology affects people and the world in which we live, the technology and engineering education subject is situated within an opportunistic context for truly implementing engineering in the K–12 school setting.


-23-

Transitioning to Technology Education A review of the transition to technology education reveals that individuals

took multiple approaches when moving forward. For example, Foster (1994) reflected on three perspectives originally identified by Pullias (1989) that individuals could have taken when implementing technology education. The first view was a revolutionary position focused on discarding the old and beginning fresh (Pullias, 1989). In retrospect, this would have been removing industrial arts completely and focusing on technology education. Secondly, the evolutionary position was when an individual preferred to keep a portion of the old, while implementing components of the new, and easing into full enactment (Pullias, 1989). This might have been comparable to still teaching industrial processes while including open-ended problem solving and better aligning with the general education disciplines. The third position was merely masking what has been done previously with a new façade or veneer (Pullias, 1989). Although all three views examined a previous initiative of transitioning to technology education, the present focus on engineering, both within science and technology education, implies comparable routes during implementation.

Similar to Pullias’ (1989) observations, the authors of this article recognize multiple implementation opportunities for engineering and identify three pathways that have seemed to present themselves. First, technology and engineering education can stay the course, continuing what has been done in the past and focusing on general technological literacy. This is similar to Pullias’s first perspective. Second, considering the close relationship from implementing engineering design, the technology and engineering education profession can further collaborate with science education, finding distinctions that clarify the differences between both fields. The last, and perhaps the most viable, option is to work with the engineering and engineering education community to establish engineering education as the primary pathway for engineering content and practices.

The purpose of this article is to bring forth promising ideas with the intent to start and continue the conversation for the future of technology and engineering education. Although the authors believe that these are not the only options that exist, they do agree that in times of change it is important to determine what is essential because “what was essential before may not be crucial now or in the future. All that we can predict is that change will happen” (Starkweather, 2005, p. 1). Balancing Traditions and Contemporary Needs

One challenge the field of technology and engineering education faces is maintaining the balance of traditions and contemporary needs. As K–12 engineering in the United States gains increased attention during STEM educational reform, addressing the traditions and contemporary needs becomes a challenge. Nearly 20 years ago, Martin (1996) commented on the challenges


-24-

faced by one industrial teacher education organization as technology education further entered the K–12 arena:

Because people create change, they must accept that there can be no perfect or permanent solutions. Similarly, finding a balance between the great traditions of the Mississippi Valley Industrial Teacher Education Conference (MVITEC) and the contemporary need of its members has no perfect or permanent solutions. In fact, finding an appropriate balance is like shooting a moving target. The balance will change hourly, daily, monthly, and yearly, and members of MVITEC must be prepared to adapt constantly. Their willingness to adapt and the methods they choose will clearly determine the very future of MVITEC. (p. 39)

A key point drawn from Martin (1996) is that there may not exist one identifiable path to meet all of the underpinnings of early industrial arts and technology education beliefs while engineering education gains significance. Rather, finding a balance between traditions of the past and contemporary needs of educators, teacher education programs, and students can provide a solid foundation for the field of technology and engineering education. Technology Education: Staying the Course

The most convenient path, the path of least resistance, is staying the course of technology education. In this context, technology education, rather than technology and engineering education, is used to allude to the issue at hand of merely adding the term engineering. This would call for little modification to the standards, curricula, and philosophical orientation of technology and engineering education. For example, early publications such as A Conceptual Framework for Technology Education (Savage & Sterry, 1990) have presented a sample philosophy of technology education as providing

Students of all grades, abilities and backgrounds with technological knowledge, skills, and attitudes necessary to become competent, contributing, and productive members of society. Through experiences in a “hands-on” cooperative environment using a systematic, problem-solving approach, students should exhibit understanding of all domains relating to technology. (p. 27)

Yet, since the addition of engineering to the title of technology and engineering education, current definitions, such as the following definition, are synonymous to early conceptualizations of the role of the discipline.

Technology and engineering education is committed to preparing students for employment and/or continuing education opportunities by teaching them to understand, design, produce, use, and manage the human-made world in order to contribute and function in a technological society. (Utah State Office of Education, 2010, para. 1).


-25-

Consequently, the current path of technology and engineering education

might be one of tradition that cannot meet recent criticism of this path that emphasizes the need to truly teach engineering rather than only adjust slightly to bumps or changes in the road. For example, staying the course does not account for the ongoing discussion of ambiguity and confusion around the term technology education. Dugger and Naik (2001) discuss the common misperception that technology education is simply computers, electronics, or educational technology. Although the mission, vision, instructional approaches, and learning outcomes of technology and engineering education are understandable to most practitioners, it is doubtful that the general populace has the same understanding of this school subject as they have regarding other core educational disciplines. Therefore, the question raised is whether theoretical understanding is more important to practitioners or if practical, immediate understanding of the overall population is a more important outcome.

Another issue the technology and engineering education profession is currently facing is the declining numbers within the discipline. Specifically, the number of “technology & engineering teacher preparation programs at colleges and universities in the United States have been in a state of decline since the 1970’s” (Litowitz, 2014, p. 73). Likewise, between 2002 and 2012, studies reported that the total number of programs nationwide preparing technology teachers has dropped from 40 to 24 programs (Bell, 2002; Litowitz, 2013; Rogers, 2012). In 2013, Strimel surveyed teachers who attended training to teach the International Technology and Engineering Educators Association’s Engineering byDesignTM curriculum and reported that nearly 70% of these teachers did not hold a degree in technology education. Furthermore, Strimel reports that over 20% of the teachers preparing to teach the Engineering byDesigntm curricula were not certified in teaching technology education. Anecdotally, one author of this article reports on the status of a metropolitan Atlanta school district containing only a small fraction of teachers who were traditionally certified in technology education, a large subset of whom were alternatively certified with little overall understanding of the scope of the technology and engineering education profession, and others who held certification in engineering with little educational experience. As a result, there are a limited number of individuals in the profession who fully understand technology and engineering education and who are able to promote its practices to progress the profession forward. Although recent initiatives to develop or sustain existing technology education programs have been conceptualized, such as Savanah State University, minimal approaches to sustain technology teacher education programs have arisen. However, viable options in relation to engineering education and possible partnerships will be discussed later in this article.


-26-

Science Education: Playing Nice in the Sandbox Although the similarity between science and technology has long been

discussed in educational literature (Gardner, 1994; Lewis, 2006), the recent release of the NGSS has further overlapped both disciplines. Specifically, the NGSS promotes the raising of engineering design to the same level of importance as scientific inquiry in science education frameworks (NGSS Lead States, 2013). As a result, science education and technology and engineering education now share a signature component. Moreover, as science education increasingly implements resources that were once exclusive to technology and engineering education, such as robotics, and recommends moving away from cheap, resourceful activities such as egg drops (Milano, 2013), technology and engineering education might proceed in collaboration with science education or otherwise potentially lose its own identity as a school subject.

As engineering design is implemented in science education, the opportunity arises for technology and engineering education to partner with science education for truly transdisciplinary approaches to Integrative STEM Education. Rather than being used only as a tool to teach science and assist in students working through scientific inquiry, technology and engineering educators can build ongoing collaborations that promote integration at the natural intersection of each discipline. For example, finding domains that require scientific inquiry and engineering design, such as biotechnology, provides opportunities for each discipline to contribute equally. For instance, existing biotechnology units such as the construction of a Microbial Fuel Cell (Wells, 2013) requires students to work through scientific inquiry to discover new scientific knowledge of ideal settings for bacteria to grow; those contributions would contribute to the engineering design process. Without knowledge of both disciplines, teachers might inadvertently situate students in a context that does not intentionally teach concepts from both disciplines.

Technology and engineering education has a great deal to offer the science education field as it moves towards more authentic educational approaches. Existing programs can work to support the teaching of science concepts and practices by providing a laboratory setting for the designing and making of new products and processes necessary to carry out realistic scientific investigations. Technology and engineering teachers are often more equipped and well trained for the acts of designing and making. These acts can be thought of as the kernel of technology and engineering education and can be considered what the profession does best. Therefore, technology and engineering programs are more often than not equipped with industry quality tools, materials, and equipment that can be used in conjunction with science education to advance student learning. The physical acts of designing and making while using current industry quality resources, can provide students with the experiences necessary for working in STEM-related careers. Additionally, the resources and abilities that technology and engineering instructors have, including lab safety, knowledge of


-27-

material processing, and correct tool use, can aid in the scientific examination of problems facing the world. In turn, these scientific investigations can then enable students to develop authentic solutions to these real-life issues using the process of engineering design.

As engineering increasingly enters the instructional practices of science educators, this path of cooperation with science educators appears as a viable option in moving forward technology and engineering education. Moreover, the technology and engineering education profession should collaborate with science education because it is a much larger profession that could assume responsibility for teaching engineering, leaving technology education without a place in a student’s general education. Science education is not only recognized and understood as a core educational subject, but it also provides a context for technology and engineering education students to apply knowledge and skills previously learned. Working closely with science education may provide a solid place for technology and engineering education in local school systems. This place can be where students actually utilize industry quality technologies to “make” solutions to engineering design problems, replacing less authentic classroom activities requiring only the use of unrealistic materials, such as Popsicle sticks, cardboard, duct tape, and hot glue.

A challenge for technology and engineering education in most states is the determination of where it fits within a student’s education. Since its historical beginning, the purpose of technology and engineering education was to provide all students the knowledge, skills, and abilities to function in a technological world. However, many states have organized technology and engineering education under the umbrella of career and technical education. As a result, technological and engineering literacy has been missing from many students general education, and many technology and engineering programs lack the necessary enrollment from all student populations to sustain the subject. Now that the NGSS includes engineering and technology as one of the core disciplines for science, the technology and engineering profession can use this to solidify its spot in the Unites States education system by leveraging the support of the much larger science education profession. This being said, some questions for the technology and engineering education profession to ponder are: (a) What if technology and engineering education becomes a core discipline of the science education profession? (b) Can technology and engineering education utilize science as a means to bring technological and engineering literacy to all students? (c) What if teacher preparation programs enable science teachers to specialize in engineering or technology much like one can specialize in chemistry or physical science? These are questions that may help guide future directions for technology and engineering education. Keep in mind, that a lack of collaboration as a profession may lead to science taking the responsibility of teaching engineering, leaving technology education with little content and practices for a student’s general education.


-28-

Routes for collaboration with science education already exist, including collocated professional organization meetings between science and technology. Yet, in moving forward, technology and engineering education might consider the implications of so closely aligning with science education and the effect that it might have on implementing similar instructional approaches. E-nough is Enough: A Final Call for Engineering

The emphasis on engineering at the K–12 level has been increasing since the turn of the century (Kelley, 2008; National Research Council [NRC], 2009). This expanded interest can be attributed to the idea that engineering education can assist in creating a better educated populace and develop a workforce ready to meet the needs of high-demand careers of the 21st century, thus providing students with the skills necessary for economic success (NRC, 2009). Today there is broad agreement among educational stakeholders that the teaching of STEM subjects in K–12 U.S. schools must be improved to prepare students with the skills necessary for success in this century (National Academy of Engineering [NAE] & National Research Council [NRC], 2014). Due to its natural ability to tie mathematics and science together through solving authentic problems, the inclusion of engineering into K–12 education is now seen as an approach to addressing concerns with the U.S. educational system (NAE & NRC, 2014; NRC, 2009). As a result, the NGSS has interwoven engineering practices within its frameworks, and the National Assessment of Educational Progress is now administering a technology and engineering literacy assessment. More recently, K–12 engineering education initiatives, such as the Chevron-funded development of an engineering education community of practice website under a 3-year project called Guiding the Implementation of K–12 Engineering Education, have been surfacing throughout the nation. As a result, engineering education programs such as Engineering is Elementary have seen increased use. However, inconsistencies exist between engineering programs as to what engineering education consists of at the K–12 level, who teaches these engineering programs, how are teachers prepared to teach engineering, how engineering is taught at the K–12 level, and where it is situated within a student’s general education.

The increased emphasis on K–12 engineering and the uncertainty of how it should be taught provide an opportunity for the technology and engineering education profession. The technology and engineering education profession can stake the claim for teaching engineering at the K–12 level, align with the engineering profession, and reform its instructional practices to reaffirm its place in the U.S. educational system. The term engineering is something that is recognized by the general population. Although it may not be fully understood by the broad populace, it is a term and a profession that is generally respected. Adding engineering to technology education brought a refreshing new view on the profession. However, the ambiguity and confusion around the term


-29-

technology continues to hinder the general understanding of the school subject. It can be easy to understand what an engineer is; it is more difficult to explain what a technologist does. Failure to align technology and engineering education with the engineering profession has caused technology and engineering education to continue to lose a foothold within local education systems. Therefore, one possibility for the profession could be a greater emphasis on engineering education and a surrender of the “T,” technology, to the educational technology that the majority of people believe that it is. As a result, the technology and engineering profession would become the provider of K–12 engineering education for all students. However, dropping the “T” will not do anything to revitalize and sustain technology and engineering education on its own. There will need to be significant work as a profession to develop a consistent and comprehensive engineering course sequence, modify preservice teacher programs, create an engineering teaching licensure, and establish clear postsecondary engineering connections and articulation pathways.

The authors of this article believe that a change to engineering education will require the development of a consistent and coherent course sequence. A major concern with technology education has always been the inconsistency of what courses students take and the content and skills that they learn from school to school. These inconsistencies can limit the ability to work as a profession to enhance technology and engineering education. However, with engineering as a focus, a core set of disciplines can be created. Much like science education has courses in physics, biology, chemistry, and earth or space science, engineering education can have coursework in the disciplines of mechanical, electrical, chemical, structural, and biological or medical engineering. These courses can be taken by all students to help better understand the designed world and do not have to focus on preparing students specifically for engineering careers. Just as students in biology class do not have to become a biologist, students in a mechanical engineering course do not have to become a mechanical engineer. However, these courses can provide all students with beneficial knowledge and skills as well as introduce them to engineering careers.

The Project Lead the Way pre-engineering program can be used as an example. The program provides core courses for introducing engineering entitled Introduction to Engineering Design and Principles of Engineering and specialization courses, such as Civil Engineering & Architecture, Biomedical Engineering, and Aerospace Engineering. However, it is still difficult to determine how this fits into a school district and whether it should be a part of student’s general education. Additionally, it can be unclear as to what types of teachers are best prepared to teach these courses.

To be able to teach engineering content, teachers need to be properly prepared. The technology and engineering education profession is sometimes criticized in regards to its ability to teach engineering at the K–12 level and rightfully so. A study conducted by Strimel (2013) showed that over 62% of the


-30-

teachers preparing to teach the Engineering byDesigntm curriculum had never completed a college-level course in trigonometry, and over 64% had never completed a college level course in calculus. Furthermore, the study reported that almost 54% of these teachers never completed a college course in physics, and almost 36% never completed a college course in chemistry and biology. The concepts in these mathematics and science courses are the foundation of theoretical engineering and are necessary for understanding the true concepts and practices of engineering professions. Many technology teacher preparation programs do not require multiple courses in mathematics and science, which is something that must be modified to produce teachers with the knowledge and skills to properly teach engineering. A study conducted by Litowitz (2014) reported there was a wide range of mathematics requirements for technology teacher education programs. The data indicated that approximately 30% of the technology teacher education programs did not require a mathematics course beyond statistics and showed that college algebra was the most frequent mathematics course required for preservice technology teachers. Additionally, his study reported that many institutions allowed technology education majors to choose any natural science course to fulfill their degree requirements.

These minimal degree requirements combined with the wide diversity of technology and engineering teacher education core curricula continues to compromise technology and engineering education. As reported by Litowitz (2014), some programs follow a traditional technology education approach that focuses on materials processing, whereas others have evolved into a more engineering design focused approach. Moreover, a study of high school students pertaining to engineering design cognition conducted by Strimel (2014a) indicated potential disconnects between technology and engineering curricula and the engineering profession. The study’s findings suggested these disconnects may have resulted in students acting and thinking in a way that does not match the engineering practices. The data indicated that students were heavily focused on the act of making a solution based on an initial idea rather than thoughtfully forecasting their designs. The study also portrayed that the majority of the students studied employed a more traditional non-engineering, trial-and-error approach to solving an engineering design problem. These students were observed dedicating little time to analytical designing, modeling, experimenting with the proper materials, and utilizing testing results to optimize their designs. This study may indicate the engineering habits of mind, which involve design, analysis, modeling, and optimization, are not emphasized or correctly practiced throughout technology and engineering curriculum and instruction. Thus, it should be essential that technology and engineering education programs clarify their purpose and, if their purpose is engineering, to enhance their standards, curriculum, and instruction to include the proper engineering practices and content.


-31-

Some teacher preparation programs have modified the curriculum to address these concerns. Some programs now require preservice technology and engineering teachers to complete coursework similar to an engineering major with some additional coursework to earn a teaching license. Some notable examples are The College of New Jersey, Ohio Northern University, and University of Maryland Baltimore County (UMBC). The College of New Jersey suggests a more rigorous sequence of mathematics and science courses for the technology/pre-engineering preservice teachers through the college’s school of engineering. The suggested course sequence provided on their website recommends students complete calculus, engineering mathematics, and general physics within their first year while progressing toward courses in structures, mechanics, analog and digital circuits, and mechanical system design (The College of New Jersey, 2016). Ohio Northern University offers an engineering education program that “directly addresses the need to develop a new generation of high school students who can contribute to solving our nation’s challenges through engineering and innovation” (Ohio Northern University, 2014, para. 2). The program combines a general engineering degree with the required education and mathematics courses to earn a teaching certification. The 4-year engineering education degree prepares graduates to become licensed secondary mathematics teachers but with a more specialized perspective on engineering-design-based learning than teachers who have a traditional education diploma.

UMBC has developed a pre-service teacher program to prepare individuals to deliver pre-engineering curriculum in middle and high schools (University of Maryland Baltimore County, 2016). Their program ties the mechanical engineering program in with Project Lead the Way training to progress towards earning a technology education certification. Lastly, according to Reed and Cantu (2016), Old Dominion University is the first institution to utilize the UTeach program to certify more technology and engineering teachers. They describe the UTeach program as an initiative that seeks to train science, mathematics, computer science, and engineering majors to become a certified teachers while earning their undergraduate degree in their content areas. Therefore, these students may obtain the content knowledge along with the pedagogical knowledge to be effective teachers of engineering and technology.

However, preceding attention to engineering teaching programs, advocacy needs to be addressed at the state level, updating current licensure agreements. Yet, this should be done carefully as to not create two parallel teacher certifications in engineering and technology. Doing so could provide the opportunity for the engineering profession to assume complete control over K–12 engineering education, eliminating the need for any former technology and engineering programs and professionals that provide expertise in authentic pedagogical practices and the use of tools and machines to make products.

Lastly, a change to engineering education requires a clear collaboration and articulation with 4-year engineering programs and 2-year engineering


-32-

technology programs. A K-12 engineering program should expose students to these career pathways and prepare them for a successful transition to postsecondary education. K–12 engineering education programs should collaborate with both engineering and engineering technology programs to ensure that the proper engineering fundamentals are taught throughout the various levels of education. Additionally, a connection between secondary and postsecondary engineering education can enable highly motivated high school students to obtain early college opportunities so they progress through their higher education studies faster and in a more affordable manner. Erekson and Custer (2008)

As we enter the 21st century it is clear that engineering education and technology education have the potential for a symbiotic alliance that will benefit both technology and engineering educators. Engineering educators have become very interested in strengthening the pathways to engineering by linking with K-12 education. At the same time, technology educators have developed national standards for K-12 education that include engineering content. Frankly, it is time to heighten and expand the discussion between the two technology and engineering educations. Collaboration between engineering educators and technology educators is an idea that needs to be further developed and put into practice. (p. 1)

Recommendations for Clarity

In consideration of the three previously discussed pathways, it is vital for the technology and engineering education profession to ruminate on what the purpose and fundamental expectations are for students. Specifically, if the field focuses on engineering education, it is essential that leaders fully consider the impact such a decision will make and that it leads to improvement in student learning. Thus, stakeholders might begin with why this change is most appropriate and what it will mean for teacher preparation, assessment, instructional approaches, and professional development. At the higher education level, additional research needs to be conducted to provide empirical evidence that engineering does indeed result in changes in students’ higher order thinking skills and increases knowledge in other domains. Such research can benefit from partnering with engineering education to ensure proper alignment with the standards of their profession. This partnership can also help enable engineering majors to become interested in teaching K–12 engineering, thus helping fill the shortage of technology and engineering educators. Teacher preparation will also need to be examined to ensure that course work for preservice teachers is consistent and comprehensive across institutions. Teaching engineering content and the engineering design process would benefit if taught through programs supported and approved by engineering education. It is also recommended that preservice engineering teachers be required to complete similar coursework as a typical engineering major in addition to coursework in pedagogy necessary for a


-33-

teaching license. Lastly, structuring pathways between K–12 schools and engineering programs would remove some of the ambiguity currently associated with technology education.

Conclusion In conclusion, the technology and engineering education profession has

multiple paths for moving forward. What appears most viable and sustainable to the authors of this article is to focus truly on engineering as a core disciplinary subject, which may help remove the long-standing confusion with technology. This shift will require increased rigor in mathematics and science applications, predictive analysis, analytical modeling and design, and executive functioning, such as decision-making, task initiation, organization, planning and prioritizing, and flexible thinking. However, the focus should ensure student opportunity for “making” or producing quality outputs that the profession may have shifted away from. Although in theory engineering has been added to the field and in practice is instilled in technology education classrooms across the country, it remains a façade or buzzword for many and has not been intentionally introduced, nor aligned to the engineering community. Most teachers are not professionally certified to teach engineering and have received little professional development to prepare them to do so. Subsequently, the thoughts and actions of technology and engineering students may not coincide with the practices of the engineering profession (Strimel, 2014a). Yet, in light of declining technology education programs and challenges for engineering education to retain students, an opportunity exists for moving forward.

As Foster (1994) suggested, when transitioning to technology education, The challenge in interpreting past practice is not to criticize it in an attempt to inflate the value of that perceived as new. It is to learn from it in an attempt to recognize the value in that established as eminent.” (p. 27).

For the technology and engineering education profession, it can’t do it all;

even in ideal aspects for what technology education does, it can’t be the best at everything. Engineering is recognized not only as a career that students can identify with but also provides the core outcomes for technology education, including problem-solving; hands-on, minds-on, creation of a product; and authentic, meaningful learning opportunities. A decision as large as changing the nature of a discipline is not one that should be made lightly. Rather, input from stakeholders, discussion on viable solutions, and consideration of the effect of such a change should all be considered.

Lastly, as Martin (2005) reflected on the origin of technology education he suggested some “individuals who provided the initial major impetus for technology education worked in isolation from their colleagues, while others worked in tandem” (para. 1). Although present opportunities of working with multiple STEM disciplines exist, it might be too early to determine if a similar


-34-

situation will arise as technology and engineering education moves forward. Yet, in any direction, collaboration is vital to the success of the technology and engineering education profession and should be considered.

References Bell, T. P. (Ed.). (2002). Industrial teacher education directory (41st ed.).

Millersville, PA: Millersville University of Pennsylvania, Department of Industry and Technology, CTTE and NAITTE.

Dugger, W. E., Jr., & Naik, N. (2001). Clarifying misconceptions between technology education and educational technology. Technology Teacher, 61(1), 31–35.

Engineering4Kids. (2015). Teaching K-12 engineering. Retrieved from https://engineering4kids.wikispaces.com/Teaching+K-12+engineering

Erekson, T. L., & Custer, R. L. (2008). Conceptual foundations: Engineering and technology education. In R. L. Custer, & T. L. Erekson (Eds.), Engineering and technology education (pp. 1–12). 57th Yearbook of the Council on Technology Teacher Education. Woodland Hills, CA: Glencoe.

Foster, P. N. (1994). Technology education: AKA industrial arts. Journal of Technology Education, 5(2), 15–30. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v5n2/pdf/

Gardner, P. L. (1994). The relationship between technology and science: Some historical and philosophical reflections. Part I. International Journal of Technology and Design Education, 4(2), 123–153. doi:10.1007/BF01204544

Grubbs, M. E. (2014). Genetically modified organisms: A design based biotechnology approach. Technology and Engineering Teacher, 73(7), 24–29.

Hill, R. B. (2006). New perspectives: Technology teacher education and engineering design. Journal of Industrial Teacher Education, 43(3), 45–63. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v43n3/pdf/hill.pdf

Hoffer, E. (1973). Reflections on the human condition. New York, NY: Harper & Row.

Kelley, T. R. (2008). Cognitive processes of students participating in engineering-focused design instruction. Journal of Technology Education, 19(2), 50–64. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v19n2/pdf/kelley.pdf

Lewis, T. (2004). A turn to engineering: The continuing struggle of technology education for legitimization as a school subject. Journal of Technology Education, 16(1), 21–39. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v16n1/pdf/lewis.pdf

Lewis, T. (2005). Coming to terms with engineering design as content. Journal of Technology Education, 16(2), 37–54. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v16n2/pdf/lewis.pdf


-35-

Lewis, T. (2006). Design and inquiry: Bases for an accommodation between science and technology education in the curriculum? Journal of Research in Science Teaching, 43(3), 255–281. doi:10.1002/tea.20111

Litowitz, L. S. (2013, December). An analysis of undergraduate technology & engineering teacher preparation programs in the United States. Paper presented at the Pupils Attitudes Towards Technology Conference, Christ Church, New Zealand.

Litowitz, L. S. (2014). A curricular analysis of undergraduate technology & engineering teacher preparation programs in the United States. Journal of Technology Education, 25(2), 73–84. Retrieved from https://scholar.lib.vt.edu/ejournals/JTE/v25n2/pdf/litowitz.pdf

Martin, G. (2005, April). From ideas to fruition—External funding and its impact on technology education. Paper presented at the meeting of the Pupils Attitudes Towards Technology conference, Haarlem, the Netherlands. Retrieved from http://iteea.org/Activities/1307/PATTConferences/52403/52448.aspx

Martin, G. E. (1996). Balancing traditions and contemporary needs. Journal of Technology Studies, 22(2), 39–42. Retrieved from http://scholar.lib.vt.edu/ejournals/JOTS/Summer-Fall-1996/PDF/7-special-section.pdf

National Academy of Engineering & National Research Council. (2014). STEM integration in K–12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. doi:10.17226/18612

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press. doi:10.17226/18290

National Research Council. (2009). Engineering in K–12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. doi:10.17226/12635

Ohio Northern University. (2011, March 8). Ohio Northern University meets critical need in engineering teacher prep: University unveils plans to launch one of nation’s first undergraduate degrees in engineering education [Press release]. Retrieved from http://www.onu.edu/page/33420

Pullias, D. (1989). Where do we go from here? ATTE Journal, Winter, 3–4. Reed, P. A., & Cantu, D. V. (2016, March). MonarchTeach. PowerPoint

presentation at the International Technology and Engineering Educators Association 2016 conference, National Harbor, MD.

Rogers, G. E. (Ed.) (2012). Engineering & technology teacher education directory (51st ed.). West Lafayette, IN: Purdue University, College of Technology, CTTE and NAITTE.

Savage, E., & Sterry, L. (Eds.). (1990). A Conceptual framework for technology education. Reston, VA: International Technology Education Association.


-36-

Starkweather, K. N. (April, 2005). PATT 15: A retrospective look at what was essential during the past 20 years. Paper presented at the meeting of the Pupils Attitudes Towards Technology conference, Haarlem, the Netherlands. Retrieved from http://iteea.org/Activities/1307/PATTConferences/52403/52430.aspx

Strimel, G. (2013). Engineering by DesignTM: Preparing STEM teachers for the 21st century. In P. J. Williams & D. Gedera (Eds.), Technology education for the future: A play on sustainability (pp. 447–456). Christchurch, New Zealand: Technology Environmental Science and Mathematics Education Research Centre, University of Waikato.

Strimel, G. J. (2014a). Engineering design: A cognitive process approach (Doctoral dissertation). ProQuest Dissertations and Theses database. (UMI No. 3662376)

Strimel, G. (2014b). Shale gas extraction: Drilling into current issues and making STEM connections. Technology and Engineering Teacher, 73(5), 16–24.

The College of New Jersey. (2016). Bachelor of Science (BS) in technology/pre-engineering education. Retrieved from http://technologicalstudies.tcnj.edu/ curriculum/bachelor-of-science-bs-in-technology-pre-engineering-education/

University of Maryland Baltimore County. (2016). Project lead the way hosted by University of Maryland, Baltimore County. Retrieved from http://pltw.umbc.edu/

Utah State Office of Education. (2010). Technology & engineering education: Hands-on, minds-on education. Retrieved from http://schools.utah.gov/cte/tech/

About the Authors

Greg Strimel ([email protected]), is Director of K-12 Initiatives and Academic Innovation at West Virginia University. Michael E. Grubbs ([email protected]), is Supervisor of Technology and Engineering Education for Baltimore County Public Schools.


-37-

Problem Decomposition and Recomposition in Engineering Design: A Comparison of Design

Behavior Between Professional Engineers, Engineering Seniors, and Engineering Freshmen

Ting Song, Kurt Becker, John Gero, Scott DeBerard,

Oenardi Lawanto, & Edward Reeve

Abstract The authors investigated the differences in using problem decomposition

and problem recomposition between dyads of engineering experts, engineering seniors, and engineering freshmen. Participants worked in dyads to complete an engineering design challenge within 1 hour. The entire design process was video and audio recorded. After the design session, members participated in a group interview. Video and audio data were transcribed, segmented, and coded to make comparisons. Results show differences between engineering experts, seniors, and freshman in design thinking. Students tend to use depth-first decomposition, and experts tend to use breadth-first decomposition in engineering design. The results also show that students spend less cognitive effort on the problem-definition stage than engineering experts. Keywords: engineering design; problem decomposition and recomposition; design thinking; expertise.

Design is recognized as the critical element of engineering thinking which differentiates engineering from other problem-solving approaches (Dym, Agogino, Eris, Frey, & Leifer, 2005). One of the primary goals of engineering design education is to equip students with the capability to become expert design engineers. To develop this capability in students, educators require a detailed knowledge of the cognitive behavior of both undergraduate students and expert design engineers. However, there is insufficient information about the cognitive behavior of expert design engineers because most studies are focused on individual student engineers or early professional engineers.

Engineering design is fundamental for engineering graduates because engineering design is a major skill required of practicing engineers. The use of design strategies plays a significant role in engineering design, and a commonly used strategy is problem decomposition or recomposition. It is frequently used by experienced engineers, especially for solving complex engineering problems (Vincenti, 1990). The process of problem decomposition involves breaking the design problem into smaller independent subproblems (Arvanitis, Todd, Gibb, &


-38-

Orihashi, 2001). Each subproblem can be further broken into even smaller problems (Arvanitis et al., 2001), and the decomposition process stops when designers can directly approach each subproblem. Problem recomposition is a bottom-up process that usually comes with problem decomposition. It is a process of recomposing all subsolutions (Chandrasekaran, 1990) in the premise of satisfying requirements of the combining design (Hall, Jackson, Lanney, Nuseibeh, & Rapanotti, 2002). Instead of focusing on a complex design problem as a whole, engineers can work on several smaller, more approachable subproblems using this process, which makes the process of engineering design more efficient. Studies have identified a gap between engineering novices and engineering experts when it comes to problem decomposition and recomposition skills in engineering design (Ball, Evans, & Dennis, 1994; Ho, 2001; McCracken, 1997).

To the extent that past works are available (e.g., Ball et al., 1994; Ho, 2001; McCracken, 1997), most studies about problem decomposition or recomposition have focused on individuals instead of groups. However, in the real world, engineers usually work in groups to solve engineering problems. By investigating this topic in the context of collaborative engineering design, researchers can have a better understanding of the development of expertise and the use of problem decomposition or recomposition in practical settings. Design is a creative, open-ended, and experiential process that aims at problem solving. Engineering design is a central part of engineering and has been emphasized as a focus for engineering education for several decades (Dym et al., 2005). Engineering design challenges are widely and effectively used in teaching engineering and in engineering education research. Engineering design challenges can be used in both formal academic circumstances and informal settings. Dym, Agogino, Eris, Frey, and Leifer (2005) believed that engineering design challenges could benefit student learning in many ways. In theory, these challenges should include the entire engineering design process, but practical engineering design challenges are extremely complex and ill structured. Classrooms educators sometimes only incorporate parts of the design process based on the needs of the curriculum (Atman et al., 2007; Katehi, Pearson, & Feder, 2009).

Research Design To investigate the gap that exists between skills developed in universities

and skills needed in the industry to become an expert engineer, a pilot study was conducted. The research question guiding this research is: In the process of engineering design, how do experts approach the design problem differently from engineering students?


-39-

Sample In this study, participants were selected using a convenience sampling

method (Gall, Gall, & Borg, 2007). Fifty participants took part in this study, including 20 college engineering freshmen, 20 engineering seniors, and 10 engineering experts. All of the participants worked in dyads. It should be noted that this research was a pilot study; therefore, data were collected from a small sample. Results of the quantitative data show preliminary findings only and cannot be generalized because the N size is small. Design Challenge

All dyads completed the same open-ended engineering design challenge. The design challenge used was a double-hung window opener that would assist the elderly in raising and lowering windows. This design challenge has been used by other researchers to study engineering design (Gero 2010; Lammi & Becker, 2013). There were various engineering and social constraints in this challenge, which made it a typical engineering design challenge. During the design session, participants had access to only five websites related to the design challenge. Participants had limited access to prevent them from searching for solutions to the design problem. They were recommended 1 hour to complete the design challenge. Participants only submitted design proposals as their final outcome. Participants received no instruction about the form or the content of the proposals. They did not build, test, and analyze their design because of the time constraint. Data Collection

The primary form of data collection was protocol analysis. In the process of engineering design, conversation happened naturally within the dyads. The researcher used audio and video recording to capture participants’ conversations and their nonverbal interactions. The researcher did not answer participants’ questions. The audio and video data complemented each other to provide rich information about the conversations and actions in engineering design process. The protocol analysis of design sessions was coded based on the function–behavior–structure (FBS) coding scheme. This ontology provides a set of irreducible foundational concepts of design and designing, which covers the acts of designing and the representation of the design. The definition and conceptualization of the ontology is illustrated in Figure 1. Function (F) represented designers’ expectations of the products, behavior (B) represented the ways that designers accomplish their goals, and structure (S) represented the solutions to the problem.


-40-

Figure 1. FBS framework (Resource: Gero, Kan, & Pourmohamadi, 2011).

The design actually is a consequence of a series of processes including the above FBS variables:

1. Formulation (process 1) transforms the design requirements, expressed in function (F), into behavior (Be) that is expected to enable this function.

2. Synthesis (process 2) transforms the expected behavior (Be) into a solution structure (S) that is intended to exhibit this desired behavior.

3. Analysis (process 3) derives the “actual” behavior (Bs) from the synthesized structure (S).

4. Evaluation (process 4) compares the behavior derived from structure (Bs) with the expected behavior to prepare the decision if the design solution is to be accepted.

5. Documentation (process 5) produces the design description (D) for constructing or manufacturing the product.

6. Reformulation type 1 (process 6) addresses changes in the design state space in terms of structure variables or ranges of values for them if the actual behavior is evaluated to be unsatisfactory.

7. Reformulation type 2 (process 7) addresses changes in the design state space in terms of behavior variables or ranges of values for them if the actual behavior is evaluated to be unsatisfactory.

8. Reformulation type 3 (process 8) addresses changes in the design state space in terms of function variables or ranges of values for them if the actual behavior is evaluated to be unsatisfactory. (Gero & Kannengiesser, 2004, p. 3)


-41-

Under the FBS ontology, there was another coding system to represent the level of the problem. Typically, engineers decompose the design problem into multiple subproblems and work on each subproblem in order to find a solution. The level of the problem ranged from 1 to 3. The meaning of each number is shown in Table 1. Gero and Mc Neill (1998) adopted this coding system in analyzing design protocols. Ho (2001) used a similar coding system to investigate engineering design strategies used by individual electrical engineers. Few studies have used levels of the problem to code; as such, there is little data available for this study, but Ho (2001) collected very similar data. Table 1 Level of the Problem

Level of the problem Definition

1: System Designers focused on the problem as an integral whole.

2: System and subsystems Designers focused on interactions between subsystems.

3: Subsystems Designers focused on details of the subsystems.

Immediately after completing the design challenge, participants took part in

a focus group interview in which they answered questions. During this semistructured interview, the researcher asked questions about how participants framed the problem, generated alternative solutions, reached agreements, and used strategies. Table 2 (continued on next page) shows the guiding interview questions. Participants’ sketches were also collected as a data resource. Table 2 Interview Guiding Questions

Number Interview Question

1 How did you define the problem? 2

How did you decide what information to get?

3 How did you develop or come across different ideas (solutions)?

4 How did you know which ideas would work and which would not work?


-42-

5 Why and how did you choose your final idea or plan?

6 Is there anything else you needed or wanted that would have helped you?

7

Did you tackle the problem as a whole or decompose it into several subproblems? If you decomposed it, why did you choose it over the other one

8

What difficulties did you meet in solving the problem?

Data Analysis

Coder training. Prior to analyzing data, two coders were trained to use the coding systems in order to reach an ideal intercoder reliability. They learned the coding systems and started coding sample data from previous studies separately. After coding separately, they compared their codes and calculated the percentage of the codes that they coded the same, which was the intercoder reliability. They also discussed the segments that they coded differently to reach a consistent understanding of the coding scheme. They repeated this process for several rounds until the intercoder reliability remained above 80%. In the social sciences, 70% intercoder reliability is acceptable (Schloss & Smith, 1999).

Data transcribing and segmenting. After participants completed the design challenge, the researcher manually transcribed participants’ conversations and movements into spreadsheets. The spreadsheet data containing participants’ conversations and movements were further broken into segments based on design issues. Each segment is a coding unit and can only contain one code.

Coding. Coders started the coding process by using the FBS ontology. Table 3 shows a piece of coded data and how coders arbitrated data. After all data were coded using the FBS ontology, Codes D, R, and O were excluded from being coded before coding the level of the problem because they do not pertain to levels of the problem. Code O is about other issues that are not related to design cognition. Code R is the requirement that is given to designers. Code D is the documentation process; the spread sheet didn’t record the content of designers’ sketches or drawing, so it was excluded as well. Table 4 shows a piece of data coded by two coders. The coding of “levels of the problem” was based on codes of the FBS ontology. The last column shows arbitrated codes, which are also the final codes of “levels of the problem.”


-43-

Table 3 Example of FBS Codes and Arbitrated Codes

Subject Utterance FBS:

Coder 1

FBS : Coder

2

FBS: Final code

A Not what I'm asking, but like how in-depth?

F R F

A Because that's like how I'm in senior drawing ...

O O O

A Like a pulley is just something you go to the store and buy. Like you… You know...Based on, like

R S R

A I don't think we are given all the numbers that we need to be able to figure what type of pulley system or what gear ratio.

S S S

B Yes, and the cost of materials S S S

A Yes, I mean, I wondering this is more just given what we are given. I don’t think we really can say we need a specific number

R R R

B I see, like 15 or whatever the number type going here

S S S


-44-

Table 4 Sample of Codes for Levels of the Problem

Subject Utterance FBS: Final code

Levels of the

problem: Coder 1

Levels of the

problem: Coder 2

Levels of the

problem: Final code


F 1 1 1


O – – –

A Like a pulley is just something you go to the store and buy. Like you. You know...Based on, like

R – – –


S 3 3 3

B Yes, and the cost of materials

S 3 1 1

A Yes, I mean, I wondering this is more just given what we are given. I don’t think we really can say we need a specific number

R – – –


R – – –


-45-

The process of problem decomposition and problem recomposition was identified by the change of the level of the problem. Table 5 (continued on next page) shows a piece of sample data of an individual dyad in which problem decomposition and problem recomposition were coded. As previously illustrated, the problem decomposition is a top-down process, whereas the problem recomposition is a bottom-up process. When the level of the problem transitions from a higher level to a lower level, it is defined as the problem decomposition, and when it transitions from a lower level to a higher level, it is defined as the problem recomposition. Table 5 Example of Problem Decomposition and Problem Recomposition

Subject Utterance FBS final code

Levels of the

problem final code

Decomposition/ recomposition


F 1 –


O – –

A Like a pulley is just something you go to the store and buy. Like you.. You know...Based on, like

R – –


S 3 D

B Yes, and the cost of materials

S 1 R

A Yes, I mean, I wondering this is more just given what we are given. I

R – –


-46-

don’t think we really can say we need a specific number


R – –

The numbers of utterances generated by each dyad were different, so simply

comparing the frequencies of each type of code would affect the validity of the study. The percentages of codes were used in order to compare the differences between dyads. The percentage of each code from each dyad was calculated by dividing the frequency of the code into the total number of effective codes of the dyad.

Interview and sketch data. The analysis of qualitative data was connected with quantitative data. The constant comparative method (Glaser, 1965) was adapted in analyzing data. The first step included going through each interview and sketch and themes were recorded. The second step was sorting themes to different categories. The third step was looking for differences between expert dyads, senior dyads, and freshman dyads within each category. The analysis of qualitative data allowed any themes or new phenomena to emerge that could not be discovered by analyzing quantitative data. Table 6 provides a small piece of interview data in order to show the first three steps of constant comparative method used in analyzing qualitative data. There were many themes and categories that emerged from the entire qualitative data, and some themes emerged repeatedly. The conclusions drawn in Step 3 were based on the analysis of the entire data instead of the small piece shown in Table 6 (please note that Table 6 has been divided into several cells for readability purposes). The fourth step of analyzing qualitative data was writing the theory which is not shown in Table 6. The analysis of sketches followed the same four steps.


-47-

Table 6 Example of Qualitative Data Analysis

Interview Question

Step 1: Themes Emerged from Answers

F- Freshmen S – Seniors E - Engineers

Step 2: Categorizing

Step 3: Comparing students and

engineers

How did you define the problem?

Increase the force (F) Problem definition

Problem definition: Engineers understood the problem better than students.

Modify existing window (F)

Problem definition

ADA Guidelines (S)

Problem definition

Old people in wheel chairs (S)

Problem definition

ADA requirements (E)

Problem definition

No major construction required (E)

Problem definition

Safety issues (E)

Problem definition

How did you know which ideas would work and which would not work?

Daily experiences (F)

Design experiences

Design Experiences: Engineers had more design experiences and intent to use their experience in solving new problems.

Don’t know if the ideas would work and how much it would cost (F)

Cost

Analyze pros and cons (S)

Alternative solutions

Comparing the design with devices used before (E)

Design experiences

Analyze clients (E)

Problem definition


-48-

Why and how did you choose your final idea or plan?

Practical and feasible (F) Alternative solutions

Cost: Students did not pay enough attention to the cost of the design. Some of them did not know the cost of materials.

Easy to work (F)


Consider how solutions fit requirements (S)


Only came up with one idea (S)


Cost effective (S)

Cost/ Problem definition

Easy to use and maintenance (E)

Problem definition

Fits the goal of the design (E)


Not block views (E)

Problem definition

What difficulties did you meet in solving the problem?

Have difficulties deciding which solution to choose (F)


Alternative solutions: Engineers evaluated alternative solution more effective than students.

They are good at solving homework problems but not the real problems. (F)

Homework problem and real life problems

The design challenge is very different from problems in class. (F)

Homework problem and real life problems

Need more constraints and criteria (S)

Problem definition

They don’t know the cost of materials (S)

Cost

They need more information about clients’ design consideration (E)

Problem definition


-49-

They wanted to talk to a window producer to make a product of their design (E)

Design experiences

Results

The researchers calculated frequencies of using problem decomposition and problem recomposition in students and engineer dyads. The means and standard deviations of percentages of using problem decomposition and problem recomposition are shown in Table 7. From these numbers, we can determine that engineer dyads used more problem decomposition more than engineering freshmen and seniors. In order to see if there are any statistically significant differences existing, p-tests were conducted, effect sizes (ES) were calculated. The results of statistical tests are shown in Table 8. In the use of problem decomposition, freshmen used problem decomposition as much as seniors did in engineering design. Engineers used more problem decomposition than both freshmen and seniors did in engineering design. In the use of problem recomposition, the results are similar; freshmen used problem recomposition as much as seniors did in engineering design, and engineers used more problem recomposition than both freshmen and seniors. Table 7 Means and Standard Deviations of Problem Decomposition and Problem Recomposition

Type of Dyad

Problem decomposition (%)

Problem recomposition (%)

Mean Standard Deviation Mean

Standard Deviation

Freshmen (N=10)

15.13 2.66 15.28 2.64

Seniors (N=10)

15.22

3.27

15.31

3.14

Engineers (N=5)

22.38

3.01

22.57

3.54


-50-

Table 8 Comparisons of Problem Decomposition and Problem Recomposition

Types of dyads compared

Problem decomposition

Problem recomposition

p value Effect Size p value Effect Size

Freshmen vs. seniors .496 N/A 980 N/A

Freshmen vs. engineers .000** 2.55 .001** 2.33

Seniors vs. engineers .010** 2.28 .001** 2.17 ** p ≤ 0.01.

The analysis of qualitative data had similar findings. In the interview, a question was “Did you tackle the problem as a whole or decompose it into several subproblems? If you decomposed it, why did you choose this over the problem as a whole?” Students’ answers varied from dyad to dyad. Some dyads broke the problem into multiple subsystems, some dyads solved the problem as a whole because they thought the problem was too simple to break down, and some student dyads did both. For engineer dyads, the answers were more consistent. They started with considering the whole problem to get the big picture, then broke it into small pieces to work on, and finally combined small pieces into the final solution.

The study also compared the effort spent on different levels of the problem. These results are shown in Table 9. Means and standard deviations from the three types of dyads show that freshmen, seniors, and engineers spent most of their cognitive effort on Level 3 (details of subsystems). All three types of dyads spent the least amount of cognitive effort on Level 1 (system). In order to see if there are any statistically significant differences existing, a series of tests were conducted, and the results of statistical tests of are shown in Table 10.


-51-

Table 9 Means and Standard Deviations of Levels of the Problem

Type of dyad

Level 1 (%) Level 2 (%) Level 3 (%)

Mean Standard Deviation



Freshmen (N = 10)

11.08 2.46 17.42 3.99 71.50 4.87

Seniors (N = 10)

12.59 3.57 16.75 4.61 70.66 4.37

Engineers (N = 5)

20.89 13.68 23.79 6.63 55.32 15.66

Table 10 Comparisons of Cognitive Effort on Different Levels of the Problem

Types of dyads compared

Level 1 Level 2 Level 3

p value Effect Size

p value Effect Size

p value Effect Size

Freshmen vs. seniors

.286 N/A .732 N/A .842 N/A

Freshmen vs. engineers

.020* 1.00 .035* 1.16 .009** 1.40

Seniors vs. engineers

.043* 0.83 .031* 1.23 .009** 1.33

*p ≤ 0.05. ** p ≤ 0.01.

On Level 1, which indicates the designer considering the problem as an integral whole, results show that freshman dyads and senior dyads spent the same amount of cognitive effort when they considered the problem as an integral whole. Engineer dyads spent more cognitive effort than freshman dyads and senior dyads when they considered the problem as an integral whole. On Level 2, which indicates designer considering interactions between subsystems, engineers spent more cognitive effort than freshmen and seniors did when they considered interactions between subsystems in engineering design. On Level 3, which indicates designer considering details of subsystems, engineers spent less cognitive effort than freshmen and seniors did when they considered details of subsystems in engineering design.


-52-

In the interview, participants were asked how they defined the problem. Engineer dyads considered many more factors in this stage. They defined the problem by thinking about both the problem and their client’s needs. All of them made sure that the design met requirements from Americans with Disabilities Act (ADA). They also considered safety issues, aesthetic issues, maintenances of the device, implementation, and cost. Some of them even considered the noises generated by the device because the device would be used in a nursing home in which a quiet environment is preferred. When freshman dyads defined the problem, the focus was to assist in opening the window. Two dyads mentioned clients of the design. A few dyads talked about the ADA, but most of them ignored ADA standard. Most senior dyads focused on the device itself, although they did better than freshman dyads. Most of them were aware of ADA standards, and a few dyads mentioned cost effectiveness as one of their criteria.

Discussion and Implications The results showed that engineer dyads used problem decomposition and

problem recomposition more than senior dyads and freshmen dyads. Qualitative data from interviews also support this result. In spite of differences in research settings, the results of this study are consistent with Ho’s (2001) study. Both studies suggested that there is a gap in using problem decomposition and recomposition between experts and novices. In fact, in interviews with engineer participants, they emphasized the importance repeatedly.

Although problem decomposition and recomposition are crucial strategies in engineering design, the results of this study showed that there was no difference between freshman dyads and senior dyads using this strategy in engineering design. This would suggest that, throughout the engineering program, students do not learn adequate knowledge about problem decomposition and problem recomposition; hence, students in the first year of the engineering program perform similar to students about to finish the engineering program.

The results of the study showed that engineer dyads, senior dyads, and freshmen dyads all spent the most cognitive effort on Level 3 and the least cognitive effort on Level 1. The quantitative data showed that on Level 1, engineer dyads spent more cognitive effort than senior dyads and freshman dyads. On Level 2, engineer dyads spent more cognitive effort than senior dyads and freshman dyads. On Level 3, engineer dyads spent less cognitive effort than senior dyads and freshman dyads.

Past studies identified two types of problem decomposition: the breadth-first approach and the depth-first approach. The breadth-first decomposition approach focuses on exploring various solutions of each subproblem and avoids deep exploration to any specific solution in the early stage, whereas depth-first decomposition tends to explore a specific subproblem in detail before other subproblems are investigated (Ormerod & Ridgway, 1999). In this research,


-53-

Level 3 represented designers considering details of subproblems. Student dyads spent more cognitive effort on this level because most of them used depth-first decomposition and spent a majority of cognitive effort exploring details of a certain subproblem. Engineer dyads used a breadth-first approach. Unlike student dyads, the distribution of their cognitive effort was more balanced across three levels of the problem.

A series of interesting findings emerged from the interviews and the analysis of participants’ sketches as well. In the process of generating alternative solutions, student dyads tended to generate too many or too few solutions compared with engineering dyads. Some dyads only generated one solution and finished their design at a premature stage. They did not make use of the time that they could have used to optimize their design. When examining engineering curriculum, we find that, in most courses, students are taught to generate only one solution instead of multiple ones. It also explains why some dyads only generated one solution through the entire design. For those dyads who generated too many alternative solutions, they spent a lot of time analyzing solutions, which lead them to either go way beyond the time limitation or to haphazardly select a final solution at the end of the design period.

In analyzing qualitative data, engineers were found to be more comfortable working in groups than students were. Student dyads had various difficulties when they worked together. A freshman dyad of students expressed their inadequacy in understanding each other’s ideas. Another freshman dyad of students had disagreements about which final solution to choose, which cost them a lot of time. A few senior dyads pointed out that they did not make good use of their time by working individually on different tasks at the same time. Typically, engineering students take foundational engineering courses before taking design classes. Most engineering fundamental courses focus on learning mathematical and scientific theories, which does not provide enough opportunities for students to work on team projects. This may be the main reason why some freshman dyads in this study had issues working with each other. As engineering students move forward in their program, they take design classes and participate in group projects, which explained why senior dyads performed better than freshman dyads when it comes to working in groups. However, the performance of senior dyads was still very different from the performance of engineer dyads. This finding is consistent with a series of previous studies (Holcombe, 2003; Meier, Williams, & Humphreys, 2000; Sageev & Romanowski, 2001; Scott & Yates, 2002). Implications

Engineering design has always been a significant content area in engineering education. Problem decomposition and recomposition strategies are frequently used by professional engineers. The results of this study showed that there is a gap between engineering students and engineering experts in using


-54-

problem decomposition and problem recomposition. In addition, no differences were found between engineering freshmen and engineering seniors, which indicates that students did not learn the skills of problem decomposition and problem recomposition in their undergraduate study. In order to better prepare students for future careers, it is extremely important to incorporate this content into engineering education. There is a need to develop supplemental teaching materials featuring problem decomposition and problem recomposition.

Considering the design problem as a whole was a common practice among professional engineers. They would analyze the big picture of the design problem, and this process is part of the problem-definition stage in engineering design. Problem definition is the first stage of engineering design. This study found that students spent significantly less time on this stage compared with engineering experts. This conclusion is consistent with previous studies (Atman et al., 2007; Jain & Sobek, 2006). Both freshman and senior dyads were found to spent significantly less effort in defining the problem, which implies that engineering education should place more importance on teaching problem definition in general.

References Arvanitis, T. N., Todd, M. J., Gibb, A. J., & Orihashi, E. (2001). Understanding

students’ problem-solving performance in the context of programming-in-the-small: An ethnographic field study. In Proceedings of the Frontiers in Education Conference (Vol. 2, pp. F1D-20-23vol.2). Washington, DC: IEEE Computer Society.

Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., & Saleem, J. J. (2007). Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education, 96(4), 359–379. doi:10.1002/j.2168-9830.2007.tb00945.x

Ball, L. J., Evans, J. St. B. T., & Dennis, I. (1994). Cognitive processes in engineering design: A longitudinal study. Ergonomics, 37(11), 1753–1786. doi:10.1080/00140139408964950

Chandrasekaran, B. (1990). Design problem solving: A task analysis. AI Magazine, 11(4), 59–70. doi:10.1609/aimag.v11i4.857

Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education, 94(1), 103–120. doi:10.1002/j.2168-9830.2005.tb00832.x

Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th ed.). Boston, MA: Pearson, Allyn & Bacon.

Gero, J. S. (2010). Generalizing design cognition research. In K. Dorst, S. C. Stewart, I.

Staudinger, B. Paton, & A. Dong (Eds.), DTRS 8: Interpreting design thinking (pp. 187–198). Sydney, Australia: University of Technology Syndey.


-55-

Gero, J. S., Kan, J. W. T., & Pourmohamadi, M. (2011). Analysing design protocols: Development of methods and tools. In A. Chakrabarti (Ed.), Research into design: Supporting sustainable product development (pp. 3–10). Bangalore, India: Research Publishing.

Gero, J. S., & Kannengiesser, U. (2004). The situated function–behaviour–structure framework. Design Studies, 25(4), 373–391. doi:10.1016/j.destud.2003.10.010

Gero, J. S., & Mc Neill, T. (1998). An approach to the analysis of design protocols. Design Studies, 19(1), 21–61. doi:10.1016/S0142-694X(97)00015-X

Glaser, B. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4), 436–445.

Hall, J. G., Jackson, M., Lanney, R. C., Nuseibeh, B., & Rapanotti, L. (2002). Relating software requirements and architectures using problem frames. In Proceedings of the IEEE Joint International Conference on Requirements Engineering (pp. 137–144). Los Alamitos, CA: Institute of Electrical and Electronics Engineers. doi:10.1109/ICRE.2002.1048516

Ho, C-H. (2001). Some phenomena of problem decomposition strategy for design thinking: Differences between novices and experts. Design Studies, 22(1), 27–45. doi:10.1016/S0142-694X(99)00030-7

Holcombe, M. L. (2003). ET grads—How’d the transition go? In Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition (pp. 8.537.1–8.537.6). Washington, DC: American Society for Engineering Education. Retrieved from https://peer.asee.org/12078

Jain, V. K., & Sobek, D. K., II. (2006). Linking design process to customer satisfaction through virtual design of experiments. Research in Engineering Design, 17(2), 59–71. doi:10.1007/s00163-006-0018-2

Katehi, L., Pearson, G., Feder, M. (Eds.). (2009). Engineering in K–12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. doi:10.17226/12635

Lammi, M., & Becker, K. (2013). Engineering design thinking. Journal of Technology Education, 24(2), 55–77. Retrieved from https://scholar.lib.vt.edu/ejournals/JTE/v24n2/pdf/lammi.pdf

McCracken, W. M. (1997). Portfolio assessment in design education. Atlanta, GA: EduTech Institute and College of Computing, Georgia Institute of Technology. Link does not work.

Meier, R. L., Williams, M. R., & Humphreys, M. A. (2000). Refocusing our efforts: Assessing non-technical competency gaps. Journal of Engineering Education, 89(3), 377–385. doi:10.1002/j.2168-9830.2000.tb00539.x

Ormerod, T. C., & Ridgway, J. (1999). Developing task design guides through cognitive studies of expertise. In S. Bagnara (Ed.), European Conference on Cognitive Science (pp. 401–410). Sienna, Italy.


-56-

Sageev, P., & Romanowski, C. J. (2001). A message from recent engineering graduates in the workplace: Results of a survey on technical communication skills. Journal of Engineering Education, 90(4), 685–692. doi:10.1002/j.2168-9830.2001.tb00660.x

Schloss, P. J., & Smith, M. A. (1999). Conducting research. Upper Saddle River, NJ: Prentice Hall.

Scott, G., & Yates, K. W. (2002). Using successful graduates to improve the quality of undergraduate engineering programmes. European Journal of Engineering Education, 27(4), 363–378. doi:10.1080/03043790210166666

Vincenti, W. G. (1990): What engineers know and how they know it: Analytical studies from aeronautical history. Baltimore, MA: Johns Hopkins University Press.

About the Authors

Ting Song ([email protected]) is Professor in the Department of Applied Science at South Puget Sound Community College in Olympia, Washington. Kurt Becker ([email protected]), is Professor in the Department of Engineering Education at Utah State University. John Gero ([email protected]), is Research Professor in the Krasnow Institute for Advanced Study at George Mason University & University of North Carolina. Scott DeBerard ([email protected]), is Associate Professor in the Department of Psychology at Utah State University. Oenardi Lawanto ([email protected]), is Professor in the Department of Engineering Education at Utah State University. Edward Reeve ([email protected]), is Professor in the Department of Technology and Engineering Education at Utah State University.


-57-

Identifying Indicators Related to Constructs for Engineering Design Outcome

Cheryl A. Wilhelmsen & Raymond A. Dixon

Abstract

This study ranked constructs articulated by Childress and Rhodes (2008) and identified the key indicators for each construct as a starting point to explore what should be included on an instrument to measure the engineering design process and outcomes of students in high schools that use the PLTW and EbDTM curricula in Idaho. A case-study design was used. Data were collected in two stages. In the first stage, a content analysis was conducted for PLTW and EbDTM curricula to identify the indicators that are associated with the six constructs articulated by Childress and Rhodes (2008). In the second stage, the constructs and key indicators or concepts were placed on a survey and sent to experts for them to rate their importance for assessment and their difficulty to assess. Main findings included engineering and human values and the application of engineering design being ranked as first and second, respectively, for inclusion on an instrument to measure the engineering design process and outcomes. In addition, a total of 141 indicators were identified for all constructs. The indicators identified provide a useful list of measures that can be used by technology and engineering teachers. Selected indicators can be used by math, science, technology, and engineering education teachers as they coordinate in the teaching of STEM concepts and collaborate in the designing of project-based activities that they engage students in solving. Keywords: assessment; EbDTM; engineering design process; PLTW; problem-based learning; project-based learning; STEM.

Introduction Problem-based learning (PBL) promotes deep thinking and problem-

solving skills (Woods, 1996). It has proven to be an effective way to learn subject knowledge, and in most PBL programs, “the goal is to empower the students with the task of creating the learning objectives that are important to them” (Woods, 2000, p. 2). Students are confronted with a scenario constructed around real-life problems, which by their nature are ill-structured, open-ended, and ambiguous, that launch students’ inquiry as they collaborate to find solutions (Banks & Barlex, 2014; Woolfolk, 2013). Project-based learning is commonly used in technology and engineering education. Because project-based learning shares many of the instructional, multidisciplinary traits as PBL, the terms are often confused or used interchangeably (Honey, Pearson, & Schweingruber, 2014).


-58-

According to Banks and Barlex (2014), the difference between project-based learning and PBL is that:

PBL has tended to be a way of configuring the curriculum and relating what the students know to actual, real-world problems which in turn leads them to find out new knowledge and skills to bring to bear on the problem. Rather, project-based learning has been more about a pupil choosing an extended activity that they are interested in and using it as a vehicle for demonstrating their current capabilities, but also including demonstrating their abilities in researching and investigating new knowledge and acquiring skills as required. (p. 141)

Project-based learning can also be built on authentic, real-world situations or problems (Edström, Soderholm, & Knutson Wedel, 2007). As they work in groups, students are not restricted on where they may look for answers. In a review of the research on project-based learning, Thomas (2000) articulated five criteria that characterized projects:

1. “Projects are central, not peripheral to the curriculum,” 2. “focused on questions or problems that ‘drive’ students to encounter

(and struggle with) the central concepts and principles of a discipline,” 3. “involve students in a constructive investigation” (p. 3), 4. “student-driven to some significant degree,” and 5. “realistic, not school-like” (p. 4).

“PBL incorporates real-life challenges where the focus is on authentic (not simulated) problems or questions and where solutions have the potential to be implemented (Gordon, 1998; as cited in Thomas, 2000, p. 4).

Assessment refers to the process of determining the extent to which students are achieving the intended learning outcomes (Gronlund, 1998). In PBL and project-based learning, assessment should emphasize problem solving, critical thinking, and reasoning skills. Creating problems that are similar to tasks accomplished in real life—authentic tasks—is a key principle for assessment used in both paradigms of instruction. However, assessment techniques that are repeatable are very challenging because of the subjective nature of PBL and project-based learning. In fact, McCracken and Waters (1997) believed that the requirement to have authentic tasks in problem solving conflicts with the requirement for assessments to be repeatable, because authentic tasks are themselves ill-structured and difficult to assess objectively. Assessments in Technology and Engineering Education

The inclusion of engineering design as a part of the Standards for Technological Literacy has resulted in more curricula promoting engineering design activities. Engineering design is a systematic and often iterative approach to designing objects, processes, and systems to meet human needs and wants (National Research Council, 2012). The Standards for Technological Literacy define engineering design as “the systematic and creative application of


-59-

scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems” (International Technology Education Association, 2007, p. 238). Engineering design problems are project-based activities during which students use the engineering design process to solve the problem while working in groups.

Traditional engineering education programs at the college level use a variety of methods to collect evidence that students are achieving intended learning outcomes. These include written and oral questions, performance ratings, product reviews, journals, portfolios, and other self-reports such as inventories and questionnaires. Written assessments include multiple-choice and other closed items, calculations, and open-ended questions. Oral questions, on the other hand, enable teachers to uncover students’ misconceptions. They require students to think on their feet and speak coherently. Journals and portfolios provide records of students’ individual and collaborative efforts in design projects. “They reveal students’ critical thinking and reasoning skills, and record the steps students followed in an engineering [design] process” (Gray, 2007, p. 161). Performance rating can be used to assess students’ process and products in engineering design. Rating scales that define the degrees of quality along with rubrics (which are a list of the quality of a performance, process, or product) are used to assess the student. Self-report measures allow “students to reflect on their learning experiences” and help “them to see more clearly the connections among the concepts they have learned, as well as the applications of these concepts to new situations” (Gray, 2007, p. 161).

Addressing the infusing of engineering design at the K–12 level, some researchers have indicated that there are still areas in assessment that are open issues (Lewis, 2005; Kelley, 2008; Wicklein, 2005). Technology educators face these issues or challenges when they seek to implement engineering design into their curriculum. For example, the past few years have seen more school districts in Idaho adopting either the Engineering by Design (EbDTM) or the Project Lead the Way curricula in their technology and engineering education programs. Some teachers and administrators, including the program director for the Technology and Engineering program in the State of Idaho, have expressed the need to explore having some assessment tool to measure engineering design outcomes that is repeatable, irrespective of whether the school uses the Engineering by Design (EbDTM) curriculum or the Project Lead the Way curriculum (PLTW). Other teachers, including science and math teachers in some smaller school districts who also teach technology education, think that it would be helpful to have some instrument that guides them in their assessment. As a starting point to explore this issue, the authors decided to identify indicators to measure the constructs associated with engineering design process and outcomes particularly for Grades 9–12 level—the level at which the EbDTM and the PLTW curricula are primarily used in Idaho.


-60-

The Framework The conceptual framework for this study drew on the work of Childress and

Rhodes (2008) in which they examined engineering design content that should be taught in high school curricula. They articulated a framework to define the engineering design curriculum content. The seven categories were identified through a modified Delphi approach. They are:

1. “Engineering design . . . [that] emphasizes the importance of creativity in designing engineered solutions to problems . . . [as well as] design iteration . . . and tradeoffs” (p. 7).

2. “Application of engineering design . . . [that] includes outcomes related to specific design activities . . . [including] experimentation, prototyping, and reverse engineering” (p. 7–8).

3. “Engineering analysis . . . [that] includes using mathematics to optimize solutions, and . . . emphasizes the use of mathematics and science in the engineering design process” (p. 8).

4. “Engineering and human values . . . [that consists of] the interaction of engineering design and society . . . [for instance,] safety and the environment versus costs and ethics” (p. 8).

5. “Engineering communication . . . [that includes] all sorts of communications important to the engineering design process” (p. 8).

6. “Engineering science . . . [that] includes many of the traditional engineering ‘sciences’ such as statics and dynamics . . . [as well as] material processes, ergonomics, energy power, etc.” (p. 8).

7. “Emerging fields of engineering . . . [that includes] nanotechnology . . . [and] genetic engineering” (p. 8).

In this research study, we used six of the seven categories. The seventh category, emerging fields of engineering, was not used because it related mainly to nanotechnology, which is not covered in the high school curriculum. Purpose of the Study

The purpose of this study was to identify indicators for each of the constructs identified by Childress and Rhodes (2008) that can be used by STEM teachers in Idaho as a guide when they are assessing design outcomes of students in high schools, irrespective of whether the curriculum in use is EbDTM or PLTW. In addition, these indicators can provide researchers in STEM with items that can be used in the development of an instrument for assessment in engineering design at the high school level. The research questions that guided this study are:

1. How are the constructs identified by Childress and Rhodes (2008) ranked by professional engineers and educators in terms of criticality for inclusion on an instrument to measure engineering design outcomes in high schools in Idaho?


-61-

2. What are the key indicators associated with each of the constructs identified by Childress and Rhodes (2008) to measure engineering design outcomes in high schools in Idaho?

Research Design

A case-study design was used. Case-study research involves the study of a case within a real-life, contemporary context or setting. It is a qualitative approach in which the investigator explores a real-life bounded case over time, using detailed data collection (Yin, 2009). A letter was sent to the program manager for the Technology and Engineering Education program requesting permission for the two schools’ participation. The Cases

Two cases were examined. One school in northern Idaho that uses the EbDTM curriculum and another school in southern Idaho that uses the PLTW curriculum. The school in northern Idaho had its own unique way of organizing and supplementing the EbDTM course material. The Fundamentals of Technology course is taught in Grade 9 and is only offered one semester. The technological design curriculum is taught in Grade 10 and covers topics such as career search, sketching, toy design (which the instructor uses for teaching shop safety, power tools, and finishing), Logo design concepts, mouse-trap cars, SolidWorks™ for bridge building, Co2 cars, and an additional design problem. The curriculum emphasizes the engineering team concept and encourages creative design for all students. The Advanced Design Applications Class, taught in Grade 12, uses a material science curriculum developed by Energy Concepts Inc. that includes solid materials, metals, polymers, ceramics, and composites. The emphasis is on the importance of materials engineering to the manufacturing process. The engineering design courses included more SolidWorks™, robotics, and the VEX curriculum as well as total quality management. Each course requires the students to complete a project.

The school in southern Idaho uses the PLTW curriculum. Introduction to Engineering is taught in Grade 9 and focuses on the design process and its application. Principles of Engineering is taught in Grade 10 and introduces major concepts that students encounter in postsecondary engineering courses, such as mechanisms, statics, materials and kinematics. There are five specialization courses within PLTW: Aerospace Engineering (AE), Biotechnical Engineering (BE), Civil Engineering and Architecture (CEA), Computer Integrated Manufacturing (CIM), and Digital Electronics (DE). Digital Electronics and Aerospace Engineering are taught in Grade 11. Engineering Design and Development (EDD) is taught in Grade 12. This is the capstone course in which students work in teams to design and develop solutions to a problem by applying the engineering design process.


-62-

Procedure Data Collection

Data were collected in two stages. In the first stage, a content analysis was conducted for PLTW and EbDTM curricula to identify the indicators that are associated with the six constructs identified by Childress and Rhodes (2008). In the second stage, the constructs and key indicators or concepts were placed on a survey form and sent to experts for them to rate their importance for assessment and their difficulty to assess. Content analysis

A qualitative content analysis of selected courses from the EbDTM and PLTW curricula used by each school was conducted to identify concepts of engineering design that were associated with the constructs identified by Childress and Rhodes (2008). These concepts are referred to as indicators in this study. Content analysis is a research tool in which researchers quantify and analyze the meanings and relationships of words and concepts within a text (Busch et al., 2012; Krippendorff, 2004).). Content analysis enables researchers to sift through large volumes of data in a systematic fashion with relative ease. It also allows inferences to be made that can then be corroborated using other methods of data collection. The curriculum materials that were analyzed from the PLTW and EbD curricula are displayed in Table 1 (below and continued on next page).

Table 1 Curriculum Materials Analyzed for PLTW and EbD

PLTW 10th Grade Curriculum Materials

EbD 10th Grade Curriculum Materials

Principles of Engineering Lessons, Activities, Projects, PowerPoint’s, Assessments, Teacher Notes, Student Resources, ABET Concepts, National Science Education Standards, Standards for School Mathematics, Standards for the English Language Arts, Standards for Technological Literacy, and Principles of Engineering PLTW textbook.

Technological Design Lessons, Activities, Projects, Assessments, Teacher Notes, and Student Resources.


-63-



Digital Electronics Lessons, Activities, Projects, PowerPoint’s, Assessments, Teacher Notes, Student Resources, ABET Concepts, National Science Education Standards, Standards for School Mathematics, Standards for the English Language Arts, Standards for Technological Literacy, and Digital Electronics PLTW textbook.

Advanced Design Applications Lessons, Activities, Projects, Assessments, Teacher Notes, Student Resources, and Material Science Textbooks.

Aerospace Lessons, Activities, Projects, PowerPoint’s, Assessments, Teacher Notes, Student Resources, ABET Concepts, National Science Education Standards, Standards for School Mathematics, Standards for the English Language Arts, and Standards for Technological Literacy.



Engineering Design & Development Lessons, Activities, Projects, PowerPoint’s, Assessments, Teacher Notes, Student Resources, ABET Concepts, National Science Education Standards, Standards for School Mathematics, Standards for the English Language Arts, and Standards for Technological Literacy.

Engineering Design & Robotics Lessons, Activities, Projects, Assessments, Teacher Notes, Student Resources and Robots program materials by Intelitek.

Coding. Two coders assigned codes to the six constructs. The curricula

were then examined to identify engineering design concepts and then categorized each of these concept under one or more of the constructs of Childress and Rhodes (2008). To ensure intercoder reliability, each coder was given a copy of Grade 10 curriculum materials for both PLTW and EbDTM. The


-64-

researcher provided instructions to the coders prior to the coding process. The coders independently highlighted words and phrases relating to engineering design concepts that were in the Grade10 curriculum materials of both PLTW and EbDTM. The coders met to review and discuss their findings. Discrepancies were discussed and resolved. The process was repeated until an inter-coder reliability of 87% was obtained. Krippendorff (2004) indicated that in order to assure the data under consideration are at least similarly interpreted by two or more coders it is customary to require an intercoder reliability of 80% or more; therefore, the intercoder reliability for this study was well within acceptable levels. The coders then proceeded to perform a content analysis of the remaining sample of curriculum materials for both PLTW and EbDTM.

Table 2 Constructs and Codes

Construct Code

Engineering design that emphasizes the importance of creativity in designing engineered solutions to problems, as well as design iterations and tradeoffs Application of engineering design that included outcomes relating to design activities, experimentation, prototyping and reverse engineering Engineering analysis that includes mathematics in optimizing solutions and the use of both science and math in the engineering design process Engineering and human values that consists of the interactions between engineering design and society such as safety and the environment versus costs and ethics Engineering communication that included all sorts of communications important to the engineering design process Engineering science that includes the traditional sciences such as statics and dynamics as well as material properties, energy, power, etc.

ED-CIT ED- EPR ED- MSO ED-HV ED-C ED-ESD

After words relating to concepts of engineering design were identified,

similar concepts were grouped together. The total number\ of words relating to engineering concepts that were identified by the coders amounted to 711, 618 of


-65-

which were common to both coders. Some of the words were a derivative of the same word, so they were reduced into a final manageable, qualitative descriptive frequency list. This process was done by including the highest frequency word found within a group of similar words. For example, a group of words found by the coders were: communicate, communication, and communications. The final word selected was communication because it had the highest frequency. A part of the final frequency list of words is shown in Table 3.

Table 3 Part of Final Frequency List

Descriptive Frequency Word Frequency

activity 1612 addition 106 aerospace 206 aircraft 285 airfoil 79 airplane 32 analysis 185 analyze 239

After the final frequency list was identified, the curriculum material was

again examined by the coders to better understand the context in which the words were used and determine which of the constructs they were related to (Busch et al., 2012). Words that appropriately related to a construct were coded using the codes identified in Table 2. So, the constructs served as categories. Brief statements containing a verb, object, and sometimes a modifier were finally used to better capture the meaning of the concept or context in which it was used. These were indicators. For example, for the word communication, which was coded as ED-C, an examination of the meaning and context produced the statement Communicating knowledge professionally. Survey

The survey instrument used was a modification of the Task Verification instruments used by Norton (1999). In the instrument Norton used, duty statements of a job or occupation are stated, and the task statements relating to each duty were listed below the duty statement. Expert workers were asked: (a) do they perform the task; (b) rate the importance of the task on a Likert scale of 0–5, with 0 meaning No Importance and 5 meaning Great Importance; and (c) rate the difficulty of a task on a Likert scale from 0–5, with 0 being Extremely


-66-

Easy and 5 being Extremely Difficult. The criticality of each task was determined by multiplying the importance index by the difficulty index.

The instrument developed by the researchers replaced the duty statements with the six constructs of Childress and Rhodes (2008):

• Engineering design, • Application of engineering design, • Engineering analysis, • Engineering communication, • Engineering and human values, and • Engineering science.

The indicator statements replaced the task statements on Norton’s (1999)

task verification instruments. The instrument asked expert participants to examine the indicators for each construct and rate each indicator on a 5-point Likert scale with 1 representing Strongly Disagree and 5 representing Strongly Agree for their (a) importance for assessment and (b) difficulty to assess engineering design process and outcomes. The criticality index for each indicator was determined by multiplying its importance score and its difficulty score. The criticality index for each construct was determined by multiplying the averaged importance score and the averaged difficulty score for the key indicators of that construct.

The survey was pilot tested by sending it to two teachers to fill out. Simple grammatical errors were corrected, and then it was sent to six experts. The experts were chosen for their experience in teaching engineering education in high school and at the college level and for practicing engineering in industry. The expert team consisted of two technology and engineering education teachers from two high schools in Idaho with combined years of teaching of over 30 years, two engineers from industry in Idaho with a combined working experience of 45 years, and two engineering education faculty from two universities with a combined experience of over 15 years in teaching and research.

Results The results obtained from an analysis of the data are presented in respect to

the two research questions posed at the beginning of the study. The first question was: How are the constructs identified by Childress and Rhodes (2008) ranked by professional engineers and educators in terms of criticality for inclusion on an instrument to measure engineering design outcomes in high schools in Idaho? Childress and Rhodes (2008) framework consisted of seven constructs, six of which were used for this study. The criticality index for each construct was derived by multiplying the indicators’ average importance index by the average difficulty index. The constructs were then rank ordered from the highest criticality index to the lowest criticality index. As indicated in Table 4,


-67-

engineering and human values had the highest criticality index and so was ranked one, and engineering science had the lowest criticality index and was ranked six.

Table 4 Criticality Ranking of the Six Constructs

Construct Category

Mf

Importance

Mf

Difficulty

Indicator of

Criticality

Engineering and Human Values

4.2 3.3 13.9

Application of Engineering Design

4.0 3.0 11.9

Engineering Communication 4.1 2.9 11.8 Engineering Design Concepts 4.0 2.9 11.6 Engineering Analysis 3.8 2.7 10.3 Engineering Science 3.5 2.3 8.3

The second research questions was: What are the key indicators associated

with each of the constructs identified by Childress and Rhodes (2008) to measure engineering design outcomes in high schools in Idaho? The category engineering and human values had six indicators (see Table 5). Five of the six indicators were rated high in importance, receiving scores ranging from 4.0 to 4.8. Three of the indicators were perceived to be difficult to assess.


-68-

Table 5 Key Indicator Results for Engineering & Human Values

Engineering & Human Values Mf

Importance Mf

Difficulty

Participate in teams 4.8 3.0 Assess the effect of technology on the environment

4.3 3.7

Understand ethical implications 4.2 3.7 Determine product’s safety in function 4.2 3.5 Apply the relationship between voltage, current, & resistance

4.0 2.7

Understand relationships among technologies 3.8 3.3

Average Mean Value 4.2 3.3

For the construct application of engineering design, 12 indicators were

identified (see Table 6). Eleven of the 12 indicators were rated high in importance, receiving scores ranging from 4.0 to 4.8. Table 6 Key Indicator Results for Application of Engineering Design

Application of Engineering Design Mf

Importance Mf

Difficulty

Provide accurate documentation 4.8 3.0 Calculate forces 4.7 2.7 Understanding measurements 4.7 2.7 Troubleshoot errors 4.3 3.5 Modify design 4.2 3.5 Use experimentation to make decisions 4.2 3.2 Apply constraints 4.2 2.8 Construct/evaluate working prototypes 4.2 2.5 Explore functions of systems 4.0 3.5 Participate in team activities 4.0 3.0 Identify manufacturing processes 4.0 2.7 Utilize flight simulators 2.0 2.1



-69-

The construct engineering communication had 20 indicators (see Table 7).

Fifteen of the 20 indicators had importance ratings at 4.0 and above. Interestingly, the indicator utilizing brainstorming methods was scored 4.5 for importance but received 4.2 for difficulty to assess, the highest difficulty score for this construct. Table 7 Key Indicator Results for Engineering Communication

Engineering Communication Mf

Importance Mf

Difficulty

Communicate knowledge professionally 4.7 2.8 Utilize modeling software 4.7 2.7 Communicate the design solution process 4.5 3.0 Engage in Problem-based learning 4.5 3.0 Apply standards 4.5 3.0 Utilize brainstorming methods 4.5 4.2 Engage in project-based learning 4.5 3.3 Develop skills in using tools 4.3 3.2 Utilize presentation software 4.3 1.8 Develop sketches 4.3 2.3 Evaluate feedback 4.2 3.3 Solve design problems 4.0 3.5 Create/deliver formal presentations 4.0 2.5 Communicate using symbols 4.0 2.3 Understand the importance of project management

4.0 3.3

Understand communication technologies 3.8 3.2 Create detailed flow charts 3.5 1.8 Improve design process & outcome 3.3 3.5 Use symbols in communicating processes 3.3 2.5 Utilize automation system programming functions

3.2 2.3


For the construct engineering design concepts, 16 indicators were identified

from the content analysis (see Table 8). Eleven of the indicators had scores ranging from 4.0 to 4.8. Each of these 11 indicators had difficulty to assess, with scores ranging from 2.3 to 3.3, indicating they are not difficult to assess in class.


-70-

Table 8 Key Indicator Results for Engineering Design Concepts

Engineering Design Concepts Mf

Importance Mf

Difficulty

Use creativity in solving problems 4.8 3.3 Document project’s progress in engineering notebook

4.7 2.3

Understand attributes of a design process 4.5 3.5 Understand core concepts of technology 4.5 2.5 Develop models 4.5 3.0 Conduct research 4.3 3.5 Create portfolios in documenting work 4.0 2.3 Understand material & equipment requirements

4.0 2.5

Optimize design solutions 4.0 3.3 Employ strategies 4.0 2.8 Understand system energy requirements 4.0 2.5 Use construction technologies 3.8 2.5 Use the method of joints strategy to determine forces in a truss

3.7 2.7

Create system control programs 3.5 2.8 Create new systems/processes 3.2 3.5 Justify discoveries or innovations 3.2 3.0


The construct engineering analysis had 30 indicators (see Table 9). Thirteen

of these indicators had scores ranging from 4.0 to 5.0. The indicator utilizing mathematics to solve problems received the highest importance score.


-71-

Table 9 Key Indicator Results for Engineering Analysis

Engineering Analysis Mf

Importance Mf

Difficulty

Utilize mathematics to solve problems 5.0 2.7 Utilize mathematical formulas to solve design problems

4.7 2.8

Use mathematical concepts in design 4.7 3.0 Know to calculate a moment 4.5 2.3 Develop solutions to problems 4.5 3.7 Understand quantitative data 4.5 2.8 Conduct testing 4.3 3.2 Evaluate design solutions 4.2 3.2 Use assessment techniques 4.0 2.8 Use decision matrix for design problems 4.0 2.7 Evaluate output work of mechanisms 4.0 2.5 Describe basic logic functions 4.0 2.3 Understand criteria in assessment rubrics 4.0 3.5 Determine angles 3.8 2.5 Identify magnitude, direction, & sense of a vector

3.8 2.2

Understand mechanical advantage ratios 3.8 2.3 Calculate mean, median, & mode 3.8 2.0 Calculate gear ratio 3.8 2.0 Weigh tradeoffs 3.6 3.2 Calculate drive ratios of mechanisms 3.5 2.0 Choose appropriate input devices of technological systems

3.3 3.0

Apply statistics 3.3 2.8 Choose appropriate output devices of technological systems

3.2 3.3

Differentiate flow rate and flow velocity 3.2 2.5 Calculate probability 3.2 2.2 Perform competitive product analyses 3.0 3.0 Locate the centroid of structural members 3.0 2.3 Understand matrix & reinforcement in composite materials

2.8 2.0

Evaluate input work of mechanisms 2.7 2.7



-72-

The last category or construct on the instrument, engineering science, had

61 indicators (see Table 10, below and continued on next page). The importance rating data indicated that 20% of the indicators ranged at 4.0 or above, which means twelve of the 61 key indicators were rated high in importance for inclusion in an engineering design assessment tool. Six of the indicators were ranked below 3.0 for importance. Only three indicators were rated at 3.0 or above in their difficulty to assess. The two indicators that were scored as least difficult to assess were differentiating and calculating velocity and differentiate digital and analog systems. Table 10 Key Indicator Results for Engineering Science

Engineering Science Mf

Importance Mf

Difficulty

Calculate mechanical advantage 4.5 2.3 Identify material properties 4.3 2.5 Use computers to organize & communicate data

4.3 2.3

Understand static equilibrium of bodies 4.3 2.3 Calculate mechanical efficiency 4.2 2.3 Develop technological knowledge 4.2 3.3 Calculate velocity 4.0 1.8 Calculating speed 4.0 2.5 Apply the relationship between voltage, current & resistance

4.0 2.3

Understand properties of metals 4.0 2.2 Distinguish between the six simple machines 4.0 2.0 Calculate mass 4.0 2.0 Use scientific concepts in design 3.9 2.8 Understand characteristics of technology 3.8 3.0 Understand compound machines 3.8 2.3 Applying thermodynamic principles 3.8 2.8 Differentiate the basic properties of materials (electrical, magnetic, etc.)

3.8 2.2

Design, build, & test truss designs 3.8 2.2 Differentiate digital & analog systems 3.8 1.8 Calculate material properties using a stress strain curve

3.7 2.3

Construct simple & compound gear systems 3.7 2.3 Identify properties of elements 3.7 2.2


-73-

Calculate torque ratio 3.7 2.0 Understand characteristics of lever systems 3.7 2.0 Calculate stress 3.7 2.0 Calculate circuit resistance, current & voltage 3.7 1.8 Identify science concepts 3.7 2.8 Understand of electrical circuits 3.7 2.7 Understand of electrical energy 3.7 2.5 Understand thermal energy transfer 3.7 2.7 Identify impacts of energy 3.5 2.8 Design, create, & test hydraulic devices 3.5 2.8 Understand the advantages & disadvantages of circuit design

3.5 2.5

Understand electronics 3.5 2.5 Define types of power 3.5 2.0 Understanding inclined plane systems 3.5 2.0 Employ kinematics equations 3.3 2.2 Identify properties & characteristics of solids 3.3 2.2 Identify & categorize energy sources 3.3 2.0 Identify components & functions of fluid power

3.3 2.0

Identify characteristics of composites 3.3 2.3 Identify engineering disciplines 3.3 2.3 Provide technical feasibility 3.2 3.3 Work with electronic assemblies 3.2 2.8 Design, create, & test pneumatic devices 3.2 2.2 Design/create/& test pulley systems 3.2 2.2 Understand recycling technology 3.2 2.2 Conduct tensile testing 3.2 2.2 Understand fuel cell technology 3.0 2.5 Classify properties of Polymers 3.0 2.5 Use transportation technologies 3.0 2.5 Design/create/& test sprocket systems 3.0 2.0 Experiment with solar hydrogen systems 2.8 2.5 Understand chemical properties 2.8 2.5 Create a simple airfoil 2.8 2.2 Understand basic aircraft design 2.7 2.5 Understand aerospace materials & structures 2.7 2.0 Differentiate ceramic materials in industry 2.5 2.0



-74-

Discussion Engineering and human values was ranked with the highest criticality for

inclusion in an instrument to measure engineering design outcomes. Not only do the experts see this construct as important, but they also see it as difficult to assess. Childress and Rhodes (2008) refer to engineering and human values as the big picture when it comes to the interaction of engineering design and society, which includes the weighing of limitations in decisions about safety and the environment versus costs and ethics. So, the expert participants believe that this should be given priority in assessment. Engineers are often required to work with teams that are diverse and interdisciplinary to solve complex problems that may have local, regional, and global consequences, and in doing so, they have to be cognizant of the ecological impact of their design. Therefore, good engineering goes beyond being technically competent but also involves understanding and making judgments about the moral implications of designs. Lau (2013) points out that engineers are largely responsible for the artifacts of the modern world, and this constructed world has both risks and benefits ranging from obvious safety and health issues, to issues of equity and environmental degradation. Engineers therefore need to “have an understanding of how their activity affects progress, and how to do that benevolently” (p. 1). In addition, he indicated that the process of solving ethical problems has many similarities to the engineering design process.

Engineering analysis and engineering science received the two lowest rankings. This might be a reflection of the perception that engineering analysis and the sciences that are associated with it must not be the predominant emphasis of engineering design at the high school level. Overall, however, the experts think that students’ engineering outcomes should be determined by their performance relating to several key indicators relating to mathematical computation and the sciences, such as calculate mechanical advantage, identify material properties, and know how to calculate a moment. It should also be noted that indicators such as utilize mathematics to solve problems, utilize mathematical formulas to solve design problems, and use mathematical concepts in design received some of the highest importance scores, emphasizing the perception that these experts have of students being able to model math as part of the engineering design outcomes in high school. This consistently reflects the opinion of other experts in science and engineering. As the National Research Council (2012) noted in their framework:

Although there are differences in how mathematics and computational thinking are applied in science and in engineering, mathematics often brings these two fields together by enabling engineers to apply the mathematical form of scientific theories and by enabling scientists to use powerful information technologies designed by engineers. Both kinds of professionals can thereby accomplish investigations and analyses and build complex models, which might otherwise be out of the question. (p. 65)


-75-

Assessments that are guided by indicators related to all six constructs should give technology and engineering educators that use the EbDTM and PLTW curricula in high school in Idaho a more holistic representation of students’ performance in engineering design. Importantly, the list of indicators relating to each construct can also help to reinforce to math and science teachers the depth of students’ immersive STEM experiences when their schools use the EbDTM and the PLTW curricula. This might motivate more collaboration across these disciplines. These indicators can also provide technology, math, and science education teachers with a list of items that can be included on performance rating forms to assess students’ process and products in engineering design. Rating scales along with rubrics are used to assess students. Selected indicators can also be included on self-report measures that allow students to reflect on their learning experience and help them see the connections among the concepts that they learned as well as the applications of these concepts in new situations (Gray, 2007). It must be mentioned that many of these indicators can be broken down further into discrete actions that can provide useful measures of student’s competency in a particular designing activity. In fact, the indicators that were viewed as difficult to assess (such as develop solutions to problems, understand attributes of a design process, and utilize brainstorming methods) may need to be broken down into more discrete action statements to provide clarity for assessment.

Conclusion This study explored ranking engineering design constructs identified by

Childress and Rhodes (2008) and identifying their indicators. The results represent preliminary work in addressing assessment of engineering design outcomes in schools in Idaho, irrespective of the curriculum in use. Admittedly, more questions still need to be answered. For example, can an instrument be developed from the indicators that validly and reliably assesses students’ outcomes in design? What indicators should be included on such an instrument? More study needs to be done to answer these questions. The indicators identified for each construct in this study, however, provide a useful list of measures that can be used by technology and engineering teachers. Selected indicators can be identified by math, science, technology, and engineering education teachers as they coordinate in the teaching of STEM concepts and collaborate in the designing of project-based activities that they will engage students in solving. But, at present, the list provides a menu that teachers can choose from that relates to their instructional objectives, which they can use to assess students learning outcomes.


-76-

References Banks, F., & Barlex, D. (2014). Teaching STEM in the secondary school:

Helping teachers meet the challenge. New York, NY: Routledge. Busch, C., De Maret, P. S., Flynn, T., Kellum, R., Le, S., Meyers, B., . . .

Palmquist, M. (2012). Writing@CSU: Content Analysis. Retrieved from http://writing.colostate.edu/guides/pdfs/guide61.pdf

Childress, V., & Rhodes, C. (2008). Engineering student outcomes for Grades 9–12. The Technology Teacher. 67(7), 5–12.

Edström, K., Soderholm, D., & Knutson Wedel, M. (2007). Teaching and learning. In E. Crawley, J. Malmqvist, S. Östlund, & D. Brodeur (Eds.), Rethinking engineering education: The CDIO approach (pp. 130–151). New York, NY: Springer. doi:10.1007/978-0-387-38290-6_6

Gray, P. J. (2007). Student learning assessment. In E. Crawley, J. Malmqvist, S. Östlund, & D. Brodeur (Eds.), Rethinking engineering education: The CDIO approach (pp. 152–165). New York, NY: Springer. doi:10.1007/978-0-387-38290-6_7

Gronlund, N. E. (1998). Assessment of student achievement (6th ed.). Needham Heights, MA: Allyn & Bacon.

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). National Academy of Engineering & National Research Council. (2014). STEM integration in K–12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press. doi:10.17226/18612

Kelley, T. R. (2008). Examination of engineering design in curriculum content and assessment practices of secondary technology education (Unpublished doctoral dissertation). University of Georgia, Athens, GA.

Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage.

Lewis, T. (2005). Coming to terms with engineering design as content. Journal of Technology Education, 16(2), 37–54. Retrieved from http://scholar.lib.vt.edu/ejournals/JTE/v16n2/pdf/lewis.pdf

Lau, A. (2013, January 13). Is ethics important to engineers? Rock Ethics Institute. Retrieved from http://rockethics.psu.edu/resources/ documents/engineering.pdf

McCracken, W. M., & Waters, R. (1997). Assessment and evaluation in problem based learning. In Proceedings of the 27th annual Frontiers in Education Conference: Teaching and learning in an era of change (pp. 689–693). New York, NY: Institute of Electrical and Electronics Engineers. doi:10.1109/fie.1997.635894

National Research Council. (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. doi:10.17226/13165.


-77-

Norton, R. E. (1999). SCID handbook. SCID: a competency-based systematic curriculum and instructional development model. Columbus, OH: Ohio State University.

International Technology Education Association. (2007). Standards for technological literacy: Content for the study of technology (3rd ed.). Reston, VA: Author.

Thomas, J. W. (2000). A review of research on project-based learning. San Rafael, CA: Autodesk Foundation. Retrieved from http://www.bobpearlman.org/BestPractices/PBL_Research.pdf

Wicklein, R. C. (2005). Critical issues and problems in technology education. The Technology Teacher, 64(4), 6–9.

Woods, D. R. (1996). Problem-based learning: Helping your students gain the most from PBL. Hamilton, Ontario, Canada: McMaster University.

Woods, D. R. (2000). Helping your students gain the most from PBL. In Tan, O. S., Little, P., Hee, S. Y., & Conway, J. (Eds.). Problem-based Learning: Educational innovation across disciplines. Singapore: Learning Academy, Temasek Polytechnic. Retrieved from http://www.tp.edu.sg/staticfiles/TP/files/centres/pbl/pbl_donaldwoods.pdf

Woolfolk, A. (2013). Educational psychology (12th ed.). Upper Saddle River, NJ: Pearson

Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Thousand Oaks, CA: Sage.

About the Authors

Cheryl A. Wilhelmsen ([email protected]) is Director of the Industrial Technology Program at the University of Idaho. Raymond A. Dixon ([email protected]) is Assistant Professor in the Department of Curriculum and Instruction at the University of Idaho.


-78-

Characterizing Design Cognition of High School Students: Initial Analyses Comparing those With and

Without Pre-Engineering Experiences

John Wells, Matthew Lammi, John Gero, Michael E. Grubbs, Marie Paretti, & Christopher Williams

Abstract

Reported in this article are initial results from of a longitudinal study to characterize the design cognition and cognitive design styles of high school students with and without pre-engineering course experience over a 2-year period, and to compare them with undergraduate engineering students.

The research followed a verbal protocol analysis based on the function–behavior–structure (FBS) ontology, which employs a task-independent approach that is distinct from a task-based or an ad hoc approach. This approach to protocol analysis is applicable across any process-based view of designing and generates results based on a common comparative measure independent of the design task.

In this article, Year 1 results are presented comparing only students in their junior year of high school who had formal pre-engineering course experience (experiment group) with those who did not have formal pre-engineering course experience (control group). Specifically, data collected from design sessions were analyzed for comparison of design issues and processes between experiment and control groups, respectively. Results from analysis of Year 1 data did not reveal any significant differences between the experiment and control groups in engineering design cognition. Based on these results, one would conclude that students with pre-engineering course experience do not demonstrate a stronger focus on the process of producing design solutions than do students without such experience. Although analysis of demographic data from high school participants indicates some degree of common prior pre-engineering experiences, it did not provide a sufficient explanation for why no significant differences in engineering design thinking were found between these groups. The researchers anticipate that Year 2 data will indicate that as the pre-engineering students continue engaging in formal engineering design experiences during their final year of high school, some degree of difference in design cognition will be demonstrated.

Keywords: design cognition; verbal protocol analysis; high school pre-engineering.


-79-

Background Engineering design used as an instructional strategy at the PK–12 level is

increasingly being embraced as a core learning method and as a pedagogical tool for integrative STEM education (Kolodner, 2002, Wells, 2010). As a key stakeholder in this trend toward integration of engineering design in K–12 STEM education curricula, it is critical that the elementary and secondary technology and engineering (T/E) education community understands the impact that such experiences have on student development of design practices. Few studies have examined the cognitive characteristics of K–12 students during T/E design-based learning (DBL) activities. Moreover, the way in which secondary students approach the engineering design process is not well understood (Katehi, Pearson, & Feder, 2009; Silk & Schunn, 2008) nor is whether that approach differs between students who have engaged in formal engineering experiences through pre-engineering course work and those who have not. Within the context of increasing opportunities for K–12 students to engage in both formal and informal T/E design activities, investigations regarding the extent to which such high school experiences contribute to a student’s capacity for design thinking (cognition) are needed. The intent of the research reported in this article was to characterize the design cognition of high school students and specifically to compare the design practices between high school students with and without formal pre-engineering design experiences.

Though few would argue that the design literature in engineering education has been somewhat singularly focused on pedagogical issues, there is a growing body of literature from studies that seek to understand the characteristics of design thinking behavior from a cognitive viewpoint (Cross, 2004; Lawson, 2004). Among these studies, protocol analysis is the research method of choice (Atman & Bursic, 1998; Dorst & Cross, 2001) for investigating design cognition and has been the basis for many of the more recent design cognition studies (Adams, Turns, & Atman, 2003; Atman et al., 2007; Christensen & Schunn, 2007). The research study presented in this article followed a verbal protocol analysis based on the function–behavior–structure (FBS) ontology developed by Gero (1990) and its extension, the situated FBS (sFBS) ontology (Gero & Kannengiesser, 2004), as a design-based coding scheme. The FBS protocol analysis employs a task-independent approach, which is distinct from a task-based or an ad hoc approach. This approach to protocol analysis is applicable across any process-based view of designing and generates results based on a common comparative measure independent of the design challenge (task). In this way, the FBS protocol analysis addresses the underlying cognitive processes, as opposed to the standard behavior-based analysis, and therefore provides a uniform basis for comparisons between students with different educational preparation and backgrounds and from different educational environments (Jiang, Gero, &Yen, 2014; Williams, Gero, Lee, & Paretti, 2011).


-80-

Function–Behavior–Structure Verbal Protocol FBS Ontology

The FBS ontology presents designing as the process of converting a set of functions into a set of design descriptions whereby those descriptions accurately convey an artifact capable of such functions (Gero, 1990). The design process is characterized in the FBS ontology (Figure 1) using three classes of ontological variables—function, behavior, and structure—as well as the external design requirements given the designer and a final description of the designed structure. Modeled in this way, function (F) is defined as the teleology of a designed object, and the behavior of that object is either what is expected (Be) from the structure or derived (Bs) from the structure. The structure (S) of an object represents individual components and the relationships among them. The external design requirements that the designer is given are designated by R, and the resultant set of design descriptions designated by D. These six ontological variables in the FBS model map onto design issues and serve as the basis for design cognition.

Figure 1. FBS framework (Resource: Kannengiesser, Gero, Wells, Lammi, 2015).

A design description is the result of a designer having progressed through a

set of eight distinct processes each of which reflects their movements (Numbers 1–8, Figure 1) among the ontological variables. The first five processes reflect an implied linear sequence of movements that include formulation, synthesis, analysis, evaluation, and documentation. In formulation (1), requirements are


-81-

transformed into functions and functions into a set of expected behaviors; synthesis (2) results in a proposed structure to satisfy expected behaviors; analysis (3) of the proposed structure produces derived behaviors; in evaluation (4), both expected behavior and behavior derived from structure are concurrently assessed; and documentation (5) generates the design description. The iterative nature of designing is captured in the movement among three types of reformulation processes, which are also denoted numerically in Figure 1: Reformulation I (6) is the reformulation of structure; Reformulation II (7) is a reformulation of expected behavior; and Reformulation III (8) is a reformulation of function. FBS Coding Scheme: Design Issues and Processes

The coding scheme adhered to in this research is based on this FBS ontology whereby the ontological variables are translated into six design issues. These design issues are coded using the FBS ontology, as exemplified in the sample of participant utterances and associated codes seen in Table 1. The selected utterances were drawn from an engineering design session in which high school participants were asked to design a device that would assist elderly clients in opening a stuck double-hung window. Transformations between the six codes used to label the design issues reflected in participant utterances generate the eight distinct design processes (Table 2).

Table 1 FBS Coding Examples

Design Issues Respective Utterance Example Design Requirements (R)

"so they need help in trying to…. for the elderly to raise windows"; "it says a significant amount of force to raise and lower the windows…"

Function (F) "but it'd have to be something that is really easy to twist."; "causes the window to expand on the frame"

Behavior Expected (Be)

"that will increase mechanical advantage"; "that may help the elderly lift or…";

Behavior from Structure (Bs)

"so if they like pull the string it actually lifts it"; "so the longer this is the more mechanical advantage you'll have so the easier it will be"

Structure (S) "So one thing I came up with is to cut a notch in the bottom frame of the window right there"; "and have the strings coming back down"

Design Description (D)

"let's draw a right side view of this thing to explain it okay I'll let you do that..."


-82-

Unidirectional transformational movements are indicated by the "→" symbol, the "↔" symbol indicates transformational comparisons, and the numbers associated with each design issue correspond to those depicted in the FBS model (Figure 1). Table 2 FBS Design Processes

Progression Design Process Transformational Movement (1) Formulation R → F, F → Be (2) Synthesis Be → S (3) Analysis S → Bs (4) Evaluation Be ↔ Bs (5) Documentation S → D (6) Reformulation I S → S (7) Reformulation II S → Be (8) Reformulation III S → F

Method

The research design followed a two-by-two factorial investigation across two exogenous variables, design experience and maturity, in which experience is formal pre-engineering coursework and maturity was the time between data collected fall of the junior and senior years of high school. The full scope of the research was to characterize the design cognition and cognitive design styles of high school pre-engineering students over a 2-year period and to compare them with undergraduate engineering students as well as high school students without such design experience. Presented in this article are Year 1 results comparing only the high school participants and only addressing the following hypothesis, which was one of six hypotheses posed in this study: High school pre-engineering students have a stronger focus than high school students with no design experience on the design process of synthesis (i.e., the process of producing solutions).

Using purposeful selection, high school students in their junior year were assigned to experiment (those with formal pre-engineering course experience) and control (those without formal pre-engineering course experience) groups. In teams of two (dyads), students engaged in a predefined engineering design task in which they were to develop a design-only solution. A dyad configuration was used because it has been found to naturally promote authentic verbal interactions during collaborations on developing acceptable engineering design solutions (Kan & Gero, 2009; Purzer, Baker, Roberts, & Krause, 2008).


-83-

Participants Participants were drawn from a convenience sample of high school juniors

attending one of three rural, mid-Atlantic high schools that offered the same ninth through twelfth grade Project Lead the Way (PLTW) pre-engineering course sequence. Student populations at each of the participating schools were of similar size. Two groups of participants, those with (experiment) and those without (control) formal PLTW pre-engineering course experience, were recruited from each high school, using a small monetary incentive. Prior PLTW course experience for the experiment group ranged from those enrolling in their first PLTW course at the start of their junior year to those with one full year of prior PLTW coursework. Within groups, students self-selected into dyads, 60% of which were mixed-gender. Of the 40 students participating in Year 1, the gender distribution within the experiment group was 64% male and 36% female, and for the control group, it was 65% male and 35% female. Procedures

Participant recruitment was conducted using typical modes of school communication. Student demographic data (e.g., age, gender, pre-engineering course experience) were collected as students arrived at their session and before dyads engaged in the design task. The design task dyads addressed was that of designing a solution to assist physically impaired elderly nursing home residents with opening difficult-to-open, double-hung windows. Instructions for completing the design task were provided as well as basic information resources regarding the construction and operation of a double-hung window. Dyads were allowed 45 minutes to collaborate on their design task and instructed to include a detailed sketch of the final solution on a whiteboard. Data Collection and Protocol Analysis

The following sequence of tasks presents the basic set of procedures used for data collection and protocol analysis.

Video Capture. Each design task session was captured using two video cameras that were arranged at two distinct vantage points ensuring sufficient recording of dyad interactions and their development of a final design description (Figure 2). Additionally, the two video cameras safeguarded against potential technological issues or difficulties encountered by either device. The first camera directly captured the white board and dyads engaged in progressive design development and sketching of their solution, while second camera recorded a general view of the entire design session. Both members of a dyad were equipped with a high-sensitivity wireless microphone to ensure that quality audio was captured for successful transcription of student verbalizations into text. The resulting videos provide a time-stamped recording of the entire design session.


-84-

Figure 2. Participants sketching final design solution descriptions.

Transcription. Video recordings of dyad design sessions were transcribed

manually with individual utterances from each dyad member entered verbatim into alternating rows of a spreadsheet. Timestamps were inserted every three minutes to establish reference points throughout the entire video. This approach to transcription resulted in a written version of the verbalizations between participants with time stamps throughout.

Segmentation and coding of text-based verbalizations. The method used to segment the text-based version of dyad verbalizations was conducted on the basis of the FBS coding that was previously described. This method involved concurrent analysis of a given transcript by independent coders. A total of six coders were involved with coding the 40 protocols. All coders participated in training using practice protocols until consistently achieving sufficient


-85-

intercoder reliability. Coders segmented and coded simultaneously, dividing the utterances until each individual segment contained a single code that reflected only one of the six possible design issues (Kan & Gero, 2007). The use of two independent coders ensured robustness and demonstrated an intercoder reliability ranging from 85% to 95%, which was consistent with prior research (Williams et al., 2011).

Arbitration. After independent coders completed the segmentation and coding of a given transcript, they would meet to arbitrate—compare, discuss, and justify—the FBS codes that they assigned to each segment. When agreement of independently coded segments occurred, a final code was assigned. Segments that differed in assigned codes required coders to engage in arbitration to dispute the assigned coding and reach agreement on the design issue addressed. If coders were unable to agree on an arbitrated code, that segment was left uncoded and was highlighted for subsequent final arbitration between the lead researchers. The final arbitration resulted in a final protocol data set that was readied for use in statistical analyses. The number of segments typically generated from the final protocol for a 45-minute design session was between 200 and 700. Because there are six codes, this implies that, on average, each code would likely appear at least 33 times. This provides a statistically significant data set. Analyses of final arbitrated protocols were conducted using LINKODER (www.linkoder.com) to generate descriptive statistics and probability analyses of the FBS ontology. Data were analyzed to determine statistical differences in design issues and processes between the control and experiment groups.

Results In this article, we report on the analyses of the first year data collected from

design sessions of participating high school juniors. These data were analyzed for comparison of design issues and processes between experiment and control groups, pre-engineering (ENG) and nonengineering (NON), respectively. Design Issues

A comparison of design issue distributions between ENG and control NON groups is illustrated in Figure 3, and descriptive statistics are presented in Table 3. The percent occurrence reflects the average within group frequency of segments associated with each of the six design issues for both groups. The data indicate that both groups expended the majority of their cognitive efforts (~ 40%) in discussions of the design structure (S), which is typical for most designers. Relatively similar total percent effort (~26–28) was expended on behavior from structure (Bs) and expected behavior (Be) combined (~14–18). Comparisons of control and experiment group data using a t-test (Table 3) revealed no significant differences among any of the design issues, although expected behavior approached it. Similarly, comparison of the total effort


-86-

expended in the problem versus the solution space (P–S Index; Jiang et al., 2014) indicated that there were no significant differences.

Figure 3. Percent occurrence of design issues: ENG vs. NON high school juniors. Error bars indicate standard deviation. Table 3 Statistical Results of Design Issues (Entire Session): ENG vs. NON High School Juniors

Design Issue t - value (%) p - value (R) Requirement -0.78 0.240

(F) Function 1.05 0.153

(Be) Expected Behavior -1.7 0.053

(Bs) Behavior from Structure 0.43 0.334

(S) Structure -0.23 0.410

(D) Description 1.09 0.145

P-S Issue Index -1.2 0.123 Design Processes

The distribution of syntactic design processes was computed to discern differences in the cognitive effort expended between control and experiment


-87-

groups. Similar to computations of design issues, analytical comparisons of the eight syntactic design processes showed no statistically significant differences between ENG and NON groups (Table 4). No statistically significant differences were observed in the P–S Processes Index between these two groups. Table 4 Statistical Results of Design Processes (Entire Session): ENG vs. NON High School Juniors

Design Process t - value (%) p - value Formulation 1.22 0.118

Synthesis -1.01 0.163

Analysis 1.48 0.077

Evaluation -0.16 0.436

Documentation 0.82 0.211

Reformulation I -0.5 0.311

Reformulation II -1.59 0.064

Reformulation III 0.55 0.295

P-S Process Index -0.92 0.183 Percent occurrences for the eight design processes (Figure 4) indicate that roughly 30% of their cognitive effort was invested in Reformulation I (S > S) and between ~17–21% on Analysis (Be<>Bs).


-88-

Figure 4. Percent design processes occurrences: ENG vs. NON high school juniors. Error bars indicate standard deviation.

Discussion and Conclusions Analysis of Year 1 data did not reveal any significant differences between

the experiment (ENG) and control (NON) groups in engineering design cognition. Based on these results, the underlying hypothesis must be rejected: Pre-engineering students do not demonstrate a stronger focus on the process of producing design solutions. To further investigate this apparent lack of difference between ENG and NON groups, the following select demographic data related to prior T/E design experiences were collected: participation in (a) middle school technology education classes, (b) T/E clubs, (c) other T/E-related activities, and because of the rural school settings, (d) farm-related activities. Analysis of these data indicated that of the ENG students, 59% had previously participated in middle school technology education classes, 14% were or had been involved in T/E clubs, 30% engaged in other T/E-related activities, and 30% had T/E-related farm experiences. In each of these demographic categories, students in the NON group had significantly less additional formal or informal T/E-related experiences, 33%, 5%, 17%, and 0% respectively. It is evident from these demographic data that students in the ENG group had far more formal and informal T/E-related experiences.

Although demographic data indicates some degree of common prior pre-engineering experiences, it does not provide sufficient explanation for finding no significant differences in engineering design thinking between these groups. Other influences such as curricular and pedagogical factors must therefore be


-89-

considered. Project Lead the Way (PLTW) program documents present entry-level course outlines that do not specifically target design thinking as a learning goal (https://www.pltw.org/our-programs/engineering). This is equally the case for the curriculum used by the middle school technology education programs at participating schools. The initial PLTW course that all pre-engineering participants engaged in was Introduction to Engineering Design (IED). A review of the detailed IED curriculum outline indicates that instructional units give attention to teaching the following set of practices and steps in the design process: technical sketching and drawing skills, modeling skills, geometry of design, documentation, and completion of a prescribed design project using computer-aided design (CAD) software. Authentic open-ended design challenges are not integral to the learning experience provided to students in this entry-level pre-engineering course. In light of this, it suggests that the pedagogical preparation provided to educators delivering the earlier courses in PLTW might not be adequate for intentionally incorporating or promoting design thinking as part of pre-engineering experiences.

Year 2 data of this longitudinal study are currently being collected. The researchers anticipate that as the pre-engineering students continue their engagement in engineering design experiences during their final year of high school, differences in design cognition will be demonstrated to some degree.

Acknowledgment The authors wish to acknowledge and thank Brooke Bellows, Yoon Lee,

Tyler Love, and Mick Smythwood for segmentation and coding of data. Appreciation is also extended to Pinelopi Kyriazi for conducting the statistical testing. This material is based upon work supported by the National Science Foundation under Grant No. 1160345. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This study has been approved by the authors’ respective Institutional Review Boards.

References Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering

designers: The role of reflective practice. Design Studies, 24(3), 275–294. doi:10.1016/S0142-694X(02)00056-X

Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosberg, S., & Saleem, J. (2007). Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education, 96(4), 359–379. doi:10.1002/j.2168-9830.2007.tb00945.x

Atman, C. J., & Bursic, K. M. (1998). Verbal protocol analysis as a method to document engineering student design processes. Journal of Engineering Education, 87(2), 121–132. doi:10.1002/j.2168-9830.1998.tb00332.x

Christensen, B. T., & Schunn, C. D. (2007). The relationship of analogical


-90-

distance to analogical function and pre-inventive structure: The case of engineering design. Memory & Cognition, 35(1), 29–38. doi:10.3758/BF03195939

Cross, N. (2004). Expertise in design: An overview. Design Studies, 25(5), 427–441. doi:10.1016/j.destud.2004.06.002

Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem-solution. Design Studies, 22(5), 425–437. doi:10.1016/S0142-694X(01)00009-6

Gero, J. S. (1990). Design prototypes: A knowledge representation schema for design. AI Magazine, 11(4), 26–36. doi:10.1609/aimag.v11i4.854

Gero, J. S., & Kannengiesser, U. (2004). The situated function–behavior–structure framework. Design Studies, 25(4), 373–391. doi:10.1016/j.destud.2003.10.010

Jiang, H., Gero, J. S., & Yen, C.-C. (2014). Exploring designing styles using a problem–solution division. In J. S. Gero (Ed.), Design computing and cognition ’12 (pp. 79–94). Dordrecht, the Netherlands: Springer. doi:10.1007/978-94-017-9112-0_5

Kan, J., & Gero, J. S. (2007). Using the FBS ontology to capture semantic design information. In J. McDonnell & P. Lloyd (Eds.), DTRS7 (155–165). London, England: University of the Arts.

Kan, J. W. T., & Gero, J. (2009). Using the FBS ontology to capture semantic design information in design protocol studies. In J. McDonnell & P. Lloyd (Eds.), About: Designing. Analysing design meetings (pp. 213–229). Boca Raton, FL: CRC Press.

Kannengiesser, U., Gero, J., Wells, J., & Lammi, M. (2015). Do high school students benefit from pre-engineering design education? Proceedings of the 20th International Conference on Engineering Design (ICED), Vol 11, pp. 267-276: Human Behaviour in Design, July 27-30, Milan, Italy.

Katehi, L., Pearson, G., & Feder, M. (Eds.). (2009). Engineering in K–12 education: Understanding the status and improving the prospects. Washington, DC: National Academies Press. doi:10.17226/12635

Kolodner, J. L. (2002). Facilitating the learning of design practices: Lessons learned from an inquiry into science education. Journal of Industrial Teacher Education, 39(3), 9–40. Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v39n3/kolodner.html

Lawson, B. (2004). Schemata, gambits and precedent: Some factors in design expertise. Design Studies, 25(5), 443–457. doi:10.1016/j.destud.2004.05.001

Purzer, S. Y., Baker, D., Roberts, C., & Krause, S. (2008). Development of a team interaction observation protocol and a self-efficacy survey using social cognitive theory as a framework. In Proceedings of the 2008 American Society for Engineering Education Annual Conference & Exposition (pp. 13.415.1–13.415.15). Washington, DC: American Society for Engineering


-91-

Education. Retrieved from https://peer.asee.org/3203 Silk, E. M., & Schunn, C. D. (2008, January). Core concepts in engineering as a

basis for understanding and improving K-12 engineering education in the United States. Paper presented at the National Academy of Engineering/National Research Council Workshop on K-12 Engineering Education,, Washington, DC.

Wells, J. G. (2010). Research on teaching and learning in science education: Potentials in technology education. In P. A. Reed & J. E. LaPorte (Eds.), Research in technology education (pp. 192–217). 59th Yearbook of the Council on Technology Teacher Education. Reston, VA: Council on Technology Teacher Education.

Williams, C. B., Gero, J., Lee, Y., & Paretti, M. (2011). Exploring the effect of design education on the design cognition of mechanical engineering students. In Proceedings of the ASME 2011 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference (pp. 607–614). Washington, DC: American Society of Mechanical Engineers. doi:10.1115/DETC2011-48357

About the Authors

John Wells ([email protected]), is Associate Professor in the Integrative STEM Education Program at Virginia Polytechnic Institute & State University. Matthew Lammi ([email protected]), is Assistant Professor in the College of Education at North Carolina State University. John Gero ([email protected]), is Research Professor in the Krasnow Institute for Advanced Study at George Mason University & University of North Carolina. Michael E. Grubbs ([email protected]), is Supervisor of Technology and Engineering Education for Baltimore County Public Schools. Marie Paretti ([email protected]) is Associate Professor and Co-Director of the Virginia Tech Engineering Communication Center at Virginia Polytechnic Institute & State University. Christopher Williams ([email protected]) is Associate Professor and Senior Faculty Fellow in the Department of Mechanical Engineering at Virginia Polytechnic Institute & State University.


-92-

Scope of the JTE The Journal of Technology Education provides a forum for scholarly discussion on topics relating to technology and engineering-related education. Manuscripts should focus on technology and engineering-related education research, philosophy, and theory. In addition, the Journal publishes book reviews, editorials, guest articles, comprehensive literature reviews, and reactions to previously published articles. Technology and Engineering Education (T&EE) is a program that resides at the P-12 school levels for all students and at post-secondary institutions for those students interested in teaching or obtaining employment in the technology or engineering fields. Technology and engineering education is primarily taught by technology and engineering teachers, with a focus on engineering design. T&EE may be considered a stand-alone discipline or part of a larger discipline in science, technology, engineering, and mathematics (STEM). Regardless of the approach, T&EE focuses on technological literacy and engineering design; engineering design is the verb tense of engineering. At the P-12 grade levels, the goal is for students to develop technological and engineering literacy, regardless of career aspirations, through hands-on, contextual applications of technological and engineering concepts. T&EE students, use a hands-on approach to solve technological problems using problem solving and creativity, while working under constraints, which involves the use of optimization and predictive analysis. At the P-5 grade levels, technology and engineering concepts are integrated into existing coursework such as reading, mathematics, science, and social studies. Typical courses students would take at the 6-12 grade levels in a T&EE program would consist of (a) information and communication technologies, including computer-aided drafting and design, (b) engineering design, (c) construction technology, (d) manufacturing technology, (e) energy, power, and transportation technology, and (f) medical, agricultural, and related biotechnologies. Within these courses, students would utilize troubleshooting, research and development, invention and innovation, and problem solving. The focus of T&EE at the P-12 levels is not to prepare future engineering majors/students, but to provide an education for all students.

Editorial/Review Process Manuscripts that appear in the Articles section have been subjected to a blind review by three or more members of the Editorial Board. This process generally takes from six to eight weeks, at which time authors are promptly notified of the status of their manuscript. Book reviews, editorials, and reactions are reviewed by the Editor.


-93-

Manuscript Submission Guidelines One paper copy of each manuscript and an electronic version in Microsoft Word format should be submitted to:

Chris Merrill, JTE Editor Department of Technology Illinois State University 215 Turner Hall Normal, IL 61790-5100

1. Overseas submissions in Microsoft Word format may be sent electronically via the Internet (to [email protected]) to expedite the review process.

2. All manuscripts must be double-spaced and must adhere to the guidelines published in Publication Guidelines of the American Psychological Association (6th Edition). Tables and figures, however, should be imbedded within the text itself rather than at the end of the document.

3. All figures and artwork must be scalable to fit within the JTE page size (4.5” x 7.25” column width and length) and included electronically within the document.

4. Line drawings and graphs must be editable within Microsoft products and in vector rather than raster format when possible.

5. Shading should not be used as a background for illustrations or graphs and within bar graphs. If needed, fill patterns consisting of lines should be used.

6. Manuscripts for articles should generally be 15-20 pages (22,000-36,000 characters in length, with 36,000 characters an absolute maximum). Book reviews, editorials, and reactions should be approximately four to eight manuscript pages (approx. 6,000-12,000 characters).

7. Authors for whom English is not the primary language must enlist a native English editor for the manuscript prior to submission. This person and his/her email address must be identified on the title page of the manuscript.


-94-

Subscription Information The Journal of Technology Education is published twice annually (Fall and Spring issues). New and renewing subscribers should copy and mail the form below: Name (please print)

Mailing Address (please print)

Email address: Fax: □New Subscription □ Renewal Subscription Make checks payable to: Journal of Technology Education. All checks must be drawn on a US bank.

Regular (USA): $20 Regular (Canada/Overseas): $30 Library (USA): $30 Library (Canada/Overseas): $40 Individual Back Issues (USA): $10 each Individual Back Issues (Canada/Overseas): $15 each

Return remittance along with this form to: Chris Merrill, JTE Editor Department of Technology Illinois State University 215 Turner Hall Normal, IL 61790-5100

JTE Co-Sponsors & Membership Information The International Technology and Engineering Educators Association (ITEEA) is a non-profit educational association concerned with advancing technological literacy. The Association functions at many levels – from international to local – in responding to member concerns. The Council on Technology and Engineering Teacher Education (CTETE), affiliated with the ITEEA, is concerned primarily with technology teacher education issues and activities. For membership information, contact: ITEEA, 1914 Association Drive, Reston, VA 22091 (703) 860-2100.


-95-

Journal of Technology Education Editor Chris Merrill, Illinois State University Technical Editor Amanda Fain, Illinois State University Previous Editors Mark Sanders, Virginia Polytechnic Institute and State University (1989-1997) James LaPorte, Virginia Polytechnic Institute and State University (1997-2010) Editorial Review Board Vincent Childress, North Carolina Agricultural and Technical University

Rodney Custer, Black Hills State University

Jenny Daugherty, Purdue University

Marc deVries, Delft University of Technology, The Netherlands

Patrick Foster, Central Connecticut State University

W. Tad Foster, Indiana State University

Brenda Gustafson, University of Alberta, Canada

Ronald Hansen, University of Western Ontario, Canada

Christine Hailey, Utah State University

Oenardi Lawanto, Utah State University

Theodore Lewis, University of Trinidad and Tobago

Stephen Petrina, University of British Columbia, Canada

Philip Reed, Old Dominion University

Mary Annette Rose, Ball State University

Kay Stables, Goldsmiths University of London, England

Ken Volk, Masdar Institute of Science and Technology, Abu Dhabi

John Wells, Virginia Polytechnic Institute and State University

John Williams, University of Waikato, Hamilton, New Zealand


-96-

Electronic Access to the JTE All issues of the Journal of Technology Education may be accessed on the World Wide Web at: http://scholar.lib.vt.edu/ejournals/JTE/ (Note: this URL is case sensitive). The views expressed in this publication are not necessarily those of the Editor or the Editorial Review Board, or the officers of the Council on Technology and Engineering Teacher Education and the International Technology and Engineering Educators Association.

journal of technology education · journal of technology education vol. 27 no. 2, spring 2016-7-...

Documents