Development of a Comprehensive Curriculum & Learning Material for Embedded Digital Signal Processing Multimedia

Applications

1. OverviewThis document describes a proposed Phase-I project submitted to the Course, Curriculum,

and Laboratories Improvement (CCLI) program of the NSF, reflecting an effort of a strong team of PIs from the Florida Institute of Technology (FIT) in Melbourne, Florida.

The goal of this project is to produce exemplary materials for courses on embedded digital signal processing multimedia applications (detailed in Section 2.3 of the Project Plan). The project will involve the following components:

1. Developing five chapters of a textbook covering digital signal processing (DSP) theory and practical implementation aspects of this theory into embedded DSP hardware platforms

2. Developing 50% of an in-depth laboratory manual containing laboratory exercises that accompanies the book

3. Developing and implementing assessment and evaluation procedures to aid stages 1. and 2. as described in the Evaluation and Assessment section

a. The student’s input assessing our project will be carefully considered

b. Several colleagues specializing in the area from a number of selected universities will be consulted (see endorsement letters supporting the project)

c. A number of professionals from industry having vested interest in successful realization of this project will be consulted, along with editors of selected publishing houses

4. Developing a companion web-site including wiki pages to disseminate developed laboratory exercises as well as final student projects

5. Setting the groundwork for the Phase II of the project

Although there are a significant number of textbooks that cover fundamentals of DSP and signals and systems, there are almost no textbooks that adequately bridge the gap between that theory and practical applications implementing such concepts in DSP embedded systems (details discussed in section 2.4). This problem causes graduating electrical and computer engineering students to have inadequate training to handle the needs of the current hi-tech market. This issue of inadequate training was also strongly emphasized by a number of companies that were consulted (see letters of endorsement and support). The urgency for a proper solution of this issue is further exacerbated by the rising problem of lower student enrollment in engineering. Bill Gates, chairman of Microsoft, issued a warning that “our lead position where we develop the best people in this country and we bring the best people from other countries is eroding”. A Google search for “shortage of engineers” reports over 54,300 links. NPR’s “Morning Edition” on April 30, 2007, reported real danger in the current growing trend of outsourcing US jobs overseas. Specifically, it was emphasized that outsourcing R&D will mark the rapid decline of

the US. Similar views are shared by the “World Views” report “Engineer shortage marks US decline” which states: “America has led the world in science and technology, but that may be coming to an end. With a looming engineer shortage in the United States and a corresponding engineering boom in China and India, we are seeing the consequences of our educational collapse”. Furthermore, current statistics show that fewer and fewer American students are going into engineering programs. There are some 100,000 fewer engineering students than there were a decade ago. At the same time the need for well-trained engineers is increasing. High tech companies reported a 27% shortage in 2005.

To address issues related to the number and quality of graduating engineers, the National Academy of Engineering (NAE), in its forward-thinking reports “The Engineer of 2020: Visions of Engineering in the New Century” and “Educating the Engineer of 2020: Adapting Engineering Education to the New Century” argued that the best way for future engineers to cope with projected changes on our planet in the new century is to train them to be “life-long learners”(the issue is discussed in more detail in section 2.1). The proposed solution applies a cyclic model for knowledge production and improvement as exemplified by Figure 2.

Significant verbal and written support for the idea received from professionals in industry and academia as well as initial feedback from students validates the merit of the proposed effort (see the attached letters of endorsement and support as well as a number of evaluations).

The general strategy for executing this Phase I project (and Phases II & III) is adapted from NFS’s successful infinity project carried out for high-school students and teacher training. It is expected that this effort will lead to outcomes similar to those achieved by the infinity project http://www.infinity-project.org/ (see also Orsak et al.). This proposal, in contrast to the infinity project, addresses junior/senior level college students who must master the complexities of dealing with development in real hardware. Properly addressing this inherent complexity requires step-wise incremental introduction of basic concepts followed by detailed issues in hardware architecture and its instruction set. The assembled FIT team collectively has significant experience in this area and thus the ability to produce this exemplary material.

Dr. Kepuska has over 12 years of industrial experience in R&D in the area of speech processing, speech recognition and text to speech. Furthermore, Dr. Kepuska, prior to becoming a resident of the US had over 4 years of teaching experience as a lecturer at the University of Prishtina. After joining FIT four years ago, Dr. Kepuska introduced three graduate level courses covering Speech Analysis, Processing and Recognition as well as Text to Speech (ece5525, 5527 & 5527). In addition, Dr. Kepuska redesigned curriculum for four undergraduate courses covering embedded DSP development and WWW application development (ece3551 & 3552, ece 3553 & 4553). All four courses require laboratory exercises. The FIT team lead by Dr. Kepuska won the first place in the first annual competition for developing Voice Assistive Device applications. This competition was sponsored by Analog Devices and the University of Massachusetts at Lowell in June of 2005. More information can be obtained under the link:

http://faculty.uml.edu/Mufeed_Mahd/UML_ADI/photogallary.htm

Drs. Anagnostopoulos and Kepuska are successfully conducting NSF EMD-MLR project 0341601. Dr. Anagnostopoulos from has expertise in DSP and Image Processing (IP). He has taught several undergraduate and graduate courses in Signals & Systems, DSP and IP over the last few years. Furthermore, Drs. Kepuska and Anagnostopoulos have closely collaborated in the past 3 years on another prototype CCLI project, which is briefly described in the last section.

Through their involvement in this project, they both gained significant experience in mentoring, engaging and interacting with undergraduate students.

Dr. Ham is an accomplished author (“Principles of Neurocomputing for Science and Engineering,” published by McGraw-Hill in 2000) as well as an experienced researcher as demonstrated by his current position of President of the International Neural Network Society.

Dr. Converse’s experience in evaluation and assessment ensures that all aspects of the project are addressed and handled successfully.

All the project’s goals directly address the Creation of Learning Materials and Teaching Strategies component of the cyclic model for knowledge production and improvement advocated in this CCLI solicitation. It is important to mention that the five aforementioned goals of the prototype project will not be attainable without the NSF’s financial support. While the intellectual merit and the broader impacts of this effort are described in a narrative fashion in Sections 2, 3 and 4, the reader is referred to the Project Summary for a succinct presentation of these elements.

2. Project PlanThis document addresses the need for improving educational processes by producing a

comprehensive curriculum and set of learning materials. These materials will target the understanding of development processes of technology for electrical and computer engineering undergraduate students. This ultimate goal is motivated by the alarming trends reporting shortages of qualified engineers caused by inadequate training and education as described in the “Shortage of Qualified Engineers” section below. The proposed solution is envisioned as a continuous process of

1. Identification of current and future trends in technology, (see Future Trends inTechnology section)

2. Discovery of the difficulties that impede meeting the requirements for future engineers, (see Requirements and Difficulty section) and

3. Development of a comprehensive action plan as a foundation for improving upon the current situation (covered in Opportunity and Action Plan).

At the core of this plan is the development of a textbook and laboratory exercises that link DSP theory with development skills and DSP hardware. This is accomplished by applying the theory in the embedded DSP-based hardware with carefully crafted lectures and laboratory exercises.

To continuously refine and apply the above mentioned paradigms, evaluation and assessment materials will be developed (see Figure 2 for details). Information obtained from assessments and evaluations will be used to improve the material and its implementation. The outcome is continuous improvement and fine tuning of the developed material as well as the teaching and delivery procedures that aid the learning process.

It is expected that the results of this Phase I proposal will be sufficiently impressive to allow us to seek further funding for the second and third stage proposal similarly to the “Infinity” project for high-school students also funded by NSF.

2.1. Shortage of Qualified Engineers Marvin Minsky, distinguished MIT professor known as the founder of AI, in his recent

presentation on April 2, 2007, at the University of Central Florida, emphasized the projected significant shortfall of highly skilled engineers and scientist by 2020 and its impact on society. Two years ago, Bill Gates, chairman of Microsoft, issued a warning that “our lead position where we develop the best people in this country and we bring the best people from other countries is eroding”. A Google search for “shortage of engineers” reports over 54,300 links. NPR’s “Morning Edition” on April 30, 2007, reported real danger in the current growing trend of outsourcing US jobs oversees. Specifically, it was emphasized that outsourcing R&D will mark the rapid decline of the US.

Similar views are shared by the “World Views” report “Engineer shortage marks US decline” which states: “America has led the world in science and technology, but that may be coming to an end. With a looming engineer shortage in the United States and a corresponding engineering boom in China and India, we are seeing the consequences of our educational collapse”. Furthermore, current statistics show that fewer and fewer American students are going into engineering programs. There are some 100,000 fewer engineering students than there were a decade ago. At the same time the need for well-trained engineers is increasing. High tech companies reported a 27% shortage in 2005. To address issues related to the number and quality of graduating engineers, the National Academy of Engineering (NAE), in its forward-thinking report “The Engineer of 2020: Visions of Engineering in the New Century” and “Educating the Engineer of 2020: Adapting Engineering Education to the New Century” argued that the best way for future engineers to cope with projected changes on our planet in the new century is to train them to be “life-long learners.” The engineering community generally agrees that future engineers will need to be trained in an entirely different manner than today. The attributes of an engineer of 2020 include the ability to communicate effectively, understand the development of technology, interact effectively in teams, and have a comprehensive understanding of engineering fundamentals. This can only be done when engineers have a solid core in math and science, understand interdisciplinary learning requirements, and have a real experience in engineering research.

2.2. Future Trends in TechnologyRecent years have witnessed proliferation of the use of personal devices with multimedia

capabilities like cellular phones, i-Pod, i-Phone, etc. Those devices incorporate state-of-the-art from several research areas including digital signal processing of speech, audio, image and video coding/compression methods, communication, and hardware (specifically digital signal processors), as well as machine learning and software development.

Requirements and DifficultyDevelopment of applications on such multimedia capable devices requires specific skill sets

in a number of areas. Those diverse areas necessitate familiarity and knowledge of

i. DSP Theory

ii. DSP Algorithm Implementation:

a. Programming language(s) such as C/C++

b. Numerical effects of finite precision number representations and operations

c. Design and analysis tools (e.g., MATLAB, LabView)

iii. Software Development for Embedded Platforms (e.g., VisualDSP++, Code Composer Studio):

a. Hardware architecture, specifically those of digital signal processors (DSPS)

b. Real-time event-driven programming and thus familiarity with hardware interrupts and event management

c. Code optimization and desirably assembly level programming of core consuming processor cycle functions

Opportunity and Action PlanTo enable college level students to be leaders in the area of development of embedded signal

processing applications, the Electrical and Computer Engineering (ECE) Department of FIT with its Advisory Board comprised of representatives from industry (NASA, Boeing, Northrop Grumman, Harris Corp, Honeywell, AFTEC, SAIC, Modus Operandi, Rockwell Collins, Synopsis, Inc, Zel Technology, Brevard Co. School Board, Intersil, Secureboration, Intelligence Data Sys, Health First Inc., Conexant Sys, Inc., Jones, Edmunds & Assoc, Progress Energy Florida), has endeavored to introduce a comprehensive curriculum that addresses the fundamentals of the scientific disciplines of DSP as well as Embedded Systems Programming and Development. This was done by recognizing the need to advance Electrical and Computer Engineering curriculum to cover the areas of embedded systems development for multimedia. Part of this strategic decision was the acquisition of the state-of-the-art software and hardware with which it became possible to start transforming outdated curriculum and laboratory equipment, providing the foundation for this proposal.

DifficultyAdoption of the state-of-the-art curriculum poses several problems as became evident at the

onset of implementing the curriculum change in two of the core courses (Microcomputer Systems 1 & 2).

1. Lack of textbooks and more generally the lack of lecture materials is the first and foremost problem.

2. Comprehensive coverage of the material requires significant and continual investment in software tools and development systems.

3. Continuous assessment and evaluation of the educational process is required for proper application of hardware, software tools, laboratory exercises, and delivery of the course content.

As expected, an extensive search for proper textbooks, lecture materials, and www/internet sites resulted in very little or no appropriate material suitable for adoption. No adequate textbooks, lecture materials, or internet sites were found that address the subject matter appropriately in an integrated and comprehensive manner that is required to deliver the curriculum properly. Obviously, partial information can be pieced together at a significant effort to fit it in a comprehensive curriculum such as the one that is being envisioned. However, such an approach is not optimal nor will it provide the desired outcome of a comprehensive curriculum. This result was expected because the cycle of new technologies is significantly

shorter than the development cycle for the teaching material. In spite of this shortcoming it is possible to advance the teaching material more rapidly by adopting a proper strategy as described in this proposal.

2.3. Solution IdeaIt has been recognized that the successful transformation into the new curriculum requires

accelerated adoption of the subject matter. This fast-track adoption is made more difficult due to the fact that the subject is spread over five core courses in the ECE Department at FIT. Furthermore, it requires significant time, effort, and expense to develop a comprehensive set of lecture materials, delivery mechanisms, and assessment and evaluation procedures.

Accelerated adoption entails development of:

1. Comprehensive textbook2. Lecture notes3. Laboratory exercises4. Internet site with reference material and links, organized and accessible as standard

internet pages as well as wiki pages,5. Partnerships with industry

a. Manufacturers (e.g., Analog Devices, Texas Instruments) b. Development tools for analysis and design of systems (e.g., MATLAB, Simulink,

LabView)6. Periodic acquisition of the state of the art of development software and hardware

platforms7. Continuous Assessment and re-Evaluation:

a. Development of the procedures for assessment and evaluation via consultations with internal as well as external experts from academia and industry

b. Peer reviewsc. Evaluation of the projects and the material d. External feedback from those in industry directly interested in the subject matter

The proposed project learning material supports three basic steps depicted in the following Figure 1:

Figure 1. Three Basic Steps of Design, Prototype and Deployment, supported with learning material. These steps are widely used in industry.

The proposed solution addresses junior and senior college students by introducing in-depth issues that deal with state-of-the art in:

Design Prototype Deploy

DSP hardware

Software development tools associated with that hardware

Development strategies and techniques

The students will learn to work at multiple levels from design of the proof-of-concept to actual implementation, utilizing available software tools (LabView or MATLAB). The learning strategies will be streamlined (see Figure 2. for details) to guarantee optimal delivery methods - ensuring that students will become proficient and comfortable with the tools as well as procedures necessary to achieve stated goals.

Figure 2. Proposed process of generating the complete course work material in close consultation with industry and continuous evaluation and assessment phase feeding back information for the next cycle of improving the delivery and the material itself.

To quickly test and develop ideas, the embedded system development cycle requires familiarity with software tools for simulation and rapid-prototyping. This step leads to a design prototype that can be tested and evaluated against design specifications. Proper use of software tools reduces the time required for development. Modern tools, like LabView and MATLAB, provide for additional software modules/toolkits that enable the designer to generate embedded code from the tested simulation. Those toolkits (if available1) further reduce the time required to port the simulation system into the embedded platform. Thus, they allow designers to focus on the solution and less on the software development. After completion of the design and its testing, most often it is necessary to (re)code the design using integrated development environment (IDE) of corresponding DSP/hardware (i.e., VisualDSP++ or Code Composer Studio) in order to achieve the best possible implementation. This, in turn, necessitates that developers be familiar with the architecture of the hardware involved as well as the C/Assembly programming language supported by IDE.

1 At the moment only few DSP chips are supported by MATLAB or LabView

Text Book

Lecture

Notes

Lab

Exercises

Wiki &

Internet

Site

Partnerships with Industry

Learning Material Development Cycle

Evaluation &

AssessmentAdoption of

Hardware &

Software Tools

Proficiency in all stages can be achieved by:

Successfully teaching and applying domain knowledge, in this case DSP theory to development of various audio, speech, image and video applications

Successful application and dissemination of the developed learning material

The developed learning system is expected to have direct impact on industry and the quality of graduating students from FIT as well as other institutions that adapt the proposed strategy and material.

The Gantt chart presented below depicts one cycle of the timeline of the proposed effort.

Figure 3. Gantt chart depicting the time-line of the major tasks for one cycle of the process presented in previous Figure 2.

In the following sections each item depicted in Figure 2 and Figure 3 is discussed in detail.

2.4. Development of a Comprehensive TextbookAlthough there are a significant number of the textbooks that cover the fundamentals of DSP

and signals and systems (Oppenheim, A.V., Schafer, W. R. (1975). Rabiner, L.R., Schafer, R.W., (1978). Jayant, N.S., Noll, P. (1984). Orfanidis, S., (1996). Zölzer, U., (1998).Oppenheim, A.V., & Schafer, W. R., Buck J.R., (1999). McClellan, J.H., Schafer, R.W., & Yoder, M.A., (1999). Embree, P.M., Damon, D., (1999). Gold, B., Morgan, B., (2000). Quatieri, T.F., (2001). Kondoz, A.M., (2004). Orsak, G.C., Wood, S.L., Douglas, S.C., Munson Jr., D.C., Treichler, J.R., Athale, R., Yoder, M.A., (2004). Bose, T., (2004). Kuo, S.M., Gan, W.S, (2005). Acharya, T., Ray, A.K., (2005). Schilling, R.J., Harris, S.L., (2005). Blanchet, G., Charbit, M., (2006). Ingle, K. V., Proakis, J.G., (2007). Gan, W.S, Kuo, S.M., (2007). Roberts, M.J. (2007).), just to name a few, there are almost no textbooks that adequately bridge the gap between that theory and practical applications implementing such concepts in DSP embedded systems.

More recently, two publications have appeared that attempted to address the issue of DSP embedded systems, namely Kuo & Gan and Gan & Kuo. Closer review of the books reveals that they are good reference material but fall short in addressing the subject with adequate detail for undergraduate students. The Kuo and Gan book addresses theory and some of the practical issues regarding assembly programming. However, the book is centered on simulation and not emulation and predominantly uses examples of assembly level programming without adequate coverage of the assembly language itself. The book refers to TMS320 Assembly Language Tool User’s Guide (see page 155). The book includes a CD with the Simulator tool, additional experiments, documentation for MATLAB and Simulink and some useful Websites.

On the other hand the Gan and Kuo book focuses entirely on the application side with very little DSP background. It is a good book for the students that already have mastered DSP theory and development process. In other words it is a great reference resource.

The fundamental idea proposed in this project is to provide a single source starting with a comprehensive textbook along with all the necessary resources as outlined above. The textbook will be aimed primarily at the undergraduate level, but will also cover advanced concepts in sufficient detail to be useful to graduate students as well as professionals.

The book will cover theoretical aspects of DSP utilizing MATLAB as the analysis and prototyping laboratory. Transition from theory to practical applications will be covered in detail emphasizing all the issues that arise when infinite precision analysis is replaced by a finite precision device that performs finite precision operations. Proper understanding of finite precision representations requires understanding of fundamental concepts in number representations. In addition it is required to analyze the effect of finite precision arithmetic in the implementation of various algorithms. The bottom line is - understanding of the effects of finite precision representations and operations is fundamental in mastering embedded systems development. Utilization of MATLAB as an analysis tool will provide for visual feedback and validation of the introduced concepts.

Since the focus of the textbook is in the application of various DSP methods, the primary development language will be C. This language is sufficiently rich and powerful while at the same time very close to the hardware to warrant efficient implementation. In addition C/C++ language is covered universally in most if not all ECE university programs.

The application areas are all relevant to today’s multimedia technologies and range from audio, speech, image and video processing. The applications range from end-user processing (e.g., surround sound, noise removal) all the way to signal coding and compression for communications systems all utilizing the theoretical concepts introduced in the first half of the book.

2.5. Lecture Notes and Laboratory MaterialSince the subject matter of this material is already being introduced in part, some of the

lecture notes are already available as PowerPoint Presentation documents. See for example the link:

http://my.fit.edu/~vkepuska/ece3551/Ch1-Introduction%20to%20DSP's.ppt

In addition to lecture notes, the lab materials, as well as various projects are made available under respective links:

http://my.fit.edu/~vkepuska/ece3551/LAB/LAB4_02_16_2007.doc

http://my.fit.edu/~vkepuska/ece3551/Projects/

Exceptional projects from Fall 2006 (the inception of the course) and Spring 2007 are:

Brandon Schmitt: “Maze”

Andrew Lash: “Bass Guitar Multi-Effects Pedal in DSP”

Elisabeth Nelson: “Simon Says”

Valerie Bastian: “Sound Generation”

Sean Powers: “Visual Audio FX”

Todd Alexander: “EzDJ”

Xerxes Beharry: “Equalizer”

Ronald Ramdhan:“Speaker and Sound Modulation”

The project reports are uploaded as supplemental documents for review and assessment considerations. Each project code implementation is also available via the link provided above.

The additional practice problems and exercises will be developed and provided as progress is made with development of the book material.

However, what is currently needed is the organization of this data into a suitable Web-based interface for easy access and search of the provided material which will be made available with the support from this project. FIT students can also access the material through the Blackboard system. However, this system does not grant public access to the posted resources.

3. Outreach Efforts Central to dissemination of the developed material is development and maintenance of an

internet site hosted by fit.edu that would enable easy access to the proper information. This implies careful organization of all the material as well as the ability to search and locate correct information within the site. Furthermore, since some of the material contained must be made available only to instructors and faculty of the educational institutions that adopt the curriculum, appropriate administrative functions should be provided within the developed www application. Due to the popularity of the wikipedia, using that technology to deliver such content efficiently is one of the primary goals. Since the PI is also the instructor for courses that cover web applications development (ece3553 Multifarious Systems), it is only logical to include students in the development of this application, utilizing learned skills.

3.1. Partnerships with IndustryDr. Kepuska’s past industrial experience ensures that the needs for qualified engineers are

properly addressed. His experience will continue to be utilized in establishing partnerships with industry. However, NSF funding of this project will add to the credibility of the assembled team to approach and seek further and more substantial collaborations for mutual benefit. For example, industry can be convinced to invest in development of a more complete development platform.

The number of submitted letters endorsing the idea of this project is also a significant indicator of future collaborations. Ultimately, upon completion of Phase I, II and III the developed program should provide a solid foundation for establishing continuous education training for professionals from industry. This outcome will leverage already developed material and enable the host institution (FIT) to support its laboratory with the most up-to-date hardware and software.

3.2. Partnerships with UniversityAt this point, FIT has secured collaboration with two universities: the University of Central

Florida and Clemson University. Those schools will be used to provide additional feedback in

the early stages of development of the learning material. Upon completion of the book and the learning material (Phase II), it is expected that more schools will adapt the materials due to their availability and exemplary content.

4. Evaluation and AssessmentThe primary goal of this project is to develop a complete set of learning materials for

embedded digital signal processing applications courses. Given this, project evaluation will focus on two main questions:

1. Content: To what extent do the course materials cover the appropriate content for courses of this type?

2. Method: How effectively do the course materials cover this content?

The first question is essentially one of content validity (e.g., Nunnally & Bernstein, 1994). Given that suitable instructional materials do not exist and there is no clear model on which to base development of these materials, a fundamental issue that must be addressed in this project is how well the content covered in the developed materials corresponds to the knowledge base needed in this area. To address this issue, subject matter experts (SMEs) from both academia and industry will be involved throughout the project to provide guidance and feedback on the content validity of the instructional materials. This type of content validity approach to evaluation has been used successfully in several curriculum development and revision projects (e.g., see Ford & Wroten, 1984; Goldstein & Ford, 2002; Teachout, Sego, & Ford, 1997/1998).

The second question relates to the effectiveness of the course materials. Clearly some approaches to covering the course content identified through the evaluation process are likely to lead to greater knowledge and skill development than others. Thus, the second major evaluation component will involve collecting data relevant to learning material efficacy. The same SMEs described above will also provide guidance and feedback regarding the likely effectiveness of the materials developed. In addition, as the materials are introduced, students enrolled in the classes will be interviewed and surveyed to collect reactions to the course materials. Student performance on lab work and final projects will also be examined as initial evidence of effectiveness. Although in-depth analysis of student performance is beyond the scope of this phase of the project given the focus on material development and refinement, examination of performance on these assessments will likely provide additional insight into areas in need of improvement.

Figure 3 provides an overview of the evaluation strategy.

Figure 3. Proposed evaluation process.

4.1. Data SourcesConsistent with recommendations for evaluation in Science, Technology, Engineering, and

Mathematics courses (e.g., National Research Council, 2003a), this project will collect information from multiple sources, each with a unique and useful perspective on the course materials. Specifically, data will be collected throughout this project from SMEs at other universities and in industry who have written letters of endorsement for this project, as well as students enrolled in the courses as the materials are introduced.

SMEs from other universities are an extremely useful source of information regarding both questions outlined above (content and method) based their knowledge of the subject area and experience in teaching related courses. Furthermore, given that the goal of this project is the development of a complete package of learning materials that will ultimately be adopted by faculty at other institutions; these individuals also represent the perspective of potential consumers of the end product. Thus, the guidance and feedback provided by these individuals will be vital to ensuring the package of materials is likely to be adopted as intended.

SMEs from industry will also be able to provide very useful information related to both questions based on their expertise in the area. In addition, these individuals have a unique perspective on what is required of individuals entering the industry, and thus can help shape the materials so that they adequately prepare students for jobs in the field.

Finally, information will also be collected from students enrolled in these courses as the materials are introduced. Specifically, student reactions and performance will be used to guide and refine the materials. These data will speak to only the second question (method), but will

Text Book

Lecture Notes

Adoption of Hardware & Software Tools

Lab Exercises

Wiki & Internet Site

SMEs in Academia and Industry

Students

Content Method

provide valuable insight related to how the materials function and are perceived from the student perspective.

4.2. Data Collection MethodsAlso consistent with recommendations for project evaluation (e.g., Frechtling, 1997, 2002),

this evaluation will collect information using both qualitative and quantitative approaches. The specific approaches used will be open-ended questions, focus groups, questionnaires, and class assignments.

For the first question (content) the SMEs from academia and industry will be asked open-ended questions focused on content validity. Input will be requested regarding three issues: deficiency, contamination, and efficiency. Deficiency refers to any content areas that are important to include in the course but are not covered in the developed material (i.e., missing topics). Contamination refers to any areas that the materials cover that are not essential to courses of this type (i.e., extraneous topics). For those areas judged to be relevant that are in fact covered, efficiency refers to the extent to which the distribution of emphasis and detail across topics is appropriate given topic importance. Greater efficiency is reflected in an appropriate distribution, where topics are not covered in too much or too little detail. The SME group will be asked open-ended questions related to each of these issues to guide and refine course materials.

For the second question (method) the SMEs will also be asked for their opinions regarding the effectiveness of the course materials. Feedback will be elicited for the full package of materials on issues including: organization, clarity, integration across components, and relevance to real-world applications.

Students will also provide feedback on the effectiveness of course materials, focusing on many of the same issues. Two methods will be used to elicit student input: web-based surveys and focus groups. Once each semester, all students in the class will be asked to respond to a web-based survey. Students will be contacted via an email that directs them to an anonymous web-based questionnaire. To ensure adequate response rates, survey implementation will follow major components of Dillman’s (2007) Tailored Design Method. This approach has been used successfully in our previous work, including in an evaluation of a teacher certification program developed by the National Board for Professional Teaching Standards, which resulted in an overall response rate of over 76% (N = 1140; Converse, Wolfe, Huang, & Oswald, 2006). The survey will focus on the effectiveness-related issues outlined above, using Likert-type items adapted from measures that have been used successfully in similar evaluation efforts (available from the Field-tested Learning Assessment Guide). In addition, once each semester, two focus groups will be organized to provide feedback on the newly developed materials. Focus groups procedures will follow suggested guidelines (e.g., Krueger & Casey, 2000), with questions emphasizing the same issues. These focus groups will help in interpreting the overall survey results and provide additional insights into strengths and weaknesses of the material and ideas for improvement.

Finally, student performance on lab work and a final project will also be examined to provide additional information on areas for improvement, based on those topics where performance is lower than expected. The final project is a particularly useful source of information, as it involves dynamic assessment including a formal presentation and complete documentation—consisting of a Word document detailing the solution and implementation and the Source Code

and Software Development Project File of Integrated Development Environment—as well as a live demonstration of the developed application (see sample projects attached as supplemental material). This is therefore a measure of application and transfer, which provides crucial information about the quality of learning experiences (e.g., see Barnett & Ceci, 2002; Bransford, Brown, & Cocking, 1999). Again, extensive assessment of student learning will not take place until after the first phase of this project, but examination of student performance on these assignments will supplement SME input and student reactions during material development and refinement.

4.3. TimelineThe process of evaluation will be iterative (National Research Council, 2003b), with student

reactions and performance collected each semester, and SME input elicited once each year. Timing of the student survey and focus groups will change across semesters, occurring later as more of the newly developed materials are introduced. However, for all semesters the survey will be implemented at least two weeks before the end of the semester to allow for enough time to achieve acceptable response rates and complete initial analyses. The focus groups will be conducted subsequently to help in the interpretation of survey results. Lab work occurs throughout the semester and the final project is presented at the end. Input from SMEs in both academia and industry will be obtained in December of each year of the project. This timing will allow for: (a) general guidance and direction after roughly one half to one third of the materials have been developed (2007), (b) more specific feedback after initial versions of all material have been developed (2008), (c) feedback after revisions are made (2009), and (d) a final evaluation after an additional year of refinement (2010).

5. Beyond Phase 1Following the first phase, a complete set of learning materials will be available for adoption.

At this point, a more extensive evaluation of outcomes will be possible, focusing on issues such as student learning and retention and adoption of the materials at other institutions. Results will be used to further refine and adjust the package as stakeholder needs change.

6. Results from Prior NSF SupportDrs. Kepuska and Anagnostopoulos are PIs of a grant, CCLI-EMD 0341601, entitled:

“PROJECT EMD-MLR: Educational Materials Development through the Integration of Machine Learning Research into the Senior Design Projects.” This is a prototype grant whose primary purpose is the development of educational materials in Machine Learning (ML) and the involvement of senior design students from FIT, as well as from another Central Florida university (University of Central Florida - UCF), and community college students from Seminole and Brevard Community Colleges (SCC and BCC) in Machine Learning Research. More details about this project can be found at the project’s web-site (http://emd-mlr.fit.edu/). Highlights of Project EMD-MLR are provided in the next three paragraphs.

Phase 1 Outcomes - Participants: In total, the project involved 11 faculties across 4 collaborating institutions, 4 graduate students at UCF, 1 graduate student at FIT, 20 junior/senior students from FIT, 30 junior/senior students from UCF, 5 sophomore students from BCC and 6 sophomore students from SCC. A total of 61 undergraduate students were impacted originating from the 4 participating institutions during the period of May 2004 to

present. These students, who were majoring in Electrical Engineering, Computer Engineering, Computer Science and Applied Mathematics, were embedded in student project teams at FIT and UCF, consisting of 2-3 students per team. The prototype project was also supported by a strong Advisory Board consisting of 8 and 9 industry and academic affiliates. Assessment and evaluation results compiled from the project participants showed well-defined, positive results with respect to the project’s impact. Due to their involvement in Project EMD-MLR, the vast majority of students felt that their participation had improved their perception of what graduate school and research is all about. Furthermore, two thirds of the undergraduate participants felt they were influenced positively in continuing their education by attending graduate school in the near future, while about half of the participating community college students stated their interest to continue their education by pursuing a bachelor’s degree in STEM disciplines. To our best knowledge, 20 or more of the junior/senior students who graduated and were involved in EMD-MLR have opted to pursue graduate studies.

Phase 1 Outcomes – Research & Products: In the course of the project teaching material has been developed for 8 teaching modules that were used to provide the project inductees with a broad overview of ML. We plan to refine these modules for widespread adoption. Additionally, the authoring of three chapters for the introductory textbook on ML for undergraduates has commenced. On the other hand, the objectives, scopes, demands and results of each team project topic were very diverse and numerous and cannot reported here due to lack of space; the interested reader is referred to project’s web site. Furthermore, in the course of the project the students developed 11 software implementations of well-known, as well as novel ML algorithms and techniques that are accompanied with related documentation. The implementations are suitable to be used as educational material for both undergraduate and introductory graduate curricula, as well as for research purposes in ML. Almost all implementations are provided as source code and MATLAB MEX files. Each implementation is supported by documentation describing the essence of each algorithm or technique. The implementations are Fuzzy Adaptive System (FAS) ART, FAS Ellipsoid ARTMAP, Fuzzy C-Means, Discriminant Adaptive Nearest Neighbor, Probabilistic Neural Network, Distributed-Learning Gaussian ARTMAP, microARTMAP Classifier, Growing Cell Structures, Growing Neural Gas, Fisher’s Exact Test for comparison of classification performances, and Analysis Tool for Speech Signal Processing and Recognition. We are currently working on standardizing their software interfaces and documentations to prepare them for on-line dissemination.

Phase 1 Outcomes – Outreach: Contributions to the STEM educational knowledge base were be made via the presentations of Anagnostopoulos, et al. 2005 and Anagnostopoulos, et al. 2006 at the Annual Conference and Exposition in 2005 and 2006 and Kepuska, et al. 2006 at the ASEE SE Section Conference, both events being sponsored by the American Society of Engineering Education (ASEE). Research findings accrued throughout the project were disseminated to a number of conference and journal venues. In total, Project EMD-MLR has produced 2 journal papers (2 additional ones are under review), 5 book chapters, and 12 conference presentations. The interested reader is referred to the corresponding entries in Part B of the References section for more information on these publications and presentations. Also worth mentioning is that software implementations of the Semi-Supervised Fuzzy ARTMAP and Ellipsoid ARTMAP classification models have already become part of a commercial data mining tool, namely KnowledgeExplorer, developed and distributed by PAI, Inc., which is owned by Dr. Mollaghasemi, one of our AF during the prototype phase of the EMD-MLR project.