operationalizing and assessing the entrepreneurial mindset: a rubric

Volume 2, Number 2, Special Issue 2011

*Associate Professor of Marketing, Pharmaceutical Business Program Coordinator

James F. Dicke College of Business Administration, Ohio Northern University

57

Operationalizing and Assessing the Entrepreneurial Mindset:

A Rubric Based Approach

Robert E. Kleine, III * and John-David Yoder

Abstract – A process for developing and assessing programming -- curricular,

co-curricular, or extracurricular -- intended to cultivate the entrepreneurial

mindset in engineering students is described. Steps of the process are to 1) define

learning-outcomes; 2) create a rubric for each learning-outcome; 3) develop

activities that generate student work; 4) collect student work; 5) conduct direct

assessment of student work by applying the rubrics to student work generated by

the activities; 6) interpret and report assessment results; and 7) modify learning-

outcomes, rubrics, activities or other program elements in response to

assessment results.

1. Introduction and Background

Stakeholders at educational institutions worldwide (e.g., students, parents, boards of trustees,

state legislators, accrediting agencies, granting agencies and foundations, employers) increasingly

expect evidence that an institution’s educational programs are effective (Arum and Roska, 2011).

The proliferation of disruptive new business models for delivering educational experiences,

enabled by advances in information technologies, increases the pressure on institutions to

document their ability to develop and deliver programs that achieve claimed outcomes

(Christensen et al., 2008). Current economic conditions, specifically declining tuition and

investment revenues experienced by many institutions, further increases the pressure to document

the degree to which programs are effective. These forces apply equally to engineering

entrepreneurial education programs.

Effective programs, courses, or class activity designs require clarity on the learning-outcomes to

achieve and on the tools that will be used to assess them (Fink, 2003; Smith, 2008). Learning-

outcomes are essential as they guide creation of programs, courses, class activities, and/or extra-

or co-curricular activities. Learning-outcomes also provide the benchmarks against which student

achievement is measured. Such assessment of student performance reveals program effectiveness

and guides program improvement efforts. Designers of engineering entrepreneurship programs

must specify carefully the learning-outcomes that motivate their programs. Business creation

(Bhide, 2000), if specified as a program outcome, will imply a program very different from a

program designed to cultivate the entrepreneurial mindset, which emphasizes cognitive or

attitudinal outcomes (Riffe et al., 2010). This article grew out of work with the Kern

Entrepreneurship Education Network (KEEN). The KEEN program, an initiative of the Kern

Family Foundation, is a network of schools working collaboratively to cultivate entrepreneurial

engineers characterized by an entrepreneurial mindset (Kriewall and Mekemson, 2010).

R. E. Kleine, III * and J.D. Yoder

The Journal of Engineering Entrepreneurship 58

Designing and implementing engineering education to cultivate students’ entrepreneurial mindset

requires clarity on three key issues:

1. What is the entrepreneurial mindset (EM)? The EM must be defined in a way that is

appropriate for development of specific learning-outcomes.

2. How can the entrepreneurial mindset be operationalized in a way that is appropriate for

guiding the design of educational activities intended to cultivate the entrepreneurial

mindset? The EM learning-outcomes must be expressed in a way useful for cultivating

EM in students.

3. How can those activities be assessed to determine whether or not they have had the

intended effects upon students’ entrepreneurial mindsets? We must be able to identify

what activities effectively cultivate the EM and those that do not.

Effectiveness insights are essential for enabling continuous program improvement. This article

leverages work completed by KEEN to define the entrepreneurial mindset to propose a learning-

outcome driven program development and assessment process model. The model is a framework

intended to guide anyone involved in cultivating the EM in students, whether an individual

faculty member, program coordinator, dean, or the director of a multi-school consortium (such as

KEEN).

The next section provides a brief overview of the proposed model of the learning-outcome driven

program development and assessment process. Subsequent sections unpack model components.

We conclude with suggestions for implementing the proposed model so as to maximize

cultivation of the entrepreneurial mindset among our students.

2. An EM Program Development and Assessment Process Model

A normative model of the learning-outcome driven program development and assessment process

for cultivating the entrepreneurial mindset is depicted in Figure 1. This model reflects inspiration

from several sources, most specifically Fink’s model (2003) for creating significant learning

experiences and the Association of the American Colleges and Universities (AAC&U) initiative

to develop normative models to guide assessment of undergraduate general education

(Association of American Colleges and Universities, 2011). The process is inherently adaptive

and recursive. Experiences applying each component of the model may precipitate modifications

intended to more closely align the system with desired objectives. The following sections

describe each of the model’s six key elements.

2.1. Learning Outcomes

Learning outcomes that reflect the entrepreneurial mindset are the foundation to this approach.

Learning outcomes are typically developed by each institution (general education learning

outcomes, for example) or units within an institution such as colleges or departments (e.g.,

learning outcomes for a major or for individual classes; see Otter (1995) for a nice overview of

learning outcomes). The process through which the EM learning outcomes were developed is

unique in that the KEEN schools collaborated to develop a set to apply across institutions.

Although a full chronology of the process employed by the KEEN schools is beyond the scope of

this paper, it is important to note that this inter-institutional approach to development of learning

outcomes is uncommon. The AAC&U LEAP initiative, which is developing a common set of

general education learning outcomes is an exemplary exception. The EM learning outcomes were

developed through a yearlong, multi-step process. The process included representatives from the

various KEEN institutions, all private institutions with ABET accredited engineering programs,

who met on several occasions with the goal of achieving consensus on a set of learning outcomes.



Volume 2, Number 1 Winter 2011 59

The resulting suite of seven EM learning outcomes defines the Entrepreneurial Mindset for the

KEEN schools (see Table 1).

Figure 1. A Model of the Entrepreneurial Mindset Learning-outcome Driven Program

Development and Assessment Process.

Table 1. Entrepreneurial Mindset Learning-outcomes Developed by the Kern Entrepreneurship

Education Network

A student should be able to:

1. Effectively collaborate in a team setting

2. Apply critical and creative thinking to ambiguous problems

3. Construct and effectively communicate a customer-appropriate value proposition

4. Persist through and learn from failure.

5. Effectively manage projects through appropriate commercialization or final delivery

process.

6. Demonstrate voluntary social responsibility.

7. Relate personal liberties and free enterprise to entrepreneurship.



This paper focuses on the role of learning outcomes in the context of developing and assessing

programs to cultivate the entrepreneurial mindset. Explication of how to construct these is beyond

the scope of this paper. For purposes of EM program development and assessment, learning

outcome statements provide direction to development and assessment of programming intended

to cultivate specific outcomes in students. When used in an inter-institutional context, as with the

KEEN schools, learning outcomes are also a vehicle for focusing the collective energies of

disparate institutions toward a cultivating a common set of student abilities.

It is important to realize that the suite of seven learning outcomes applies to the totality of a

program. An individual activity, or even a single course, cannot cultivate all of the abilities

implied by the EM learning- outcomes. A specific activity, such as an in-class egg case design

exercise, strives to cultivate a focused subset of the EM learning outcomes. The myriad EM

cultivating activities deployed by a program, in aggregate, yield the total impact of a program.

One challenge with learning- outcomes, even those that are well crafted , is that they are often

subject to multiple interpretations. Different interpretations of learning outcomes by individual

faculty members or by disparate institutions induce variance in what abilities students develop

with regard to a particular learning outcome. Rubrics are a tool commonly used to enhance

uniformity of understanding of learning outcomes.

2.2. Rubrics to Operationalize the Entrepreneurial Mindset

A rubric is an explicit set of criteria used for assessing a particular type of work or performance

(The TLT Group, 2011). Rubrics serve several vital functions. This paper emphasizes the utility

of rubrics for EM program development and assessment. Rubrics can serve other important

functions. For example, rubrics can function as a communication tool among faculty within an

institution or across institutions, as was mentioned above. When shared with students, rubrics

afford an efficient tool for conveying performance expectations. Applied in the assessment

process, rubrics serve as measuring devices for systematizing the assessment of student work.

Rubrics are especially helpful when, as is often the case, the learning outcome has a subjective

component. A rubric is a tool that helps objectify the assessment of subjective phenomena. A

rubric provides a set of criteria developed to operationalize a learning outcome.

The unit of application must be identified when developing learning outcomes and hence rubrics.

The entrepreneurial mindset rubrics (reproduced in the Appendix) were developed for the purpose

of assessing the work of an individual, as opposed to groups of students. The rubric is applied to a

unit of work that can be identified as having been produced by a specific student. Individual

student performance on a group activity can be assessed if the individual’s contribution is distinct

and separable from the work of the other group members. For example, effective verbal

communication for an individual could be determined from their part of a group presentation if

there is a portion of the presentation during which the individual is the only group member

speaking. In contrast, a group-produced written report would not be an appropriate artifact for

assessing written communication effectiveness for individual students.

Assessment rubrics should be shared with students at the time relevant activities are assigned.

The rubrics thus afford students a useful guide for how to approach the activity. It is also

productive to instruct students to self-assess their work with the relevant rubrics. The self-

assessment might afford the foundation for a reflection paper. The self-assessment might also be

directed toward closing the self-identified weaknesses in the work.




2.2.1. A Recipe for Rubric Creation

A general recipe for constructing an assessment rubric is as follows:

a) Dimensionalize the learning outcome according to the distinct student tasks or abilities

specified in the learning outcome. Rubrics are commonly summarized in matrix form. The

dimensions typically form the rows of a rubric. This step can be tricky when a learning outcome

includes a fundamental ability that is an implied or assumed but not explicitly stated. For

example, consider the learning outcomes ―Persist through and learn from failure.‖ This learning

outcome contains two explicit dimensions: persist through failure and learn from failure. Implicit

in this learning-outcome statement is the ability for a student to recognize when a failure has

occurred. An individual must recognize that a failure has occurred before they find evidence of

persistence and learning. Consequently, the rubric for this learning outcome has three dimensions:

―failure recognition,‖ ―persist through failure,‖ and ―learn from failure‖ (see the Appendix). For

this reason, the individual(s) creating a rubric must be proficient in the domain addressed by the

rubric.

b) Determine the number of performance levels. Performance levels typically form the columns

of a rubric matrix. Performance levels delineate qualitatively distinct levels of student

performance. The performance levels progress monotonically and typically range from absence of

the desired ability (at the low end) to proficiency (at the high end). The number of intermediate

performance levels included must strike a balance between making distinctions that capture

incremental variation in qualitatively distinct levels of student performance and discriminating

the results by all who will use the rubric. In general, ease of use declines as the number of

performance levels increases. Although we have not found research on this topic, four levels of

performance appear to strike the balance between capturing essential performance variance and

ease of use.

Four performance levels are used for the entrepreneurial mindset learning-outcome rubrics:

Does not meet expectations, Developing, Meets Expectations, and Proficient.

―Does not meet expectations‖ characterizes a student performance that does not display

any of the desired target activity.

―Developing‖ characterizes a student performance that evidences some of the desired

target activity.

―Meets expectations‖ designates student performance that evidences the minimal level of

ability expected.

―Proficient‖ designates student performance that exceeds ―meets expectations‖ and

evidences mastery of the target activity.

c) Identify student abilities that reflect performance characteristic of each dimension. It is useful

to ask the question: What would provide evidence that a student has mastered this dimension?

The resulting answer can form the foundation for describing the ―proficient‖ level of the

dimension. For example, proficiency at the ―failure recognition‖ dimension was determined to be

indicated by two activities: recognition that a failure has occurred and the ability to properly

identify (categorize) the failure. Bloom’s taxonomy (Bloom et al., 1956) is a useful aid for

articulating the desired behavior characteristic of proficiency.

The act of describing what constitutes ―proficient‖ behavior may reveal that a dimension is very

complex and should be disaggregated into sub-dimensions to fully capture the breadth of the

domain encompassed by the learning outcome. The Effective Communication rubric reflects the

complexity of the learning outcome. This rubric includes two levels of sub division. First, it



differentiates three types of communication—written, oral, and visual--on recognition that each is

a unique behavioral domain. Second, written and oral share the ―overall organization‖

dimension—the behaviors that reflect that behavior differ by communication modality. The rubric

descriptors reflect these differences.

Each cell in the rubric matrix presents a description of the student behavior characteristics of the

specific ability level along that dimension. The ability levels increase monotonically. Each ability

level subsumes the performance abilities of the levels below it. Pragmatically, it is often easiest to

write the description of the highest-level indicator and then back fill the lower levels. For

example, proficiency of failure recognition is characterized by evidence of three conceptually

related events: failure recognition, failure identification, and the ability to think critically about

the consequences of the failure. Conceptually, recognition that a failure has occurred must occur

before identification and apprehension of the consequences. Put differently, if a student is

unaware that a failure has occurred, none of the subsequent behaviors can be present.

Consequently, the descriptors progress from the following behaviors: not evident; evident at a

low level (recognition that failure occurred); evident at a qualitatively distinct higher level

(recognition that failure occurred and proper identification of the failure); evident at a level that

reflects mastery of the dimension (recognition that failure has occurred, correctly identifying the

failure, and explaining the consequences of the failure).

The specific student behaviors that reflect each level will depend on the context in which they are

applied. For example, evidence of failure recognition in the context of a freshman engineering

activity, in which team constructed bridges that were stress-tested, will differ from the evidence

of failure recognition if the student was analyzing a business model.

Rubric construction can be expedited or avoided altogether by first searching for existing rubrics

constructed by others (rubistar.com and tltgroup.org provide inventories of rubrics and tools for

rubric construction). When developing rubrics for the EM learning outcomes, we leveraged the

fact that several of the learning outcomes overlap with general education learning outcomes

developed by the AAC&U and Ohio Northern University (ONU). The AAC&U have published a

set of general education learning outcomes and rubrics with the intent that institutions use them as

meta-rubrics to inspire institution- or program-specific rubrics. The AAC&U rubrics were

developed through the assistance of faculty from institutions around the world. The rubrics

developed by Ohio Northern University (2009), some of which were inspired by the AAC&U

efforts, were created to support ONU’s new general education program. ONU rubrics were

constructed by teams comprised of ONU faculty with relevant expertise via a modified Delphi

technique facilitated by EduMetry (http://edumetry.com/), a consultancy that specializes in higher

education program assessment. The use of teams, comprised of faculty from across the ONU

campus, was employed to increase the likelihood that the resulting rubrics, as they result from

faculty engagement, would be more readily accepted and adopted across disciplines.

2.2.2. EM Rubrics: An Overview

A brief overview of the rubric developed for each EM learning outcome follows. It is important

to keep in mind that rubrics are typically part of a suite of tools that faculty, programs, or

foundations might use to assess the effectiveness of a program. Also, these rubrics are intended to

provide a starting point. It is expected that the rubrics will be modified as application data

becomes available. These overviews are intended to provide background on the origins and logic

underlying each of the EM learning-outcome rubrics. The rubrics themselves are presented in the

Appendix.




Effectively collaborating in a team setting – This rubric built on the foundation afforded by the

AAC&U teamwork rubric, specifically, the AAC&U rubric’s five dimensions: contributes to

team meetings, facilitates the contributions of team members, the individual’s contributions

outside of team meetings, and fosters constructive team climate. Working in collaboration with

two ONU management faculty, the target behaviors were adjusted in ways that better align with

salient target attributes based on their understanding of the teamwork literature and experience

with senior capstone groups. Emphasis was placed on ―soft skill‖ attributes that are sufficiently

objective such that their presence or absence can be reliably detected. The ―contributes to team

meetings‖ dimension captures the degree to which an individual effectively engages during team

meetings. ―Facilitates the contributions of team members‖ focuses on the degree to which team

members respect and enable contributions by other team members. ―Individual’s contribution

outside of team meetings‖ captures the degree to which an individual completes his/her team

assignments in accordance with the agreed upon time schedule and assures that the work done is

of a quality that advances the project effort toward its goals. ―Fosters constructive team climate‖

focuses on the degree to which an individual respects, motivates, encourages, and supports team

members. Conflict is common in group activities. How individuals handle conflicts can determine

whether a group becomes dysfunctional or succeeds and proceeds forward. The ―responds to

conflict‖ dimension captures the degree to which an individual engages and resolves conflict in

an appropriate way.

Applying critical and creative thinking to ambiguous problems – Critical and creative thinking

are considered by many to be the holy grail of learning outcomes; cultivating critical and creative

thinking is regarded by many as the primary purpose of an undergraduate college education

(Arum and Roska, 2011). The ability to think critically and creatively is fundamental to

entrepreneurial activity. Ambiguous problems refer to the ill-structured nature of the situations to

which the entrepreneurial mindset is applied. Ill-structured problems contrast with well-structured

problems. Well-structured problems have an unambiguously correct solution. For example, ―what

is the load bearing capacity of a bridge made from carbon fiber of a specified density and

dimensionality?‖ In contrast, ―what is the best way to study for my statistics exam?‖ is an ill-

structured problem. There are myriad possible solutions. The challenge is to apply critical and

creative thinking to identify the solutions that are most appropriate given the constraints and

resources available.

The rubric presented here was developed by a team of ONU faculty for application with ONU’s

general education program. The learning outcome is elaborated as the following: students are able

to correctly identify the underlying problems or issues in both theoretical and practical realms;

apply appropriate analytical and creative skills to develop feasible alternatives while considering

multiple perspectives; and provide creative and logical solutions. The ONU team process was

seeded by two AAC&U rubrics: critical thinking and creative thinking. The resulting rubric has

three sections: problem identification, research and analysis, and solution development.

Constructing and effectively communicating a customer-appropriate value proposition –

Innovation and entrepreneurial activity are complementary yet distinct activities. Entrepreneurial

activity concerns innovation harnessed in the service of providing value to an identified market.

This learning outcome has two distinct components: constructing a customer-appropriate value

proposition and the ability to effectively communicate that value proposition to key

constituencies (e.g., end users, purchasers, investors).

A search failed to yield a rubric appropriate for the customer-appropriate value proposition

learning outcome. SRI International’s NABC model—Need, Approach, Benefits relative to costs,



and Competition—for formulating a value proposition (Carlson and Wilmont, 2006) was thus

adopted to operationalize this learning outcome for three reasons. 1) The NABC model has a

proven record of success (Carlson and Wilmont, 2006). 2), The NABC model has proven useful

as a pedagogical aid in Kleine’s Product Development class. NABC provides a framework

students grasp readily and can apply productively when assessing and expressing a value

proposition. 3) The KEEN community, the ecosystem in which the rubric is to be deployed, is

familiar with the SRI model. This familiarity is conducive to accelerating adoption of the rubric

across the KEEN schools.

The rubric operationalizes the key dimensions of the NABC formulation:

Is an important customer and market need addressed?

Is the proposed approach meaningfully unique or distinctive relative to alternatives?

Does the proposed approach offer distinctive benefits relative to costs?

How do the benefits/costs of the proposed solution compare to existing alternatives?

Are the benefits/costs of the proposed solution quantitatively superior to existing

alternatives?

Are key constituencies identified and an appropriate variation of the value proposition

formulated for each? Key constituencies might be prospective investors, channel partners,

and end-users. The value proposition appropriate for an end-user would not be

appropriate for a channel partner or a prospective investor.

This rubric implicitly subsumes myriad sub-activities necessary to effectively support each level.

For example, the ability to identify key constituencies and identify and quantify market needs

presumes market research and analysis are conducted. The rubric is crafted such that a

comprehensive business plan is not mandatory. The rubric also does not presume that a working

prototype is crafted, but rather the emphasis is on making the case for the potential of an idea.

Effective communication is one of ONU’s general education learning outcomes. In the interest of

efficiency, ONU’s rubric for effective communication general was adopted. This also illustrates

that the rubrics proposed are modular. If an institution has effective communication as a general

education or other learning-outcome and has a rubric developed for assessing it, that rubric could

be adopted to leverage existing infrastructure on campus. ONU’s effective communication

learning outcome has three major components: effective written communication, effective verbal

communication, and effective nonverbal communication.

Persisting through and learning from failure – An individual’s ability to overcome adversity, to

stick with a problem until an effective solution is found, to ―fail-forward‖ effectively (Maxwell,

2000), is characteristic of ―growth mindset‖ individuals (Dweck, 2000). Others, such as Dwek’s

―fixed mindset‖ individuals are less likely to try again upon experiencing failure. Dwek’s

research shows that ―fixed mindset‖ individuals can be taught how to approach situations using a

growth mindset approach. This research suggests that the ability of an individual to persist

through and learn from failure is a skill and not a personality characteristic inherent in

individuals. Consequently, individuals can cultivate their ability to negotiate failure.

This learning outcome has three fundamental dimensions that build on one another: failure

recognition, persist through failure, and learn from failure. The ability to recognize that a failure

has occurred is essential as it sets the stage for corrective action. Consequently, the first row of

the rubric captures the ability to identify, categorize, and apprehend the consequences of the

identified failure. Implicit in this formulation is the possibility that a recognized failure is




misidentified. Improper identification or categorization of the failure will have implications for

the individual’s ability to effectively mitigate the failure.

Identification that a failure has occurred sets the stage for figuring out how to deal with

the failure. Persisting through failure is operationalized in terms of an individual’s ability to

identify and articulate a course of action that could mitigate, reduce or eliminate the

consequences of the identified failure.

Effectively managing projects through appropriate commercialization or final delivery process –

This learning outcome recognizes that a good customer-appropriate value proposition must be

translated into reality. That translation requires understanding of the process through which an

idea is transformed into a valuable entity. The rubric has four primary dimensions: 1) project

level, 2) task level, 3) evaluation and 4) deliverables.

The project level dimension captures activities relevant to overall project management: goal

setting, steps to achieve goals, optimal use of resources to achieve those goals, and the ability to

create and maintain a budget. This rubric was developed with the recognition that different

programs teach different process models. For example, some programs emphasize the stage-gate

model (Cooper, 2011) for managing the product development process. Consequently, students

would be expected to evidence understanding of the stage-gate model in how they approach

project management.

Embedded within the overall structure of a project are the myriad tasks that must be identified,

defined, sequenced and implemented in a way that advances the project toward its ultimate goals.

These task-level project activities include defining and sequencing appropriate tasks, calculating

the time a task will require, executing the task, and managing the task process, which may require

delegating and/or coordinating with team members.

Evaluation captures the ability to effectively assess progress toward the identified goal(s). This

includes the ability to articulate procedures to use to assess project effectiveness into the future.

Deliverables, the materials that must be created and supplied to designated constituencies by an

identified time form the final dimension. This dimension reflects that projects typically require

generation of materials that provide evidence of project activity. The specific materials to be

delivered will depend on the project and its goals. For example, one course may have creation of

a development plan as its key deliverable. Another course may entail a working prototype with

supporting documentation as expected deliverable materials.

Demonstrating voluntary social responsibility – This learning outcome parallels ONU’s general

education learning outcome for ―informed and ethical responses to personal, civic and global

needs. ― These learning-outcomes parallel the AAC&U ―Civic Engagement‖ learning outcome.

However, the KEEN learning outcome is not isomorphic with these two learning outcomes.

Accordingly, the rubric (see the Appendix) leverages those foundations, yet embraces the

uniqueness of the KEEN learning outcome. The target behaviors are that students display an

understanding of personal, civic and social needs; identify underlying ethical issues in specific

situations; and make informed ethical responses to those needs. This parsing strives to embrace

the spirit of the KEEN learning outcome while maintaining as much overlap as is possible with

the ONU and AAC&U general education rubrics so as to maximize utility to schools that may

apply it.

The rubric has three dimensions that operationalize the three parts of the learning outcome. The

first dimension concerns the degree to which the individual evidences an understanding of needs



that are grounded in data as opposed to personal opinion. The second dimension concerns an

individual’s ability to discern the ethical issues that arise from social needs. The third dimension

concerns the individual’s ability to respond to identify needs in an appropriate way by applying

appropriate analytical frameworks.

Relating personal liberties and free enterprise to entrepreneurship – This learning outcome

highlights individuals’ understanding of personal liberties and free enterprise and how each is

reflected in and enabled by entrepreneurship (Schramm, 2006). As such, this learning outcome

focuses on the intersection of values, government regulation, and individual initiative directed

toward creating and delivering value. The rubric’s three dimensions flow naturally from the

learning outcome. Entrepreneurship, the first dimension, focuses on the student’s understanding

of entrepreneurial behavior and attributes of successful entrepreneurial behavior. The second

dimension taps the student’s ability to provide evidence of a connection between personal

liberties and entrepreneurship. The third dimension focuses on individuals understanding of free

enterprise and entrepreneurial behavior.

2.3. Developing curricular, co-curricular, and extra-curricular activities, and programs that

cultivate the EM

An outcomes-based approach to program development implies that learning outcomes are

simultaneously the beginning and the result of the education cycle. Rubrics, by providing an

elaboration of each learning outcome, afford a tool useful for aiding the design of educational

activities. ―Activity‖ is used here as a generic term for curricular, extra-curricular and/or co-

curricular activities intended to further a learning outcome.

Student abilities are typically cultivated via modular units such as majors, minors, courses, extra-

curricular activities such as student organizations, and co-curricular activities such as idea pitch

competitions. The smaller the modular activity unit, the more specific the learning outcomes the

activity will develop. In the case of the KEEN program, for example, all seven of the

entrepreneurial mindset learning-outcomes apply. Program-level design involves asking the

question: how will we cultivate the abilities of our students with the goal of them achieving

proficiency? As with all unstructured problems, there is no single best answer. What is a best

approach for one school may not be appropriate for another school.

2.3.1. The Program Matrix

A Program Matrix is a visual tool useful for guiding program development. Form a Program

Matrix by expressing the learning outcomes as columns. Express the various EM cultivating

activities available on a campus as the rows. Place a check mark or other symbol in the table cells

to designate the specific EM learning outcome(s) addressed by a particular activity. The result is a

convenient visual summary of an institution’s EM cultivating initiatives. Review the summary

relative to the objectives established for your institution’s EM initiative. Identify learning

outcomes lacking coverage. These voids should be addressed. Similarly, review the summary to

identify over-emphasized learning outcomes. Over emphasized areas could be scaled back. The

resources freed up (e.g., class time) can be reallocated to develop activities that cultivate the

underserved learning outcomes.

As a first pass, the matrix may include only the seven learning outcomes. As the program design

becomes better specified, expand the learning outcomes into their constituent dimensions and

rubric rows. Review the resulting Program Matrix to identify voids. The goal is for each learning

outcome to map onto at least one activity.




2.3.2. Aligning Activities with Learning Outcomes, Rubrics and Rows

Activities should be closely aligned with the EM learning outcomes and rubrics. Activity/rubric

alignment requires addressing two questions: 1) What learning outcome(s) will this activity

cultivate? This step identifies the relevant rubrics. 2) What row(s) of the relevant rubrics apply?

Typically, an activity will cultivate one or two of the EM learning outcomes and specific rows of

the relevant rubrics. A single assignment created for a particular course might focus on parts of

one or more learning outcomes. For example, an activity that requires the student to develop and

deliver an elevator pitch might be used to develop a student’s ability to construct a customer

appropriate value proposition and her oral communication skills. When developing the activity,

care must be taken to align the activity task with the behaviors emphasized by the relevant EM

rubric rows.

Rarely will a single activity suffice for bringing students to the target proficiency level for a

specific learning outcome. One activity will generally impact incrementally the degree to which

participating students are proficient. In many cases, multiple activities that cultivate a single

ability will be required for students to achieve proficiency. For example, critical and creative

thinking might be developed throughout a student’s undergraduate program. A single activity

might simultaneously cultivate multiple learning outcomes. For example, a group activity focused

on developing criteria for screening ideas, may simultaneously cultivate critical thinking to solve

ambiguous problems, team collaboration skills, and an element of constructing a customer-

appropriate value proposition. It is useful to prioritize the intended learning. It is also possible

that an activity will cultivate specific parts of a learning outcome. For example, an activity that

requires students to prepare a slide deck to support defense of a selected alternative might be

identified as an activity that cultivates visual communication skills.

The rubrics for the learning outcomes (and sub-parts) to be emphasized in a particular activity

provide useful guidance on the types of work the students might do to cultivate the outcome.

They provide guidance on the specific skills that should be emphasized or that may be developed

prior to commencing an activity. The rubrics can also be used as a way to check activities to

verify they are appropriate for the targeted learning outcomes.

As a result of engaging the activities created to cultivate learning outcomes, students generate

artifacts or work. The artifacts provide data useful for assessing proficiency at the student level,

the activity level, and the program level.

2.4. Accumulating evidence of student ability

Data that provides direct evidence of student ability is fundamental to the proposed model.

Program accrediting agencies, such as ABET, expect direct assessment of student work. Evidence

of student EM abilities is termed an artifact. An artifact reflects the work of an individual student

that is relevant to one or more rows of the EM learning-outcome rubrics. An artifact may take

myriad forms: a reflection paper written by a student; a video of a student presentation; a

student’s response to an exam question; or a term paper or project report and more.

A system or systems for accumulating and storing artifacts is needed. Although the details of

artifact archive systems are beyond the scope of this paper, we outline some basic parameters.

Procedures for collecting EM artifacts also need to be established. Many artifacts will be

generated in response to a class assignment. These artifacts can be collected using normal class

procedures, which may range from students handing a tangible assignment to the instructor to

uploading the assignment to a class management system (CMS) such as BlackboardTM. The

system used to collect and archive student artifacts should be compatible with the assessment



system (see below). As with all educational materials, the procedures developed should be

compliant with the Federal Educational Records Privacy Act (FERPA). The artifact collection

system should be designed such that artifacts can be retrieved when needed by those responsible

for EM assessment.

2.5. Assessment

Assessment is about gaining insights into whether or not program learning outcomes are being

realized. Assessment can be applied to the totality of the EM program or to each activity that

cultivates EM abilities. Assessment, in the proposed model, involves applying the rubrics to the

student artifacts. This may involve applying specific rubric rows, rather than a rubric in its

entirety. The rubric rows applied will be the those designated when constructing the activity that

generated the student artifact.

Artifact scoring involves applying the appropriate rubric row(s) to an artifact. For each applicable

rubric row, the artifact is scored into one of the performance levels. For the entrepreneurial

mindset rubrics, that would entail categorizing an artifact as ―does not meet expectations,‖

―developing,‖ ―meets expectations,‖ or ―proficient.‖ In Blackboard and in other LMS systems,

rubrics can be enabled via grading forms to streamline the scoring process. A grading form

(rubric) can be attached to an assignment. LMS based approaches such as this one automatically

capture how an artifact is scored on each rubric row and aggregate the scoring data. EpsilenTM

(www.epsilen.com) is the only CMS of which we are aware that enables sharing of rubrics across

courses, programs, and institutions. Rubric sharing makes it possible for one source to publish

rubrics that can then be easily used by many scorers at different institutions, as well as those

teaching different courses. The outcome of the artifact scoring process is a matrix of students by

rubric rows.

Assessment of an activity or a program is based on aggregated data. Data aggregation involves

determining the frequency for each rubric row at which each proficiency column was evident in

the student artifacts. These summaries provide a useful snapshot for answering the question,

―how are we doing?‖ Interpretation is simplified if the scores are normalized as percentages and

displayed visually as bar charts. A preponderance of scores in the ―meets expectations‖ and

―proficient‖ categories for a rubric row would suggest students are performing well at that ability.

A preponderance of scores in the ―does not meet expectations‖ or ―developing‖ categories signals

ability on which more development is needed. Low ability is expected, for example, of freshman

students making their first attempts at creating a customer appropriate value offering. A pattern of

low ability evident in the work of senior standing students, on the other hand, could point to a gap

in program design or a weakness in relevant program activities.

Figure 2 illustrates three scenarios based on simulated data for n=20 artifacts. Each bar represents

the number of student artifacts scored at each of the four proficiency levels. Let us assume that

the goal is for at least 85% of artifacts to be scored as ―meets expectations‖ or ―proficient.‖ In the

left-most scenario, 15 of the 20 artifacts were scored ―does not meet expectations.‖ Only 15% of

the artifacts reach the goal levels. This distribution reflects a lack of proficiency. In the middle

scenario, the scores are concentrated in the ―developing‖ and ―meets expectations‖ categories.

The middle scenario provides evidence that some progress is being made to achieve the learning

outcome, however, only 45% of the artifacts meet the minimum expectations. Clearly there is

room for improvement. The bar set on the right reveals that the 60% of artifacts are scored at

―meets expectations‖, and 25% of the artifacts are scored as ―proficient.‖ Thus, the goal is

realized as 85% of the artifacts are scored as ―meets expectations‖ or ―proficient.‖




Figure 2. Three assessment scenarios

Who scores the artifacts? Artifact scoring should be objective, repeatable, and performed by

knowledgeable individuals familiar with the artifacts and the rubric(s) they are applying. Units

vary in terms of how artifact scoring is handled. Faculty in the College of Engineering at ONU,

for example, are responsible for scoring artifacts generated by assignments in classes that they

teach. In contrast, ONU’s new outcome-driven general education program places the

responsibility for scoring artifacts under the jurisdiction of the University Assessment Committee.

Companies such as EduMetry provide an artifact scoring services. Departments and colleges will

need to develop artifact-scoring procedures that fit with the resources available.

A database of artifact scores should be designed to allow assessment of student ability at key

program points (e.g., freshman vs. seniors). The ability to assess intra-student development over

time is also a desirable feature, as it would allow tracking student proficiency changes over time.

Artifact score data can also be compared across institutions.

An example of how assessment data might be aggregated is provided in Figure 3. Figure 4, which

presents simulated data, illustrates how assessment data might be reported for each learning

outcome. Each bar represents the percentage of artifacts that meet or exceed the identified

minimum expectation. Reprising the standard used earlier of at least 85% of artifacts scoring at

the ―meets expectations‖ level or better, the example (Figure 4) suggests that program targeted

proficiency is achieved on all but the first two learning outcomes. This suggests that changes may

be needed in regards to how student collaboration and critical thinking abilities are cultivated.



EM Learning-outcome Does Not Meet

Expectations Developing

Meets

Expectations Proficient

Effectively collaborate in a

team setting

Apply critical and creative

thinking to ambiguous

problems

Construct and effectively

communicate a customer-

appropriate value

proposition

Persist through and learn

from failure.

Effectively manage

projects through

appropriate

commercialization or final

delivery process.

Demonstrate voluntary

social responsibility.

Relate personal liberties

and free enterprise to

entrepreneurship.

Figure 3. Example EM Program Dash Board

Figure 4. Example Program Summary Graphic




2.6. What should be changed?

Assessment enables continuous improvement. When there is evidence that student performance is

not achieving or exceeding target levels, those areas of underperformance should be examined

and adjusted. Our hypothetical score card data revealed failure to achieve target performance

levels for the effective collaboration and critical thinking learning outcomes. These failures may

be due to the activities developed to cultivate the learning outcome, problems with the rubric, or

specific rows of the rubric, or with the learning outcome. Each is discussed.

2.6.1. Reviewing then modifying existing or create new activities

Lack of performance may reflect problems with the activities designed to cultivate the learning

outcome. Examine the portfolio of activities developed to cultivate the learning outcome. The

Program Matrix, described above, will be very useful for this task. Perhaps, examination of the

Program Matrix reveals an absence of activities designed to cultivate the learning outcome: no

activities equals no development of relevant student abilities. Activity development is the obvious

next step. If the performance matrix reveals activities intended to cultivate abilities relevant to the

learning outcome, then drill down in the assessment data to determine whether failure to achieve

the target performance level can be isolated to specific rows of the rubric. Determine whether

activities exist to cultivate the identified problem rows. If no activities exist, they should be

created or existing activities should be modified to include them. If all rubric rows are addressed

by existing activities, those activities should be reviewed and modified or replaced. Activities

should be reviewed to ensure they are aligned with the abilities emphasized by the rubric.

Activities not aligned with the rubric should be modified to bring them into alignment. It is

possible that each activity will be modified in some way in response to its application and

assessment.

2.6.2. Modify the rubric?

It is possible that failure to achieve target performance levels is due to errors, omissions, or

misspecifications in the rubric. It is possible that the level of performance formalized into the

rubric is too ambitious for an undergraduate program, or not possible given available resources,

for example. It is also possible that individuals applying the rubric are misinterpreting the

language describing the performance levels. This may be discovered through conversation with

individuals that score the artifacts. These exemplify some of the reasons that assessment data may

suggest modifying a rubric to remove these sources of error or bias. After making modifications

to a rubric, the activities relevant to that rubric should be reviewed and modified, if necessary.

The assessment process should enable and inform continuous refinement of the rubrics.

2.6.3. Modify the learning-outcome?

Modifying the learning outcome is a third area of possible change. Failure to achieve

performance levels on a specific learning outcome may reflect a learning outcome that is too

broadly defined or unattainable. The assessment data patterns may suggest ways to modify the

learning outcome to one that is well defined or attainable. Changes to learning outcomes may also

be changed to reflect shifting priorities or new philosophies. Of the three areas of possible

change, modifications to learning outcomes are the least frequent.

3. Conclusions and Future Work

This article describes a learning-outcome driven approach to developing and assessing programs

to cultivate the entrepreneurial mindset in undergraduate students. The entrepreneurial mindset is

characterized by seven learning outcomes. Each learning outcome is operationalized via a rubric.



The rubrics afford specific guidance to faculty, program coordinators and others in their efforts to

create educational activities intended to develop specific abilities reflective of the entrepreneurial

mindset. The rubrics also provide a basis for assessing student performance so as to ascertain the

effectiveness of an individual activity, a course, or a program. The rubrics as presented represent

an initial effort to operationalize the seven KEEN learning outcomes.

Future work is needed to refine and validate the rubrics. This requires that faculty employ the

rubrics and provide feedback on their experiences. We emphasize that the rubrics as presented are

intended as a starting point. It is anticipated that the rubrics will evolve and improve through use

across KEEN. Further, it is possible that individual institutions may see fit to modify one or more

of the rubrics to better reflect local approaches to cultivating aspects of the entrepreneurial

mindset. At the time of this writing, rubrics for three of the seven EM learning outcomes are

undergoing field trial by faculty at participating KEEN institutions. Feedback received through

this process will be used to revise the rubrics. It is expected that two rubrics will be subjected to

field trial each academic term on an ongoing rotation.

Faculty members around the world are creating activities that cultivate some aspect of the

entrepreneurial mindset. It would seem desirable to create an easily accessible EM Activity

Repository of these activities. The repository should designate each activity according to the

entrepreneurial mindset learning outcome(s) it is designed to cultivate and also allow designation

of the rubrics and the rows of the rubrics that should be used for assessing student work generated

by the activity. A tool such as this would allow faculty or program designers, perhaps after

reviewing their Program Matrix, to quickly identify activities that cultivate specific student

abilities.

Faculty members seek EM cultivating activities that are effective. It would be desirable for the

EM Activity Repository to also accumulate assessment results from faculty who have used the

activities. It would also seem desirable that the Repository would allow faculty to submit updated

versions of an activity that reflects modifications based on experience applying the activity.

An assessment dashboard would be desirable. The dashboard would display a metric that reveals

the degree to which students in a program are realizing each of the EM learning outcomes. To

gain understanding of the effectiveness of a multi-school initiative, dashboard data could be

aggregated across institutions. Ideally, the dashboard would provide a summary at the learning-

outcome level and also allow fine-grained analysis of program effectiveness for each learning-

outcome sub-area. A dashboard that allows drilling down to the rubric-row level would yield such

fine-grained insights.

References

Association of American Colleges and Universities. (2011). Valid assessment of learning in

undergraduate education. http://www.aacu.org/value/rubrics/ (last accessed February 2011).

Arum, R. and Roska J. (2011). Academically Adrift: Limited Learning on College Campuses.

Chicago: University of Chicago Press.

Bhide, A.V. (2000). The Origin and Evolution of New Business. New York: Oxford.

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). Taxonomy

of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive

Domain. New York: Longmans, Green, 1956.

Carlson, C.R. and Wilmot W. (2006). Innovation: The Five Disciplines for Creating What

Customers Want. New York: Crown.




Christensen, C.M., Horn, M.B., and Johnson, C.W. (2008). Disrupting Class: How Disruptive

Innovation Will Change the Way the World Learns. New York: McGraw-Hill.

Cooper, R.G. (2011). Winning at New Products: Creating Value Through Innovation. New York:

Basic Books.

Dweck, C.S. (2000). Self-Theories: Their Role in Motivation, Personality, and Development.

Florence, KY: Psychology Press.

Fink, L.D. (2003). Creating Significant Learning Experiences: An Integrated Approach to

Designing College Courses. San Francisco, CA: Jossey-Bass.

Kriewall, T.J., and Mekemson, K. (2010). Instilling the entrepreneurial mindset into engineering

undergraduates, The Journal of Entrepreneurial Engineering, 1(1): 5-19.

Ohio Northern University. (2009). Final Report of the General Education Committee.

http://www.onu.edu/files/academic_affairs_general_education_final_report_i_23april2009_a

nd_rubrics_-_updated_03252011.pdf (last accessed August 2011).

Otter, S. (1995). Learning-outcomes in higher education. In J. Burke, ed., Outcomes, Learning,

and the Curriculum: Implications for NVQ’s, GNVQ’s and Other Qualifications. Bristol, PA:

Falmer Press, 271-283.

Maxwell, (2000). Failing forward: Turning mistakes into stepping stones for success. Nashville,

TN: Thomas Nelson, Inc.

Riffe, W.J., Tavakoli, M., and Harris, M. (2010). ―Design, implementation and assessment of

entrepreneurial workshops for an entire university faculty.‖ NCIIA Open 2010,

http://nciia.org/sites/default/files/conf2010papers/riffe.pdf (Last accessed August, 2011).

Schramm, C.J. (2006). The Entrepreneurial Imperative: How America’s Economic Miracle will

Reshape the World (and Change Your Life). New York: Harper Collins.

Smith, R.M. (2008). Conquering the Content: A step-by-Step Guide to Online Course Design.

New York: Jossey-Bass.

The TLT Group: Teaching, Learning, and Technology. (2011). ―Rubrics: Definition, Tools,

Examples, References,‖ http://www.tltgroup.org/resources/flashlight/rubrics.htm (Last

accessed February 2011).



Appendix

Draft Rubrics Developed to Operationalize the Entrepreneurial Mindset

KEEN Learning Outcome: Effectively collaborate in a team setting

Student is able to contribute to team meetings in ways that advances the group’s work,

facilitates the contributions of other team members, contributes to the project effort

outside of team meetings, fosters a constructive team climate and responds effectively

to conflict that may arise within the team effort.

Does Not Meet Expectations

Developing Meets


Con

trib

ute

s to

Tea

m M

eeti

ng

s

Does not share

ideas or solutions

that advance the

work of the

group.

Offers new ideas or solutions to advance the work of the

group.

Offers alternative ideas or solutions

or courses of action that build on the

ideas of others.

Helps the team

move forward by

articulating the

merits of

alternative ideas

or solutions.

Fa

cili

tate

s th

e C

on

trib

uti

on

s of

Tea

m M

ember

s

Does not engage

team members

by taking turns

and listening to

others without

interrupting.

Engages team members by taking turns and listening to

others without interrupting.

Engages team members in ways that facilitate their

contributions to meetings by restating the views of other

team members and/or asking questions for clarification.

Engages team members in ways that

facilitate their contributions to

meetings by constructively building

upon or synthesizing the

contributions of others.

Notices when

someone is not

participating and

invites them to

engage.




KEEN Learning Outcome: Effectively collaborate in a team setting (Continued)

Ind

ivid

ual

Co

ntr

ibu

tio

ns

Ou

tsid

e

of

Tea

m M

eeti

ngs

Does not

complete all

assigned tasks by

deadline.

Completes all assigned tasks by deadline

Work accomplished advances the project.

Work accomplished is thorough and

comprehensive.

Proactively helps

other team

members

complete their

assigned tasks to

a similar level of

excellence.

Fost

ers

Con

stru

ctiv

e T

eam

Cli

mate

Does not support

a constructive

team climate;

does none of the

following:

Supports a

constructive

team climate by

doing any two of

the following:

Supports a

constructive

team climate by

doing any three

of the following:

Supports a

constructive

team climate by

doing all of the

following:

Treats team members respectfully by being polite and constructive in

communication.

Uses positive vocal or written tone, facial expressions, and/or body language

to convey a positive attitude about the team and its work.

Motivates teammates by expressing confidence about the importance of the

task and the team's ability to accomplish it.

Provides assistance and/or encouragement to team members.

Res

po

nd

s to

Co

nfl

ict

Passively accepts

alternate

viewpoints/ideas/

opinions; fosters

conflict.

Identifies and acknowledges conflict.

Redirects focus toward common ground, toward task at

hand (away from conflict).

Stays engaged with conflict until it is

resolved.

Directly and

constructively

helps to

manage/resolve

conflict in a way

that strengthens

overall team

cohesiveness and

future

effectiveness.

Derived from the AAC&U Teamwork Value rubric

(http://www.aacu.org/value/rubrics/Teamwork.cfm)



KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems

Students are able to correctly identify the underlying problems or issues in both

theoretical and practical realms, apply appropriate analytical and creative skills to

develop feasible alternatives while considering multiple perspectives, and provide

creative and logical situations.

Section 1 of 3: Problem Identification


Developing Meets


Iden

tifi

es k

ey

issu

e(s)

an

d

con

vert

s to

a p

roble

m s

tate

men

t

Does not show

comprehension

of the situation

to be able to

identify the key

issue(s)

Shows

comprehension

of the situation

and correctly

identifies some

key issue(s) but

does not

articulate a

reasonable

problem

statement

Shows good

comprehension

of the situation

and correctly

identifies most

key issue(s) and

articulates

reasonable

problem

statement(s)

Comprehends

situation to be

able to identify

key issue(s) and

articulates valid

problem

statement(s)

Co

nsi

der

s th

e si

tua

tio

n f

rom

mu

ltip

le p

ersp

ecti

ves

Does not

consider the

problem(s) from

different

perspectives to

view it

comprehensively

Brings out some

of the

perspectives

applicable to the

issue(s), but

leaves out some

aspects; hence,

problem

identification

suffers from this

bias/skew

Considers the

issue(s) using a

few applicable

alternate

perspectives, but

does not use this

to ―frame‖ the

key issue(s)

realistically/

originally

Considers the

issue(s) from

multiple

perspectives and

uses this to

―frame‖ the key

issue(s)

realistically/

originally

Doesn't show the

ability to

combine these

different

perspectives to

view the issue(s)

more

comprehensively

Shows evidence

of combining

some of the

alternate

perspectives to

present well

rounded issue(s)

Is able to

combine

different

perspectives and

relate them to

each other

effectively to

comprehend the

issue(s) more

comprehensively




KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)

Section 2 of 3: Research and Analysis


Developing Meets


Dem

on

stra

tes

ab

ilit

y to

id

enti

fy a

nd

eva

luate

in

form

ati

on

No evidence of

search, selection

or source

evaluation skills

Demonstrates

ability to search

and select

information

sources, but

evaluation of

sources is not

quite adequate

Demonstrates

adequate skills in

searching,

selecting and

evaluating

information

sources to meet

information

needs

Evidence of

search, selection

and source

evaluation skills

to meet

information

needs

Displays ability

to identify

uniquely salient

sources

Sta

tes

pla

usi

ble

rea

son

ing f

or

posi

tion

or

act

ion

s

des

crib

ed

Selects and

applies data to

situation or

rejects it without

any justification

Applies some

rudimentary

reasoning in

consideration

and application

of data to

situations, which

is often

incorrect/

inadequate

Spells out the

reasoning

applied in

consideration

and application

of data to

situations with

few lapses in

adequacy of data

Spells out the

reasoning

applied in

consideration

and application

of data to

situations

Analysis is

inappropriate,

inadequate and

superficial, at

best

Uses some

analytical tools/

methods

Analytical tools

and methods

employed are

appropriate and

about accurate,

but not the most

suitable ones,

against the need

Analytical tools

and methods

employed are

appropriate and

adequate for the

need

An

aly

zes

info

rmati

on

usi

ng

appro

pri

ate

co

nce

ptu

al

fram

ewo

rk/t

oo

ls

Does not explain

which specific

tools or methods

are relevant to

the issue(s) at

hand

Analysis is

minimal against

the need and

does not fully

clarify the issues

and facilitate

decision-making

Analysis

somewhat

clarifies the

issues and

facilitates

decision-making

Analysis clarifies

the issues and

facilitates

decision-making

Does not clearly

and explain

which specific

tools or methods

are relevant to

the issue(s) at

hand

Partly explains

why specific

tools or methods

are relevant to

the issue(s) at

hand

Explains why

specific tools or

methods are

relevant to the

issue(s) at hand



KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)

Section 3 of 3: Solution Development

Does Not Meet


Meets


Dis

pla

ys a

bil

ity

to e

xplo

re a

lter

nate

opti

on

s to

th

e id

enti

fied

pro

ble

m

Does not come

up with alternate

feasible and

original options

Makes an effort

to minimally

explore alternate

solutions but

plays safe, does

not risk

uncertainty

associated with

original solutions

Explores

alternate

solutions to the

problem but

these are not

very original,

tends to play safe

to an extent

Explores and

comes up with

original, relevant

alternate

solutions to the

problem

Ma

kes

con

vin

cin

g a

rgu

men

ts f

or

reco

mm

ended

opti

on

, ali

gn

ed t

o

the

giv

en c

on

text

Fails to identify

conclusions,

implications, and

consequences of

the issue

Does not clearly

identify

conclusions,

implications, and

consequences of

the issue

Analyzes

conclusions,

implications, and

some

consequences of

the issue

Analyzes

conclusions,

implications, and

the majority of

consequences of

the issue

Doesn’t propose

solution to

problem

Proposes

solution to

problem(s)

without much

investigation and

analysis

Proposes

solution to

problem(s) with

some

investigation and

analysis

Proposes

solution to

problem(s) based

on detailed

analysis

Justifies the

recommended

option and

recognizes the

implications




KEEN Learning Outcome: Construct a customer-appropriate value proposition

This rubric is based on the NABC model for expressing value propositions as presented

in Carlson and Wilmot’s book Innovation: The Five Disciplines for Creating What

Customers Want.

Does Not Meet


Meets


Import

an

t cu

stom

er a

nd m

ark

et

nee

d (

op

po

rtu

nit

y)

Customer and

market needs not

identified.

Prospective customer and market needs and

characteristics are identified and described.

Identified customer and market

needs and characteristics are

quantified.

All key

constituencies

are identified

(e.g., target

customer,

investors); the

above are

replicated for

each key

constituency.

Wh

at

is t

he

un

iqu

e ap

pro

ach

for

addre

ssin

g t

he

nee

d?

No approach for

addressing the

need is provided.

An approach for addressing the need is described.

The approach is appropriate relative to the identified

customer and market needs.

The approach

evidences

incremental

uniqueness

relative to

existing

alternatives.

The approach

has a

distinguishable

advance in

design,

technology,

process, or

business model.

The approach

has an important

advance in some

combination of

design,

technology,

process, or

business model.



KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)

Does Not Meet


Meets


Wh

at

are

th

e sp

ecif

ic B

enef

its/

Cost

s th

at

resu

lt f

rom

th

is

appro

ach

No customer

benefits per costs

are provided.

Key customer benefits of the approach are identified

using specific language.

Customer costs of the approach are

identified using specific language.

The identified

key customer

benefits are

quantified in an

appropriate way.

The identified

customer costs of

the approach are

quantified in an

appropriate way.

Benefits relative

to costs of the

approach are

clearly

expressed.

How

are

th

ese

ben

efit

s/co

sts

super

ior

to a

vail

able

alt

ern

ati

ve s

olu

tio

ns

for

solv

ing t

he

pro

ble

m?

No alternative

solutions are

mentioned.

Key direct alternative solutions are identified and the

benefits and costs of each are quantified and explained in

specific language.

Key indirect alternative solutions are

identified and the benefits and costs

of each are quantified and explained

in specific language.

The superiority

of the proposed

alternative’s

benefits and

costs relative to

key direct and

indirect

alternatives is

quantified and

explained in

specific

language.




KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)

Does Not Meet


Meets


How

are

th

e ben

efit

s/co

sts

of

the

pro

po

sed s

olu

tio

n

super

ior

to t

he

Com

pet

itio

n?

No competition

is mentioned.

Key direct competitors are identified and the benefits and

costs of each are quantified and explained in specific

language.

Key indirect competitors are

identified and the benefits and costs

of each are quantified and explained

in specific language.

The superiority

of the proposed

alternative’s

benefits and

costs relative to

key direct and

indirect

competitors is

quantified and

explained in

specific

language.

Mu

ltip

le v

alu

e

pro

posi

tion

s fo

r m

ult

iple

con

stit

uen

cies

No

constituencies

identified.

All key constituencies are identified.

An effective

value proposition

is developed for

each of some of

the identified key

constituencies.

An effective

value proposition

is developed for

each key

constituency.



ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition

Students demonstrate the written, oral and visual communication skills necessary to

communicate professionally and effectively as responsible members of their

organizations and their communities.

Section 1 of 3: Written Communication

Does Not Meet


Meets


Ove

rall

Org

an

iza

tio

n Thesis statement

is unclear or

absent

Provides a thesis

statement that is

somewhat

developed

Provides a thesis

statement that is

generally clear

Provides a

strong, clear

thesis statement

Provides a

structure and

organization that

is not cohesive

or coherent

Provides a

structure and

organization that

is somewhat

cohesive and

coherent

Provides a

structure and

organization that

is generally

cohesive and

coherent

Provides a

structure and

organization that

is strongly

cohesive and

coherent

Para

gra

ph

Dev

elopm

ent

Writes

paragraphs that

are

underdeveloped,

with topic

sentences that

are missing or

unsupported

Writes

paragraphs that

are developed

inconsistently,

with topic

sentences that

are present, but

not fully

supported

Writes

paragraphs that

are generally

well-developed,

with topic

sentences that

are present and

supported

Writes

paragraphs that

are well-

developed, with

strong, focused

topic sentences

that are fully

supported

Form

at

an

d S

tyle

Demonstrates

improper use of

appropriate

document

formatting

guidelines.

Demonstrates

inconsistent use

of appropriate

document

formatting

guidelines.

Demonstrates

appropriate use

of appropriate

document

formatting

guidelines.

Demonstrates

excellent use of

appropriate

document

formatting

guidelines.

Integrates

quotations and

exhibits

inappropriately

Integrates

quotations and

exhibits

inconsistently

Integrates

quotations and

exhibits

appropriately

Integrates

quotations and

exhibits

effectively into

the analysis

Wri

tten

La

ng

ua

ge

Mech

an

ics

Writes sentences

that are unclear

or indirect

Writes sentences

that are

occasionally

unclear, indirect

or grammatically

incorrect

Writes sentences

that are generally

clear, concise,

and direct.

Writes sentences

that are

consistently

clear, concise,

and direct and

grammatically

correct

Minor problems

with grammar





(Continued) Section 2 of 3: Oral Communication

Does Not Meet


Meets


Ove

rall

Org

an

izati

on

of

pre

sen

tati

on

The central idea

is not clearly

stated and

remains unclear

or unidentified.

The central idea

is implied but not

explicitly stated

or communicated.

The central idea

of the

presentation is

completely and

clearly

presented.

Proposes a clear

central idea.

The presentation

moves from one

point to the other

in a disconnected

manner, without

regard to the

objective.

The

organizational

pattern of the

presentation does

not adequately

meet the

audience’s needs

or develop the

thesis

appropriately.

The

organization of

the presentation

is generally

logical and

meets the

audience’s

needs.

The entire presentation

is a cohesive piece in

which the ideas

developed support the

thesis.

Key ideas are

easily overlooked

and important

aspects of the

thesis are not

addressed.

There is lack of

clarity regarding

the purpose and

only a few of the

key ideas have

been addressed.

Presentation

begins with a

clear sense of

purpose, but the

presence of

irrelevant

content in some

parts leads to

digression from

the central idea.

The presentation leads

naturally to a strong

conclusion that

summarizes the key

points and leaves the

audience with a clear

message.

Arg

um

ent

an

d R

het

ori

c

Presentation lacks

an identifiable

central message.

Presentation

contains an

identifiable

central message,

but it argues from

a false premise or

attempts to prove

the null

hypothesis.

Presentation

contains a

significant

central message,

with valid and

true outcome

adequately

supported by

evidence and

logic.

Presentation contains a

compelling central

message, with a valid

and true outcome fully

supported by evidence

and logic.

Presentation does

not rely on

credible evidence,

or form a valid

and true outcome.

Argument is not

fully supported by

credible evidence;

the outcome is

either invalid or

valid but untrue.

Able to answer

basic objections

to their

conclusions.

Able to answer basic

and (at least some)

more

advanced/sophisticated

objections to their

conclusions.

Unable to answer

basic objections to their

conclusions




(Continued) Section 2 of 3: Oral Communication

Does Not Meet


Meets


Ove

rall

Org

an

izati

on

of

pre

sen

tati

on

The central idea

is not clearly

stated and

remains unclear

or unidentified.

The central idea

is implied but not

explicitly stated

or communicated.

The central idea

of the

presentation is

completely and

clearly

presented.

Proposes a clear

central idea.

The presentation

moves from one

point to the other

in a disconnected

manner, without

regard to the

objective.

The

organizational

pattern of the

presentation does

not adequately

meet the

audience’s needs

or develop the

thesis

appropriately.

The

organization of

the presentation

is generally

logical and

meets the

audience’s

needs.

The entire presentation

is a cohesive piece in

which the ideas

developed support the

thesis.

Key ideas are

easily overlooked

and important

aspects of the

thesis are not

addressed.

There is lack of

clarity regarding

the purpose and

only a few of the

key ideas have

been addressed.

Presentation

begins with a

clear sense of

purpose, but the

presence of

irrelevant

content in some

parts leads to

digression from

the central idea.

The presentation leads

naturally to a strong

conclusion that

summarizes the key

points and leaves the

audience with a clear

message.

Arg

um

ent

an

d R

het

ori

c

Presentation lacks

an identifiable

central message.

Presentation

contains an

identifiable

central message,

but it argues from

a false premise or

attempts to prove

the null

hypothesis.

Presentation

contains a

significant

central message,

with valid and

true outcome

adequately

supported by

evidence and

logic.

Presentation contains a

compelling central

message, with a valid

and true outcome fully

supported by evidence

and logic.

Presentation does

not rely on

credible evidence,

or form a valid

and true outcome.

Argument is not

fully supported by

credible evidence;

the outcome is

either invalid or

valid but untrue.

Able to answer

basic objections

to their

conclusions.

Able to answer basic

and (at least some)

more

advanced/sophisticated

objections to their

conclusions.

Unable to answer

basic objections

to their

conclusions




KEEN Learning Outcome: Demonstrate voluntary social responsibility

Students display an understanding of personal, civic and social needs; identify

underlying ethical issues in specific situations; and make informed, ethical responses to

those needs.

Does Not Meet


Meets


Un

der

sta

nd

ing

of

per

son

al,

civi

c, a

nd s

oci

al

nee

ds Does not base

understanding of

the needs of

persons, civil

societies, or the

global

community on a

body of pertinent

information.

Bases

understanding of

the needs of

persons, civil

societies, or the

global

community on a

limited body of

pertinent

information.

Bases

understanding of

the needs of

persons, civil

societies, or the

global

community on a

substantive body

of pertinent

information.

Bases

understanding of

the needs of

persons, civil

societies, or the

global

community on a

substantive and

diverse body of

pertinent

information.

Iden

tify

Un

der

lyin

g

Eth

ical

Issu

es i

n

Spec

ific

Sit

uati

on

s

Does not identify

ethical issues in

personal, civic,

or global life.

Identifies ethical

issues in

personal, civic or

global life.

Identifies ethical

issues;

recognizes that

these ethical

issues shape

personal, civic,

or global life.

Discerns the

underlying

needs, values,

and perspectives

that create

ethical issues in

personal, civic,

or global life.

Ma

ke I

nfo

rmed

, E

thic

al

Res

pon

ses

to

Th

ose

Nee

ds

Responds to

ethical issues

without regard to

pertinent

information;

does not apply

an analytical

framework.

Responds to

ethical issues by

considering

pertinent

information;

does not defend

chosen position

within an

applicable

analytical

framework.

Responds to

ethical issues by

considering

pertinent

information;

defends chosen

position within

an applicable

analytical

framework.

Responds to

ethical issues by

considering

pertinent

information;

defends chosen

position within

an applicable

analytical

framework;

evaluates

assumptions and

implications of

divergent ethical

perspectives.



KEEN Learning Outcome: Relate personal liberties and free enterprise to entrepreneurship.

The student is familiar with defining attributes of entrepreneurial behavior, can explain

the relationship between personal liberties and entrepreneurship, and can connect the

concept of free enterprise to entrepreneurial behavior.

Does Not Meet


Meets


En

trep

ren

eurs

hip

No defining

attributes of

entrepreneurial

behavior are

identified.

Key defining attributes of entrepreneurial behavior are

identified.

Can identify attributes of successful

entrepreneurial behavior.

Provides relevant

examples to

successful

entrepreneurial

behavior.

Rel

ate

per

son

al

liber

ties

to

entr

epre

neu

rsh

ip

No evidence of

relating personal

liberties and

entrepreneurial

behavior.

Some evidence

of relating

personal liberties

and

entrepreneurial

behavior.

Can describe the relationships

between personal liberties and


Provides relevant

examples to

illustrate the

relationships

among personal

liberties and

entrepreneurial

behavior.

Rel

ate

fre

e en

terp

rise

to

entr

epre

neu

rsh

ip

No evidence of

relating free

enterprise and

entrepreneurial

behavior.

Some evidence

of relating free

enterprise and

entrepreneurial

behavior.

Can describe the relationships

between free enterprise and


Provides relevant

examples to

illustrate the

relationship

between free

enterprise and

entrepreneurial

behavior.

operationalizing and assessing the entrepreneurial mindset: a rubric

Documents