measuring the impact of ifle grant programs evidence-based grant design & evaluation u.s....

Measuring the Impact of IFLE Grant Programs Evidence-Based Grant Design & Evaluation

U.S. Department of Education, International & Foreign Language Education Program

KimOanh Nguyen-Lam, Ph.D.Advanced Training & Research

Division Director

Andy GluckTechnical DirectorICF International

IMPROVING IFLE EVALUATION – A WORKING SESSION

3

WRITING MEASURABLE OBJECTIVES

An objective is a ‘foreseen end that gives direction to an activity.”

John Dewey, 1977• A goal is where you want to be and objectives are the steps taken to reach the goal.

• Goals are the large statements of what you hope to accomplish but usually aren’t very measurable.

• Measurable objectives communicate in specific and concrete terms the observable results to be attained. Measurable means that someone can assess the extent to which the project achieved the objectives.

4

WRITING MEASURABLE OBJECTIVES• When writing objectives, use ACTION VERBS such as

perform, identify, list and match. Avoid vague and immeasurable VERBS (appreciate, realize and be aware of)

• Each objective should answer the following questions WHAT, WHO, WHEN and HOW MUCH

What (measurable) change or benefit is expected?What should occur?Who is expected to change or benefit?Who will benefit from the project?How much change or benefit is expected?How much change must occur to prove the project’s success?When is the change or benefit expected to happen?When will the change happen?

5

“If you’re not measuring it, you’re not managing it.”

S.M.A.R.T. Objectives

6

S.M.A.R.T. Objectives SPECIFIC

Concrete Use action verbs What specific change is

going to occur?

MEASURABLE

Numeric or descriptive Quantity, quality cost How will you know that a

change has taken place?

ASSIGNED Relates to a specific target

audience. Who will change?

REALISTIC The objective is achievable. It can

be accomplished. Measures outputs or results.

What is the baseline (The starting point-Increase from what to what)? How much improvement is anticipated?

TIMELY Relates to the specific time to

achieve the objective. Identifies target date When will the change take place?

Developing Clear Goals and Objectives

Consider the following when developing measurable objectives and planning data collection:

1. How do we know whether the project is meeting its goals?2. What types and sources of data will best demonstrate the project's

progress toward achieving its objectives?3. How will the data be collected? Describe access and frequency.4. How will the data be analyzed and reported? Describe the

methodology and key personnel responsible.5. Would the results be able to demonstrate the project’s proposed

outcome and impact (e.g., an increase in qualified language instructors, higher graduation rate in international studies, better rate of graduate placement, etc.)?

Examples of possible project-specific quantitative objectives include:

Increase the number of students completing advanced courses in XYZ priority languages;

Increase the number of students from business, health, or science majors graduating with foreign language skills;

Increase the number of study abroad opportunities for students on campus;

Increase the number of certificates and degrees conferred in relevant programs of study.

Slide 8

Examples of possible project-specific qualitative objectives include:

Improved employment opportunities for students who possess advanced language skills and international experience;

Stronger collaboration between foreign language departments, international education, and other disciplines;

Improved quality of assessment tools for priority and/or less-commonly-taught languages.

Slide 9

Progress indicators that relate to the quantitative and qualitative examples cited in the above sections include, but are not limited to, the following:

Number of new faculty positions in priority and/or less-commonly-taught languages, area studies courses, or interdisciplinary courses that are institutionalized after grant support has ended;

Number and type of courses developed, piloted, and subsequently submitted to the institution’s review board for inclusion in the college catalog for the upcoming academic year.

Slide 10

Developing Evaluation QuestionsFormulate evaluation questions that are of interest to all stakeholders and audiences related to the proposed project, and align questions with appropriate information gathering techniques.

– Who/what will change?– When do you expect the change(s) to take place?– How much change is expected?– How will change be measured, recorded, or

documented?

Slide 11

Planning Data Collection and Analysis Establish baseline data Develop data collection instrument (surveys, standardized

tests, exams, focus groups, topic guides, etc. )

Specify the types and sources of data that will be collected

Describe how the data may be collected (including

access and frequency) Describe how the data will be compiled, analyzed,

and reported Describe the methodology and key personnel

responsible for these tasks Slide 12

Instructions for Completing GPRA PMF

All applicants filled in a GPRA PMF according to the following instruction steps 1-6. Step #7 was added to allow grantees to report actual results. Grantees need to edit the original GPRA PMF they submitted at the time of application to include the A1-A4 columns. 1. Performance Measures – The GPRA measures are already filled in.2. Activities – Fill in the major activities your institution will undertake to support achievement

of the GPRA performance measure goals.3. Data / Indicators – State the data or indicator that will be used to track progress of each

activity stated in #2. Note that one of the indicators should be synonymous with the performance measure, while other indicators play a supporting role to track progress toward meeting the performance measure. Since the performance measure might not change from the baseline in the early year(s) of the grant, supporting indicators are used to track and demonstrate progress.

4. Frequency – State periodicity of measurement (e.g., Quarterly, by Semester, or Annual).5. Data Source – State the source of the data (e.g., Grantee records or University registrar).6. Baseline (BL) and Targets (T1-T4) – State the “Baseline” number of participants (if any) that

meet the GPRA measure requirement prior to the current grant that is being applied for. Please state how many participants you intend to have been of service to at the end of each grant year as your answers for T1 through T4.

7. Actuals (A1-A4) – State these figures as the actual results that can be compared against the targets in the preceding columns.

Slide 13

PMF for GPRA Measures for BIE Grantees

1. Performance Measures

2. Activitie

s

3. Data/ Indicato

rs

4. Freq.

5. Data Source

6. Baseline and Targets, and 7. ActualsBL T1 A1 T2 A2 T3 A3 T4 A

4

A) An increase in the number of courses for degrees with an international business focus

B) An increase in the number of programs offering international business work experience or internships, domestically or abroad

C) An increase in the number of training programs for business personnel and program participants

Sample PMF for GPRA Measures for BIE Applicants1. Performance

Measures2. Activities 3. Data/ Indicators 4.Freq 5. Data

Source6. Baseline and Targets

BL T1 T2 T3 T4

A) An increase in the number of courses for degrees with an international business focus

A.1 Establish an inter-departmental working group to explore the possibility of offering new courses in international business.

A.2 Propose and refine recommended new courses in international business to appropriate approval committees.

A.3 Offer and perform outreach to recruit students for new course(s) in international business.

The number of new faculty volunteering to serve on the interdepartmental working group

The number of new courses in international business that are under development

The number of new courses offered in international business

The number of students enrolling in the

new international business courses

Annual

Annual

Annual

Annual

BIE grantee records

BIE grantee

records

University registrar


0 0 0

0

5 2 0

N/A

5 2 2

30

5 2 4

60

5 2 6

90

B) An increase in the number of international business work experience or internships offered, both domestically and abroad.

B.1 Develop relationships with US businesses / associations / organizations carrying on international economic activity with the BRIC nations.

B.2 Invite input into the development of curricula that would meet the needs of new partners involved in trade with BRIC nations.

B.3 Develop joint internships with businesses involved in trade with BRIC nations.

The number of organizations involved in trade with BRIC nations that attend functions sponsored by the international business department.

The number of courses with infusions of new material or topics suggested by partner organizations that are involved in trade with BRIC nations.

The number of internship positions with businesses involved in trade with BRIC nations.

Annual

Annual

Annual

BIE grantee records

BIE grantee records

BIE grantee records

0

0 0

6 1 4

12 2 8

15 3

12

18 4

15

C) An increase in the number of training programs for business personnel and program participants.

C.1 Develop relationships with US businesses / associations / organizations carrying on international economic activity with the BRIC nations.

C.2 Invite input into the development of curricula that would meet the needs of new partners involved in trade with BRIC nations.

C.3 Jointly develop and offer training programs for business personnel involved in trade with BRIC nations

The number of organizations that have signed MOUs to be part of a coalition of companies seeking to increase trade with BRIC nations.

The number of courses with infusions of new material or topics suggested by partner organizations that are involved in trade with BRIC nations.

The number of trainees who complete international business training courses intended to increase trade with BRIC nations.

Annual

Annual

Annual

BIE grantee records

BIE grantee records

BIE grantee records

0 0 0

4 1

100

10 2

250

12 3

300

15 4

400

16

1. Project Goal Statement2.

Performance Measures

3. Activities

4. Data/ Indicators

5. Freq.

6. Data

Source

7. Baseline and TargetsBL T1 T2 T3 T

4

PMF for Project-Specific Measures

1. Project Goal Statement – Each project will have more than one larger goal such as “Development of an international internship program for foreign language students and business students at minority-serving community colleges,” or “Provide international learning experiences to STEM and Business faculty and students, with a special emphasis on part-time adult, nontraditional students.” Each project goal will require a separate PMF template row to be completed. State the first project-specific goal in the first (header) row of the template. Proceed to steps 2-7 below. Repeat the process for each project-specific goal.

2. Performance Measures – State the project-specific measure in an objective and time-bound manner. Make sure the units of measure (e.g., # of courses, # of students, etc.) are well defined.

3. Activities – Fill in the major activities your institution will undertake to support achievement of the project-specific performance measure goal.

4. Data / Indicators – State the data or indicator that will be used to track progress of each activity stated in #2. Note that one of the indicators should be synonymous with the performance measure, and the other indicators play a supporting role to track progress toward meeting the performance measure. Since the performance measure might not change from the baseline in the early year(s) of the grant, supporting indicators are used to track and demonstrate progress.

5. Frequency – State periodicity of measurement (e.g., Quarterly, by Semester, or Annual).6. Data Source – State the source of the data (e.g., Center records or University registrar).7. Baseline (BL) and Targets (T1-T4) – State the level of each measure at the time of application submission

as the “Baseline” and provide the target levels at the end of each grant year. Where the measure specifies “new,” the baseline is stated as the existing total at the time of the application and the targets are stated as the incremental increase.

Instructions for Completing Project-Specific PMF

18

Sample PMF for Project-Specific Measures for BIE Applicants

1. Project Goal Statement: To internationalize the Business Education Program at JC College through curriculum and faculty development so that at least 200 students who graduate with a business degree in four years will have taken at least two BRIC language or BRIC internationalized business education courses

2. Performance Measures

3. Activities 4. Data/ Indicators 5. Freq 6. Data Source

7. BL T1 T2 T3 T4

A) Internationalize 25% of the business education curriculum with new courses or course modules with a focus on the BRIC (Brazil, Russia, India, and China) over the next four years when compared against the baseline.

A.1 Create a faculty advisory committee that includes faculty from international studies and business studies.

A. 2 Review existing business education curricula and identify appropriate curricular components that could be internationalized.

A.2 Develop new courses or course modules to internationalize curricula with a focus on the BRIC countries.

The number of new courses developed with a focus on BRIC countries

The number of new course modules developed to internationalize curricula

Annual

Annual

University Registrar


0 0

2 6

4

12

6

18

8

24

B) Increase inter-national business offerings by 15% and enhance 30% of existing business courses with cultural content within a four-year period when compared against the baseline.

B.1 Recruit and hire additional qualified instructors.

B.2 Develop conversational BRIC language courses based on cross-cultural business interactions.

B.3 Enhance existing business courses with cultural or area studies content.

The number of new instructors hired

The number of foreign language courses developed for the business degree program

The number of business courses enhanced with cultural or area studies content

Annual

Annual

Annual

BIE grantee Records


BIE grantee

Records

20

60

60

1 3 5

1 3 5

1 3 5

1 0 3

C) Increase the number of students who have completed internationalized business education courses each year so that the university is able to reach its four-year Project Goal.

C.1 Inform incoming business degree students of new graduation expectation on language and culture.

C.2 Disseminate information on business education courses that have international components.

C.3 Raise student awareness regarding the value of international business/ international studies/foreign language and global education through campus-wide international activities

Number of students who completed at least one IB/IS or BRIC language course

Number of students who completed at least one BRIC internationalized business education course

Number of students who completed at least two BRIC language or BRIC inter-nationalized business education courses by the time they graduate

Annual

Annual

Annual




30 0

20

45

50

35

90

150

70

150

250

150

200

350

200

measuring the impact of ifle grant programs evidence-based grant design & evaluation u.s....

Documents