white paper: impact of business intelligence on institutional research

Abstract Business intelligence software is ubiquitous in private industry to measure, monitor,

and communicate progress toward strategic goals; this process is being adopted in

higher education. This report explores the degree to which institutional research

offices are involved in institutions’ definitions and use of key performance indicators

and the selection and use of business intelligence software.

TTaabbllee ooff CCoonntteennttss Key Findings ................................................................................................................................................. 1

Section 1: Key Performance Indicators ........................................................................................................ 3

Institutional KPIs ...................................................................................................................................... 3

KPI Process ............................................................................................................................................... 4

KPI Categories .......................................................................................................................................... 4

Reporting KPIs .......................................................................................................................................... 5

Section 2: Business Intelligence Software ................................................................................................... 6

Choosing Reporting and Analytics Software Tools .................................................................................. 6

Evaluating Software Tools for IR Work .................................................................................................... 7

Section 3: Changing IR Role ......................................................................................................................... 9

Section 4: IR Budget ................................................................................................................................... 11

Appendix .................................................................................................................................................... 13

Survey Methodology and Respondent Characteristics .......................................................................... 14

Testing of Differences in Means ............................................................................................................ 15

Survey Instrument ................................................................................................................................. 16

Copyright AIR, 2015 and used with permission by Ellucian PAGE 1

IInnttrroodduuccttiioonn As higher education institutions focus on performance improvement, business intelligence garners increased attention, and the professionals who manage it are more visible. We now know that vast stores of data amount to a cacophony without the tools and expertise needed to strip away the noise and find real meaning that informs decisions.

So how are institutions finding that meaning? What investments are they making in business intelligence and performance management? Who are the people most engaged in the processes of turning data into information?

The Association for Institutional Research (AIR), with assistance from Ellucian, developed a survey to answer these questions. Ellucian’s expertise in working with postsecondary education institutions provided useful insights that were incorporated into the survey of AIR members. Thanks go to the 542 respondents who gave us a grassroots view of these important issues. We were delighted to hear from a wide range of institutions including public four-‐year, private four-‐year, and public two-‐year colleges and universities. We now have a better sense of the tools, methodologies, and budgets committed to measuring and reporting performance.

Thanks also go to Drs. Darlena Jones, Amelia Parnell, and Leah Ewing Ross of AIR who served as the lead researchers for this project. They developed the survey, compiled the data, conducted the analyses, and wrote the report. Mr. Christopher Coogan served as AIR’s liaison with Ellucian in order to achieve this robust examination of a topic of great importance to higher education.

We are pleased to share our findings and look forward to continued discussions on this topic. Key findings are reported below and are followed by the full research report.

I hope these findings and the detailed information contained in this report encourage productive campus discussions that focus on improving the use of data to prove and improve higher education.

Randy L. Swing, Ph.D. Executive Director Association for Institutional Research July 30, 2014 Tallahassee, Florida

KKeeyy FFiinnddiinnggss Demand for information is exploding in higher education, and institutional researchers are the center of their institutions’ data and information hubs. Because of this need for data, business intelligence (the skills, applications, and technologies used to leverage an institution’s data to support data-‐informed decision-‐making) is growing in popularity among senior leaders. The Association for Institutional Research (AIR) recently conducted a survey to determine institutions’ use of business intelligence (BI) and members’ roles in BI. Using KPIs: Most higher education institutions are adopting key performance indicators (KPIs), and institutional researchers are leaders in this arena. Survey evidence:


• 90% of respondents said KPIs are important to their institutions.

• 79% of respondents said their IR offices will be very involved in data collection to support KPI reporting within the next three years.

• 80% of respondents said their institutions track enrollment KPIs.

• 71% of respondents said their IR offices are the primary authorities on campus to collect KPI data.

• 41% of respondents said KPIs originated in the IR office. Selecting Analytical Software: In selecting institutional technology solutions, IR offices are part of decision making teams. Survey evidence:

• 42% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions’ enterprise resource planning (ERP) systems.

• 68% of respondents said IR offices either participate in decisions or are the decision makers in choosing their institutions’ BI solutions.

Changing IR Role: The IR role will change for some IR professionals in the near future. Survey evidence:

• 53% of respondents said they anticipate a role change in IR within the next three years.

• 38% of respondents predicted their roles will evolve toward being more strategic within the next three years.

• 27% of respondents noted that changes in analytical technologies will provide opportunities to conduct more advanced analytics.

Budget: IR professionals are somewhat optimistic for stable or expanded office budgets in the near future. Survey evidence:

• 32% of respondents expect expansion of their IR office budgets over the next three years, 46% anticipate no changes in their office budgets, and 22% expect their budgets to be reduced.


7% 8% 9%

16% 29% 32%

37% 41%

47% 51%

Other Don’t know

Other funding organizaaons Department commibees

Federal Internal commibees

State Insatuaonal Research Academic leadership

President/Cabinet

Chart 2: Originaaon of KPIs

57%

64%

90%

Importance of KPIs

It is very important for KPIs to have consistent definiaons for

benchmarking

KPIs are a major responsibility for IR office

KPIs are somewhat or very important to insatuaon

Chart 1: Importance of KPIs

SSeecctt iioonn 11:: KKeeyy PPeerrffoorrmmaannccee IInnddiiccaattoorrss Key Performance Indicators (KPIs) are metrics that provide leaders information to evaluate an institution’s success or its progress toward a strategic goal. Examples of KPIs for higher education are enrollment, retention, and graduation rates.

IInnssttiittuuttiioonnaall KKPPIIss Nearly every respondent (90%) reported that KPIs are somewhat to very important to their institutions (Chart 1). When asked if KPIs are a major responsibility for IR offices, 64% said yes. Over half of respondents (57%) said it is very important that KPIs have consistent definitions across institutions to support benchmarking efforts. We asked respondents to identify the areas from which their KPIs originate. More than half of respondents reported that KPIs originate from the president/cabinet, 47% reported academic leadership, and 41% reported IR offices. External groups, such as state or federal agencies, play smaller roles (Chart 2).

NOTE: This question was multi-‐response; responses total more than 100%.


KKPPII PPrroocceessss The KPI process includes identification of performance metrics, collection of data to measure current performance, reporting of data, strategic planning based on the outcomes of those metrics, and other institutional improvement initiatives. AIR asked survey respondents the degree to which their IR offices are involved in the KPI process. The percentages of respondents who noted that they are “very involved” in different stages of the KPI process are presented in Table 1. We also asked respondents what they anticipate their IR offices’ participation with KPIs will be in three years; they predict higher levels of future involvement for every stage of the KPI process. Creation of KPIs and work on other institutional performance improvement initiatives are the areas in which survey respondents expect the greatest increases in levels of involvement.

Table 1: How involved was the IR office with the following processes at your institution? How involved do you expect to be three years from now? Cu

rren

t: Ve

ry in

volv

ed

3 ye

ars f

rom

now

: Ve

ry in

volv

ed

Chan

ge

Creation of Key Performance Indicators (KPIs) 53% 69% 16% Data collection to support KPI reporting 73% 79% 6%

Reporting on KPIs 69% 78% 9% Strategic planning institution-‐wide 53% 65% 12%

Other institutional performance improvement initiatives 58% 75% 17%

KKPPII CCaatteeggoorriieess Performance metrics can be used in many areas of an institution. Respondents were asked to indicate in which areas their institutions track KPIs (Table 2). We found that most institutions track enrollment, while few track workforce management. We also asked respondents if their IR offices would have increased involvement over the next three years in these KPI categories.

Table 2: Which of the following KPI categories does your institution currently track? How will the IR office’s involvement change over the next three years? In

stitu

tion

Trac

ks K

PI

Incr

ease

d in

volv

emen

t

Enrollment 80% 57% Academics 73% 66%

Recruitment/Admissions 65% 58% Finance and Administration 53% 39%

Student Services/Student Life 35% 48% Alumni Relations/Advancement 35% 34%

Workforce Management 22% 35%


2%

3%

3%

5%

6%

7%

8%

9%

58%

Informaaon Technology

VP Student Services

Operaaons/COO

Other

Finance/CFO

Department Leaders

President

Academic/Provost Office

Insatuaonal Research

Chart 4: Primary Reporters of KPIs

8%

16%

29%

29%

31%

34%

39%

40%

67%

Other

Funding agencies

The public

Website

Staff focused on measurement

Faculty

State/federal agencies

Accrediang bodies

Leadership

Chart 5: Reporang KPIs

1%

1%

2%

4%

6%

6%

10%

71%

Finance/CFO

Operaaons/COO

Informaaon Technology

Academic/Provost Office

Department leaders

Other

Don’t know

Insatuaonal research

Chart 3: Primary Authority to Collect KPI Data

RReeppoorrttiinngg KKPPIIss Overwhelmingly, respondents said that their IR offices are the primary authorities to collect data on their institutions’ KPIs (Chart 3).

When asked who reports information on KPIs at their institutions, 58% of respondents cited their IR offices (Chart 4.) Other KPI reporters include provosts, presidents, and department leaders.

We asked respondents to identify all of the areas to which KPIs are reported (Chart 5.) Two-‐thirds of respondents said their institutions’ KPIs are reported to leadership and 40% said their accrediting bodies received KPI data. Nearly one-‐third (29%) of respondents said KPI data are reported on their websites and to the public. NOTE: This question was multi-‐response; responses total more than 100%.


ERP system BI soluaon Decision maker 1% 18%

Paracipated 41% 50%

No involvement 58% 32%

58%

32%

41%

50%

1% 18%

Chart 6: Choosing ERP and BI Solware

Current insatuaon tools Future IR tools

Decision maker 26% 21%

Paracipated 50% 75%

No involvement 24% 4%

24% 4%

50% 75%

26% 21%

Chart 7: Choosing Reporang and Analyacs Solware Tools

SSeecctt iioonn 22:: BBuussiinneessss IInntteell ll iiggeennccee SSooffttwwaarree

CChhoooossiinngg RReeppoorrttiinngg aanndd AAnnaallyyttiiccss SSooffttwwaarree TToooollss We asked respondents if their IR offices are part of decision-‐making processes to select their institutions’ enterprise resource planning (ERP) and business intelligence (BI) reporting and analytics software tools (Chart 6). We found that 42% of respondents’ IR offices either participate in decisions or are the decision makers in selecting their institutions’ ERP systems. We also found that 68% of respondents said their IR offices either participate in decisions or are the decision makers in selecting their institutions’ BI solutions. Regarding institutions’ reporting and analytics tools/software, 76% of respondents’ IR offices either participate in decisions or are primary decision makers (Chart 7). When respondents were asked what they anticipate in terms of their IR offices’ influence on future purchases of reporting and analytics software tools, 96% believe their IR offices will participate in the decisions or will be decision makers.


EEvvaalluuaattiinngg SSooffttwwaarree TToooollss ffoorr IIRR WWoorrkk We asked respondents to identify the ideal features for software tools (Chart 8.) Nearly three-‐fourths noted that they desire a tool that provides access to all required data, and 69% would like the ability to capture historical data and decisions. NOTE: This question was multi-‐response; responses total more than 100%.

Respondents described the features their ideal software tools would contain.

48%

50%

53%

69%

74%

Built-‐in KPIs and metrics for tracking performance

Out of the box integraaon with ERP

Ability to engage all major departments electronically

Ability to capture historical data and decisions

Access to all required data

Chart 8: Features of an Ideal Solware Tool

In their words…

Be user friendly to someone who doesn't know how to code or do SQL. I should be able to build queries and reports as simply as it is to use Office or surf the web. Publish directly to and provide interactive use on websites without specialized web programming. Ease of use and ease of learning. Organization and easy search for reports etc. We don't have a high-‐powered predictive analytics tool, and that's what we need. Be able to integrate data from multiple sources and be interactive in filtering fields (users get to pick fields and criteria) and also be able to do a drill down (e.g. start with university overall enrollment and then add layers like UG/Grad, In state-‐Out-‐of-‐state, Male/Female, etc.) and be available via web interfaces It should readily interface with other databases. It should allow for real-‐time (or at least near-‐real time) visualizations of data out of the box. It should provide a user interface designed to be easy for a beginner to learn, with options available for more advanced users.


We presented survey respondents with a list of KPI tools currently on the market and asked them to indicate their satisfaction with the tools they use (Table 3).

Table 3: Software Tools

% Respondents whose

institutions use tool for KPIs

Satisfaction of respondent using tool in IR work

Dissatisfied (Answer = 1, 2)

Neutral (Answer = 3)

Satisfied (Answer = 4, 5)

Microsoft Excel 60% 2% 7% 90% Business Objects (SAP) 6% 10% 25% 65%

Entrinsik 5% 25% 13% 63% IBM Cognos 14% 25% 14% 61%

Crystal Reports 16% 7% 36% 57% SharePoint 16% 12% 35% 53%

Blackboard Analytics 8% 15% 35% 50% Nuventive TracDat 2% 0% 50% 50%

Evisions Argos 11% 23% 46% 31% We asked respondents to list other software tools used in their IR work, which resulted in a list of 34 additional tools; SPSS, SAS, Access, and Tableau were the most common. We asked respondents to identify all barriers they experience when using their current software tools for IR work (Chart 9). The need for more training and the cost of the tools were noted as the most common barriers.

24%

28%

29%

30%

33%

38%

39%

No enterprise resource planning integraaon

Lack of historical data and decision capture

Solware is not user-‐friendly

Inability to engage with all insatuaonal departments

The right data aren't available

Cost

Need more training

Chart 9: Barriers to Using Tools


SSeecctt iioonn 33:: CChhaannggiinngg IIRR RRoollee We asked respondents what changes they expect in their IR roles in the next three years. The predominant theme in respondents’ predictions was that IR will have a more strategic role (Table 4).

Table 4: Changing IR Role in Next Three Years

% of Responses

More strategic/planning 38% More data/analytics focused 17%

More assessment focused 8% Use of more BI tools 7%

More reporting 6% More technical 1%

More involvement in educating users 1% More of a support role 1%

Leaving the job 1% In a new position 1%

Don't know 5% No change 5%

In their words…

The role of IR within the university setting is changing particularly in the aspect of strategic planning, data support for decision making.

Involved in more strategic planning and assessment activities

More automated required reporting to different agencies. More time spent answering and asking questions related to student success and college goals.

My role will evolve into more of a support capacity. We have developed automated program review report application and we have implemented a more user-‐friendly Academic Assessment tool.

I believe that we will be more in the area of outcomes assessment.

IR will take over a more direct role in the long-‐term planning of BI with also hands-‐on development of BI (ODS/EDW build-‐out, meta-‐data, report /dashboard development, data governance). IR will go from strictly BI consumer to a key player in expanding and maintaining BI and NEIU.


We asked respondents to reflect on how advances in analytical technologies will change the responsibilities of their jobs. The predominant theme in respondents’ predictions was that IR professionals will have the abilities to conduct more analyses (Table 5).

Table 5. Change in Responsibilities Due to Advances in Analytical Technologies

% of Responses

Allow for more analyses 27% Improve efficiencies 10%

More reporting 10% Don't know/No answer 8%

General statement 8% Other users accessing data 8%

Need to adapt to new technologies 6% No changes to job 6%

New hires with different skills 4% Automated reporting 2%

Data story teller 2% Managing data 2%

More programming design 2% Needs training 2%

Pushing beyond current tasks 2% Support role 2%

In their words…

I will do less simple ad hoc and more in-‐depth analysis of data. I'm looking forward to the upcoming changes— they can't come soon enough.

The hope is that analytics allows our office to spend less time on standard and ad hoc enrollment reporting (because it is much less manual and more seamless) and allows our office to spend more time on analysis, evaluation, and assessment.

It will make data readily available to end users. It will ascertain the accuracy of data so that we can always have consistent data report no matter the time of the year the data is pulled from the student administration system.

If done properly, it will make us more productive (better and faster analysis without the issues of data gathering and cleansing, which now take up more than half of the time).

New hires must be more proficient and experienced in using such technologies. Will need to walk in the door with the basic training already completed.


SSeecctt iioonn 44:: IIRR BBuuddggeett We asked respondents to anticipate how their IR offices’ budgets will change over the next three years; the responses were very different by sector (Chart 10).

• All Respondents: We found 46% of respondents expect no change in their budgets and 32% expect their budget to expand over the next three years.

• Public 4-‐Year Institutions: Disaggregating by U.S. sector, we find more than half of respondents from public 4-‐year institutions believe their budgets will stay the same and 28% anticipate expansion of their budgets over the next three years.

• Public 2-‐Year Institutions: Over 90% of respondents from public 2-‐year institutions in the U.S. anticipate their budgets will stay the same or expand over the next three years.

• Private 4-‐Year, Not-‐for-‐Profit Institutions: Respondents from private, 4-‐year, not-‐for-‐profit institutions in the U.S. were less positive. We found 50% of respondents expect their budgets to shrink over the next three years.

NOTE: We only report responses from the three sectors listed above because of lack of responses in other sectors. Respondents from public 4-‐year institutions rated this item statistically higher than respondents from private, 4-‐year, not-‐for-‐profit institutions (p < .05).

All Respondents Public, 4-‐Year Public, 2-‐Year Private 4-‐Year, Non-‐profit

Expand by 25% or more 3% 2% 6% 0%

Expand up to 25% 29% 26% 43% 10%

Stay the same 46% 52% 43% 40%

Reduce up to 25% 17% 16% 6% 35%

Reduce by 25% or more 5% 4% 2% 15%

5% 4% 2% 15%

17% 16%

6%

35%

46% 52%

43%

40%

29% 26%

43%

10% 3% 2% 6%

Chart 10: Anacipated Change to IR Budget Over Next Three Years


14%

11%

32%

42%

$10,000 or more

$5,001 to $9,999

$1,001 to $5,000

Up to $1,000

Chart 11: Approximate Annual Amount Spent by IR Office on Analyacal Solware

Yes, all, 15%

Yes, some, 54%

No, 31%

Chart 12: Analyacal Solware Expenditures from IR Office Budget

We asked respondents to approximate the amount spent annually by their IR offices on analytical software tools. The largest segment, 42%, noted that less than $1,000 is spent annually, and only 25% of respondents reported that their offices spend $5,000 or more annually (Chart 11).

In a separate, but related question, we asked respondents to identify the amount of the single largest purchase made by their IR offices in the past year. One-‐quarter noted that they spent less than $1,000 and 46% said $5,000 or more.

We asked respondents whether analytical software expenditures come out of IR offices’ budgets. Nearly one-‐third (31%) said “no” and 15% reported that all expenditures come from IR office budgets (Chart 12).


CClloossiinngg There’s no doubt about it: institutional research professionals have an increasingly vital role in helping institutions identify, understand, and act upon opportunities for performance improvement. More than half of survey respondents said they foresaw their roles changing in the future, and nearly 40% said they expected to adopt a more strategic role in their institutions’ decision-‐making processes. That all makes sense given the overwhelming emphasis on key performance indicators today. A full 90% of respondents said KPIs are important to their institution. That importance is here to stay. And that’s exciting. With the right tools, training, and expertise, institutions can get the insights they need to truly elevate and increase overall performance to better serve students. Ellucian Perform is for campus leaders who have multiple sources of data and want to drive a culture of data-‐based decision making throughout the entire institution. To learn more about the industry’s first-‐to-‐market platform for establishing and maintaining a data-‐driven culture of performance and how this solution can significantly support the changing role of IR, visit Ellucian.com.

Wayne Bovier, Vice President of Product Management, Ellucian

Association for Institutional Research 1435 E. Piedmont Drive, Suite 211, Tallahassee, FL 32308 850.385.4155 (phone), 850.385.5180 (fax) Ellucian Headquarters: 4375 Fair Lakes Ct, Fairfax, VA 22033, USA Phone: +1 800.223.7036 To download copies of this research, visit www.ellucian.com


AAppppeennddiixx

SSuurrvveeyy MMeetthhooddoollooggyy aanndd RReessppoonnddeenntt CChhaarraacctteerriisstt iiccss Survey metrics were written by assessment experts at AIR in collaboration with staff from Ellucian, a company that specializes in business intelligence (BI) software. Sixty-‐five questions on the survey explored the types of key performance indicators (KPIs) used on respondents’ campuses, use of BI software, and respondents’ personal experiences with BI software. AIR randomly selected 1,546 members of the Association to participate in the study. The survey participants were divided into three groups, and the survey was divided into three sections; each survey group received one section of the survey in order to reduce respondents’ survey burden. An online surveying system was used to collect data April 22—May 6, 2014. One invitation email and two reminder emails were sent to survey participants. We collected 542 responses for a 35% response rate. Of the 542 responses, 536 individuals self-‐identified their IR roles and the locations of their institutions (Table 6).

Table 6: IR Role by Institution Location

Location of Institution U.S./U.S. territories Canada

Outside the U.S. and Canada Total

College or university, in an IR office 415 9 7 431 College or university, not in an IR office 64 0 3 67

System office 16 1 0 17 Non-‐profit organization 4 0 2 6

Retired 2 0 0 2 Other 13 0 0 13 TOTAL 514 10 12 536

For the purpose of this report, we focused on the 424 respondents who work in college or university IR offices in the U.S./U.S. territories or Canada. (In the report, reference to institutions in the U.S. includes U.S. territories.) The distribution of the 424 responses by survey is shown in Table 7.

Table 7: Survey # Responded Response Rate Survey 1 158 37% Survey 2 127 30% Survey 3 139 33% TOTAL 424


Senior institutional research (IR)/institutional effectiveness (IE) officers comprised nearly 60% of the response pool (Table 8). Note that 13 respondents failed to answer this item. NOTE: Due to rounding errors, values may appear to not be equal to 100%.

Table 8: Respondent's Role # of Respondents % of Respondents Senior IR/IE Officer 240 59%

Associate/Assistant IR/IR Officer 43 11% IR Analyst/Researcher 98 24%

Senior Assessment Officer 2 1% Assessment analyst/researcher 15 3%

Other 12 3% Total 410

Respondents who work at post-‐secondary institutions in the U.S. were matched with their institutions’ IPEDS sectors. The largest sector represented in the data set was private, not-‐for-‐profit, 4-‐year institutions followed by public 4-‐year institutions (Table 9.)

Table 9: Institutional Sector # of Respondents % of Respondents Private not-‐for-‐profit, 4-‐year or above 154 38%

Public, 4-‐year or above 145 36% Public, 2-‐year 87 22%

Private for-‐profit, 4-‐year or above 14 4% Private not-‐for-‐profit, 2-‐year 1 0%

Total 401

TTeessttiinngg ooff DDiiffffeerreenncceess iinn MMeeaannss We performed t-‐tests on survey metrics between respondents in different sectors. There were no statistical differences in means by sector except for the metric describing the anticipated change in IR budget over the next three years, as noted. In that metric, respondents from public 4-‐year institutions rated that item statistically higher than respondents from private, 4-‐year, not-‐for-‐profit institutions.


SSuurrvveeyy IInnssttrruummeenntt

Which best describes your current place of employment?

College or university, in an Institutional Research office

College or university, not in an Institutional Research office System office

Non-‐profit organization

I am retired Other (please specify)

Where is your primary work location? In the U.S./U.S. territories In Canada

Outside of the U.S. and Canada

How involved should IR offices be in choosing their institutions’ business intelligence solutions?

Should have primary responsibility for decision Should participate in decision-‐making

Should have no involvement Don’t know

How involved should an IR office be in choosing their institutions’ reporting and analytics tools/software?

Should have primary responsibility for decision Should participate in decision-‐making

Should have no involvement

Don’t know

How do you expect an institution’s IR office budget to change in the next 3 years?

Reduce by 25% or more

Reduce up to 25% Stay the same

Expand up to 25%

Expand by 25% or more Don’t know

Which best describes your role in Institutional Research?

Senior IR/IE Officer Associate/Assistant IR/IR Officer

IR Analyst/Researcher Senior Assessment Officer

Assessment analyst/researcher

Other (please specify)

How involved was your IR office in choosing your institution’s business intelligence solution?

Primary responsibility for decision

Participated in decision-‐making No involvement

Don’t know

How involved was your IR office in choosing your institution’s reporting and analytics tools/software?

Primary responsibility for decision Participated in decision-‐making

No involvement Don’t know


How involved was your IR office in choosing your institution’s ERP (enterprise resource planning) system (e.g., Ellucian, Jenzabar, PeopleSoft)?

Primary responsibility for decision Participated in decision-‐making

No involvement

Don’t know Institution doesn't have an ERP system

How do you see your role changing in the next few years? That is, do you anticipate that it will be more or less strategic, encompass different responsibilities, etc.?

How frequently do you interact with each of the following? (Scale = 1) Daily,

2) Weekly, 3) Monthly, 4) Less than monthly, 5) Rarely or never)

Information Technology

Finance

Registrar Academic Department Leaders

Administrative Department Leaders

Senior Leaders (CEO, CAO, etc.) Cabinet/Board of Trustees

How involved was the IR office with the following processes at your

institution? (Scale = 1) Not involved at all, 2) Somewhat involved, 3) Very

involved, 4) Institution does not use KPIs, 5) Don’t know)

Creation of Key Performance Indicators (KPIs)

Data collection to support KPI reporting Reporting on KPIs

Strategic planning institution-‐wide Other institutional performance improvement initiatives

How involved will the IR office expect to be with these processes in the next 3 years? (Scale = 1) Not involved at all,

2) Somewhat involved, 3) Very involved, 4) Institution does not use

KPIs, 5) Don’t know)

Creation of Key Performance Indicators (KPIs)

Reporting on KPIs

Data collection to support KPI reporting

Strategic planning Other institutional performance improvement initiatives

Are KPIs (Key Performance Indicators) a major responsibility for the IR office?

Yes No

Where do the KPIs used at your institution originate? Please select all that apply.

State Federal

Other funding organizations

President/Cabinet Academic leadership

Institutional Research Internal committees

Department committees

Other (please specify) Don’t know


Who has the primary authority to collect data on KPIs?

Institutional Research Information Technology

Department Leaders

Finance/CFO Operations/COO

Academic/Provost Office Other (please specify)

Don’t know

Who is the primary reporter on KPIs?

Institutional Research President

Information Technology VP Student Services

Department Leaders Finance/CFO

Operations/COO

Academic/Provost Office Other (please specify)

Don’t know

Where are KPIs reported? Please select all that apply.

Leadership

Staff focused on measurement

Faculty The public

Accrediting bodies Funding agencies

Government (state/federal) agencies Website


Don’t know

How are KPIs being used at your institution?

Formal strategic planning

Obtain funding resources Accreditation

IPEDS reporting

Other external reporting Other (please specify)

Don’t know


What software tools does your institution currently use for tracking KPIs (Key Performance Indicators) and reporting on data? Please select all that apply.

Blackboard Analytics Business Objects (SAP)

Crystal Reports

Entrinsik Evisions Argos

IBM Cognos Microsoft Excel

Nuventive TracDat

SharePoint Other (please specify)

Don't know Please list any other software tools used for IR tasks with which you are familiar.

Thinking of the ideal software tool for your role, what features would it include? Please select all that apply.

Access to all required data Out-‐of-‐the-‐box integration with ERP

Ability to engage all major departments electronically

Ability to capture historical data and decisions Built-‐in KPIs and metrics for tracking performance

Other (please specify) Don’t know

Please indicate which of the following KPI categories your institution currently tracks. Please select all that apply.

Recruitment/Admissions

Enrollment Student Services/Student Life

Academics Alumni Relations/Advancement

Finance and Administration Workforce Management


None of these Don’t know

How will the IR office’s involvement with the following KPI categories

change in the next 3 years? (Scale = 1) Decreased involvement, 2)

Involvement will stay the same, 3) Increase involvement, 4) Institution doesn't use this category, 5) Don't

know)

Recruitment/Admissions

Enrollment Academics

Alumni Relations/Advancement

Finance and Administration

Workforce Management

What is the level of importance placed on KPIs by your institution? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know)


How important is having KPIs defined across institutions to support benchmarking? That is, how useful would common definitions across institutions for KPIs be in order to support comparisons? (Scale = 1) Not at all important, 2) Somewhat important, 3) Very important, 4) My institution doesn't use KPIs, 5) Don't know)

What will be your IR office’s future involvement with these KPIs categories?

Thinking about the software tools that you use for Institutional Research, how satisfied are you with each? (Scale = 1)

Very dissatisfied, 2) Dissatisfied, 3) Neutral, 4) Satisfied, 5) Very satisfied,

6) Don't use this)

Blackboard Analytics

Business Objects (SAP) Crystal Reports

Entrinsik

Evisions Argos IBM Cognos

Microsoft Excel Nuventive TracDat

SharePoint

What would your ideal software tool be able to do that your current tool doesn't? Please explain.

Thinking about the software tools you are currently using for Institutional Research, what are the barriers to using these tools? Please select all that apply.

There are no barriers to the tools I currently use

The right data aren't available Need more training

Software is not user-‐friendly Cost

No enterprise resource planning integration

Lack of historical data and decision capture Inability to engage with all institutional departments


Which of the following would help you to retrieve data from your systems? Please select all that apply.

Additional services engagement with system vendor

Additional services from your Information Technology department

Workarounds (e.g., export data to Excel) Manual data collection/entry


How do you see the budget for your IR office changing in the next 3 years?

Reduce by 25% or more

Reduce up to 25% Stay the same

Expand up to 25%

Expand by 25% or more Don't know


What is the approximate value of largest purchase your IR office has authorized in the past 12 months?

Up to $1,000 $1,001 to $5,000

$5,001 to $9,999

$10,000 or more Don't know

Approximately how much does your IR office spend annually on analytical software tools?

Less than $1,000 $1,000 to $4,999

$5,000 to $9,999

$10,000 or more Don’t know

Do analytical software expenditures come out of the IR office's budget?

Yes, all Yes, some

No Don’t know

What kind of influence do you think the IR office will have in the selection of analytical software tools for Institutional Research in the next 3 years?

Will have primary responsibility for decision

Will participate in decision-‐making Will have no involvement

Don’t know

How do you think that the advances in analytical technologies will change the responsibilities of your job?

white paper: impact of business intelligence on institutional research

Documents