lauren weiner, ed.d. director, associated students administration university of california-san diego...

Lauren Weiner, Ed.D.Director, Associated Students Administration

University of California-San DiegoMarilee J. Bresciani, Ph.D.

Professor, Postsecondary Educational LeadershipSan Diego State University

Ben Gillig, M.A. CandidateAssistant Coordinator of Academic Initiatives

San Diego State University

Lisa McCully, M.A.Director, College of Education Office of Student Services

San Diego State University

Utilizing an Outcomes-Based Program Review to Inform

Resource Allocations

Bresciani, Gillig, Weiner, & McCully

Session Overview

Presentation of framing questionsReview of research studyDiscussion of findings and application to practice

Presentation of a preliminary framework

Discussion with small groups


Framing Questions

What evidence is used to inform your resource re-allocation or allocation processes?

How does your outcomes-based program review process inform resource re-allocation or allocation process?

Why would you even want to have this conversation?


Background

Resource allocation is about more than just money… Human resources Facilities IT Time spent by senior leaders Budget allocations Evidence that priorities are being effectively met

Resource allocation and re-allocation are cornerstones of senior student affairs officers’ jobs

How resources are allocated says something about how the division operates and how it sets its priorities


Student Affairs Leaders Pulling The Pieces Together

Resource AllocationResource Allocation


Literature Review

Budgeting Frameworks Program budgeting Formula budgeting Zero-base budgeting Performance budgeting Incremental budgeting

Budgeting Frameworks Program budgeting Formula budgeting Zero-base budgeting Performance budgeting Incremental budgeting

Require a set of purposes, priorities,

and/or strategies to

work.

What framework do we use to ensure that our purposes, priorities, and/or strategies will be effective in accomplishing

intended results?Bresciani, Gillig, Weiner, & McCully

Purpose of the Study

This mixed method study explores how institutional/divisional leadership use outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development.

Explores the need to utilize outcomes-based assessment in times of budget cuts, reduction in revenues, and a higher demand for services

The findings intend to identify frameworks that align resource allocations and re-allocations with stated priorities and provide evidence to refine programs and services that align with those priorities.


Hypotheses

The manner in which institutions are funded does not influence how institutional leadership uses outcomes-based assessment results to inform resource allocations and re-allocations for student learning and development.

The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development.


Hypotheses, Cont.

The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development.

The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development.


Hypotheses, Cont.

The manner in which institutional leadership engages in outcomes-based assessment program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development.


Methodology

This mixed methods study included the following data collection processes:

Survey list started with institutional members (n=1,030)

Researchers stratified the institution by typeA random sample of 257 invitations were then sent

through Student Voice.Survey invitations were sent to the institution’s

Senior Student Affairs Officer E.g., VP of Student Affairs or Student Services


Methodology

After data was collected, researchers analyzed the information using descriptive statistics

Researchers looked for correlations between institutional/divisional budget process and how they utilized their outcomes-based assessment results for their resource allocations and re-allocations

Researchers conducted a document analysis based on the information that was provided from their institutions URL addresses

Interviews will be conducted in the upcoming weeks


Survey Participant Demographics

Institution typePrivate 4-year Liberal Arts Colleges (35%)Public 4-year Comprehensive (20%)Public 2-year (15%)

Institution sizeLess than 4,000 (37%)4,000 to 8,000 students (24%)15,001 to 25,000 students (22%)

Position titleSenior Student Affairs Officer (93%)


Instrument Development

• Survey included 25 questions and a request for institutions to provide their URL address for additional assessment documents

Survey focused on asking institutional leaders to discuss if they utilize (a) outcomes-based assessment, (b) program review, (c) annual reports, and (d) strategic planning to inform the budget and resource allocation and re-allocation process at the institutional and divisional level

Survey included questions pertaining to institutional/divisional leadership practices and questions relating to the practices used for allocating and re-allocating resources to student learning and development programs


Document Analysis

• Survey respondents were asked to provide their URL addresses• Document analysis selection criteria:• All materials needed to be official institutional documents,

which included:• Annual reporting processes• Outcomes-based assessment processes• Program review processes• Assessment plans from the institutional, divisional, and

departmental levels• Researchers analyzed documents and developed themes by

using the comparative method for category development• Researchers determined essential descriptive groupings during

the initial coding phase


Instrument Development

Interview Protocol:• Researchers will conduct semi-structured interviews with

institutional leaders, such as vice-presidents, deans, faculty, and administrators directly involved in the outcomes-based assessment program review process

• Researchers will also conduct interviews with institutional leaders involved in the resource allocation and re-allocation processes

Questions focus on:• Types of evidence based decision-making processes that

institutional leadership use to inform the resource allocation and re-allocation process for student learning and development

• The alignment between the results generated from outcomes-based program review and resource re-allocations and allocations


Limitations

Since the researchers used a self-reporting survey, participants did not have to provide their actual budgeting documents, thus researchers cannot triangulate that data; however, outcomes-based and planning practices can be triangulated through document analysis

25% response rate; survey is back out in the field


Preliminary Findings


WE DO NOT HAVE CONCLUSIVE DATA FOR REJECTING THIS HYPOTHESIS

The manner in which institutions are funded does not influence how institutional leadership

uses outcomes-based assessment results to inform resource allocations and re-allocations

for student learning and development.


Use of tuition and fee revenue was associated with use of OBA to formulate budget requests at both the institutional and division levels but not to inform actual resource allocation (t=-.43, p>.01; t=-.48, p>.01).Use of state allocated revenue was associated with OBA playing a role in re-allocation only (t=-.34, p=.02).Use of state allocation revenue was also associated with OBA playing a weighted role in the formulas used to allocate money to departments (t=.83, p=.01).Use of endowment revenue was associated with OBA playing a weighted role in the formulas used to allocate money to divisions (t=.31, p=.04).Use of revenue from auxiliary services was associated with OBA playing no role in resource allocation/re-allocation at all (t=-.32, p=.02).

Based on these preliminary findings, we will be conducting interviews and gathering additional documents to clarify


There were significant correlations between certain revenue sources and use of outcomes based assessment at certain points in the

resource allocation/re-allocation process.

The timeframe in which institutional leadership allocates and re-allocates resources for student learning and development does not influence how they use outcomes-

based assessment results to inform the allocation and re-allocation of resources for student learning and development.

• 98% of survey participants responded that their resource allocation process takes place on an annual basis

• Since the majority of respondents selected the same timeframe, we can not determine the extent that the time frame influences this process.

• Thus, we can not accept or reject this hypothesis


The budgeting framework that institutional leadership uses to allocate and re-allocate resources for student learning and development does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development.

46% report their institutions using incremental budgeting processes, while 35% use program-based at the divisional level

There are no significant correlations between budgeting framework and the use of OBA for resource allocation and re-allocation

We accept this hypothesisBresciani, Gillig, Weiner, & McCully

The manner in which institutional leadership engages in strategic planning does not influence how they use outcomes-based assessment results to inform the allocation and re-allocation of resources for student learning and development.

• Strategic planning is the most common practice used to inform the resource allocation or re-allocation process

• Institution level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process

• Institution level: 20% of survey respondents reported that strategic planning played a role in the way they construct the budget, but not the allocation of resources

• Division level: 22% of survey respondents reported that strategic planning informs the resource re-allocation process

• Division level: 22% of survey respondents also reported that strategic planning played a weighted role in the formulas used to allocate dollars

• There are no significant correlations between the use of strategic planning and the use of OBA for resource allocation and re-allocation

• We accept this hypothesis.


The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development.

• Institution level: 24% of survey respondents reported that program review does not have any role in the allocation or re-allocation of resources

• Institution level: 20% of survey respondents reported that program review plays a role in the re-allocation of resources

• Division level: 27% of survey respondents reported that program review plays a role in the re-allocation of resources

• Division level: 22% of survey respondents reported that program review does not play any role in the allocation or re-allocation of resources


The manner in which institutional leadership engages in outcomes-based program review does not influence how they use outcomes-based assessment results to inform the resource allocation and re-allocation of resources for student learning and development.

• However, uses of outcomes-based program review data was correlated with several uses of outcomes-based assessment data:• Use of program review data in a weighted role in budget formulas used to allocate

money to departments was correlated with the use of outcomes based assessment data for the same purpose (t=.43, p<.01).

• Use of program review data in a weighted role in the budget formula used to allocate money to programs and processes correlated with the use of outcomes based assessment data for the same purpose (t=.47, p<.01).

• Use of program review data to construct budget proposals, but not necessarily to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.67, p<.001).

• Use of program review data to construct budget proposals, but definitely not to allocate resources was correlated with the use of outcomes based assessment data for the same purpose (t=.69, p<.001).

• Thus, we cannot reject this hypothesis.


Preliminary Findings from the Document Analysis

Documents analyzed from the URL addresses included annual reports, assessment plans, program reviews, strategic planning processes, assessment committee structures, and institutional planning documents

Themes from the documents: Departmental goals and objectives align with the

student affairs and institutional goals and objectives• Institutions use assessment results to inform the

program improvement process, not in the resource allocation process

• Identification of departmental and divisional successes, but do not inform the resource allocation process


Similarities and Differences Between Survey and Documents

• Institutional leaders identified and communicated values, objectives, and priorities

Departments aligned programs with the institutional goals and objectives

Input being solicited from faculty, staff, and administrators

Strong participation of faculty, staff, and administrators in the planning process

Differences between survey and documents:Survey respondents indicated that resource re-allocations

were influenced by strategic planning and outcomes-based assessment program review findings; documents showed no linkage to the resource re-allocation process


Summary of Conclusions

• There is a disconnect between what revenue sources influence versus what the budgeting process influences.

• There is a disconnect between the institutional budgeting processes and the ability to use allocation of resources to improve student learning and development. Does that make sense to continue?

• If 74% of the initial allocation of funding is done annually, why aren’t outcomes-based program review results used to inform initial allocations of resources?


Summary of Conclusions, Cont.

• If strategic planning and outcomes-based assessment program review findings only influence resource re-allocations and not initial allocations, how will leadership be able to influence long-term improvement in student learning and development?

• More to come….


Framework with Modifications(Bresciani, 2010 & Bresciani, et.al, in progress)

Identify and articulate valuesPrioritize valuesAllocate resourcesAlign outcomes to values and Implement

outcomes-based assessmentDefine the criteria for quality within the context

of the values and identify capacity for meeting the criteria of quality

Gather the results and determine at which level the decision for resource re-allocation or allocation resides


Framework with Modifications(Bresciani, 2010 & Bresciani, et.al, in progress)

Allocate or re-allocate resources to improve your outcomes within your context and capacity for quality and in alignment with your values


Small Group Discussion Questions

How well do your resource budgeting and allocation processes align?

How much do you utilize planning and assessment processes to allocate or re-allocate resources to refine your priorities?

Does this proposed framework offer any practical value for your division/institution?

What are some immediate next steps that you can implement?


Planned Next Steps for the Study

Complete correlation analysisRe-invite participants to complete the

surveyRequest more URLs for expanded analysis

- particularly budget process documentsConduct interviews to test the viability of

the frameworkWrite-up results for publication

submission


Special Thanks to…

Student VoiceJD White Annemieke Rice

Participating institutionsTo you


Session Citation

This presentation: Weiner, L., Bresciani, M.J., Gillig, B., McCully, L.

(2011). Using outcomes-based assessment to inform resource allocation in higher education. Workshop presented at the annual meeting of the American College Personnel Association, Baltimore, MD.

Future paper: Bresciani, M.J., Weiner, L., Gillig, B., McCully, L.(in

preparation). Using outcomes-based assessment to inform resource allocation in higher education. Journal of Student Affairs Research and Practice submission


References

Bresciani, M.J. (2010). Aligning values with resources and assessment results. Student affairs leader (38), 13, p. 1-2.

Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating student success in student affairs. Sterling, VA: Stylus Publishing.

Berg, D.J. & Skogley, G.M. (1985). Editors’ notes. In D.J. Berg & G.M. Skogley (Eds.), Making the budget process work. New Directions for Higher Education No. 52. San Francisco: Jossey-Bass.

Carnegie Council on Policy Studies in Higher Education (1980). Three thousand futures: The next twenty years for higher education. San Francisco: Jossey-Bass.

Clark, R. & d’Ambrosio’s, M. (2006). The new balancing act in the business of higher education. Northamton, MA: Edward Elgar Books.


References, Cont.

Merriam, S. B. (1998). Qualitative research and case study application in education. San Francisco: Jossey-Bass.

Mayhew, L. (1979). Surviving the eighties: Strategies and priorities for solving fiscal and enrollment problems. San Francisco: Jossey-Bass.

Strauss, A., & Corbin, J. (1990). Basics of qualitative research. Newbury Park, CA: Sage.

Taylor, S. J., & Bogdan, R. (1998). Introduction to qualitative research methods. New York: John Wiley.


Contact Information

Marilee J. Bresciani – [email protected]

Ben Gillig –[email protected]

Lauren Weiner – [email protected]

Lisa McCully – [email protected]
