are you using technology to assess learning or assessing learning to improve technology? marilee j....

Are you Using Technology to Assess Learning or Assessing Learning to Improve Technology?

Marilee J. Bresciani, Ph.D.Associate Professor, Postsecondary Education andCo-Director of the Center for Educational Leadership, Innovation, and PolicySan Diego State University3590 Camino Del Rio NorthSan Diego, California, U.S.A.619-594-8318Marilee.Bresciani@mail.sdsu.edu

Overview

Overview of Outcomes-Based Assessment

Examples of Using Technology to Assess Learning or Assessing Learning to Improve Technology

Questions to Consider .

Ask Yourself These Questions

How would you describe what outcomes-based assessment is to your faculty, staff, and students?

How would you explain its importance?

Ask Yourself These Questions

How would your description change if you were describing assessment for

how well technology enhances student learning?

how well technology can be used to assess and report out on student learning ?

The Assessment Cycle (Bresciani, 2006)

The key questions…• What are we trying to do and why? or• What is my program supposed to accomplish?

or• What do I want students to be able to do and/or

know as a result of my course/workshop/orientation/program?

• How well are we doing it?• How do we know?• How do we use the information to improve or

celebrate successes?• Do the improvements we make contribute

to our intended end results?

The IterativeSystematicAssessment CycleBresciani, 2006

Mission/Purposes

Goals

Outcomes

Implement Methods to

Deliver Outcomes and

Methods to Gather Data

Gather Data

Interpret Evidence

Make decisions to improve programs; enhance student learning and development;

inform institutional decision-making, planning,

budgeting, policy, public accountability

Drivers

Regional Accreditation Requirement since 1985

National Commission on the Future of Higher Education (2006)

Demand for Public Information about Performance

• Transparency of outcomes and results• Comparable measures of quality• Demonstration of value-added of the

entire educational experience

Drivers, Cont.

Accountability requirements handed back to statesPerformance indicatorsStandardized tests

Discipline Standards could be designed by disciplines

Increased focus on private education Still no focus on evaluating

transferability of learning

Now turn to each other

Would you change any of your responses to the aforementioned questions?How would you describe what

outcomes-based assessment is?

How would you explain its importance?

This Assessment Process Can be Used To…

1. Assess learning, where technology is adata collection tool or teaching tool (to promote deep

learning)

2. Improve technology as a teaching tool or a data collection tool in order to make assessment of learning more effective

This Assessment Process Can be Used To…

3. Improve professional development for use of technology

4. Improve technological skills of students, faculty, and staff

5. Improve process of planning for improved student learning by using technology as an instructional organizational tool

So, One Process,

which is outcomes-based assessment,

for many uses

The Key is to Determine

What you are assessing?Therefore, what are your

outcomes?Informs DesignInforms MethodologyInforms Decisions

Student Learning Example

Students will be able to identify potential ethical research dilemmas and conflicts of interest and propose solutions to address themCase Study exercise in classAssignment - Student’s actual

research project using Blackboard discussion

Student Learning Example, Cont.

The majority of students were able to demonstrate this learning on blackboard and challenged their peers’ comments as well.

However, three students did not perform as well as they had on the in-class case studies.

What was the problem?

Lack of student learning

Or concern with technology as a data collection tool?

Two things Going on at Once

1. Assess learning, where technology is adata collection tool or teaching tool

2. Improve technology as a data collection tool in order to make assessment of learning more effective

Another Example

Assessing effectiveness of professional development for faculty using TracDat to manage their process for evaluating student learningEvidence of assessment plans and

reports entered

Another Example

All 10 who attended the workshop entered assessment plans But 6 of the 10 had poorly written

outcomes, and disconnected evaluation methods even though all items were in their “correct” box

Only 2 of the 10 faculty had entered assessment reports

What was the problem?

Poor professional development workshop

Or concern with faculty and their technology skills?

Three Things Going on at Once

2. Improve technology as a data collection tool in order to make assessment of learning more effective

3. Improve professional development for use of technology

4. Improve technological skills of students, faculty, and staff

One Final Example

Assessed instructional technology for its accessibility from the Pacific Islands Delays in assignments postedTroubles posting to discussion

board

Example, Cont.

Poor accessibility by students but could not discern if it was the software or the students

Interestingly, students did not complain about access and the designer said it was because the course expectations were so clear

What was the problem?

Poor technology

Or concern with students and their technological skills?

Three Things Going On

2. Improve technology as a teaching tool in order to make assessment of learning more effective

4. Improve technological skills of students, faculty, and staff

5. Improve process of planning for improved student learning by using technology as an instructional organizational tool

Re-Visiting: The Key is to Determine

What you are assessing?Therefore, what are your

outcomes?Informs DesignInforms MethodologyInforms Decisions

Take a moment and share

with each other a scenario where you determined that you needed to clarify whether you were assessing technology or learning

Questions to Consider (Bresciani, 2007)

When considering the use of Technology to promote outcomes-based assessment…Does the use of this technology reinforce the

reasons we are engaged in outcomes-based assessment?

How does the use of this technology promote formative assessment, summative assessment, or the assessment of transferable skills/knowledge?

Questions to Consider, Cont. (Bresciani, 2007)

Is our professional development to train users of our technology coupled with the professional development for outcomes-based assessment or instructional design?

How does the use of this technology enable us to better use the results to improve student learning and development?

Questions to Consider, Cont. (Bresciani, 2007)

How does the use of this technology improve efficiencies? And who benefits from those improved efficiencies?

How does this technology constrain or promote creativity/flexibility within each discipline?

How does this technology aid the coordination of the process? Or does it contribute to isolated thinking and doing?

Questions to Consider, Cont. (Bresciani, 2007)

How can data that this technological tool gathers help us evaluate the extent that we are transforming our organization through the use of evidence-based decision-making?“Keep your eye on the prize”

Challenge Questions to Consider (Bresciani, 2007)

How does this technology help illustrate the transferability of learning/skills?

How does this technology help us illustrate, evaluate, and improve the learning/skills as they relate to the requisite skills required by employers?

Challenge Questions to Consider , Cont. (Milne, 2007)

How does this technology help us determine whether the technology is promoting or prohibiting social interaction?

How do we ensure that the technology is assuring the deepest engagement with the subject matter?

Challenge Questions to Consider , Cont. (Bresciani, 2007)

How will you determine that technology is promoting deep learning?

What do you need to refine in order to make this happen?

Which questions do you need to focus on as you move forward with your learning at this conference?

Questions?

Marilee.Bresciani@mail.sdsu.edu

References

Bresciani, M.J. (2006). Outcomes-Based Academic and Co-Curricular Program Review: A Compilation of Institutional Good Practices. Sterling, VA: Stylus Publishing

Bresciani, M.J. (2007). Implementing Lessons Learned: Establishing A Culture of Evidence-Based Assessment. Closing Keynote at Course Technology Conference. Las Vegas, NV.