the art of evaluation combined slides - arts midwest · – etienne wenger touchstone center ......

October 11

1

The Art of Evaluation:

Reflective Learning Loops

October 17, 2011

Welcome

Opening Remarks: Sharon DeMark

Minnesota Philanthropy Partners

David ScheiePresident

Touchstone Center for Collaborative Inquiry


Building a Learning Culture within Your

Organization or Project

The Art of Evaluation: Reflective Learning Loops

October 17, 2011


What is a “learning culture?”

“A culture of learning is an environment that supports and encourages the collective discovery, sharing, and application of knowledge.”

-- Stephen Gill (2008)


Organizational or project culture:

“Shared history, expectations, rules (written & unwritten), values, relationships, and customs that affect everyone’s behavior.” (Conner & Clawson)

“The sum of solutions to yesterday’s problems.” (Schein)

October 11

2


Organizational or project culture:

Appears at 3 levels:

Level 1: Physical artifacts

Level 2: Intangible policies, rituals, procedures, and networks of relationships

Level 3: Values, assumptions, beliefs, and expectations that underlie levels 1 and 2.


A project with a learning culture is like a …

Learning organization: “an organization where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are included, where collective aspiration is set free, and where people are continually learning to see the whole together.” -- Peter Senge

Community of practice: “a group of people who share a passion for something that they know how to do and who interact regularly to learn how to do it better.” – Etienne Wenger


A learning culture benefits…

Individuals: improves performance, expands opportunities, enhances self-worth, prevents boredom

Organizations: improves morale, effectiveness, efficiency, impact

Communities: more assets, dynamic innovation, better problem analysis & resolution


Assessing your learning culture: self-audit

Pro-learning culture

People ask questions and share stories about successes, failures, and what they have learned.

People take at least some time to reflect on what has happened and what may happen.

Senior managers are willing to explore their underlying values, assumptions, beliefs, and expectations.

Anti-learning culture

Information is shared on “need to know” basis. People keep secrets & don’t describe how events really happened.

Little time or attention is given to understanding lessons learned from projects.

Senior managers are defensive and unwilling to explore their underlying values, assumptions, beliefs, and expectations.


Learning culture self-audit, continued…

Pro-learning culture

People are treated as complex individuals.

People are hired and promoted on the basis of their capacity for learning and adapting to new situations.

Performance reviews include attention to what people have learned

Anti-learning culture

People are treated like objects or resources without attention to their individuality.

People are hired and promoted based on their technical expertise as demonstrated by credentials.

Performance reviews focus almost exclusively on what people have done.


Learning culture: healthy practices

New ways of…

Asking questions

Running meetings

Conducting performance reviews

October 11

3


Learning culture: healthy practices…

Participating in group discussion

Having one-on-one conversations

Reading about new ideas

Watching experts duel over cutting-edge issues

Reflecting on new data


Relevant metrics and data

Who’s coming, and how many? Demographics

Roles, e.g. , student, spectator, volunteer

What difference are we making for people? Attitudes

Behaviors

Relationships


Relevant metrics…

What influence on other institutions and the broader community?

Shifts in programs, practices, policies, partnerships

Participants who go on to contribute to the community as artists, leaders, teachers, etc.


Relevant metrics: gathering data

Talk-back sessions

Facebook, Twitter posts

Visual documentation, observation

Administrative, attendance records

Surveys

Interviews

Focus groups



Meetings include time for task coordination and joint reflection

Annual rhythm includes short meetings and longer learning conferences,

reflection sessions and retreats

1:1, small group and large group interactions

People interact in the private spaces between meetings



Program designs often start with fewer elements, more open space than in traditional program design Leaves room for discovery and innovation

Small, everyday interactions are among the most valuable community activities informal discussions to solve a problem one-on-one exchanges of information about a tool,

approach or situation

October 11

4



People move between multiple levels of participation:

Core members who steer, initiate and participate extensively

Active members who participate regularly

Peripheral members who watch the interactions of core & active members

Neighbors who have an interest



Successful learning communities offer the familiar comforts of a hometown, but also enough varied events to keep new ideas and new peoplecycling into the community.

Familiarity, stability => candor, relationships

New ideas, people, events (divergent thinking) => excitement, a sense of common adventure, & challenges to assumptions



Vibrant communities of practice have a rhythm.

If the beat is too fast, the community feels breathless; people stop participating because they are overwhelmed.

When the beat is too slow, the community feels sluggish.

When the beat is strong and rhythmic, the community has a sense of movement and liveliness.


Tensions & challenges with a learning culture

Expansive reflection may conflict with task completion Do we have time to reflect? To wonder? To think?

Ambiguity & uncertainty can be stressful


Learning culture in organizational and collaborative projects

Simpler to implement within an organizational project Collaborative partners must agree to invest in, commit

to the learning culture

But contrasts among partners offer rich fuel for learning! And collaborative partners must learn if they are to

collaborate successfully


Learning culture: implications for leadership

Traditional view: leaders set vision, give direction, make key decisions, energize troops Assumes most people lack vision, need direction, can’t

make good decisions, lack energy Hierarchical, charismatic models of leadership

Learning culture view: leaders are designers, stewards, teachers Assumes people possess vision, imagination, capacity

to solve problems and create things of value Shared, flat, collaborative models of leadership

October 11

5


Learning culture: implications for leadership…

Leaders as

Designers (designing environments for learning)

Stewards(recognizing and developing others’ talents)

Teachers (facilitating others’ learning)

Panelist 1: BARBARA PlunkettMarketing Director,

MacPhail Center for Music

MARKET RESEARCHFindings, Action, Results

M A C P H A I L C E N T E R F O R M U S I C

MacPhail Center for Music

WALLACE GOALS

Launch “Customer Centered Approach” organization-wide, resulting in…

Increased enrollment and event attendance among 25-54 year olds

OVERVIEW:Customer Centered Approach

STEP 1:Customer Service

Communication Systems

STEP 2:STEP 3:

RESULTS:

CUSTOMER

October 11

6




STEP 2:STEP 3:

RESULTS:

CUSTOMER


STEP 1:•Customer Service•Communication

Systems

STEP 2:Apply findings about

customers: alter Product,

Place, Price and/orPromotion to better

meet needs

STEP 3:

RESULTS:

CUSTOMER



Systems


Customers: alter Product,


meet needs

STEP 3:Incorporate new

Knowledge of External Variables

RESULTS:

CUSTOMER



Systems

STEP 2:Apply findings

aboutCustomers: alter

Product,Place, Price and/orPromotion to better

meet needs



RESULTS:Satisfied customers

Positive teaching experienceIncreased

Enrollment

CUSTOMER

Data Collection

One Time: 26 focus groups; current and potential, 6 Event attendee surveys

Bi-Annual: Demographic Survey

Annual: New Student Satisfaction Surveys

Quarterly: Web and Google analytics

Monthly: Coupon tracking, Student Services feedback

Analyzing Data

Marketing Department: Compiles, conducts 1st

analysis, creates summary report.

Program Leadership: Deeper analysis by area.

Marketing/Leadership: Identify specific action, build into annual/weekly plans. Action plan created.

Staff and Board: Share findings, action and results

October 11

7

26 Adult Focus Groups –One time research

Current: Strong praise for their teacher, class offerings Feel front desk is intimidating at times Lack of communication issue

Potential: No awareness of classes/ensembles, assume

Individual Instruction only Assumed expensive – yet their “fair” price = ours Want clarity on skill levels

Sample Adult Catalog

Adult Focus Groups –Action and Result

Altered price listing, by session and semester

Added “great for beginners”

Created adult-specific ads on public radio and TV stations

Restructured Student Services area

Result: 5% increase in adults since FY09

Early Childhood Music Parents –One time researchCurrent: Praised high-quality program, grounded in child

development, LOVE teachers, reputation NOT clear pathway after ECM Take home materials mixed

Potential: One-third never heard of MacPhail Assumed just for older “serious” musicians Competitors take home materials better Concerns about website usability

Early Childhood Music Sequence

Early Childhood Music –E-lesson sample

October 11

8

Early Childhood Music –Action and Results

Launching new website next week

Cut a new ECM CD as give away

Created ECM specific ads on public radio and TV stations

Results: Positive response. Over 100% enrollment in 2 of 3 sites

EVENTS SURVEYS – One Time

450 events – Onsite 6 surveys at Spotlight, Jazz and Bach’s Lunch

EVENTS SURVEYS -- Findings

98% Agree/Strongly Agree “high-quality” performance

Detailed demographic info/event

Only 5% Likely/Very Likely to become students

Only 13% current students

EVENTS SURVEYS – Action, Results

Action

Continue success of Spotlight Series and Jazz

Maximize faculty performances, decrease guests

Add internal communication to increase students

Introduce Free Family Music Series

Results

Sold out 3 of 6 concerts

Increased student attendance at events

Student Satisfaction Survey –Annual Research

Selected communication and customer satisfaction items to measure longitudinally

By program

By service area

By individual teacher

Student Satisfaction Survey -ACTION

Faculty to/from Student Services New role

Student Services to Students New placement process (use script, send ques.)

Re-org space (sit in front, clear counters, attire)

Conduct Customer Service training

Staff to Faculty Intro new orientation

Spread the Word kits

Professional Development

October 11

9

Sample Student Satisfaction

49

ECM2010

ECM2011

Group 2010

Group2011

Suzuki

2010

Suzuki2011

Individual

Instruction

2010

Individual

Instruction

2011

Music Therapy

2011

Materials and repertoireExcellent

69% 72% 70% 82% 83% 78% 82% 83% 65%

Good 30% 26% 29% 15% 16% 20% 17% 16% 20%Fair

2%2%

1%3%

1%1% 1% 5%

PoorN/A 1% 1% 1% 10%Total 100% 100% 100% 100% 100% 100% 100% 100% 100%Preparedness of teacherExcellent

84% 79% 78% 83% 81% 88% 87% 88% 90%

Good 15% 21% 17% 15% 17% 10% 11% 11% 10%Fair

1%5% 2%

2%1%

2%1%

PoorN/ATotal 100% 100% 100% 100% 100% 100% 100% 100% 100%Guidelines and expectations for home learningExcellent

40% 49% 55% 65% 71% 75% 78% 80% 45%

Good 38% 35% 35% 29% 24% 22% 20% 17% 30%Fair

8%9% 7% 5%

5%3%

1%2% 15%

CONCLUSION:Customer Centered Approach



STEP 2:STEP 3:

RESULTS:

CUSTOMER



Systems


customers: alter Product,


meet needs

STEP 3:

RESULTS:

CUSTOMER



Systems


Customers: alter Product,


meet needs



RESULTS:

CUSTOMER



Systems

STEP 2:Apply findings

aboutCustomers: alter

Product,Place, Price and/orPromotion to better

meet needs



RESULTS:Satisfied customers

Positive teaching experienceIncreased

Enrollment

CUSTOMER

QUESTIONS

October 11

10

Panelist 2: PJ DoyleManaging Director, Mixed Blood Theatre

Radical EvaluationThe Art of Evaluation: Reflective Learning Loops

October 17, 2011

What is…

Mixed Blood Theatre

Our VisionTo be the definitive destination where theater artists and audiences representing the global village can create and share work that spawns a ripple effect of social change and revolutionizes access to theater.

57

What is…

Radical Hospitality

Radical Hospitalityprovides no-cost access to mainstage productions for allaudience members beginning with the 2011–12 season. An expansion of Mixed Blood’s egalitarian mission, Radical Hospitalityerases economic barriers in pursuit of building a truly inclusive, global audience.

58

Today’s…

Strategic Question

Howdo we implement and measure the Radical Hospitality model with particular attention to engaging individuals to whom cost is a barrier to an arts experience?

59

WHO are the stakeholders?

AudiencesContributors

Artists

Create theProgram

Deliver the Program

Assess the impact

Communicatewith stakeholders

Plan the Programming

Mixed

Blood staff

Board

of D

irectors

5

October 11

11

Data CollectionAsking questions•Traditional audience surveys• Intrinsic impact surveys• Free ForUms• Post performance panel discussions• Social media

6162

Analyze, Assess, Crunch

What does it all mean?• Statistical comparisons• Measurable outcomes• Demographics

…and to whom?• Staff• Board• Audience• Contributors• Artists

63

How findings influence Mixed

Blood’s stakeholders

Now what?• Program planning• Community connections• Resource allocation

• Financial• Staffing

• Strategic thinking• Reporting

…and how?• Funder reporting requirements• Annual report• Dashboards

63

WHO are the stakeholders?

AudiencesContributors

Artists

Create theProgram

Deliver the Program

Assess the impact

Communicatewith stakeholders

Plan the Programming

Mixed

Blood staff

Board

of D

irectors

9

Mixed Blood’s VisionTo be the definitive destination where theater artists and audiences representing the global village can create

and share work that spawns a ripple effect of social change and revolutionizes access to theater

65 66

October 11

12

67

Are any of the scenes, lines or music from the performance still bouncing around in your head? 3.88

• Post-Performance: Impact Echo - indicates the extent to which respondents report that scenes or lines from the play [or musical] are still present in their minds

Afterwards, did you discuss the performance with others who attended? 36% intense discussion

• Post-Performance: Meaning-Making through Dialogue with Others Who Attended - indicates the percentage of respondents who reported having an "intense exchange" about the work with others who attended (a form of intellectual stimulation)

Overall, at what level were your expectations fulfilled for this performance? 4.16

• Summative Satisfaction - percentage of respondents who report that their expectations for the experience were fulfilled

68

Questions

69

Panelist 3: Kristin PrestegaardDirector of Marketing and Communications,

Minneapolis Institute of Arts

To enrich the community by,

collecting, preserving and

making accessible

outstanding works of art from

the world’s diverse cultures.

VISION & MISSION

Inspiring wonder through the power of art.

October 11

13

How can we engage

the next generation of

art lovers, ages 20-45?

KEY STRATEGIC QUESTION METRICS

Target Visitor Focus Groups

Museum-wide Visitor Survey

Survey As We Go

Social Media Feedback

METRICS – Focus Groups

• Who is NOT coming?

• Arts interested people who said “the MIA isn’t for me”

• Shifting to a younger age –46 to 39

• 20 to 30 year-old crowd increases from 2010 (33%) to 2011 (45%)

• Increased size of 21 to 45-year-old Third Thursday audience to 85% vs. 55%

METRICS - Annual Visitor Survey

• Direct Feedback on a Day to Day Basis

• What do you do with this?

• How do you use it?

• What’s important?

METRICS - Social Media

• Avinash Kaushik, Analytics Evangelist for Google

• October 10, 2011 blog

•http://www.kaushik.net/avinash/best-social-media-metrics-conversation-amplification-applause-economic-value/

METRICS - Social Media

Conversion Rate, Amplification, Applause Rate, Economic Value

October 11

14

METRICS - Survey As We Go

Is Something Better Than Nothing?

• Third Thursdays

• Music in the galleries

• Audio guides

• Lectures

• Art Carts

Sharing Visitor Survey

1. Share with our senior leadership team

2. All staff brown-bag lunch

3. Top results shared at all staff meeting

4. More in depth information accessible via internal Intranet

INTERNAL LEARNING – Staff input

Partner With Groups That Know Their Networks Best

INTERNAL LEARNING – Partner input

Insert photos

• Flickr albums

• Reflect what we want the event to look like to external audiences

TELLING OUR STORY TO EXTERNAL AUDIENCES

• Programming reflects what we are saying in our communications… you really can “Come As You Are”

TELLING OUR STORY TO EXTERNAL AUDIENCES TELLING OUR STORY TO EXTERNAL AUDIENCES

• Come As You Are

• “The MIA isn’t for me”

• “I don’t see myself at the MIA”

• Literally tell them they are welcome here

October 11

15

B I L L B O A R D S B I L L B O A R D S

P R I N TD O W N T O W N M I N N E A P O L I S

S K Y W A Y A D S

• Mirror display ads:

T E L E V I S I O N

KARE 11 TV spot

Visitor Response:

• Visitor Tweets

•Visitor Video

• Age and Demographic Switch

TELLING OUR STORY TO EXTERNAL AUDIENCES

October 11

16

Discussion andQ & A

Closing Remarks: Susan ChandlerArts Midwest

Adjourn

Thank you!

the art of evaluation combined slides - arts midwest · – etienne wenger touchstone center ......

Documents