Large Effects With Small Efforts: What Does it Take to Leverage User Data?

Joe Zucca, Director for Planning and Organizational Intelligence, Penn Libraries

SLA, Philadelphia, April 2, 2014

PrologueProducing Large Effects With Small Effort?

Count of User Type Column Labels        

Row Labels FAC Other Staff StudentGrand Total

Canvas Features 217 2 53 16 288

Access 200 2 58 69 329

Site Request 103  41  144

Administration 8 1 7  16

Account Request 1 1    2

Other 1  4  5

Grand Total 530 6 163 85 784

Courseware Problem Incidents: Frequency by Incident & User Group

How Well Are They Managing Complexity (in the LMS)?

Migrate content 41

Settings 29

Course content 27

Assignments 21

Copy content 17

Other (Features) 15

Announcements 14Conversations/messages 13

Multimedia 11

LTI Tools (other) 8

Quizzes/surveys 6

Gradebook 6

Modules 4

Discussions 3

Turnitin 1

Learning Objects 1

Average of lagColumn Labels        

Row Labels FAC Staff Student Other Grand Total

Administration 3.0 5.4  5.0 3.9

Canvas Features 1.9 4.0 0.6 27.5 2.4

Site Request 1.4 1.8    1.5

Access 1.3 0.4 1.0 1.5 1.1

Account Request 0.0    7.0 3.5

Other 0.0 3.3    3.2

Grand Total 1.6 2.2 0.9 11.7 1.8

Courseware Problem Incidents: AVG resolution time by Incident & User Group

How Well Are We Managing Complexity (in the LMS)?

Quizzes/surveys 9.2

Multimedia 4.0

Gradebook 3.8

Assignments 2.4

Migrate content 2.0

Copy content 1.8

Course content 1.6

Settings 1.6

Turnitin 1.0

Modules 0.5

LTI Tools (other) 0.4

Other (Features) 0.4

Announcements 0.4Conversations/messages 0.3

Discussions 0.0

E-reserves 0.0

Learning Objects 0.0

The Challenge: The Scope & Complexity of Service

What products and services does your library offer or directly support for the university?

Acquisition and licensing services Access to massive e-content Discovery service Supply chain services [local circulation & third party - ILL] Inventory control Research | Instructional services Courseware service (e.g., Blackboard) Learning environments (e.g. MOOCs) Author service Copyright compliance service Technology support services (e.g., desktops, digital media,

scanning) Digitization services (image production, description, curation) Metadata service Data services Repository services (internal and public-facing digital

archives) Researcher intelligence services (e.g., VIVO)

COUNTER

SFX

SOLRlog

ILLIAD

VIVO

Canvas LMS

BePress

FootPrints

Fedora Repo

Banner IDM

RelaisGoogle Analytics

ARES

HR db

STAFF

SpringShare

AEON

ILSAPACHE

log

EZPROXYlog

AnalyticsRepository

A B.I. Framework

AnalyticsRepository

Extract Trnsfrm Load

Data Mart AnalysisEnterprise Products &Services

Resolvers

a framework for:

Extracting activity data from systems

Transforming those data into readable, normalized formats

Loading transformed/normalized payload into a repository

Supporting analysis through local and collaborative dissemination channels.

https://github.com/upenn-libraries/metridoc-app

Abstracts 4 key functions, exposes interfaces for interoperability

Target Source, e.g. Relais, Illiad, ILS

Ingest Log

Parse

Format

Refined output

1. Extract

Resolution Sources e.g. IdM, WorldCat

Refined output

Resolve Codes & IDs

Normalize

2. Transform

Query Srvc

Data Repo

3. Load

MetriDoc Tiered Architecture

User Interface

LocalData Stores

Results Document

Query Document

4. Query

Economic failure of German colonies in AfricaUsing Google Fusion TablesPapers on reaction kinetics on ethylene oxide hydrationHistory of marketing listerinemonetization of contemporary art especially digital artThe folklore of the supernatural in the mediaArab nationalism at the beginning of the 20th century, and relations with the Ottoman Empire.Medicaid eligibility rules by state dataUsing GIS for studying health in neighborhoodsResearching Democratic party's views on charter schools and high-stakes testing.Health of pregnant women in prisonLocate the full list of proscribed authors under the Yuanyou ban 20thh-century China.Pediatric Tuberculosis Caregivers Attitudeshistory of Chinatown developments in the U.S./CanadaHow the Illiad reflects political realities in Bronze Age Greece.Economic effects Sykes-Picot agreement in Syria.The effect of the Nike settlement on labor relations in the footwear industry around the world.educational linguistics, semiotics, reflectivity and meta-languagetype 1 diabetes and telecommunicationsNative American slaveholders (specifically the Cherokee)Import substitution in Brazil under Vargashousehold interventions for asthma prevention in urban settingdata sets on toilets in MozambiqueThe changing Confucian-influenced perceptions among Song-dynasty writers of the Kitans.Familial lineages in Commedia del ArteWalter Mischel and original marshmallow, gratification studiesResearch on democratically elected Presidents of Brazil since the 1980s.Why Employment Non-Discrimination Act is taking so long to pass.Economic consequences of French colonialism in North Africa.Raster creation in arc gisPossible use of Shoah VHA in class about the Holocaust in Italy.Relation between U.S. intervention in Philippines and Cuban independence.glycogen storage disease and mortalityPutting together an annotated bibliography on the Cristero rebellion.Researching medieval pilgrimages to HebronLliterature by Christian missionaries in the Middle East during the late 19th and early 20th centuries.

EpilogueRaising the Organizational I.Q.

The O.I.Q. NOT about

Software, IT Systems, Network Plumbing, Assessment Plans, or Culture…

…it IS about creating an infrastructure for learning.

AnalyticsRepository

Services Analytics Organizational Knowledge

Understand how users work, what they want, need, and expect, and what’s expected of them

And What Is It We’re Learning to Do?

Guide the library through the morass of strategic choices…

Improve our response to change and the value of our services

Heighten the observational power of our organizations.

“…the key requirement for institutional success is to move from scalable efficiency to scalable learning.

Said differently, the rate of learning, innovation, and performance improvement within the institution must match (or exceed) that of the surrounding environment

if the institution is to survive (or thrive).”

-- John Hagel III and John Seely Brown The New Organization Model: Learning at Scale HBR Blog Network

Joe Zucca | zucca@upenn.eduPENN Libraries

With the support of

Producing Large Effects With Small Efforts

MetriDoc: https://github.com/upenn-libraries/metridoc-app