fossetcon 2014 - metrics are fun, but which ones really matter?

32
metrics are fun, but which ones really matter? james falkner liferay community manager @schtool 2014 fossetcon orlando, florida #fossetcon

Upload: schtool

Post on 08-Jul-2015

148 views

Category:

Software


0 download

DESCRIPTION

Slide deck from my September 2014 talk @ FOSSETCON in Orlando, FL. Abstract: There is no shortage of algorithms, techniques, and software to produce virtually any metric one may desire. Too often, projects concentrate on vanity metrics that may make the project look good, or the CEO happy, but hides the true nature of the community and how it is evolving. True value is only realized when metrics are chosen that most closely align with a project's goals. These slides were presented as part of a session whose goal was to: learn which goals and resulting metrics we chose to track and respond to in the Liferay project (a large, open source portal project), why they were (and continue to be) important, how they are produced, and what we are doing with them.

TRANSCRIPT

Page 1: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

metrics are fun, but which ones really matter?

james falkner liferay community manager

@schtool

2014 fossetcon orlando, florida #fossetcon

Page 2: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

James Falkner, Liferay Community Manager

Page 3: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

liferay community metrics

13 years 4.3M LOC

120k registered site members 102 external contributors in 2013

150 SI partners 300 marketplace apps

800 marketplace developers 43 user groups

114 translators for 43 languages 300k forum posts (~100/day)

25k forum participants 2 Top 50 github devs ( , )

#5 #45

Page 4: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

body count

Page 5: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

MOAR METRICS!!

Page 6: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

… and more

Page 7: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

allthingsd.com/20121217/andreessen-and-mixpanel-call-for-an-end-to-bullshit-metrics/

Page 8: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

“companies need to start using a new set of metrics that don’t simply make them feel good. They should use actionable metrics that provide insight, provide guidance, and help businesses make better decisions.”

- Suhail Doshi, mixpanel founder

Page 9: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

core contributions

Page 10: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

downloads

Page 11: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

step #1

“search your feelings”

Page 12: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

step #2 identify your

goals for metrics

Page 13: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

goal #1: increase the value of

participation

Page 14: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

www.youtube.com/watch?v=u6XAPnuFjJc

Page 15: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?
Page 16: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

goal #2: Measure the effect

of business decisions

Page 17: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

goal #3: understand the

relationship between metrics and business

performance

Page 18: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?
Page 19: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

step #3 decide what to measure

and actions to take

Page 20: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

basic 1st-order metrics for the liferay community

# of apps/reviews # of pull requests

# of commits/lines # of authors

# of bug reports # of forum posts/answers

# of downloads # open tickets / code reviews

Page 21: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

1st Order Metrics

WE DON’T REPORT THEM

Page 22: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

more interesting 2nd-order metrics

non-liferay vs. liferay time between contribution->codebase

location of community members evolution of contributor age over time

evolution of active/inactive users time of bug report -> fixed

time from forum q to a # ignored messages

# of ignored contributions distribution & evolution of commits across

functional areas

Page 23: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

metrics.forEach(function(metric) {

How accurate is it? How can it help achieve your goals? What to do when it turns good/bad? What is its relationship to others?

});

Page 24: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?
Page 25: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?
Page 26: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

Forums

Page 27: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

Demographics

Page 28: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

Community Contribution Performance

Page 29: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

future: cmty career evolution

Page 30: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

newbie: 0-5 posts, 0 answers, 1 download, 1 fork, …

junior: 20 posts, 10% engaged, >1 patch, 5 tweets, 1 app

senior: 50 posts, 30% engaged, 10 tweets, 5 apps, moderator, …

champion: 100 posts, 50% engaged, bugsquad, …

Page 31: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

summary identify values and goals

avoid focusing on vanity metrics decide actions ahead of time

assume inaccuracies experiment

Page 32: FOSSETCON 2014 - Metrics Are Fun, But Which Ones Really Matter?

thank you!

james falkner liferay community manager

@schtool

2014 fossetcon orlando, florida #fossetcon