fossetcon 2014 - metrics are fun, but which ones really matter?
DESCRIPTION
Slide deck from my September 2014 talk @ FOSSETCON in Orlando, FL. Abstract: There is no shortage of algorithms, techniques, and software to produce virtually any metric one may desire. Too often, projects concentrate on vanity metrics that may make the project look good, or the CEO happy, but hides the true nature of the community and how it is evolving. True value is only realized when metrics are chosen that most closely align with a project's goals. These slides were presented as part of a session whose goal was to: learn which goals and resulting metrics we chose to track and respond to in the Liferay project (a large, open source portal project), why they were (and continue to be) important, how they are produced, and what we are doing with them.TRANSCRIPT
metrics are fun, but which ones really matter?
james falkner liferay community manager
@schtool
2014 fossetcon orlando, florida #fossetcon
James Falkner, Liferay Community Manager
liferay community metrics
13 years 4.3M LOC
120k registered site members 102 external contributors in 2013
150 SI partners 300 marketplace apps
800 marketplace developers 43 user groups
114 translators for 43 languages 300k forum posts (~100/day)
25k forum participants 2 Top 50 github devs ( , )
#5 #45
body count
MOAR METRICS!!
… and more
allthingsd.com/20121217/andreessen-and-mixpanel-call-for-an-end-to-bullshit-metrics/
“companies need to start using a new set of metrics that don’t simply make them feel good. They should use actionable metrics that provide insight, provide guidance, and help businesses make better decisions.”
- Suhail Doshi, mixpanel founder
core contributions
downloads
step #1
“search your feelings”
step #2 identify your
goals for metrics
goal #1: increase the value of
participation
www.youtube.com/watch?v=u6XAPnuFjJc
goal #2: Measure the effect
of business decisions
goal #3: understand the
relationship between metrics and business
performance
step #3 decide what to measure
and actions to take
basic 1st-order metrics for the liferay community
# of apps/reviews # of pull requests
# of commits/lines # of authors
# of bug reports # of forum posts/answers
# of downloads # open tickets / code reviews
1st Order Metrics
WE DON’T REPORT THEM
more interesting 2nd-order metrics
non-liferay vs. liferay time between contribution->codebase
location of community members evolution of contributor age over time
evolution of active/inactive users time of bug report -> fixed
time from forum q to a # ignored messages
# of ignored contributions distribution & evolution of commits across
functional areas
metrics.forEach(function(metric) {
How accurate is it? How can it help achieve your goals? What to do when it turns good/bad? What is its relationship to others?
});
Forums
Demographics
Community Contribution Performance
future: cmty career evolution
newbie: 0-5 posts, 0 answers, 1 download, 1 fork, …
junior: 20 posts, 10% engaged, >1 patch, 5 tweets, 1 app
senior: 50 posts, 30% engaged, 10 tweets, 5 apps, moderator, …
champion: 100 posts, 50% engaged, bugsquad, …
summary identify values and goals
avoid focusing on vanity metrics decide actions ahead of time
assume inaccuracies experiment
thank you!
james falkner liferay community manager
@schtool
2014 fossetcon orlando, florida #fossetcon