netflix oss meetup season 4 episode 4

Netflix Open SourceSeason 4, Episode 4

Introduction

Why does Netflix Open Source?

Improve Engineering● Great feedback from wider community● Collaborate through open code

Recruit new and retain engineering talent● Hard problems are openly worked on

Industry Alignment

Why does Netflix Open Source?

Netflix movesto cloud

2008

2013

2016

Netflix Github -http://netflix.github.io

Netflix’s approach to open source

Form a small cross-functional team working group that centralizes OSS competence, assisting decentralized teams working with OSS spend less time focusing on the administrative aspects (legal, tooling, branding, monitoring, and community promotion).

Open source enabler - OSS Interest Group

● Internal mailing list● Meets once per month● Topics from developers● Help each other with

common problems

Agenda

Assisting open source at Netflix● Github management and security● Build, CI and release tools

Engaging in open source● Transparency & OSS maturity● Fostering communities

@rusmeshenberg

@SonOfGarr

@rfletcherEW

mikegrima

Agenda

Open Source Offices and the TODO Group

Nithya A. RuffDirector, Open Source Office, Western Digital

@nithyaruff

Github Management & SecurityMike Grima

Security Tools - Sensitive Data Leakage

● We scan source code for○ Access keys, passwords, tokens,

hostnames

● We scan code automatically and frequently

Scumblr

Security Tools - User Management

● Provide tools and automation for user access○ Adding / Removing users○ Performing privileged tasks

● We aim for self-service as much as possible!

Github Organizational Management

● Management must be easy○ Otherwise, teams will go it alone (BAD)

● Less is more: fewer orgs = Good

Github Organizational ManagementBYOGID:● User links to internal ID● All tools then can

associate identity

Two Factor Auth Enforcement● Automation to boot users who don’t● Be careful - education on recovery!

Github Organizational Management

● Owners○ Limited group - due to power○ Broker owner actions via ChatOps

● Netflixer group○ Full write permissions on all repos

● Outside collaborators○ Added by netflixers, validated over time

ChatOps for GitHub Management

More advanced commands & DUO!

Building Netflix OSSMike McGarr

OSS builds needs...

● Consistent build automation● Continuous integration● Release software versions● Publish to JCenter and Maven central● Validate license files● Simple and consistent

http://nebula-plugins.github.io

build.gradle file

● Add contacts to build.gradle● Bundle build metadata into .jar ● Publish .jar files to Bintray● Git tag to build release versions● Optional: lock dependencies

Reduce boilerplate

● Jenkins on Cloudbees● Setup builds with the Job DSL plugin

Continuous integration

● What we didn’t like?○ Lacked declarative builds○ Config not in source○ Complex setup○ Not as common in OSS community

Travis CI

All commits to master./gradlew -Prelease.travisci=true build snapshot

Releases w/ Nebula + TravisCI

Every pull request./gradlew build

Commit tagged with vX.Y.Z-rc#./gradlew -Prelease.travisci=true candidate

Commit tagged with vX.Y.Z./gradlew -Prelease.travisci=true final

https://jcenter.bintray.com

Distributing the binaries

What happens when...

Backup source Netflix OSS

● Backup Github repos to internal Git

git remote add internal \ ssh://[email protected]/foo/bar.git

Transparency and OSS MaturityRuslan Meshenberg@rusmeshenberg

4 seasons of NetflixOSS

Many OSS Projects

In 4 years

All wildly successful?

Some - yes

Some… not so much

What are some of the challenges?

Lack of OSS transparency / direction


Internal / OSS Divergence


Maturing and EOL of projects


Separating ideas from code


All leading to variable levels of support

What are we doing about it?

Data to the rescue!

Org Health TrackingProject Health TrackingBacklog of PRs and Issues

Overall Org Health Tracking

Metrics we track

● Issues○ open, closed, TTC

● Pull Requests○ open, closed, TTC

● Last commit timing● Stars/forks● Num contributors

Project Health Tracking

github.com/Netflix/

OSSTracker

Transparency about project lifecycle

OSSMETADATA file:

● Active● Maintenance● Archived

Project Ownership

All projects have● Development lead, Management lead● Shepherd from OSS function areaOnly projects with active leads stay active!

Transparency about project evolution

Converging internal and OSS

Less maintenance for us

You get exactly what we use

Fostering communitiesRob Fletcher

What’s in it for us?Tangible contributions

● Leverage enhancements made for other cloud providers

● Titus integration made easier● Role-based authentication

Intangible benefits

● Influencing the conversation● Validation of concept & implementation● Recruitment● Retention

Spinnaker contributions — Clouddriver

Spinnaker contributions — Orca

Encouraging engagementContributors

● Public roadmap● “no” > ignoring people● We can’t do everything — encourage

contributions ● Review community PRs & issues regularly● Make the 1st step easy● Don’t let docs mislead

Open tools

● GitHub● Slack● Readme.io● Travis CI

You gotta do it every day…that’s the hard part

TODO GroupNithya A. Ruff

• Many of us who ran open source program offices shared a private mailing list to commiserate…

• It was an avenue to discuss issues in private and even find ways to collaborate on open source projects…

• Focused on Silicon Valley companies initially

• In 2014 we had an idea of scaling and opening up the the community more…

• Announced the TODO Group @Scale 2014 conference!

• TODO Group is a group of companies who want to– collaborate on best practices on running open source program

offices– share open source policies and training material– codify quality criteria for well-run open source projects and

communities– build and share tools to maintain those quality standards

• As we scaled our open source programs, we realized we all built similar tools for the purposes of corporate scale open source…

• What is corporate scale open source?

• Corporate participants in open source have a number of unique concerns:– scale (i.e., Google and Microsoft have many open source projects)– insights– cultural– legal / governance

• Companies doing open source generally want to be good community citizens, to be open and inclusive, to operate meritocracies. They also need to run a business and be aware of responsibilities to their employees, shareholders and the broader community.

• To establish the TODO Group as a legitimate legal entity, we partnered with the Linux Foundation to make the TODO Group an official collaborative project!

• The LF helps with legal paperwork, running events and gives the TODO Group access to its 650+ members

Western Digital and Open Source

• Started the Open Source office at SanDisk to engage with the community and go past consumption and compliance

• Branded it Open @ SanDisk and became a visible supporter of events and communities

• Increased contribution and competency inside the company around open source development models

• With the acquisition by Western Digital, created a single office across WD, HGST and SanDisk

Demo Stations

Conductor(workflow)

Questions?

netflix oss meetup season 4 episode 4

Technology