netflix nebula - gradle summit 2014
DESCRIPTION
Netflix has open sourced many of our Gradle plugins under the name Nebula. These plugins are there to lend our expertise and experience to building responsible projects, internally and externally. This talk will cover some of the ones we've published, why we want to share these with the community, how we tested and published them, and most importantly how you can contribute back to them. Nebula started off as a set of strong opinions to make Gradle simple to use for our developers. But we quickly learned that we could use the same assumptions on our open source projects and on other Gradle plugins to make them easy to build, test and deploy. By standardizing plugin development, we've lowered the barrier to generating them, allowing us to keep our build modular and composable.TRANSCRIPT
@quidryan @quidryan http://www.slideshare.net/quidryan
Netflix Build Language
Justin Ryan <[email protected]>
@quidryan @quidryan http://www.slideshare.net/quidryan
Justin Ryan <[email protected]>
Why Netflix Uses Gradle
• Better Dependency Management story
• Flexible lifecycle
• Groovy
How Netflix Uses Gradle
• JVM Languages
• Resolution
• Code Quality
• Publishing
• Deployment Orchestration
How Netflix Sets up Gradle
• Patched Gradle
• Custom Distribution
• Custom Wrapper
apply plugin: ‘nebula’!apply plugin: ‘java’!!nebula {! readyForJava7 = true!}!!dependencies {! compile ‘netflix:platform:latest.release’!}
build.gradle
@NetflixOSS
github.com/Netflix
netflix.github.io
Unite two builds
• Model a responsible project
• Componentize via plugins
nebula-plugins
• Infrastructure
• Use Github
• Use CloudBees
• Use Bintray
• Mailing List
• nebula-plugin-plugin
Github Repositoriesnebula-* or gradle-*-plugin
Ensuring Github• https://github.com/nebula-plugins/ensure
• Ensure repository has
• Description
• Web Hooks
• Ensure “contrib" team has all repositories in it
• Ensure a contrib team exists for every repository
Continuous Integration
• Job DSL to create jobs
• Per branch
• Snapshot job
• Release job
• Lock job
CloudBees jobsRelease, snapshot, and pull request per branch
Job DSL
Bintray Packages
Ensuring Bintray• https://github.com/nebula-plugins/ensure
• Ensure every repository has a package
• Ensure package has the description as Github
• Ensure license is set to Apache 2.0
• Ensure labels are “gradle" and “nebula”
nebula-plugin-plugin
nebula-test• ProjectSpec
• PluginProjectSpec
• IntegrationSpec
• Thanks to Marcin and Luke
• Runs with Tooling API or GradleLauncher
class PluginExampleSpec extends PluginProjectSpec {! @Override! String getPluginName() { return 'plugin-example' }!! def ‘run task’() {! when:! project.plugins.apply(PluginExample)!! then:! def t = project.tasks.get(‘example’)!! when:! t.run()!! then:! new File(projectDir, ‘build/example.txt’).exists()! }!}
PluginProjectSpec
!def 'setup and run build'() {! buildFile << '''! apply plugin: 'java'! '''.stripIndent()!! when:! writeHelloWorld('nebula.hello')!! then:! fileExists('src/main/java/nebula/hello/HelloWorld.java')!! when:! def result = runTasksSuccessfully(‘build’, ‘-v’)!! then:! fileExists(‘build/classes/main/nebula/hello/HelloWorld.class')! result.wasExecuted(':compileTestJava')! def output = result.standardOutput! output.contains('Skipping task \’:compileTestJava\'')!}
IntegrationSpec
nebula-core• Collection of tasks
• Download
• Untar
• Unzip
• AlternativeArchiveTask
• CopySpecHelper
• GradleHelper
class GradleHelper {!! def beforeEvaluate(Closure beforeEvaluateClosure)!! def getTempDir(String taskBaseName)!! def addDefaultGroup(String defaultGroup)!}!!class CopySpecHelper {!! def visitCopySpec(CopySpecInternal copySpec, Closure closure)!! def findCopySpec(CopySpecInternal delegateCopySpec, closure)!}!!class ClassHelper {!! String findSpecificationVersion(Class clazz)!! Manifest findManifest(Class clazz)!! def findManifestValue(Class clazz, String key, defaultValue)!}
nebula-core helpers
nebula-publishing-plugin• Artifact plugins
• nebula-javadoc-jar
• nebula-source-jar
• nebula-test-jar
• Publishing plugins
• resolved-ivy
• resolved-maven
• nebula-sign
apply plugin: ‘nebula-maven-publishing‘!apply plugin: ‘nebula-source-jar'!apply plugin: ‘nebula-javadoc-jar'!apply plugin: ‘nebula-test-jar'!apply plugin: ‘nebula-sign'!apply plugin: 'java'
nebula-publishing.gradle
my-plugin-1.12.0-javadoc.jar!my-plugin-1.12.0-javadoc.jar.md5!my-plugin-1.12.0-javadoc.jar.sha1!my-plugin-1.12.0-javadoc.jar.asc!my-plugin-1.12.0-sources.jar!my-plugin-1.12.0-sources.jar.md5!my-plugin-1.12.0-sources.jar.sha1!my-plugin-1.12.0-sources.jar.asc!my-plugin-1.12.0-tests.jar!my-plugin-1.12.0-tests.jar.md5!my-plugin-1.12.0-tests.jar.sha1!my-plugin-1.12.0-tests.jar.asc!my-plugin-1.12.0.jar!my-plugin-1.12.0.jar.md5!my-plugin-1.12.0.jar.sha1!my-plugin-1.12.0.jar.asc!my-plugin-1.12.0.pom!my-plugin-1.12.0.pom.md5!my-plugin-1.12.0.pom.sha1!my-plugin-1.12.0.pom.asc
gradle-info-plugin• Collects meta data
• ‘info-java'
• ‘info-ci'
• 'info-scm'
• Reports in key/value pairs
• ‘info-jar'
• ‘info-props'
buildscript {!! repositories { jcenter() }!! dependencies { !! ! classpath 'com.netflix.nebula:gradle-info-plugin:1.12.+'!! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'!! }!}!apply plugin: 'info'
info.gradle
Manifest-Version=1.0!Implementation-Title=com.netflix.nebula#my-plugin;1.12.1-SNAPSHOT!Implementation-Version=1.12.1-SNAPSHOT!Built-Status=integration!Built-By=jryan!Build-Date=2014-06-10_13:30:44!Gradle-Version=1.12-20140608201532+0000!Module-Source=!Module-Origin=git@github.com:nebula-plugins/my-plugin.git!Change=976292c!Build-Host=localhost!Build-Job=LOCAL!Build-Number=LOCAL!Build-Id=LOCAL!Created-By=1.7.0_45-b18 (Oracle Corporation)[email protected][email protected]!X-Compile-Target-JDK=1.7!X-Compile-Source-JDK=1.7
info.gradle output
gradle-contacts-plugin• Express people involved in the project
• Make people and roles available to other plugins
• contacts-manifest
• contacts-pom
apply plugin: 'contacts'!contacts '[email protected]', ‘[email protected]'!!contacts {! '[email protected]'! '[email protected]' {! roles 'notify', 'owner'! }! '[email protected]' {! role 'techwriter'! }! '[email protected]'! role 'notify'! }!}!
contacts.gradle
gradle-scm-plugin• Attempt to provide SCM abstraction for other plugins
• E.g. gradle-dependency-lock-plugin and gradle-info-plugin
gradle-dependency-lock-plugin
• Developer declare their ideal situation
• Save resolved version
• If tests pass, commit to SCM
{! "com.github.townsfolk:gradle-release": { !! ! "locked": "1.2", "requested": "1.2" },! "com.jfrog.bintray.gradle:gradle-bintray-plugin": { !! ! "locked": "0.3", "requested": "0.3" },! "com.netflix.nebula:nebula-project-plugin": { !! ! "locked": "1.12.0", "requested": "1.12.+" },! "com.netflix.nebula:nebula-test": { !! ! "locked": "1.12.0", "requested": "1.12.+" },! "org.codehaus.groovy.modules.http-builder:http-builder": { !! ! "locked": "0.7.1", "requested": “latest.release" },! "org.jfrog.buildinfo:build-info-extractor-gradle": { !! ! "locked": "2.2.4", "requested": "2.2.+" }!}
apply plugin: ‘gradle-dependency-lock'!
lock.gradle
./gradlew generateLock
nebula-integtest-plugin• Sets up integTest source set
• Adds integTestCompile and integTestRuntime configurations
• Creates integrationTest task
nebula-project-plugin
• Pull together other plugins
• Responsible projects
nebula-plugin-plugin
• Used by plugins
• Strong opinions on how to publish
• Force nebula-project-plugin on projects
gradle-ospackage-plugin• Merging ubuntu-packager-plugin and gradle-rpm-
plugin
• Uses CopySpec definition
• Via just Java, generates RPMs and DEBs
ospackage {!! os = LINUX!! into '/opt/foo'!! from ('dist') {!! ! user 'builds'!! ! exclude '**/*.md'!! }!! postInstall file('scripts/postInstall.sh')!}!!buildRpm {!! requires('bar', '2.2', GREATER | EQUAL)!! from (‘build/metadata.properties’)!! link(‘/etc/init.d/foo’, '/opt/foo/bin/foo.sysv',)!}!!buildDeb {!! link('/etc/init/foo', '/opt/foo/bin/foo.upstart')!}
ospackage.gradle
gradle-override-plugin• Take command line arguments
• Intelligently apply in afterEvaluate
• E.g. -Nfindbugs.enabled=false
Internal Plugins
• netflix-repos
• nebula-ospackage
• nebula-grails
• nebula-findbugs, etc
• ivyimport
• nebula-fixexcludes
• nebula-intellij
buildscript {!! repositories { jcenter() }!! dependencies { !! ! classpath ‘com.netflix.nebula:nebula-plugin-plugin:1.12.+'!! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'!! }!}!!description ‘Example Plugin'!apply plugin: ‘nebula-plugin'!!contacts {! ‘[email protected]’ {! moniker 'Justin Ryan'! github 'quidryan'! }!}
plugin.gradle
curl -s get.gvmtool.net | bash!gvm install lazybones!cd ~/Projects/github/nebula-plugins!lazybones create nebula-plugin <name-of-project>!cd <name-of-project>!git init!git remote add origin [email protected]:nebula-plugins/<name-of-project>.git!./gradlew clean build!git add -A!git add -f gradle/wrapper/gradle-wrapper.jar!git commit -m "Initial template”!git push --set-upstream origin master
Making a plugin
https://github.com/nebula-plugins/nebula-plugins.github.io/wiki/New-Plugins
Getting it out the door • Let “ensure” run
• Run <name-of-project>-release
• Link package to jcenter
• Link package to gradle-plugins
Unopinionated Plugins
Opinionated Plugins
Gotchas
• NamedDomainObjectSet
• Debugging Tooling
• File as @Output
• afterEvaluate
Outstanding• CloudBees permissions
• Bot to create repositories
Participating
• Use individual plugins
• Get on nebula-plugins Google Group
• Move your plugin to nebula-plugins
• Start a new plugin in nebula-plugins
@quidryan @quidryan http://www.slideshare.net/quidryan
We’re hiring
Justin Ryan <[email protected]>
HOUSE of GRADLE