zen and-the-art-of-build-script-maintenance

ZENand the art of

Build Script Maintenance

An Inquiry into Build Automation Quality

Tuesday, 22 June 2010

John Ferguson SmartWakaleo Consulting

Web: http://www.wakaleo.comEmail: [email protected]: wakaleo


http://www.wakaleo.com/blog


mailto:[email protected]


AgendaWhat are we discussing today?What makes a good build script?

Smelly build scripts

Choosing your tools

Maven tips

Ant tips


IntroductionQuality build scripts - why botherMaintenance costs

Learning curve

Turn-over

Portability

Automation


Build quality - quality buildsWhat makes a good build script?Gold Standard

Portable

Reproducible

Standard

Maintainable


Build quality - quality buildsGold StandardReference build process

Reference binaries


Build quality - quality buildsPortableRuns anywhere

Runs on any OS

No local dependencies

Environment-specific configurations

Specially-installed software or databases

...


Build quality - quality buildsReproducible“Play it again, Sam”


Build quality - quality buildsStandardKnowing what to expect


Build quality - quality buildsMaintainableThink of the next dude


Smelly buildsSo what makes a poor build script?

1) The hard coded build

2) The OS-specific build

3) The IDE-only build

4) The Magic Machine build

5) The Oral Tradition build

6) The Nested build

7) The Messy Dependencies build

8) The DYI build

9) The untrustworthy build

10) The slow build


Smelly buildsThe hard coded build

C:/bea/weblogic-9.1/..

Paths

http://testserver.acme.com:7001

URLs

<svn username="scott" password="tiger"...>Passwords

<property name="dir.jboss" value="${env.JBOSS_HOME}"/>

Environment variables


http://testserver:7001























































































<target name="checkstyle">

<delete dir="./reports" quiet="true" /> <mkdir dir="./reports" /> <checkstyle config="./docs/sun_checks.xml"> <formatter type="xml" tofile="./reports/checkstyle.xml"/> <fileset dir="./src" includes="**/*.java"/> </checkstyle>

<style in="./reports/checkstyle.xml" out="./reports/checkstyle.html" style="checkstyle.xsl"/>

</target>

Hard-coded directories











































































































<target name="checkstyle">

<delete dir="./reports" quiet="true" /> <mkdir dir="./reports" /> <checkstyle config="./docs/sun_checks.xml"> <formatter type="xml" tofile="./reports/checkstyle.xml"/> <fileset dir="./src" includes="**/*.java"/> </checkstyle>

<style in="./reports/checkstyle.xml" out="./reports/checkstyle.html" style="checkstyle.xsl"/>

</target>

<property name=”reports.checkstyle.dir” value=”${basedir}/reports”/>

<target name="checkstyle"> <delete dir="${reports.checkstyle.dir}" quiet="true" /> <mkdir dir="${reports.checkstyle.dir}" /> <checkstyle config="./docs/sun_checks.xml"> <formatter type="xml" tofile="${reports.checkstyle.dir}/checkstyle.xml"/> <fileset dir="./src" includes="**/*.java"/> </checkstyle>

<style in="${reports.checkstyle.dir}/checkstyle.xml" out="${reports.checkstyle.dir}/checkstyle.html" style="checkstyle.xsl"/>

</target>

Project-relative directory

DRY





























































































































<target name="war" > <war destfile="c:/tomcat/jakarta-tomcat-5.0.19/webapps/app.war" webxml="${src}/app.xml" basedir="${bin}" /></target>

Hard-coded directories




































<property name="wardir" location="c:/tomcat/jakarta-tomcat-5.0.19/webapps"/>

<target name="war" > <war destfile="${wardir}" webxml="${src}/app.xml" basedir="${bin}" /></target>


Still hard-coded













































<svn username="scott" password="tiger"> <checkout url="http://subversion.acme.com/myapp/trunk" destPath="${subproject.dir}" /></svn>

Hard-coded username/password


http://subversion.acme.com/someproject/trunk











<property environment="env"/><property name="dir.jboss" value="${env.JBOSS_HOME}"/>

Environment variable


Smelly buildsThe OS-specific build

<exec command="grep \"@\" ${build.dir} | wc -l" outputproperty="token.count"/>


Smelly buildsThe OS-specific build

...CALL PAUSE.CMD...

build.cmd

...:: Check for a non-existent IP address:: Note: this causes a small extra delay!IF NOT DEFINED NonExist SET NonExist=10.255.255.254PING %NonExist% -n 1 -w 100 2>NUL | FIND "TTL=" >NUL...

pause.cmd


Smelly buildsThe IDE-only build


Smelly buildsThe Magic Machine build

Directories

App servers

Databases

Configuration files


Installed software or tools


Smelly buildsThe Magic Machine build

Directories

App servers

Databases

Configuration files


Installed software or tools

<proprerty weblogic.dir="/u01/app/bea/weblogic-9.1"/>


Smelly buildsThe Oral Tradition build


Smelly buildsThe Nested Build

#! /bin/shANT_HOME=/u01/app/tools/ant-1.7.1...$ANT_HOME/ant $1project/tools/ant.sh


Smelly buildsThe Nested Build

<target name="build-subproject"> <svn username="scott" password="tiger"> <checkout url="http://subversion.acme.com/someproject/trunk" destPath="${subproject.dir}" /> </svn> <ant dir="${subproject.dir}" target="build-all" /></target>

build.xml










































The Messy Dependencies buildJAR files in the SCM

Unversioned JAR files

Unclear dependencies

Smelly builds


The DYI build“Not invented here”

DYI dependencies

DYI deployments

DYI Maven releases

...

Smelly builds


Smelly buildsThe untrustworthy build<junit fork="yes" haltonfailure="false" dir="${basedir}"> <classpath refid="test.class.path" /> <classpath refid="project.class.path"/> <formatter type="plain" usefile="true" /> <formatter type="xml" usefile="true" /> <batchtest fork="yes" todir="${logs.junit.dir}"> <fileset dir="${test.unit.dir}"> <patternset refid="test.sources.pattern"/> </fileset> </batchtest></junit>




















































































Smelly buildsThe slow build


Choosing your toolsFlexibility verses ConventionWhat’s better: flexibility or standards?

It depends what you’re doing...


Choosing your toolsStandards and Conventions

No standards

Make up your own standards

Support standards

Encourage/enforce standards

Standards and ConventionsAd-hoc scripting

Eas

y to

rea

dH

ard

to

rea

d

23


Choosing your toolsFlexibility and expressiveness

Eas

y to

rea

dH

ard

to

rea

d

Easy to do whatever you wantMakes you stick to conventions

3

Encourage/enforce standards

Do whatever you want

2


Choosing your toolsFlexibility verses ConventionBuild Scripting Rule 1

“A build script will tend to reflect the personality of it’s developer”


Choosing your toolsFlexibility verses ConventionBuild Scripting Rule 2

“The more flexible a build script, the more likely it is to become unmaintainable”


Choosing your toolsFlexibility verses ConventionFlexibility is great for some jobs:

Ad-hoc tasks

Some deployment tasks

“Out-of-the-box” stuff


Choosing your toolsFlexibility verses ConventionBut convention pay off in lower maintenance costs


Ant tipsBetter Ant scriptsConsistent conventions

Declare your dependencies

Make it readable

Tidy up your mess

Avoid long scripts


Ant tipsBe consistent

Standardize target names

Document your public targets


Ant tipsDeclare your dependencies

Use an Enterprise Repository Manager

Several tool choices:

Maven Ant Tasks

Ivy


Ant tipsUsing the Maven Ant Tasks

Declare dependencies

Deploy to a Maven Enterprise Repository

<artifact:dependencies pathId="dependency.classpath"> <dependency groupId="junit" artifactId="junit" version="3.8.2" scope="test"/> <dependency groupId="javax.servlet" artifactId="servlet-api" version="2.4" scope="provided"/></artifact:dependencies>


Ant tipsMake it readable

Write a build script like your source code...

Avoid long targets

Avoid long build scripts

Use descriptive target names


Ant tipsTidy up your mess

Always define a ‘clean’ target.


Ant tipsMove to Maven ;-)


Maven tipsBetter Maven scriptsSimple

Portable

Reproducible

Clean

Automated


Maven tipsKeep it simpleUse modules

Use an organization-level POM


Maven tipsKeep it portableNo hard-coding

Define sensible defaults for properties and profiles

Avoid resource filtering for production code


Maven tipsKeep it reproducibleAvoid external snapshots

Specify plugin versions

Use consistent environments


Maven tipsConsistent environmentsEnforcing a minimum Maven version

<?xml version="1.0"?><project...> <modelVersion>4.0.0</modelVersion> <groupId>com.ciwithhudson.gameoflife</groupId> <artifactId>gameoflife</artifactId> <version>0.0.1-SNAPSHOT</version> <name>gameoflife</name> <prerequisites> <maven>2.2.1</maven> </prerequisites>

Minimum Maven version


Maven tipsConsistent environmentsUse the same version of Maven

Use a “standard” Maven installation across the organization

Use a global settings.xml file

Store a copy in SCM

Enforce a minimum Maven version in your projects


Maven tipsEnforcing consistency with the enforcer pluginMaven version

JDK version

Snapshots

Plugin versions

OS

...


Maven tipsEnforce the Maven version

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>1.0-beta-1</version> <executions> <execution> <id>enforce-maven-version</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <requireMavenVersion> <version>2.2.1</version> </requireMavenVersion> </rules> </configuration> </execution> </executions> </plugin>

Minimum Maven version


Maven tipsEnforce the JDK versionAll developers should be using the same JDKs

Incompatible bytecode

Different XML parsers

Different Maven behaviour


Enforce the JDK version

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>1.0-beta-1</version> <executions> <execution> <id>enforce-jdk-version</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <requireJavaVersion> <version>[1.5.0,1.6.0)</version> </requireJavaVersion> </rules> </configuration> </execution> </executions> </plugin>

Maven tips

Authorized JDK versions


Maven tipsSpecify your plugin versionsUndeclared version numbers are bad

Inconsistent builds across different machines

Non-repeatable builds

Plugin changes can break the build

Don’t use SNAPSHOT plugins either


Specify your plugin versions

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>1.0-beta-1</version> <executions> <execution> <id>enforce-versions</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <requirePluginVersions/> </rules> </configuration> </execution> </executions> </plugin>

Maven tips

Plugin versions must be defined


Maven tipsKeep it cleanKeep tabs on your dependencies:

What dependencies are you actually using?

What dependencies do you really need?


Maven tipsDependency list What dependencies are you actually using?$ mvn dependency:list[INFO] Scanning for projects...[INFO] Searching repository for plugin with prefix: 'dependency'.[INFO] ------------------------------------------------------------------------[INFO] Building babble-core[INFO] task-segment: [dependency:list][INFO] ------------------------------------------------------------------------[INFO] [dependency:list][INFO] [INFO] The following files have been resolved:[INFO] antlr:antlr:jar:2.7.6:compile...[INFO] commons-collections:commons-collections:jar:2.1.1:compile[INFO] commons-logging:commons-logging:jar:1.0.4:compile[INFO] dom4j:dom4j:jar:1.6.1:compile[INFO] javax.persistence:persistence-api:jar:1.0:compile[INFO] javax.transaction:jta:jar:1.0.1B:compile[INFO] junit:junit:jar:4.5:test[INFO] net.sf.ehcache:ehcache:jar:1.2:compile[INFO] org.hamcrest:hamcrest-all:jar:1.1:compile[INFO] org.hibernate:hibernate:jar:3.2.0.ga:compile[INFO] org.hibernate:hibernate-annotations:jar:3.2.0.ga:compile[INFO] [INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESSFUL[INFO] ------------------------------------------------------------------------

mvn dependency:list


Maven tipsDependency tree Where do they come from?$ mvn dependency:tree[INFO] Scanning for projects...[INFO] Searching repository for plugin with prefix: 'dependency'.[INFO] ------------------------------------------------------------------------[INFO] Building babble-core[INFO] task-segment: [dependency:tree][INFO] ------------------------------------------------------------------------[INFO] [dependency:tree][INFO] com.sonatype.training:babble-core:jar:1.0-SNAPSHOT[INFO] +- org.hibernate:hibernate:jar:3.2.0.ga:compile[INFO] | +- net.sf.ehcache:ehcache:jar:1.2:compile[INFO] | +- javax.transaction:jta:jar:1.0.1B:compile[INFO] | +- commons-logging:commons-logging:jar:1.0.4:compile[INFO] | +- asm:asm-attrs:jar:1.5.3:compile[INFO] | +- dom4j:dom4j:jar:1.6.1:compile[INFO] | +- antlr:antlr:jar:2.7.6:compile[INFO] | +- cglib:cglib:jar:2.1_3:compile[INFO] | +- asm:asm:jar:1.5.3:compile[INFO] | \- commons-collections:commons-collections:jar:2.1.1:compile[INFO] +- org.hibernate:hibernate-annotations:jar:3.2.0.ga:compile[INFO] | \- javax.persistence:persistence-api:jar:1.0:compile[INFO] +- junit:junit:jar:4.5:test[INFO] \- org.hamcrest:hamcrest-all:jar:1.1:compile[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESSFUL[INFO] ------------------------------------------------------------------------

mvn dependency:tree


Maven tipsDependencies in EclipseEclipse has an equivalent screen


Maven tipsDependency analyse What dependencies do you really need?$ mvn dependency:analyze[INFO] Scanning for projects...[INFO] Searching repository for plugin with prefix: 'dependency'.[INFO] ------------------------------------------------------------------------[INFO] Building babble-core[INFO] task-segment: [dependency:analyze][INFO] ------------------------------------------------------------------------[INFO] Preparing dependency:analyze...[INFO] [dependency:analyze][WARNING] Used undeclared dependencies found:[WARNING] javax.persistence:persistence-api:jar:1.0:compile[WARNING] Unused declared dependencies found:[WARNING] org.hibernate:hibernate-annotations:jar:3.2.0.ga:compile[WARNING] org.hibernate:hibernate:jar:3.2.0.ga:compile[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESSFUL[INFO] ------------------------------------------------------------------------

mvn dependency:analyse

Used but not declared

Declared but not used


Maven tipsExcluding dependenciesWhat if you don’t want a dependency? <dependencies> <dependency> <groupId>org.springframework</groupId> <artifactId>spring</artifactId> <version>2.5.5</version> <exclusions> <exclusion> <groupId>javax.jms</groupId> <artifactId>jms<artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.geronimo.specs</groupId> <artifactId>geronimo-jms_1.1_spec</artifact> <version>1.1</version> </dependency> <dependencies>

Don’t include JMS

























































































































Maven tipsStandardizing versionsUse dependencyManagement for consistency

<dependencyManagement> <dependencies> <dependency> ! <groupId>mysql</groupId> ! <artifactId>mysql-connector-java</artifactId> ! <version>5.1.6</version> </dependency> <dependency> ! <groupId>postgres</groupId> ! <artifactId>postgres</artifactId> ! <version>7.3.2</version> </dependency> </dependencies></dependencyManagement>

<dependencies> <dependency> !<groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> </dependency></dependencies>

Parent pom

Child pom
































































































































Maven tipsKeep it automatedPlan your release strategy

Use a Repository Manager

Automatic snapshot deployments

Automated releases


Maven tipsMaven best practices for CI buildsUse batch mode (-B)

Always check or snapshot updates (-U)

Use a repository per project

Print test failures to stdout (-Dsurefire.useFile=false)


Maven tipsKnow when to script itGroovy or Ant scripting is easy in Maven

Call external scripts when appropriate


Maven tipsKnow when to script itIt’s pretty easy in Maven 2...<project> <build> <plugins> <plugin> <groupId>org.codehaus.groovy.maven</groupId> <artifactId>gmaven-plugin</artifactId> <version>1.0-rc-5</version> <executions> <execution> <phase>compile</phase> <goals> <goal>execute</goal> </goals> <configuration> <source> println "Hi there I’m compiling ${project.name}" </source> </configuration> </execution> </executions> </plugin> </plugins> </build> ...


























































































































Maven tipsKnow when to script itIt’s even easier in Maven 3...project { build { $execute(id: 'compilation-script', phase: 'compile') { println "Hi there I’m compiling ${project.name}" } $execute(id: 'validation-script', phase: 'validate') { println "Hi there I’m validating ${project.name}" } ... }}








































































Thank You

John Ferguson SmartWakaleo Consulting

Web: http://www.wakaleo.comEmail: [email protected]: wakaleo






zen and-the-art-of-build-script-maintenance

Technology