continuous integration for open source distros v 3.0
DESCRIPTION
These are the slides of my talk at Foss.IN 2012 on Continuous Integration for Open Source Distribution ecosystems. I have covered how existing Continuous Integration efforts can be improved to ensure greate collaboration between open source projects, developers and distributions to make earlier rTRANSCRIPT
Continuous Integration
For
Open Source Distributions
Sriram Narayanan
@sriramnrn
About me
Belenix Tester + Developer
Thoughtworker
Why we work on distros
● Innovate● more versatile installation● faster boot up times● experimental drivers
● Learn package management (Rolling updates!)
● Learn how a distro works (LFS,BLFS + more)
Pain points for a Distro Maintainer
● Repeat Tasks
● Dependency Tracking and Management
● Manual Testing
Pain points for a Distro Maintainer
● Repeat Tasks● Track upstream changes and fixes● Regular Distro builds● Changelog and Release Notes● Documentation Updates
● Dependency Tracking and Management
● Manual Testing
Pain points for a Distro Maintainer
● Repeat Tasks
● Dependency Tracking and Management● Dependencies Expressed by not Tested● Insufficient Integration Tests● Heavily dependent on manual testing
● Manual Testing
Pain points for a Distro Maintainer
● Repeat Tasks● Dependency Tracking and Management● Manual Tests
● Package Integration● Installation● Desktop Application Functionality● Service Functionality
● Testing at too many levels● Not just exploratory (More exploratory is good)
Continuous Integration - Basics
● “Integrate changes continuously”● Checkout source, make, make test, package● Usual Implementation
● Automated Agents that perform tasks● Farm of Agents depending upon scale and budget
– Great for packaging multiple apps in parallel– Parallel Tests
Continuous Integration - Benefits
● Detect issues earlier● Package software earlier● Automate various tasks
● Packaging● Testing● Integration● Generate ChangeLog + Release Notes skeleton● Pending Documentation Hints to Documentation
Team● Build the ISO!
Continuous Integration Today
● Isolated build environments● Based on chroot jails● Used by Fedora, Debian, Suse, et al
● Continuous Integration servers● Jenkins/Hudson● Koji● Others (distro specific)
Continuous Integration – What's missing
● Dependency awareness● A “Green” upstream should trigger downstream
builds● Complex Dependency graph builds and tests● Prevents Automated Consolidation Releases
● A Variety of Testing● “make test” is not enough!
Continuous Integration – What's missing
● Dependency Awareness
apr
httpd
subversion
Image Credits: http://cd-with-thoughtworks-go.blogspot.in/
Why “make test” is not enough
● Tests the app only on the compiled system● No tests post installation
● Packaging occurs after make test● Packaging errors are not detected
– Incorrect placement, ownership and permissions– Conflicting requirements– Missing files
● Integration Tests needed
./configure makemake make test package
Testing
Unit
Service Layer(Integration)
UserInterface
The Test Pyramid – Mike Cohn
http://martinfowler.com/bliki/TestPyramid.html
About Unit Tests
● Unit Tests● Requirements in executable form● Detect what exact functionality has broken
About Integration Tests
● Service Level Tests● Is ComponentA correctly calling ComponentB?● Is ComponentA fulfilling its public API contract?
About UI Level Tests
● Test End to End Functionality● e.g. Sahi based tests
What Tests tell you
● UI Tests - “Something in this feature broke”
● Integration Tests - “Component B returned an unexpected response to Component A”
● Unit Tests - “When we introduce functionality requirement #13 into Component B, then our code causes requirements #7 and #9 to break”.
Post – Installation Integration Tests
● CentOS package level tests● A good start!
● Some useful tests for distro maintainers– Is httpd able to serve files ? (selinux issues!)– Is SSH accepting connections ?– Did the network subsystem initialize correctly?– Did the installer set the MBR + active partition correctly?– Is the DNS server accepting queries ?
● Is the application behaving as required?
CI in Belenix
● Distro tests + Consolidation level smoke testskernel
httpd
postgres
ssh
WebStack
nginx
python
httpd
vi Userland
shell
awkIntegration
Tests
kernel
AutomatedInstall
Post InstallTests
Distro + package levelsmoke tests
Roadmap - CI and Belenix
● Package early● Consolidations – kernel, userland, subsystems
(storage, firewall, load balancer)● Custom distros – CoreOS, storage, firewall,
load balancer● Each with its own functional test suite!
● CI for open source projects that we bundle● Potential inter-distro inter-project collaboration