new devops tools for native code · new devops tools for native code linuxcon north america...
TRANSCRIPT
![Page 1: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/1.jpg)
C. Thomas Stoverwww.thomasstover.com
New DevOps Tools for Native Code
LINUXCON North AmericaWednesday, September 18th, 2013Hyatt HotelNew Orleans, Louisiana11:50 CST
![Page 2: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/2.jpg)
Revision Control Source Configuration
Collaboration
Packaging &Deployment
Build Automation
Continuous Integration
Test Analysis
Project Management
Some DevOps Interest AreasSome DevOps Interest Areas
(Scope of BCA and Build Matrix)
![Page 3: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/3.jpg)
Build Configuration Adjust (BCA)
•Web Site: bca.stoverenterprises.com•Git Hub: ctstover/Build-Configuration-Adjust•Google Groups Mailing List: bca-discus•License: GPLv3
NONE.NONE.PROJECT_NAME = "Hello World "BINARY.MAIN.NAME = helloBINARY.MAIN.FILES = ./src/hello.c
![Page 4: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/4.jpg)
$ ./configureconfigure: running ./bca --configureconfigure: creating directories for build host NATIVE...configure: directory . already exists.configure: directory ./obj already exists.configure: generating Makefile.bcaconfigure: Next use GNU Make to process Makefile.bca, ie "make -f Makefile.bca" or "gmake -f Makefile.bca".$ make -f Makefile.bcagcc -c hello.c -o ./obj/MAIN-hello.ogcc ./obj/MAIN-hello.o -o ./native/hello$ ./native/hellohello world
![Page 5: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/5.jpg)
Build Tool Roles
•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)
![Page 6: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/6.jpg)
Build Tool Roles
•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)
![Page 7: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/7.jpg)
BCA Design Ideas
● “normal” build logic: ./configure; make; make install● “normal” command line usage: --disable-*, --enable-*, --without-*, --host=, --prefix=, etc.●installing additional programs not required●human readable Makefiles●human readable project files●human readable build configuration files●Full command line / scriptable, and text file interfaces●Always a simple manual override procedure●minimal to no dependencies: C99-ish●Configure time, and Build time performance really do matter●just abandon non-*nix style build hosts●ground up incorporation of pkg-config●build visualization (human readable graphviz dot output) helps.
![Page 8: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/8.jpg)
Unix Wars -> Portability in 2013
•Many of the detection tests in Autoconf are: valuable, esoteric, complicated, not worth replacing, not worth discarding, etc.
•Really, package configure (pkg-confg), is a good idea, really.
“Solution” - platform details checking now means either 1) pkg-config, 2) slightly modified autoconf script, 3) any other external script you want to make.
![Page 9: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/9.jpg)
Big deal:
pkg-config files (.pc) are used for all component interrelationships, both external and internal.
(btw Hacking a .pc to force something to work is infinitely superior to forcing LibTool to do anything.)
![Page 10: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/10.jpg)
External dependency
Internal dependency
![Page 11: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/11.jpg)
Packaging vs. Source Installations
•A practical, non-hateful, binary package support abstraction for all meaningful platforms is simply not possible. Really. No seriously.
•You know what? That doesn't matter.
•Source installations are the original portable distribution format.
•Source builds/installations that ACTUALLY WORK, perpetuate maintained, quality in-distribution packages. (and happy users)
![Page 12: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/12.jpg)
Optional GUI may increase source installation useability.
![Page 13: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/13.jpg)
Fighting Recursive Makefiles
•Miller, P.A. (1998), Recursive Make Considered Harmful•Console output utterly disgusting and confusing•Parallel build configurations can be accommodated by a single monolithic Makefile.•Use make -j X to full potential
$ bca --generate-graphviz$ dot -Tpng -o bcaproject.png bcaproject.dot
![Page 14: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/14.jpg)
New Work flow Model
Run a greater and greater number ofbuild configurations, before handingoff to continuous integration.
![Page 15: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/15.jpg)
New Work flow Model
Corollary: Let the build system - notcontinuous integration - driveparametrized builds.
![Page 16: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/16.jpg)
Build Matrix•Web Site: buildmatrix.stoverenterprises.com•Git Hub: ctstover/Build-Matrix•Google Groups Mailing List: build-matrix-disc
![Page 17: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/17.jpg)
Redundant Facilities in Continuous Integration (CI)
●SSH●Cron●Bash / Scripting Languages●User Access Paradigms●GNU Screen (slightly)●mail●Linux / Unix in general●build system logic (slightly)●version control logic (slightly)
![Page 18: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/18.jpg)
What are the missing pieces?
•A Scoreboard•A Test History•A Build History
![Page 19: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/19.jpg)
Build Matrix Design Ideas
● ssh (or fork()/execv() ) is the only relevant transport for this●use system users ●SQLite + directory tree for big files as backing store●distributed: peer to peer push + pull + list operations●web interfaces, GUI tools, test analytics can each get there own full copy to play with / mine●again just C99 ●very low overhead●no persistent daemon●just a “dumping ground” for build output and test results●minimum scope possible
![Page 20: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/20.jpg)
Leaving Plantation J/X-Unit
suite: stringstest: escapenull: passedtest: escapeempty: passedtest: escapenotneeded: passedtest: escapeqoutes: passedtest: escapespaces: passedtest: escapemixed: passedtest: containsnullsource: passedtest: containsnullsearch: passedtest: containsnot: passedtest: containstring: passedtest: stringarrayaddnull: passedtest: stringarrayaddstr: passedtest: freestringarray: passedtest: stringarraynoduplicates: passedtest: pathextractnullpath: passedtest: pathextractnulloutput: passed
![Page 21: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/21.jpg)
![Page 22: New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America Wednesday, September 18th, 2013 Hyatt Hotel New Orleans, Louisiana 11:50 CST. Revision](https://reader033.vdocuments.net/reader033/viewer/2022053012/5f0fa8b57e708231d445410a/html5/thumbnails/22.jpg)
BCA - very near first release candidateBuild Matrix - nearing feature complete alpha prototype
cts at thomasstover.com