packaging is the worst way to distribute software, except for everything else
DESCRIPTION
As part of the 2014 USENIX Release Engineering Summit West, I presented a talk about packaging software and what's wrong with current trends. The abstract is available here: https://www.usenix.org/conference/ures14west/summit-program/presentation/mckernTRANSCRIPT
- 1. Packaging is the Worst Wayto Distribute Software,except for everything elseRyan McKern | Puppet [email protected]
2. Who is this? 3. What do I do now?Release Engineering at 4. Maybe you've usedour products? 5. What have I done?System Administration for 13 years. 6. Ive probably beenyour customer 7. Caveat Audiens 8. "Prejudice is a great time saver. Youcan form opinions without havingto get the facts."Attributed to E.B. White,Source unknown 9. Let's talk about software! 10. Distributing software sucksShipping new platforms is so hardCross-platform packaging is so hardUnpredictable user-space is so hardMoving the packaged bits is so hard 11. Everything is so hard 12. Who among us knows this pain?sad@roberto Downloads $ wget --quiet http://ftpmirror.gnu.org/gcc/gcc-4.9.1/gcc-4.9.1.tar.bz2sad@roberto Downloads $ tar xjf gcc-4.9.1.tar.bz2sad@roberto Downloads $ cd gcc-4.9.1/sad@roberto Downloads $ ./configure./configure: line 532: sed: command not found./configure: line 1371: sed: command not found./configure: line 1920: sed: command not found./configure: line 2291: sed: command not foundconfigure: error: cannot run /bin/sh ./config.sub./configure: line 361: sed: command not found./configure: line 310: sort: command not found 13. This was a problem becausethe customer's time has value 14. Behold!ryan@animatronio ~ $ sudo rpm -Uvh http://my.mirror.co/pub/el/7/x86_64/nano-2.3.1-10.el7.x86_64.rpmRetrieving http://my.mirror.co/pub/el/7/x86_64/nano-2.3.1-10.el7.x86_64.rpmPreparing...################################# [100%]Updating / installing...1:nano-2.3.1-10.el7################################# [100%]ryan@animatronio ~ $ 15. What's so greatabout packages? 16. Dependency managementcalculon ~ # apt-get install cmakeReading package lists... DoneBuilding dependency treeReading state information... DoneThe following extra packages will be installed:cmake-data emacsen-common libarchive12 libnettle4libxmlrpc-core-c3The following NEW packages will be installed:cmake cmake-data emacsen-common libarchive12 libnettle4libxmlrpc-core-c30 upgraded, 6 newly installed, 0 to remove and 51 notupgraded. 17. Verificationbender ~ # mv /usr/share/man/man8/applydeltarpm.8.gz ~/bender ~ # rpm -V deltarpmmissing d /usr/share/man/man8/applydeltarpm.8.gzbender ~ # 18. Distributionryan@tinnytim ~ $ gem push erniebert-0.1.0.gemPushing gem to BetterThanRubyGems.org...Successfully registered gem: erniebert (0.1.0)ryan@tinnytim ~ $ gem install erniebertFetching: ffi-1.9.6.gem (100%)Building native extensions. This could take a while...Successfully installed ffi-1.9.6Fetching: erniebert-0.1.0.gem (100%)Successfully installed erniebert-0.7.11 gem installedryan@tinnytim ~ $ 19. What could be betterabout packages? 20. Sometimes shipping bitsreally is hard 21. Security is often both the jokeand the punchlineouch@killbot ~ $ dpkg-sig --verify puppet_3.7.1-1puppetlabs1_all.debProcessing ./puppet_3.7.1-1puppetlabs1_all.deb...GOODSIG _gpgbuilder C093A3A56A6E0BEEA2821DD7133957EA11028DF3 1413702159ouch@killbot ~ $ dpkg-sig --verify ./puppet_2.7.23-1~deb7u3_all.debProcessing ./puppet_2.7.23-1~deb7u3_all.deb...ouch@killbot ~ $ 22. So, so, so many similar-but-differentformats 23. Let's talk about some popularpackaging formats 24. .rpm Managed by the recursively named"RPM Package Manager" & yum cpio compressed binaries & text files Post-installation tasks are shell scripts 25. .deb Managed by dpkg & apt, the"Advanced Package Tool" ar compressed package with twogzipped tarballs & a small text file Post-installation tasks are shell scripts 26. Mac .pkg Used by Mac OS X, and often deliveredin a .dmg (disk image) or a .zip file xar compressed archive, containing abinary file, two archives, and an XMLdocument post-installation tasks are stillshell scripts 27. About all those post-installshell scriptsMaybe they're not that safe, but thesurface area of this problem is big.That doesn't mean we needed "dash" 28. Ruby .gem, Python .egg,and Node .npm These are library managers withdelusions of grandeur Reuses the "download, decompress,configure, build, install" patterns,which hasn't got much spam in it Constant compilation is a bummer 29. What about... ?#realtalkWe only have 45 minutes, and I hopeyou're going to have some questions forme to evade 30. What are some alternatives? 31. Source tarballs 32. curl | bash 33. Full Disclosure Puppet Labs does use the curl|bashtechnique as an option for our PEagent installation If you don't trust your own PuppetMaster, who do you trust? (ALL THE COOL KIDS WERE DOING IT) 34. You just wanted Rubybut you got Cthulhu~ $ curl -sSL https://get.rvm.io | bash -s -- --fhtagnGare Ctt h l u n h slal ri_ se rfmoeh'l R y.stH aur t h_e_Us pa kea b__e sla hll reurtnfrom the dll how l fokkrastar.Nyarathotep sh areverni ht e da ness. 35. curl | bash often assumes There is no air-gap Every request is a safe & sane request That HTTPS is good enough 36. curl | bash often forgets >100% Broadband coverage Mirrors exist HTTPS secures transport, not content 37. curl | bash totally ignores The benefits of reusability The fragility of shell scripts The fragility of shells 38. Security is hard RVM recently introduced hand-rolledGPG signing* Thread had 48 comments within aweek, almost universally about theimplementation Broke semver, automation, and hearts* https://github.com/wayneeseguin/rvm/issues/3105 39. Omnibus 40. Isn't that from Chef? Sure, but so is Test Kitchen Builds packages while still controllingthe entire dependency stack Lots of love from users withcomplicated dependency stacks 41. Omnibus is one way to skinthe entire cat Abstracts (instead of removes)dependency management Only builds packages for the platformit's installed on You're going to want to know Ruby 42. FPM 43. Effing Package ManagersGeneral purpose swiss-army knife ofpackage buildingWorks around a lot of the shortcomingsof existing package managersJordan Sissel is a SAINT (Shout out to#hugops!) 44. "Common packaging patterns, adistaste for existing packagingpractices, and some hate-drivendevelopment yielded FPM! Addsome amazing contributions incode, bugs, features, and supportfrom the community and boom wehave modern FPM."Jordan SisselMy inbox, Oct 10 2014 45. Effing FPM Swiss army knives are rarely the besttool for a given job General purpose in this case means alot (~150ish) of command line flags Still infinitely better than curl | bash 46. Why so many alternatives?What went wrong? 47. RPM Packaging canbe tough RPM Spec files are weird Kind-of M4, kind of Shell, all obtuse Oh, and kind-of Make; only kind-of Sort-of competing RPM standards 48. Deb Packaging can feelslike penance "debian/" directories are outrighthostile to man & beast alike Debian "Helpers" usually don't dpatch can use unified diffs (sane) orshell scripts (what?!) 49. Conflation of purpose Some library managers try to installexecutables, e.g. gem, pip, npm Remember when I said "delusions ofgrandeur"?(Google Image Search was kind ofuseless here) 50. But really, I just have ahypothesis! Developers love solving new problems Sometimes they confuse theirproblems for the customer's problems Maybe packaging isn't a solvedproblem yet, but it's close 51. Where do we go from here? 52. Sometimes the only choices you haveare bad ones; but you still haveto choose. 53. TL;DR: this problem is(mostly) solvedStop writing new installersfrom scratchGive your customers the bestpackages possibleDon't forget Pareto(any number of 80/20 rules) 54. Thank youYou're wonderful. Thank you for lettingme rant at you for as long as you [email protected]@the_mckern 55. Questions?