puppetconf 2016: delivering premium quality modules: using beaker and vmpooler for multi-platform...
TRANSCRIPT
DeliveringPremiumQualityPuppetModules
RenéLast&DimitriTischenkoPuppetconf2016
Using Beaker and VMpooler for Mul4-Pla7orm Tes4ng
Whoarewe?
Agenda
AboutKPNOurproblemOursoluBon
AboutKPN
Ourproblem
KPNdecidedtomanage10Kserversforvariousclients
withPuppet…
Ourproblem
…mostofthemWindows.
Ourproblem–consequences(1)
NeedtosupportWindows
Ourproblem–consequences(2)
Needtoguaranteestableandpredictableinfrastructurestate
Ourproblem–consequences(3)
Needtoguaranteecodequality
OursoluEon(1)
Adedicatedteamtodevelopmodules
OursoluEon(2)
AnautomatedtesBngfactory
TesEngFactoryRequirements
AutomatedpipelineconfiguraBonPuppet3.xtestsPuppet4.xtestsUnittestsonLinuxandWindowsAcceptancetestsonallplaQorms(currently8)Runtestsinparallel
ModuleCI/CDPipelineOverview
syntaxcheck
Auto
puppet-lint
rspectest
acceptancetest
ReleasetoTEST
Auto Auto Manual
ReleasetoPROD
ManualChangein
git
Periodictrigger
NoEfybymail&Hipchat
Fail Fail Fail Fail
PuppetCIComponents
JenkinsMaster
JenkinsLinuxSlave
JenkinsWindowsSlave
vmpooler
GithubEnterprise
vcenter
beaker
CreateVMs
Linuxpipeline
WindowspipelineMulE-PlaXormAcceptanceTests
Syntax,lint&rspec
Syntax,lint&rspec
SystemsUnderTest
Jenkinsmasterandbuildslaves
rtyler/jenkinspuppetmodule
Ruby
maestrodev/rvmPIKoruru!=rvmL
Jenkinsjobs
Jenkinsjobbuilderstankevich/python
PuppeEzingJenkinsJobs
puppet Yamlfile Jenkinsjobbuilder xml Jenkinsjob
Howtoaddamodule
1. kpn_jenkins::pipeline::module{'kpn-puppet-forge/puppet-kpn-hosts':2. ensure=>present,3. test_platform=>['linux','windows'],4. pe3_syntax=>false,5. pe4_syntax=>true,6. pe4_unit=>true,7. pe3_unit=>false,8. acceptance_test=>true,9. acceptance_platform=>['centos-7-x64-pe4',10. 'windows-2008r2-x64',11. 'windows-2012r2-x64',12. 'windows-2008r2-x64-pe4',13. 'windows-2012r2-x64-pe4',14. 'windows-2016-x64-pe4',15. ],16. hipchat_notification=>true,17. hipchat_room=>'Moduleteam',18. email=>‘[email protected]',19. }
Howtoaddamodule
JenkinsMaster
JenkinsLinuxSlave
JenkinsWindowsSlave
vmpooler
GithubEnterprise
vcenterCreateVMs
WindowspipelineMulE-PlaXormAcceptanceTests
Syntax,lint&rspec
Syntax,lint&rspec
SystemsUnderTest
Beaker
beaker
Linuxpipeline
Beaker
AtestharnessfocusedonacceptancetesBngusing(virtual)machines
Beaker
Runstestson“real”servers
Beaker
SupportsvmproviderslikeVagrant,vSphere,docker,AWSorvmpooler
BeakerInstallaEon
#geminstallbeakerbeaker-rspecbeaker-puppet_install_helper
BeakerConfiguraEonspec/acceptance/nodesets/*.yml 1. HOSTS:2. centos-7-x64-pe4:3. roles:4. -agent5. platform:el-7-x86_646. hypervisor:vmpooler7. template:centos-7-x648. CONFIG:9. masterless:true10. pe_ver:'2016.2.0'11. pe_promoted_builds_url:'http://repo.kpn.com/pe'12. pooling_api:'http://vmpooler.kpn.com:4567'13. ssh:14. auth_methods:15. -password16. user:root17. password:XXXXXXXX
PrepareaSystemForTesEngspec/spec_helper_acceptance.rb 1. require'beaker-rspec/spec_helper'2. require'beaker-rspec/helpers/serverspec'3. require'beaker/puppet_install_helper'4. 5. #Installpuppetagentonanode6. run_puppet_install_helper7. 8. #CopyourmoduledependenciestotheSUT9. system('[email protected]/stdlibspec/fixtures/modules/stdlib')10. 11. copy_module_to(hosts,:source=>'spec/fixtures/modules/stdlib',12. :module=>'stdlib')13. 14. #Orifyouhaveinternetaccess15. install_puppet_module_via_pmt_on(host,:module_name=>'stdlib')
ExampleTestspec/acceptance/*_spec.rb 1. require'spec_helper_acceptance'2. 3. describe'kpn-centerity_agentmodule'do4. context'installcenterity_agentsoftware'do5. pp=<<-EOS6. class{'centerity_agent':}7. EOS8. it'1stapplyshouldrunwithouterrors'do9. apply_manifest(pp,:catch_failures=>true)10. end11. it'2ndapplyshouldrunwithoutchanges'do12. apply_manifest(pp,:catch_changes=>true)13. end14. end15. end
TesEngforInstalledPackages
1. context'Testinstalledsoftware'do2. casefact('osfamily')3. when/RedHat/4. describepackage('sshd')do5. it{shouldbe_installed}6. end7. when/windows/8. describecommand('C:\Programs\Chocolatey\bin\clist-l')do9. its(:stdout){shouldmatch/sshd/}10. end11. end12. end13. end
Run,beaker,Run!
#exportBEAKER_setfile=~/nodesets/${platform}.yml
#rakebeaker
RulesforWriEngAcceptanceTests(1)
Onlytestyourpuppetcode,
notthesupplier’ssocware
RulesforWriEngAcceptanceTests(2)
Abeakertestshouldbeself-contained
RulesforWriEngAcceptanceTests(3)
Ifyoucanrspecit,don’tbeakerit
ImprovingBeakerDocumentaEon
heps://github.com/RARYates/beaker
JenkinsMaster
JenkinsLinuxSlave
JenkinsWindowsSlave
GithubEnterprise
vcenterCreateVMs
WindowspipelineMulE-PlaXormAcceptanceTests
Syntax,lint&rspec
Syntax,lint&rspec
SystemsUnderTestbeaker
vmpooler
vmpoolerLinuxpipeline
vmpooler
VmpoolerpoolsVMs
Providesconfigurablepoolsofinstantly-available
(running)virtualmachines
vmpooler
NeedsitsownDHCPzone
andproperlyconfiguredDDNS
vmpooler
Systemrunningbeakermustusevmpooler’s
nameserver
vmpooler
vmpoolerhasastrangelogo
vmpoolerinstallaEon
include‘vmpooler’
vmpooler
❝Templateset-upislecasanexercisetothereader.Somehow,eitherviaPXE,embeddedbootstrapscripts,orsomeothermethod--clonesofVMtemplatesneedtobeabletoset
theirhostname,registerthemselvesinyourDNS,andberesolvablebythevmpoolerapplicaBonacercompleBngtheclonetaskandbooBngup.
❞
vmpoolerhostnamingpuzzle
vmname!=hostnameneedtosethostnameautomaBcallytovmnamedon’twanttogiveeachvmvspherecredenBals
so-howtodoit?
Just5linesofcode(2monthslater)#Annotatewithcreationtime,origintemplate,etc.#AddextraconfigoptionsthatcanbequeriedbyvmtoolsconfigSpec=RbVmomi::VIM.VirtualMachineConfigSpec(annotation:JSON.pretty_generate(name:vm['hostname'],created_by:$config[:vsphere]['username'],base_template:vm['template'],creation_timestamp:Time.now.utc),extraConfig:[{key:'guestinfo.hostname',value:vm['hostname']}])
Seangthehostname
Linux:#hostname=$(vmtoolsd--cmd"info-getguestinfo.hostname")Windows:>$hostname=cmd.exe/c'C:\Progra~1\VMware\vmtoolsd.exe--cmd"info-getguestinfo.hostname"'
vmpoolerlessonslearned
• Canusemorelogging(PRsforthcoming)• Poolnameneedstoequaltemplatename• Docsrefertovmtemplates,needtobevms• WillnotinstallifESXhostisnot“green”• CanusebeeerexcepBonhandling
Conclusions
AutomatedpipelineconfiguraBonPuppet3.xtestsPuppet4.xtestsUnittestsonLinuxandWindowsAcceptancetestsonallplaQorms(currently8)Runtestsinparallel
PPP
P P
P
Whatdidwelearn?
+ Regressiontest+ MulBplaQorm
+ Catcherrorsearly
Whatdidwelearn?
FutureImprovements
Speedoftests
FutureImprovements
StabilityofvmpoolerWindowssupport
Andnow…forsomenumbers
ModulesperOperaEngSystem
Windows32
Linux43
LinuxandWindows
16
TestTypes
102 9479
0
20
40
60
80
100
120
Syntax Rspec(unit) Acceptance
#modules
TestRunsFlow8%
Init14%
Syntax21%
Unit18%
Acceptance39%
Top10ModuleswithFailedTests
Vmpooler:AverageTimetoReady
Q&A
WindowsModules
• kpn-ad_ds• kpn-dotnet• kpn-rdp• kpn-kms• kpn-chocolatey_client• kpn-chocolatey_server• kpn-local_security_policy