ixp manager workshop v4 deep dive · ixp manager workshop v4 deep dive barry o’donovan & nick...
TRANSCRIPT
IXP Manager Workshop
V4 Deep Dive
BarryO’Donovan&NickHilliard27th Euro-IXForum
Berlin,Germany– October25th 2015
Introduction / House Keeping / Agenda
Whatdoyouwanttocoverhere?Suggestions/preparedstuff:– Currentwishlists/plans– AdeeplookatIXPManagerV4
• Mytoolstack forthoseinterested
– Longtermgoal– funding,sponsorship,etc.– Somepracticalscripting
CurrentWishLists/Plans
Current Wish Lists / Plans (and WIP)
HelpdeskIntegrationL2ACLsAdvancingP2PfunctionalityResellerportsviap-tagPatchpanelmanagementIntelligentProvisioning– Portconfigurationtemplatesandautomaticconfiguration
L2 ACLs
Dynamicportsecurityjustdoesnotwork inanIXP– MostofushavealreadymovedtostaticL2ACLs
NeedthisfunctionalitysupportedinIXPManager:– Preventdrivererror/PEBKAC– AllcustomerstoupdateL2ACLs->nomore2AMphonecalls
Usualconstraintsexist:– Security– Switchvendoragnosticwithreferenceimplementation(s)– Someknobsanddials
Advancing P2P Functionality
CurrentlyP2PfunctionalityisviaRRDs– Thislimitsfeaturedevelopmentforanythingbutsimplyp2pgraphs
EvaluatingOpenTSDB andGraphite/Carbon/Whisper– WhisperislikeRRDinthatitisafixedsizedatabase– However,OpenTSDB ismuchmorecomplexandisdistributed
DiskIOisanissue– butnotinsurmountableWithamoredatabaselikebackend,wecanpresentmoreinterestingdetailstoourcustomers:topnpeers,burstiest peers,presentationofthedatacanbecomealotmoreuserfriendly,…
Reseller Ports via p-tag
IXPManagersupportsthefan-outportmodelforresellersasisNewequipmentatINEXallowsthep-tagmodelThisbreaksalotofthings:– Port/trafficgraphsviamrtg /snmp– MACaddresslearningforp2pgraphs– WehavetwomemberswherewehavenovisibilityL
L2ACLs+advancesinP2P/sflow collectionshouldfixallthisJ
Patch Panel Management
Currentlywemanage~25datacentre patchpanelsviawikiLHavehopedtoimplementthisinIXPManagerfor~7yearsLAnumberoffalsestarts.Why?– We’vebeencompletelyoverthinkingthiswithwiz-bangfeatures– Wejustneedthewikiequivlent forcryingoutloud!– Nofancygraphing,nomanaginginternalinterrack panels…
Patch Panel Management
PatchPanelManagementto:– Recordpatchpanelreferenceandmediatype,numberofports– Onaperportbasis,assigntoacustomer– Recordthedatacentre assignedID– Connection(anddisconnection)date– Optionallyassigntoaswitchport
• Needtomanagecustomerportmovesasaprocessnow!– Oroptionallyassigntocustomerco-locatedequipment
Nomorebike-sheddingthis!(ourselvesincluded!)
Intelligent Provisioning
Anoftenrequestedfeature– eveninternallyatINEXCreateaprovisioningflowforcommontaskssuchas:– Joiningtheexchange(westilluseaWorddocument,yuck!L )– Provisioningaport->verymanual
Intelligent Provisioning
Provisioningaport->verymanual– Assignport– Assignpatchpanelportandsendcrossconnectdetails– Configureport(quarantine)– ConfirmcrossconnectisinplaceandrecordID– Testportandensurespeed/duplexarematched– Quarantineport(checktraffic,checkBGProutesadvertised)– PutportliveinproductionpeeringLAN– Setuproutecollector,routeserverandAS112sessions– Sendannouncement
Intelligent Provisioning
Provisioningaport->verymanual– Assignport– Assignpatchpanelportandsendcrossconnectdetails– Configureport(quarantine)– ConfirmcrossconnectisinplaceandrecordID– Testportandensurespeed/duplexarematched– Quarantineport(checktraffic,checkBGProutesadvertised)– PutportliveinproductionpeeringLAN– Setuproutecollector,routeserverandAS112sessions– Sendannouncement
Intelligent Provisioning
Provisioningaport->verymanual– Assignportand patchpanelportandsendcrossconnectdetails– Configureport(quarantine)– ConfirmcrossconnectisinplaceandrecordID– Testportandensurespeed/duplexarematched– Quarantineport(checktraffic,checkBGProutesadvertised)– PutportliveinproductionpeeringLAN– Setuproutecollector,routeserverandAS112sessions– Sendannouncement
Intelligent Provisioning
So,howdowedesignaprovisioningsystemforeveryIXP?Carefully.Intelligently.Dynamically.Eachstephascommonattributes:– ActionpendingcustomerorIXP– Testtodetermineifstepiscompletecorrectly/incorrectly– Actiontotakeoncompletion?– Messaging– Advancetonextstep
Createaframeworkusingcontractstobuildupaprocesswithsteps
Intelligent ProvisioningPort Configuration Templates
Standardised portconfigurationsat(obviously)essentialatanIXPINEXperformedaforkliftupgradefromBrocadetoExtremerecentlyUsedIXPManager’sdatabasetoconfigureallportsonnewswitches– Timeefficient,nooperatorerrors,ensuredstandardconfigs withoutmissing
elements
CouldbeeasilyintegratedintoIXPManagerforcopyandpaste,but:– Wouldpreferittoactually:shutdownport+wipe+configure+enableagain– WouldalsolikeittovalidateRANCID/Oxidizedconfigs forerrors/ommissions
Vendoragnostic!
IXPManagerV4
IXP Manager V4
Majorversionchangesusuallymeanmajorchanges– Thisistruehere->butmainlyonthebackend
IXPManagerisyourtypicalMVCstack– We’recompletelychangingtwoelementsofthis:
• Thecontroller• Theview
IXP Manager V4 – Doctrine ORM
Let’stalkaboutthemodelfirst:thedatabaseLayerbetweenMySQL/MariaDBandthePHPapplicationisDoctrineORMV3ofIXPManagerchangedfromDoctrine1.2toDoctrine2– ActivemodeltoORM(ObjectRelationalMapping)
Anexample:
IXP Manager V4 – Doctrine ORM
<?php$c = new Entities\Customer;$c->setName( "Big ISP Ltd" );$c->setAutsys(64496);// ...D2EM::persist( $c );
IXP Manager V4 – Doctrine ORM
$custRepo= D2EM->getRepository( “Entities\Customer” );
$customers = $custRepo->findAll();
foreach( $customers as $c ) {// do something
}
IXP Manager V4 – Doctrine ORM
Stableproject– aroundsince2006It’sactuallyanumberoflibrariesthatstacktogether.Mainly:– DoctrineDBAL:DatabaseAbstractionLayer– DoctrineORM:ObjectRelationalMapper/ing
http://www.doctrine-project.org/
IXP Manager V4 – Doctrine ORM
Keyconcepts:– Entities:representasingledatabaserowfromagiventable– Proxies:compiled entitiesthatarefullytransparenttoyourcode.
• Allowsforlazyloading,loadingincompletedetails,etc.
– Repositories:handlessetsofentities• Mostofourcomplexqueriesarehandledbywayofproxies:
IXP Manager V4 – Doctrine ORM
class Customer extends EntityRepository {/** * Utility function to provide a array of * all active and current customers. */ public function getCurrentActive(
$asArray =false, $trafficing =false, $externalOnly = false, $ixp = false )
{…
} …
}
IXP Manager V4
HowdidwestarttalkingaboutDoctrine?
IXPManagerisyourtypicalMVCstack– We’recompletelychangingtwoelementsofthis:
• Thecontroller• Theview
Let’slookattheviewnext…
IXP Manager V4 – The View
What’saviewcomponent?Whydowehaveit/needit?– Separateslogicfrompresentation– Inlargerprojects,UIdesignersdon’tneedtobeabletocodetomanagethe
frontend– Eradicatesspaghetticode– Allowsfortemplating withlayouts– Allowsforskinning
Foryears,theonlygameintowninPHPwasSmarty– It’sgodawfulL Itstinkstohighhell.– Okay,that’sprettyharsh.It’sofits’ageandhasn’tmovedforward…
IXP Manager V4 – The View
ContenderstoSmarty:Twig– modernOOPdesign– goodextensibility– wellsupportedandwidelyused
Blade– thebuilt-indefacto viewforLaravel
Problem:viewsadddeveloperoverhead:moresyntax,libraries,functions,etc.tolearnL
IXP Manager V4 – The View
PHPPlates– NativePHPtemplates– nonewsyntaxtolearn– InspiredbyTwig– Supportslayoutsandinheritance– Easytoextendwithfucntions andextensions
– NB:Platesisafulltemplatesystem,notspagetti codebyanothername
Inreality,IXPManagerv4supportsSmarty,BladeandPlatesoutofthebox.
IXP Manager V4 – The Controller
ThebiggestpartofMVCisC->thecontrollerHandles:– Routingofrequests– Middleware– Inputvalidation– Controllers– Responses
OurnewcontrollerisLaravel.Notjustacontroller,aframework.
IXP Manager V4 – The Controller
Whychangeframeworkatall?– Developerapathywhichleadsto:– Stagnationofthecodebase– Neworprospectivedevelopersareturnedoff– Newfeaturesremainunimplementedbecausetherearebetterways– Staymoderntoleveragenewtechniquesandservices
IXP Manager V4 – The Controller
TheLaravel framework providesnewtechniquesandintegrations:– Serviceproviderframework– Events– Queues– Taskscheduling– Testing– Migrations– Packagemanagement
IXP Manager V4 – The Controller
Wecan’tthrowawaytheexistingcodebasethough.OverthecourseofV4’slifetime,we’llmigratefromZend toLaravel– i.e.Zend/Smartywillco-existwithLaravel/Plates forquitesometime– Thismeansnewfeaturescanbeimplementedimmediatelyusingthelatest
technologies– Wedon’tneedtodisappearforsixmonthstorewritetheentirecodebase
Howwillthisbeachieved?– An(in)elegent solution!
IXP Manager V4 – The Controller
Laravel isnowthedefaultframeworkandroutesrequestsIfarequesthitsLaravel foraroutethatdoesnotexist/isnotimplementinLaravel:– Itthrowsa404exception
Inapp/Http/Kernel.php wecatchthat404exception– AndspinuptheZend Framework
Zend willthenhandleifpossibleorthrowanother404– (handledasapagenotfoundinZend andpresentedtotheuser)
IXP Manager V4 – The Controller
try { return $this->sendRequestThroughRouter($request);
} catch( \Symfony\Component\HttpKernel\Exception\NotFoundHttpException $e ) {
require_once 'Zend/Application.php';
$application = new \Zend_Application(APPLICATION_ENV, APPLICATION_PATH . '/configs/application.ini’
);
$application->bootstrap()->run();}
IXP Manager V4 – Events and Queues
Eventsprovideasimpleobserverimplementation– Youcansubscribeandlistenforeventsinpackages/extentions!– Eventscanbefired whensomethingsignificant happens– Eventlistenerscanqueuetheeventforofflineprocessing
Laravel queuessupportBeanstalkd,IronMQ,AmazonSQS,Redis andsynchronous(local,immediate)
Whatkindofthingscanwedowiththis..?
IXP Manager V4 – Events and Queues
PhysicalinterfacechangedinIXPManager:firephysIntChangedEvent– MRTGlistenercancheckforportorportspeedchangeandregenerateMRTG
configurationandreloadthedaemon– Billingnotificationslistenercancheckforspeedchangeandemailaccounts for
billingpurposes– Switchconfigurationlistenercanrolloutconfigurationchangetoswitch(firesevent)
• Physicalinterfacestatuslistenercaninspectinterfaceformatchingspeed/duplex
– Patchpanellistenercouldtakesome actionifaportischangedthathasaconnectedcrossconnect
IXP Manager V4 – Events and Queues
VLANinterfacechangedinIXPManager:firevlanIntChangedEvent– AS112listenercan(de)configureBGPsessionasnecessary– Routecollectorcan(de) configureBGPsessionasnecessary– Routeserverscan(de)configureBGPsessionasnecessary– Othereventlistenersmayinclude:regeneratingSmokeping&Nagios configurations,
DNSPTRentries.– EnablingIPv6couldadditionallysendemailwithdetailsorstarttheIPv6enable
processtowalkthecustomerthroughconfigurating sessionstoroutecollector,servers,as112,etc.
IXP Manager V4 – Service Providers
CentralpieceofLarabel’s applicationbootstrappingRegisters:– Controllersandroutes– Eventlisteners– Middleware
CanbeusedtoextendIXPManagerwithouthackingthemaincodebase
IXP Manager V4 – Contracts
InterfacesthatdefinecoreservicesprovidedbyLaravelAlsohowwewilldevelopextensionstoIXPManager– Designacontract– Developreferenceimplementation(s)tothatcontract
Example:Helpdeskintegration– Firstabitofhistory….
New Helpdesk - [email protected]
Up to 2008 - Shared IMAP Mailbox2009 - Cerberus
Served us well but extreme feature creep2013 - Realisation that we need something new
Helpdesk research => maximum pain
From an INEX Members’ Update
New Helpdesk - [email protected] an INEX Members’ Update
New Helpdesk - [email protected]
Up to 2008 - Shared IMAP Mailbox2009 - Cerberus
Server us well but extreme feature creep2013 - Realisation that we need something new
Helpdesk research => maximum painPain so great, we stuck with what we had
2015 - Try again, new methodology => pain killerz
From an INEX Members’ Update
New Helpdesk - [email protected] an INEX Members’ Update
New Helpdesk - [email protected]
Candidates included:Freshdesk, GrooveHQ, Zendesk, Kayako, Cerb5 and many more…
Contrary to my initial preconceptions, the winner was ZendeskExcellent API (essential as we needed to import old tickets)Easily configurable triggers and automationsSupports markdownNice UI plus iOS / Android appsPoor reporting (@ Zendesk Regular anyway
From an INEX Members’ Update
IXP Manager V4 – Helpdesk Integration
Needtobeableto:– Createcustomers(organisations)onthehelpdesksystem– Createusersonthehelpdesksystem– Findticketsbyorganisation– Createtickets– Update/closetickets
It’sworkinprogressbutmostoftheintegrationisdone
IXP Manager V4 – Helpdesk Integration
Contract:app/Contracts/Helpdesk.php– ticketsFindAll()– organisationNeedsUpdating($custLocal,$custHelpdesk )– organisationCreate($cust )– organisationUpdate($helpdeskId,$customer)– organisationFind($id)– contactNeedsUpdating(…)– userCreate()– userUpdate()– …
IXP Manager V4 – Helpdesk Integration
Zendesk ReferenceImplementation– app/Services/Helpdesk/Zendesk.php
ServiceProvider– app/Providers/HelpdeskServiceProvider.php– Thisfileneedsupdatingfornewimplementations
Instantiation:– $helpdesk=App::make('IXP\Contracts\Helpdesk');
Configuration:– config/helpdesk.php– EnvironmentconfigurationviaPHPDotEnv
IXP Manager V4 – Helpdesk Integration
Sample.env forZendesk:
HELPDESK_BACKEND=zendeskHELPDESK_ZENDESK_SUBDOMAIN=ixpHELPDESK_ZENDESK_TOKEN=yyyHELPDESK_ZENDESK_EMAIL=john.doe@example.com
IXP Manager V4 – Statistics Backend
IXPManagercurrentlyonlysupportsMRTG/logforportstatsFromearlier,wealsowanttosupportportstatsviasflow /p2p– WouldalsoliketosupportMRTG/rrd
DE-CIXhaveofferedabountyforthisworkWillbeimplementedinthesamewayasthehelpdesk– i.e.anybackendcouldbesubstitued onceitisimplementedagainstthe
providedcontract
Threereferenceimplementations:sflow/p2p,MRTG/log,MRTG/rrd
IXP Manager V4 – Installation Changes
ThePHPdevelopmenttoolchainhaschangedsincev3– Gitsubmodules nolongernecessary
PHPhasapackagemanagementsystemcalledcomposer– Allthirdpartydependancies nowinstalledviacomposer– Includes:ZF1,Laravel,Smarty,Plates,Zendesk API,Doctrine,etc.
Frontendassetshandledsimilarlyviabower– Includesjquery,Bootstrap,etc.
IXP Manager V4 – Vagrant
IXP Manager V4 – Vagrant
IXP Manager V4 – Vagrant
1. InstallVagrant(http://www.vagrantup.com/ )2. InstallVirtualBox (http://www.virtualbox.org/ )3. CloneIXPManager,checkoutv4andinstalldependancies:
git clone https://github.com/inex/IXP-Manager.git ixpmanagercd ixpmanagergit checkout v4
composer update
IXP Manager V4 – Vagrant
4. Start-upVagrant:vagrant up5. Thiswilltakeawhile– itexecutesbootstrap.sh whichwill:
– apt-getupdate,upgradeandinstallalldependancies forIXPManager’sLAMPenvironment
– ConfigureMySQLandphpMyAdmin– Installcomposerandbower– ConfigureandpopulatetheIXPManagerdatabasewithsampledata– ConfigureApacheandIXPManager
IXP Manager V4 – Vagrant
6. Onceit’scomplete,youcan:– AccessIXPManagerat:http://localhost:8088/
• Adminusernameandpassword:vagrant/vagrant1– SSHintothevirtualmachinewith:vagrant ssh– Yourixpmanager directoryismountedunder/vagrant– MySQLisavailablevia:mysql –uroot–ppassword ixp
• Orhttp://localhost:8088/phpmyadmin
IXP Manager V4 – Vagrant
ManagingyourVM:– Shutdowncleanlybylogginginand: sudo shutdown –h now– Tosuspend: vagrant suspend– Toforceshutdown: vagrant halt– Tobringup: vagrant up– Forstatus: vagrant status
IXP Manager V4 – Documentation
Remember:v4isabridgingversionfromZF1toLaravelAssuch,documentationisalsoahalfwayhouseExistingdocumentationavailableat:– https://github.com/inex/IXP-Manager/wiki
Newdocumentationwillbe:– https://ixp-manager.readthedocs.org/en/latest/– Source:https://github.com/inex/ixp-manager-docs
IXP Manager V4 – My Tool Stack
AppleOSXwithHomebrewfor:– php,bash,bgpq3,git,joe,mariadb,node,sshfs andmuchmore
Atomasatexteditor– Withlanguage-php,linter(sameforCSS,JS,etc)andDash
Vagrant(latestUbuntuLTS)Git,GitHub,TravisCISkipper(ORMGUI,http://www.skipper18.com/ )
QuickCodingExample?
Quick Coding Example
Let’smakeanewArtisancommand– ArtisanistheCLIcomponentofLaravel
./artisan make:console DemoListCustomers
Quick Coding Example
Nowlet’sedittheresultantfile:– app/Console/Commands/DemoListCustomers.php
GivethecommandanameanddescriptionAndlet’sseeifitworks…
Quick Coding Example
Andlet’sseeifitworks…nopeL
Extendingthewrongclass– needtouseandextend:– useIXP\Console\Commands\CommandasIXPCommand;– =>classDemoListCustomers extendsIXPCommand {
Nooptions/argumentsrequiredNeedtoregisterthecommandinapp/Console/Kernel:– protected$commands=[…]
Nowitworks!Butdoesnothing...
Quick Coding Example
Let’sgetandlistallcustomers:Completethefire()method:
$customers = \D2EM::getRepository( 'Entities\Customer' )->getCurrentActive();
foreach( $customers as $c )$this->info( $c->getName() );
Thanks for listening!
https://github.com/inex/IXP-Manager
Mailing list:https://www.inex.ie/mailman/listinfo/ixpmanager