who pulls the strings?jeffg/presentations/conf/txlf2012/...who pulls the strings? integrating...

Who Pulls the Strings? Integrating OpenNMS with Modern

Configuration Management

Texas Linux Fest 2012, 04 August 2012Jeff Gehlbach Ronny Trommer

[email protected] [email protected](Not present today)

mailto:[email protected]

mailto:[email protected]

04 Aug 2012 Texas Linux Fest 2012

“World's First Enterprise-Grade Network Management Platform Developed Under the Open Source Model”

Started in 1999 by ex-OpenView hackers

Maintained by the Order of the Green Polo

Supported, sponsored by my employer

Consistent model designed for huge scale

100% GPLv3 codebase

Will never suck

Will always be Free (as in Freedom)

What is OpenNMS?®

Fauxpen Source


Built from the ground up

100% GPLv3 code base, Java

Makes extensive use of good libraries

Does not duct-tape in other apps

→ That way lies the end of scalability

→ Not to mention maintainability

Architectural decisions dictated by requirement to scale huge.

Not “Based On Tool X”


If you want a monitoring app that works just the way you want right out of the box, keep looking. OpenNMS is a platform, not a fixed-function application.

It is designed to “front-load” the effort involved in a given task. The payoff comes in easy, automatic repetition of that task at scale.

Designed to Save You Time


Routers

Switches

Firewalls

Load balancers

WAN accelerators

VPN concentrators

... does it have an IP address?*

Network Management


Describes the capabilities that a network management platform may offer:

Fault management

Configuration management

Accounting

Performance management

Security management

OSI FCAPS Model


OpenNMS sticks to ~2.5 of these concerns


How to add nodes to be managed?

Services up and responding quickly? (F)

Something happened, how to know? (FS)

What's happening under the hood? (P)

OpenNMS Concerns Itself...


Even after cutting FCAPS in half, it's still a big problem domain, especially

when done at large scale!


Configuration Management

• Another big problem space!

• Basic challenges:– Define what a system's config should be– Automate the “make it so” part

• Plenty of free platforms do a good job– Cfengine– Puppet– Chef


Plenty of smart people work on OpenNMS, so we could extend its

core capabilities into the configuration management space...

Why Innovate...


When We Can Integrate?

...but that would be silly.


ProvisioningAutomatic Provisioning: Seed an IP address; scan for interfaces and services.

Directed Provisioning: Seed an exact set of known IP interfaces and services.

Policy-Based Provisioning: Seed an IP address; scan for interfaces and services, deciding on persistence, data collection, service monitoring, categorization...


Provisioning (cont'd)

External provisioning sources...

DNS import: Create nodes and interfaces from A / AAAA records in a zone

ReST API: Push-wise from outside

Your DB: Make a CGI that generates XML describing your systems, feed URL to Provisiond, watch magic happen


Directed Provisioning

• Every node created this way is part of a requisition and has:– Foreign Source: a string that groups a set

of nodes; identical to the name of the containing requisition. Slightly analogous to Puppet's environments.

– Foreign ID: a string that uniquely identifies a node within a requisition.

• Foreign-Source:Foreign-ID makes an identifying tuple for a node.


Your Configuration Management platform knows everything about the

configuration of your nodes.

Hey, I know!

Use this knowledge to feed OpenNMS!


So “We” Started Hacking

• I wanted an excuse to go to FOSDEM

• Duped Ronny (DE) into doing all the hard work on a Puppet integration POC

• Talk accepted. Cold in Brussels!

• Now people have expectations...

• Starting with the Polish guy who browbeat us afterwards for not covering Chef!


• How we can get data from the CM system?

• What has to be written in OpenNMS?

• Restrictions?

• Further improvements?

Thoughts for the integration work


(With client SSL certificate authentication)

- patches.mydomain.net

- swlab.mydomain.net

- itchy.mydomain.net

- scratchy.mydomain.net

- lvps.mydomain.net

curl -k -H "Accept: yaml" \

https://puppetmaster:8140/production/facts_search/search

Puppet ReST API


curl -k -H "Accept: yaml" \

https://{puppetmaster}:8140/{environment}/node/{puppetNode}


Oops. 401 Authorization Required.

Mixlib::Authentication complicates things a bit, since nobody else uses this scheme

curl -k -H "Accept: application/json" \

http://chef-server:4000/search/node

Chef ReST API


knife search -Fj node '*:*'

Things Go Better with knife


You have to build this structure

Got Data; Now What?


/opt/opennms/etc/imports/production.xmlTa-Da! Nodes from Puppet


/opt/opennms/etc/imports/test.xml

Huzzah! Nodes from Chef


Puppet

<requisition-def import-name="production"

import-url-resource="puppet://puppetmaster:8140/production">

<cron-schedule>0 9 21 * * ? *</cron-schedule>

</requisition-def>

OPENMS_HOME/etc/provisiond-configuration.xml

21 3

Feed URLs to Provisiond


• Used Jersey API for ReST – it's already in the project

• Add snakeYAML library dependency

• Created URL handler for puppet://

• Dealt with some “special” bits in Puppet's YAML output heading

---object:Puppet::Node• Accept: application/pson (bug)

Code Dependencies - Puppet


Chef

<requisition-def import-name="hosted_chef"

import-url-resource=

"chef://[email protected]/https/hosted_chef/organizations/jeffg_testing">

<cron-schedule>0 13 14 * * ? *</cron-schedule>

</requisition-def>

OPENMS_HOME/etc/provisiond-configuration.xml

21 3

Feed URLs to Provisiond

4 5


• Became a committer to jclouds.org– Added support for Chef 10 environments

• Add jclouds-chef library dependency

• Created URL handler for chef://

• Created ChefRequisitionConfiguration class to encapsulate some details

• Remainder is just making clever, configurable mappings for the data

Code Dependencies - Chef


Notification configuration


Notification details


• Currently this code is in two git feature branches

– feature-puppet

– feature-chef-requisitions

• Normalize conventions (as feasible) between Ronny's Puppet work and Jeff's Chef work

• Get some real-world shops to try this out

• Better unit test coverage – send us your weird nodes' YAML / JSON :)

Restrictions and Improvements: Common


• Currently we don't have a “Java-Puppet-Node-Model”

• WEBrick with ReST and scalabilityhttp://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

• Filter import for Nodes based on a fact search like search?facts.productname=bla

• if (possible) ? One ReST call for nodes and facts : leave it as it is

Restrictions and Improvements: Puppet


• Current newest jclouds release (1.5.0.beta.8) does not include Chef environments, but it's in for the next release.

• Wider testing – so far only Hosted Chef, need to test against other flavors

• Solidify mappings between Chef and OpenNMS concepts

• Consider using databags and attributes like the other monitoring platform recipes

Restrictions and Improvements: Chef


Puppet / Chef

• http://docs.puppetlabs.com/guides/rest_auth_conf.html

• http://docs.puppetlabs.com/guides/rest_api.html

• http://wiki.opscode.com/display/chef/Server+API

• https://github.com/jclouds/jclouds-chef

OpenNMS

• http://www.opennms.org/wiki/Developing_with_Git

• http://opennms.org/wiki/Eclipse_and_OpenNMSgit checkout -b ft-puppet origin/feature-puppet

git checkout -b ft-chef origin/feature-chef-requisitions

• irc.freenode.net – #opennms, #puppet, #chef

• http://www.opennms.org/wiki/Mailing_lists

This could be helpful

http://docs.puppetlabs.com/guides/rest_auth_conf.html

http://docs.puppetlabs.com/guides/rest_api.html

http://wiki.opscode.com/display/chef/Server+API

https://github.com/jclouds/jclouds-chef

http://www.opennms.org/wiki/Developing_with_Git

http://opennms.org/wiki/Eclipse_and_OpenNMS


Push-to-OpenNMS: Puppet

• Still a work in progress

• Puppet: External Node Classifier for OpenNMS– Iterate the OpenNMS Nodes ReST Service

using HTTParty or equivalent gem– Output YAML to populate puppetmaster– Jason Aras has written a prototype

• https://gitorious.org/opennms-puppet-node-pusher

• It's called a pusher yet it pulls. I know :p

• It's ugly, POC-quality code. Deal with it.

https://gitorious.org/opennms-puppet-node-pusher


Push-to-OpenNMS: Chef

• Still basically conceptual only

• Use Chef Notification Handlers to drive the OpenNMS Requisitions ReST Service– Update the OpenNMS Requisitions ReST

Service when Chef detects adds, updates, deletes

– Moves the sophisticated logic to the Chef side, which is well-suited to deal with it


TAMTTWTDI!

Translation: > 2 approaches∃

Example: a guy in our IRC channel has been hacking on his own idea for an

MCollective integration:

https://github.com/InfoSec812/OpenNMS-MCollective-Client


FIN


Questions, Contact

Jeff Gehlbachidenti.ca: @jeffg / !opennmsE-mail: [email protected] (Freenode): jeffg,

#opennmsG+: http://gplus.to/jeffgdotorg


License

This work is licensed under the terms of the Creative Commons Attribution-ShareAlike 3.0 license.

http://creativecommons.org/licenses/by-sa/3.0/

http://creativecommons.org/licenses/by-sa/3.0/

who pulls the strings?jeffg/presentations/conf/txlf2012/...who pulls the strings? integrating...

Documents