using omni to build tools

Sponsored by the National Science Foundation

Using Omni to Build Tools

Sarah Edwards, GENI Project Office

23 July 2013

$ omni.py createsliver aliceslice myRSpec.xml INFO:omni:Loading config file omni_config INFO:omni:Using control framework pgeni INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab. expires within 1 day on 2011-07-07 INFO:omni:Creating sliver(s) from rspec fileINFO:omni:Writing result of createsliver for INFO:omni:Writing to ‘aliceslice-manifest-rspeINFO:omni: -----------------------------------INFO:omni: Completed createsliver:

Options as run: aggregate: https://www.emulab. framework: pgeni native: True

Args: createsliver aliceslice myRSpec.xml

Result Summary: Slice urn:publicid:IDN+pgeniReserved resources on https://www.emulab.net/p Saved createsliver results to aliceslice-manINFO:omni: ===================================

Sponsored by the National Science Foundation 2GEC17: July 23, 2013

Outline

• What is omni?• Using omni as a script to build other tools• Example WIP: Awesome Omni• What’s next?


What is omni?

• Omni is a command line client tool for use with the GENI Aggregate Manager (AM) API

• Supports reserving resources from any aggregate which supports the AM API

• Supports complete resource reservation workflow• Distributed with gcf which is a set of developer tools,

including an AM, which allow for testing the AM API

client aggregateFlackOmni(-based):

• omni• portal.geni.net

gcf-am.pyFlavors:

• InstaGENI• ExoGENI• FOAM (OpenFlow)

AMSoil-based


Strengths and Weaknesses

Strength• Reliably at the forefront of

implementing changes to the AM API

• Easy and powerful to script with– e.g. readyToLogin.py– Returns easy to use python

objects (no parsing CLI output)

Weakness• Changes fairly frequently• User must be aware of

differences in v1/v2 vs v3• Lots of command line

options (-h is very long)• Packaging• System exit codes

In addition, to the usual strengths and weakness of CLIs…


(Not So) Recent Additions• Multiple aggregates specified in one command (use –a multiple times)• omni-configure.py configure omni for ProtoGENI, Portal and

PlanetLab user account• readyToLogin.py --no-keys gets login info even if don’t have

ssh keys – works well with multi-user slices• stitcher.py dynamically reserve cross-domain VLANs• Aggregate nicknames In an upcoming release• RSpec nicknames

– Specify in omni_config vm = http://...../onevm_rspec.xml

– Look in default directory or urldefault_rspec_location = http://rspecs.geni.net

default_rspec_extension = rspec

omni createsliver myslice myspecialtopology -a ig-utah

RSpec nickname

agg nickname


omni-configure.py

• Distributed with omni• Not built on omni (ie not an omni script)• Automatically generates your omni_config• Supports ProtoGENI, PlanetLab, and GENI Portal

accounts• Reasonable defaults for the common case

– Just run: omni-configure.py• All defaults can be overridden with command line

options


readyToLogin.py• Distributed with and built on omni • Automatically determines your ssh login info• Supports ProtoGENI, ExoGENI resources

$ readyToLogin.py egtest10 -a pg-ky my-node's geni_status is: ready (am_status:ready)User sedwards logs in to my-node using:

ssh -p 33594 -i …/geni_key_portal [email protected] -p 33594 -i …/geni_cert_portal_key [email protected] -p 33594 -i …/geni_cert_portal_key [email protected]

$ readyToLogin.py egtest10 -a pg-ky --no-keysmy-node's geni_status is: ready (am_status:ready)User sedwards logs in to my-node using:

ssh -p 33594 [email protected]


Omni scripting#!/usr/bin/env pythonimport omni

# Get a parser from omni that understands omni optionsparser = omni.getParser()parser.set_usage("%s [options] username"%sys.argv[0])options, args = parser.parse_args(sys.argv[1:])<snip># (1) Run equivalent of 'omni.py listmyslices username’text, sliceList = omni.call( ['listmyslices', username], options ) <snip># (2) For each returned slicename run equivalent of: # 'omni.py print_slice_expiration slicename’for slicename in sliceList: text, expiration = omni.call( ['print_slice_expiration', slicename], options ) printStr += "%s\n"%(str(expiration))printStr += "="*80 return printStr

================================================================================User sedwards has 2 slice(s):Slice urn:publicid:IDN+ch.geni.net:tutorial+slice+portal99 expires on 2013-07-13 18:54:28 UTCSlice urn:publicid:IDN+ch.geni.net:tutorial+slice+portal01 expires on 2013-07-12 14:03:25 UTC================================================================================

gcf/examples/expirationofmyslices.py

Easily execute and parse the output of omni calls from within another script

Returns python objects


Omni scripting

#!/usr/bin/env pythonimport omni

# Get a parser from omni that understands omni optionsparser = omni.getParser() <snip>

# Add additional optparse.OptionParser style options for your script as needed.# Be sure not to re-use options already in use by omni parser.add_option("--myScriptPrivateOption", help="A non-omni option added by %s"%sys.argv[0], action="store_true", default=False)# options is an optparse.Values object, and args is a listoptions, args = parser.parse_args(sys.argv[1:])if options.myScriptPrivateOption: # do something special for your private script's options print "Got myScriptOption” <snip>

gcf/examples/myscript.py

Inherit command line options from omni while adding custom options

Easily inherit omni’s command line arguments…

…while adding your own

Get -a, -c, --api-version for free


Tools built using omni scriptingFor experimentersgcf/src/stitcher.py

• Tool which supports stitching (as seen in yesterday’s plenary)

gcf/examples/readyToLogin.py• determines ssh command to login to any node

gcf/examples/remote-execute.py• remotely runs commands on the nodes in your slices

For developersgcf/acceptance_tests/AM_API/am_api_accept.py

– AM API acceptance tests • verifies compliance with the AM API specification • supports AM API v1, v2, and v3• Looking for an example code snippet? Look here

For operators• Aggregate health checks run by GPO Lab’s nagios are simple omni

scripts


Tools using omni under the covers

GENI Experimenter Portal (https://portal.geni.net)• runs omni under the covers to talk to aggregates

GENI Desktop and GIMI• Each use omni for parts of their GENI Portal integration

HyperNets a.k.a. Shufeng Huang’s PhD Thesis


Best Practices Using Omni to Build a Tool

• Cache slice credentialomni.py getslicecred mySlice -o # save slice cred to a fileomni.py createsliver --slicecredfile myslice-cred.xml

• Cache user credentialomni.py getusercred mySlice -o # save slice cred to a fileomni.py listmyslices --usercredfile jdoe-usercred.xml

• GetVersion cache by default reduces calls to GetVersion but may or may not be appropriate to your situationTo not use the cache use --NoGetVersionCacheTo use the cache use --ForceUseGetVersionCacheTo age out the cache --GetVersionCacheAgeTo set the location of the cache file --GetVersionCacheName

• Cache a list of aggregates used in each slice– So only have to query used aggregates


What next for omni?Possible next steps? What would be most useful?• Packaging and installing omni -> easier to build tools using

omni– Completed initial installation on Windows – Unix package

• Cache user & slice creds -> fewer/quicker calls • Ability to run multiple commands in one invocation • Exit codes returned when run from the command line

make it hard to shell script with omni• Smooth over difference in v1/v2 vs v3 easier for users• Common CH API support

– Multi-user slice support: dynamically query CH for users and keys for each slice instead of using a statically configured list

• Others?


Where to get it?

Where to get omni/gcf?Can be downloaded from the gcf wiki:

http://trac.gpolab.bbn.com/gcf/wiki

Have an idea for an improvement?Send a suggestions to

[email protected], make a ticket (requires a wiki account)

http://trac.gpolab.bbn.com/gcf/newticket

Sponsored by the National Science Foundation

Awesome Omni:Or how I learned to stop worrying

and love Omni

inspired by Niky Riga

Work

In Progress


Motivation

• Omni originated as a developer test tool and has been adapted to support advanced experimenters

• Omni’s user interface reflects this history– Many (~30) commands and many (~40) options– The output is thorough (aka long and verbose)

• omni-configure.py, readyToLogin.py, and stitcher.py are separate tools


Imagine that omni were more awesome.

Imagine it worked like this….


Configure and setup

awemni init * awemni configured! ** Try: * awemni help *awemni help* .....


A complete workflowawemni create vm as topo1 at utah_rack * Created one virtual machine called ”topo1" at aggregate "utah_rack” * To login/ssh, try: * awemni connect node1 * To release resources try: * awemni delete node1 awemni connect topo1 at utah_rack* ”topo1" is booting...wait a minute and try again

awemni connect topo1 at utah_rack* ”topo1" is ready...logging in *vm> hostnamevm.utah.geniracks.comvm> exit

awemni delete topo1* Deleted virtual machines


Or even more simply…

awemni create vm * Created one virtual machine named "novelSlice” * at each of: utah_rack, mass_rack * To login, try: * awemni connect novelSlice at utah_rack * To release resources try: * awemni delete novelSlice

awemni connect novelSlice at utah_rack* "novelSlice" is ready...logging in vm> hostnamevm.utah.geniracks.comvm> exit

awemni delete vm* Deleted virtual machines


Reserving a custom stitched topology

With stitching this becomes even easier…

awemni create nikysawesometopology.....


What’s needed to do this?

Have omni support nicknames for RSpecs A way to store a small amount of current state

A wrapper to: parse the natural language-ish syntax into the underlying omni parse the omni outputs into something natural language-ish pull in the functionality of omni-configure.py to support init pull in the functionality of readyToLogin.py to support connect parse RSpec into something human readable

Testing


Demo

using omni to build tools

Documents

omni scriptautomatically

config file omni

createsliver aliceslice

config info

login info

createsliver results

writing result of createsliver

control framework pgeni