cloud foundry api for fun and ops
Post on 21-Mar-2017
660 Views
Preview:
TRANSCRIPT
EXPLORING THE CLOUD FOUNDRY API FOR FUN AND OPS
Josh Kruckprofessional doer of things
github.com/krujos@krujos
WHO AM I?• Today
• pivotal customer success
• cf cli plugin author
• service broker author
• lots of little utilities for getting shit done
• Past
• a lot of storage and application management
• Why
• use software to make work more fun and be happier
maybe you have heard?
https://lynxbat.files.wordpress.com/2012/06/api.jpg
done. good game. thanks everyone!
http://www.wired.com/2013/11/pivotal-one/
now what?
SOME IDEAS
• apps (dashboard, provisioner, automation, crypt keeper)
• cli plugins
• scripts (why aren’t you writing a cli plugin?)
• A one time thing? (jq will change your life)
SCRIPTS ARE AN ANTI PATTERNCLI plugins FTW!
What are the things that got API on them?
Service Brokers(more of a spec really)
Lots of other people talk about service brokers and how to author them, i won’t be covering them.
UAA(User Account and Authentication Server)
This is where we go to login. If you’re writing CLI plugins and scripts, it’s a non issue, but for any kind of app we’ll have to deal with it.
Firehose(the tattletale)
the emitter of most of the things
Collector(older tattletale)
emits a lot of the readable things
Cloud Controller(the brains)
The CC is where we got to listen and act.
but wait!Router
(eventually)
A feature that’s coming down the pipe is “router services”. While I don’t have anything to say on this today it is something to be aware of. It will allow us to do some inline things at the router that we previously had to insert between our LB’s and routers.
THE DETAILS
UAA(User Account and Authentication Server)
This is where we go to login. If you’re writing CLI plugins and scripts, it’s a non issue, but for any kind of app we’ll have to deal with it.
WHEN/WHY DO WE INTERACT WITH UAA?
• When we’re writing an application and DON’T want to inherit the credentials of the user
• Dashboards, system automation and on boarding
System automation examples: Clean up orphaned services Check for rouge users Reports (new URL’s for nessus, apps pushed in the last day, crash summaries… whatevs).
WHEN/WHY DO WE INTERACT WITH UAA?
• When we’re writing an (web) application and we DO want to inherit the credentials of the user
• cli plugins and scripts can usually get this for free
System automation examples: Onboarding, cleanup, CI stuff (slackbots)
UN HUMANS
• For apps that do not act on behalf of a human we use client_credentials
• See the OAuth 2.0 spec for more information
• https://tools.ietf.org/html/rfc6749#section-1.3.4
• http://tools.ietf.org/html/rfc6749#section-4.4
If you google, you will find people who have translated this into somewhat more complicated english than the spec provides.
GOuri, _ := url.Parse(uaaURI.String() + "/oauth/token?grant_type=client_credentials") creds := &UaaClientCredentials{ uaaURI: uri, clientID: clientID, clientSecret: clientSecret, } client := http.Client{tls.Config{}} req, _ := http.NewRequest("POST", creds.uaaURI.String(), nil) req.SetBasicAuth(creds.clientID, creds.clientSecret) resp, _ := client.Do(req)
auth = requests.auth.HTTPBasicAuth(client_id, client_secret) r = requests.post( url=uaa_uri, headers={'accept': 'application/json'}, params={'grant_type': ‘client_credentials'}, auth=client_auth)
PYTHON
both of those examples assume certificate validation, see the docs for skipping it.
➜ curl “https://user:pass@uaa.mycf.xip.io/oauth/token" -d grant_type=client_credentials -X POST | jq “.” { "access_token": “eyJhbG…C6Fvww”, "token_type": "bearer", "expires_in": 43199, "scope": "doppler.firehose", "jti": "2d56398d-ba65-4dbd-9f43-4d1a302709a1" }
STOP WITH YOUR TRICKERY AND GIVE ME A SHELL MAN
THE IMPORTANT THINGS
1. UAA exposes an API protected by HTTP Basic Auth that gives us a token http(s)://uaa.example.com/oauth/token
2. We use a client_id & client_secret for auth
3. We pass it the grant type as a parameter grant_type=client_credentials
4. It gives us back a token and time to live
note that you can indeed send your credentials in clear text. that’s a bad idea.you can, and should prevent http requests from getting to uaa as your traffic can be sniffed.
UAA IS FOR PEOPLE TOO?• CLI Plugins you’ve already assumed the identity of the user
• Scripts should use & assume the identity of the logged in user via cf curl
• password_grant otherwise (but srsly, why do it the hard way?)
• Web apps use the regular OAuth flow
For applications that act on behalf of humans we have a few options.
For scripts cf curl is a good option as it means you can avoid (hopefully) managing usernames and passwords in the scripts.
require 'bundler' require 'sinatra' require 'omniauth' require 'omniauth-uaa-oauth2'
use Rack::Session::Cookie, :secret => ‘its a secret' use OmniAuth::Builder do provider :cloudfoundry, 'app', 'appclientsecret', {:auth_server_url => “https://login.run.cf.io", :token_server_url => “https://uaa.run.cf.io"} end
class App < Sinatra::Base get '/' do <ul><li><a href='/auth/cloudfoundry'>Sign in with Cloud Foundry</a></li></ul> end
get '/auth/cloudfoundry/callback' do content_type 'application/json' request.env['omniauth.auth'].to_hash.to_json rescue "No Data" end get '/auth/failure' do content_type 'text/plain' request.env['omniauth.auth'].to_hash.inspect rescue "No Data" end end
run App.new
This one doesn’t do a redirect like I normally do, use has to click the auth link.
Firehose(the tattletale)
WHAT IS THE FIREHOSE?
• The firehose is where we go to listen
• Combined stream of logs from all apps, plus metrics data from CF components
• Exposed as a web socket endpoint
• Accessed via doppler.firehose scope
The scope means it’s unlikely your user accounts have access to it. Login w/ adminOr create user in uaac.
WHATS IN THE FIREHOSE• A lot, but not everything
• https://github.com/cloudfoundry/loggregator/wiki/Firehose-Metric-Catalog
• https://docs.google.com/spreadsheets/d/176yIaJChXEmvm-CjybmwopdRGQfDGrSzo3J_Mx8mMnk/edit#gid=472741134
• heartbeats are being removed (rejoice!)
• https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/message/EPTBPAJLBV4LAG72GOSK3KYO7MKZPL7B/
I find the most interesting things in the firehose to be http related. I can tell the status of every request in and out of the system, as well as it’s response time. This helps me paint a picture of “normal”. It also helps me figure out “lights are on but no one’s home” situations. CF health check just make a connection on a socket, they don’t check http response codes. So the firehose http metrics give me a good view of what my users are seeing.
HOW TO DISCOVER STUFF?
• noaa firehose sample app and grep
• ➜ export DOPPLER_ADDR=wss://doppler.10.244.0.34.xip.io:443
• ➜ CF_ACCESS_TOKEN=`cf oauth-token | grep bearer` ./firehose_sample
• https://github.com/cloudfoundry/noaa/blob/master/firehose_sample/main.go
*note the app ignores cert validation errors.
WHAT’S INTERESTING• POST’s to the Cloud Controller
origin:"router__0" eventType:HttpStartStop timestamp:1439755833436264212 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439755833419943436 stopTimestamp:1439755833436264212 requestId:<low:8452822747789629966 high:15632988695117975649 > peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/organizations" remoteAddress:"10.0.2.15:39470" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1298 parentRequestId:<low:5137755037457773227 high:6003671325381183557 > instanceId:"10.244.0.134:9022" >
WHAT’S INTERESTING• POST’s to the Cloud Controller
origin:"router__0" eventType:HttpStartStop timestamp:1439755833436264212 … peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/organizations" remoteAddress:"10.0.2.15:39470" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 … >
Someone just created an org successfully.
WHATS INTERESTING➜ egrep "method:PUT|method:POST" /tmp/firehose.out | egrep -v "uaa|login" origin:"router__0" eventType:HttpStartStop timestamp:1439756656563113538 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 requestId:<low:5639804705379164088 high:13655764414813434448 > peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1485 parentRequestId:<low:9677111003368022686 high:16762836456174757743 > instanceId:"10.244.0.134:9022" > origin:"router__0" eventType:HttpStartStop timestamp:1439756656563296157 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656523150050 stopTimestamp:1439756656563296157 requestId:<low:9677111003368022686 high:16762836456174757743 > peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1485 > origin:"router__0" eventType:HttpStartStop timestamp:1439756656712256742 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 requestId:<low:8304746205643063272 high:4491838666105706059 > peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1503 parentRequestId:<low:16449840403178608098 high:13013670532211215483 > instanceId:"10.244.0.134:9022" > origin:"legacy" eventType:LogMessage deployment:"cf-warden" job:"api_z1" index:"0" ip:"10.244.0.134" logMessage:<message:"Updated app with guid d2cd4770-64e0-44fe-a997-f2e7e6dc5c79 ({\"route\"=>\"15879527-b813-4165-9b04-8dc3f5af2a31\"})" message_type:OUT timestamp:143975\ 6656694498836 app_id:"d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" source_type:"API" source_instance:"0" > origin:"router__0" eventType:HttpStartStop timestamp:1439756656712519692 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656666355668 stopTimestamp:1439756656712519692 requestId:<low:16449840403178608098 high:13013670532211215483 > peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:1503 > origin:"router__0" eventType:HttpStartStop timestamp:1439756656786930549 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 requestId:<low:15224145774697211575 high:9071476560656655465 > peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:270 parentRequestId:<low:13565402856673872720 high:11582024932942486143 > instanceId:"10.244.0.134:9022" > origin:"router__0" eventType:HttpStartStop timestamp:1439756656787167491 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756656765234958 stopTimestamp:1439756656787167491 requestId:<low:13565402856673872720 high:11582024932942486143 > peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:270 > origin:"router__0" eventType:HttpStartStop timestamp:1439756663113749547 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 requestId:<low:10468161744519186019 high:3081438611027230039 > peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:4073 parentRequestId:<low:16375959389728685162 high:5351609031812398929 > instanceId:"10.244.0.134:9022" > origin:"router__0" eventType:HttpStartStop timestamp:1439756663114085243 deployment:"cf-warden" job:"router_z1" index:"0" ip:"10.244.0.22" httpStartStop:<startTimestamp:1439756661872568200 stopTimestamp:1439756663114085243 requestId:<low:16375959389728685162 high:5351609031812398929 > peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 contentLength:4073 >
lets distill this and just look at app interactions
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656563296157 httpStartStop:<startTimestamp:1439756656523150050 stopTimestamp:1439756656563296157 peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712519692 httpStartStop:<startTimestamp:1439756656666355668 stopTimestamp:1439756656712519692 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656787167491 httpStartStop:<startTimestamp:1439756656765234958 stopTimestamp:1439756656787167491 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663114085243 httpStartStop:<startTimestamp:1439756661872568200 stopTimestamp:1439756663114085243 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 >
lets distill further and just grab some of the fields we’re interested in. We can see some requests being passed through the system by looking at the http traffic. Kind of a poor mans after the fact zipkin.
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656563296157 httpStartStop:<startTimestamp:1439756656523150050 stopTimestamp:1439756656563296157 peerType:Server method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712519692 httpStartStop:<startTimestamp:1439756656666355668 stopTimestamp:1439756656712519692 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656787167491 httpStartStop:<startTimestamp:1439756656765234958 stopTimestamp:1439756656787167491 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663114085243 httpStartStop:<startTimestamp:1439756661872568200 stopTimestamp:1439756663114085243 peerType:Server method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 >
So now we can see that we have a client talking to a server and a server talking to a server
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
lets distill further, to just client operations.
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
Lets make sure we’re tracing one user’s experience through the system.
note the port changed? Why? I suspect the CLI had to open a new connection..
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
lets distill further, to just client operations, we can trace one user’s experience through the system.
How long did their operation take? (6.5 seconds), firehose timestamps are nano seconds.
WHATS INTERESTINGtimestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795 stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905 stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31" remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916 stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" > timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559 stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
What did they do what was the output?
First one crates the app. second one uploads the bitsthird one does a state change to start apps.
WHY IS THAT INTERESTING?
• If I can learn what normal interaction patterns are, I can identify abnormal ones and investigate
• Are apps failing to start after a push?
• For all teams?
to me anyway.
this lets us identify and view actions across the entirety of the user base.
Could you do this with platform logs? Probably. The beauty of the firehose is it gives you all of it, I omitted the logs from my example for readability and dramatic effect.
Collector(older tattletale)
WHAT IS THE COLLECTOR• The collector is the “legacy” aggregator of CF metrics
• No log messages
• Grabbed all the /varz and /healthz data
• The collector still has a lot of relevant runtime information in it
• available_stagers, healthy, system.cpu.wait
Most of the collector usage is for dashboards.
WHAT IS THE COLLECTOR FOR• It’s also how we export metrics in open source via historian plugins
• https://github.com/cloudfoundry/collector/tree/master/lib/collector/historian
• Vendors (well Pivotal at least) have their own spin
• Pivotal Ops Metrics (JMX)
The historian plugins change the way data is presented. They are usually monitoring solution specific.
COLLECTOR DOWNSIDE
• Vendors don’t install all the plugins in their distribution
• Consumption can be irritating, lots of guesswork
• You may end up pretending to be a historian plugin, or consuming ones output.
Cloud Controller(the brains)
The CC is where we got to listen and act.
CLOUD CONTROLLER
• The CC API is what we talk to the most
• api.run.pivotal.io
• The CC API is where we go to act and report
THE CC API IS UNSURPRISING• Which is to say, the api has all of the things you would expect
• /v2/apps
• /v3/apps (experimental)
• /v2/organizations
• /v2/events
• /v2/spaces
or kinda boring
API DETAILS
• the v3 api is totes HATEOAS compliant!!!
• the v2 api has some links conventions
THE API TEAM COMMITTED TO MAINTAINING BACKWARDS COMPATIBILITY
• v3 api’s will have v2 compatibility layers
• I was there, they said it at cf summit, it’s like they wrote it in their own blood
API DETAILS
• The v2 api is very, very, very verbose
• The v3 api is more targeted
API DETAILS
• v3 api is fun to talk about, but as of today much of it doesn’t exist outside of documentation.
• we will see some examples later.
VERSIONSdoes thing you read about exist in your
version
EXPERIMENTAL REALLY MEANS EXPERIMENTAL.
Theres a good chance it won’t be there (or be the same) in your vendors next release
(I’m looking at you async service broker api)
SOMETIMES IT’S NOT PRETTY
➜ cf curl /v2/apps/<a guid> | grep buildpack "buildpack": null, "detected_buildpack": "PHP",
TOKENS MUST BE ACQUIREDAND THEY EXPIRE
https://github.com/cloudfoundry/cf-uaa-lib (Ruby)https://github.com/cloudfoundry/omniauth-uaa-oauth2 (more ruby)
https://github.com/krujos/uaaclientcredentials (Go)
CLIENTS VS USERS
https://docs.google.com/document/d/1YU01vrGmIhSHCKOFIRY7EhU8qCbUHJsw3ZXgdnUIfeI/editTODO: move that somewhere permanent, and more public.
What’s a client?What’s a user?
UAA
http? https?be safe!
SCOPES, AUTHORITY AND THE GRANTING OF POWER
SO HOW DO I BUILD A THING?
• it starts with cf curl and jq
• or the firehose and grep
• lets see what the firehose has to say about started apps?
export DOPPLER_ADDR=wss://doppler.10.244.0.34.xip.io:443CF_ACCESS_TOKEN=`cf oauth-token | grep bearer` ./firehose_sample | grep -i started
cf stop hello-nodecf start hello-node
• how about orphaned user provided services?
cf cups my-svc -p '{"foo":"bar"}'cf curl /v2/user_provided_service_instancescf curl `cf curl /v2/user_provided_service_instances | jq -r '.resources[0].entity.service_bindings_url'` | jq ‘.total_results'
Think about what we would have had to do to report on that with the CLI?
• lets look at something more complex.
• how many ai’s are you running?
• How does the usage-report plugin work?
• https://github.com/krujos/usagereport-plugin
demo the plugin
sometimes its about mashing things up
Why is how many ai’s your running a hard question to answer?Why is how many ai’s your running an important question to answer? As someone who’s paying for IT charge back, am I paying too much?How much ram am I consuming, are my quotas right?make note of the lack of auth, plugin assumes current user, works for everyone based on their cf access.magic starts here: https://github.com/krujos/usagereport-plugin/blob/master/usagereport.go#L57https://github.com/krujos/usagereport-plugin/blob/master/apihelper/apihelper.go
• What’s fun? what should we try to figure out?
top related