2
Has this happened to you?• Email to users results in
50+ undeliverable
• Need to verify the users in Active Directory
• Then “deactivate” former employees in Crowd
• 750 mouse clicks later, you’re done! http://www.flickr.com/photos/left-hand/4231405740/
Scripting Atlassian applications with Python
Dave ThomasConfiguration Management Architect, FIS
3
Automate That!
4
Agenda
• Use cases for scripting
• Atlassian APIs available for scripting
• The awesome power and simplicity of python
• Examples
5
When is scripting useful?
• Automate time consuming tasks
• Perform data analysis
• Cross-reference data from multiple systems
6
Some specific use cases
• Crowd – Deactivate Users and remove from all groups
• Bamboo – Disable all plans in a project
• JIRA – Release Notes
• Subversion – custom commit acceptance
• Custom build processes – pull code linked to a specific issue into a patch archive
Why Scripts?Why Not Plugins?
7
• I’m not a Java Developer
• Installing new plugins can require a restart
• Prefer to minimize ad hoc changes on the server
• Need to correlate information from several systems
• Need an agile process to accommodate changing requirements
8
APIs for scripting(that we avoid if possible)
• The user interface• Can do anything a user can do• Reporting tasks are relatively easy (particularly when xml is available)• Actions are relatively hard (and prone to breakage)• Capture browser traffic with livehttpheaders, firebug, etc• Form token checking can be an obstacle
• XML-RPC and SOAP• Relatively low-level interface• Many actions available• Relatively complex to use
9
More APIs for scripting(the ones we prefer to use)
• RESTful Remote APIs (now deprecated)• High level interface• Supports a handful of actions
• Now emerging: “real” REST interfaces• High level interface• Supports a handful of actions• http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassi
an+REST+API+Design
10
Why Python?• Powerful standard libraries
• Http(s) with cookie handling• XML and JSON• Unicode
• Third Party Libraries• SOAP• REST• Templates• Subversion
• Portable, cross-platform
11
Python Versions
• 2.x• Ships with most linux distributions• Lots of third-party packages available
• 3.x• Latest version• Deliberately incompatible with 2.x• Not as many third-party libraries
12
HTTP(s) with Python• Python 2
• httplib – low level, all HTTP verbs• urllib – GET and POST, utilities• urllib2 – GET and POST using Request class, easier manipulation
of headers, handlers for cookies, proxies, etc.• Python 3
• http.client – low level, all HTTP verbs• http.parse - utilities• urllib.request – similar to urllib2
• Third-Party• httplib2 – high-level interface with all HTTP verbs, plus caching,
compression, etc.
13
Example 1JIRA Issue Query & Retrieval
14
Discovering URLs for XML
15
Simple Issue Retrievalimport urllib, httplibimport xml.etree.ElementTree as etree
jira_serverurl = 'http://jira.atlassian.com'jira_userid = 'myuserid'jira_password = 'mypassword'
detailsURL = jira_serverurl + \"/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + \"?os_username=" + jira_userid + "&os_password=" + jira_password
f = urllib.urlopen(detailsURL)tree=etree.parse(f)f.close()
Construct a URL that looks like the one in the UI, with extra parms for
our user auth
Open the URL with one line!Parse the XML with one
line!
16
Find details in XML
details = tree.getroot()print "Issue: " + details.find("channel/item/key").textprint "Status: " + details.find("channel/item/status").textprint "Summary: " + details.find("channel/item/summary").textprint "Description: " + details.find("channel/item/description").text
Issue: JRA-9Status: OpenSummary: User Preference: User Time ZonesDescription: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>
Find based on tag name or path to
element
17
Behind the scenes…cookies!httplib.HTTPConnection.debuglevel = 1f = urllib.urlopen(detailsURL)
send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0\r\nHost: jira.atlassian.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'reply: 'HTTP/1.1 200 OK\r\n'header: Date: Wed, 20 Apr 2011 12:04:37 GMTheader: Server: Apache-Coyote/1.1header: X-AREQUESTID: 424x2804517x1header: X-Seraph-LoginReason: OKheader: X-AUSERNAME: myuseridheader: X-ASESSIONID: 19b3b8oheader: Content-Type: text/xml;charset=UTF-8header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f8434fde36326fb340e8b7|lin; Path=/header: Connection: close
Turn on debugging and see exactly what’s
happening
JSESSIONID cookie sent from JIRA
18
Authentication• User credentials determine:
• The data returned• The operations allowed
• Methods Available:• Basic Authentication• JSESSIONID Cookie• Token Method
19
Basic Authentication
• Authentication credentials passed with each request
• Can be used with REST API
20
JSESSIONID Cookie• Authentication credentials passed once;
then cookie is used
• Used when scripting the user interface
• Can be used with REST API for JIRA, Confluence, and Bamboo
21
Token Method• Authentication credentials passed once;
then token is used
• Used with Fisheye/Crucible REST
• Used with Deprecated Bamboo Remote API
22
Obtaining a cookie
• Scripting the user interface login page
• Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword”
• Using the JIRA REST API
23
JIRA REST Authenticationimport urllib, urllib2, cookielib, json
# set up cookiejar for handling URLscookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
creds = { "username" : jira_userid, "password" : jira_password }queryurl = jira_serverurl + "/rest/auth/latest/session"req = urllib2.Request(queryurl)req.add_data(json.dumps(creds))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req) fp.close()
urllib2 handles cookies automatically. We just need to
give it a CookieJar
Request and response are both
JSONWe don’t care about response, just the
cookie
24
Submitting a JIRA Querywith the user interface
# Search using JQLqueryJQL = urllib.quote("key in watchedIssues()")queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + \ "?tempMax=1000&jqlQuery=" + queryJQLfp = myopener.open(queryURL)
# Search using an existing filterfilterId = "20124"queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/" + \ "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId)fp = myopener.open(queryURL)
Pass any JQL Query
Or Pass the ID of an existing shared filter
25
A JQL Query using REST# Search using JQLqueryJQL = "key in watchedIssues()"IssuesQuery = { "jql" : queryJQL, "startAt" : 0, "maxResults" : 1000 }queryURL = jira_serverurl + "/rest/api/latest/search"req = urllib2.Request(queryURL)req.add_data(json.dumps(IssuesQuery))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req)data = json.load(fp)fp.close()
Pass any JQL Query
Request and response are both
JSON
26
XML returned from user interface query
An RSS Feed with all issues and requested
fields that have values
27
JSON returnedfrom a REST query
{u'total': 83, u'startAt': 0, u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969',
u'key': u'JRA-23969'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138',
u'key': u'JRA-23138'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770',
u'key': u'BAM-2770'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489',
u'key': u'BAM-2489'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410',
u'key': u'BAM-1410'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143',
u'key': u'BAM-1143'}], u'maxResults': 200}
A list of the issues found, with links to
retrieve more information
28
JSON issue details
All applicable fields are returned, even if
there’s no value
Expand the html property to get rendered html for description,
comments
29
What’s the difference?<reporter username="mlassau">Mark Lassau [Atlassian]</reporter><customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented">
<customfieldname>Last commented</customfieldname><customfieldvalues>
1 week ago</customfieldvalues>
</customfield>
u'reporter': {u'type': u'com.opensymphony.user.User', u'name': u'reporter', u'value': {
u'self': u'http://jira.atlassian.com/rest/api/latest/user?username=mlassau', u'displayName': u'Mark Lassau [Atlassian]', u'name': u'mlassau'}},
u'customfield_10160': {u'type': u'com.atlassian.jira.toolkit:dayslastcommented', u'name': u'Last commented', u'value': 604800},
XML values are display strings
REST values are type-dependent
REST vs. non-REST
30
REST
• More roundtrips to query JIRA and get issue details
• Returns all fields
• Values require type-specific interpretation
• Easier to transition issues
• Easier to get info for projects, components
Non-REST
• Can query based on existing filter
• XML returns only fields that contain values
• Values always one or more display strings
• Can do anything a user can do (with a little work)
31
Example 2Cross-referencing JIRA, Fisheye, and
Bamboo build results
32
Which build resolved my issue?• Bamboo keeps track of “related issues” (based on issue IDs
included in commit comments), but doesn’t know when issues are resolved.
• If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue”
• Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully.
33
To Fisheye for related commits!
queryURL = FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + \ "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue)
req = urllib2.Request(queryURL)auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password)base64string = base64.encodestring(auth_string)[:-1]req.add_header("Authorization", "Basic {0}".format(base64string))response = myopener.open(req)issuecommits=etree.parse(response).getroot()response.close()
Query a specific fisheye repository for a commit with our JIRA issue ID in the
comments
Use basic auth headers to
authenticate
34
Fisheye changesets returned<results expand="changesets"> <changesets> <changeset>
<csid>130948</csid> <date>2011-04-29T12:35:56.150-04:00</date> <author>lc6081</author> <branch>trunk</branch> <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment> <revisions size="1" />
</changeset> </changesets></results>
35
Parsing the changesetscommits = []for changeset in issuecommits.findall("changesets/changeset"): commits.append(changeset.findtext("csid"))
commits.sort()print "Highest commit is: " + commits[-1]
Highest commit is: 130948
36
Logging into Bamboo
urllib2.HTTPCookieProcessor(cookiejar))
cookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
queryURL = bambooServer + "/userlogin!default.action“
params = urllib.urlencode({ "os_username" : bambooUserid, "os_password" : bambooPassword})response = myopener.open(queryURL, params) response.close()
Using a POST to the user interface login screen to
retrieve a JSESSIONID cookie
37
Querying for build results
# Warning: This is a very resource-intensive operation. # You should consider limiting the number of builds returnedqueryURL = bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + \ "?expand=results[-10:-1].result.jiraIssues"
req = urllib2.Request(queryURL)req.add_header("Accept", "application/xml")response = myopener.open(req)results=etree.parse(response).getroot()response.close()
Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list
Request the related issues
Ask for XML(JSON also available)
38
Example (partial) build results
<results expand="results"><link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" /><results expand="result" max-result="25" size="46" start-index="0"><result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful">
<link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" /><buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime><buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime><buildRelativeTime>4 days ago</buildRelativeTime><vcsRevisionKey>4483</vcsRevisionKey><buildReason>Code has changed</buildReason><comments max-result="0" size="0" start-index="0" /><labels max-result="0" size="0" start-index="0" /><jiraIssues max-result="1" size="1" start-index="0">
<issue iconUrl="http://myjira.domain.com/images/icons/bug.gif" issueType="Defect" key="MYJIRAPROJECT-1629" summary="Need to display an error message when balance is zero."><url href="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" />
</issue></jiraIssues><stages max-result="1" size="1" start-index="0" />
</result></results></results>
Can also expand comments, labels, and
stages
jiraIssues property has been expanded here
39
Walking through build resultsfor result in results.findall("results/result"): print result.get("key") + ":" print "\tRevision: " + result.findtext("vcsRevisionKey") issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")] print "\tIssues: " + ", ".join(issues)
MYPROJECT-MYPLAN-31: Revision: 4489 Issues: MYJIRAPROJECT-1658MYPROJECT-MYPLAN-30: Revision: 4486 Issues: MYJIRAPROJECT-1630MYPROJECT-MYPLAN-29: Revision: 4485 Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663
40
Example 3Removing a user from a Crowd group
41
Beyond GET and POST
connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080')operation = 'DELETE'urlpath = "/rest/usermanagement/latest/user/group/direct" + \ "?username={0}&groupname={1}".format(userToRemove, fromGroup)body = Noneauth_string = '{0}:{1}'.format(crowdAppName, crowdAppPassword)base64string = base64.encodestring(auth_string)[:-1]headers = {'Authorization' : "Basic {0}".format(base64string)}connection.request(operation, urlpath, body, headers)response = connection.getresponse()print response.status, response.reasonconnection.close()
204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found
Lower-level HTTPConnection needed
Authenticate as a Crowd
Application
42
A few loose ends
• Be prepared to handle Unicode strings
• Error handling – not shown here, but important!
• Formatting output – several python libraries for handling templates are available
• REST Interfaces – you can write your own!http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services
43
Links for more information
• http://confluence.atlassian.com/display/JIRA/Displaying+Search+Results+in+XML
• http://confluence.atlassian.com/display/JIRA/JIRA+REST+API+(Alpha)+Tutorial
• http://confluence.atlassian.com/display/CONFDEV/Confluence+REST+APIs
• http://confluence.atlassian.com/display/FECRUDEV/REST+API+Guide
• http://confluence.atlassian.com/display/BAMBOO/Bamboo+REST+APIs
• http://confluence.atlassian.com/display/CROWDDEV/Crowd+REST+APIs