clearquest xml server with clearcase integration northwest rational user’s group february 22, 2007...
TRANSCRIPT
ClearQuest XML Serverwith ClearCase Integration
ClearQuest XML Serverwith ClearCase IntegrationNorthwest Rational User’s GroupFebruary 22, 2007Northwest Rational User’s GroupFebruary 22, 2007
Frank Scholz
Casey Stewart
Frank Scholz
Casey Stewart
© 2007 Mentor Graphics Corporation
All Rights Reserved
© 2007 Mentor Graphics Corporation
All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
22
Agenda
Mentor Graphics Environment Requirements High-Level Overview Samples Architecture Details Findings ClearCase Client Demonstration
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
33
Mentor Graphics Environment
Product development effort is widely distributed.— 3000+ developers— 30+ development sites worldwide
ClearQuest deployment is 100% web-based. ClearQuest sits behind the corporate firewall. No ClearQuest groups. Implementation by the numbers:
120,000+350,000+1,350,000
+3,500,000
+1
stateless recordsdefects recordsnotes recordshistory recordsdatabase
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
44
Requirements
No client installs. No remote servers. Perform well over the WAN. Scalable. Utilize a generic interface. Enforce permission restrictions. Password encryption. Notification on errors.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
55
High-Level Overview
CQ Request Manager
Database
CQ Native Client
Browser Client
ToolsWeb Servers
X
User Written Client
XML Server
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
66
High-Level Overview
The ClearQuest XML Interface (CQ/XML) uses industry standard communication methods and formats to communicate to the ClearQuest Request Manager.— Immediate updates to ClearQuest data.— ClearQuest business rules are enforced.— Development activity can have direct
relationships to defect data. “Industry standard communication…”
— method: platform-independent TCP sockets— format: XML
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
77
High-Level Overview: Features
All CQ/XML communication is done via XML.— input, output, errors, logs and configuration— W3C standard
Read, write and create ClearQuest defect records.
Performs well over the WAN. Supports RC4 password encryption. Permissions can be limited to a single user,
multiple systems or anywhere in between. Allows interactive or batch response.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
88
High-Level Overview: Features
Users can run existing ClearQuest queries, including prompted queries.
Special functionality has been added to support user-specific interactivity.
ClearQuest defect histories record XML updates.
Logs are reviewable in all of the current browsers.
CQ/XML server can email users ClearQuest errors.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
99
Sample: view a defect’s headline
<!-- the login root element --> <ClearQuest login="cq_ user" password="* * * * * " repo="cqrepos" db="dev28"> <defect id="dev2800000048" action="view" wait="yes"> <headline/> </defect> </ClearQuest> <!-- CQ/XML RETURNS --> <ClearQuest db="dev28" login="cq_ user" cqtan="06318n000001"
client="w35d125" ip="147.34.35.125"> <defect id="dev2800000048" action="view" status="ok"> <headline>Breakout length changed when joining designs.</headline> </defect> </ClearQuest>
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1010
Sample: submit a defect
<ClearQuest login="cq_ user" password="* * * * * " repo="cqrepos" db="dev28"> <defect id="0" action="submit" wait="yes"> <component>Component A1</component> <description>sample submit description</description> <headline>sample submit</headline> <product>Product A</product> <version_reported>v1</version_reported> <severity>3-Average</severity> </defect> </ClearQuest> <!-- CQ/XML RETURNS --> <ClearQuest db="dev28" login="cq_ user" cqtan="06318n000002"
client="w35d125" ip="147.34.35.125"> <defect id="0" action="submit" status="ok"> <id>dev2800007374</id> </defect> </ClearQuest>
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1111
Sample: “adding” to defect fields
<ClearQuest login="cq_ user" password="* * * * * " repo="cqrepos" db="dev28"> <defect id="dev2800000048" action="modify" wait="yes"> <platform behavior="add">Linux Red Hat Ent 3 x86</platform> <description behavior="add">added 2 days to est. time</description> <time_estimate behavior="add">2</time_estimate> </defect> </ClearQuest> <!-- CQ/XML RETURNS --> <ClearQuest db="dev28" login="cq_ user" cqtan="06318n000003"
client="w35d125" ip="147.34.35.125"> <defect id="dev2800000048" action="modify" status="ok" /> </ClearQuest>
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1212
Sample: prompted query
<ClearQuest login="cq_ user" password="* * * * * " repo="cqrepos" db="dev28"> <query name="Personal Queries/ DEIwPrompts" wait="yes"> <prompt1>cq_ user</prompt1> <prompt2 operator="Does Not Equal">Closed</prompt2> </query> </ClearQuest> <!-- CQ/XML RETURNS --> <ClearQuest db="dev28" login="cq_ user" cqtan="06318n000004"
client="w35d125" ip="147.34.35.125"> <query name="Personal Queries/ DEIwPrompts" status="ok"> <defect row="0"> <headline>TC01 Headline String</headline> <id>dev2800007311</id> <State>Submitted</State> </defect> <defect row="1">...</defect> <defect row="2">...</defect> </query> </ClearQuest>
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1313
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1414
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
1. accept socket
2. assign conn. id
3. fork
4. read socket
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1515
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
5. pre-parse XML
6. write queue file
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1616
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
7. parse XML
8. restructure XML obj
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1717
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
9. login to CQ
10. run CQ commands
11. logout of CQ
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1818
Architecture
ClientLive
ServerQueue
Manager
ClearQuest Request Manager
Logs
Queuesocket
XML
XML
XML
XM
L
CQ
AP
I
CQ
AP
I
Logs
XM
L
ClearQuest Web
user site
Wilsonville
system visible
user visible
12. send status
13. close socket
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
1919
Details: Live vs. Queued
Allows faster response for jobs that don’t require a return message.
Can run on a second machine to lighten the load on the live server.
Every successfully parsed job is written to disk.
Queue manager is a scheduled process that runs every 5 minutes to process queued jobs and clean the disk.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2020
Details: XML Parser
Since you can’t compile modules for cqperl/ratlperl, the only stand-alone parser available is XML::Mini.
— XML::Mini can get in some recursive loops so we pre-parse and reformat the XML before calling the parser.
— XML::Mini doesn’t differentiate between attributes and elements so we made all attribute names unique from element names.
— Resulting data structure is complex and differs based on the input.
— Results may not be in the same order as the input. Our users aren’t well-versed on XML so we ignored
DTDs, external entities, parameter entities, XML Schema, etc.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2121
Details: <info…> Elements “Special functionality has been added to support
user-specific interactivity.” Can return a list of personal, public or all queries. Can return login or database information.
<ClearQuest login="cq_ user" password="* * * * * " repo="cqrepos" db="dev28"> <info type="Personal Queries" wait="yes"/> </ClearQuest> <!-- CQ/XML RETURNS --> <ClearQuest db="dev28" login="cq_ user" cqtan="06318n000005"
client="w35d125" ip="147.34.35.125"> <info type="Personal Queries" status="ok"> <queryname row="0">Personal Queries/ State & Release</queryname> ... <queryname row="36">Personal Queries/ WebQueries/ Closed</queryname> </ info> </ClearQuest>
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2222
Details: Miscellaneous
Encryption implemented via Crypt::RC4.— Stand-alone Perl package.— Very simple and fast.
Appending to XML logs requires overwriting the root element end.
Prompting dates was error prone and dropped from the implementation.
Wrap every ClearQuest call with an eval{} so a failure in ClearQuest doesn’t kill the entire server.
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2323
Findings
Stable— No application-level failures encountered
through seven months of testing and three months of production deployment.
Scales extremely well— Under Grid testing, the CQ/XML server was
still handling requests with over 350 near-simultaneous connections.
Response time meets user requirements— defect update jobs < 3 seconds— defect view jobs < 1 second— query jobs < 1 second
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2424
ClearCase Client
Why not use IBM ClearCase/ClearQuest integration?— Requires Native Clients
CCRC currently only supports CQ UCM integration— Does not work over WAN and MultiSite does
not work for us— ClearCase and ClearQuest user must have
same version of client installed— Not configurable
Most of the CC information captured in CQ is unusable
Need to Configure differently on a per VOB basis (different processes and requirements per group)
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2525
ClearCase Client: Implementation Details
Uses CC Triggers to send data to CQ via XML— Triggers use ratlperl/ccperl
Allows users to Associate Defect(s) to file during checkout/checkin— Uses either Command Line prompts or GUI — CC Data: File Path, Version, User, Date,
Checkin Comment Removes Defect Associations during
uncheckout CC Uses CrmRequest hyperlink to link to
Defects
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2626
DemonstrationClearQuest / ClearCase Integration via
CQ/XML
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
ClearQuest XML Server with ClearCase IntegrationNW Rational User’s Group – February 22, 2007© 2007 Mentor Graphics Corporation, All Rights Reserved
2727