open grid computing environments tutorial marlon pierce, suresh marru, gopi kandaswamy, gregor von...
Post on 19-Jan-2016
227 Views
Preview:
TRANSCRIPT
Open Grid Computing Environments TutorialMarlon Pierce, Suresh Marru, Gopi Kandaswamy, Gregor von Laszewski, and Tom Scavo
Tutorial OverviewPart 1: OGCE Portal (Marlon)o Introduction and overviewo Portal Demoo How to build Grid portlets and gadgets
Part 2: OGCE Workflow Tools (Suresh and Gopi)o Part A: Demo and overviewo Part B: Hands on demo, deploying new
applications
Part 3: Web 2.0 for Grid Portals (Gregor)
Part 4: GridShib and Gateway security (Tom)
Goals of TutorialGoalso Review portal and gateway technologies old and new.o Demonstrate how to execute workflows and wrap
TeraGrid applications.o Discuss security o Hear from TG Gateway community: what do you
want?
We have a lot of material to cover so we will sample many things.o You will have enough information to get going as a
homework assignment.o Please also check with us at TG08
Please see Nancy’s “Gateway Parade” this afternoon.
Slides and Demo SiteTutorial slides are available from http://www.collab-
ogce.org
We run a permanent demo portal at https://community.ucs.indiana.edu:8443/gridsphere/o Also aliased as
https://ogceportal.iu.teragrid.org:8443/gridsphere
Demo services are also available from Communityo [NEED URLS]
Accounts train30-train49 have been created for the workshop. Password is the same as the account name.
We also have training accounts with same names that can be used to retrieve TG proxy credentials.
Portal and Gadget DemosDemonstration of basic capabilities
Getting the Code
What You Can DownloadOGCE Portalo Various portlet samplers
Web and Grid Serviceso XRegistry: Web Service
Registryo GFAC: Application Factory
Serviceo Grid Portal Information
Repository
Programming Librarieso Java and JavaScript COGo Grid Tag Libraries and
Beans (GTLAB)o Information Service Client
APIs
Project Release ScheduleWe have two major releases per year with
intermediate updates.o TeraGrid (June)o Supercomputing (November)
Focus of this release cycle: the workflow suite tools shown in Part 2.
We do not currently package services and portlets together. o These are separate downloads.
Project Build and TestSource Code Version Control
and Building: We use Maven 2 and SourceForge’s SVNo Anonymous checkoutso We are Open Source
Nightly Build Testing: We use the NMI Build and Test facility at UWo Mac and Linux flavors
System Testing: We use Apache JMeter to verify portal builds function correctly.o You can download this from
uso You can also view the nightly
dashboard
Project InformationWeb site: http://www.collab-ogce.org
Announcement Blog: http://collab-ogce.blogspot.com
RSS/Atom Feeds: http://collab-ogce.blogspot.com/atom.xml oThis is also embedded in the wiki as
“News”
Contact us: discuss@ogce.org
Building the Portal
Choosing a Host ComputerSelect your host computer.o You will need a real IP addresso VPN also OK (?)o Know general Globus et al firewall issues.
I recommend turning firewalls and SELinux off for development.
Operating Systemso We test with Mac OS X and many Linux flavorso Other Unix should be fine (assuming Java works)o We have not had any requests for Windows.
Software Environmento The portal download comes with everything you need
EXCEPT Java. You need the JDK, not the JRE.o We test with Java 1.5. Don’t use Java 1.4.
The more memory, the better.
Downloading and Building the PortalDownload the portal from
http://www.collab-ogce.org.o Unpack in $HOME.
You should also download the Maven repository.o This is optional but
recommended
Edit ogce-portal-home/pom.xml
Run the command “mvn clean install”.o Use “mvn –o clean install”
to build offline.
Build will typically take 5 minutes (new Mac) to 30+ minutes (old Linux PC).
You can also check out the latest code from SVN. Check the “Nightly Build” dashboard to verify.
Practical ConsiderationsIncrease your JVM Memoryo export JAVA_OPTS="-server -Xms512m -
Xmx1024m -XX:MaxPermSize=256m”o Xmx2048m, 4096m, etc if you can.
Monitor catalina.out for problemso tail –f catalina.out
Make sure that Tomcat shuts downo Kill if necessary
NFS can really slow down buildingo Use /usr/local/ if you have problems
Editing pom.xmlYou only need to edit the
top level ogce-portal-only/pom.xml.
You only need to change <properties> for local settings.
You MUST update the IP address.
You may want to add or remove GRAM and GridFTP hosts.
Update project.home if you are building someplace besides $HOME.
<properties> <portal.server.ip> 156.56.104.143 </portal.server.ip>
<host.base.url> http://${portal.server.ip}:8080/ </host.base.url> <gridftp.host.names> …. </gridftp.host.names> <gram.host.names> …. </gram.host.names>
<project.home> ${env.HOME}/ogce-portal-only </project.home>
</properties>
Rebuilding the PortalRunning “mvn clean install” will completely
wipe out the existing installation.o You may not want to do this.
If you want to rebuild individual components, just run the following command from the ogce-portal-only directory.o mvn clean install –f
portlets/myportlet/pom.xml
You can do this for all <modules> in ogce-portal-deploy/pom.xml
OGCE Portal Project LayoutWe follow standard Maven 2
conventions for project layouts for each component.omoduleName/src/main/webappomoduleName/src/main/java
Modules are listed on next slideoSee ogce-portal-only/pom.xml. oEach module is a separate directory
and has its own pom.xml.
Module Description
Portal_deploy This is where your portlets are deployed into Tomcat.
Containers Specific build instructions for containers. Currently we only have GridSphere 2.1.5 here.
Lnf_portal Controls look and feel for the container.
Global-config Deploys COG jars into /shared/lib and applies Tomcat custom configs.
Certificates CA certs. Includes download of all TG certs.
Portlets Each portlet has its own module.
Developing Grid User Interface ComponentsPortlets, gadgets, etc.
Under the Hood: Building ComponentsOGCE portlets use the Java COG Abstraction Layer to
access Grid services.o Provides a buffer over different Grid toolkits and
versions.o Provides ability to compose graph-based workflows.o Tutorial in itself
Portlets can be built with our Apache Velocity bridge.o “Template” or “steal this code” approach.
We have also developed Grid Tag Libraries and Beans (GTLAB) to simplify this process.
GTLAB FeaturesExtends Java Server Faces.o Tag components wrap major COG Abstraction
Layer features
Allows you to do standalone development.
Use JSF portlet bridge to convert into portlets No new coding, just change, add config files and jars.
Or forget about portlets and o Convert into Google Gadgetso Develop as a Facebook applicationo Etc.
Installing GTLABYou need to install the OGCE portal o GTLAB depends on many shared jars.
Download the release code from www.collab-ogce.org or check out from SVN:o svn checkout
https://ogce.svn.sourceforge.net/svnroot/ogce/GTLAB
Edit GTLAB/jsf_standalone/pom.xml o <catalina.home> should point to your OGCE portal server.
Edit GTLAB/jsf_standalone/src/main/resources/resources.properties.
Run “mvn clean install” in GTLAB/jsf_standalone/
Levels of Abstraction in GTLAB code
You can use prepackaged gadgets/portlets.
You can mix and match repeatable sections of tags via “include” statements.
You can develop new applications using existing tag libraries.
You can hack the codeNote JSF is compatible with JSP, so you can
mix and match.
And you can embed JavaScripto YUI, Scriptaculous user interface goodies.o Google and Facebook APIs.
MyProxy Example<h:form id="form"> <h:outputText value="Username:"/> <h:inputText value="#{resource.username}" required="true"/>
<h:outputText value="Password: "/> <h:inputSecret value="#{resource.password}" required="true"/>
<h:outputText value="MyProxy Server: "/> <h:inputText value="#{resource.myproxyHostname}" required="true"/>
<o:submit id="submit" value="Submit" binding="#{builder.body}" action="#{builder.junkAction}" >
<o:multitask id="multi" persistent="true" taskname="#{resource.taskname}"> <o:myproxy id="mypr" hostname="#{resource.myproxyHostname}"
lifetime="2" password="#{resource.password}" port="7512"
username="#{resource.username}"/> </o:multitask> </o:submit> </h:form>
Show the Proxy Credential<h:dataTable value="#{myproxy.storedProxyList}" border="1"
binding="#{myproxy.storedProxyTable}" var="proxyEntry" id="proxyTable">
<h:column> <f:facet name="header"> <h:outputFormat id="subject" escape="false”
value="Subject"/> </f:facet>
<h:outputText id="proxySubject" value="#{proxyEntry.subject}"/> </h:column>
<h:column> <f:facet name="header"> <h:outputFormat id="Remove" escape="false”
value="Delete"/> </f:facet>
<h:commandButton id="deleteProxy" value="Remove” actionListener="#{myproxy.deleteCredential}"/>
</h:column></h:dataTable>
Executing Multiple Task<o:submit id="submit" value="Submit" binding="#{builder.body}"
action="#{builder.junkAction}" > <o:multitask id="multi" persistent="true"
taskname="#{resource.taskname}"> <o:jobsubmit id="js" myfaces="true"
arguments="#{resource.arguments}" executable="#{resource.executable}" hostname="#{resource.hostname}" provider="#{resource.provider}" stdout="#{resource.stdout}"/>
<o:filetransfer id="ft" myfaces="true" from="#{resource.from}"
to="#{resource.to}" /> <o:dependency id="dep" task="ft"
dependsOn="js"/> </o:multitask></o:submit>
GTLAB Example Description
MyProxyExample Get a credential and store in OGCE global proxy store.
JobSubmitExamples
Various examples showing how to submit a grid job and get output.
Dashboard Example
Runs showq on Big Red and formats output.
DirectoryBrowser Displays and downloads files on a remote host via GridFTP
MultipleTaskExample
Shows how to couple file transfers, operations, and
TaskMonitorExample
View task state and metadata.
These will work as gadgets, portlets, or standalone applications
Making GTLAB Gadgets into PortletsDo all development in jsf_standalone.
When you are ready to convert:o cd GTLAB/transitiono mvn clean process-resourceso cd GTLAB/portletso mvn clean install
You will need to manually edit portlet.xml, group.xml, and layout.xml in GTLAB/portlet/src/main/webapp/WEB-INF/
That’s it.
Making GTLAB Google GadgetsA Gadget is a standalone application that is integrated
into your personal iGoogle display.o They can run on your Web server.o You can use SSL/HTTPS
Google Gadgets have two versionso http: these use Google’s JavaScript APIso url: these wrap external applications with IFrames.
Limitations:o Not much real estate (but location, location,
location) o Cookies manage state on gadgets from the same
server but we need a way to integrate multiple independent servers.
Some GTLAB Gadgets
Example Gadget Config File
<?xml version="1.0" encoding="UTF-8" ?> <Module> <ModulePrefs title="MyProxy Gadget Example"
scrolling="true" height="500"/>
<Content type="url" href="http://…/:8080/GTLAB/examples/
MyProxyExample.jsf"> </Content> </Module>
Save this as MyProxy.xml and place in a Web accessible place.
Some Web 2.0 ResourcesSee recent tutorials by Geoffrey Fox and me:o http://grids.ucs.indiana.edu/ptliupages/
presentations/Web2.0Tutorial.ppto http://grids.ucs.indiana.edu/ptliupages/
presentations/CTSTechFutures_May19-08.pptxo Includes material on Google Gdata, Facebook
API, Open Social, Microformats, etc.
See also full examples at http://communitygrids.blogspot.com
TeraGrid Information Services: Portlets and Gadget LibrariesSlides courtesy of Maytal Dahan
TG Information ServicesWhat are TG Information Services?o Services that publish metadata about
TeraGridd TG Software, TG resources, TG Services
o Software and applications can query TG Info services for latest information
Existing TG Info Services:o Currently we are relying on the REST
interface of TG Info Services TG Also has WebMDS interfaces
o Service includes CTSS4 information
Libraries for TG Info ServicesUsing the REST Interfaces we areo Developing JavaScript Libraries that
automatically query and return service information
o Handling all the querying and xml parsing necessary and return objects through libraries
o API will include the ability to (and more) getTGSites() – Get a list of all TG sites getResoures() – Get list of resources at a site getServices() – Get list of TG resources
Late summer availability
Demo portlets available now
Sample Portlet Interfaces
Talks, Demos, and BOFs 37
Monday 1:00 pm Science Gateways on Parade (Nancy Wilkins-Diehr and a cast of 1000’s)
Tuesday 11:00 am TALK: TeraGrid Open Life Science Gateway (Wenjun Wu)
Tuesday 1:30 pm TALK: Engaging with the LEAD Science Gateway Project: Lessons Learned in Successfully Deploying Complex System Solutions on TeraGrid
Tuesday 3:30 pm TALK: Reliable and Scalable Job Submission: LEAD Science Gateway's Testing and Experiences with WS GRAM on TeraGrid Resources
Tuesday 3:30 pm TALK: The LEAD-WxChallenge Pilot Project: The Potential of Grid-Enabled Learning
Tuesday 5:30 pm BOF: TeraGrid User Portal BOF (Maytal Dahan)
Wednesday 10:30 am TALK: Workflow Infrastructure for Multi-scale Science Gateways
Wednesday 2:00 pm TALK: Increasing TeraGrid User Productivity through Integration of Information and Interactive Services (Maytal Dahan)
Wednesday 4:00 pm TALK: On-Demand Severe Weather Forecasts Using TeraGrid via the LEAD Portal
Wednesday 6:30 pm POSTER/DEMO: TeraGrid Open Life Science Gateway
Wednesday 6:30 pm POSTER/DEMO: Fault Tolerance and Recovery of Scientific Workflows in LEAD (Gopi)
top related