acknowledgements - monash university...overview of eclipse and eclipse parallel tools platform (ptp)...

Acknowledgements ª  Portions of this material are supported by or based upon work supported by the

Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR0011-07-9-0002, the United States Department of Energy under Contract No. DE-FG02-06ER25752, the Blue Waters sustained petascale computing project, which is supported by the National Science Foundation under award number OCI 07-25070, and the SI2-SSI Productive and Accessible Development Workbench for HPC Applications, which is supported by the National Science Foundation under award number OCI 1047956

ª  The SI2-SSI team is lead by Jay Alameda (NCSA), Greg Watson (IBM), Steven Brandt (LSU), and Allen Malony (U Oregon). Team members and senior personnel include Beth Tibbitts (IBM), Ralph Johnson (U Illinois), Chris Navarro (NCSA), Sameer Shende (U Oregon), Wyatt Spear (U Oregon), Brian Jewett (U Illinois), Galen Arnold (NCSA), and Rui Liu (NCSA)

1

2

Outline ª  Overview of Eclipse and Eclipse Parallel Tools Platform

(PTP) ª  Overview of WHPC: NSF-funded SI2-SSI project to

produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª  Improvements to Eclipse PTP

ª  Software Engineering Practices Enabled by Eclipse PTP ª  Code visibility ª Multi-system build management ª  Performance tuning ª  Source code control ª  Issue Tracking ª Documentation

ª  Eclipse PTP Resources

3






4

What is Eclipse?

ª A vendor-neutral open-source workbench for multi-language development

ª A extensible platform for tool integration ª Plug-in based framework to create, integrate

and utilize software tools

5

Launching & Monitoring

Eclipse Parallel Tools Platform (PTP)

Debugging

Coding & Analysis

Performance Tuning

6

Parallel Tools Platform (PTP)

ª  The Parallel Tools Platform aims to provide a highly integrated environment specifically designed for parallel application development

ª  Features include: ª  An integrated development environment (IDE) that

supports a wide range of parallel architectures and runtime systems

ª  A scalable parallel debugger ª  Parallel programming tools

(MPI, OpenMP, UPC, etc.) ª  Support for the integration

of parallel tools ª  An environment that simplifies the

end-user interaction with parallel systems ª  http://www.eclipse.org/ptp

7






Why WHPC?

ª Stable, portable platform for tool development ª Focus on tool functionality, manage rapid evolution

of HPC platforms ª Encourage consistent tool look and feel ª Support for HPC application development practices

ª Edit, build, test, debug, maintain, for maximum developer productivity

ª Remote development, batch execution mandatory ª  Track, store, search, browse code artifact provenance ª  Share tool functionality through an integration framework ª Maintain tool identity

ª Provides for independent tool development pathways and funding

8

Why Parallel Tools Platform?

ª High potential to meet needs of a WHPC. ª Target next generation of HPC developers

growing up with IDEs (Eclipse, Visual Studio, ...)

ª For PTP to become a WHPC need to: ª Cultivate community of users ª Make substantial improvements to PTP around two

themes: ª Improving usability ª Improving productivity

9

10






Requirements and Impact

ª Application-centric approach ª Use real application codes, with PTP, on production

computational resources ª Identify specific goals to accomplish with each

application ª Use Eclipse PTP to accomplish the goals ª Identify shortcomings in Eclipse PTP that need to

be rectified for Eclipse PTP to be effective with that application workplan

ª  This is part of our project team’s responsibility ª Work with application community and learn from their

experience with Eclipse PTP

11

Requirements and Impact (2)

ª Application-centric approach ª Work with application community and learn from their

experience with Eclipse PTP ª Bridge to TeraGrid and (now) XSEDE Advanced

User Support ª Work with targeted organizations to assist with

adoption of PTP ª Monthly user calls ª Annual user group meeting ª Hands on tutorials ª Conference Birds of a Feather

12

13






Improvements

ª Work within Eclipse release cycle ª Major (API-breaking) improvements with coordinated

June release ª Last major release Eclipse 4.2 “Juno” released June 27,

2012 ª Minor enhancements and bug-fixes with two

coordinated service releases in September and February ª Eclipse 4.2 SR1 released Sept 28, 2012

ª Foci of improvements ª Improve usability ª Improve productivity

14

Improve Usability

ª Remote support and scalability enhancements ª Broaden support of remote capabilities to full PTP ª Provide for easy platform configuration management ª Provide additional remote features

ª Automatic remote service deployment ª Multiple authentication mechanism ª Support wide range of resource managers ª Full remote debug support

15

Improve Usability

ª Integration with other tools ª Improve External Tools Framework (ETFw)

ª Full remote support ª Integration of tool output with Eclipse views

ª Improve and broaden parallel paradigm support ª Driven by user needs and feedback

16

Improve Productivity

ª Provide support for performance driven refactoring

ª Track source and executable code provenance

17

Significant Recent Improvements

ª User-configurable machine configuration ª Wide variety of configurations now available: ª Documentation, tutorial at

ª http://wiki.eclipse.org/PTP/designs/Resource_Manager_Configuration

18

Scalable System Monitoring ª  System view

ª  Jobs running on system

ª  Active jobs

ª  Inactive jobs

ª  Messages

ª  Console

Run-19 Running an Application

Sync-20

Synchronized Projects ª  Projects types can be:

-20

File Service Index Service

Launch Service

Build Service

Debug Service

Local source code

Source code copy

Local Remote

Run

Debug

Compute

Edit Search/Index NavigaAon

Synchronize

Executable

Synchronized Projects

Blue Waters Enhancements

•  Blue Waters: Cray XE6/XK7 at NCSA •  PTP did not work with Crays “out of the box”

•  Could not submit jobs with appropriate aprun options •  Could not monitor status of compute nodes •  Could not set environment modules for build •  Did not recognize Cray, PGI compilers’ errors messages •  Did not support OpenACC •  …

•  Less than 6 months to fix these for PTP 6.0 (!)

21 The Eclipse Parallel Tools Platform


Integrated OpenACC documentation and PLDT support (added for BW)

Documentation also available for MPI, OpenMP

23 The Eclipse Parallel Tools Platform 23 The Eclipse Parallel Tools Platform Code completion for OpenACC directives (added for BW)


After the build, compiler errors, warnings, and loopmark information are shown in the Problems view and source code editor

(Cray, PGI support added for BW)

25 The Eclipse Parallel Tools Platform Graphical interface for launching a job (customized for BW)

Additional Plug-ins from NCSA

ª NCSA publishes additional plug-ins can be added onto an existing PTP installation ª  http://forecaster.ncsa.uiuc.edu/help/index.jsp

ª Contribute a System menu to the menu bar with XSEDE- and NCSA-specific commands

NCSA-26 Advanced Features: NCSA/XSEDE

System Menu

ª  Open Web content in Eclipse: ª  Open XSEDE User Portal

ª  Open User Guide for a machine

ª  Open an SSH terminal (as an Eclipse view)


Eclipse-integrated SSH terminals are provided by the Remote System Explorer (RSE), one of the features that is included in the Eclipse for Parallel Application Developers package.

System Menu

ª  Shortcuts for common PTP tasks: ª  Add Remote Environment adds a

Remote Tools connection for a particular machine

ª  Add System Monitor opens the System Monitoring perspective and begins monitoring a particular machine


System Menu

ª  The plug-in is preconfigured with information about XSEDE and NCSA resources

ª  The bottom four commands generally prompt for a system

ª  Select System can be used to eliminate this prompt, so these commands always act on a particular system


MyProxy Logon

ª  MyProxy Logon allows you to authenticate with a MyProxy server ª  Often myproxy.teragrid.org

ª  It stores a “credential,” which is usually valid for 12 hours

ª  During these 12 hours, SSH connections to XSEDE resources will not require a password; they can use the stored credential ª  However, you must enter the

correct username for that machine!


Improvements in the works…

ª External Tools Framework (ETFW) being transitioned to use the resource manager “JAXB” XML descriptions. ª Enhanced flexibility for tool integration

ª Environment Management enhancements ª Ordering of modules (important for some machines) ª Evaluate proper role and relationship between

environment management, scanner-discovery and tool chains ª Would like to properly set remote include paths

(semi)-automatically! ª Separate Build system from CDT build

ª Cleaner multi-system build management ª Builds in any language

Improvements in the works…

ª External Tools Framework (ETFW) being transitioned to use the resource manager “JAXB” XML descriptions. ª Enhanced flexibility for tool integration

ª Environment Management enhancements ª Ordering of modules (important for some machines) ª Evaluate proper role and relationship between

environment management, scanner-discovery and tool chains ª Would like to properly set remote include paths

(semi)-automatically! ª Separate Build system from CDT build

ª Cleaner multi-system build management ª Builds in any language

Increasing ambitiousness!

33






Software Engineering

ª Code Visibility

34



35

Code navigation



36

Code navigation

Syntax-aware editing (navigate to program units and declarations)



37

Code navigation

Syntax-aware editing (navigate to program units and declarations)

Code Outline


ª Code visibility: deducing call hierarchy

38

Would like to understand call hierachy of this code in relation to “main()” in startup.c

Software Engineering: Call Hierarchy (C/C++)

ª After selecting main, right click and select <Open Call Hierarchy>

39

40






Multi-machine build management

ª Local ª Source is located on local machine, builds happen

locally ª Synchronized

ª Source is local, then synchronized with remote machine(s)

ª Building and launching happens remotely (can also happen locally)

ª Remote ª Source is located on remote machine(s), build and

launch takes place on remote machine(s)

41

Synchronized Projects ª  Projects types can be:

42

File Service Index Service

Launch Service

Build Service

Debug Service

Local source code

Source code copy

Local Remote

Run

Debug

Compute

Edit Search/Index NavigaAon

Synchronize

Executable

43






Performance Tuning: PTP TAU plug-ins http://www.cs.uoregon.edu/research/tau

ª  TAU (Tuning and Analysis Utilities) ª  First implementation of External Tools Framework (ETFw) ª  Eclipse plug-ins wrap TAU functions, make them available

from Eclipse ª  Full GUI support for the TAU command line interface ª  Performance analysis integrated with development

environment

44

45






Source Code Control: “Team” Features

ª Eclipse supports integration with multiple version control systems (VCS) ª CVS, SVN, Git, and others ª Collectively known as “Team” services

ª Many features are common across VCS ª Compare/merge ª History ª Check-in/check-out

ª Some differences ª Version numbers ª Branching

46

CVS Features

ª Shows version numbers next to each resource

ª Marks resources that have changed ª Can also change color (preference

option) ª Context menu for Team

operations ª Compare to latest, another

branch, or history ª Synchronize whole project (or any

selected resources) 47

48






Issue Tracking

ª Mylyn Bridge ª Tracks tasks, links to source and bug repositories

49

Connections to Jira, bugzilla, …

50






Eclipse Documentation

ª Eclipse Help System – built in and standalone (http://help.eclipse.org)

51

Adapting Eclipse Documentation to Other Projects: QMCPack

ª See http://code.google.com/p/qmcpack-doc/

52

53






Online Information

ª  Information about PTP ª Main web site for downloads, documentation, etc.

ª http://eclipse.org/ptp ª Wiki for designs, planning, meetings, etc.

ª http://wiki.eclipse.org/PTP ª  Articles and other documents

ª http://wiki.eclipse.org/PTP/articles

ª  Information about Photran ª Main web site for downloads, documentation, etc.

ª http://eclipse.org/photran ª  User’s manuals

ª http://wiki.eclipse.org/PTP/photran/documentation

54

Mailing Lists

ª  PTP Mailing lists ª Major announcements (new releases, etc.) - low volume

ª http://dev.eclipse.org/mailman/listinfo/ptp-announce ª  User discussion and queries - medium volume

ª http://dev.eclipse.org/mailman/listinfo/ptp-user ª Developer discussions - high volume

ª http://dev.eclipse.org/mailman/listinfo/ptp-dev

ª  Photran Mailing lists ª  User discussion and queries

ª  http://dev.eclipse.org/mailman/listinfo/photran ª Developer discussions –

ª  http://dev.eclipse.org/mailman/listinfo/photran-dev

55

Getting Involved

ª See http://eclipse.org/ptp ª Read the developer documentation on the wiki ª Join the mailing lists ª Attend the monthly developer meetings

ª Conf Call Monthly: Second Tuesday, 1:00 pm ET ª  Details on the PTP wiki

ª Attend the monthly user meetings ª Teleconference Monthly ª Each 4th Wednesday, 2:00 pm ET ª Details on the PTP wiki

PTP will only succeed with your participation!

56

acknowledgements - monash university...overview of eclipse and eclipse parallel tools platform (ptp)...

Documents