Top Ten Grid Questions

Written by:Jennifer M. SchopfBill NitzbergPresented by:Allen Lee

Also presenting…

“10 Things We Hate About the Grid” Also by Jennifer M. SchopfPowerpoint Presentation http://www.mcs.anl.gov/~jms/Pubs/TopTenTalk.ppt

Topics Covered

How is the Grid different? Why isn’t the Grid a reality?

The Grid isn’t a new concept

Using multiple distributed resources to cooperatively work on a single application has been around for decades Networked OS (70s) Distributed OS (80s and 90s) Heterogeneous Computing

http://www.acis.ufl.edu/hcw2006/ Parallel Distributed Computing Metacomputing

http://www.cnds.jhu.edu/research/metacomputing/

How is a grid different?

Grids focus on site autonomy Grids involve heterogeneity Grids involve more resources than just

computers and networks Grids focus on the user

Grid Criticisms…

“The Grid is a solution looking for a problem.”

Grid Criticisms…

“We tried to install Globus and found out that it was too hard to do. So we decided to just write our own...”

#1 Why don’t Grids have basic functionality yet? Grids aren’t being used for what they were

“meant to be”“Application developers aren’t using multiple

machines at geographically distant sites to coordinate and solve a single application.”

Used to solve embarrassingly parallel applications such as Seti@home and Entropia

#1 Why don’t Grids have basic functionality yet? What is functionality?

The ability to run an application in a straightforward manner using single commands for a user to just run.

#1 Why don’t Grids have basic functionality yet? Running a process isn’t simple Steps:

Determine what software is on the machine and install it, then check it

Make sure all security credentials are setup (certificates, keys, accounts, etc.)

Determine availabile resources Submit to a Grid computer (set up environment) Run Cleanup

#1 Why don’t Grids have basic functionality yet? “The few current users have had to go through

heroic measures to achieve any functionality at all.”

Complicated due to socio-political problems It is difficult setting up and maintaining a piece of

new software on one system let alone many systems

Resistance from users who would have to learn a new environment

Moral #1: Before we can have a successful Grid, we must have a functional Grid

Grid Criticisms…

“Cynics reckon that the Grid is merely an excuse by computer scientists to milk the political system for more research grants so they can write yet more lines of useless code” – Economist, June 2001

#2 Why aren’t there more Grid application developers and users? There is a need for user input to guide research,

but users are hard to find and input is hard to receive despite large communities at NSF supercomputer centers

Groups are running specialized code targeted to specific platforms

Even when they use the Grid, they use it “improperly” they use it to gain access to specific machines, not as the Grid was envisioned.

#2 Why aren’t there more Grid application developers and users? Making apps Grid enabled is seen by

some as a distraction from getting real science done

Grid work is a buzzword for funding, but takes away from other software development work

Moral #2: Without users there can be no Grid Better software tools to ease transition to a

new environment Standards for uniform access to Grid

services Deployment of Grid software must be

made easier

#3 Where are the Grid software tools to aid application developers? Parallel programming was difficult so few

people attempted it until debuggers, fast compilers and other flexible tools were available

There is active work on a set of software tools for the Grid such as Globus, however it has a steep learning curve

#3 Where are the Grid software tools to aid application developers? This is due to socio-political problems

Most projects are academic Little or no funding funding for hardening code

Chicken and Egg problem To make tools, you need use cases Without tools to use, you don’t know what to ask for Constructing a reasonable use case is difficult, it

needs specific details without too much overwhelming detail, especially if you don’t know what is possible

#3 Where are the Grid software tools to aid application developers? Different language/communication styles

Computer Science-Centric projectsApplication-Centric

Moral #3: The Grid needs tools that facilitate the use of a Grid

#4 How do we make Grids secure?

For many systems, the mandatory “security is important” paragraph is all the depth problem receives

GSI is accepted security infrastructure and is used in things such as CVS, OpenSSH

However, that only resolves Authentication

#4 How do we make Grids secure?

Local accounts are still needed and the issue of authorization and usage policy is not address uniformly

Moral #4: Without a security infrastructure(including higher-level tools) users will not take advantage of the Grid

#5 How can we define standard interfaces and definitions for the Grid?

No simple answer for the following questions:How do I run a job on the Grid?What sort of monitoring is available?Where do I get information about the Grid

resources?How do I make sure this operation is secure?

#5 How can we define standard interfaces and definitions for the Grid?

A lingua franca needs to be created to promote greater interoperability between current toolsLack of agreement between developers over

simple terms such as: Job Resource

#5 How can we define standard interfaces and definitions for the Grid?

Example Globus Metacomputing Directory Service (MDS) and

Grid Monitoring Architecture (GMA) MDS concentrates on resource discovery (2 protocols) GMA concentrates on the provision of data (3

implementations) However, rather than interoperate to work off each other’s

strengths, there is little collaboration between these standards, interfaces and protocols

#5 How can we define standard interfaces and definitions for the Grid?

The Global Grid Forum has standardization as a goal

But so does Peer-to-Peer Working Group and the New Productivity Initiative for example

Who will standardize between the standardization bodies?

Moral #5: The lack of standards continues to hinder the interoperability needed for the Grid

#6 How can we manage variance on the Grid? Grid resource behavior is unpredictable

Unsure of who else is using a resource and in what way

Users want not only fast execution times from their applications, but unpredictable behavior, and would be willing to sacrifice performance in order ot have reliable run times.

Moral #6: Varying behavior is a fact in the Grid and must be addressed Teach users how variance can affect their

performance Scheduling algorithms have to be made to

incorporate variance Information services have to incorporate

the handling of the additional variance information

#7 How can deployment be made easier? Deploying a test bed is difficult Firewalls are often a major stumbling block Accounting methodology

Accounts have to be made for resources that never had policies for Grid access in mind

Moral #7: If the deployment and setup of software isn’t seamless, getting users to adapt their systems to be Grid compatible will not happen

Halfway solutions can be worse than no solution

#8 Where are the benefits to encourage sharing on the Grid? Grid computing has high appeal to the end

user and to the top-level policy makersEnd Users: access to resources spread out

geographically and across several domainsTop-Level: Enterprise-wide organization policy

(NASA could direct 90% of its resources to deflecting near-Earth collision)

#8 Where are the benefits to encourage sharing on the Grid? Tragedy of the Commons

Users don’t play nice; they will act in their own interests and degrade and overuse any shared resource

Middle-Level resource owners and managers are caught in the middle Users will exploit Grid technology increasingly to consume more

resources Grid accounting can’t provide the accounting or control needed

so ssers end up getting a free ride How can Middle-Level managers prove their value to

policymakers while controlling user consumption?

Moral #8: Benefits to middle-level resource ownersk must be made clear in order fo the Grids to be deployed

#9 How can we fund the work needed for a functioning Grid? Work to coordinate sites and resources are seen

as not new enough so it doesn’t get funding Fund groups to work on the same problem but

using different methods. Then fund to combine them. Funding several groups is seen as wasteful

Tensions caused by fight between Best Approach and Standard

Moral #9: Without support for basic work in functionality, standards and software engineering, the Grid will not live up to its potential

Funding agencies need to be educated on the needs of Grid researchers

Money should be made for mundane tasks to get software beyond beta versions, standardization, setting up test beds, functionality

#10 Where are the performance metrics for success? Success can not be determined without

metrics to compare a project against

Moral #10: There isn’t a moral

Proposed Success MetricsNo more “Grid” papers

Instead, papers with the following footnote, “This work was achieved using the Grid”

When supercomputer centers don’t give a user the choice of using their machines or the Grid

They will just use the Grid

Moral #10: There isn’t a moral

Proposed Success Metrics (Cont.)When a SuperComputing demo can be run

any time of the year The SuperComputing demo runs every year since

1988 in November for one week “For one week every year, it’s the world’s fastest

nework, providing direct connectivity to a multitude of worldwide research networks.” Source: SC|05 http://sc05.supercomputing.org

Agnostic 1

In the introduction, the authors point out some differences between old distributed systems and today’s Grids. Are these all the major differences? If not, can you suggests others grid distinctive characteristics?

Agnostic 2 - “1. Why don’t Grids have basic functionality yet?”

According to the authors, grids do not have basic functionality because to run a grid application one has to follow a sequence of intricate steps, like installing software, setting security, finding the appropriate resource and submitting a job, which look daunting and extremely complex. Do you think this is a fair description as for users running applications on the grid? How is the Service Oriented Architecture helping improve that scenario?

Agnostic 3 - “3. Where are the Grid software tools to aid application developers?”

Do you agree with the statement that the difficulty of developing useful tools for the grid is because there are not well defined use cases? Is it really a chicken and egg problem?

Agnostic 4 - “4. How do we make Grids secure?” It is mentioned that the authorization issue has not been

fully addressed, and typically imply using a local account to access a particular resource. This would difficult tasks like auditing the usage of a certain resource. Could you provide some information as for the advances in this matter?

Another aspect of grid security is reliance on the grid services. A service can mimic a real application and obtain sensitive information from the users. What work has been done in this respect?

Agnostic 5 - “5. How can we define standard interfaces and definitions for the Grid?”

It is mentioned during the introduction that none of the standards defined at the time of writing the paper (2002) were standing still. Is this assertion still true?

Could you briefly talk about the current status of the Global Grid Forum and the Enterprise Grid Alliance groups as for grid standards?

Agnostic 6 -“8. Where are the benefits to encourage sharing on the Grid?”

Could you suggest some ways to enforce the user to “play nicely”? In particular, how to manage multiple user applications that can potentially exhaust the grid resources?

Also, can you think of good reasons to convince managers for sharing resources?

Agnostic 7 -“10. Where are the performance metrics for success?”

The second success metric says that users are not given a choice of using or not the grid, but just use it. What is your view of this statement? Aren’t the authors assuming that all problems are suitable to get solved in a grid environment?