N Tropy: A Framework for Knowledge Discovery in a

Virtual Universe

Harnessing the Power of Parallel Grid Resources for Astrophysical

Data AnalysisJeffrey P. GardnerJeffrey P. Gardner

Pittsburgh Supercomputing CenterPittsburgh Supercomputing CenterUniversity of PittsburghUniversity of Pittsburgh

Carnegie Mellon UniversityCarnegie Mellon University

Mining the Universe can be (Computationally) Expensive

Computational Astrophysics: Size of simulations are frequently

limited by inability to process simulation output (no parallel group finders).

Current MPPs are 1000’s of processors…this is already too large for serial processing.

Next generation of MPPs will be 100,000’s of processors!

Mining the Universe can be (Computationally) Expensive

Observational Astronomy: Paradigm shift is astronomy: Sky Surveys Observers now generates ~ 1TB data per

night With Virtual Observatories, one can pool data

from multiple catalogs. Computational requirements are growing at a

faster rate than computational power. There will be some problems that would be

impossible without parallel machines. There will be many problems for which

throughput can be substantially enhanced by parallel machines.

The Challenge of Parallel Data Analysis

Parallel programs are hard to write! Steep learning curve to learn parallel programming Lengthy development time

Parallel world is dominated by simulations: Code is often reused for many years by many

people Therefore, you can afford to spend lots of time

writing the code. Data Analysis does not work this way:

Rapidly changing scientific inqueries Less code reuse

Data Mining paradigm mandates rapid software development!

Tightly-Coupled Parallelism(what this talk is about)

Data and computational domains overlap Computational elements must

communicate with one another Examples:

Group finding N-Point correlation functions New object classification Density estimation

Solution(?): N Tropy

The Goal

GOAL: Minimize development time for parallel applications.

GOAL: Enable scientists with no parallel programming background (or time to learn) to still implement their algorithms in parallel.

GOAL: Provide seamless scalability from single processor machines to MPPs…potentially even several MPPs in a computational Grid.

GOAL: Do not restrict inquiry space.

Methodology Limited Data Structures:

Astronomy deals with point-like data in an N-dimension parameter space

Most efficient methods on these kind of data use trees. Limited Methods:

Analysis methods perform a limited number of fundamental operations on these data structures.

N Tropy DesignPKDGRAV already provides a number of

advanced services PKDGRAV benefits to keep:

Flexible client-server scheduling architecture Threads respond to service requests issued by master. To do a new task, simply add a new service.

Portability Interprocessor communication occurs by high-level

requests to “Machine-Dependent Layer” (MDL) which is rewritten to take advantage of each parallel architecture.

Advanced interprocessor data caching < 1 in 100,000 off-PE requests actually result in

communication.

N Tropy New Features

Dynamic load balancing (available now) Workload and processor domain boundaries

can be dynamically reallocated as computation progresses.

Data pre-fetching?? (To be implemented) Predict request off-PE data that will be

needed for upcoming tree nodes. mdlSlurp(): Prefetch a big block of data to a special

block of local memory Intelligent prediction: Investigate active learning

algorithms to prefetch off-PE data.

Performance

10 million particlesSpatial 3-Point3->4 Mpc

(SDSS DR1 takes lessthan 1 minute with perfect load balancing)

PHASE 1 Performance

10 million particlesProjected 3-Point0->3 Mpc

NTropy Conceptual Schematic

Computational Steering LayerC, C++, Python (Fortran?)

Framework (“Black Box”)

User serial collective staging and processing routines

Web Service Layer (at least from Python)

Domain Decomposition/Tree Building

Tree TraversalParallel I/O

User serial I/O routines

VO

WSDL?SOAP? Key:

Framework ComponentsTree ServicesUser Supplied

Collectives

Dynamic Workload Management

User tree traversal routines

User tree and particle data

N Tropy Design Use only Parallel Management Layer (pst) and

Interprocessor Communication Layer (mdl) of PKDGRAV.

Rewrite everything else from scratch

Computational Steering Layer

Parallel Client/Server Layer(pst)

Serial Layer

Gravity Calculator Hydro Calculator

PKDGRAV Functional Layout

Executes on master processor

Coordinates execution and data distribution among processors

Executes on all processors

Interprocessor Communication Layer(mdl)

Passes data between processors

N Tropy Functional Layout

NTropy Computational Steering API

Thread services layer (executed on all threads)

PKDGRAV Parallel Client/Server Layer

“User” Supplied Methods

Executed Executed on master on master thread thread onlyonly

MDL Interprocessor communication

layer

Layers retained from PKDGRAVLayers completely rewritten

Key:

Application Steering script

Application “Tasks” (serial code)

Tree node and particle navigation methods:

(e.g. Acquire node/particle, next node/particle, parent node, child node

Executed Executed on all on all threadsthreads

ntropy_* functions

ntp_* functions

Using Ntropy (example: N-body)Computational Steering Layer (nbody.c)

#include “ntropy.h” /* Mandatory */#include “nbody.h” /* My application-specific stuff */

int main(int argc,char **argv) { NTROPY ntropy; /* Mandatory */ struct nbodyStruct nbody; /* My App-specific struct */

/* Start ntropy. Only the master thread will ever return from this function. * The slave threads will return to "main_ch" in ntropy.c. */ ntropy = ntropy_Initialize(argc, argv);

/* Process command line arguments */ /* Add nbody-specific command line arguments */ nbodyAddParams(ntropy, &nbody); /* Read in command line arguments */ ntropy_ParseParams(ntropy, argc, argv); /* Now that the command-line parameters have been parsed, we can * examine them. */ nbodyProcParams(ntropy, &nbody);

/* Open log file and write header */ fpLogFile = nbodyLogParams(ntropy, &nbody);

NTropy stuffApp-specific stuff

Using Ntropy (example: N-body)Computational Steering Layer (nbody.c)

/* Start threads by calling ntropyThreadStart. This starts NTropy computational services on all compute threads, pushes the data in Global and Param structs to all threads, and runs the function tskLocalInit on all threads. tskLocalInit is the constructor for your thread environment, and is convenient if you want to set up any local structs. You can also provide tskLocalFinish which is destructor for your thread environment. */ nThreads = ntropy_ThreadStart(ntropy, &(nbody.global), sizeof(nbody.global), &(nbody.param), sizeof(nbody.param), (*tskLocalInit), (*tskLocalFinish));

And in task_fof.c:void *tskLocalInit(NTP ntp) { struct nbodyLocalStruct *local; /* A struct that I invent that will store * all “thread-global” variables that I * will need for this computation */

local = (nbodyLocalStruct *)malloc(sizeof(struct nbodyLocalStruct)); return (void *)local;}

void tskLocalFinish(NTP ntp) { struct nbodyLocalStruct *local = ntp_GetLocal(ntp); free(local);}

Using Ntropy (example: N-body)Computational Steering Layer (nbody.c)

N-body structs:/* This struct has everything that I read in from the command line */struct nbodyParamStruct { double dOmega0; int nSteps; …}/* This struct has everything that I don’t read in from the command line, but still want all threads to know */struct nbodyGlobalStruct { double dRedshift; double dExpansionFactor; …}/* This struct has everything that I was store store locally in each compute thread. */struct nbodyLocalStruct { int nParticleInteractions; int nCellInteractions; …}

Using Ntropy (example: N-body)Computational Steering Layer (nbody.c)

Main N-body loop:

nbody->iStep = 0;nbody->iOut = 0;while(nbody->iStep==0 || nbody->iOut < nbodySteps(&nbody)) { /* Build the tree */ ntropy_BuildTree(ntropy, (*tskCellAccum), (*tskBucketAccum));

/* Do the Gravity walk using dynamic load balancing */ ntropy_Dynamic(ntropy, (*tskGravityInit), (*tskGravityFinish), (*tskGravity)); /* Or, if you don’t want dynamic load balancing for this function, use ntropy_Static() */

/* Write results if needed (nbodyWriteOutput checks if it needs to output. If so, it calls ntropy_WriteParticles(), then increments msr->iOut.) */ nbodyWriteOutput(ntropy, &nbody); ++(msr->iStep);}

/* Finish */ntropy_Finish(ntropy);

Using Ntropy (example: N-body)Task Layer (nbody_task.c)

void tskGravity(NTP ntp, NS *pNodeSpecStart) { /* Get all the structs that I will need */ struct nbodyParamStruct *param = ntp_GetParam(ntp); struct nbodyGlobalStruct *global = ntp_GetGlobal(ntp); struct nbodyLocalStruct *local = ntp_GetLocal(ntp); NS nodeSpecBucket; /* Handle for the current bucket node */ NS nodePtrBucket; /* Pointer to the data of the current bucket */ NS nodeSpecDone; /* The handle of the node that is “next” for nodeSpecStart. * When we walk to this node, we will be done. */

ntpNS_Next(ntp, pNodeSpecStart, &nodeSpecDone); ntpNS_Copy(ntp, pNodeSpecStart, &nodeSpecBucket); /*Find each bucket that is in pNodeSpecStart’s domain and do a treewalk for it*/ while(ntpNS_Compare(ntp, nodeSpecBucket, nodeSpecDone) { AQCUIRE_NODEPTR(ntp, nodePtr, nodeSpec); while(ntpNP_Type(ntp, nodePtr) != NTP_NODE_BUCKET) { ntpNS_ReplaceLower(ntp, &nodeSpec, nodePtr); ACQUIRE_NODEPTR(ntp, nodePtr, nodeSpec); } /* Now nodeSpecBucket and nodePtrBucket are pointing at the next bucket to process. */ MyWalkBucket(ntp, nodeSpecBucket, nodePtrBucket); }}

Using Ntropy (example: N-body)Task Layer (nbody_task.c)

void MyWalkBucket(NTP ntp, NS nodeSpecBucket, NP nodePtrBucket) {

NS nodeSpec; /* Handle for the node currently being looked at */ NP nodePtr; /* Pointer to the data of the node currently being looked at */ ntpNS_PstStart(ntp, &nodeSpec); /* Get the handle of the root of the PST */ while (iWalkResult != NTP_NULL_PST) { ACQUIRE_NODEPTR(ntp, nodePtr, nodeSpec); iInteractResult = MyCellCellInteract(ntp, nodeSpecBucket, nodePtrBucket, nodePtr, nodeSpec); if (iInteractResult == MY_CELL_OPEN) iWalkResult = ntpNS_ReplaceLower(ntp, &nodeSpec, nodePtr); else /* iInteractResult == MY_CELL_NEXT */ iWalkResult = ntpNS_ReplaceNext(ntp, &nodeSpec, nodePtr); }}

N Tropy Functional Layout

NTropy Computational Steering API

Thread services layer (executed on all threads)

PKDGRAV Parallel Client/Server Layer

“User” Supplied Methods

Executed Executed on master on master thread thread onlyonly

MDL Interprocessor communication

layer

Layers retained from PKDGRAVLayers completely rewritten

Key:

Application Steering script

Application “Tasks” (serial code)

Tree node and particle navigation methods:

(e.g. Acquire node/particle, next node/particle, parent node, child node

Executed Executed on all on all threadsthreads

ntropy_* functions

ntp_* functions

Further NTropy Features

Custom cell and particle data Selectable cache types

Read-only Combiner

Explicit control over cache starting and stopping

Cache statistics 10 user-selectable timers and 4

“automatic” timers

Further NTropy Features

Custom command-line and parameter file options

Automatic reduction variables Range of collectives (AllGather,

AllToAll, AllReduce) I/O Primitives (TIPSY “array” and

“vector” files) as well as flexible user-defined I/O.

Conclusions Most data analysis in astronomy is

done using trees as the fundamental data structure.

Most operations on these tree structures are functionally identical.

Based on our studies so far, it appears feasible to construct a general purpose parallel framework that users can rapidly customize to their needs.