Slide 1

CS6703 GRID AND CLOUD COMPUTINGUnit 4Dr Gnanasekaran ThangavelProfessor and HeadFaculty of Information TechnologyR M K College of Engineering and Technology

UNIT IVPROGRAMMING MODELOpen source grid middleware packages Globus Toolkit (GT4) Architecture , Configuration Usage of Globus Main components and Programming model - Introduction to Hadoop Framework - Mapreduce, Input splitting, map and reduce functions, specifying input and output parameters, configuring and running a job Design of Hadoop file system, HDFS concepts, command line and java interface, dataflow of File read & File write.24-Aug-162Dr Gnanasekaran Thangavel

Open source grid middleware packages24-Aug-16Dr Gnanasekaran Thangavel3The Open Grid Forum and Object Management are two well- formed organizations behind the standardsMiddleware is the software layer that connects software components. It lies between operating system and the applications.Grid middleware is specially designed a layer between hardware and software, enable the sharing of heterogeneous resources and managing virtual organizations created around the grid. The popular grid middleware areBOINC -Berkeley Open Infrastructure for Network Computing.UNICORE - Middleware developed by the German grid computing community.Globus (GT4) - A middleware library jointly developed by Argonne National Lab., Univ. of Chicago, and USC Information Science Institute, funded by DARPA, NSF, and NIH.CGSP in ChinaGrid - The CGSP (ChinaGrid Support Platform) is a middleware library developed by 20 top universities in China as part of the ChinaGrid Project

Open source grid middleware packages conti24-Aug-16Dr Gnanasekaran Thangavel4Condor-G - Originally developed at the Univ. of Wisconsin for general distributed computing, and later extended to Condor-G for grid job management.Sun Grid Engine (SGE) - Developed by Sun Microsystems for business grid applications. Applied to private grids and local clusters within enterprises or campuses.gLight -Born from the collaborative efforts of more than 80 people in 12 different academic and industrial research centers as part of the EGEE Project, gLite provided a framework for building grid applications tapping into the power of distributed computing and storage resources across the Internet.

The Globus Toolkit Architecture (GT4)24-Aug-16Dr Gnanasekaran Thangavel5The Globus Toolkit, is an open middleware library for the grid computing communities. These open source software libraries support many operational grids and their applications on an international basis. The toolkit addresses common problems and issues related to grid resource discovery, management, communication, security, fault detection, and portability. The software itself provides a variety of components and capabilities. The library includes a rich set of service implementations. The implemented software supports grid infrastructure management, provides tools for building new web services in Java , C, and Python, builds a powerful standard-based security infrastructure and client API s (in different languages), and offers comprehensive command-line programs for accessing various grid services. TThe Globus Toolkit was initially motivated by a desire to remove obstacles that prevent seamless collaboration, and thus sharing of resources and services, in scientific and engineering applications. The shared resources can be computers, storage, data, services, networks, science instruments (e.g., sensors), and so on. The Globus library version GT4, is conceptually shown in Figure

The Globus Toolkit24-Aug-16Dr Gnanasekaran Thangavel6

The Globus Toolkit24-Aug-16Dr Gnanasekaran Thangavel7GT4 offers the middle-level core services in grid applications. The high-level services and tools, such as MPI , Condor-G, and Nirod/G, are developed by third parties for general purpose distributed computing applications. The local services, such as LSF, TCP, Linux, and Condor, are at the boom level and are fundamental tools supplied by other developers.As a de facto standard in grid middleware, GT4 is based on industry-standard web service technologies.

Functionalities of GT424-Aug-16Dr Gnanasekaran Thangavel8Global Resource Allocation Manager (GRAM ) - Grid Resource Access and Management (HTTP-based)Communication (Nexus ) - Unicast and multicast communicationGrid Security Infrastructure (GSI ) - Authentication and related security servicesMonitory and Discovery Service (MDS ) - Distributed access to structure and state informationHealth and Status (HBM ) - Heartbeat monitoring of system componentsGlobal Access of Secondary Storage (GASS ) - Grid access of data in remote secondary storageGrid File Transfer (GridFTP ) Inter-node fast file transfer

Globus Job Workflow24-Aug-16Dr Gnanasekaran Thangavel9

Globus Job Workflow24-Aug-16Dr Gnanasekaran Thangavel10A typical job execution sequence proceeds as follows: The user delegates his credentials to a delegation service. The user submits a j ob request to GRAM with the delegation identifier as a parameter. GRAM parses the request, retrieves the user proxy certificate from the delegation service, and then acts on behalf of the user. GRAM sends a transfer request to the RFT (Reliable File Transfer), which applies GridFTP to bring in the necessary files.GRAM invokes a local scheduler via a GRAM adapter and the SEG (Scheduler EventGenerator) initiates a set of user j obs. The local scheduler reports the j ob state to the SEG. Once the j ob is complete, GRAM uses RFT and GridFTP to stage out the resultant files. The grid monitors the progress of these operations and sends the user a notification

Client-Globus Interactions24-Aug-16Dr Gnanasekaran Thangavel11There are strong interactions between provider programs and user code. GT4 makes heavy use of industry-standard web service protocols and mechanisms in service description, discovery, access, authentication, authorization. GT4 makes extensive use of java, C, and Python to write user code. Web service mechanisms define specific interfaces for grid computing. Web services provide flexible, extensible, and widely adopted XML-based interfaces.

Data Management Using GT424-Aug-16Dr Gnanasekaran Thangavel12Grid applications one need to provide access to and/or integrate large quantities of data at multiple sites. The GT4 tools can be used individually or in conj unction with other tools to develop interesting solutions to efficient data access. The following list briefly introduces these GT4 tools:1. Grid FTP supports reliable, secure, and fast memory-to-memory and disk-to-disk data movement over high-bandwidth WANs. Based on the popular FTP protocol for internet file transfer, Grid FTP adds additional features such as parallel data transfer, third-party data transfer, and striped data transfer. I n addition, Grid FTP benefits from using the strong Globus Security Infra structure for securing data channels with authentication and reusability. It has been reported that the grid has achieved 27 Gbit/second end-to-end transfer speeds over some WANs.2. RFT provides reliable management of multiple Grid FTP transfers. I t has been used to orchestrate the transfer of millions of files among many sites simultaneously.3. RLS (Replica Location Service) is a scalable system for maintaining and providing access to information about the location of replicated files and data sets.4. OGSA-DAI (Globus Data Access and Integration) tools were developed by the UK eScience program and provide access to relational and XML databases.

24-Aug-16Dr Gnanasekaran Thangavel13Apache top level project, open-source implementation of frameworks for reliable, scalable, distributed computing and data storage.It is a flexible and highly-available architecture for large scale computation and data processing on a network of commodity hardware.Hadoop offers a software platform that was originally developed by a Yahoo! group. The package enables users to write and run applications over vast amounts of distributed data. Users can easily scale Hadoop to store and process petabytes of data in the web space. Hadoop is economical in that it comes with an open source version of MapReduce that minimizes overhead in task spawning and massive data communication.It is efficient, as it processes data with a high degree of parallelism across a large number of commodity nodes, and it is reliable in that it automatically keeps multiple data copies to facilitate redeployment of computing tasks upon unexpected system failures.

Hadoop24-Aug-16Dr Gnanasekaran Thangavel14Hadoop:

an open-source software framework that supports data-intensive distributed applications, licensed under the Apache v2 license.

Goals / Requirements:

Abstract and facilitate the storage and processing of large and/or rapidly growing data setsStructured and non-structured dataSimple programming models

High scalability and availability

Use commodity (cheap!) hardware with little redundancy

Fault-tolerance

Move computation rather than data

Hadoop Framework Tools24-Aug-16Dr Gnanasekaran Thangavel15

Hadoops Architecture24-Aug-16Dr Gnanasekaran Thangavel16Distributed, with some centralization

Main nodes of cluster are where most of the computational power and storage of the system lies

Main nodes run TaskTracker to accept and reply to MapReduce tasks, and also DataNode to store needed blocks closely as possible

Central control node runs NameNode to keep track of HDFS directories & files, and JobTracker to dispatch compute tasks to TaskTracker

Written in Java, also supports Python and Ruby

Hadoops Architecture24-Aug-16Dr Gnanasekaran Thangavel17Hadoop Distributed FilesystemTailored to needs of MapReduce Targeted towards many reads of filestreamsWrites are more costly High degree of data replication (3x by default)No need for RAID on normal nodesLarge blocksize (64MB)Location awareness of DataNodes in network

Hadoops Architecture24-Aug-16Dr Gnanasekaran Thangavel18NameNode:Stores metadata for the files, like the directory structure of a typical FS.The server holding the NameNode instance is quite crucial, as there is only one. Transaction log for file deletes/adds, etc. Does not use transactions for whole blocks or file-streams, only metadata.Handles creation of more replica blocks when necessary after a DataNode failureDataNode:Stores the actual data in HDFSCan run on any underlying filesystem (ext3/4, NTFS, etc)Notifies NameNode of what blocks it hasNameNode replicates blocks 2x in local rack, 1x elsewhere

Hadoops Architecture24-Aug-16Dr Gnanasekaran Thangavel19

Hadoops Architecture24-Aug-16Dr Gnanasekaran Thangavel20MapReduce Engine:JobTracker & TaskTrackerJobTracker splits up data into smaller tasks(Map) and sends it to the TaskTracker process in each nodeTaskTracker reports back to the JobTracker node and reports on job progress, sends data (Reduce) or requests new jobsNone of these components are necessarily limited to using HDFSMany other distributed file-systems with quite different architectures workMany other software packages besides Hadoop's MapReduce platform make use of HDFS

Hadoop in the Wild24-Aug-16Dr Gnanasekaran Thangavel21Hadoop is in use at most organizations that handle big data: Yahoo! FacebookAmazonNetflixEtc

Some examples of scale: Yahoo!s Search Webmap runs on 10,000 core Linux cluster and powers Yahoo! Web search

FBs Hadoop cluster hosts 100+ PB of data (July, 2012) & growing at PB/day (Nov, 2012)

Three main applications of Hadoop24-Aug-16Dr Gnanasekaran Thangavel22Advertisement (Mining user behavior to generate recommendations)

Searches (group related documents)

Security (search for uncommon patterns)

24-Aug-16Dr Gnanasekaran Thangavel23Hadoop HighlightsDistributed File SystemFault ToleranceOpen Data FormatFlexible SchemaQueryable DatabaseWhy use Hadoop?Need to process Multi Petabyte DatasetsData may not have strict schemaExpensive to build reliability in each applicationNodes fails everydayNeed common infrastructureVery Large Distributed File SystemAssumes Commodity HardwareOptimized for Batch ProcessingRuns on heterogeneous OS

24-Aug-16Dr Gnanasekaran Thangavel24DataNodeA Block SeverStores data in local file systemStores meta-data of a block - checksumServes data and meta-data to clientsBlock ReportPeriodically sends a report of all existing blocks to NameNodeFacilitate Pipelining of DataForwards data to other specified DataNodes

24-Aug-16Dr Gnanasekaran Thangavel25Block PlacementReplication StrategyOne replica on local nodeSecond replica on a remote rackThird replica on same remote rackAdditional replicas are randomly placedClients read from nearest replica

24-Aug-16Dr Gnanasekaran Thangavel26Data CorrectnessUse Checksums to validate data CRC32File CreationClient computes checksum per 512 byteDataNode stores the checksumFile AccessClient retrieves the data and checksum from DataNodeIf validation fails, client tries other replicas

24-Aug-16Dr Gnanasekaran Thangavel27Data PipeliningClient retrieves a list of DataNodes on which to place replicas of a blockClient writes block to the first DataNodeThe first DataNode forwards the data to the next DataNode in the PipelineWhen all replicas are written, the client moves on to write the next block in file

24-Aug-16Dr Gnanasekaran Thangavel28MapReduce UsageLog processingWeb search indexingAd-hoc queries

24-Aug-16Dr Gnanasekaran Thangavel29MapReduce Process (org.apache.hadoop.mapred)JobClientSubmit jobJobTrackerManage and schedule job, split job into tasksTaskTrackerStart and monitor the task executionChildThe process that really execute the task

24-Aug-16Dr Gnanasekaran Thangavel30Inter Process CommunicationProtocol

JobClient JobTracker

TaskTracker JobTracker

TaskTracker ChildJobTracker impliments both protocol and works as server in both IPCTaskTracker implements the TaskUmbilicalProtocol; Child gets task information and reports task status through it.JobSubmissionProtocolInterTrackerProtocolTaskUmbilicalProtocol

24-Aug-16Dr Gnanasekaran Thangavel31JobClient.submitJob - 1Check input and output, e.g. check if the output directory is already existingjob.getInputFormat().validateInput(job);job.getOutputFormat().checkOutputSpecs(fs, job);Get InputSplits, sort, and write output to HDFSInputSplit[] splits = job.getInputFormat(). getSplits(job, job.getNumMapTasks());writeSplitsFile(splits, out); // out is $SYSTEMDIR/$JOBID/job.split

24-Aug-16Dr Gnanasekaran Thangavel32JobClient.submitJob - 2The jar file and configuration file will be uploaded to HDFS system directoryjob.write(out); // out is $SYSTEMDIR/$JOBID/job.xmlJobStatus status = jobSubmitClient.submitJob(jobId);This is an RPC invocation, jobSubmitClient is a proxy created in the initialization

24-Aug-16Dr Gnanasekaran Thangavel33Job initialization on JobTracker - 1JobTracker.submitJob(jobID)