Job Scheduling: History and Evolution

November 17, 2010Welkin Associates

Job Scheduler Terminology and Components

The Scheduler Study Overview

Brief History of Job Scheduling

Job Scheduler Functions

Job Scheduler Architectures

Evaluation of Job Schedulers and Vendors

Q/A

Agenda

Terminology Job: An executable unit. Can be a systems command, a script, an application program, or a collection of executable units. Jobs can have one or more steps, each containing an executable unit. Job-Dependencies: Logical events or conditions that must be satisfied in order for jobs to be scheduled for execution. Predecessor and Successor.Job-Flow: A collection of two or more jobs related by job-dependencies.Schedule: The current active set of scheduled jobs and job-flows ready to be released for execution and monitored by the Master Scheduler.

ComponentsMaster Scheduler: Central software component that defines, modifies, stores job and job-flow definitions, dependencies, security information, events, and schedules. Provides automatic submission of executables, monitors status of all jobs and job-flows, provides alerts, controls starting, stopping, restarting of jobs and job-flows. Creates Audit trail of all events.Agent: A program that resides on each server that is controlled by the Master. The agent monitors jobs on its server, communicates with the Master about the status of jobs, and executes commands relayed by the Master.GUI Console: Console display for Administrators and Operators.Relational Database: The central repository used by the Master to store all job, job-flow, dependencies, security information, schedules and audit trails.

Job Scheduler Terminology and Components

Job Scheduler Terminology and Components (Cont.)

Agent

Master Scheduler

RDMS

Operator GUI console

Administrator

Agent

Agent

Logs

Administrator GUI console

Operator

Job-Flows

Job-Flows

Job-Flows

OPUS: Legacy Job Scheduler used for Satellite data processing

Developed by Space Telescope Science Institute (STScI ) in 1994 for HST Very fast and efficient pipeline processor. Powerful job-flow features and GUI consoleAdopted and customized for NOAA as a Job Scheduler on AIX p-series for migration from z/OS Mainframe (2005)OPUS requires replacement by a COTS Scheduler:

STScI no longer supports OPUSOPUS GUI written in Java has compatibility issues with latest version of Java

The Scheduler Study

Evaluated using two in-house Schedulers (NDE and STARS)Evaluated COTS Schedulers and vendors.

Scheduler Study Overview

In the beginning… there were 80 column Hollerith cards The term “batch” was derived from stacks of punched cards The first true Job Schedulers were people and card readers Therefore, originally Job Scheduling known as “batch processing” 1964 IBM OS/360 JCL presented first job dependency functions

Brief History of Job Scheduling

Early Mainframe era: Batch processing using card decks

Later Mainframe era: JCL on IBM Mainframes.

Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns

Evolution of 3rd party vendor Mainframe Job Schedulers IBM Tivoli Workload Scheduler (TWS) CA 7 Mainframe Scheduler BMC Control-M ASG Zeke Scheduler

Unix systems: crons

Today: Job Scheduling has greatly matured, with a number of software vendors that have developed and marketed their Job Scheduler products for over 25 years

Brief History of Job Scheduling (Cont.)

Workload Automation Maturity Pyramid

Brief History of Job Scheduling (Cont.)

Gartner: The future automation model will be “IT Workload Automation Broker” (ITWAB)

Brief History of Job Scheduling (Cont.)

Scheduler Functions:

Create and edit jobs, job-flows, users, groups, schedules Calendar scheduling File-Watchers (Trigger file events) Job parameters within job-flows available Job Queues, Classes, Priority Virtual resources to control jobs and files High Availability / Failover capability Reliable and fast Scheduler database with open schema Scheduler commands to start, stop, suspend, resume, and restart jobs Alerts generated for job errors and sent to Operators (GUI console, email, cell) Reporting tools

GUI console: Windows, Web-based, Unix/Linux Flexible and configurable for Administrators and Operators, easy to use Graphical display of jobs and job-flows. Click and drag highly preferable Detailed display of job-flow and job status, dependencies, errors/alerts

Job Scheduler Functions

Architecture: Master-Agent vs. Peer-to-Peer

Job Scheduler Architectures

Architecture: Master – Agent

Job Scheduler Architectures (Cont.)

EMA Criteria and Methodology for Evaluating Job Schedulers

Research/Advisory Firms Job Scheduler Evaluations

EMA Radar Report for Job Schedulers 01/2010

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

Forrester List of Evaluated Job Schedulers

November 2009 “Market Overview: Workload Automation, Q3 2009”Workload Automation Vendors

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

Gartner Job Scheduler Magic Quadrant Report 04/2009

The Gartner Magic Quadrant is copyrighted (04/28/2009) by Gartner, Inc., and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner’s analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the “Leaders” quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Source: Gartner, Inc.

Magic Quadrant for Job Scheduling

28 April 2009, Report Id Number: G00166953

Milind Govekar, Gartner Analyst

Research/Advisory Firms Job Scheduler Evaluations (Cont.)

Q/A

Additional Slides for reference…

18

Workload Automation Example

19

Future Growth of WLA and Job Scheduling

Forrester: Forecast for Growth of WLA and Job Scheduling

Evaluation of Job Schedulers and Vendors

Vendor Scheduler Product Name Open Source,

Freeware, COTS

ASCI ActiveBatch COTS

Absyss Visual TOM COTS

ActivMindz TaskForest Open Source

APM Software GmbH APX/PCC COTS

Apple XGRID COTS

Argent Job Scheduler COTS

ASG Zena for Distributed Systems COTS

ASG Zeke for z/OS Mainframes COTS

Automation Anywhere Automation Anywhere COTS

Ayehu eyeShare COTS

BMC Control-M COTS

Camellia Software Batch Job Server (BJS) COTS

CISCO Tidal Enterprise Scheduler (TES) COTS

Cluster Resources Maui Cluster Scheduler Open Source

Computer Associates AutoSys COTS

Evaluation of Job Schedulers and Vendors

Vendor Scheduler Product Name Open Source,

Freeware, COTS

Corporate Practical Solutions Bqplus COTS

Cronless.com cronless Open SourceDavid G. Walker php JobScheduler Free Ware

Exact JobBOSS COTSExspans Automan COTS

Flux Corp. Flux COTS

Grand Logic JobServer COTS

Help/Systems Robot/SCHEDULE Enterprise COTSHonico Batchman COTS

Hux Software HQ Scheduler COTS

IBMTivoli Workload Automation Suite

COTS

IndependIT BICsuite Scheduler Open Source

ISE EnterpriseSCHEDULE COTS

JOX IT JOX COTSLXI LXI Scheduler COTS

Evaluation of Job Schedulers and Vendors

Vendor Scheduler Product Name Open Source,

Freeware, COTS

MJTNet Macro Scheduler COTSMVP Systems JAMS COTSNeteject.com VisualCron COTS

Network Automation Automate7 COTSNguyen Anthuan SetCronJob Free Ware

OAR Team OAR Open SourceOpen Systems Mgmt Cosbatch COTS

OpsWise Automation Center COTSORSYP Dollar Universe COTS

OW2 Consortium ProActive Parallel Suite Open SourcePlatform Computing Load Sharing Facility (LSF) COTS

PTC Software PTC Scheduler COTSRedwood Software Cronacle COTS

ROC Software Maestro Scheduler COTSRuntime Design

AutomationNetwork Computer (NC) COTS

SMA OpCon/xps COTSSoftTree Tech 24x7 Scheduler COTS

Evaluation of Job Schedulers and Vendors

Vendor Scheduler Product Name Open Source,

Freeware, COTS

Sourceforge Jcrontab Open Source

Stonebranch Indesca COTS

Sun Microsystems N1 Grid Engine COTS

Sypsoft Job2Do COTS

Tenth Generation Job Scheduler for Windows COTS

Terracotta Quartz Enterprise Job Scheduler Open Source

The Coens Group Jobstream Open Source

UC4 Workload Automation Suite COTS

Univa UD Grid MP COTS

University of Warwick TITAN Grid Scheduler Open Source

University of Wisconsin Condor Open Source

Vinzant Global ECS (GECS) COTS

Xi Software GNUbatch Open Source

XuiS Software EnterpriseSchedule COTS