© 2003 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

SmartFrog OverviewHP LabsVersion: 0.6 (Draft)Localized for US English

page 2July 2003

What is SmartFrog?

SmartFrog: “Smart Framework for Object Groups”

• Describes services as collections of components• Activates services by realizing service descriptions

• Framework elements:– language– engine– components

• A framework to build from, not a packaged solution• Mature; already embedded in products

page 3July 2003

SmartFrog framework

language

parser

processmodel

transportprotocol

transactionallifecycle

discovery

naming

scripting

transforms

templates

componentmodel

reliabilityservices

deployment

binding

live-nesssecuritymodel

configservers

workflows

config.models

Language

Components

Engine

page 4July 2003

What problem does it solve?

• How to configure & automatically activate complex, distributed apps– flexible– repeatable– sequencing, failure-recovery,

evolution, adaptation, …

• Ideal for the utility computing model– automatic app activation

on utility resources– enables rapid resource

repurposing for different apps

page 5July 2003

How does SmartFrog address the problem?1. The common problems

• No standard method to capture service config., multiple: – definitions of each value– mechanisms for access– notations (XML, ini files, SQL, ...)

• No method of validation for the configuration as a whole– variable lifecycles– no up-front consideration of complete system

startup/shutdown– lack of facility to compose systems from sub-systems

• No separation of concerns– functionality and configuration mixed together– rigid location and binding integrated into code

• Lack of auto-discovery, self-monitoring, and automation

page 6July 2003

How does SmartFrog address the problem?2. Philosophy

• Services are collections of components– software, data: disk files, hardware

• Components collaborate to achieve some goal for which they must be appropriately organised:– required components must be defined– correctly initialised, with appropriate attributes– be able to locate each other as required– exchange information regarding their state

page 7July 2003

How does SmartFrog address the problem?3. Approach

• Language allows rich system descriptions• Engine automates application deployment,

activation, management and shutdown• Component model allows the engine to be

extended

page 8July 2003

Examples

• Network monitoring application– many monitoring components spread throughout the

network– each component must be correctly configured– service as a whole must be correctly configured /

customized– service components must come up in the correct

sequence

• Three-tier web application– database, application logic engine and web-server

must each be installed, configured and started correctly– must be started on the correct host(s), in the correct

sequence

page 9July 2003

SmartFrog elements

• Language– templates /

descriptions• Engine

– interprets descriptions to activate running services

• Components– make up the running

service– deployed, configured

& activated by the engine

Language

Engine

activates running

components

managed, monitored through lifecycle

• which service components?• running where?• how is each component

initialised?• how are components related?• how are the component

lifecycles sequenced?

page 10July 2003

Language properties

• Attribute:value pairs in a structured containment hierarchy

• Properties– declarative (a data description language, not a

programming language)– instance-inheritance (prototype-based);

allow templates to be progressively redefined– flexible linking between attributes and values– value resolution can be delayed to deployment time– composition: build larger descriptions out of smaller

parts

• Note: not XML-based, but can use XML as an input or output format if needed

page 11July 2003

Language example

webServerTemplate extends { sfProcessHost “localhost”; port 80; useDB;}

dbTemplate extends { userTable extends { columns 4; rows 3; } dataTable extends { columns 2; rows 5;}

#include “wstemplate.sf”#include “dbtemplate.sf”

sfConfig extends { commonPort 8080; ws1 extends webServerTemplate { sfProcessHost “webserver1.hpl.hp.com”; port ATTRIB commonPort; } ws2 extends webServerTemplate { sfProcessHost “webserver2.hpl.hp.com”; port ATTRIB commonPort; useDB LAZY ATTRIB db; } db extends dbTemplate { userTable:rows 6; }}

wstemplate.sf

dbtemplate.sf

webservice.sf

page 12July 2003

Language: composing descriptions

E.g. Apache Web-server

e.g. reliability & recovery thresholding

e.g. servers, storage networking

e.g. n-tiers including firewall rules

e.g. 3-tier web service on Linux cluster front-end, Superdome back-end.

fully-specifiedsolution

description

e.g. SLAs

softwarecomponentdescriptions

applicationpatterns

composed:solution

templates

‘farm’patterns

resourcedescriptions

customerpreferences

parameterized

page 13July 2003

Engine

• Fully distributed engine comprised of SmartFrog daemons that run on each node– deploy service descriptions on any node and the

engine will create components in the right place• Daemons interpret SmartFrog descriptions

– load components in the right sequence, in the right place, with correct configuration parameters

– orchestrate the whole system by stepping components through their complete lifecycles

• Provides liveness and component-tree navigation mechanisms

• Many services can be deployed simultaneously using the same engine

• Service un-deployment allows clean service removal

page 14July 2003

Security

• Engine provides secure deployment based on PKI

• All nodes have certificates installed• All component code and descriptions must be

signed• If node has valid certificate and

code/descriptions are correctly signed, then deployment is allowed

• Single-level security model (all deployed services operate within the same trust domain for a given certificate set)

page 15July 2003

Component model

• SmartFrog components all implement a simple lifecycle that allows the engine to control them

• Allows systems of components to be started consistently

import com.hp.SmartFrog.Prim.*;import java.rmi.*; public class MyPrim extends PrimImpl implements Prim, … {

/* any component specific declarations */  public MyPrim() throws RemoteException {}  public void sfDeploy() throws Exception { super.sfDeploy(); /* any component specific initialization code */ }  public void sfStart() throws Exception { super.sfStart(); /* any component specific start-up code */ }  public void sfTerminateWith(TerminationRecored tr) { /* any component specific termination code */ super.sfTerminateWith(tr); } /* any component specific methods */}

page 16July 2003

Components

• SmartFrog has an extensible component set• Some basic components form the core of the system,

e.g.:– “Prim”: the primitive component – everything inherits

from it– “Compound”: implements a “shared-fate” lifecycle for

groups of components– Sequence: implements a sequential lifecycle for

components• Other components provide system services: discovery

mechanisms, reliability building blocks, host scripting, JMX support, …

• Components are written as needed to support new services and wrap application components

page 17July 2003

Encapsulating non-SmartFrog components• We don’t need to write our whole

application/service in SmartFrog Java components -- it’s easy to encapsulate non-SmartFrog things, e.g.

• Apache web-server: wrapper components to– install and configure software (and remove)– stop and start the service

• We describe Apache and its required configuration in a SmartFrog description (using an Apache SF template)

• When deployed, the SF Apache component can install, configure, start, stop and remove Apache

• Apache can be used as a component of larger services

page 18July 2003

Implementation details

• Implemented completely in Java (using JDK1.4)– works across all standard Java platforms

• Uses Java RMI as transport• Core system size

– ~10K non-commented lines of Java– ~750 lines of description files– binaries: 800-1200 KBytes depending on services

included

page 19July 2003

Research questions

• Rich research space spanning language, engine, and components– reliability (basic mechanisms, generic recovery

patterns)– security– combining reliability and security– scalability– language specification– service debugging tools– standardized interfaces to resource management

subsystems– standardized interfaces to node image deployment

subsystems– generic system/service visualization tools

page 20July 2003

Business opportunity

• App developers– better enable apps for Grid and utility computer models– optimize application for specific hardware configurations– improve development (rapid deployment)

• Solution providers– improve app configuration, deployment, and management– more secure and reliable solutions– quicker time to deployment and adaptation

• Happier users– more stable systems– customization to specific needs– more control over solutions

page 21July 2003

Framework status

• Developing an open source (LGPL-licensed) release of the core framework (availability calendar ‘03)

• Earlier preview / alpha releases planned• Deployed within HP/Agilent products

• Experience with:– Apache– BEA Weblogic– Maya CGI rendering engine– Oracle

page 22July 2003

Some related work

• Grid standards for Grid/utility computing• Imaging / provisioning systems: Altiris, Rembo,

Novadigm (+ many others)• MS Dynamic Systems Initiative, MS Provisioning

System• Adaptive systems: ThinkDynamics, Corosoft• IBM eLiza, autonomic computing

• Proprietary (product-specific) solutions in place today for installation / configuration / activation

page 23July 2003

Standardization goals

• Improve Grid applications– specification, deployment, configuration & life-

cycle mgmt

• Standardize SF language, app templates, and SF engine

• Create multiple reference implementations of– engine– app templates (Application Server, JBOSS,

WebLogic)• Define SF integration with Grid• Form a GGF WG

page 24July 2003

SmartFrog benefits

• Increased operational reliability• Improved quality• Assured correctness and consistency• Increased security

• Reduced cost• Improved customer experience

page 25July 2003

Summary

• SmartFrog framework (language, templates, engine) supports:– specification, deployment, configuration, and life-cycle

mgmt• Well suited for a utility computing model • Addresses needs of complex distributed applications• Business benefits for application developers, ISPs, &

users

• We are seeking partners & lighthouse users– jointly standardize key elements– develop with reference implementations– create templates for specific applications– establish a wide technical community

HP logo