Download - Event stream processing with WebSphere eXtreme Scale

The use of the term event might be confusingat first. The term is used both to describe anactivity that occurred, and something thatrepresents an activity that occurred in acomputer system. When discussing eventprocessing, it is common to allow theoverloading of the term to continue.Hopefully the context within which the termis used will be sufficient to avoid ambiguity.

Event stream processing with WebSphere eXtremeScaleLevel: Intermediate

Dr. Alan Hopkins ([email protected]), Senior IT Specialist, IBM　

03 Sep 2008

IBM® WebSphere® eXtreme Scale exposes a rich set of APIs that enable access to data residing withina distributed, resilient, and high performance cache. These APIs support a wide range of applicationprogramming patterns. One such pattern enables repeated cache updates to be treated as a temporal

sequence of events. Applications written to monitor these event sequences are able to correlate and aggregatepatterns of lower level events in order to derive real-time insight into business-significant situations. This articledescribes the implementation of a simple WebSphere eXtreme Scale-based scenario that illustrates the real-timeprocessing of heterogeneous event streams.

From IBM WebSphere Developer Technical Journal.

IntroductionA common characteristic of modern enterprise architectures is the inclusion of components procured from different sources,based upon non-consistent foundation technologies, and implemented at different points in time. Not surprisingly, the resultantinfrastructure is often too complex to completely understand at an IT level, let alone at a higher level business abstraction.

The occurrence of activities in these disparate subsystems represent the inner workings of today’s modern enterprise, orbeyond. As a result of completely independent threads of execution, these activities will frequently occur in isolation fromeach other. Collectively, these heterogeneous events represent an often untapped source of insight into business threats andopportunities.

A common approach to externalizing the occurrence of a significant activitywithin a computer system is via the emission of an event to describe theactivity. The physical representation of the event is of lesser interest at thispoint. What is important is that activities occurring continuously withinthese heterogeneous subsystems can be made available for analysis from across-subsystem perspective. This set of heterogeneous events is sometimesreferred to as an event cloud.

IBM WebSphere eXtreme Scale provides the functionality required tocapture and perform real-time analysis of the event cloud. This articlepresents the implementation of a simple stock portfolio stop loss capability for a Web application. This implementationleverages some of the key capabilities of WebSphere eXtreme Scale that facilitate the real-time processing of heterogeneousevent streams.

Scenario: Stock portfolio stop lossThis example scenario provides a simple stock portfolio stop loss capability. In this scenario, client stock portfoliosencapsulate any number of stocks that might be traded in either US dollars or UK pounds. Portfolio valuations are expressedin UK pounds, and are recalculated dynamically each time an incoming stock ticker event results in a change to a givenportfolio stock holding.

Event stream processing with WebSphere eXtreme Scale http://www.ibm.com/developerworks/websphere/techjournal/0...

1 of 16 7/23/09 2:15 PM

Figure 1. Stock portfolio stop loss scenario

If the aggregate valuation of a portfolio changes by more than 10% on a given trading day, an alert is raised. In this example,the alert is implemented as an informational message written to the system log. (In a more realistic implementation, this alertcould result in a warning rendered on a business monitoring dashboard, or an automated trade to sell some or all of themonitored stock portfolio.)

Essential aspects of WebSphere eXtreme ScaleWebSphere eXtreme Scale provides a runtime platform into which applications with ultra-demanding performance, scalability,and availability requirements can be deployed. A complete discussion of the all the capabilities provided by this technology isbeyond the scope of this article, but brief descriptions of some of the capabilities leveraged in this solution are presentedbelow. This information provides a contextual background to help you understand the content of our example scenario. (SeeResources for a complete description of all WebSphere eXtreme Scale capabilities.)

In-memory cache

The central capability provided by WebSphere eXtreme Scale is an in-memory cache that can be partitioned acrosslarge numbers of operating system processes that reside on disparate machines. This distributed cache can beconfigured to provide enterprise class qualities of service such as performance, scalability, and resiliency. From anadministrative perspective, the high-level container for this in-memory cache is the ObjectGrid. Within an ObjectGridinstance, data is encapsulated within one or more maps. Application programs can read/write data from/to these mapsin the form of data objects, identified by a unique key. Data integrity is assured by the inherent transactional capabilitiesof WebSphere eXtreme Scale.

In order to respond to discrete events in the incoming stock/foreign exchange event streams in our example, you willmake use of an ObjectGrid event listener plug-in. Because this type of plug-in is configured with a scope of anObjectGrid instance, you will use two configured ObjectGrid instances in the present scenario. This lets you achieveisolation with respect to incoming events. Since the data residing within this cache is held entirely within memory, akey benefit of using WebSphere eXtreme Scale is the speed at which operations can be performed against cached data.

Programming models


2 of 16 7/23/09 2:15 PM

WebSphere eXtreme Scale exposes a rich set of APIs that collectively support the access of cached data via a numberof application programming patterns. In this scenario, you will exploit two of these patterns:

In the first instance, you will use a simple map API to write entries to the cache. The WebSphere eXtreme ScaleMap API is similar to the Java™ Collections Map API in that you essentially write entries identified by a uniquekey to a map object.In addition, you will also use the WebSphere eXtreme Scale Stream Query API to handle data updatesrepresenting continually changing stock and foreign currency exchange rates. This enables updates to aWebSphere eXtreme Scale map to be treated as a temporal event stream. This capability is provided as anextension to the core map capability. Support for event stream handling introduces the concept of a stream mapand a view map. A stream map is used to capture one or more incoming raw event streams. The precise nature ofthe extraction and aggregation of data elements from the raw data stream that are used to populate the view mapis expressed using Stream Query Language, an SQL-like language that contains specific extensions to facilitatethe explicit processing of temporal event streams. Processing of this raw event stream is handled by the StreamProcessing Technology (SPT) engine. SPT engine output is written to the corresponding stream view map.

Figure 2 shows a pictorial representation of the processing flow through these WebSphere eXtreme Scale components.

Figure 2. Event stream processing with WebSphere eXtreme Scale

ObjectGrid event listener

WebSphere eXtreme Scale is architected to provide pluggable extension points through which user code can beincorporated. The ObjectGrid event listener extension point enable the incorporation of code that is invoked whensignificant ObjectGrid lifecycle events occur. The sample scenario leverages this capability to introduce code that willbe executed every time a transaction is committed against one of the two ObjectGrids that are used here. This pluggableextension point provides the mechanism through which you will incorporate code that dynamically recalculatesaggregate portfolio valuations every time you receive a stock ticker event.

ImplementationEssential elements of this sample scenario are described below. The development of this scenario was performed with thesesoftware product versions:

IBM WebSphere Integration Developer Version 6.1 (Java perspective)IBM WebSphere eXtreme Scale Version 6.1IBM Java Runtime Environment Version 1.5.0

Although the underlying Java runtime environment in the development scenario was J2SE™, this application should runequally well in an IBM WebSphere Application Server-based environment.

WebSphere eXtreme Scale configuration

Configure WebSphere eXtreme Scale using two complementary configuration files. The first, objectGrid.xml (Listing 1)contains the definition of the two ObjectGrids that are used in this scenario:

StocksGrid is used to encapsulate incoming event streams that contain foreign exchange and stock ticker events.


3 of 16 7/23/09 2:15 PM

Within this ObjectGrid you define four backing maps:

Incoming raw stock and foreign exchange event streams are written to stockStreamMap and forexStreamMap,respectively.View data derived from these two initial maps by the STP engine, subject to the specification of thecorresponding streams and views, is written to stockViewMap and forexViewMap.

Specification of the data object attributes that are relevant to the scenario can be found within the stream definitions.The time-based aggregation of data from these streams is provided via the view definitions.

One further point to note in the configuration data below is the specification of the ObjectGridEventListener plug-in onthe StocksGrid ObjectGrid instance.

CustomerPortfolioGrid is used to contain the details of individual customer portfolios. This ObjectGrid instancecontains two backing map instances: StockOwners enables you to determine the owners of any given stock, whilstPortfolios enables you to retrieve the details of all monitored Customer portfolios.

Listing 1. Configuration file objectGrid.xml<objectGrids> <objectGrid name="StocksGrid"> <bean id="ObjectGridEventListener" className="wxs.plugins.ObjectGridEventListener"/> <backingMap name="stockStreamMap" streamRef="stockStream"/> <backingMap name="forexStreamMap" streamRef="forexStream"/> <backingMap name="stockViewMap" viewRef="stockView"/> <backingMap name="forexViewMap" viewRef="forexView"/>

<streamQuerySet name="stockMonitoringSQS"> <stream name="stockStream" valueClass="wxs.streamquery.example.StockQuote" sql="create stream stockStream keyed by t (price DECIMAL (9,2), tickerSymbol VARCHAR(100));" access="FIELD" > </stream> <stream name="forexStream" valueClass="wxs.streamquery.example.ForexQuote" sql="create stream forexStream keyed by t (rate DECIMAL (9,4), type VARCHAR(100) );" access="FIELD" > </stream> <view name="stockView" sql="CREATE VIEW stockView AS SELECT tickerSymbol, avg(price) as avgPrice FROM (SELECT * FROM stockStream FETCH LATEST 5 MINUTES) group by tickerSymbol" access="FIELD"> </view> <view name="forexView" sql="CREATE VIEW forexView AS SELECT type, avg(rate) as avgRate FROM SELECT * FROM forexStream FETCH LATEST 5 MINUTES) group by type" access="FIELD"> </view> </streamQuerySet></objectGrid> <objectGrid name="CustomerPortfolioGrid"> <backingMap name="StockOwners" readOnly="false" lockStrategy="PESSIMISTIC"/> <backingMap name="Portfolios" readOnly="false" /> <ObjectGrid> </objectGrids>

A secondary configuration file, objectGridDeployment.xml, defines characteristics of your cache topology. The essentialelements of this topology configuration is presented in Listing 2, which shows that each ObjectGrid instance is made up of asingle partition.

Listing 2. Configuration file objectGridDeployment.xml<objectgridDeployment objectgridName="StocksGrid"> <mapSet name="mapSet1" numberOfPartitions="1"


4 of 16 7/23/09 2:15 PM

minSyncReplicas="0" maxSyncReplicas="0" maxAsyncReplicas="0" numInitialContainers="1"> <map ref="stockStreamMap"/> <map ref="forexStreamMap"/>

<map ref="forexViewMap"/> <map ref="stockViewMap"/>

</mapSet></objectgridDeployment>

<objectgridDeployment objectgridName="CustomerPortfolioGrid"> <mapSet name="mapSet1" numberOfPartitions="1" minSyncReplicas="0" maxSyncReplicas="0" maxAsyncReplicas="0" numInitialContainers="1"> <map ref="StockOwners"/> <map ref="Portfolios"/> </mapSet></objectgridDeployment>

Cached object data model

Define two sets of POJOs to encapsulate the data that is cached within your ObjectGrid instances:

The first set, used to hold state data describing customer, stock portfolios, and stock holdings, is defined duringscenario initialization and remains static throughout the scenario lifetime (Figure 3).

Figure 3. Static data model

The second set of data objects represent stock and currency exchange rate events that make up the event streamsconsumed by this scenario (Figure 4).

Figure 4. Event stream data model

Code artifacts

This implementation involves the Java code artifacts listed in Table 1.

Table 1. Code artifactsClass name Description

wxs.streamquery.example.StockOwners POJO that encapsulates set of portfolio IDs that own a given stock.

wxs.streamquery.example.Portfolio POJO that encapsulates a customer portfolio.

wxs.streamquery.example.StockQuote POJO that encapsulates a stock ticker event.

wxs.streamquery.example.ForexQuote POJO that encapsulates an exchange rate ticker event.

wxs.streamquery.example.StockHolding POJO that encapsulates the portfolio holding of a single stock.


5 of 16 7/23/09 2:15 PM

Wxs.streamquery.example.StatusFlag POJO used to denote that scenario is initializing.

wxs.streamquery.example.ScenarioInit Command line Java program that initializes the stop loss scenario.

wxs.streamquery.emitter.ForexEventStreamEmitter Command line Java program used to emit a stream of exchange rate ticker events.

wxs.streamquery.emitter.StockEventStreamEmitter Command line Java program used to emit a stream of stock ticker events.

wxs.streamquery.emitter.SingleStockEventEmitter Command line Java program used to emit a single stock ticker event.

wxs.plugins.ObjectGridEventListener Listener program that is invoked for every occurrence of a state change to a givenObjectGrid instance. In this scenario, you use this program to react to committedchanges to the StocksGrid ObjectGrid instance.

wxs.utility.CustomerPortfolioGridHelper Utility program that manages the acquisition of a handle to theCustomerPortfolioGrid ObjectGrid singleton instance.

wxs.utility.StocksGridHelper Utility program that manages the acquisition of a handle to the StocksGridObjectGrid singleton instance.

The next sections take a closer look at some of these artifacts. Because the intent here is to present an overview of the mostsignificant aspects of these code artifacts, exhaustive descriptions of all the Java programs that make up this scenario are notincluded here. However, all code artifacts described in the table above can be found in the download file that accompanies thisarticle.

Scenario initializationInitialization of the scenario is encapsulated within ScenarioInit, a command line program responsible for creating and writinginstances of the StockOwners and Portfolio POJOs to the CustomerPortfolioGrid ObjectGrid instance. This state informationis subsequently used during the detection of stop loss situations when ObjectGridEventListener is running. Scenarioinitialization is also responsible for writing initial valuations for each stock used within the scenario, and the initial GB poundsto US dollars currency exchange rate to the StocksGrid ObjectGrid instance.

State data definition

If you examine ScenarioInit, you will see that the details of each stock, portfolio, and exchange rate is hardcoded at the start ofthe class source code.

Listing 3. Definition of stock portfolios and initial stock pricesstatic String[] stockList = {"AIG", "AXP", "ATT", "INTC", "MOT", "IBM", "CSCO", "JAVA", "MSFT", "ORCL", "HBOS", "HSBA", "BAY", "BGY", "BLND", "GSK", "ITV", "LAND", "III", "BT"};

static float[] stockPrices = {49.12f, 52.40f, 24.95f, 23.56f, 10.07f, 123.29f, 26.92f, 13.14f, 29.43f, 21.55f, 4.71f, 8.82f, 2.09f, 7.14f, 8.04f, 11.53f, 0.61f, 14.67f, 8.97f, 2.33f};

static int[][] portfolioVolumes = { {1000, 500, 2000, 850, 4000, 2000, 7000, 2500, 5000, 9000, 2000},{1000, 2500, 4000, 9000, 500, 900, 600, 2000},

{8000, 900, 2500, 1000, 2000, 3000},{3000, 1000, 2500, 5500, 500, 2000, 7500, 1000},{3000, 5500, 4000, 500, 3000, 7000, 9000},{3500, 2500, 9000, 6500, 3000, 1000, 2500, 1000},{1000, 7000, 8000, 3000, 500, 3500, 5500},{9000, 3000, 5500, 1000, 8000, 4000, 3000, 2000, 1500},{2500, 500, 4000, 3000, 9000, 8000, 7000, 6000},{5500, 1000, 2500, 500, 3000, 8000, 500, 3000}};

static String[][] portfolioStocks = {{"AIG", "AXP", "ATT", "INTC", "MOT", "IBM", "CSCO", "JAVA", "MSFT", "HBOS"},{"IBM", "CSCO", "JAVA", "MSFT", "ORCL", "BAY", "BGY", "BLND"},


6 of 16 7/23/09 2:15 PM

{"IBM", "CSCO", "JAVA", "GSK", "ITV", "LAND"},{"ATT", "INTC", "MOT", "IBM", "CSCO","III", "BT", "HBOS"},{"AIG", "ATT", "INTC", "MOT", "JAVA", "ORCL","HSBA"},{"ATT", "INTC", "MOT", "IBM", "CSCO", "ORCL", "BAY", "BGY"},{"AIG", "AXP", "INTC", "JAVA", "ORCL","BLND", "GSK"},{"AIG", "AXP", "INTC", "IBM", "CSCO", "JAVA", "ORCL","ITV","LAND"},{"AIG", "ATT", "MOT", "CSCO", "MSFT", "ORCL","III","BT"},{"AXP", "ATT", "INTC", "IBM", "MSFT", "ORCL","GSK", "ITV"}}; float initialDollarToPoundsForexRate = 1.9876f;

Stock ownership state data creation

The code fragment in Listing 4 illustrates how you can write stock ownership data to the StockOwners map within theCustomerPortfolioGrid ObjectGrid instance. For each stock referenced within the scenario, as defined within the stockListarray in Listing 3, you create a StockOwners object instance with a key value set to the stock ticker symbol from stockList[].You then construct an array of CustomerIDs that contains an entry for every portfolio that contains the stock represented bythe StockOwners object instance. This array is set as the value of the customerIDs attribute, before writing the object to theStockOwners map.

After the code in Listing 4 has run, the StockOwners map contains one instance of the StockOwners object for every stockdefined within the stockList[] array.

Listing 4. Population of StockOwners map with static dataportfolioGrid = CustomerPortfolioGridHelper.getOG("CustomerPortfolioGrid");stocksGrid = StocksGridHelper.getOG("StocksGrid"); /* -------------------------------------------------------- *//* Get ObjectGrid Session *//* -------------------------------------------------------- */Session portfolioSession = portfolioGrid.getSession();

/* -------------------------------------------------------- *//* Write Stock Ownership StockOwners Map *//* -------------------------------------------------------- */ObjectMap ownersMap = portfolioSession.getMap("StockOwners"); portfolioSession.begin(); // Create array of StockOwners objectsfor (int i = 0; i < stockList.length; i++){

ArrayList customerIDs = new ArrayList();StockOwners stockOwners = new StockOwners(stockList[i]);stockOwners.setTickerSymbol(stockList[i]);

for (int j = 0; j < portfolioStocks.length; j++){

for (int k = 0; k < ScenarioInit.portfolioStocks[j].length; k++) { if (ScenarioInit.portfolioStocks[j][k].equals(stockList[i])) { customerIDs.add(String.valueOf(j + 1)); } } } stockOwners.setCustomerIDs(customerIDs); ownersMap.put(stockOwners.getTickerSymbol(), stockOwners); }portfolioSession.commit();

Acquisition of ObjectGrid handle

In Listing 4, the acquisition of a handle to the configured ObjectGridInstances CustomerPortfolioGrid and StocksGrid isencapsulated within two similar helper classes. Figure 5 shows the essential aspects of one of these helper classes.


7 of 16 7/23/09 2:15 PM

Listing 5. Population of Portfolios map with static datasynchronized static public ObjectGrid getOG(String objectGrid){ if (og == null) { try {

URL url = Thread.currentThread().getContextClassLoader().getResource("SQ_ObjectGrid.xml");

ClientClusterContext ccc = bjectGridManagerFactory.getObjectGridManager().connect("localhost:6000", null,url);

ObjectGridManager ogm = ObjectGridManagerFactory.getObjectGridManager();

og = ogm.getObjectGrid(ccc,objectGrid);

} catch (Exception e) {

e.printStackTrace(); } } else { System.out.println("StocksGridHelper.getOG() - OG already initialised"); } return og;}

Customer portfolio state data creation

You must now write the state data that defines customer stock portfolios into the Portfolios map within theCustomerPortfolioGrid ObjectGrid instance. Once again, this data is used whilst monitoring changes to the aggregate value ofcustomer portfolios in order to provide the stop loss capability. The code fragment used to write portfolio state data intoWebSphere eXtreme Scale is shown in Listing 6. The initial aggregate portfolio valuation is calculated for each portfolio, andthe value is copied into the initialValuation attribute of each portfolio object instance.

When the code in Listing 6 has run successfully, one portfolio object instance has been written into the Portfolios ObjectMapfor each defined customer stock portfolio.

Listing 6. Population of Portfolios map with static dataObjectMap portfoliosMap = portfolioSession.getMap("Portfolios"); for (int j = 0; j < ScenarioInit.portfolioStocks.length; j++){ float portfolioValuation = 0.0f; Portfolio portfolio = new Portfolio(String.valueOf(j + 1)); ArrayList<StockHolding> stockHoldingList = new ArrayList<StockHolding>(); portfolioSession.begin(); for (int k = 0; k < ScenarioInit.portfolioStocks[j].length; k++) { int numUnits = ScenarioInit.portfolioVolumes[j][k]; StockHolding stockHolding = new StockHolding(ScenarioInit.portfolioStocks[j][k]); stockHolding.setNumUnits(numUnits); if (isUSStock(ScenarioInit.portfolioStocks[j][k])) { stockValuation = numUnits * initialStockPrice(portfolioStocks[j][k])/ initialDollarToPoundsForexRate; } else


8 of 16 7/23/09 2:15 PM

{ stockValuation = numUnits * initialStockPrice(portfolioStocks[j][k]); } stockHolding.setLatestValuation(stockValuation); stockHoldingList.add(stockHolding); portfolioValuation += stockValuation; } portfolio.setStocks(stockHoldingList); portfolio.setInitialValuation(portfolioValuation); portfoliosMap.insert(portfolio.getCustomerID(), portfolio); }portfolioSession.commit();

Write initial events to event stream maps

Next, you must prime the stock and currency exchange event streams by inserting objects that represent the opening prices foreach stock within the scenario into stockStreamMap, and the opening GB pound to US dollar exchange rate intoforexStreamMap, respectively.

Essential elements of the code required to insert these events is presented in Listing 7.

Listing 7. Scenario initialization: Insertion of initial eventsSession stockStreamSession = stocksGrid.getSession();

ObjectMap stockStreamMap = stockStreamSession.getMap("stockStreamMap");ObjectMap stockViewMap = stockStreamSession.getViewMap("stockViewMap"); stockStreamSession.begin(); StockQuote.QuoteCurrency currency = null; for (int i = 0; i < stockList.length; i++){ if (i < 10) { currency = StockQuote.QuoteCurrency.USDOLLAR; } else { currency = StockQuote.QuoteCurrency.GBPOUND; }

stockStreamMap.insert(stockList[i], new StockQuote(stockList[i], stockPrices[i], currency)); }

// ForexforexStreamMap.insert("USDOLLAR_TO_GBPOUND", new ForexQuote("USDOLLAR_TO_GBPOUND", initialDollarToPoundsForexRate));

stockStreamSession.commit();

ObjectGridEventListener

As described earlier, WebSphere eXtreme Scale is designed to provide a number of extension points that enable the defaultcapability of the technology to be extended. Code modules can be specified to run at specific defined points in the overallWebSphere eXtreme Scale application lifecycle.

The ObjectGridEventListener plug-in module is provided to enable the incorporation of code that will be invoked at strategicpoints during transaction lifecycle management. You can leverage this capability to introduce functionality that runs each timea transaction is committed to the StocksGrid ObjectGrid instance to which all stock and foreign exchange ticker events arewritten. As such, the implementation of the ObjectGridEventListener is at the heart of the event-driven scenario presentedhere. The plug-in is responsible for processing events that represent updates to views derived from both stock and foreign


9 of 16 7/23/09 2:15 PM

exchange tickers.

Aggregate portfolio value recalculations are performed after an update to the stockView view, which is derived from thestockStream event ticker stream. These aggregate values are computed using time-averaged stock and foreign exchange valuesderived from their respective event streams.

The most important aspects of the endTransaction() method are presented in Listing 8. (The code shown here is condensed forsimplicity, and is intended only to convey the essence of how the code works. For a full code listing, refer to the includeddownload file.) A collection object encapsulating the changes contained within a given transactional bracket is passed in as aparameter on the method invocation. The first section of code focuses on determining the most recent time-averaged forexvalue. To retrieve this value, you navigate through interrelated entity API objects to eventually arrive at the current timeaveraged value.

Listing 8. ObjectGridEventListener: Retrieval of time-averaged forex valuepublic void transactionEnd(String txid, boolean isWriteThruEnabled, boolean committed, Collection changes) {

Iterator logSequenceIterator = changes.iterator();

while (logSequenceIterator.hasNext()) { LogSequence logSequence = (LogSequence)logSequenceIterator.next(); String mapName = logSequence.getMapName(); Iterator forexChangeIterator = logSequence.getAllChanges();

if ((mapName.equals("forexViewMap") && logSequence.isDirty())) {

while (changeIterator.hasNext()){ try { LogElement logElement = (LogElement)changeIterator.next(); Tuple forexKeyTuple = (Tuple)(logElement.getCacheEntry().getKey()); String key = (String)forexKeyTuple.getAttribute(0); ogSession = instrumentGrid.getSession(); ObjectMap forexViewMap = ogSession.getViewMap("forexViewMap"); EntityMetadata emd = forexViewMap.getEntityMetadata(); TupleMetadata keyMD = emd.getKeyMetadata(); Tuple forexTuple = keyMD.createTuple(); forexTuple.setAttribute(0, "USDOLLAR_TO_GBPOUND"); //Could use key value Tuple forexValueTuple = (Tuple)forexViewMap.get(forexTuple); timeAveragedForexValuation = (Float)forexValueTuple.getAttribute(0);}catch ( Exception e)

{......} } } }}

You then need to handle the case in which the transaction that just committed had resulted from an incoming stock tickerevent; that is, the transaction commit that caused the invocation of your transactionEnd() method was against the stockViewmap. For this case, you first establish the key of the event, which is set to the stock ticker symbol, and then retrieve the currenttime-averaged stock price valuation for this ticker symbol.

Listing 9. ObjectGridEventListener: Retrieval of time-averaged stock valueselse if (mapName.equals("stockViewMap") && logSequence.isDirty()){ while (changeIterator.hasNext()) { try {

LogElement logElement = (LogElement)changeIterator.next();


10 of 16 7/23/09 2:15 PM

LogElement.Type type = logElement.getType(); Tuple keyTuple = (Tuple)logElement.getCacheEntry().getKey(); String key = (String)keyTuple.getAttribute(0);

Session instrumentSession = instrumentGrid.getSession(); instrumentSession.beginNoWriteThrough(); ObjectMap stockViewMap = instrumentSession.getViewMap("stockViewMap");

EntityMetadata emd = stockViewMap.getEntityMetadata(); TupleMetadata keyMD = emd.getKeyMetadata(); Tuple ibmKey = keyMD.createTuple(); ibmKey.setAttribute(0, key);

Tuple ibmValue = (Tuple) stockViewMap.get(ibmKey); Float timeAveragedValuation = (Float)ibmValue.getAttribute(0);

info("Stock Ticker Event :: Average price in the last 5 minutes for " + key + " = " + timeAveragedValuation);

instrumentSession.commit();

Once you have established the stock ticker symbol and current time-averaged stock price, you can go ahead and recalculatethe aggregate value of all portfolios that contain a holding of this stock. First, retrieve a list of the portfolios that hold the stockrepresented by the event that is currently being processed (Listing 10).

Listing 10. ObjectGridEventListener: Retrieval of portfolios containing current stock/* ------------------------------------------------------------- *//* Retrieve List of Portfolios that hold this stock *//* ------------------------------------------------------------- */ObjectMap portfoliosMap = portfolioSession.getMap("Portfolios");

StockOwners stockOwners = (StockOwners) holdingsMap.get(key);

ArrayList portfolioIDs = stockOwners.getCustomerIDs();

Iterator portfolioIterator = portfolioIDs.iterator();

Then, iterate through each stock held in the returned portfolio list and recalculate the aggregate portfolio valuation using themost recent time-averaged stock and forex prices (Listing 11).

Listing 11. ObjectGridEventListener -- Recalculation of aggregate portfolio valuationswhile (portfolioIterator.hasNext()){portfolioSession.begin();

String portfolioID = (String)portfolioIterator.next();Portfolio portfolio = (Portfolio)portfoliosMap.get(portfolioID);ArrayList stocksList = portfolio.getStocks();ArrayList updatedStocksList = new ArrayList();Iterator stocksIterator = stocksList.iterator();

float newPortfolioValuation = 0.0f;while (stocksIterator.hasNext()){

StockHolding stockHolding = (StockHolding) stocksIterator.next();ibmKey.setAttribute(0, stockHolding.getTickerSymbol());ibmValue = (Tuple) stockViewMap.get(ibmKey);timeAveragedValuation = (Float)ibmValue.getAttribute(0);info("Stock Ticker Event :: Average price in the last 5 minutes for "

+ stockHolding.getTickerSymbol() + " = " + timeAveragedValuation);

boolean usStockFlag = isUSStock(stockHolding.getTickerSymbol());if (usStockFlag)

{ newStockValuation = stockHolding.getNumUnits() *

timeAveragedValuation.floatValue()/timeAveragedForexValuation;}


11 of 16 7/23/09 2:15 PM

else{ newStockValuation = stockHolding.getNumUnits() *

timeAveragedValuation.floatValue();}newPortfolioValuation += newStockValuation;

}portfolioSession.commit();

Finally, check whether the aggregate valuation of each portfolio has changed by more than 10% of its original value (Listing12).

Listing 12. ObjectGridEventListener: Check whether portfolio valuation has changed by more than 10%float relativeChange = newPortfolioValuation/portfolio.getInitialValuation();

/* ---------------------------------------------------------------- *//* If relative portfolio valuation is > 1.10 or < 0.90 raise Alert *//* ---------------------------------------------------------------- */if ((relativeChange > 1.10) || (relativeChange < 0.90)){info("!************************************************************");

info("!*");info("!* Portfolio Valuation Alert");info("!* Portfolio ID = " + portfolioID );info("!* Initial Valuation = " + portfolio.getInitialValuation());info("!* New Valuation = " + newPortfolioValuation);info("!* Relative Valuation Change = " + relativeChange);info("!*");info("!* Alert Triggering Stock = " + key);info("!*");info("!************************************************************");

}

Event emissionTo illustrate the event-driven characteristics of this stock portfolio stop loss capability, you must be able to simulate theemission of events. This capability is provided by a set of command line utility programs that are effectively WebSphereeXtreme Scale client programs. Because these programs are all coded in an essentially consistent manner, the essential aspectsof only one of these programs, StockEventStreamEmitter, is shown in Listing 13.

Listing 13. Event emissionObjectMap stockMap = getObjectMap("stockStreamMap");

for (int i = 0; i < numEvents; i++){int stockIndex = random(stockList.length);

int priceIndex = random(stockPrices[stockIndex].length);float stockPrice = stockPrices[stockIndex][priceIndex];

if (stockIndex < 10){

quoteCurrency = StockQuote.QuoteCurrency.USDOLLAR;}else{

quoteCurrency = StockQuote.QuoteCurrency.GBPOUND;}

StockQuote stockQuote = new StockQuote(stockList[stockIndex],stockPrice, quoteCurrency );


12 of 16 7/23/09 2:15 PM

stockMap.update(stockQuote.getTickerSymbol(), stockQuote);

Thread.sleep(sleepTime);}

For the purpose of this example, this code iteratively instantiates a StockQuote object using a ticker symbol and correspondingprice, selected pseudo-randomly from hardcoded sets of possible values. These StockQuote Objects are then written to theObjectMap defined as the basis of your stock event stream.

Scenario executionBe aware that the execution of this event stream processing scenario is simplified through the use of scripts that encapsulatethe invocation of a number of executables. The scripts that were used during the testing of this scenario are provided in theincluded download file, and can be modified for running your own sample scenario.

Start catalog server

WebSphere eXtreme Scale makes use of a catalog server to provide essential services to the distributed elements of theruntime topology. Start a catalog server instance from the command line by executing a command resembling the followingfrom within the ObjectGrid bin directory:

C:\IBM\ObjectGrid\ObjectGrid\bin>startCatalog.bat cs1 hostname 6000

Notice that this command starts a catalog server instance named cs1 listening on port 6000. Replace hostname with the nameof the machine on which you are running this scenario. Look for a message resembling the following as confirmation that thecatalog server has initialized successfully:

[22/05/08 11:30:18:328 BST] fc00fc ServerImpl I CWOBJ1001I: ObjectGrid Server cs1is ready to process requests.

Start container server

Cached data resides within the JVM heap of one or more container servers. The containier server in this scenario also providesthe runtime container within which both the SPT engine and your ObjectGridEventListener will execute. As such, you need toensure that your executable code artifacts are available on the container server classpath.

These files are packaged within the download file that accompanies this article:

EventStreamProcessing.jarObjectGrid.xmlObjectGridDeployment.xml

To execute the scenario, it is necessary to add EventStreamProcessing.jar to the classpath of the WebSphere eXtreme ScaleContainerServer. Copy the ObjectGrid.xml and ObjectGridDeployment.xml configuration files, described above, to anappropriate location in your file system and specify them on the parameters within your container server startup script. In ourtest environment, the container server startup script in Listing 14 was used.

Listing 14. Container server startup scriptcall "%~dp0setupCmdLine.bat"

"C:\IBM\ObjectGrid\java\jre/bin/java" "-Xmx512m" "-classpath" "C:\IBM\ObjectGrid\java\lib\tools.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\objectgrid.jar;C:\IBM\ObjectGrid\ObjectGrid\session\lib\sessionobjectgrid.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\cglib.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\ogstreamquery.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\castor.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\commons-io.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\mx4j.jar;


13 of 16 7/23/09 2:15 PM

C:\IBM\ObjectGrid\ObjectGrid\lib\mx4j-remote.jar;C:\IBM\ObjectGrid\ObjectGrid\lib\mx4j-tools.jar;C:\IBM\ObjectGrid\ObjectGrid\properties;c:\IBM\ObjectGrid\EventProcessing\EventStreamProcessing.jar"

"com.ibm.ws.objectgrid.InitializationService" %1 -objectgridFile c:\IBM\ObjectGrid\EventProcessing\ObjectGrid.xml -deploymentPolicyFile c:\IBM\ObjectGrid\EventProcessing\ObjectGridDeployment.xml -catalogServiceEndpoints localhost:6000

An arbitrary container server name is specified upon the invocation of this script; for example, if the script resides withinc:\IBM\ObjectGrid\ObjectGrid\bin and a container server name of c0 is required:

c:\IBM\ObjectGrid\ObjectGrid\bin startEventProcessingContainer.bat c0

Upon successfully starting the container server, you should see output messages that confirm the availability of our twoconfigured ObjectGrid instances, as shown in Listing 15.

Listing 15. Successful container server startup[22/05/08 11:31:47:671 BST] fc00fc ServerImpl I CWOBJ1001I: ObjectGrid Server c0 is ready to process requests.[22/05/08 11:31:48:312 BST] 4b8e4b8e ReplicatedPar I CWOBJ1511I: CustomerPortfolioGrid:IBM_SYSTEM_ENTITYMANAGER_MAPSET:0 (primary) is open for business.[22/05/08 11:31:48:343 BST] 4d204d2 ReplicatedPar I CWOBJ1511I: CustomerPortfolioGrid:mapSet1:0 (primary) is open for business.[22/05/08 11:31:48:390 BST] 1d781d78 PeerManagerIm I CWOBJ8601I: PeerManager found peers of size 1[22/05/08 11:31:48:390 BST] 1d781d78 ServerAgent I CWOBJ7206I: New leader is (9.145.21.53:3912). Old leader was (<null>).[22/05/08 11:31:48:390 BST] 1d781d78 ServerAgent I CWOBJ7203I: Leader changed. New leader (9.145.21.53:3912) is elected in core group (DefaultZoneCG0) and reported to catalog server.[22/05/08 11:31:48:515 BST] 4d204d2 ReplicatedPar I CWOBJ1511I: StocksGrid:mapSet1:0 (primary) is open for business.[22/05/08 11:31:48:578 BST] 4b8e4b8e ReplicatedPar I CWOBJ1511I: StocksGrid:IBM_SYSTEM_ENTITYMANAGER_MAPSET:0 (primary) is open for business.

Initialize scenario

Once the container server has successfully started, it is necessary to prime the scenario, which involves the insertion of statedata into the CustomerPortfolioGrid ObjectGrid instance, defining portfolio holdings. This exercise also writes initial stockand foreign exchange ticker events into the StocksGrid ObjectGrid instance. A command line program has been provided forthis purpose. This program can be run from the command line using the following command, provided that the currentdirectory contains EventStreamProcessing.jar, and Java 5 is specified on the PATH variable.

Listing 16. Write initial stock and foreign exchange ticker eventsjava "-classpath" "c:\IBM\ObjectGrid\ObjectGrid\lib\castor.jar;c:\IBM\ObjectGrid\ObjectGrid\lib\commons-io.jar;c:\IBM\ObjectGrid\ObjectGrid\lib\objectgrid.jar;c:\IBM\ObjectGrid\ObjectGrid\lib\ogstreamquery.jar;c:\IBM\ObjectGrid\java\lib\tools.jar;EventStreamProcessing.jar"wxs.streamquery.example.ScenarioInit

Event stream emission

After checking that ScenarioInit ran successfully, you can commence to emit stock and foreign exchange ticker events. Onceagain, sample utility programs are provided in the download file for this purpose:

wxs.streamquery.emitter.StockEventStreamEmitterwxs.streamquery.emitter.ForexEventStreamEmitter

Example scripts are provided to encapsulate the command string required to execute each of these utility programs. Be aware


14 of 16 7/23/09 2:15 PM

that each utility requires two parameters: the first specifies the number of events to emit, and the second specifies a time delayin milliseconds between consecutive events.

For example, to emit 500 stock ticker events at intervals of 500 milliseconds, enter:

C:\IBM\ObjectGrid\EventProcessing>emitStockTickerEvents.bat 100 500

To emit 500 forex ticker events at intervals of 500 milliseconds, enter:

C:\IBM\ObjectGrid\EventProcessing>emitForexTickerEvents.bat 100 500

To facilitate the demonstration of this scenario, a utility program is provided that combines a “tail” function that works onWindows® platforms and has a highlighting capability. Configuration of the program to display the output written to thecontainer server SystemOut.log file, and to display stock and foreign exchange event output in contrasting colours, results insteady state output that resembles the following:

Figure 5. Output of time-averaged stock and forex values

To illustrate the portfolio stop loss detection capability, you must emit an event that will cause the calculated aggregateportfolio values to stray outside the allowable range. Once again, a sample program and script are provided for this purpose.

Open a command window and execute the emitSingleStockTickerevent.bat script. Notice that this command takes twoparameters: the ticker symbol of the stock, and the stock value:

C:\IBM\ObjectGrid\EventProcessing>emitSingleStockTickerEvent.bat IBM 10.0

In our development environment, this was sufficient to trigger a stock portfolio valuation alert that resulted in the followingoutput in our log display, highlighting utility program. (It might be necessary for you to inject several low value stock tickerevents to drive the averaged value low enough to trigger an alert.)

Figure 6. Output of aggregate portfolio valuation alert


15 of 16 7/23/09 2:15 PM

DownloadDescription Name Size Download method

Code sample EventStream_Attachment_23072008.zip 52 KB HTTP

Information about download methods

ResourcesWebSphere eXtreme Scale product information

WebSphere eXtreme Scale Information Center

The Complex Event Processing Blog

WebSphere eXtreme Scale wiki

IBM developerWorks WebSphere Extended Deployment zone

About the author

Alan is a Senior IT Specialist with IBM Software Group Services for WebSphere with more than 16 yearsexperience in middleware and internet-related technologies. For much of his career he has focused onTransactional Middleware systems, and their usage in the e-Business space. Most recently Alan has been dedicatedto the strategically important area of Business Process Integration. Of specific interest is the application ofopen-standards-based approaches to Business Process Integration, including the evolution of Service-OrientedArchitectures and their deployment upon an Enterprise Services Bus based infrastructure. At present Alan is amember of the Business Integration Solution Services Team, who form part of IBM Software Services forWebSphere, based at IBM's Hursley Park Laboratory in England.

Share this....Digg this story del.icio.us Slashdot it!


16 of 16 7/23/09 2:15 PM

Download - Event stream processing with WebSphere eXtreme Scale

Top Related