dynamic data exchanges with the java flow processor presenter: scott bowers date: april 25, 2007
TRANSCRIPT
2
History of Java Flow Processor (JFP)
Developed in 2004 for NJDEP to facilitate dynamic data publishing and dynamic data submittal services
Used at New Jersey DEP to support Facility Registry System (FRS) submittal and query services
(production), RCRA Handler and Permitting submittal and query services
(production) Used at Mississippi DEQ to support
FRS submittal and query services (testing only) Use at Kentucky DEP to support
RCRA Handler, Permitting, and Corrective Action (testing only) This utility is freely available to any Exchange Network
partner!
3
What is the Flow Processor?
The Flow Processor: Is a generic Java application that knows nothing about
specific Data Flows. Is aware that data is retrieved from a database and needs
to be formatted as XML. Is dynamic, allowing SQL Selects and output XML to be
changed without modifying the core Java code. Is extensible, allowing additional Java extensions to be
created and assigned to data flows. Can export or return data (submit to CDX or save to
permanent storage) in a variety of formats (XML, ZIP, or TXT).
Can be invoked with or without a Node
4
What is a Flow? The Flow Processor operates on an entity called a
Flow A Flow is identified by a unique name The only mandatory component of a Flow is the
model The model consists of a primary SQL query, and Zero or more detailed queries
The remaining components of a Flow are optional Zero or more PreProcessors (Java class) An XML Translation (XSLT) Zero or more ExportManagers (Java classes)
5
Simple Generic XML Submittal to CDX
Get Flow Config Data
XML(SQL)
JFP
ExportGenerate
XMLFrom SQL
Transform
XSLT
ExportMgr
Source
ExecuteSQL
TempXML
XSLT
FinalXML CDX
*Execution is from Left to Right JFP Step
6
Data Retrieval Processing: Contact Example
Primary Queryselect contact_id “id”
from ContactsWhere dept=‘Sales’
id100200300
Detailed Query 1select contact_id, address, city, state, zip from AddressWhere contact_id=#id#
Detailed Query 2select contact_id, home, cell from Phone_NumbersWhere contact_id=#id#
Detailed Query 3select contact_id,region from Sales_RegionsWhere contact_id=#id#
<address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip></address_info><address_info> … </address_info><address_info> … </address_info>
<phone_info> … </phone_info><phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell></phone_info><phone_info> … </phone_info>
<region_info> ... </region_info><region_info> ... </region_info><region_info> <contact_id>300</contact_id> <region>Southwest</region></region_info>
7
Transformation Step: Contact Example
<address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip></address_info><address_info> … </address_info><address_info> … </address_info>
<phone_info> … </phone_info><phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell></phone_info><phone_info> … </phone_info><region_info> ... </region_info><region_info> ... </region_info><region_info> <contact_id>300</contact_id> <region>Southwest</region></region_info>
<get_contact_data>
</get_contact_data>
<SalesForceDataService version="1"> <SalesContact id="100" region="Midwest"> <ContactInformation> <Address>100 Maple Ave.</Address> <City>Detroit</City> <State>MI</State> <ZipCode>11111</ZipCode> <PhoneNumbers> <Number type="Cell">717-555-3456</Number> <Number type="Home">717-555-1278</Number> </PhoneNumbers> </ContactInformationn> </SalesContact> <SalesContact id="200" region=“Midwest"> ... </SalesContact> <SalesContact id="300" region=“Midwest"> ... </SalesContact></SalesForceDataService>
XSLT
Temporary XML from Data Retrieval
Final XML OutputPost Transformation
8
Extending the Java Flow Processor The Flow Processor can be extended by developing custom
PreProcessors and ExportManagers classes. At runtime, the Flow Processor Engine uses Java Reflection to
instantiate a PreProcessor or ExportManager class. A Java class that acts as a PreProcessor must implement the
Java Interface FlowPreProcessManager. A Java class that acts as a ExportManager must implement the
Java Interface FlowExportManager. Several Java extensions are included in the JFP:
DefaultPreProcessManager – Issues zero or more INSERT or UPDATE statements defined in the Flow Processor Settings database tables.
CDXExportManager – Submits an XML payload to a URL. FileExportManager – Writes the resulting output to a flat file.
9
Flow Processor Complete Processing StepsFlo
w P
roce
ssor
Eng
ine
Time
Use the configuration file to locate the Data Access Definition files
Configuration
Data Access Definitions
Use the Data Access Definition files to locate and access the Flow Settings
Flow SettingsUse the Flow Settings to determine the XSL Stylesheets, FlowPreProcess Managers and FlowExportManagers
XSL Stylesheets
Flow PreProcess Manager(s)
Flow Export Manager(s)
Execute each FlowPreProcess Manager class in the order specified by the Flow Settings
Use the Data Access Definitions and Flow Settings to retrieve the data of the Flow
Source Database
Final Output File
For each data set retrieved, build the generic XML data format, and apply the XSL Stylesheets to convert the data into the final format
Execute each FlowExportManager class in the order specified by the Flow Settings
10
Deployable Components The Java Flow Processor engine, default Pre-
Processors, and Export Managers are packaged into a single Java Archive file (cgi-flow-process.jar)
It requires several third-party Open Source Java libraries, most notably: iBATIS – Provides the data access layer from the
source database and SQL mapping utilities Apache – Several Java libraries provide support for
XML parsing, XML Transformation, Connection Pooling, and Web Services (AXIS)
Log4J – Provides logging service
11
Invoking the Flow Processor Invoked from a DOS command line or Unix shell script
using the FlowProcessorCmdLine class. Accepts just one configuration file to configure the Flow
Processor Called within existing Java applications using the
FlowProcessor class directly. Two public methods are available: performSubmit – This requires only a Flow name and
returns a message of the resulting operation performQuery – This requires a Flow name and a Java
Map of potential query arguments. This returns the resulting byte stream from the JFP engine.
12
Node Functional Specification 1.1
<message name='Query'> <part name='securityToken' type='xsd:string'/> <part name='request' type='xsd:string'/> <part name='rowId' type='xsd:integer/> <part name='maxRows' type='xsd:integer/> <part name='parameters' type='typens:ArrayOfstring'/> </message> <message name='QueryResponse'> <part name='return' type=’xsd:string'/> </message>
Dynamic Node Query Services with JFP
FlowProcessor class
public byte[] performQuery( String flow_code, int start_row, int max_rows, Map arguments)
Node query()FlowProcessor performQuery()
JFP Engine
Byte StreamXML
13
Dynamic Node Query Processing Steps Convert Node Functional Specifications 1.1 WSDL into Java
code (e.g., Node11 class) public String query(String securityToken, String request, String
startRow, String maxRows, String[] parameters) Modify Node11’s query() method to retrieve an instance of the
FlowProcessor class Either use the incoming request parameter as the Flow code or
translate the incoming request into the appropriate Flow code Convert the incoming parameters array into a Java Map Call the FlowProcessor’s performQuery() method with correct
Flow code and Map of incoming parameters Convert the returning byte[] into a String and return this as the
result of the Query method
14
Using Apache Formatting Object Processing (FOP), stylesheets can transform XML into PDF
SQL Reports can be built and stylesheets can transform output into HTML reports.
Other Output Possibilities with JFP
Source
Gather Configuration Information
JFP Configuration
Data
HTML
Desired Output XSLT
SQL ConfigurationAs XML
1 2a
Temporary XML
2
1b
1a
Generate Temporary XML
Payload
3a
Transform XML into Final Payload
Desired Output XSLT
3b
2b
Final XML Payload
Text File
3
3c
3c
3c
4 Export Payload
CDX
File System
Other System
3c
4a
4a
4a4a
4b
4b
4b