4 march 2008kaiser: coms e61251 coms e6125 web-enhanced information management (whim) prof. gail...

4 March 2008 Kaiser: COMS E6125 1

COMS E6125 Web-COMS E6125 Web-enHanced Information enHanced Information Management (WHIM)Management (WHIM)


Prof. Gail KaiserProf. Gail Kaiser

Spring 2008Spring 2008


Important Notice• Course website now located at

http://bank.cs.columbia.edu/classes/cs6125

• Tell me ([email protected]) about any problems, broken links or otherwise


Today’s Topic:

• More Web Services – some of the WS-* specifications


Reprise: Web Services• Web Services = distributed

applications, services and components, described using XML-encoded WSDL interfaces, that process XML-encoded SOAP messages

• XML, SOAP and WSDL constitute baseline specifications that provide a foundation for application integration


But…• Additional standards beyond this

baseline become necessary as WS applications become more complex, integrating multiple components across multiple organizations

• Otherwise, WS developers are compelled to implement higher-level functionality in proprietary and non-interoperable ways

• Solution?: WS-* set of standards


What Higher-Level Functionality?


Example: Department Store Chain Enterprise Application Integration

• Background: The chain discovered that different credit approval applications had been developed in various parts of the company.

• Solution: The chain exposed one credit approval application as a Web Service. They linked this to their point-of-sale, warehouse and financial applications.


Example: Department Store Chain Enterprise Application Integration

• Business Benefits: The chain was able to use the same credit approval application with the three distinct applications. As a result:

• Credit approvals became more consistent • Maintenance costs decreased


Tier 1 -- Enterprise Application Integration

• Companies initially use Web Services to integrate internal applications

• Web Services allow them to expose legacy applications in heterogeneous environments without having to rewrite significant amounts of code


Example: Car Rental Company

Interoperability with Key Partner

• Background: A major airline approached the car rental company about putting a link to the car reservation system on the airline’s website. Linking the two proprietary reservation systems presented an extreme challenge.

• Solution: The car rental company created a translation engine for sending data between the two systems.


Example: Car Rental Company

Interoperability with Key Partner

• Business Benefits: • Car rental company developed

another large sales channel

• Solution got to market quickly


Tier 2 -- Interoperability with Key Partners

• The next step is to integrate one or two key partners outside the company

• Web Services allow for interoperability between applications across the public Internet

• But companies must agree upon the technologies they will use to develop these interoperating Web Services


Example: Insurance Company

Interoperability across Several Companies

• Background: A large insurer needed to generate quotes for dental coverage and make them available on the intranet of one of their large corporate customers. But it had outsourced the maintenance of the dental providers directory and the credit rating service.

• Solution: The insurance company, credit rating service, and dental provider orchestrated these applications to generate a quote that was requested by the customer on a corporate intranet.


Example: Insurance Company

Interoperability across Several Companies

• Business Benefits: The insurance company considered this a transformational competitive advantage for the following reasons:

• It generated quotes in half the time of its competitors and provided them via a corporate intranet to one of its major customers.

• It automated existing business relationships at the level of multiple, interoperating applications. As a result, outsourcing became much more valuable, cutting the cost of quote generation by one third.


Tier 3 -- Interoperability across Multiple

CompaniesCompanies want to extend their computing out

to more partners and customers to build business ecosystems


Business Ecosystem Requirements: Security• The most common concern for

companies implementing Web Services solutions

• Developers need an end-to-end security architecture that is straightforward to implement across companies and trust boundaries


Business Ecosystem Requirements:

Addressing• Companies building Web Services

solutions are concerned about the scalability and fault-tolerance of the business ecosystems they are building

• Developers need a way of specifying messaging paths and the ability to configure those message paths dynamically


Business Ecosystem Requirements:

Reliable Messaging

• A key requirement for mission-critical applications

• Developers need an end-to-end guarantee of message delivery across a range of semantics such as: – at-least-once– at-most-once– exactly once


Business Ecosystem Requirements: Coordination

• Some applications require database-like transactions across companies

• ACID properties – atomic, consistent, isolated, durable

• But because of the nature of decentralized computing, developers need flexible compensation-based transaction schemes


Building Business Ecosystems

• The basic Web Services specifications enable interoperability between software components developed by different companies and residing on different infrastructures

• But most component model frameworks for use within an enterprise support security, reliability, transactions, etc.

• How can we add those capabilities to WS?


Composable Services• By adding more specialized Web Service

specifications that are independent but can be combined

• For example, it is possible to independently add transaction identifiers and reliable messaging sequence numbers

• The two extensions do not conflict with each other and are compatible with pre-existing message structures

• Developers and providers can integrate selected specifications that fulfill the requirements of their communicating processes


SOAP Inherently Supports Composition• SOAP uses a regular, multi-part message

structure: New message elements supporting new services may be added to message headers in a manner that does not alter the processing of existing functionality

• SOAP body is for the ultimate recipient, SOAP header blocks may be targeted at any entity along the message path


Messaging


Transports• HTTP, HTTPS, SMTP (Simple Mail

Transport Protocol)• Core communication mechanisms • Move blocks of "bytes" between Web

Services• This is only useful if participants can

convert the bytes into data structures that their code processes


Messaging• XML, SOAP• XML and XML Schema Definition (XSD)

provide the mechanisms for abstractly agreeing on message [data] structures

• SOAP defines the standard encoding for representing XML messages in the byte information that Web Services exchange over transports


Addressing• Messages and responses both go

somewhere and come from somewhere (and errors also need to be reported somewhere)

• By default, SOAP encodes the destination for a message with a URL placed in the HTTP transport

• The destination for the response is determined by the HTTP return address

• Builds on the basic browser-server model


Addressing

• The source and destination information are not part of the message itself

• But information can be lost if a transport connection terminates (e.g., if the response takes a long time and the connection times out)

• Or if the message is forwarded by an intermediary, perhaps routed over multiple transports

• Also does not allow for directing a response to a third party (e.g., request sent over HTTP but returned via SMTP)


WS-Addressing

• Provides a mechanism to place the target, source and other addressing information directly within the message

• Decouples address information from any specific transport model

• Supports both asynchronous and extended duration communication patterns

• Across multiple endpoint references• Does not match very well the request/response

model over a single HTTP connection (see blog entry), more applicable to other transports

http://www.w3.org/TR/ws-addr-core/


WS-Addressing Endpoint Reference (EPR)• A standard XML element providing a structured

approach to encoding fine-grained addressing • Only the address is required• May include reference properties that

distinguish between multiple services (or multiple versions of the same service) at the same address

• May include reference parameters that identify resources managed by a particular service


Message Addressing Properties

• To -- message destination, required (URI)

• Action -- an action value indicating the semantics of the message, corresponds to WSDL porttype, required (URI)

• From -- the endpoint of the service that dispatched this message (EPR)


Message Addressing Properties

• ReplyTo -- the endpoint to which reply messages should be dispatched (EPR)

• FaultTo -- the endpoint to which fault messages should be dispatched (EPR)

• Unique MessageId, required if there will be any response (URI)

• RelatesTo previous messages (a pair of URIs, indicating previous From and MessageId)

Example with Simple Endpoint References

<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wsa="http://www.w3.org/2004/12/addressing"> <S:Header> <wsa:MessageID> http://example.com/SomeUniqueMessageIdString </wsa:MessageID> <wsa:ReplyTo>

<wsa:Address>http://myClient.example.com/someClientUser </wsa:Address>

</wsa:ReplyTo> <wsa:FaultTo> <wsa:Address>http://myserver.example.com/DemoErrorHandler </wsa:Address> </wsa:FaultTo> <wsa:To>mailto:[email protected]</wsa:To> <wsa:Action>http://myserver.example.com/DoSomething</wsa:Action> </S:Header> <S:Body> … </S:Body></S:Envelope>


Example Extended Endpoint Reference

<wsa:EndpointReference xmlns:wsa="..." xmlns:example="...">

<wsa:Address>http://example.com/weather</wsa:Address>

<wsa:ReferenceProperties> <example:ServiceLevel>Basic </example:ServiceLevel>

</wsa:ReferenceProperties> <wsa:ReferenceParameters> <example:CityCode>NYC</example:CityCode> </wsa:ReferenceParameters></wsa:EndpointReference>


Security


Assurances

• It is not enough to simply exchange messages

• Applications and services reside in middleware and systems that provide valuable component services such as security, reliable messaging and transacted operations

• Web Services need a mechanism for interoperability between these facilities


Security• Supports authentication, message

integrity, confidentiality, trust, privacy• Federation of security between

different organizations


Security Requirements• A sends a message to service B• B partially processes the message and

forwards it to service C• HTTPS allows authentication, integrity, and

confidentiality between A-B and B-C• However, C and A cannot authenticate each

other, or hide information from B • For A, B and C to use userid/password for

authentication, they must share the same replicated user and password information

• Need “end to end” security


WS-Security• Defines mechanisms for associating

security related claims with a message• Signed, encrypted security tokens

– Username/password (BASIC-Auth)– x509 certificates (public key infrastructure)– Kerberos tickets (secret key)– XrML eXtensible rights Markup Language

(digital property rights)– SAML Security Assertion Markup Language

(single sign-on)

http://www.oasis-open.org/committees/download.php/21257/wss-v1.1-spec-errata-os-SOAPMessageSecurity.htm

http://web.mit.edu/Kerberos/

http://www.xrml.org/

http://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf


WS-Security

• A can generate a token that C can verify as having come from A, B cannot forge the token

• A can sign selected elements or the entire message, this allows B and C to confirm that the message has not changed since A sent it

• A can seal the message or selected elements, this ensures that only the intended service for those elements can use the information and prevents B from seeing information intended for C and vice versa


Trust• Security relies on pre-defined trust

relationships• Kerberos works because participants

trust the Kerberos Key Distribution Center

• PKI (public key infrastructure) works because participants trust the root certificate authorities

http://www.pki-page.org/


WS-Trust• Defines an extensible model for setting up and

verifying trust relationships• Allows Web Services to set up and agree on which

security servers they "trust" and to rely on these servers

• The key concept is a Security Token Service (STS) - a distinguished Web Service that issues, validates and exchanges security tokens

• An STS might issue a Kerberos token asserting that the key holder is Susan, based on Susan’s X.509 certificate issued by a trusted Certificate Authority

• Enables organizations using different security technologies to federate

http://docs.oasis-open.org/ws-sx/ws-trust/v1.3/ws-trust.html


STS


Secure Conversations• Some Web Service scenarios only involve the short

sporadic exchange of a few messages – which is what WS-Security was intended for

• Other scenarios involve long duration, multi-message conversations between the Web Services

• WS-Security may not be good enough for:– Repeated use of computationally expensive

cryptographic operations such as public key validation

– Sending and receiving many messages using the same cryptographic keys, providing more information that allows brute force attacks to "break the code"


Secure Conversations• Protocols like HTTPS use public keys to

perform a simple negotiation that defines conversation specific keys

• This key exchange allows more efficient security implementations and also decreases the amount of information encrypted with a specific set of keys


WS-SecureConversation

• WS-SecureConversation provides similar support to WS-Security

• Participants may use WS-Security with public keys to start a "conversation" or "session ", and then use WS-SecureConversation to agree on session specific keys for signing and encrypting information

http://docs.oasis-open.org/ws-sx/ws-secureconversation/v1.3/ws-secureconversation.html

http://docs.oasis-open.org/ws-sx/ws-secureconversation/v1.3/ws-secureconversation.html


Federation• Sometimes a set of organizations need to

establish a single virtual security domain• For example, a travel agent, an airline and a

hotel chain may set up such a federation• An end-user that "logs into" any member of

the federation has effectively logged into all of the members


WS-Federation• WS-Federation defines several models

for providing federated security through WS-Trust and WS-SecureConversation

• Customers often have "properties" when they deal with an enterprise

• An example is a preference for window or aisle seats, or a midsize car

• WS-Federation allows the members to set up a federated property space

http://www.oasis-open.org/committees/download.php/26323/ws-federation-1.2-spec-ed-03.doc


WS-Federation• Properties about individuals may be closely

held for privacy protection or because the information provides a competitive advantage to a specific federation member

• WS-Federation supports a pseudonym model• For example, users that have authenticated to

the travel agency have agency-generated "aliases" in their interactions with the airline or hotel


Reliable Messaging


Reliable Messaging• In an Internet world, almost all communication channels

are unreliable - messages disappear or are duplicated, connections break

• Without a reliable messaging standard, Web Service application developers must build these functions into their applications

• The basic approaches and techniques are well understood, e.g., many middleware systems ensure messages have unique identifiers, provide sequence numbers, and use retransmission when messages are lost

• If Web Service developers implement these models in their applications, they may make different assumptions or design choices, resulting in little if any reliable messaging


WS-ReliableMessaging• Defines mechanisms that enable Web

Services to ensure delivery of messages over unreliable communication networks

• Supports bridging two different infrastructures into a single, logically complete, end-to-end model


WS-ReliableMessaging• The RM Source MUST assign each reliable

message a sequence number beginning at 1 and increasing by exactly 1 for each subsequent reliable message

• Every acknowledgement issued by the RM Destination MUST include within an acknowledgement the range or ranges of the sequence numbers of every message successfully received by the RM Destination and MUST exclude sequence numbers of any messages not yet received


WS-ReliableMessaging• Delivery Assurances – AtMostOnce,

AtLeastOnce, ExactlyOnce, InOrder• Protocol Elements – Sequence,

Sequence Acknowledgement, Request Acknowledgement, Sequence Creation, Sequence Termination

• Policy Assertions – SequenceCreation, SequenceExpiration, InactivityTimeout, RetransmissionInterval, AcknowledgementInterval


Transactions


Transactions• A complex business scenario may require

multiple parties to exchange multiple sets of messages

• The multiple messages exchanged between participants constitute a logical "task" or "objective"

• The parties must be able to: – Start new coordinated tasks. – Associate operations with their logical task -

the parties may be performing multiple such tasks at the same time

– Agree on the outcome of the computation


WS-Coordination• General mechanism for starting and

agreeing on the outcome of multiparty, multi-message Web service tasks

• Coordination context is a message element that flows on all messages that Web Services exchange during the computation

• The coordination context contains the WS-Addressing endpoint reference to the coordination service and the endpoint contains information to identify the specific task being coordinated


Coordination Service• Starts a coordinated task, terminates a

coordinated task, allows a participant to register in a task, and produces a coordination context that is part of all messages within a group

• Includes an interface that participating services use in order to be informed of the outcome of the coordinated task


WS-AtomicTransaction• Defines a specific set of protocols that

plug into WS-Coordination to implement traditional two-phase atomic transactions

• For activities that require the traditional atomic, consistent, isolated, and durable (ACID) properties

• Usually short-lived


WS-AtomicTransaction• Atomic, two-phase model is only with

respect to the services involved• Sites or infrastructure offering services

may advertise two-phase commit, but use some other intra-enterprise model like compensation or versioning

• This makes a simple two-phase commit model more useful for long-running Internet computations


Business Activities• May consume many resources over a long

duration• May involve a significant number of atomic

transactions• Individual tasks within a business activity can

be “seen” prior to the completion of the business activity - their results may have an impact outside of the computer system

• Responding to a request may take a very long time - human approval, assembly, manufacturing or delivery may have to take place before a response can be sent


Business Activities• In the case where a business exception

requires an activity to be logically undone, transactional abort is typically impractical/impossible

• Exception handling mechanisms may require business logic, e.g., in the form of a compensation task, to reverse the effects of a completed business task


WS-BusinessActivity• Another set of protocols that plug into WS-

Coordination, to coordinate activities that apply business logic to handle business exceptions

• Actions are applied immediately and are permanent

• Compensating actions may be invoked in the event of an error

• Enables existing business process and work flow systems to wrap their proprietary mechanisms and interoperate across trust boundaries and different vendor implementations


Metadata


Description• The transport and message specs allow Web

services to communicate using messages• But how do participants know what the

messages are? • How does a Web Service describe the messages

it sends and receives? • Using a Web Service requires an understanding

of the messages the Web Service will consume and produce


XSD and WSDL• XML Schema allows developers and

service providers to define XML types for data structures, e.g., a purchase order, and messages, e.g., the CreatePO message

• WSDL allows a Web Service to document the messages it receives and sends - what "actions" or "functions" the service performs in terms of the messages it receives and sends


Requirements and Expectations between

Requesters and Receivers• WSDL and XSD define the service's interface

syntax, but do not express information about what the service requires/expects of the caller and vice versa

• For example, does the service require security or implement transactions?

• WS-Policy enables a service to specify the functional assurances that they expect from and provide to callers (and intermediaries)


WS-Policy

• Extensible language for expressing the requirements, capabilities and preferences of a service

• Assertions represent an individual preference, requirement or capability

• Usage: Required, Optional, Rejected, Observed, Ignored

• Operators: All, ExactlyOne, OneOrMore• Applied to domain-specific policy subjects (e.g.,

from WS-SecurityPolicy)


Example<wsp:Policy xmlns:wsse="..." xmlns:wsp="...">

<wsp:ExactlyOne> <wsse:SecurityToken wsp:Usage="wsp:Required" wsp:Preference="100"> <wsse:TokenType>wsse:Kerberosv5TGT

</wsse:TokenType> </wsse:SecurityToken> <wsse:SecurityToken

wsp:Usage="wsp:Required" wsp:Preference="1"> <wsse:TokenType>wsse:X509v3

</wsse:TokenType> </wsse:SecurityToken></wsp:ExactlyOne>

</wsp:Policy>


Obtaining Descriptions• XML, XSD, WSDL and WS-Policy support

describing the interface and service assurances for a Web Service

• But how do potential users of the service find this information?

• Currently the most common approach is through email exchanges, manual (human) online lookup, or word of mouth

• A more general purpose, scalable model is necessary – although has not (yet) proven practical outside an intranet setting


UDDI

• UDDI = Universal Description and Discovery Interface

• Query UDDI at design time, to find services compatible with requirements

• Query UDDI at runtime, when the caller "knows" the interface it requires and searches for a service that meets its functional requirements or is provided by a well-known partner


WS-MetadataExchange• The caller service may go directly to

the callee service to obtain information via SOAP messages following WS-MetadataExchange specification

• Used when developers already have a reference to a service and need to obtain details

• Obtains relevant WSDL, XSD, WS-Policy, etc.


Business Processes• The “big picture”, or workflow, of an

enterprise process that spans multiple applications

• Either intra- or inter- organizational• May take an orchestration (one owner)

or choreography (joint owners) point of view

• Many “business process management” products predate Web Services


WS-BPEL• Previously known as BPEL4WS = Business

Process Execution Language for Web Services• Supports service composition• Enables developers to define the structure and

behavior of a set of Web Services that together implement a shared business solution

• The composed solution is itself a Web Service, which supports HTTP/SOAP messages and defines its interface using WSDL (and WS-Policy)


WS-BPEL Processes• Abstract processes specify the mutually

visible message exchange behavior of each of the parties involved in the protocol, without revealing their internal behavior

• Executable processes model actual behavior of each participant in a business interaction

• An entire programming system where conventional Web Services are the atomic functions


BPEL Servers• BPEL typically used as on server

side• May be deployed to serve and

track customer requests, possibly as a proxy to legacy systems

• Process instances created on demand


BPEL Chain of Events• Web Service client (user) requests something

from the server (perhaps via a browser or another server)

• Client request is received by the BPEL server. • Client request is identified as a new request and

a new BPEL process is used to serve that client • Client continues to interact with the BPEL

process, making requests, etc., until the interaction is complete

• BPEL process disappears and client goes about it's business


Some BPEL Notation• import – import WSDL file (for endpoint

descriptions) or XSD file (for data types)• partnerLink – maps to an instance of a

web service port (partnerRole and/or myRole)

• variable – container for an XSD value or WSDL message

• assign - Variable assignment• invoke – invokes web service endpoint


BPEL Example<?xml version="1.0" encoding="UTF-8"?><process

xmlns="http://schemas.xmlsoap.org/ws/2003/03/business-process/"xmlns:print="http://www.eclipse.org/tptp/choreography/2004/engine/Print"

<import importType="http://schemas.xmlsoap.org/wsdl/"

location="../../test_bucket/service_libraries/tptp_EnginePrinterPort.wsdl" namespace="http://www.eclipse.org/tptp/choreography/2004/engine/Print" />

<partnerLinks><partnerLink name="printService"

partnerLinkType="print:printLink" partnerRole="printService"/>

</partnerLinks>

Cont


BPEL Example Cont

<variables><variable name="hello_world"

messageType="print:PrintMessage" /></variables>

<assign><copy>

<from><literal>Hello World</literal></from><to>$hello_world.value</to>

</copy></assign>

<invoke partnerLink="printService" operation="print" inputVariable="hello_world" />

</process>


Many More…• WS-Transfer• WS-Enumeration• WS-Eventing• WS-ResourceFramework• WS-MetadataExchange• WS-Notification• WS-Management• …


Summary• WS-* specs add orthogonal features to

SOAP headers• Implement a component model

framework focused primarily on security, reliability and fault tolerance

• Ease development of cross-organizational applications, organized using WS-BPEL or otherwise


Next Assignment: Project Proposal

• Preliminary Proposal due Monday March 10th

• Two pages• Post in Preliminary Project

Proposals folder on CourseWorks


• Build a new system or extend an existing system – submit code, demo system

• OR evaluate/compare one or more existing system(s) – submit procedures and findings, show system(s)

• You may "continue" your paper topic towards the project, or do something entirely different




• Sketch the project you have in mind, including both the functionality or evaluation you aim to achieve and the technology you plan to use to do so

• In the case of multi-student teams, also propose a "management structure“– who is in charge of scheduling team meetings– who is in charge of the code repository and version

control (e.g., cvs, svn)– who is in charge of collecting and editing documentation

• You will have the opportunity to submit a revised project proposal (with further details) following feedback from the teaching staff


Also Keep In Mind Your Upcoming Full Paper

• Due Friday March 14th

• Approximately 15 pages• Must be in a format I can read!• File (uploaded to CourseWorks)

must follow filename convention


Make Sure To Include In Full Paper

• Title• Author name and contact information• Abstract (approximately 200 words)• Introduction, several body sections,

Conclusions• Reference list – must contain the full

bibliographic information available and most of the entries must be cited in the appropriate place(s) within the prose


Reminders

• Preliminary project proposal due March 10th

• Revised project proposal due March 31st

• Full paper due Friday March 14th


Important Notice• Course website now located at

http://bank.cs.columbia.edu/classes/cs6125

• Tell me ([email protected]) about any problems, broken links or otherwise




Prof. Gail KaiserProf. Gail Kaiser

Spring 2008Spring 2008

4 march 2008kaiser: coms e61251 coms e6125 web-enhanced information management (whim) prof. gail...

Documents