CA*net 4 Research Program Update – UCLP Roadmap for creating User Controlled and Architected Networks using Service

Oriented Architecture

Draft—Draft—Draft—Draft—Draft—Draft—Draft—Draft

You are free to distribute this paper as long as the following the disclaimer is included:

“This is a discussion paper intended to provoke further thought and debate on the implications of web services and web services workflow for allowing users to architect their own Articulated Private Networks

on a common network substrate connecting and managing geographically distributed router, switches research instruments and sensors to networks based on the original User Controlled LightPath (UCLP)

software developed for CANARIE to manage geographically distributed optical and SONET/SDH cross connects and switches. Please send all comments, suggestions and criticisms to the author”

Author: Bill.St.Arnaud@canarie.caLast revised draft: January 3, 2006

Abstract

Around the world there are several initiatives to develop the next generation canonical Internet network architecture . This discussion paper provides an alternative approach largely based on two significant development, first: the trend for more users to own, control and manage their own network resources and secondly: the demand by big science and large enterprises to have dedicated network resources for the data flows generated by their high end applications. As opposed to other approaches a key assumption in this discussion paper in regards to network architecture is that there is NO canonical network architecture. Instead this document describes new features and enhancements to the current implementations of the User Controlled LightPath (UCLP) software which will enable users to define their own packet or switched based network architecture including topology, routing, virtual routers, switches virtual machines and protocols based on the concept of many separate and independently managed Articulated Private Networks (APN) operating on top of one or more network substrates across different ownership domains. APNs can be considered as a next generation Virtual Private Network where, rather than signaling for a single end to end connection, a user can create a complex network structure, in any way they wish, by binding together layer 1 through 3 network links, instruments, computers, time slices and virtual or real routing and/or switching nodes. This capability is enabled through UCLP by representing all such network element, devices and links as web services, and by using web services workflow as the tool to allow the user to bind together their various web services to create a long lived APN instantiation. With web services workflow the user also has the ability to offer all, or portions of their APN as a web service (or set of services) in is own right to other downstream users.

1

1.0 Introduction

On August 28, 2005 the National Science Foundation (NSF) in the United States announced the Global Environment for Networking Investigations (GENI) [NSF-GENI]. Research that falls under the broad conceptual umbrella of this initiative will focus on designing new network architectures and services that range from new wireless and sensor devices to customized routers and optical switches to control and management software. The objective of the program is an attempt to “de-ossify” the Internet and develop and architecture that will allow an easy migration strategy such that users and applications can migrate effortlessly from the current Internet to a future more robust and secure environment. As such it is felt that a national testbed is necessary to carry out research in defining new network architectures.

One proposed architecture under the GENI framework is “Network Virtualization” [VIRTUALIZATION] which provides for multiple virtual networks to co-exist on top of a shared “substrate”. Different virtual networks provide alternate end-to-end packet delivery systems and may use different protocols and packet formats. Virtual networks may also be interconnected with virtual routers and interconnect wireless, instrument and sensor networks as illustrated in Figure 1.0

Figure 1.0 GENI Network Virtualization

The original concept for User Controlled LightPaths (UCLP)[UCLP-DESIGN] as subsequently updated with the UCLPv2 program [UCLPv2] has many similar objectives as that with the proposed GENI program in particular relation to the concept of network virtualization. The major difference between these concepts is that UCLP assumes there is no predefined canonical network architecture. Rather the user is given the tools to build any variation of virtual (and real) network architecture that so desire on top of one or

2

more substrates across different ownership domains. This user defined network can be packet or switched and the user decides what type of topology, restoral, protection and routing that may be necessary for their particular requirements. In essence the future network architecture has no predefined network architecture. Instead a substrate is provided which is capable of supporting many separate user defined networks which can be assembled through a series of lego block type web services as illustrated in Figure 1.1.

Figure 1.1 UCLP Network Virtualization

The major differences between UCLP and GENI is that in UCLPv1 the network was assumed to be made up only of wavelengths and optical switches which could be partitioned into smaller lightpaths and where the lightpath owner could be given control of the optical or SONET cross connects that touched that particular lightpath. UCLPv2 allows for the user control of both wavelengths, MPLS tunnels, IP packet networks and Ethernet networks.

UCLP allows for a creation of a set of VPNs which can be configured into a topology by the user decided upon by the user including meshes, rings, etc. As well UCLP allows the user to reconfigure the topology of this mesh of VPNs and change their shape, size and path. Hence the moniker of “articulated” Private Networks.

Articulated Private Networks (APNs), in many ways are similar to the GENI concept of many virtual networks sharing a common network substrate linking together virtual instruments, router and devices. With APNs all the virtual links and devices are represented as web services and are linked together via web service workflow tools as

3

Substrate Router

InstrumentWS

SubstrateSwitch

ParentLightpathWS

TimesliceWS

Child Lightpath WS(may run over IPEthernet, MPLS, etc

GMPLSDaemon WS

APN

VirtualRouterWS Wireless Sensor

Network

Substrate Router

InstrumentWS

SubstrateSwitch

ParentLightpathWS

TimesliceWS

Child Lightpath WS(may run over IPEthernet, MPLS, etc

GMPLSDaemon WS

APN

VirtualRouterWS Wireless Sensor

Network

described in more detail in the following section. In GENI there is yet no defined process for creating virtual links and connecting them to virtual devices and/or time slices.

By representing the substrate, child lightpaths, switches, routers and time slices or processes the user with UCLPv2 has complete flexibility and control to architect the resulting network in way they wish using web service workflow tools.

1.1.1 Market drivers for UCLP

One of the drivers for UCLP is the growing trend for organizations such as universities, hospitals, schools and businesses to acquire their own fiber and point to point wavelengths from different suppliers. These organizations would like to manage these facilities as part of their own network domain, and, in turn offer a variety of downstream services to their users such as APNs, VPNs, etc. These needs are very similar to the “carrier’s carrier” market where carriers want to service customers outside of their facilities based footprint by acquiring leased layer 1 or layer 3 services from some other carrier and interconnect these facilities with virtual routers and switches. The driver for “carrier’s carrier” solutions is increasingly be driven by the large enterprise as well as other competitive local exchange carriers.

Another important driver for UCLP is the growing recognition around the world that there are many large remote sensor and instrumentation projects whose large data flows will need to be delivered to computing facilities many thousands of kilometers away. As well, researchers will want to remotely control these instruments and, from time to time, re-direct the flow of data to different computing destinations. Examples of such facilities include the European eVLBI project- JIVE [E-VLBI], the US-Canada undersea network Neptune [NEPTUNE], the Canadian Light Source Synchrotron project (CLS) [CLS], the Ottawa University-NRC Nuclear Magnetic Resonance Instrument Facility [NRC-NMR] , and of course, the granddaddy of them all the CERN Large Hadron Collider (LHC) [LHC] .

Up to now the interconnection of these instruments to networks and remote computers was static and pre-configured by network or software engineers using VPNs or lightpaths. Increasingly researchers need to do their own traffic engineering and control how these instruments are connected to the network and redirect or share the data flow amongst different geographically distributed researchers. As well they may want to control the transport mechanisms on how the data is delivered as for example through their own IP routed network with guaranteed QoS parameters.

In some cases the expected data flow demands of these projects, in terms of throughput and duration will easily exceed by order of magnitude, that of conventional Internet traffic flows. In anticipation of these requirements CANARIE has been leading the development of User Controlled LightPaths [UCLP] in order to permit researchers to create “discipline” or “application” specific Layer 1 VPN or IP networks which would create separate high speed networks for these specific applications. As opposed to traditional VPNs the UCLP allows end users to do their own traffic engineering, change

4

the topology of their collection of VPNs, and cross connect their collection of VPNs over multiple domains to VPNs operated by other users as shown in Figure 1.2.

In this example we see an APN created from three separate independent managed domains and in turn a “child” APN has been created out of the mult-domain APN. The nodes can be either virtual routers or switches. The nodes that indicate that they have been partitioned indicates that control of the virtual router or switch has also been handed over to the owner of the APN or child APN, which ever the case may be.

Figure 1.2 Creation of APN from multiple independent managed domains

1.1 Web Services Workflow

With UCLPv2 it was proposed that Service Oriented Architecture (SOA) [SOA] new technologies of web services workflow and/or orchestration, that were originally conceived for eBusiness applications, would be the underpinning architectural framework for extending UCLP to allow the interconnection of instruments, time slices and sensors and for incorporating virtual routers and switches.. Specifically Web Services Description Language (WSDL) [WSDL] and Work flow languages such as Business Process Engineering Language (BPEL) [BPEL-INFO] were proposed as the implementation tools for such a reference framework.

One of the major attractions of web service and work flow tools is that they provide a consistent and standardized set of interfaces to all sorts of process including instruments and networks and allow users to bind processes together as required for particular applications, unconstrained by the original design requirements.

5

Domain CDomain A Domain B

Multi-Domain APN

Child APNPartitioned Node

Pass Through Node

Domain CDomain A Domain B

Multi-Domain APN

Child APNPartitioned Node

Pass Through Node

Another attraction of many work flow tools is their service oriented nature as opposed to the document oriented architecture of most web service standards. For example every BPEL orchestration is represented as new WSDL web service and so it is possible to encapsulate web service orchestrations within other web service orchestrations allowing for a truly recursive object oriented architecture. BPEL as well supports life time management tools for the leasing and consumption of resources.

There are many other possible tool service and object oriented sets including some of the original distributed object oriented middleware such as DCOM and CORBA. Alternatively Jini/Javaspaces as also been used in such applications. There are many approaches with various pros and cons that would be an entire paper in itself. But for a variety of reasons there seems to be an industry trend to web services and web services work flow language tools as a common platform for such applications. Various standards bodies are migrating their protocols to web services including the Object Management Group Unified Modeling Language (UMLv2), UPnP Forum for connecting devices in the home, The Instrumentation, Systems, and Automation Society (ISA) and many others.

1.2 e-Business tools and instrument/sensor networks

The application of e-Business tools to the control of networks, instrumentation and sensors may also have large impact on many industrial and business process control applications. To date the e-Business revolution has had the biggest impact on “information" industries and processes such as financial trading, business transactions, on-line shopping and information services. But with UCLP and web services for instruments the same tools can be used to interconnect "physical" processes and process control systems in manufacturing, agriculture, pulp and paper, oil and gas, power distribution, marine, transport, environmental etc.

To date many of these process control systems and Supervisory Control And Data Acquisition (SCADA) systems use proprietary or industry unique protocols and network architectures. The use of web services and web service workflow technologies promises to provide a standardized and open interface to these systems and allow them to be seamlessly integrated into existing e-Business applications. Just as importantly the web service and grid security standards can then also be adopted to these systems, many of which have only rudimentary security.

Although many of these industrial applications will only encapsulate instruments and sensors connected to local area networks, there will be requirements for wide area network applications as well. A simple illustrative example is to use these tools to interconnect oil and gas sensors on the ocean floor across a network to computer systems in oil refineries to accurately control the flow of raw crude to the refinery with less inventory in ships and storage tanks [IBM-MQ]. Another example is to connect agriculture sensors networks in wheat fields over the network with remote sensing databases to provide future traders accurate and instant estimates of crop production. Or weather and environmental sensors that can simultaneously feed data to remote

6

supercomputers for weather forecasts as well as to local users who may need more immediate and local information.

1.3 Definition of Layer 1 VPNs, LightPaths and Articulated Private Networks

The terms “LightPaths”, “Layer 1 Virtual Private Networks (VPNs)”[L1VPN] and “Articulated Private Networks (APNs)” are used extensively in this document. VPNs have normally been associated with statistical multiplexed networks while layer 1 facilities are usually circuits like STS channels or optical wavelengths. Layer 1 VPNs are generic to describe a number of layer “1” real or virtual network connections such as optical VPNs, concatenation of STS circuits etc. This is also consistent with the ITU definition of a Layer 1 VPN [ITU_Y.1312].

In the original UCLP design document [UCLP-DESIGN] a lightpath was defined as any communications link that could be treated as a circuit with fixed and guaranteed bandwidth. In addition some types of lightpaths can be subtended into smaller lightpaths while retaining the original attributes. Optical wavelengths, STS channels, MPLS with QoS, ATM CBR PVCs and 802.1 p/q links were therefore considered as lightpaths under that original definition.

For the purposes of the UCLP roadmap it is proposed that the fixed bandwidth restriction be removed in defining a lightpath, and instead the definition be expanded to include any channel or link where an user can control the end points and topology.

To distinguish between UCLP services and traditional end-to-end VPNs, whether at layer 3 or layer 1 we propose to use the term “Articulated Private Networks (APNs). Because the term lightpath is used with UCLP, many people assume it is a protocol to provide end-toend bandwidth on demand or allow for scheduled end-to-end bandwidth. Although UCLP can be used for these types of solutions, this was not its primary intended application. It’s primary purpose is to allow the partitioning of lightpaths and switches to be managed by third parties for the creation of multiple virtual VPNs sharing a common substrate or parent VPN. The outcome of this partitioning may coincidentally result in the creation of an end-to-end lightpath and/or inter-domain optical VPN, but not necessarily so. The defining characteristic of UCLP is that the user can reconfigure the partitioned lightpaths and cross connects at each node resulting in an “Articulated” private network. The articulated private network concept can be extended beyond the physical network links to devices outside of the network such as instruments, sensors, computers and so forth. As well the APN can incorporate internal virtual routers, switches and other devices to make up an entire network solution. Again this does not necessarily mean the APN must be “end-to-end”. A single APN can be thought of as a complex “object” linking together both virtual and real network elements, instruments and so forth, which when linked to other APNs may result in a complete end-to-end solution.

1.4 Definition of a “User”

7

The term “user” is used frequently in this document. It causes considerable confusion amongst readers as they assume it mean an “end-user”. However as explained later in this document, because UCLP exhibits inheritance the term user is generic reference to anyone who consumes the web services offered by an entity. The user could be a physical network administrator, a campus network administrator, a departmental network administrator or ultimately ( but probably least likely) a member of the general public. In each case the parent may aggregate and abstract a collection of web services and advertise them as single instantiated web service to their child users.

1.5 Security, Authorization and Authentication

With more and more devices and software processes linked to local and wide areas networks, security, authentication and authorization becomes a major concern. It has becoming increasingly recognized that centralized single system firewalls and authentication/authorization system are inadequate. Many enterprise and process control networks are extremely complex systems and have numerous backdoors, multiple network connections and other weaknesses.

The alternate approach is to compartmentalize security by building a complete security suite around each instrument and software web service. Each service or aggregation of services is then responsible for its own internal authorization, authentication and accounting (AAA) each of which also can be exposed as web services and orchestrated into a new service offering. This approach therefore necessitate some sort of delegated certificate security architecture.

The whole of security for distributed resources in multiple independent management domains with computer to computer interaction is an entire separate discussion topic in its own right and so will only be briefly touched upon in this document. However, it is believed that by complying to some of the grid standard for security which employ the concepts of delegated certificates and virtual organizations may be the best approach for the management and control of APNs

1.6 Important concepts and assumption presented in this document

There are several important inter-related concepts and assumptions presented in this document that will require further analysis and discussion. These are highlighted here to guide the reader through some of the basic design parameters and objectives. It is possible that these design parameters or objectives may not be fully realized in any actual working implementation, but they are presented here to set a framework and a “mind set” of the underlying architectural philosophy, namely:

(a) With UCLPv2 the first rule of network architecture is that there is no network architecture;

(b) Exposing all software (and/or time slices), hardware processes, network elements, individual lightpath and cross connect as web services;

8

(c) Extension of the “network” into the application and/or computer;

(d) Using web services and web services workflow to build a universal control plane capability across instruments, lightpaths, cross connects, networks and software with connections and work flow state maintained by the end user orchestration;

(e) Eliminating the concept of network layers which only communicate to the layer above and below such as the OSI 7 layer stack and replacing them with a stack of services which can communicate on a peer to peer basis with any other web services, anywhere in the stack or even outside the network stack; and

(f) Using the semantic web with specially developed ontologies or RDF/OWL vocabularies [RDF] for UCLP and instrument web service discovery.

These concepts and assumptions are explained in more detail as follows:

1.6.1 No pre-defined network architecture

One of the fundamental concepts with UCLP is that there is no need to define, a priori, a network architecture. Instead the user (which may be a network administrator or in some cases an actual end user) may create a network of their choice including topology, routing, network protocols and so forth. One user may wish to create an IP over optical switched network, while another may wish to build an IPv6 or for that matter an ATM network.

It is important to note that the resulting network is not necessarily a fixed static architecture. The user can change the network architecture at any time they so wish to adopt to new applications or requirements.

The first question that springs to mind with such an approach is how to assure end-to-end universality, which is one of the defining hallmarks of the Internet today. But since one of the fundamental drivers of UCLP is the assumption that the customer will control and/or possibly own the network, and therefore it is in the customer’s own interest to deploy an architecture that provides the greatest end-to-end universality and peers with similar customer owned networks which are similarly motivated. This is in stark contrast with today’s network environment where service providers are trying to build walled gardens, and want to limit universality as it is seen as a competitive threat.

Peering between customers may exist at several different levels, and again the assumption here is that is the user or customer who chooses with whom and in what way, they would like to peer with like minded networks. In some case user controlled networks may peer at the optical plane, in other cases they may peer at the IP layer, and in other cases they may peer only at the application layer as for example with VoIP or port 80 peering. Again the important point to stress that is the user or owner of the

9

network, not a nameless, faceless network engineer who makes the decision of how the networks should interconnect and peer.

Of course, in many cases, there will be many users who will want to keep their network closed and private and have no interest in peering with others at any plane. Again this should be a user’s choice.

Similar to GENI, this adaptability of having no predefined network architecture allows users and researchers to explore new network architectures and concepts. For example one area of interesting research would be to investigate other possible addressing and naming mechanisms as for example using URIs (Uniform Resources Indicators) for end host names and addresses and more traditional IPv4/IPv6 for network addressing. In essence this would eliminate the need for DNS as the actual host name is in the edge forwarding table. Of course there are all sorts of issues with regard to performance and speed as well as memory with effectively unbounded size limits on addresses. This type of research would not be possible on a conventional R&E network.

1.6.2 Exposing web services

In terms of the first concept of “exposing” or “atomizing” web services is an assumption that web services will increasingly be less of a “shell” or “wrapper” around a legacy application and be replaced by small native web service applications running on a mini-web server or chip set equivalent in functionality to an Axis/Apache/Linux kernel proxy. Many manufacturers of instruments, software processes, network equipment are already adding these capabilities with the new single chip or single card processors.

Exposing services is also one of the key underlying principles of UCLP. UCLP pioneered the concept of rather than building one common generic IP network service for many users, one instead could deploy many parallel networks for individual groups of users. Every lightpath and every cross connect is a service and the concatenation or subtending of these services is also represented as a service. Each of these individual networks on a common substrate has their own separate routing topology and discovery mechanisms. This architecture then makes it possible to extend some of these networks into new areas, which would be impractical with a general purpose IP network, such as deep into the application because every routing node may be in fact a software abstraction rather than a physical device.

1.6.3 Extending the network into the application

”Networks” today are often thought of as physical entities that route or forward packets between “physical” interfaces on computers, routers or switches. By exposing individual application and software processes there is no reason why the network itself can be extended beyond the physical realm deep into the computer or application itself and route packets not only between hardware interfaces, but software interfaces as well.

10

In fact, this was how the original routers in the Internet were originally constructed. The move to specialized high performance routers was required over time because of the sustained high volume packets these devices were required to forward and the increased complexity of routing over the global general purpose Internet network. However by moving from a common IP network with high performance routers to many smaller parallel customer controlled (or owned) or application specific networks or APNs (i.e. routable and reconfigurable VPNs), may enable some these networks to be extended past the physical interface of a computer and be connected directly to the application.

Currently routing of packets between software processes is largely done through overlay networks or through virtual interfaces e.g. XML routing, peer to peer networking etc. However by extending the physical network into the software might provide for a more global, seamless and portable applications. URI (Uniform Resources Indicators) addressing could be used rather than IPv4 or IPv6 with ports and sockets. This is described in more detail in Section 4.3

This concept again is similar to the proposed GENI Virtualization architecture to have different time slices on a host linked to different virtual networks.

1.6.4 Web services workflow as a universal control plane

This leads us to the third concept of using web services and web services work flow as a common open standard for the control plane, not only of the network but the equipment, routers, switches, time slices and sensors as well. The attraction of using web services as opposed to a myriad number of other control plane mechanisms like GMPLS, ASON, O-UNI, etc is that web services can not only interact with each of these control planes through a wrapper, but also communicate directly with other control plane web services that may or may not be network related.

1.6.5 Network service stack instead of network layers

The above concepts can be further extended by thinking of the network itself as a set of exposed web services rather than layers, in such a way that any network service can communicate with any other service, network or not. This has the potential to provide a lot more flexibility into the network layer and also allow the many software concepts of object oriented design, inheritance, etc to be applied to the network in what is sometimes referred to Object Oriented Networks (OON).

1.6.6 Using the Semantic web as a discovery tool

To date most network architectures and instrument systems have had a very fixed and defined device and topology discovery protocol generally using a very simple syntactic structure for query and expression. The semantic web with carefully crafted vocabularies and/or ontologies may provide a greater degree of flexibility and allow the intersection of different ontologies for different instrument types and network connectivity.

11

For example an end user is not going to know, or care, about a syntactic structure of end to end network connection made up of AS numbers, IP address and port/slot/channel assignments. Rather they will want to say (most likely use a graphical orchestration tool) "I want a lightpath from my bio-informatics application on my computer at University X to your visualization engine on a computer University Y" The end user does not want to be bothered by complex hierarchical naming structures.

In this roadmap we proposed that he UCLP or instrument WSDL query a well known service for a UCLP semantic web server. The semantic web server would then do a distributed search with an ontology cross reference that matched these input requirements. Some third party such as graduate student or another researcher may have already created such a specific end to end lightpath with that name and has offered it up for lease to other parties which would be discovered by the semantic web.

If there is no existing lightpath exists then the UCLP semantic web might resolve "University X” to "AS 506" and "University Y" to "AS 403". In turn the semantic web would continue to resolve these highest level hierarchical names to lower names until it can establish a sequence of connection interfaces and cross connects that would form an end to end lightpath. 1.7 Layout of this document

This document in Section 2.0 first provides an update and background on UCLP in its current configuration. Understanding the drivers for the development of UCLP will hopefully provide the context as to its adaptability to instrument and sensor networks.

Section 3.0 describes new features for UCLP that are required to complete its functionality for use in networks, but more importantly will have applicability in the new emerging sensor and instrumentation grids.

Section 4.0 lays out the architecture vision for the UCLP road map followed by specific example scenario in Section 5.0 on how UCLP would work for interconnecting sensor and instrument to a network.

12

2.0 UCLP status to date

In September 2003 CANARIE issued a call for proposals under the Directed Research program to invite university researchers and business to develop a proof of concept for User Control of Lightpaths over the CA*net 4 network. Three teams subsequently undertook to develop the first implementations of the UCLP. Details on these developments can be found at the CA*net 4 UCLP web [UCLP].

Each team undertook an entirely different design approach to the problem. This has proved quite useful in identifying and comparing strengths and weaknesses in the different implementations as well as providing useful directions for this roadmap document.

CANARIE staff has deployed and are testing the various UCLP implementations. This UCLP roadmap document is an outcome of these ongoing tests and end user evaluations. As well new developments in Web services and grid technology, particularly the new evolving Web Service Work Flow and Orchestration have provided insight to possible new features for the next version of the UCLP software.

2.1 UCLP description

It is important to review the original objectives of the UCLP program and as they are still critical to the future evolution of the concept of user control not only of networks, but also of instruments and software process. This will be fully elaborated further in this document.

There still remains considerable misunderstanding about the purpose of UCLP. Some people think it is an alternative optical routing/switching protocol to GMPLS or ASON, while others think that it is an “on-demand” inter-domain or end-to-end VPN protocol. Some people also think it is a bandwidth scheduling and reservation mechanism. It is none of these. UCLP can very simply be thought of as a configuration and partition manager that exposes each lightpath in a physical network and each network element associated with a lightpath as an “object” or “service” that can be put under the control of different network users to create their own logical IP network topologies. Related concepts have been developed elsewhere such as the ITU standards Y.1312 [ITU_Y.1312] and Y.1313 [ITU_Y.1313], and the IETF MEGACO [MEGACO] and General Switch Management Protocols (GSMP) [GSMP].

The primary purpose of UCLP is not yet another optical switching-routing protocol, or to create on-demand, inter-domain and/or end-to-end VPNs. Nor is it intended for the scheduling and reservation of bandwidth. However, as a secondary benefit, the many implementations of UCLP can be used for all these purposes.

The major difference between other proposed partition manager protocols and UCLP is the object oriented recursive architecture of UCLP. Rather than one layer of partitioning UCLP assumes that owners of switches and lightpaths will create “root” lightpath (or

13

APN) objects. Once a root lightpath (or APN) object is created, the root can independently create “child” lightpath objects. Each one of these child lightpath objects can have its own independent AAA, routing, topology and end to end lightpath services. In fact it is quite conceivable for the child APNs to use GMPS or other routing protocols across their constituent components.

A key assumption when using UCLP is that all the resultant APNs will mostly be used to support the interconnection of “long lived” IP networks. This greatly simplifies the need for inter layer communication and puts the onus of link failure detection and routing on the owner of the APN as opposed to the partition manager.

Given that no commercial switch or router supports partition management across multiple domains or this recursive feature of lightpath objects, UCLP was developed as a “proxy” server in front of existing switches and provide this virtual capability of lightpath objects in terms of partitioning both the switches and the associated lightpaths.

Grid technology using web services (Globus Toolkit 3.0) as well as Jini/Javaspaces were used as implementation strategies to provide this proxy service for the current UCLP proxy solutions. The advantage of using such tools is that they incorporate proven solutions for security and authentication as well as advertising and discovery of services across independent managed domains.

2.2 UCLP objectives

UCLP was intended to address a number of unique and novel business and network models that are only now starting to appear in the marketplace. These new models, for the most part are starting in the same community where the Internet was first developed – universities and research centers. Briefly they are:

(a) Customer owned and managed IP networks;

(b) Discipline or application specific fine grained IP networks; and

(c) User controlled traffic engineering of IP networks

These objectives are described in more detail as follows.

2.2.1 Customer owned and managed networks

Many large enterprises such as universities, regional networks, government departments and commercial organizations are acquiring their own dark fiber and unprotected, point-to-point wavelengths for their wide area networks. Ideally these institutions want to integrate these wavelengths and dark fiber from different suppliers within their own network management domain. They want be able to do their own topology and discovery and protection and restoral as well as offer value added services such as VPNs (layer 1 through layer 3) as well as APNs to their downstream users. This is not possible if the

14

organization purchases current VPN services, optical or otherwise, from today’s carriers. Current VPN architectures are only single domain, do not allow cross connection between VPNs within a domain, and, as yet do not allow VPNs to cross connected to VPNs in other domains.

The UCLP architecture allows a carrier and/or condominium wavelength provider to partition their optical switches and wavelengths to enable different organizations to manage their own partitions independently and link those partitions to partitions in other independently managed networks so that the a comprehensive wide area network solution can be operated by the enterprise. The enterprise can then run its own routing and topology discovery protocols in these partitions and incorporate them within their existing LAN or WAN network.

Figure 2.0 National LambdaRail Condominium Wavelength Network

A good example of such a model is the National LambdaRail [NLR] network in the United States. This is a classic condominium wavelength network where different organizations own and operate different wavelengths for different purposes. Since each wavelength owner may operate a separate network, and may wish to integrate that lambda with external network services it is not practical to have a central management plane for topology discovery and routing that is typical of most traditional DWDM carrier networks. This is illustrated in Figure 2.0 where one of the condominium partners CAVEwave [CAVEwave], which has acquired one of the lambdas that is part of NLR and wants to do its own add/drop, wavelength routing and partitioning on that specific wavelength independent of the other wavelength owners. It also at some point in the future wish to acquire an additional wavelength outside of the NLR and yet

15

CAVEwave acquires a separate wavelength between Seattle and Chicago and wants to manage it as part of its network including add/drop, routing, partition etc

NLR Condominium lambda network

OriginalCAVEwave

CAVEwave acquires a separate wavelength between Seattle and Chicago and wants to manage it as part of its network including add/drop, routing, partition etc

NLR Condominium lambda network

OriginalCAVEwave

interconnect that new wavelength and use it as an integral part of network with its existing NLR wavelength. UCLP was developed specifically to allow for this integration of wavelengths from other suppliers in different management domains.

2.2.2 Specialized Internet networks for high end applications and/or disciplines

Today’s Internet networks (both research and commercial) are generic in design and optimized to serve the average traffic flows and requirements of many thousands of users with relatively small flows. Increasingly however there are small communities of users who have extreme traffic volume per flow requirements, that in many cases dwarf the traffic volumes of regular IP networks. These traffic flows may originate from the newly emerging grids of high performance computers, but are also likely to originate with specialized instruments and sensors. A good example is the collection of data from radio dishes involved in Very Long Baseline Interferometers (VBLI). Another example is the data streams that originate at high energy physics detectors located in various facilities around the world including TRIUMF, Fermilab, CERN etc. Additional examples include acquiring the data streams directly from the charge coupled devices used in astronomy and synchrotron detectors.

Figure 2.1 Traditional Hierarchical Internet networks

Figure 2.x Discipline Specific Internet networks

Because of the size these data flow volumes it makes sense to enable the physical network topology to be more dynamic and responsive to their specific application needs. However, rather than having a network engineer undertaking traffic engineering decisions, as is done today, these can be done by the end users or by their community with UCLP, in effect, creating discipline specific IP networks as illustrated in Figure 2.x

16

CommodityInternet

Bio-informaticsNetwork

University

University

University

CERN

University

University

High EnergyPhysics Network

eVLBINetwork

Dept

Research Network

See next slide for more details

Lab

Common DWDM substrate

CommodityInternet

Bio-informaticsNetwork

University

University

University

CERN

University

University

High EnergyPhysics Network

eVLBINetwork

Dept

Research Network

See next slide for more details

Lab

Common DWDM substrate

Another advantage of such specialized IP networks is that they can be designed to bypass campus firewalls and LAN networks through direct fiber or campus wavelength connections as illustrated in Figure 2.y.

Figure 2.y Direct Campus Connections

This obviates the need for the campus network manager to make a large investment in high end routers, switches and firewall to provide a solution which meets the needs of only a very small subset of their client base who need such capabilities. Campus or corporate network security policies can still be applied to discipline or application specific networks especially with the move away from centralized security systems using firewalls to more compartmentalized security systems based on web services standards.

2.2.3 User Controlled Traffic Engineering

A related concept is the enabling of network managers to dynamically undertake their own traffic engineering, based on source-destination IP address block pairs as well as AS paths. Already there are a number of commercial products that look at BGP routing and allow the network manager at the edge, to “pref” the best BGP path based on throughput and other metrics, but this approach can affect the routing of all traffic going through a particular BGP peering. The UCLP approach allows an end user to “create” a new BGP path rather than selecting the best BGP path that may be advertised by setting up direct optical links between managed partitions on UCLP enabled network.

In addition with a plethora of lightpaths such as with the National Lambda LambdaRail, it is possible for regional networks and institutes to establish direct peering with each

University Hospital

PhysicsDepartment

RadioTelescope

Firewall

Main campusNetwork Borde

rRouter

University

Internet

HealthNetwork

GlobalPhysicsNetwork

GlobaleVLBINetwork

NREN

Campus DWDM

17

other rather than exchanging traffic through a hierarchy of IP networks as illustrated in Figure 2.1.

Figure 2.1 Traditional Hierarchical Internet networks

With UCLP and a national DWDM network like CA*net 4 or NLR a much richer peering mesh is possible between regional networks, but also between institutions, and in some cases individual departments as shown in Figure 2.2. In this example a number of UCLP lightpaths (or APNs) have been created over a common DWDM network to enable the direct peering. Some of the lightpaths are across the national DWDM network, but others may be implemented through direct physical connection of the regional networks themselves.

The concept of inheritance in the creation of child lightpaths out of parent lightpaths is also illustrated. In this example, the four optical regional advanced networks (ORANs) could each own a wavelength across the national DWDM network such as the NLR. The ORANs could use these wavelengths to create their own national IP networks, directly peer with each other or international partners, or interconnect to national IP network for international peering. Some of the ORANs may partition their wavelength into smaller lightpaths and through UCLP assign control and management of that lightpaths to a downstream client such as university. In turn the university may also further partition a lightpath such that an individual department or server may have a direct lightpath connection. Alternatively a researcher or department could acquire from the regional network or the national DWDM network a direct wavelength or lightpath. Any number of interconnections and peerings are possible.

2.3 UCLP discovery and advertisement

University

Regional

National or Pan-Nationl IP Network

Other national networks

NREN A NREN B NREN C NREN D

University

RegionalRegional

National or Pan-Nationl IP Network

Other national networks

NREN A NREN B NREN C NREN D

18

In order for end users to discover networks and or devices it was proposed that UCLP use the BGP AS path information as a Uniform Resource Indicator (URI) to a well known UCLP (or AAA) server. Upon finding an end to end path the BGP routing information at each end of the link would be updated to indicate the new path. This process is referred to as Optical BGP (OBGP). As well it was proposed that an existing end to end lightpath which been previously established across multiple partitions and was subsequently not being used by the partition owner could be advertised through well known resource advertisement and discovery tools such as UDDI or WSIL.

Figure 2.2 Richer mesh of direct peering enabled with UCLP

In the case of OBGP, the end user’s application would invoke an agent that would query the local border router to get the BGP AS path information to the destination IP address. The agent would then query well known URIs, as for example http://ASxxx.net/UCLP-server to see if a partition could be created to provide a lightpath across that domain to the next associated AS in the BGP path.

If a path could be successfully set up then the OBGP agent at each end of the connection would insert a new static route in the forwarding table, so that data packets with the associated destination address would now be redirected along this new route. Because the connection is layer 1 physical connection between interfaces in different domains ARP proxy services may be required, or prior agreement on sharing the same address space.

An Internet RFC draft [OPTICAL_BGP] was also written which outlined a process of how lightpath information could be carried in BGP updates. This would allow as user to

19

World

UniversityRegional

Server

World World

National DWDMNetwork

ORAN A ORAN BORAN C ORAN D

ChildLightpaths

Child Lightpaths

World

UniversityRegionalRegional

Server

World World

National DWDMNetwork

ORAN A ORAN BORAN C ORAN D

ChildLightpaths

Child Lightpaths

immediately know which AS system provided lightpath capability. The user would still have to query an OBGP server associated with a given AS as to the availability and policy of the lightpaths. However to date this version of OBGP has not been deployed as it would require router manufacturers to implement it in their routing code.

2.4 UCLP design assumptions and criteria

There is often considerable confusion as to whether UCLP is a competing optical network protocol to GMPLS, ASON or other inter-domain optical switching reservation bandwidth protocols. UCLP can be thought of a technology that operates on the layer below these protocols or any other protocol that operate on the network topology discovery and routing layer.

Although UCLP can be used for reservation and scheduling of bandwidth and/or lightpaths, that is not its primary intended purpose. In fact, it is still an open question as to whether reservation and scheduling of bandwidth or lightpaths will ever be required in networks. Any application that is “bursty” and requires a lot of bandwidth for a short period of time is probably best served with a high bandwidth routed network. High bandwidth IP networks are designed for statistical multiplexing such types of flows. UCLP, on the other hand is intended to be a user controlled traffic engineering protocol that allows users such as ISPs, enterprises and, in some cases, individual users to setup direct high throughput, or high QoS, long duration IP connections.

As mentioned previously UCLP is akin to a partition manager, but with much more functionality and extensibility. Once network elements and links have been partitioned, the respective owners of those partitions may elect to implement their choice of network topology discovery and routing protocol. So rather than one topology discovery and routing plane there may be several parallel topology discovery and routing planes each with independent topologies and routing protocols.

As such, with UCLP there is no primary requirement in terms of traditional optical/IP routing to support routing, topology discovery, reservation, scheduling, authorization, accounting, or other related services. However, variations of these services are largely implemented in the current versions of UCLP not so much to support traditional optical/IP routing but to enable partition management applications. It is expected that most of the optical/IP network functionality will be provided by higher layer services operating above the partition layer. Although the services are similar it is important to distinguish between the respective objectives of these services in terms of an optical/IP routing network using GMPLS or ASON versus a partition management service:

(a) UCLP does not require a central control plane for the establishment of end to end lightpaths (although a discovery service is required to find the switches that offer partition services and creation of “root” objects);

20

(b) UCLP primary purpose is NOT to offer reservation or scheduling mechanisms of lightpaths, but owners of partitions or root lightpath object may offer this service to users of any child lightpaths that are inherited from the parent or root lightpath;

(c) UCLP does not require millisecond provisioning as lightpath root objects or partitions are assumed to be provisioned on a long term basis

(d) UCLP does not necessarily require node or link failure error messages to be redirected to partition owner or the root lightpath object owner (as that owner may in fact reside in another independent network domain and like the Internet itself restoral and protection is done at layer 3 independent of failure on the underlying transport); and

(e) UCLP does not require crank-back or fail over mechanisms to be provided in the event of a link or node failure as this is a service that can be implemented above the partition or root lightpath object.

It should be noted that some lightpath owners may want to operate a routable VPN (APN) where the lightpaths, rather than the packets are re-routed in the event of a link failure. In that case link status information needs to be delivered to the UCLP partition owner in order that the cross connects in that partition can re-route the lightpath.

However in most cases it is assumed that a partition owner (as opposed to the switch and facilities operator) who will run and maintain a set of layer 3 routing, topology and fail-over daemons or protocols to provide such services between the partitions on different switches which are under the control or ownership of that specific partition owner. These routing daemons may be a logical router or a blade router that is integrated with the switch. It is the subsequent partition owner’s layer 3 routing protocol that determines link failures, rather than depending on the underlying switch or optical fabric to report outages.

It is important to note that a user may have ownership of partitions on switches that are in different physical network domains. As such the user must run their own keep alive or hello messages between each link to detect link or node failure, as well as provide restoral and protection services if required. The underlying physical network providers will have no visibility or awareness of the other networks and cannot provide such a capability.

Of course, in an optical/SONET network the routing information can not be done in band and as such the partition owner must use an out of band channel, such as the Internet itself or a common channel provided by the switch owners for this purpose. The later arrangement is the most common in optical/SONET networks and so it is incumbent on the switch owner to provide accessibility to the switch partition through the out of band control channel to the partition owner. This can be easily achieved through industry standard security techniques.

21

UCLP is designed with the assumption that the creation and assignment of partitions of both network elements and lightpaths will be for long duration, in the order of hours and days if not weeks. UCLP was never intended to be a fast switching algorithm for optical circuits. Fast optical switching is required if maximum utilization of limited network resources is required. It is expected that the demand for fast switching, or scheduling and reservation, of optical circuits will become obsolete as the cost of wavelengths continues to drop. Once wavelengths start following the same price curve as semi-conductors users will waste bandwidth rather than incurring higher operational costs on optimizing their utilization through fast switching, or scheduling and reservation mechanisms.

Finally it should be pointed out that UCLP can operate on a physical switch or a virtual abstraction of a physical switch. It is not necessary that a network operator expose and partition all their network switches and links within their network. Instead they can offer a “virtual” UCLP service through a well known interface such as O-UNI. To the end user the network cloud appears as single switch entity, but in fact, the “cross connects” across the virtual switch may in fact be facilitated through the creation of edge to edge VPNs across a network cloud. In reality, this is likely to be the most common way for network operators to provide “intra-domain” UCLP services.

22

3.0 UCLPv2 additional features

As mentioned previously not all the features described in the original UCLP architecture have been implemented in the current versions largely for cost and complexity reasons. The first versions of UCLP were developed as a proof of concept to provide the minimal functionality required in a research network like CA*net 4. These additional features representing new services that need to be added to the UCLP software to make it more scaleable and extensible, particularly in new application areas beyond networks. These new features are described briefly as follows:

(a) Support for providing layer 3 and/or layer 2 routing and links , topology discovery, and link status daemons in each partition through the use of virtual routers, link failure detection, etc;

(b) Graphical tools to allow partition owners to link, create end to end lightpaths and manage their partitions;

(c) Resolvers to map IP addresses to port/slot/channel numbers on the switches such as Semantic Web tools, as most applications use IP addresses for end points and interfaces;

(d) UCLP extensions for LAN campus networks;

(e) UCLP extensions to support interconnection of virtual and/or logical routers and switches;

(f) Exposure of all lightpath, cross connect and lightpath creation services as WSDL web services; and

(g) Semantic web ontology for describing UCLP services to be used by advertising and discovery service and for describing WSDL web service port types across heterogeneous network types from IP to optical networks

As described later in this document it is believed that web services using work flow tools such as BPEL can enable most of these enhancements without major modification to the existing UCLP software.

3.1 Adaptation of UCLP to sensor and instrumentation facilities

Early on in the UCLP design process it was recognized that the challenges and problems that UCLP was supposed to address in allowing partitioning and providing user control of an optical network could equally apply in providing users the ability in the management and in the connection of data instruments to the network.

23

One of the primary applications for UCLP was to allow the deployment of application or discipline specific IP networks for specialized high volume uses where instruments that produce large volumes of data need to be connected to computers, grids and/or databases in far away locations. It would make sense therefore not to limit user control to just the network elements but also extend this capability to specific network instruments at one end, and software process at the other end to allow researcher and other users to completely control the entire research equipment connectivity.

The grid community is actively addressing the ability for remote user control of software processes through the application of web services to control software process on distributed independent managed on high performance computing facilities. But little has been done to extend this same concept to the other end of the network where the instruments and facilities would be located.

This document proposes to lay out an architecture that will allow UCLP to be adapted to instrument and sensor networks as well as addressing some of the newly required features that have been identified for UCLP to support greater functionality for use in CA*net 4 and similar infrastructures.

3.2 Adaptation of UCLP to IP Routed Networks

Many modern day routers support sophisticated IP flow structures with advanced queue management. It is proposed in this UCLP Roadmap that UCLP be modified to provide end users to modify certain IP flow management properties through a web services interface and in some cases IP flows be mapped to optical lightpaths and vice versa.

The GENI Virtualization program has largely focused on this aspect of the problem. Researchers like Jon Turner et al have done considerable in depth work in this area in the architecture of special routers that can handle the forwarding of packets for the various virtual networks. [TURNER]

In UCLPv2 we do not propose to duplicate this effort in creating the special router technology to allow the identification and separation of flows through a router. Instead it is proposed that web services be wrapped around existing tools for creating logical or virtual routers on a larger parent router. The web service tools will allow the owner of the substrate or parent router to create a virtual or logical router and assign one or more interfaces to that router. The interfaces may be SONET STS channels, MPLS tunnels, or GMPLS tunnels. The owner can also assign to a given APN manager and create proper access control tools to the virtual or logical router.

The designated APN manager then has access to a limited set of router commands applicable to that virtual or logical router. The range of commands and tools will be largely dependent on the specific of the manufacture and type of router.

Once the virtual router is represented as a web service the APN manager has some limited to do configuration and topology management – but this will be far more limited

24

than what is possible on a switch which allows for full inheritance and many sub child arrangements.

25

4.0 UCLPv2

To address the requirements for these new features in the UCLP software as described in Section 3.0, including extensibility for the integration of sensor and instrumentation grids a new high level functionality for UCLP will be described. This will serve as framework for other investigation into detailed implementation issues and identification of appropriate architectures and tools to support the implementation.

The driving technology that will enable this new functionality is the emerging grid/web services technology called Service Oriented Architectures (SOA) built on integrating web services using the emerging web services work flow and/or orchestration tools and platforms for the creation of end to end solutions and the use of the Semantic Web and ontologies for describing and finding resources.

4.1 Definition of a user

To fully describe the proposed architecture for the UCLP roadmap a common understanding and meaning must be given to who and what is expected of a “user”. An important philosophical underpinning to the UCLP roadmap is that, as much as possible, all services should be exposed to an end-user and that the end-user has direct control of the various web services and work flow orchestrations. Clearly only some users will be interested in this type of flexibility and control, while to others it represents unnecessary complexity. So to accommodate both requirements we must define different types of users as follows:

(a) An “End-user” in many cases will be a researcher or a non-technical user who will have little interest in the concepts of web services and web services work flow and simply wants a working solution or a single abstraction of all the underlying services. An “End-user” does not necessarily need to be a human being. It can also be another web services orchestration which incorporates the current orchestration into another web services abstraction.

(b) A “Super-user” will generally manipulate or aggregate published services to produce partial or complete end-to- end solutions for some end-user. Some of these services may in fact be offered as new web services or orchestrations to other super users who will incorporate them within their own work flows or orchestration. Super users may work closely with end users as technical support staff or they may also be third parties such as Virtual Organizations (VOs) who specialize in aggregating complex service orchestrations across multiple administrative domains into new single web service entity.

(c) An “Administrative-User” is a special type of “super-user” who has ownership or management control of a given domain and its resources. An administrative- user is a super user within an administrative domain and is responsible for aggregating or orchestrating internal services and/or creating “child” services or work flows that will be advertised as a web service to an external user. An

26

Administrative-user can be an optical network and/or equipment facilities manager or operator. For example an administrative user may aggregate a AAA service with an instrument device and/or internal optical network link to produce a single service instance for secure access to a specific instrument or network device.

In this document the generic form of “user” applies to all 3 definitions and does not necessarily imply specifically an end-user or super-user.

4.2 Service Granularity and exposing of services

Although all the original UCLP implementations support the use of Open Grid Services Architecture (web services) to varying degrees there was no consistent implementation of such services. As well most of the UCLP implementations developed their own internal database architecture for keeping track of the lightpath objects and their state. A large part of this was driven by the technical nature of SONET/SDH multiplex hierarchy where centralized records had to be maintained in order keep track of the creation of child lightpath objects and their connection status.

In this UCLP roadmap it is proposed that a new architecture framework be established that as much as practically possible will push web service constructs down to the smallest network or instrument element and use Web services workflow or service orchestration to bind or concatenate these “services’ together to build a complete network-instrument sensor solution.

In the current UCLP implementations only the lightpath creation process is exposed as a web service and each lightpath segment is a simple database entry describing its end points, lightpath ID, owner ID, bandwidth and other relevant parameters.

In the UCLP roadmap It is proposed that each lightpath and cross connect be represented as a WSDL web service and that concatenation of lightpaths be done with a web services orchestration tool which creates a new lightpath WSDL web service. Conversely each partitioning of a lightpath and/or cross connect will result in the spawning of a new lightpath service representing the child lightpath and/or cross connect. Probably the easiest way of achieving this is to allow many instances of the current implementations of UCLP (wrapped in web services) to operate on the same proxy platform.

More importantly rather than building a priori a hierarchical set of services or abstractions that are permanently linked together, this new framework approach permits a hierarchical set of services to be defined by super-users as required. A simple example is to provide secure access to a UCLP web service for either a single lightpath or a network cloud. Rather than definitely binding a UCLP web service to a security interface such as a AAA server, they can be linked together a service orchestration by a super-user and exposed a single web service to other users – end-users or other super-users. The advantage of this approach is that the AAA service can be orchestrated as single service

27

entry point to a domain of switches or devices (i.e. network cloud) or linked to individual lightpaths and devices as required. Or, if the switch or device can be partitioned into child services it may be desirable to bind the AAA service to only specific child services or partitions.

A good model of this more generic solution is to look at WSFL and work flow tools for a variety of grid eScience applications such as the Taverna open source workflow project Taverna [TAVERNA], the Feefluo [FREEFLUO] project, Keppler [KEPPLER] or Active End-Points [ACTIVE]. The Taverna and Active-Endpoint systems are graphic tools that allow users to create web service work flows which constituent parts are various web services.

Another good model of a workflow Graphics User Interface (GUI) is LabView [LABVIEW] which is a popular tool for virtualzing instruments and creating workflows by connecting instruments together through a GUI.

Graphical workflow tools are analogous to “real time” Unified Modeling Language (UML) tools where software architects can create complex linkages and inter-relationships between many different software sub-routines and modules.

Web services are much more loosely coupled than traditional software modules, and through SOAP and HTTP, they now have a common interface language. Both LabView and the new UMLv2 [UMLv2] standard support web service interfaces.

This abstraction of a set of services into a workflow web service can allow “super-users” to create innovative, custom, end-to-end workflow solutions that in turn can be advertised to a user community, who may or may not be interested in creating these workflow solutions themselves.

4.3 Moving from network “layers” to network “services”

Current network architecture perspectives are largely proscribed by the network layer stack model where each network layer only communicates with layer above and below it and is largely hidden from the end user.

However another view is that network is less of an layered set of services, but more akin to a software service, conceptually like the Unix “pipe (|) ” command. The Unix pipe command is a utility that links the output of one software program to the input of another. This concept can now be applied to network services where the network is a software utility that allows data to piped from one application to another.

Exposing all network links as individual web services would enable this concept of Unix piping in the wider area not only between end points but between traditional network layers. By viewing everything as a service from the traditional application layer to the physical layer new relationships and linkages can be established between applications, users and networks.

28

For example, traditional routing protocols were designed to forward packets between physical network connected devices. But by abstracting the network into services, the “network” can now be extended past the physical interface into the middle of a computer to a “soft” interface on a software process or web service as shown in Figure 4.0. GENI has a similar concept where individual time slices on a host are connected to different virtual networks.

Daemons for layer 3 routing protocols like OSPF, BGP, etc can also be exposed as services and be linked to the software process with web service workflow orchestration tools. So routing of packets can now occur, not only between devices but between processes (time slices) and/or between processes and network devices. Within a single “physical” computer software processes or time slices could be forwarding packets to different interfaces and/or other internal processes at the same time. Moreover different processes or services within a single machine could be integrated into different network APNs.

Figure 4.0 Extending the network into the application

Our whole concept of a network now completely changes. Rather than being a physical infrastructure layer that only connects physical devices and lies below the application layer, it can now be a seamless part of the entire distributed application. The “network” can now be deeply embedded into the application and provide a more open and transparent method for forwarding packets from one application to another.

Figure 4.0 illustrates to separate APNs or virtual networks with their own separate virtual routers interconnecting to separate processes or time slices on the host computer. In addition on the host computer, in this example, there is a common routing daemon which allows packet forwarding between the two processes. Again all the elements of these

Time slice or software process

Web service

Time slice orSoftware process

Web service

Routing daemon

Web service

User A

User B

Sin

gle

Com

pute

r or W

S

inst

ance

of

an

orch

estr

atio

n Interface Card or portWith URI addressing

VPN extends into computer to specific processes

DWDMNetwork

http://10.0.0.1

yyyy:410:0:1

zzzz:410:0:1

Layer 3 WSVirtual Routershttp://

10.0.0.2

http://10.0.0.3

http://10.0.0.4http://10.0.0.5

http://10.0.0.6

29

APNs are exposed as web services and therefore the APN manager can manipulate or change the architecture and topology any time they wish.

Of course, if software processes and/or embedded web services can communicate via a routing protocol with other devices and/or processes the demand for address space will increase significantly. Rather than using URIs for handles and ports, they can also be used to identify interfaces and neighbors for these embedded software networked devices. A physical interface on a computer may advertise itself not only to physical neighbors such as routers and switches but to internal software processes and/or interfaces. Similarly these devices would advertise their URI addresses to the physical interface for packet forwarding.

In Figure 4.0 there are two separate routable APNs. The APNs can be at layer 1 or higher layers. But in this example we assume they are operating at layer 1 with UCLP routing daemons located at each optical/SONET cross connect switch. As noted previously each VLAN can be operated independently by the APN owner with their own separate routing protocols and services. More importantly because of the granularity of the APNs it is possible to build one APN with software routing daemons that are embedded within an application as show in the diagram. This would be impractical on a normal routed IP network because of the size of the routing tables and the need for a comprehensive set of protocols for router management, etc.

Another simple example illustrates the concept of exposing each layer of the OSI network stack as a network service which can communicate with any other layer or with services outside of the stack. Today many applications send and receive data through a well known port and TCP/IP stack on their computer. The TCP/IP stack assembles the data into packets and manages the actual TCP session with a remote host. Generally there is little flexibility in this architecture. If the application needs to do a large file transfer over long distances this type of “default” arrangement may not be adequate. The current solution today in these situations is to custom assemble special applications protocols for the file transfer. In fact there is a community of researchers who are actively involved in this type of work [PFLDnet]. However, if the application was exposed as a web service then it might be able orchestrate different file transfer techniques based on the size of the file, distance, time to deliver, etc. Each file transfer technique would also be exposed as web services – one might be for best efforts transmission while another may be integrated directly with a optical/SONET channel to provide guaranteed throughput over longer distances.

4.4 Web services for control plane versus data plane

To date the primary use of web services has been to encapsulate transaction data for financial and commercial transactions. Consequently the biggest application for web services and related work flow and orchestration technologies has been for e-business applications.

30

Individual financial transactions generally have small data volumes and so the overhead cost of encapsulating transactions within a verbose web services framework has not been that onerous. However the data flows for instruments, sensors and networks can be quite significant, sometimes on occasion in the order of Gigabits per second. Clearly encapsulating this data would be impractical due to the high overhead costs. Therefore in this document it is proposed that the web services constructs and workflows or orchestration be used with the control plane to setup “pipes” and or linkages to enable the rapid flow of un-encapsulated data between services.

Traditional web services are designed for transaction based interactions, but since the establishment of pipes or linkages requires maintaining state, “state full” web services are required such as WSRF. However, another approach is to use orchestration and workflow tools such as BPEL or Keppler to maintain state. There is still an open debate as to which is the best approach. But increasingly it is seen that tools like BPEL offer true object oriented characteristics with inheritance and long lived state and as such are more suited to instrument grids, as opposed to WSRF.

Over the past decade there have been several message passing and object oriented protocols have been developed such as CORBA, DCOM, Jini/Javaspaces etc. Each protocol has had a difficult time dealing with the issues of dealing with generic failures, healing of state-full partitions, graceful degradation of guarantees in the presence of partial failures, etc.  The attraction of WSDL and workflow tools such as BPEL and Keppler is that they are not designed to replace existing protocols that handle these issues extremely well.  What WSDL and web service workflow tools enable is a common interface between existing and new processes.  WSDL and BPEL are more akin to provisioning and configuration tools to link together existing state-full processes.  The underlying processes and protocols maintain actual state of the process and handle disruptions, outages, etc. Although BPEL specifically supports state through correlation sets it is not the intent in this framework document to use such tools to maintain state in the BPEL linked processes. Hence there is no intent to replace existing routing protocols like OSPF or BGP, but instead give the user greater flexibility of WSDL and BPEL in configuring networks and software processes.

The other aspect of WSDL and BPEL that is appealing is that they are built on a much simpler paradigm of the web itself.  They allow for the very simple concept of linking text and images through embedded http pointers to be extended to processes.  And because WSDL and BPEL (which really is a specialized version of WSDL)  use URI structures there is no predefined data types and constructs that must be adhered to by developers in order for their applications to communicate with each other.  The data types and the message passing is defined through http itself using SOAP bindings.

4.5 UCLP (WSDL) Proxy for Instruments and Sensors

Most instruments and sensors used in research applications and process control have some characteristics similar to that of optical cross connect switches. They assume a

31

single management entity and are intended to connected to a communication medium through a fixed long term configuration, usually through a single LAN interface or serial port.

However increasingly researchers are recognizing it would be useful to be able to independently link or “bind” these devices to different systems such as networks, APNs and software services, particularly when these instruments may be shared by different teams of scientists at multiple institutions. Connecting the instrument to a lightpath may be one of many types of “bindings” that could be required. Other possible bindings would be to connect the instrument or sensor to an IPsec tunnel for secure transmission of data, or to another instrument in order to correlate or calibrate data flows.

Of course, today this can all be done by network managers just as easily as they can setup lightpaths across an optical network without using UCLP. But the purpose of UCLP is to allow end users to do their own configuration management and bindings without the need to signal or request permission from a network operator to provide such a service. The same principle could be applied to connection of remote instrumentation to computers and servers located at a home institution.

As part of the UCLP roadmap it is hoped that a couple of sensor or instrumentation facilities can be identified that would be willing to implement a UCLP proxy server to allow user control of the instrument as well as the ability to create daughter services and other features that are currently available with UCLP for optical switches. An open question remains is whether child services should be described as workflows (orchestrations) or state-full web services.

In order to do this, the current versions of UCLP must become more generic and not assume that the only things they are concatenating or binding are lightpaths. More importantly it would be useful to describe lightpath objects as a web service - WSDL (or perhaps a web service work flow which can also be expressed as web service) in their own right. This allow a user to link together an instrument and a software process through a UCLP “service” as is currently available today, but also to discover existing pre-configured lightpaths that may have been created by a previous invocation of UCLP.

Many of these other services already exist in one form or another and are used by specific science teams around the world. But few of them have been adapted to serve a number of different science disciplines or allow seamless integration with other services. A good example of a prototypical web service based sensor network is ROADnet [ROADNET] -. And a good example of a prototypical set of common database and network services for multiple science disciplines is the US Integrated Ocean Observing System Plan [ORION] using the Open Source Project for Network Data Access Protocol (OPeNDAP) [OPENDAP].4.6 Semantic web, WSIL and UDDI for path discovery and resolution

As in the original UCLP design one of the key “services” that must be first invoked is a path discovery and harvesting service. The path discovery and harvesting service can be

32

a web service in its own right or be a “harvesting” tool to find and locate appropriate web services that describe all the possible segments between a source and destination. Path information can consist of AS numbers, IP addresses of routing/switching devices along the path, and/or slot port/channel interface identifiers of devices along the path.

Figure 4.2 Possible graphical representation of UCLP web services

The path service can query a border router for the AS inter-domain path and/or an internal routing metric such as OSPF or ISIS to get the intra-domain path information. But the path discovery service must be also capable of discovering pre-configured partial or complete end to end lightpaths that bypass existing hop by hop connections. So a combination of UDDI, WSIL and topology discovery tools is needed. It is expected that most APNs, and/or lightpaths will be expressed as existing WSDL services (or factories). They will not created on demand.

The pre-established web services most likely will be “permission sets” to an existing UCLP service where there those who meet the required criteria will be to create a lightpath within a limited domain, or actual nailed up lightpaths between specific end-points.

Path connections may also use different sets of descriptors for bandwidth (e.g. OC3 versus 155 Mbps), connection types and channels. Intersection of ontologies or RDF vocabularies may also be needed as different disciplines may describe connection port types differently or with different relationships.

Ultimately it is essential that this information harvested from topology discovery services, UDDI and WSIL resource brokers be presented graphically with each UCLP

33

VancouverCA*net4

WinnipegCA*net4

SeattleCA*net4

ChicagoCA*net4

MontrealCA*net4

ChicagoSTAR LIGHT

New YorkMAN LAN

SeattlePwave

UCLPLightpathWS

UCLPCross ConnectWS

Third Party LightpathBidirectional -1 GbpsVancouver: Port x/Slot y/Channel zMontreal: Port x/Slot y/Channel zPartitionableAvailable until 2006 to all VancouverCA*net 4 peers

NeptuneInstrument WS

BCnet

VancouverCA*net4

WinnipegCA*net4

SeattleCA*net4

ChicagoCA*net4

MontrealCA*net4

ChicagoSTAR LIGHT

New YorkMAN LANNew YorkMAN LAN

SeattlePwaveSeattlePwave

UCLPLightpathWS

UCLPCross ConnectWS

Third Party LightpathBidirectional -1 GbpsVancouver: Port x/Slot y/Channel zMontreal: Port x/Slot y/Channel zPartitionableAvailable until 2006 to all VancouverCA*net 4 peers

NeptuneInstrument WS

BCnet

web service graphically illustrating the various connections available to other web services. This is illustrated in Figure 4.2

A graph connections tool can be built on top of any Keppler, Taverna or one of the commercial BPEL visual scripting engines. One possible scenario would be for the user to click and drag the required connections to create an end to end lightpath from the Neptune instrument to the visualization engine. With each click and drag operation the underlying web service for either a lightpath or cross connect would be queried for the availability of a service that meets the specified requirements of the lightpath.

4.7 Extending UCLP to the LAN

There are already several tools that allow end users to create their own VALNs within a campus network environment [VLAN-UBC], [VLAN-McGILL]. Some of these tools also allow users to create nested VLANs similar in concept to creating child lightpaths. It therefore is not a great leap in complexity to extend the UCLP concept to user controlled campus VLANs. All that is required is to wrap these existing services in web services and allowing multiple instantiations of such services.

Network administrators or “orchestration engineers” could provide orchestration services to users, or create orchestration on behalf of users linking a VLAN web service to an external lightpath web service. This is illustrated in Figure 4.3

Figure 4.3 Exposing Nested 802.1 p/q VLANs as web services

Campus Border Router

802.1 p/q VLANWeb ServiceLightpath Creation

Workflow Service

VLAN

End user

Standard Ethernet Links

ExternalLightpath

VLAN to LightPathCross Connect Web Service

Campus Border Router

802.1 p/q VLANWeb ServiceLightpath Creation

Workflow Service

VLAN

End user

Standard Ethernet Links

ExternalLightpath

VLAN to LightPathCross Connect Web Service

34

5.0 UCLP Next Generation Scenarios using workflow

The following scenarios will attempt to illustrate the use of web services work flow or orchestration as a tool for linking remote instruments to users or computers over a user controlled optical network and create a private application specific IP network. The details presented are examples of how the underlying web services may be scripted, but to the actual user it is expected that the primary interface will be through a specialized GUI modeled on Taverna or Keppler such that the user can click and drag to connect the various services together.

5.1 Creating a simple APN linking instrument to a visualization engine

This scenario is made up of several components to illustrate aspects of the proposed roadmap for UCLP as illustrated in Figure 5.0. The scenario will demonstrate the connection of a remote undersea instrument such like that would be used in the Neptune project to a visualization engine located many thousands of kilometers away across four independently managed networks – BCnet, CA*net 4, CAVEwave ( using a partitioned wavelengths on NLR) and OMNInet. It is assumed in this scenario that the instrument will produce extremely large data volumes and therefore will require dedicated lightpaths. However, the same scenario could be used for small data flows where some other end to end network service could be required such as IPsec.

Figure 5.0 Orchestration scenario to connect instrument over network

There will be 4 different types of networks used in this scenario. It is assumed the Neptune Instrument Web service (which in fact is a proxy running on a land based

35

NeptuneInstrument WS

OMNInet

Winnipeg

Calgary

Chicago

Seattle

Optiputer

CA*net 4

NLR

Neptune Lightpath

CAVEwaveLightpath

VisualizationEngine

NeptuneInstrument WS

OMNInetOMNInet

Winnipeg

Calgary

Chicago

Seattle

Optiputer

CA*net 4

NLR

Neptune Lightpath

CAVEwaveLightpath

VisualizationEngine

server) will be connected via an IP flow QoS service or MPLS tunnel to a BCnet router. The router interface supporting that flow is then connected to lightpath on the CA*net 4 network, which is then cross connected to a partitioned lightpath provide by the CAVEwave at the Pacific Wave exchange point in Seattle. The CAVEwave lightpath (which is a partitioned lambda on the NLR) is then connected to the OMNInet O-UNI network in Chicago. From there the lightpath finally terminates on a visualization engine at the Electronic Visualization Laboratory at the University of Illinois.

Each network will have its own independent authorization and policy management mechanism. In this scenario the CA*net 4 lightpath for Neptune will also orchestrate a separate policy and access mechanism that is different than that for the underlying CA*net 4 network itself.

As a first step CAVEwave and Neptune must acquire lightpaths from CA*net 4 and NLR respectively. It is assumed that NLR and CA*net 4 have UCLP implemented on their respective networks. Both NLR and CA*net 4 engineering staff would create APNs on their respective networks for the use of Neptune/BCnet and CAVEwave respectively. The researcher or Neptune “super-user” would then see a number of web services for lightpaths and cross connects that have been “harvested” and/or discovered as illustrated in Figure 4.2. The super-user would then create a complete end to end orchestration as detailed further in this document. The Neptune “super-user” described in Section 5.2.2 has the most important job of “harvesting” or “discovering” the various web services representing APNs or lightpaths in each domain and then linking these together to create a complete end-to-end solution from instrument to visualization engine.

In this scenario there will be at least four “Administrative-users” who will be have control of networks and facilities in their respective independent managed domains. The Administrative-users performing the critical role of defining which web services will be exposed and how they will be advertised to third parties. The Neptune super-user will then harvest these advertised web services to create their own orchestration.

5.2 Different users and different orchestration views

Probably the best way to understand service orchestration is to think of it being analogous to of the familiar concept of specialized web pages where web-masters or “super-users” create web pages on specific topics that link to other web pages. This makes it easy for “end-users” who have a particular interest on a given topic to find one canonical source to all related web sites for that specific topic. It is expected that similar “orchestrations” of web services will be done by super-users rather than end users by themselves. As well users of all kinds will be able to use search and discovery tools to find appropriate services that are of specific interest to them.

It is assumed that every user, whether they are an end-user or super-user has a specialized graphics orchestration work flow similar to Keppler, except that it can display geographical relationships as well as workflows. The orchestration graphics tool actually creates the orchestration XML scripts for the binding together of various selected web

36

services. Each icon on the graphics interface would represent a web service and be of sufficient detail to view its portTypes, handles and other interfaces. To create an orchestration of web services a user would highlight and drag a connection between each portType and/or handle to a matching portType and handle on a corresponding service. Figure 4.2 is an example of what a graphics representation of such an orchestration where the links show the possible UCLP connections.

WSDL web services representing lightpaths and cross connects would have some unique port types conditions that reflected the fact that they can only connect to other services where a physical connection can exist. So, in Figure 4.2 if user tries to connect a Chicago cross connect web service directly to a Vancouver lightpath an error condition would result. But a Chicago to Montreal, New York, Winnipeg or Toronto would work.

Once the user has completed an orchestration a compilation engine would generate the necessary WS scripts which would then expose the orchestration as a new web service complete with its own WDSL description of portTypes and Handles.

5.2.1 End-user orchestration view

In the scenario of an ocean observatory, an end-user such as an oceanographer may have a “Taverna” or “Keppler” like service work flow or orchestration interface where they can link together services that are of particular interest to them. It is important to note that none of these services may be related to the specific task of linking instruments to optical networks. An oceanographer may be more interested in services and service orchestration related to various oceanographic datasets and data manipulation tools such as OPeNDAP. So on the oceanographer’s orchestration web page there may be icons for various OPeNDAP web services for databases and data manipulation tools, of which one single icon may be for a web service for a Neptune undersea instrument that is providing a continuous live data feed. In fact this instrument web service may actually be a complex orchestration exposed as single abstracted web service created by some third party ”super-user” as described in Section 5.2.2.The complexity of connecting the instrument to the network to the “end-user” is hidden by the orchestration contained in the single exposed web service.

The oceanographer end user may use work flow orchestration tools to graphically paste and link the Neptune instrument end-to-end visualization web service created in this scenario to a calibration and “de-instrumentation” web service that may be available elsewhere on the global Internet. Or they may graphically connect (or spawn) the Neptune instrument end-to-end visualization web service to some other web service such as OPeNDAP. But in the scenario further described in this document we will not look at the oceanographic examples cited here, but instead explore in depth the creation of the Neptune instrument end-to-end visualization web service on the end user’s orchestration graphical interface and the connection to an optical lightpath across an optical network like CA*net 4.

5.2.2 Super-user orchestration view

37

It is assumed that some “super-user” will create this abstracted service of linking the instrument to the visualization engine as illustrated in Figure 5.1. The super user in this case could be local technical staff such as a network technician or graduate student. Alternatively it may be created by some third party which specializes in building web service orchestrations across networks or even someone at the Neptune or ORION project administrative offices.

The focus of this scenario will largely be on how the super-user creates end to end orchestrations by combining lightpath web services, cross connect web services, instrument web services, possible campus VLAN services into a new web service that is then exposed on the “end users” desktop as a single web service instance for the end-to-end instrument visualization web service, as described in the previous section.

It is assumed that the super–user also has a special graphics interface built on tools like Taverna or Keppler in order to graphically create orchestrations of various lightpath web services as described in Section 5.0 previously. On the super-user’s graphic interface there would be icons representing web services for a variety of tasks such as harvesting, discovery and geographic displaying of lightpath web services as in Figure 4.2. In addition the graphics tool may provide for discovery and harvesting of instrument services with connection port types, notification services for fault and error handling, and mid span routing services to create routable VPNs (APNs), etc

Figure 5.1 Orchestration of orchestrations for example scenario

Discovery and harvesting mechanisms could be simple tools like Google that search on key words or more specialized discovery and advertisement tools like the semantic web with a lightpath ontology or web service directories like UDDI and/or WSIL. In either

Neptune/ORIONInstrumentWS

VisualizationWS

IP FlowQoS

WS

OMNInetBandwidthReservationWS

LightpathWS

NeptuneInstrumentServicePT

Ban

dwid

thR

eser

vatio

nPT

Lig

htPa

thC

onec

tionP

T

Lig

htPa

thC

onec

tionP

T

InstrumentNetworkServicePT

Super user orchestration

1

2 3 4

5

1

2 3

4

5

End user orchestrationNeptune admin orchestration

XconnectWS

LightpathWS

XconnectWS

Neptune/ORIONInstrumentWS

VisualizationWS

IP FlowQoS

WS

OMNInetBandwidthReservationWS

LightpathWSLightpathWS

NeptuneInstrumentServicePT

Ban

dwid

thR

eser

vatio

nPT

Lig

htPa

thC

onec

tionP

T

Lig

htPa

thC

onec

tionP

T

InstrumentNetworkServicePT

Super user orchestration

1

2 3 4

5

1

2 3

4

5

End user orchestrationNeptune admin orchestration

XconnectWSXconnectWS

LightpathWSLightpathWS

XconnectWSXconnectWS

38

case it is assumed that the super-user can quickly retrieve a URI (or an icon representing a URI) for a specific instrument WSDL web service directly from the Neptune web/orchestration site or by using a variety of search tools. Through a pre-arranged certificate proxy service the user can reserve the resource by clicking on the icon. Alternatively the icon may pop up with a separate web service in order for the user to log-on and authenticate themselves with the specific resource.

Each of the administrative users in BCnet, CA*net 4, CAVEwave and OMNInet would authorize specific lightpath resources to be made available to the Neptune super-user. These resources can be allocated wavelengths or APNs with fixed end points and optical or STS channels, or they could be a group of resources which are then fixed to specifioc end points or channels upon request.

Alternatively a more hierarchical structure could be created where, for example, where a CAVEwave super-user sets up an end-to-end lightpath or APN from the Seattle Pwave across the OMNInet network to the visualization engine. It is assumed that OMNInet may have given CAVEwave privileges to setup a lightpath across OMNInet, and such a privilege may not be available to other third party users. The CAVEwave super-user would represent this APN orchestration as a single WSDL web services a shown in Figure 5.0. Similarly BCnet could arrange an end-to-end lightpath or APN between the Neptune instrument and the Seattle Pwave using pre-allocated lightpath services from CANARIE’s CA*net 4 and linking that to an “on-demand” IP flow based lightpath across the BCnet network. Again the assumption in this approach is that BCnet super-user might have authorization to CA*net 4 lightpaths or APNs which might not be available to a Neptune super-user.

Any number of permutations and combinations are possible in how lightpaths and APNs might be created. The examples given here are only a small sample of what is possible.

Finally, as illustrated in Figure 5.1 the Neptune super-user would graphically connect the various advertised web services in sequence to produce an end to end solution all the way from the network instrument web service to the destination end user’s desktop or computer system.

5.2.3 Administrative-user orchestration view

The Neptune as well as the intervening network web services administrators would also have a geographic based GUI similar to Taverna or Keppler in order to create a host of various web service orchestrations that would be exposed as WSDL web services to external users, either other super users and/or and users.

Many of their clients may not have the authorization to access internal or external network facilities or instrumentation. So the administrative user, in some cases may also be a super-user in order to create complete, or partial orchestrations for their clients. For example, these could include linking an instrument to a network stub that terminates on a local high speed network, or perhaps to a distant exchange point. Rather than offering

39

the only instrument as a web service, it may be more practical to have a partial orchestration such as this, which can then linked to other network APNs or lightpaths to enable an end-to-end orchestration.

It is unlikely that Neptune administrator would allow any external user direct control over a given instrument. Similarly no network manager will give entire control of a switch or network to an external user. Instead it is assumed that the administrative user will orchestrate a number of internal services in to a single web service instance that would be exposed to external users. For example the administrator may only want to expose only specific control functions of the instrument or network device to end users and attach a AAA service to allow only authorized users access to that specific instrument or network device. In addition the administrative-user may want to incorporate local archiving and calibration of the data from the instrument before the data is forwarded to the end user. Each one of these functions can be exposed as web services, or in some cases they may be recursive web service instances of more complex orchestrations of other services.

As noted previously many instruments will not have the intelligence to support web services natively and a proxy or “shell” will have to be built around the instrument on some shore based computer that would represent the instrument as a web service. This is the same with optical networks where in the past it was assumed only one network manager would have access to the device. The same principles of this scenario apply regardless of the location and form of the web service.

The administrative user would have a host of icons on their orchestration graphics desktop representing the various instruments and services that would be internal to the operations of Neptune or the network. The administrative user job is to basically link these internal web service together into an orchestration exposed as a single web service to an external user.

5.2.4 Administrative orchestration for Neptune or ORION

In this scenario the Neptune administrative user has linked the data output of the instrument web service (which was created for Neptune internal purposes) and linked it to a local area network service and then to a archiving/forking web service.

A simple web service instrument proxy is shown in Figure 5.3. In this example the instrument is connected locally, or more likely remotely, to a small computer running a web services platform such as the popular open source tool set Tomcat/Apache Axis/Linux. It is assumed in this example that the instrument may have an existing Java control interface, but the control interface could be as simple as a serial port or alternatively be complex facility with many control buttons and knobs. In either case, the Neptune/ORION administrator may want to pass on some of the controls, (or a subset of the functionality of a control) through to another user or application. In this example the direct instrument controls are not made available but the simple “datapathConnectionPT” port type which allows a user to select the path has been exposed.

40

Figure 5.2 ORION/Neptune Instrument Orchestration

In the scenario described in this document the Neptune/ORION administrative operator does not pass on the “dataPathConnection” port type to an end user, but instead orchestrates a direct link to the next web services process which in this is a local area network connection between the instrument web service and an archiving/fork web service as illustrated in Figure 5.2.

Figure 5.3 Simple web service instrument proxy

The web service orchestration, in essence, is the control plane mechanism for the connection of the data forwarding path between the instrument web service, LAN network service and archiving fork service. The resulting orchestration in itself becomes a new WSDL service that can be linked to other services as detailed previously in Figure 5.1. Note that instrument controls, that the administrative user does not want exposed to

41

Control Port(s)

Data Port(s)

JavaStub

Instrument

instrumentControlPT

Data Path A

Data Path B

Tomcat/Apache/LinuxServer

dataPathConnectionPT

WSDLInterface

Control Port(s)

Data Port(s)

JavaStub

Instrument

instrumentControlPT

Data Path A

Data Path B

Tomcat/Apache/LinuxServer

dataPathConnectionPT

WSDLInterface

NeptuneInstrumentServicePT

Instrument WS Proxy

LAN WS

Archive & Fork WS

instrumentControlPT

Neptune Instrument WSda

taPa

thC

onne

ctio

nPT

LA

Nne

twor

kCon

nect

ionP

T

arch

iveF

orkP

T

1

Data Flow Path1

Path A

Path BNeptuneInstrumentServicePT

Instrument WS Proxy

LAN WS

Archive & Fork WS

instrumentControlPT

Neptune Instrument WSda

taPa

thC

onne

ctio

nPT

LA

Nne

twor

kCon

nect

ionP

T

arch

iveF

orkP

T

1

Data Flow Path1

Path A

Path B

Instrument WS Proxy

LAN WS

Archive & Fork WS

instrumentControlPT

Neptune Instrument WSda

taPa

thC

onne

ctio

nPT

LA

Nne

twor

kCon

nect

ionP

T

arch

iveF

orkP

T

1

Data Flow Path1

Path A

Path B

an external user can be easily removed from the external WSDL description of the service.

Independently the administrative user may advertise additional services which an external user can optionally invoke. One example may be a AAA or MyProxy service to authenticate users and another is an OSPF daemon web service which an external user can invoke to bind to the end user’s network to permit routing of packets across private routable VPN.

The administrative user, through the graphics interface, will also define the partner link requirements, the flow process and the invocation procedure and other requirements to build the internal web service orchestration.

5.2.4.1 Orchestration of Power Consumption and Instrument Usage

One of the big challenges for cable observatories like Neptune is the management of power consumption for various instruments. Neptune has a complex high voltage distribution architecture [NEPTUNE-POWER] for providing power to the various instruments and devices connected to the network. However the Neptune administrators still have to carefully monitor the power utilization by various instruments and in some cases may only authorize certain instruments or devices to be turned on, if they are satisfied there is sufficient power available at given node.

Figure 5.4 Orchestration of power and instrumentation control

The power distribution decision process would be an ideal environment in which to implement a BPEL work flow process. As the power system is quite complex different nodes can have different power resources based on their distance from the shore station

42

Control Port(s)

Data Port(s)

JavaStub

Instrument

instrumentControlPT

Dat

a P

ath

A

Dat

a P

ath

B

Axis/Apache/LinuxServer

dataPathConnectionPT

WSDLInterface

instrumentEnablelPT

To user’s WSDL

Power Enabler WSDLInstrument WSDL

New Instrument WSDL

Control Port(s)

Data Port(s)

JavaStub

Instrument

instrumentControlPT

Dat

a P

ath

A

Dat

a P

ath

B

Axis/Apache/LinuxServer

dataPathConnectionPT

WSDLInterface

instrumentEnablelPT

To user’s WSDL

Power Enabler WSDLInstrument WSDL

New Instrument WSDL

and local consumption in use by different instruments. The whole power system could be represented as a single WSDL web service, which may or may not be a BPEL orchestration of a more complex set of web services for the power distribution at each node and other sites. Initially a single WSDL abstraction using a shore based proxy system of the power system may be the most practical implementation giving the critical timing and reliability issues that surround the architecture of the power system. In any event, which ever approach is taken, the Neptune administrator can use BPEL to link together the power web service with the instrument web service to enable the activation of an instrument. A simple illustrative example of how this may be implemented is shown in Figure 5.4.

5.2.6 Administrative orchestration for UCLP network

The UCLP administrator can offer two types of web services –a set of pre-configured APNs that are made available to a specific community or custom made APNs manually created for a specific end user upon demand. As well, some current UCLP implementations will allow users to set up an end-to-end lightpath on demand.

Some typical APNs may include pre-configured end to end lighpath from Chicago to Amsterdam for access by researchers in Chicago, a layer 1 transit lightpath from NYC to Chicago for international users, an Ethernet interface stub to a specific institution or ISP at an Internet exchange point, or a set of APNs along a given network path and so forth. An external user would discover all these pre-configured APNs, for which they have been authorized, when they wanted to orchestrate a new APN.

The administrative user of a given UCLP domain would create a WSDL web service for all APN cross connects and lightpaths that they wish to expose to external users. In some cases the cross connects and lightpaths can be orchestrations of much more complex internal web service APNs For example some users may want to represent a complex network cloud as a single APN service where an external user would not see the intermediary switches and lightpaths.

It is important to note that the orchestration engine of these web services resulting in the expression of a new web service can be completely independent of the engine(s) that support the component web services.

For example an administrative user can create APNs by dragging and clicking links across a special graphics map. Every “click” could represent the instantiation of a new web service that might be a constituent component of the larger APN. Each web service, in fact could be a simple interface, factory or proxy to a single underlying UCLP system or to multiple UCLPs for several independent instantiations. The APN web services would actually be related to real physical SONET channels and cross connects.

These web services would be only available to certain users and have a given lifetime in the order of weeks, months or year. Each web service that is exposed to an external user would have 2 primary port types:

43

(a) the ability to connect another lightpath web service limited, of course by what is physically possible; and

(b) the ability to create a child lightpath.

It is important to note that an APN would be made up of a concatentation of web services representing each link and cross connect in the path of the APN. This way users can “articulate” the APN from time to time by reconfiguring the web services to create new topologies, to create child APNs and/or to interconnect to other APNs that might be available to the user.

An example of some possible articulations would be to re-configure an APN for traffic engineering purposes at an Internet exchange point. Initially an APN may be connected to one peer, but at later time the end user may want to change the peering relationship to another peer, or have 2 simultaneous peers by creating a child APN.

Another possible articulation would be to change the topology of an APN in the event of a long time service outage on a given link.

5.2.6 Administrative orchestration for GMPLS or O-UNI network

The administrative requirements to configure a WSDL for the orchestration of a GMPLS, JIT or O-UNI as would be used in OMNInet are very similar to creating an on-deamnd end to end lightpath using UCLP. However, in this case most of the “orchestration” is done by proprietary or hidden processes. The external user has no choice or influence over the internal orchestration on how an end to end lightpath is setup across the network cloud. The process results in binary outcome – successful or unsuccessful.

All the administrative user needs to create a simple web service describing the endpoints and bandwidth for the underlying network interface. The port types for this web service would be similar to those of UCLP in that the user has to define the inbound and next hop requirements whether they be AS, IP number or slot/port/channel as well as the requested bandwidth.

Given the architecture of GMPLS, O-UNI, etc a number of features would not be available as with UCLP such as the ability to partition a lightpath, cross connect a lightpath with other third parties etc

5.2.7 Administrative orchestration IP routed network

The administrative requirements to configure a WSDL for the orchestration of an IP flow network such as that would be used in BCnet would be similar to that for GMPLS. All the administrative user needs to create a simple web service describing the IP endpoints and the web service for the virtual router. Given the architecture of IP routers, a number of features would not be available as with switches such as the ability to partition a

44

lightpath, cross connect a lightpath with other third parties etc. However some routing manufacturers do allow for nested VPNs, and so in that case the web services interface would allow the end user to create daughter lightpaths or VPNs but no daughter virtual router.

5.3 APN for a high energy physics network

In this scenario we will go through the steps that would be undertaken to allow Canada’s high energy physicists to have their own discipline specific IP network for the transmission of data from the CERN Large Hadron Collider to the Canadian Tier 1 site at TRIUMF British Columbia, and from there distribution of the data to the two Tier 2 sites and finally a private IP distribution network for physicists at the various universities can access the data.

In 2005 CANARIE has acquired, in partnership with SURFnet in The Netherlands, a 10G lambda from New York to the NetherLight in Amsterdam and from there an additional 10G facility on the GEANT 2 network to CERN in Switzerland.

In this scenario it is assumed that the 10G lightpath from New York through Netherlight to CERN is part of the SURFnet management domain. But the facility is offered directly to CANARIE and/or Canada’s High Physics community network – HEPnet.

Figure 5.5 is a conceptual screen shot of how a CANARIE super-user would create and assign APN resources and lightpath objects to the HEPnet administrator. This is complex diagram that attempts to illustrate a number of key concepts with UCLPv2.

Figure 5.4 Orchestration of power and instrumentation control

Figure 5.5 Screen shot of APN resources for high energy physics network

45

CANARIE ONS NetworkResources

STAR LIGHT HDXMAN LAN HDX

Pwave HDX

TRIUMF

OME

YVR

YEG

YCG

WinnipegYYZ

YUL

YOW

Halifax

Seattle

Vancouver

Chicago

Toronto Ottawa

Montreal

New York

ONSONS

STAR LIGHT HDX

ONS

MAN LAN HDX

BCnet

Amsterdam

New York

Toronto

Vancouver

Victoria

Edmonton

Ottawa

Geneva

Montreal

To Fermi

To BrookhavenNew APN Resource list composition

ONS

New York Geneva

SURFnet APN resources advertised to CANARIE

Amsterdam

CANARIE OME Network Resources

Lightpath Object Creation

Edmonton

Chicgao

EdmontonToronto

Chicago is hidden

1

2

3 4

5

CANARIE ONS NetworkResources

STAR LIGHT HDXMAN LAN HDX

Pwave HDX

TRIUMF

OME

YVR

YEG

YCG

WinnipegYYZ

YUL

YOW

Halifax

STAR LIGHT HDXMAN LAN HDX

Pwave HDX

TRIUMF

OME

YVR

YEG

YCG

WinnipegYYZ

YUL

YOW

Halifax

Seattle

Vancouver

Chicago

Toronto Ottawa

Montreal

New York

ONSONS

STAR LIGHT HDX

ONS

MAN LAN HDX

BCnet

Amsterdam

New York

Toronto

Vancouver

Victoria

Edmonton

Ottawa

Geneva

Montreal

To Fermi

To BrookhavenNew APN Resource list composition

ONS

New York Geneva

SURFnet APN resources advertised to CANARIE

Amsterdam

ONS

New York Geneva

SURFnet APN resources advertised to CANARIE

Amsterdam

CANARIE OME Network Resources

Lightpath Object Creation

Edmonton

Chicgao

EdmontonToronto

Chicago is hidden

1

2

3 4

5

The dotted lines represent APN resources i.e. individual Web services that have not yet been compiled into a functioning APN using BPEL. The solid lines represent lightpath objects or existing running APNs that are BPEL scripts running as web services on a CA*net 4 or other server.

The creation of the HEPnet APN involves the following steps:

(a) Identification by the physical network administrative user of those APN resources that are to be assigned to a third party such as HEPnet

(b) Optionally aggregating some of these resources into lightpath objects and thereby hiding some of the complexity to the third party such as HEPnet

(c) HEPnet accepting these resources including lightpath objects and concatenating them into a working APN

(d) HEPnet partitioning the APN and offer child APNs or lighpath objects to third parties

First of all it should be noted that CA*net 4 is in fact made up of essentially 2 separate physical networks: one network that contains Cisco ONS15454 switches and the other made up of Nortel OME 6500s. Given CANARIE’s unique mandate of NOT attempting to build a homogenous single manage network like a a traditional telecom company this separation of the network by equipment manufacturer makes a lot of sense. These 2 separate physical networks can be treated as facilities in two separate management domains in addition to the SURFnet domain.

The CA*net 4 network administrative staff would decide which windows and resources that they wanted to display in the various windows on their APN screen. In this example the administrator has show the 2 CA*net 4 networks in 2 separate windows as if they were networks in two separate physical management domains.

The administrator user in each network domain must first identify those network resources that are to made available to third parties. These resources, which are each represented as an individual web service can be lightpaths, interfaces, virtual routers, cross connects, instruments and aggregation of other resources. This can be done a number of ways – for example by highlighting each resource and assigning permissions, time to lease, SONET channel, etc and dragging or dropping them into the APN resource list composition window as shown in Figure 5.5 (3) and (4). These selected APN resources can then be advertised directly to the intended recipient, or included in a list of at well known URL, and be “discovered” by the intended recipient through a directory or web crawler.

In some cases the physical domain administrator may wish not to advertise all these resources separately, but instead aggregate a number of resources into a single advertised resource. This is illustrated in Figure 5.5 where the lightpath network elements between Edmonton and Toronto are integrated into a single lightpath object (1) . This aggregation can be a BPEL script window running on a local server that concatenates these resources

46

together. The resulting aggregation is in itself a Web service which can then be assigned the appropriate permissions and dragged into the APN resource composition window (2).

Finally as shown in this example the SURFnet resources that were originally assigned to CANARIE can also be re-assigned to HEPnet and dragged into the APN resource composition window (5). The CA*net 4 administrator has two ways of doing this – they can pass on the APN resources intact; or they can convert the resources into a running web service lightpath object running on a CANARIE server. In this example it is assumed that the resources have been passed through to HEPnet directly.

Note, at this point in time, no functioning APN has been created – only a set of lightpath resources has been identified with permission assigned to the designated recipient. The next stage is for the recipient of the APN resource list to compose these resources into a functioning APN as illustrated in Figure 5.6.

The intended recipient, in this case HEPnet, also has an APN management screen/GUI similar to CANARIE’s where they can now have two choices:

(a) they compose the various WS resources into one or more functioning APNs and/or lightpath object WS; or(b) they can re-assign some or more the WS resources including newly created lightpath object WS to their parties

Figure 5.6 Screen shot of APN composition high energy physics network

47

Amsterdam

New York

Toronto

Vancouver

Victoria

Edmonton

Ottawa

Geneva

Montreal

To Fermi

To Brookhaven

TRIUMF APN

UoVic Campus802.11 LightpathObject

UBC CampusCWDM LightpathObject

Victoria

Vancouver

Lightpath Object for 2 GbpTiier 2between TRIUMF and UoVic

TRIUMFUoVic

Composition Window

Amsterdam

New York

Toronto

Vancouver

Victoria

Edmonton

Ottawa

Geneva

Montreal

To Fermi

To Brookhaven

TRIUMF APN

UoVic Campus802.11 LightpathObject

UBC CampusCWDM LightpathObject

Victoria

Vancouver

Lightpath Object for 2 GbpTiier 2between TRIUMF and UoVic

TRIUMFUoVic

Composition Window

The dotted lines in the largest window represent APN resources i.e. individual Web services that have not yet been compiled into a functioning APN using BPEL. Solid lines represent lightpath object WS or existing running APNs

In this scenario the HEPnet administrator drags APN resources from the resources provided by CA*net 4 as well as other resources from UoVic and UBC campus to create a lightpath object WS or running APN. The APN or lightpath object WS is a 2 Gbps circuit between the UoVic and UBC campuses. It is assumed that this circuit will be used for the Tier 2 connection between TRIUMF and UoVic.

Note that UoVic and UBC resources are campus LAN or DWM resources assigned to HEPnet. It is assumed the UoVIC and UBC network administrators have their own UCLP systems with their own GUIs in which they can assign their network resources to whatever third party they wish.

The HEPnet administrator can continue to create a number of additional APNs including a 5 Gbps lightpath between TRIUMF and CERN, additional Tier 2 lightpaths and finally a 1GBps routed network between the various Tier 3 high energy physics labs across the country as illustrated in Figure 5.7

Figure 5.7 Resultant APN compositions for high energy physics network

In this scenario there are 3 APN compositions that were created by the HEPnet administrator:

(a) one 5 Gbps lightpath from TRIUMF to CERN;(b) two 2 GBps Tier 2 lightpaths from TRIUMF to UoVic and UoToronto

respectively; and(c) one 1 Gbps routed lightpath connecting the various Tier 3 sites across the country

linking together virtual router web services on the CA*net 4 substrate routers.

48

1G HEPnet daisy chainrouted

UoToronto PhysicsTier 2

UoVictoria PhysicsTier 2

TRIUMFTier 1

CERNTier 0

Amsterdam

New York

Chicago

Toronto

Vancouver

Victoria

FERMITier 1 Brookhaven

Tier 1

UBCPhysics

UAPhysics

UoTPhysics

CarletonPhysics

UdMPhysics

CA*net 4

Edmonton

Ottawa

To other physics users at smaller universities Geneav

CWDMCWDM

5G Tier 1 data

2G Tier 2 data

Optionalinterfaces

Note: Typical View on TRIUMF UCLP GUI

1G HEPnet daisy chainrouted

UoToronto PhysicsTier 2

UoVictoria PhysicsTier 2

TRIUMFTier 1

CERNTier 0

Amsterdam

New York

Chicago

Toronto

Vancouver

Victoria

FERMITier 1 Brookhaven

Tier 1

UBCPhysics

UAPhysics

UoTPhysics

CarletonPhysics

UdMPhysics

CA*net 4

Edmonton

Ottawa

To other physics users at smaller universities Geneav

CWDMCWDM

5G Tier 1 data

2G Tier 2 data

Optionalinterfaces

Note: Typical View on TRIUMF UCLP GUI

HEPnet, if it so wishes can assign management and control of each one of these APNs to different entities. For example it would make sense for TRIUMF to manage the 5 GBps APN to CERN – that way it could control the re-routing of the APN to Brookhaven or Fermi Lab if there was a disruption in the direct connection to CERN.

Similarly the two 2 Gbps lightpaths for the Tier 2 sites could be handed to the administrator of those sites so that they could have more flexibility in the re-routing and partitioning of those APNs.

It is important to note that the HEPnet administrator has two choices in how they hand off APNs to third parties like TRIUMF or the Tier 2 site administrators:

(a) HEPnet could re-assign the permissions on those specific resources it received from CA*net 4; or

(b) HEPnet could create the APNs or lightpath object WS themselves and hand these over to the designated recipients

The first approach provides the greatest flexibility to the recipient as they get a collection of web services that they can then integrate into their own workflow schema and/or combines with other web services. The disadvantage is that they require their own BPEL server in which they convert the BPEL scripts into running web services

The second approach is much simpler for the recipient as they don’t need their own BPEL server, but they are then restricted in the flexibility they have in terms of manipulating the APN and integrating it with other web services.

49

6.0 Conclusions and Next Steps

In the spring of 2005 CANARIE issued a call for proposals for the next generation of UCLP based on the framework described in this document. Three teams have been selected to do development of 3 different versions of UCLP. These projects are now in their early prototype development stage with full functional products expected to be available in the first quarter of 2006. All 3 versions of UCLPv2 will be available as open source.

Those who are interested in participating in the UCLPv2 program are encouraged to contact the author.

.

50

7.0 Acknowledgements

The CANARIE CA*net 4 program is funded by a grant from the Government of Canada through Industry Canada.

Thanks to the following individuals for their comments, suggestions and editorial comments:Dave Macneil, Rene Hatem, Darcy Quesnel of CANARIEEric Byres of BCITLoki Jorgenson of Apparent NetworksBill Rutherford of RXX Network Inc Michel Savoie, Scott Campbell, Mathieu Lemay, Jing Wu of CRC

51

8.0 References

[GRID-WFL] http://www.extreme.indiana.edu/groc/Worflow-call.html[BPEL-ENG] http://www.activebpel.org/info/intro.html[BPEL-INFO] http://searchwebservices.techtarget.com/originalContent/0,289142,sid26_gci880731,00.html[CAVE-WAVE] http://www.evl.uic.edu/events/eve_project.php3?indi=298[CLS] http://www.cls.usask.ca/[E-VLBI] http://www.evlbi.org/evlbi/evlbi.html[FREEFLUO] http://freefluo.sourceforge.net/[GSMP] http://www.ietf.org/html.charters/gsmp-charter.html[IBM-MQ] http://www-306.ibm.com/software/integration/mqfamily/integrator/telemetry/[ITU_Y.1312] http://www.itu.int/itudoc/itu-t/aap/sg13aap/history/y1312/[ITU_Y.1313] http://www.itu.int/itudoc/itu-t/aap/sg13aap/history/y1313/[KEPPLER] http://kepler-project.org/index.html[LABVIEW] http://www.ni.com/labview/[LHC] http://lhc-new-homepage.web.cern.ch/lhc-new-homepage/[L1VPN] T. Takeda, et al, “Layer 1 Virtual Private Networks: Service Concepts, Architecture Requirements, and Related Advances in Standardization,” IEEE Comm. Magazine, Vol. 42, No. 6, June 2004, pp. 132-138. http://www3.ietf.org/proceedings/05aug/l1vpn.html[MEGACO] http://www.ietf.org/html.charters/megaco-charter.html[NEPTUNE] http://www.neptunecanada.ca[NLR] http://www.nlr.net/[NRC-NMR] http://www.canarie.ca/conferences/advnet2004/ppt/impey.ppt[NSF-GENI] http://www.nsf.gov/cise/geni/[OPENDAP] http://opendap.org[OPTICAL_BGP] Optical BGP (OBGP): InterAS lightpath provisioning - draft-parent-obgp-01.txt[ORION]http://www.dmac.ocean.us/dacsc/imp_plan.jsp[PFLDnet] http://www-didc.lbl.gov/PFLDnet2004/[RDF] http://www.w3.org/RDF/[ROADNET] http://roadnet.ucsd.edu/[SOA] www.soasystems.com[TAVERNA] http://taverna.sourceforge.net/[TURNER] http://www.arl.wustl.edu/~jst/talks/keynote05.pdf[UCLP] http://www.canarie.ca/canet4/uclp/uclp_software.html.[UCLP-DESIGN] http://www.canarie.ca/canet4/library/c4design/canet4_design_document.pdf[UCLPv2] http://www.canarie.ca/canet4/uclp/UCLP_Roadmap.doc[UMLv2] http://www.openwflow.com/[VIRTUALIZATION] http://www.arl.wustl.edu/netv/main.html[WORKFLOW-INFO] http://www.ifan-online.org/isoportalsupport/help/webwork/wf_overview.html[WSDL] http://www.canarie.ca/canet4/uclp/UCLP_Roadmap.doc[WS-SEC] http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wss

52

53