rights / license: research collection in copyright - non ...1424/eth... · listings 3.1 menu de...

Research Collection

Master Thesis

Generalizing the origo back-end for large-scale distributedapplications using Aranea

Author(s): Herlig, Beat

Publication Date: 2009

Permanent Link: https://doi.org/10.3929/ethz-a-005906501

Rights / License: In Copyright - Non-Commercial Use Permitted

This page was generated automatically upon download from the ETH Zurich Research Collection. For moreinformation please consult the Terms of use.

ETH Library

https://doi.org/10.3929/ethz-a-005906501

http://rightsstatements.org/page/InC-NC/1.0/

https://www.research-collection.ethz.ch

https://www.research-collection.ethz.ch/terms-of-use

��

��

��

�� !�∀��∀��

#��∀!�∀��

��∃�%��&∋�()&�&∗+

Abstract

Origo's back-end is designed to scale. From the start this was made one ofthe primary design goals. To that e�ect, the back-end consists of multipleapplications � nodes � that communicate by message passing. As all nodesare implemented in Ei�el, not much attention was given to data-serialization:payload data of messages would simply be serialized in an Ei�el-speci�c for-mat.

The goal of this thesis is to amend this situation. By replacing the currentserialization protocol with JSON � a language-independent, cross-platformformat � we open many new possibilities for Origo, as nodes are not re-stricted exclusively to be written in Ei�el anymore.

Further achievement of this thesis is the upgrade of the content man-agement system of Origo's front-end. As substantial code refactoring wasrequired, many inconsistencies could be eradicated at the same time.

i

Acknowledgments

I would like to thank the entire Origo team for all their support, especiallyalong my �rst steps with Origo. Special thanks to my supervisor Till Bay, forhis ever competent advice and general help will all matters at hand. Further,I'd like to thank Bernd Schoeller and Patrick Ruckstuhl for many constructivediscussions about Origo and its design. Finally, I thank my family and friendsfor their continuous encouragement and support throughout my studies andthis thesis.

B. HerligZürich, August 2009

iii

Contents

Abstract i

Acknowledgments iii

Contents v

List of Figures ix

Listings xi

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Origo 32.1 Back-End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.1 Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1.2 Aranea . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.3 Use case . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Front-End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2.1 Drupal . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Drupal Migration 133.1 Development Setup . . . . . . . . . . . . . . . . . . . . . . . . 143.2 Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2.1 New menu system . . . . . . . . . . . . . . . . . . . . . 153.2.2 FormAPI improvements . . . . . . . . . . . . . . . . . 183.2.3 The arguments to url() and l() have changed . . . . 203.2.4 Simpli�ed �le upload . . . . . . . . . . . . . . . . . . . 213.2.5 Porting the theme . . . . . . . . . . . . . . . . . . . . 223.2.6 Porting the install pro�le . . . . . . . . . . . . . . . . . 223.2.7 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . 24

v

vi CONTENTS

3.3 Other Changes to the Front-End . . . . . . . . . . . . . . . . 283.3.1 Hook extension for install pro�le . . . . . . . . . . . . 283.3.2 Module origo_global . . . . . . . . . . . . . . . . . . 293.3.3 Proper database updates � with Drush . . . . . . . . 303.3.4 Project access via subdirectory . . . . . . . . . . . . . 32

3.4 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.5 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4 Aranea Message Transport 354.1 Serialization Protocols . . . . . . . . . . . . . . . . . . . . . . 374.2 Aranea Messages . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2.1 Sending and receiving . . . . . . . . . . . . . . . . . . 424.2.2 Convert clauses . . . . . . . . . . . . . . . . . . . . . . 44

4.3 Message Generation using oAW . . . . . . . . . . . . . . . . . 464.3.1 openArchitectureWare . . . . . . . . . . . . . . . . . . 464.3.2 Domain speci�c language and its grammar . . . . . . . 464.3.3 Code generation . . . . . . . . . . . . . . . . . . . . . . 50

4.4 Serialization and Deserialization . . . . . . . . . . . . . . . . . 554.4.1 Value classes . . . . . . . . . . . . . . . . . . . . . . . 564.4.2 Serialization . . . . . . . . . . . . . . . . . . . . . . . . 574.4.3 Deserialization . . . . . . . . . . . . . . . . . . . . . . 59

4.5 Using the Message Framework . . . . . . . . . . . . . . . . . . 644.5.1 Creating primitive types . . . . . . . . . . . . . . . . . 644.5.2 Creating sequence types . . . . . . . . . . . . . . . . . 644.5.3 Creating messages . . . . . . . . . . . . . . . . . . . . 654.5.4 Accessing message values . . . . . . . . . . . . . . . . . 65

4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5 Conclusions 675.1 Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.2.1 Transport of binary data . . . . . . . . . . . . . . . . . 685.2.2 Front-end unit testing . . . . . . . . . . . . . . . . . . 685.2.3 Other improvements . . . . . . . . . . . . . . . . . . . 685.2.4 User administration . . . . . . . . . . . . . . . . . . . . 695.2.5 Administrative functions accessible through web . . . . 69

Appendices 73

A Origo Case Study for Drupal.org 73

CONTENTS vii

B Comparison of Marshalling Methods 81

C JSON 85

D Messages Provided by Aranea 89

E Aranea's New Message Transport 93

Bibliography 99

viii CONTENTS

List of Figures

2.1 Origo work item icons . . . . . . . . . . . . . . . . . . . . . . 32.2 Origo architecture . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Origo nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.4 Use case: login . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.1 PHP debugging with PDT . . . . . . . . . . . . . . . . . . . . 15

4.1 Aranea connecting to ActiveMQ . . . . . . . . . . . . . . . . . 364.2 Benchmark of Serialization protocols . . . . . . . . . . . . . . 394.3 Aranea Message Framework . . . . . . . . . . . . . . . . . . . 41

ix

x LIST OF FIGURES

Listings

3.1 Menu de�nition in Drupal 5 . . . . . . . . . . . . . . . . . . . 163.2 Menu de�nition in Drupal 6 . . . . . . . . . . . . . . . . . . . 173.3 Wildcard loader function in Drupal 6 . . . . . . . . . . . . . . 173.4 Submit function in Drupal 5 . . . . . . . . . . . . . . . . . . . 183.5 Submit function in Drupal 6 . . . . . . . . . . . . . . . . . . . 193.6 Link function in Drupal 5 . . . . . . . . . . . . . . . . . . . . 203.7 Link function in Drupal 6 . . . . . . . . . . . . . . . . . . . . 203.8 File upload in Drupal 5 . . . . . . . . . . . . . . . . . . . . . . 213.9 File upload in Drupal 6 . . . . . . . . . . . . . . . . . . . . . . 213.10 De�ning custom menu-paths in Drupal 5 . . . . . . . . . . . . 243.11 De�ning custom menu-paths in Drupal 6 . . . . . . . . . . . . 243.12 Adding issue-status tags in Drupal 5 . . . . . . . . . . . . . . 263.13 Adding issue-status tags in Drupal 6 . . . . . . . . . . . . . . 263.14 Utility function for retrieving issue-tags vocabulary IDs . . . . 263.15 Utility function for retrieving MediaWiki's �lter ID . . . . . . 273.16 Hook extension to origo.profile . . . . . . . . . . . . . . . . 283.17 Excerpt from the shell script upgrading Drupal to version 6.x . 314.1 Feature new_ems_text_message of class A_MESSAGE . . . . 434.2 Feature make_from_ems_text_message of class A_MESSAGE 444.3 Grammar of message properties an their type . . . . . . . . . 464.4 Grammar of Aranea namespace and message . . . . . . . . . . 484.5 An extension function for Aranea's DSL's custom editor plug-in 494.6 A check function for Aranea's DSL's custom editor plug-in . . 504.7 A DEFINE block in XTend/oAW . . . . . . . . . . . . . . . . . 514.8 A EXPAND block in XTend/oAW . . . . . . . . . . . . . . . . . 514.9 Suppressing whitespace output in XTend/oAW . . . . . . . . . 524.10 Namespace template in Main.xpt . . . . . . . . . . . . . . . . 534.11 Generating feature serialization_list . . . . . . . . . . . . 544.12 Serialization of A_BOOLEAN_VALUE . . . . . . . . . . . . 574.13 Serialization of A_SEQUENCE_VALUE . . . . . . . . . . . . 574.14 Serialization feature of A_PARSABLE . . . . . . . . . . . . . 58

xi

xii LISTINGS

4.15 Simpli�ed serialization feature of a message extending anotherone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.16 make_from_tokenizer as implemented in A_PARSABLE . . 604.17 parse_tag: recreating �elds according to the parser's last read

�eld-name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.18 make_from_tokenizer from class A_MESSAGE: the entry

point of deserialization . . . . . . . . . . . . . . . . . . . . . . 624.19 make_from_tokenizer from class A_INTEGER_VALUE . . 624.20 make_from_tokenizer from class A_SEQUENCE_VALUE . 634.21 Create primitive Aranea framework types . . . . . . . . . . . . 644.22 Create A_SEQUENCE_VALUEs . . . . . . . . . . . . . . . . 644.23 Creating a message object . . . . . . . . . . . . . . . . . . . . 654.24 Value access of Aranea's composite types . . . . . . . . . . . . 66

Chapter 1

Introduction

1.1 Motivation

Origo's architecture using discrete computational units for data processing� nodes in the back-end (see section 2.1), Drupal sites in the front-end (seesection 2.2) � is designed to scale: more processing power can be addedby simply running more nodes, potentially on di�erent servers, and moreproject-sites can be used by adding more webservers. But all these unitsneed to communicate, send and receive data. Typically this involves variousmethods of interprocess communication. One of these methods � which canbe used when communicating across seperate hardware � is message pass-ing.Message passing enables systems to communicate across hardware bound-aries: processes do not need to reside on the same computer, but may shareinformation over a network. This enables us to build software systems thatcan harness more processing power than a single computer could provide:services split over an array of servers, but still working together.But the separation of processes, and thus of memory that holds their datastructures, has one big inherent problem. When passing messages, howshould we represent the data being transmitted? This requires a proper,unambiguous serialization and deserialization protocol.

Past versions of Origo somewhat sidestepped this: as all of Origo's nodesare implemented in Ei�el, a serialization class of Ei�el's base library wasused to transform objects into a series of bytes. While this works perfectly,and is fully transparent to clients of the transport mechanism, it restrictsdevelopment of nodes to the Ei�el programming language.

1

2 CHAPTER 1. INTRODUCTION

The goal of this thesis is open up more development possibilities forOrigo's back-end, by refactoring inter-node communication, i.e. replacingthe existing serialization protocol with a language and platform independentone.

A secondary goal was to update Origo's front-end to the current majorversion of Drupal, the CMS framework that the front-end builds upon. Assome of Drupal's API calls changed between these versions, this resulted insubstantial code refactoring. As a positive side e�ect of this, many inconsis-tencies could be eradicated.

1.2 Overview

Chapter 2 gives an overview over Origo and its architecture, while chapter 3details the work of upgrading the Drupal front-end to version 6. Chapter 4documents Aranea and its new message transport system. The �nal chapter 5concludes the thesis, and gives an overview over future work.

Chapter 2

Origo

Origo [25, 41] is a platform for distributed software development. It is anopen, modular and extensible software development, management and distri-bution platform that incorporates various applications and tools. It providesservices such as source control management, issue tracking, release hostingand also a public API that can be used to integrate Origo into externalapplications. Origo consists of two main parts: a back-end responsible fordata management and processing, and a front-end displaying information re-trieved from the back-end, and providing a user interface to Origo. The twoparts interact through API calls (see [26]) using XML-RPC [47]. Figure 2.2gives a schematic of Origo's architecture.

Figure 2.1: Origo work item icons: blog entry, commit, issue, release, wikientry and comment icon (from left)

An important feature of Origo is the notion of a work item. Work itemsare noti�cation on changes in a project, and help developers to keep track ofproject evolution. Work items can be used by other platform users who wantto stay informed about the development of projects that they are interestedin. Work items are created on key events in the system, e.g. when a wikipage is edited, Subversion commits take place or a blog entry is made. Anoverview of all Origo work items can be seen in �gure 2.1. For more details,see chapter 4 in [48] or section 6.1.1 in [41].

3

4 CHAPTER 2. ORIGO

Figure 2.2: Architecture of the Origo platform

2.1 Back-End

The controller of Origo is Origo Core [45]. It is the middleware and controllayer of the Origo platform. Origo Core allows implementing use cases (seesection 2.1.3) and controls the various Origo components accordingly. Itcontrols and distributes the �ow of messages through the back-end.

2.1.1 Nodes

The back-end consists of disjoint data processing units called nodes. Thereare di�erent types of nodes, each having a di�erent function within the plat-form. In line with Origo's design goals of scalability and extendability, thenodes are message oriented [35], and communicate on an enterprise servicebus using Apache ActiveMQ (more in chapter 4). The message orientationallows to run arbitrary numbers of each node type, thus providing good scal-ability and redundancy. Messages can either be sent to a group of nodesimplementing a certain service (identi�ed by their common node type), oraddressed individually by specifying a node's name. Figure 2.3 shows thearchitecture of the platform.

We here present a quick overview over the nodes currently employed in

2.1. BACK-END 5

Figure 2.3: Origo nodes forming the back-end

Origo:

Core node Message bus and controller of the back-end. It controls the othernodes according to user de�ned use cases by creating and distributingcontrol instructions or relaying messages to speci�c nodes.

API node Provides an XML-RPC [47] interface for Origo using Goanna[29].For each incoming XML-RPC request a speci�c message is constructed,and sent to the Core node. An API node can be started in normalor internal mode. The internal mode provides special services onlyavailable to Origo's front-end, but not to 3rd party applications.

Con�g node Used to execute commands, and read or write �les on theoperating system (mostly shell scripts and con�guration �les). This isused e.g. for FTP and Subversion [33] access rights management.

Storage node Responsible for managing all Origo related data. Internallya MySQL [23] database is used to store data persistently.

Mail node Allows the sending of emails. It can use a local or remote mailserver to deliver mails.

6 CHAPTER 2. ORIGO

Statistics node Creates and maintains statistical data for projects. Cur-rently the web-server, issues and work items serve as data sources.

RRD node Creates and maintains RRDTool [31] �les with data aggregatedby the statistics node.

All of Origo's nodes are implemented in Ei�el [43].

2.1.2 Aranea

Aranea is an Ei�el library that provides messaging [35] capabilities for appli-cations. It is a middleware infrastructure implementing an enterprise servicebus (ESB). Aranea uses Apache ActiveMQ [2] as the underlying MessagePassing system. Aranea-enabled applications can be distributed into com-putational units and communicate via Message Passing.

Historically, JXTA [20] was the messaging layer of Origo, implementedwith Vampeer [46], a peer-to-peer framework in Ei�el implementing theJXTA protocol (which was exchanged in fall 2008 for ActiveMQ). Realiz-ing that not only Origo, but other applications could bene�t from an Ei�elESB technology stack, it was since extracted from Origo and refactored intoa separate library. Aranea abstracts and consolidates the way that Ei�elnodes use messages to communicate with each other. Besides the messagingframework, Aranea comes with two node implementations: a mail node anda con�g node (see section 2.1.1 for a description).

Aranea's messaging system will be detailed in chapter 4.

2.1.3 Use case

Origo Core is service oriented [36]: services are provided by nodes and thecore is the message bus and controller. Based on programmed use cases,Origo's core controls the other nodes by sending control instructions. Usecases are de�ned in Ei�el classes. Control instructions are sent as asyn-chronous messages over a reliable message layer � this is detailed in chap-ter 4. An example for a simple use case is a user login:

1. The user enters his credentials in the web front-end: an XML-RPC callis generated and received by the back-end's API-node.

2.2. FRONT-END 7

Figure 2.4: An example of a use case: The login process

2. The API node generates an Ei�el message from the call, and sends itto the controller, i.e. the core node.

3. The core node passes the message on to the storage node, for veri�ca-tion.

4. The storage node veri�es the user's credentials, and generates either anerror message, or opens a user-session.

5. The core node relays the storage node's answer to the API-node.

6. The API-node returns the XML-RPC call with either the session iden-ti�er, or an error message.

Figure 2.4 depicts a UML diagram of this use case.

2.2 Front-End

The integrated platform built using Origo Core combines several externalapplications. Each of these applications can possibly have its own web in-terface, e.g. a web front-end to a subversion repository. But also externalapplications that have no web interface themselves o�er their functionalityto Origo Core. To make the platform usable as a central point for managingproject life, we provide a single web interface for all services o�ered by Origo.It should provide all necessary functionality to manage a project hosted on

8 CHAPTER 2. ORIGO

the platform. It uses and extends the use cases de�ned in Origo Core. Addi-tionally a work item system is integrated into the web interface. Work itemsdisplay noti�cations about changes and evolution of a project on a user'sOrigo Home page. Work items can be used by a developer or a user to keeptrack of changes in his own and bookmarked projects.

2.2.1 Drupal

Origo's web interface uses the free open-source content management sys-tem Drupal [8], a fully modular framework that suits Origo's needs particu-larly well. Drupal provides an extensive API through public PHP functionsand themable output. It provides a publish/subscribe model through hooks,where aptly named functions are called at various stages in Drupal's work-�ow. Using this extension mechanism, it is normally not necessary to modifycode in Drupal Core itself, as data can be manipulated at various stages byimplementing hooks in a module. This makes Drupal easily extensible, andindeed its big user community has produced numerous modules that can befreely combined. A template system, built upon the same notion of hookfunctions, is provided for customizing the site's design, e.g. freely placingspecial blocks (i.e. areas that modules can �ll with content); this process iscalled theming. Another Drupal feature that serves Origo well is its sup-port for multi-site installations: arbitrary numbers of completely disjointsites (instances), i.e. with separated databases and thus separate user andcontent data, can share a common codebase and themes. This makes codedeployment, and maintenance of the sites particularly easy.

The Origo web front-end is built using both custom theming and modules,and is run as a Drupal multi-site: each Origo project is a fully functionalDrupal site � with its own database, but with a common theme and sharedmodules.

The separation of the Drupal databases mandates that users navigatingbetween instances possess (separate) user accounts in all of them, i.e. signingup and activating user accounts multiple times. Furthermore, login creden-tials (i.e. PHP active sessions) would not be shared. With Origo, this wouldresult in users having to log on to each project separately. To remove this re-striction, a single sign-on mechanism was developed: by storing an additionalcookie on the client, we make the Origo user name and encrypted passwordavailable in all Drupal instances. Thus we can simply log the user on to everyinstance by using the back-end's login function. For details see section 6.2.1in [41].

2.2. FRONT-END 9

Drupal organizes its content in nodes � a term that can lead to confusion, asOrigo's back-end, too, uses this term. Drupal nodes are content elements ofthe web front-end, for Origo: wiki-pages, issues, blog-entries and comments.

Here's an overview over the Drupal modules that were developed forOrigo:

User authentication is handled by the back-end and replicated backto Drupal. The custom module uses XML-RPC to authenticate usersin the back-end. The authentication module works by implementinghook_form_alter, where Drupal's form validation- and submit func-tions are swapped against custom ones that authenticate with the back-end. Furthermore, Drupal's session-handling functions are overwrittenwith slightly changed ones that synchronize session information withthe back-end. This interception of Drupal's login mechanism is lever-aged to implement a single-sign-on system: users that change betweenprojects, i.e. Drupal sites, are automatically logged in at the new site,and the still valid back-end authentication recreates a Drupal sessionat the new site-instance.The module further provides an XML-RPC framework for other Origo-API functions, both authenticated (which require a valid user-session)and anonymous API calls.

is another custom module providing a bulk of Origo's features. It han-dles all user related functions, such as pro�les, settings, networking(user friendships and communities), as well as work items (includingRSS feed). It o�ers input masks for requesting projects and communi-ties, and displays an aggregated view of issues reported by the user. Italso provides the interface to Origo's release hosting, i.e. displaying thelist of a project's releases, as well as forms for uploading and categoriz-ing releases. Furthermore it implements an interface for the system'sadministrative functions, such as mass mailings to all users.

While most origo_home functions are straightforward form processing,its main task is bringing the front- and back-end together. On the Dru-pal side this results in intercepting all processed nodes, hook_nodeapiis crucial for this: it checks which action is performed on what type ofnode, and registers this action accordingly with the back-end.

Origo uses its own bug tracking module which integrates with the back-end. Issues are a custom node type with its own vocabulary. This way,issues can be tagged, indexed and searched by using Drupal's taxonomy

10 CHAPTER 2. ORIGO

module. Metadata of an issue (such as open/closed-, assignment- orresolution-status) is added to a issue-node's tags, and thus seamlesslyintegrated with other vocabulary data. The main page of the issuetracker provides an interface to this data, and besides displaying apaginated list of all issues, gives the user the possibility to search and�lter these tags. The di�erent �lters can be customized in any way,and the resulting combination can be saved.

issue_tracker implements quite a few Drupal hooks, e.g. hook_view,hook_form_alter, hook_update, hook_insert or hook_nodeapi, be-cause it is necessary to intercept node processing at various stages, andsynchronize issue-nodes with data from the back-end.

When using Origo's Mylyn Connector [22, 44], issues and correspond-ing comments can be added from within Eclipse. As the connector'spreview function cannot render MediaWiki syntax (which is the onlyinput format allowed by Origo), this small module provides an XML-RPC API call for rendering MediaWiki syntax as it will appear in theissue_tracker.

This module aims to provide content spanning across all of Origo'sDrupal instances (i.e. projects sites), such as Origo's Terms of Use. Itreads static content (that is not bound to any Drupal database) froma �le, and provides it at a location speci�ed in the module's code. Itis designed to be modi�ed by the maintainer of an Origo instance, andthus �le names and URI paths are hard coded into the module.

This module uses RRDTool to generate images from statistical datagathered in the back-end (see �Statistics node� in section 2.1.1).

is a replacement module for some Origo pages, namely the release andproject list. Its main goal is to provide a solid framework for pagingand ordering results from XML-RPC calls to Origo's back-end.

The following modules were developed for use with Origo, but are self-contained: the wiki-functionality provided by them could be used in otherDrupal installations.

enables users to restrict access to wiki pages and issues to project mem-bers. This is achieved by checking appropriate Drupal access rights.

wraps MediaWiki's library for parsing and rendering MediaWiki syn-tax [21]. The MediaWiki �lter is enabled for all of Origo's content:

2.2. FRONT-END 11

issues, blogs, forum posts, etc, thus enabling users to use this syntaxthroughout Origo.

provides the possibility to edit individual sections of a wiki page.

12 CHAPTER 2. ORIGO

Chapter 3

Drupal Migration

Today's world of software development is very fast-paced: users demandfast implementation of feature requests, often encouraged and supported bymanagement. More important (from a developer's view) are security threats,which require constant awareness, and probably patching of holes in one'sapplication. Drupal is no di�erent. Regular releases of major and minorversions � the latter often including security �xes � were one of the reasonsthe Origo team decided to implement the front-end on top of Drupal.

Drupal has its own take on maintaining releases, though. From theirmission statement1:

origo_authorigo_homeissue_trackerformatting_serviceorigo_globalorigo_statisticsorigo_xmlrpc_tabledeveloper_pagesmediawiki_filtersection_editThe drop is always moving

Everyone considering Drupal should understand that Drupal de-velopment is always on the cutting edge and with each majorrelease there will be radical improvements. While the upgradepath will reliably preserve your data, there is no backward com-patibility with the previous Drupal code.. . .Therefore, people adopting Drupal for their web or CMS projectshould plan for periodic upgrades of their project to the latestmajor release (every 12-24 months) in order to bene�t from theongoing active support of one of the �nest open source develop-ment communities.

In a nutshell: Drupal prepares reliable ways to update sites and their data,but they won't do so for very old versions.

1http://drupal.org/node/65922

13

http://drupal.org/node/65922

14 CHAPTER 3. DRUPAL MIGRATION

With Origo's �rst public release in August 2007, its front-end used Drupalversion 5.x 2. At the start of this thesis, Origo still used the branch of version5, then the current release 5.12. Accordingly, the �rst task of this thesis wasto migrate Origo to the most recent version of Drupal's version 6.

3.1 Development Setup

Upgrading Origo's front-end would break a lot of things until every part of itwas properly adjusted to the new Drupal version. In order not to interruptthe live system, it was essential to have a complete, all-encompassing Origoinstallation on a local machine: not only for debugging when upgradingthe modules, but also for proper testing. This ensured that regular codemaintenance was still possible for other team members.

Following Origo's setup guide3 � and correcting outdated bits of infor-mation � a local system was soon set up. Adding custom entries to the ma-chine's hosts �le not only allowed accessing the local Origo instance througha proper URI (instead of using the loop-back address), but also allowed todrive multiple front-ends (by con�guring multiple virtual hosts in Apache'scon�guration) with the same back-end. This parallel setup of both Drupal5 and 6 allowed for quick comparison between the versions, and testing ofproper functionality of the modules with Drupal 6.

Drupal, including its modules and theme template system, is implementedin PHP. Use of the PHP Development Tools framework (abbr. PDT [28])for Eclipse, together with XDebug (a PHP extension enabling debugging,see [37]) enabled a comfortable debugging and testing environment.

Another very helpful tool employed was Coder [5], a Drupal module de-signed for assisting code upgrades and code reviews. Upon enabling it pro-vides the possibility to scan a module's �les and presents a list of �ndings:critical errors (i.e. code statements that won't work in the next version),normal warnings (e.g. syntax that violate Drupal's coding standard [12]) orminor faults (e.g. missing whitespace). Each reported item is annotated, ex-plaining why it needs mending, most of them with a link to the appropriatelocation in Drupal's documentation or upgrade guide.

2Drupal often abbreviates minor versions: �5.x� and �6.x� denote the most recent stableversion of version 5 and 6 respectively.

3http://www.origo.ethz.ch/wiki/development#Installing_Origo

http://www.origo.ethz.ch/wiki/development#Installing_Origo

3.2. CHANGES 15

Figure 3.1: Debugging Origo from within Eclipse using PDT plug-in

3.2 Changes

Drupal's upgrade guide for modules4 and themes5 lists 99 changes in betweenversions 5.x and 6.x. Addressing all of these required careful conversion andmanual checking and testing of over 20 000 lines of code6. In this section wewill look at the major changes that required the most work.

3.2.1 New menu system

Drupal's menu system handles not only visible navigation items, i.e. links, butalso maps URL paths to callback functions, including proper access checks.Each module may implement hook_menu to register its own callback func-tions for menu-paths. In Drupal 5 this was somewhat cumbersome: pathcomponents had to be checked with if-clauses, dynamic components (e.g. auser name in the path) passed extracted by a function call. These complex

4http://drupal.org/node/1147745http://drupal.org/node/1324426The number denotes Origo's web front-end code, i.e. Drupal modules, installation

pro�les and themes.




structures were then stored in Drupal's database as a serialized array, whichneeded to be unserialized (which uses lots of memory) upon each call to anuncached page, i.e. menu-paths with dynamic components.

In Drupal 6, the menu system was completely overhauled to reduce thiscomplexity: menu data is kept unserialized in two tables, {menu_links}7

and {menu_router}. The de�nition of entries in hook_menu was greatlysimpli�ed: the de�ning array in hook_menu is now an associative array, withits key being the desired menu-path. To integrate dynamic URL components,loader functions can be de�ned, and addressed by including wildcard loaderarguments into the array's keys. Callbacks for access-checks were split intofunction and function-arguments.

As an example, below the de�nition of the callback for sending a messageto a user.

Listing 3.1: Menu de�nition in Drupal 5

1 if (arg (0) == 'message ' && is_string(arg(1))) {

23 $items [] = array(

4 'title' => t('Send Message to ' . arg (1)),

5 'path' => 'message ',

6 'callback ' => 'origo_home_message_page ',

7 'callback arguments ' => array(arg(1)),

8 'access ' => $user ->uid > 0,

9 'type' => MENU_CALLBACK ,

10 );

1112 }

Line 6 de�nes the callback function that will be executed (with the argumentsdenoted on line 7, here: the component at index 1 of the URL path) whenthe condition on line 1 is ful�lled. The title of the page generated by thatcallback is speci�ed on line 4 (including string concatenation of the dynamiccomponent � the user name to send the message to), and access check forthis menu-path is the inlined condition on line 8: the current user needsto have an user-ID greater then zero, i.e. only signed in users are grantedaccess8.

Here is the same, much more elegant de�nition in newer versions oforigo_home for Drupal 6:

7Surrounding table names with curly braces is special Drupal syntax: such table nameswill get expanded to include possible speci�ed database-pre�xes.

8Anonymous users, i.e. anonymous visitors that did not log in, receive the user-ID 0.

3.2. CHANGES 17

Listing 3.2: Menu de�nition in Drupal 6

1 $items['message /% origo_username '] = array(

2 'title' => 'Send Message to @recipient ',

3 'title arguments ' => array('@recipient ' => 1),

4 'page callback ' => 'origo_home_message_page ',

5 'page arguments ' => array (1),

6 'access callback ' => 'origo_home_existing_user_access ',

7 'type' => MENU_CALLBACK ,

8 );

Here, the menu-path is a key of the associative array $items. As no �titlecallback� is de�ned, it defaults to Drupal's translation function t()9: argu-ments passed contain replacement values, here this will result in �@recipient�being replaced. The number 1 means that the index 1 (i.e. the second partof the menu-path) of the URL will be passed. There another replacementhappens: the �origo_username� part of �%origo_username� tells Drupal topass the second part of the path to function named by the part after "% " plus"_load ". For "%origo_username ", this is "origo_username" plus "_load",origo_username_load(). Therefore by browsing to message/user_foo, thefunction origo_username_load("user_foo") will be called. Here it is:

Listing 3.3: Wildcard loader function in Drupal 6

1 function origo_username_load($username) {

2 if (! is_string($username)) {

3 return FALSE;

4 }

5 return $username;

6 }

We just check for the parameter to be of type string � but much moreelaborate checks could be included here, e.g. escaping special characters,checking the back-end if it is a valid user name or whether the user exists.We can omit such checks as they are not needed, we already do these checkswhen processing the login credentials in the back-end.

Wildcard handling can get much more complicated, e.g. when multiplewildcard arguments are needed, or special wildcards are used (e.g. �%map�), orany of Drupal 6's prede�ned loader functions are used. For detailed documen-tation and examples see Drupal's documentation of the new menu system [7].

9http://api.drupal.org/api/function/t

http://api.drupal.org/api/function/t


3.2.2 FormAPI improvements

Form handling is one of the central parts of Drupal. As a content man-agement system, most of the user-generated data is passed through forms:creating or editing Drupal-nodes, but also login data or search queries, amongother things. The FormAPI allows to interact with the processing work�ow,most importantly through validation- and submit functions. Through theuse of e.g. hook_form_alter arbitrary forms (including Drupal Core forms)can be changed by modules.

In Drupal 6 the FormAPI underwent major changes10, most speci�callyintended to improve the consistency and reliability of the API. While eachindividual change is relatively easy to implement, they a�ect all form pro-cessing code.Below, we list the two most important FormAPI changes a�ecting Origo.

Parameter changes

FormAPI functions were refactored for consistency. This also includes newor changed parameters, some of which are now passed by reference. Af-fected are all form validation and submission functions, hook_form_alter(),form_set_value(), drupal_retrieve_form() and hook_forms().

The $form_state variable

Previous versions of FormAPI used a combination of $form_values, globalvariables, and custom �ags in the form de�nition itself to capture informa-tion about the form's work�ow and its current state during processing. InDrupal 6 a single associative array � $form_state � is passed by referencealong through each stage of form processing. It contains several standardkeys, most importantly $form_state['values'] which contains incoming,sanitized $_POST data.

As an example, we list an excerpt of the issue_tracker's comment-submission function.

Listing 3.4: Submit function in Drupal 5

1 function issue_tracker_comment_submit ($form_id ,

2 $form_values) {

10http://drupal.org/node/144132 gives an overview of all FormAPI changes

3.2. CHANGES 19

34 if ($form_id == 'comment_form ') {

5 $node = node_load($form_values['nid']);

67 if (user_access('edit issue')) {

89 // Add special tags (status , assigned)

10 if ($form_values['taxonomy ']['tags'][2] == '') {

11 $form_values['taxonomy ']['tags'][2] =

$form_values['issuestatus '];

12 }

13 else {

14 $form_values['taxonomy ']['tags'][2] .= ', ' .


15 }

1617 ...

Note the use of a separate parameter for the form's ID, which is checkedon line 4. Also, the $_POST data is now held by the second parameter$form_values, which is read and assigned e.g. on line 10.

Here's the same excerpt in Drupal 6:

Listing 3.5: Submit function in Drupal 6

1 function issue_tracker_comment_submit($form ,

2 &$form_state) {

34 if ($form_state['values ']['form_id '] == 'comment_form ') {

5 $node = node_load($form_state['values ']['nid']);

67 if (user_access('edit issue')) {

89 // Add special tags (status , assigned)

10 if ($form_state['values ']['taxonomy ']['tags'][2] ==

'') {

11 $form_state['values ']['taxonomy ']['tags'][2] =

$form_state['values ']['issuestatus '];

12 }

13 else {

14 $form_state['values ']['taxonomy ']['tags'][2] .= ', '.


15 }

1617 ...

Note the di�erent parameters on line 1 (another change in the FormAPI):


data resides in the associative array $form_state['values'], which alsoholds the form-ID (cf. lines 4 and 11 respectively).

Most FormAPI changes did not require complicated refactoring in Origo'smodules, but although simple, the changes a�ected large areas of the code.

3.2.3 The arguments to url() and l() have changed

Drupal provides these two utility functions for generating and formattinglinks, url() for generic, l() for Drupal-internal links. The arguments tourl() and l() have changed in Drupal 6. Instead of a long line of singlearguments (the exact options and order of which were hard to remember),both of these functions now take an associative array to specify the optionalarguments.

Here's an example how the link function was used in Drupal 5:

Listing 3.6: Link function in Drupal 5

1 $output .= l(

2 '<img src="misc/feed.png" style ="vertical -align: middle ;"

alt="" />',

3 $feedUrl . '/' . $project['id'],

4 NULL ,

5 NULL ,

6 NULL ,

7 TRUE ,

8 TRUE

9 );

Here the current usage in 6.x:

Listing 3.7: Link function in Drupal 6

1 $output .= l(

2 '<img src="misc/feed.png" style ="vertical -align: middle ;"

alt="" />',

3 $feed_url .'/'. $project['id'],

4 array(

5 'absolute ' => TRUE ,

6 'html' => TRUE

7 )

8 );

The �rst two (mandatory) parameters are unchanged, but through the useof the associative array, unneeded optional parameters are omitted.

3.2. CHANGES 21

3.2.4 Simpli�ed �le upload

Uploading a �le in Drupal 5 was a process involving multiple steps: �rstfile_check_upload() veri�ed that the speci�ed resource was indeed a valid�le, and copied it to a temporary location. This returned a Drupal �leobject, which then could be validated (e.g. an image for size-constraints).Afterwards, file_save_upload was called to save it to the speci�ed location.

In Drupal 6, the check-function was merged into file_save_upload. Thefunction can now be passed callback names of validation functions, possiblyDrupal's prede�ned ones. Uploads are granted temporary status, which canbe changed by calling file_set_status().

Here's an excerpt from a �le upload in Origo, uploading a project logo.

Listing 3.8: File upload in Drupal 5

1 function

origo_home_project_settings_logo_form_submit($form_id ,

$form_values) {

23 if ($form_values['op'] == t('Save')) {

45 if ($file = file_check_upload('upload ')) {

6 $result = file_save_upload('upload ',

variable_get('file_directory_path ', 'xyz'), FALSE);

78 if ($result == 0) {

9 form_set_error('upload ', t('Invalid file'));

10 }

11 else {

1213 ...

Note that the image is not validated, this would have gone in between lines5 and 6.

Here's the same �le upload in Drupal 6

Listing 3.9: File upload in Drupal 6

1 function origo_home_project_settings_logo_form_submit($form ,

&$form_state) {

23 if ($form_state['values ']['op'] == t('Save')) {

4 $validators = array(

5 'file_validate_is_image ' => array (),

6 'file_validate_size ' => array (2097152) // 2MB


7 );

89 $result = file_save_upload('upload ', $validators ,

file_directory_path (), FALSE);

10 if ($result == 0) {

11 form_set_error('upload ', t('Invalid file'));

12 }

13 else {

14 file_set_status($result , FILE_STATUS_PERMANENT);

1516 ...

Two validation functions are used: checking that the �le is indeed an image,and that its �le-size does not exceed 2MB. We do not need to check the imagesize, as it will be resized by using a function provided by the image modulelater on.

3.2.5 Porting the theme

Upgrading the theme was for the largest part straightforward, thanks toDrupal's upgrade guide11. Registering the theme's blocks, and renaming ofvariables were quickly done. A bit trickier was to spot di�erences in CSS styleattributes: some elements generated by Origo's modules relied on Drupalsupplied CSS de�nitions, e.g. the pagination numbers of the issue tracker.Drupal removed some (by Drupal Core) unneeded de�nitions, and the designof parts of Origo's modules were slightly o�. This was �xed by tracking downthe correct CSS de�nition, and adding it to Origo's theme CSS �le.

3.2.6 Porting the install pro�le

Drupal installation pro�les are a way to pre-con�gure almost all aspects ofan installation: it lets you de�ne which modules to enable, set con�gura-tion variables, or run arbitrary code, e.g. for database manipulation. Installpro�les can be chosen upon installation of a new site, and enable it to beused immediately after installing � without the need to con�gure modulesby hand.

To this e�ect, it is an essential tool for Origo, as not only a great deal ofcon�guration, but also navigation menus, prede�ned wiki pages (the default

11http://drupal.org/node/132442


3.2. CHANGES 23

front-page, �Screenshots�, �Doc� and �Development�) and user-permissionsare set through the custom Origo pro�le. Furthermore it de�nes the customnode-type �issue�, including issue-tags for adding metadata to them andcreates custom blocks where content can be displayed.

Migrating the installation pro�le to Drupal 6 was largely dominated bysetting all correct variables, because Drupal itself, as well as most contributedmodules had many new or changed settings to be integrated into the pro�le.A much bigger challenge related to the installation pro�le, was how to use itin Drupal 6.

Command line installation

Drupal has very little support for command line operations. One of the fewexceptions is Drupal's cron.php script, which can be used to periodicallytrigger maintenance functions. But even then it has to be executed by aninterpreter (the php binary, or through a webserver using wget or curl).Drush (see 3.3.3 below) recti�es some of the de�ciencies, but there's onearea where even this tool fails (because it depends on a running instance):installing a new Drupal site.

There's only one way to install a Drupal site, be it the very �rst or anadditional multi-site: running install.php, and �lling out the presentedforms. Previous versions of Origo bene�ted from the simplicity of the formpresented by the install site. It was possible to pass all form elements tothe site with a curl call with _POST parameters. The install form used byDrupal 6 though uses a multipage form, i.e. multiple forms after each other.As the forms contain dynamically generated ID �elds, it was not possible tosimply place multiple _POST calls from the command line.

The solution came with �Drupal CLI Utils�[9], a small toolset for per-forming Drupal tasks from the command line. Its ability to install Drupalsites is provided through screen-scraping: it places _POST calls using curl,and parses the received output for generating the call to the next form. Theutility was modi�ed to integrate into Origo's setup, i.e. only the functionalityto install Drupal sites was kept, and adapted to be called from another script(origo-create-project.php).


3.2.7 Refactoring

Besides changes directly mandated by the new Drupal versions, several im-plementations were refactored, mostly for removing hard-coded values (alsosee 3.6). Below are some selected examples.

Install pro�le: Creation of Navigation items

As mentioned in section 3.2.6, Origo uses installation pro�les to pre-con�gurethe project-sites. Among these things, navigation � menu-links (cf 3.2.1) areinserted into the database for the �Home�, �Screenshot�, �Documentation�,�Issue� and �Development� page. Here's how this was done:

Listing 3.10: De�ning custom menu-paths in Drupal 5

1 db_query("INSERT INTO `menu ` (`mid `, `pid `, `path `, `title `,

`description `, `weight `, `type `) VALUES ".

2 "(144, 2, 'download ', 'Download ', 'Download ', -5, 118),".

3 "(145, 2, 'node/5', 'Home ', 'Home ', -7, 118),".

4 "(146, 2, 'node/1', 'Screenshots ', 'Screenshots ', -4,

118),".

5 "(147, 2, 'forum ', 'Forum ', 'Forum ', 1, 118),".

6 "(148, 2, 'node/3', 'Doc ', 'Doc ', 0, 118),".

7 "(149, 2, 'node/4', 'Development ', 'Development ', 7,

118),".

8 "(150, 2, 'issues ', 'Issues ', 'Issues ', 5, 118),".

9 "(151, 2, 'blog ', 'Blog ', 'Blog ', 3, 118),".

10 "(152, 2, 'statistics ', 'Statistics ', 'Statistics ', 5,

118)");

Note that column mid (the primary key of the table), is inserted explicitly.Apparently, it was once determined that a clean Drupal 5 installation de�nes143 menu-paths, and that Origo can claim the next few paths. Of course thisonly works as long as nothing menu-related changes: no changes in DrupalCore, nor in other modules � the menu-path would be overridden, leadingto unpredictable e�ects.

Here's the corresponding, refactored code of the install pro�le currentlyin use:

Listing 3.11: De�ning custom menu-paths in Drupal 6

1 $last_id = db_result(db_query("SELECT COUNT (*) FROM

{menu_links}"));

23 // downloads

3.2. CHANGES 25

4 $last_id ++;

5 $options = 'a:1:{s:10:" attributes ";'.

6 'a:1:{s:5:" title";s:8:" Download ";}}';

7 db_query("INSERT INTO {menu_links} (`menu_name `,

`link_path `, `router_path `, `link_title `, `options `,

`module `, `weight `, `depth `, `customized `, `p1 `) "

8 ."VALUES ('primary -links ', 'download ', 'download ',

'Download ', '%s', 'menu ', -5, 1, 1, %d)", $options ,

$last_id);

9 // home

10 $last_id ++;

11 $options = 'a:1:{s:10:" attributes ";'.

12 a:1:{s:5:"title";s:4:"Home";}}';

13 db_query (" INSERT INTO {menu_links} (`menu_name `,

`link_path `, `router_path `, `link_title `, `options `,

`module `, `weight `, `depth `, `customized `, `p1 `) "

14 ." VALUES ('primary -links ', 'node/5', 'node/%', 'Home',

'%s', 'menu', -7, 1, 1, %d)", $options , $last_id);

1516 ...

As the menu-system changed, the data is inserted into a di�erent table, butinserting the ID is still needed.We �rst retrieve the current number of relations in the table, and increase itbefore inserting the next relation. In this way, the menu-path data is alwaysappended to the end of the table.

Issue tracker: retrieving issue tags

Drupal's taxonomy module [11] provides a way to systematically classify con-tent by assigning keywords to Drupal nodes. These keywords � called tags� are organized by the module into vocabularies, which consist of one ormore terms. Vocabularies group terms that describe an aspect of a node.This allows for separating tags attributing di�erent types of nodes.

Tags are used extensively in Origo's issue tracker. Besides user-de�nedtags, issue-metadata such as the issue's status (open/closed) or due-date arestored as tags. This allows for easy search and �ltering of issues by keywords.The issue tracker's tags are organized into a vocabulary of its own, issue. Inprevious versions of Origo, this vocabulary received the ID 2 (when addingthe vocabulary category in the install pro�le), ID 1 was used for blog tags.In Drupal 6.x forum nodes were given its own vocabulary. As forum as wellas blog are core modules, they receive the IDs 1 and 2, respectively, with the


vocabulary issue receiving ID 3. This revealed that the issue tracker reliedon the hard-coded ID of the vocabulary.

Here's an excerpt from the comment form:

Listing 3.12: Adding issue-status tags in Drupal 5

1 if ($form_values['taxonomy ']['tags'][2] == '') {

2 $form_values['taxonomy ']['tags'][2] =


3 }

4 else {

5 $form_values['taxonomy ']['tags'][2] .= ', ' .


6 }

78 ...

910 taxonomy_node_save($form_values['nid'],

$form_values['taxonomy ']);

Project members can change the status of an issue when submitting a com-ment. The lines shown above assign the issue's status-tag to the node, byreferencing the vocabulary ID directly (by storing the form's data into index2 of the tags array).

Here's the corresponding code excerpt, as currently used in Origo.

Listing 3.13: Adding issue-status tags in Drupal 6

1 $vid = _issue_tracker_get_issue_vocabulary_id ();

23 if ($form_state['values ']['taxonomy ']['tags'][$vid] == '') {

4 $form_state['values ']['taxonomy ']['tags'][$vid] =


5 }

6 else {

7 $form_state['values ']['taxonomy ']['tags'][$vid] .= ', '.


8 }

910 ...

1112 taxonomy_node_save($node , $form_state['values ']['taxonomy ']);

Here, the vocabulary ID is retrieved dynamically by calling the function_issue_tracker_get_issue_vocabulary_id(), as shown below.

Listing 3.14: Utility function for retrieving issue-tags vocabulary IDs

3.2. CHANGES 27

1 function _issue_tracker_get_issue_vocabulary_id () {

2 global $issue_vid;

34 if(! isset($issue_vid)) {

5 $vocab = taxonomy_get_vocabularies('issue ');

6 $issue_vid = key($vocab);

78 return $issue_vid;

9 } else {

10 return $issue_vid;

11 }

12 }

1314 global $issue_vid;

Issue tracker: retrieving the content's �lter

Each Drupal node can be assigned multiple filters, modules that processtext, e.g. for HTML tags. The issue tracker module processes all nodes usingmediawiki_filter, allowing users to input MediaWiki format. To assign a�lter to a node, its ID has to be added upon creating the node. Quite similarto retrieving issue tags, this �lter ID was hardcoded to a speci�c number.To eliminate the risk of confounding di�erent �lters (e.g. when a standard�lter added by Drupal Core receives the hardcoded value), �lter IDs are nowretrieved by calling a special utility function, as listed below.

Listing 3.15: Utility function for retrieving MediaWiki's �lter ID

1 function _issue_tracker_get_filter_format_id () {

2 global $issue_filter_format_id;

34 if(! isset($issue_vid)) {

5 $list = filter_formats ();

6 foreach ($list as $filter) {

7 if ($filter ->name == "Mediawiki format") {

8 return $filter ->format;

9 }

10 }

11 return 0;

12 }

13 else {

14 return $issue_filter_format_id;

15 }

16 }

1718 global $issue_filter_format_id;


3.3 Other Changes to the Front-End

Part of this thesis' tasks included regular maintenance and user-support ofthe Origo system at ETH. In this function, other improvements besides themain topics of this thesis have been done: many bugs were �xed, new func-tionality added and new Drupal modules introduces. Some of these wererequested by end-users or suggested by Origo team members, including theauthor.

3.3.1 Hook extension for install pro�le

Maintainers of an Origo instance, e.g. for a company internal system, mightwant to customize Origo. A simple hook system (inspired by Drupal's thor-ough use of hook functions) was added for Origo's install pro�le to aid in thistask. Selected functions in the regular pro�le origo.profile are extendedin such a way, that �les placed into a special folder may contain hooks tobe executed. This allows us to split inclusion and con�guration of modulesinto separate �les. Only by placing them into the special folder they will beinstalled and con�gured.

This extension is currently used for installing and con�guring Origo'smodules google_cse and googleanalytics. Listing 3.16 shows the imple-mentation of the hook system.

Listing 3.16: Hook extension to origo.profile

1 define('ORIGO_PROFILE_INCLUDE_DIR ', dirname(__FILE__)

.'/include ');

23 // include all additional files

4 foreach(glob("" . ORIGO_PROFILE_INCLUDE_DIR . "/*. inc") as

$inc_file) {

5 include_once $inc_file;

6 }

78 ...

910 function origo_profile_modules () {

11 $list = array(

12 'block', 'filter ', 'node', 'system ', 'user', ...

13 );

1415 // get additional modules

16 $additional_modules = array ();

3.3. OTHER CHANGES TO THE FRONT-END 29

17 foreach (_origo_profile_include_list () as $inc) {

18 $function = $inc .'_modules ';

19 if (function_exists($function)) {

20 $additional_list = $function ();

21 $additional_modules =

22 array_merge($additional_modules , $additional_list);

23 }

24 }

25 // merge with module list defined here

26 return array_merge($list , $additional_modules);

27 }

282930 function _origo_profile_include_list () {

31 $list = array ();

3233 //get all profile -include files

34 $path = ORIGO_PROFILE_INCLUDE_DIR . "/*. inc";

35 $full_path = glob($path);

3637 foreach($full_path as $inc_file) {

38 // extract filename & cut off ".inc"

39 $name = basename($inc_file , ".inc");

4041 // replace dots with underscores

42 $replaced = preg_replace('/\./', '_', $name);

43 $list [] = $replaced;

44 }

45 return $list;

46 }

Function origo_profile_modules de�nes which modules to enable wheninstalling a site using Origo's install pro�le. Regular modules are listed online 12 (only the �rst few shown), while the hook-system calls correspond-ing hook-functions on line 16�. Function origo_profile_eth_modules()

de�ned in include/origo.profile.eth.inc is called, and returns an addi-tional list of modules to enable.

3.3.2 Module origo_global

Drupal provides no support for content spanning a multi-site installation.With all content being stored as Drupal nodes inside each site's database,Origo's projects can't access content common to all of them; such content hasto be replicated and stored within each project. This can be an annoyancefor read-only data, e.g. FAQ pages. While disk-space is hardly a problem,


maintenance of such common content is unnessecary complicated: a singleupdate to a FAQ page needs to be replicated into all Origo projects to avoidstale information.

The new module origo_global provides a way to serve content from acentral location to all sites simultaneously. It is aimed for Origo administra-tors to de�ne a content to be shared, and assign a menu-path to access it. Itis currently used for serving Origo's �Terms of Service� page.To be database-independent, origo_global retrieves its data from �les. It isrendered with Origo's standard mediawiki_filter, which enables the �lesto be written with the same syntax used elsewhere in Origo's front-end.

The module is in its early stages, and currently can't be con�guredthrough an interface. The �les to read from disk, and the menu-path as-signed to their contents is hard-coded in the module's code.

3.3.3 Proper database updates � with Drush

Drupal provides an API function for updating a module's database infor-mation. hook_update_N12 (where N is a number, sequentially increased foreach update) functions of all modules are called when updating a Drupalinstallation through update.php. Unfortunately, the form in update.php

is not designed for automated submission: it contains drop-down �elds ofeach module's update-number, and thus � just like installing a site (seesection 3.2.6) � can't be pre-selected using $_POST calls. And automateddeployment is essential for Origo's thousands of Drupal sites.

Updates of modules typically involve very few database updates, oftenonly one or two hook_update functions. Previous versions of Origo extractedall involved SQL statements, either by dissecting the update-functions, or bycomparing database-dumps before and after an executed update. The state-ments were then added to a special update-script, update_database.php13,which executed them on all of Origo's project databases.This process, while complicated and tedious, worked for small incrementalupdates, such as patches or updates in minor-versions of modules. But usingit for upgrading Drupal 5.x to 6.x is a di�erent kind of beast: there were hun-dreds of updates, with data being moved in between tables, obsolete tables

12http://api.drupal.org/api/function/hook_update_N/613Visit http://svn.origo.ethz.ch/viewvc/origo/web/tags/20090419/

development/data_management/update_database.php?view=markup to view the�le.

http://api.drupal.org/api/function/hook_update_N/6

http://svn.origo.ethz.ch/viewvc/origo/web/tags/20090419/development/data_management/update_database.php?view=markup

http://svn.origo.ethz.ch/viewvc/origo/web/tags/20090419/development/data_management/update_database.php?view=markup

3.3. OTHER CHANGES TO THE FRONT-END 31

removed, new ones added, etc. Extrapolating the necessary SQL statementscovering all possible cases was not feasible.

The solution came with the discovery of the wonderful Drush tool [13], anabbreviation of �Drupal Shell�. It is a command line shell and Unix scriptinginterface for Drupal, and as such perfect for Origo's needs. It provides accessto a great many functions of Drupal from the command-line, such as

cron Run all cron hooks.

updatedb Execute the update.php process from the command line.

cache clear Clear all caches.

en-/disable Enable/Disable one or more modules.

dl Download core Drupal or contributed modules - this will always downloadthe most recent stable version.

sql cli Open a SQL command-line interface using Drupal's credentials. Thiscan also be used to execute SQL scripts.

As mentioned in section 3.2.6, Drush needs an existing Drupal site tooperate on, and as such can't be used to install a fresh site. The reason forthis is that Drush extracts runtime information from a site's settings.php�le. For example, it uses each project's database-connection to execute SQLstatements on the correct database, logging in as the user assigned to thesite, and detects the current version of Drupal (as it provides di�erent func-tionality for di�erent versions of Drupal).

By executing the update-script from the command-line, we can use theregular hook_update_N functions as de�ned by each module's maintainers.The additional possibility of running arbitrary SQL statements on a project'sdatabase makes Drush the ideal choice for updates of the Origo platform.Figure 3.17 shows an excerpt from the upgrade script used to upgrade Origo'sDrupal sites from 5.x to 6.x.

Listing 3.17: Excerpt from the shell script upgrading Drupal to version 6.x

1 find ${DRUPAL_ROOT }/ sites -maxdepth 1 -type l -printf '%f\n'

| while read FILENAME;

2 do

3 #add new views tables. this is done first , so no warnings

will be printed later on during the db-update.

4 drush $DRUSH_OPTS -l http:// $FILENAME sql cli <

drupal_views_structure.sql


56 #add geshi variables


drupal_geshi_dump.sql

89 # update the database (to drupal6)

10 drush $DRUSH_OPTS -l http:// $FILENAME updatedb

1112 #disable modules openid , php (filter) and tql

13 drush $DRUSH_OPTS -y -l http:// $FILENAME disable openid php

tql

1415 #add new views tables. this is done first , so no warnings

will be printed later on during the db-update.


drupal_views_dump.sql

1718 # run cron

19 drush $DRUSH_OPTS -l http:// $FILENAME cron

2021 # update the database (for modules)

22 drush $DRUSH_OPTS -l http:// $FILENAME updatedb

2324 # insert additional data (variables , permissions , etc)


drupal_additional.sql

2627 # drop old , unused tables


drupal_table_drops.sql

2930 # clear cache , so various new settings are rendered

properly

31 drush $DRUSH_OPTS -l http:// $FILENAME cache clear

32 done

The while loop on line 1 iterates over all Origo projects, executing the sameupgrade steps for all of them. The updatedb statement on line 22 triggersthe main update procedure, converting all data to the new version.

3.3.4 Project access via subdirectory

Each project hosted on Origo can be accessed by its own subdomain, for ex-ample myproject.origo.ethz.ch. While this is a convenient way that helpsremembering one's project's URL, this requires a wildcard DNS record [42]

myproject.origo.ethz.ch

3.4. TESTING 33

to route requests to Origo's server. This setup might not be feasible for allstakeholders, and might even deter from using Origo in the �rst place.

Drupal also supports accessing di�erent instances using multiple subdi-rectories [10]. To use this feature with Origo, several functions in both thefront- and in the back-end, as well as in shell-scripts (e.g. project-creation)had to be adapted. A new con�guration variable was added to Origo's con�g�le to control e.g. generation of links in work items. This variable is accessedfrom both Origo nodes and various shell-scripts. In the front-end modules,subdirectory or subdomain project access is controlled automatically, by an-alyzing the referring URL.

The possibility of having a single URL for all projects of an Origo-instanceis very attractive, as the number of needed URL's has now an upper bound(there's still the need for subdomains for e.g. subversion- and ftp-servers orrelease-hosting).

3.4 Testing

Unfortunately, Origo doesn't have a test-suite for the web front-end (cf. sec-tion 5.2.2). Testing the new Origo front-end building on Drupal 6.x wastherefore done manually. As outlined in section 3.1, the development setupwas capable to run both the new and the old versions simultaneously. Test-ing was thus done by navigation to every possible page (also using the menu-paths for checking that all pages were visited), and comparing the screenoutput of both versions. Additionally, both Origo Team members and unre-lated persons were given access to a development version for testing.

While not perfect, all but very minor errors were discovered and could be�xed before deployment.

3.5 Case Study

One of Origo's users indicated to the Origo team that Drupal provides asection of its web portal for showcasing projects that use Drupal, and thatthought Origo to be a good addition.

The team agreed that presenting Origo and its use of the CMS to theDrupal community was a good way to improve discussions and collaboration


with the Drupal community, as well as promoting Origo to a wider audience.This thesis' work on Origo's front-end was thus concluded by writing a casestudy of Origo, with an emphasis on its use of Drupal. It was posted onDrupal's message board [27], and can be found in appendix A. It was featuredon drupal.org's front-page on August 18th, 2009.

3.6 Summary

Upgrading Origo's front-end to the most recent Drupal version was longoverdue, especially with Drupal 7 on the horizon (the code freeze for newfeatures is on September 1st, 2009). While Drupal does provide good up-grade instructions, the process was unfortunately not that simple for Origo:there are some areas where Origo circumvents Drupal's API, sometimes un-wittingly, sometimes out of necessity. Furthermore, hard-coded values (thatwere nowhere mentioned) do generally not �t into any upgrade scheme. It isthose areas where the upgrade process turns out to be the most di�cult, asthis is not intended by Drupal's developers, and needs to be resolved manu-ally.On the plus side, the now refactored code should prove to be much easier toupgrade to future versions of Drupal.

Upgrading the whole front-end had the added bene�t of not only gettingan overview, but an in-depth knowledge of all of Origo's modules, and theiruse within Drupal. This allowed bug-�xing, as well as continued developmentof various Origo's modules.

Chapter 4

Aranea Message Transport

Aranea enables communication by providing a messaging layer. It interfaceswith Apache ActiveMQ (abbr. AMQ, [2]) by using ActiveMQ-CPP [1], an im-plementation of CMS, a C++ API to Java Message Service. The C++ libraryis wrapped for use in Ei�el by Ei�el Messaging Service (abbr. EMS, [15]). Amessage broker server is running in on the network: ActiveMQ. Figure 4.1depicts a schematic of the libraries involved.

ActiveMQ-CPP provides di�erent kinds of messages, three of which areavailable in Ei�el through EMS:

MAP_MESSAGE A message containing arbitrary many payload key-value pairs, where the keys are strings, and the values Ei�el basic typesor strings.

TEXT_MESSAGE A message containing a single payload string, the sizeof which can be up to a theoretical limit of 2GB (although messagesabove a few MB are discouraged).

BYTES_MESSAGE A message holding its payload data in a single arrayof bytes.

Additionally, each message can have metadata �elds, referred to as properties(see section 4.2.1).

The task at hand is: how to map arbitrary Aranea-messages to EMS-Messages? And how to do it in a transparent way? Being a middlewarelibrary, Aranea provides an abstraction layer between its clients and EMS/

35

36 CHAPTER 4. ARANEA MESSAGE TRANSPORT

Figure 4.1: Libraries involved when connecting Aranea to ActiveMQ

AMQ, therefore the mapping of Aranea Messages (see section 4.2 for a de-tailed explanation of this term) to EMS message-types should be fully trans-parent.

Past versions of Aranea used a combination of MAP_MESSAGEs andBYTES_MESSAGEs: simple messages consisting of basic types and stringswere transformed into MAP_MESSAGEs, more complex (i.e. containing ar-bitrary objects), or composite (e.g. ARRAYs) objects were serialized into astream of bytes by using SED_INDEPENDENT_SERIALIZER1 (and forthe reverse operation its counterpart SED_SESSION_DESERIALIZER).While using Ei�el Base's serializer classes is fully transparent to Aranea'sclients (because it handles arbitrary complex Ei�el data structures), it re-quires all Aranea-enabled programs to be written in Ei�el, as no other lan-guage can handle Ei�el Base's serialization protocol.

This chapter details this thesis' main task: opening up many new pos-sibilities for Origo by exchanging Aranea's serialization protocol to a cross-platform, language-independent one. Section 4.1 details why JSON [19] waschosen as the designated protocol. Section 4.2 takes a look at Aranea Mes-sages, their components and transmission. Section 4.3 explains how to designand specify Aranea messages, and the process of creating message-classes us-

1http://docs.eiffel.com/static/libraries/base/sed_independent_

serializer_flatshort.html

http://docs.eiffel.com/static/libraries/base/sed_independent_serializer_flatshort.html

http://docs.eiffel.com/static/libraries/base/sed_independent_serializer_flatshort.html

4.1. SERIALIZATION PROTOCOLS 37

ing oAW. Section 4.4 will show the new serialization mechanism in detail.Section 4.5 shows how the new framework is used when creating messages,and retrieving values from them.

4.1 Serialization Protocols

Many di�erent serialization protocols have been developed, to accommodatevarious needs. When evaluating a replacement protocol for Aranea, the �rststep was to gather the team's requirements for Aranea:

Speed Aranea, particularly as it's being used in Origo, should support large-scale systems. A protocol sporting fast serialization and deserializationof payload data is essential.

Size of Wire Format The more data, the longer the transmission, and alsode-/serialization time. A good trade-o� of size and speed is needed.

Cross-Language We want a protocol that works across di�erent languages.This is key reason to switch away from Ei�el-serialization.

Ease of Use The protocol should be easy to integrate and handle in Aranea.

Documentation A proper, extensive documentation is essential, as mostlikely the protocol will be ported to Ei�el (and possibly other lan-guages).

Protocol Versioning Does the protocol inherently support versioning?

Open Source We only want to build upon open-source software.

Spread /Acceptance The protocol should be used by a wide audience: aniche protocol hinders development, particularly across many di�erentlanguages.

Ei�el Binding: Work�ow and Code Generation The protocols Ei�elbinding should seamlessly integrate with Aranea's message generationwork�ow (see section 4.3).

Supported Languages, Community There should be a broad list of lan-guages that support the protocol, as well as a large, active community.


From the numerous of cross-language protocols, the following were lookedat:

• Protocol Bu�ers ([30], also known as protobuf ), a binary protocol de-veloped and used by Google throughout all their applications

• Thrift [34], developed and used by Facebook [17], uses binary wireformat

• JSON ([19], JavaScript Object Notation), �rst became popular withAJAX calls, but is since used in many other places as well

• Etch [16], a framework and protocol developed by Cisco

• Hessian [18], a protocol �rst conceived as a binary replacement of SOAP

• CORBA /GIOP [6] GIOP (General Inter-ORB Protocol) is the messageformat used in CORBA

• XML-RPC, SOAP and YAML: three XML based protocols

Of these, Hessian and Etch were immediately discarded, as there's hardlyany information available about them. Furthermore, Hessian's communityactivity seems non-existent (for an example: the mean number of topics onHessian's mailing-list2 in the �rst half of 2009 is 6.5). XML-based protocolswere discarded, too, as they were found to be quite resource-intensive toboth serialize and deserialize. GIOP was eliminated as being to complex tointegrate in the existing Aranea setup.

The remaining protocols � Protocol Bu�ers, Thrift and JSON � werefound to be largely similar in comparison: all of them are open-source, havenumerous language-bindings, good documentation and support largely thesame data types. While Protocol Bu�ers and Thrift were the fastest protocolswith the smallest message size, JSON was found to be only marginally slower,due to its low overhead for data structures and general simplicity. Table 4.1gives a comparison of the three, and YAML as an example of an XML basedprotocol. Figure 4.2 shows a benchmark comparison of various protocols,from a test suite implemented in Java.

A discussion of this comparison led the Origo Team to choose JSON asthe designated serialization protocol. Despite it being slightly slower thanProtocol Bu�ers or Thrift, its simplicity � and therefore the integration into

2http://maillist.caucho.com/pipermail/hessian-interest

http://maillist.caucho.com/pipermail/hessian-interest

4.1. SERIALIZATION PROTOCOLS 39

Protocol Time in seconds Size in bytes

Serialization

thrift 0.560 370009protobuf 4.850 171650json 0.080 680680yaml 17.330 610680

Serialization and compression

thrift 0.620 111326protobuf 3.980 98571json 0.110 124919yaml 17.160 121065

Serialization and Deserialization

thrift 2.130protobuf 7.550json 0.130yaml 56.740

Table 4.1: Comparison of di�erent serialization protocolsSource: http://bouncybouncy.net/ramblings/posts/json_vs_thrift_and_protocol_buffers_round_2

Figure 4.2: Benchmark of Serialization protocolsSource: http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking

http://bouncybouncy.net/ramblings/posts/json_vs_thrift_and_protocol_buffers_round_2

http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking


Aranea's message-generation work�ow (detailed in 4.3) � was valued higher.Starting with Aranea 1.1, all data transported by Aranea is serialized usingJSON.

JSON was favored as a serialization format, amongst other reasons, be-cause

• it is platform and language independent

• the small metadata overhead makes it a fast protocol (as compared to,e.g. XML)

• bindings in numerous languages exist

• it is a simple format (which makes it easy to implement in languagesthat don't have a binding yet)

• it is well accepted and in use in many applications.

JSON provides us with maximum �exibility with regards to supported tech-nologies, nodes can be written in any language and deployed on any platform- as long as it's possible to read and write JSON data. Aranea supports anduses all of JSON's features, except for the keyword null.

4.2 Aranea Messages

An Aranea message (hereafter simply messages) is an Ei�el object that canbe used to communicate between Aranea enabled nodes. Aranea providesa framework for describing, sending and receiving messages (and also forabstractly describing messages and generating Ei�el code from it, see sec-tion 4.3). This section describes the concept, implementation and usage ofAranea Messages.

Aranea messages and their framework can be categorized into three parts

1. (Generated) Messages that are sent and received between nodesAranea users will typically de�ne messages tailored to their application.Messages are de�ned using Aranea's DSL (Domain Speci�c Language,see section 4.3), and the corresponding Ei�el code is generated by run-ning an oAW work�ow to transform the message speci�cation.

4.2. ARANEA MESSAGES 41

Figure 4.3: Aranea Message Framework

2. Framework classes that provide de-/serialization and message trans-portThese classes provide features for transparent de-/serialization andtransmission support.

3. VALUE classes, the basic building blocksDe-/Serialization requires all class �elds of messages to provide appro-priate features, thus it is required that all messages are composed ofdata types provided by Aranea.

Figure 4.2 shows the general design of the framework: VALUE classesdepicted to the right, regular messages on the bottom level on the left. Theremaining ones belong to the framework.


4.2.1 Sending and receiving

Aranea uses ActiveMQ as message broker, which provides both global andpersonal message queues. Each node connected to a running AMQ instanceopens a personal queue identi�ed by this node's name (command-line argu-ment -n NAME). Messages sent to a personal queue will be handled only bythe corresponding node. Nodes can create and subscribe to global queues,where multiple nodes may access the same queue, and atomically retrieve amessage for processing (once a message is retrieved by a node, it's not avail-able in the queue anymore. AMQ also supports persistent queues, wheremessages are only removed by explicitly requesting it, but this is not usedby either Aranea, nor Origo).

As listed at the beginning of this chapter, EMS provides several of AMQ'smessage types. As we us JSON serialization, we only need a single string

payload �eld, and thus only use TEXT_MESSAGEs. Besides the payloadmessage �eld, Aranea uses the following properties:

destination The queue where an Aranea message is sent to: either a globalor a private message queue.

reply handler /reply_destination (Optional) Information about whichfeature to call, or destination queue to send to, when returning a resultmessage.

ID Each message automatically receives an ID of the form914d1a9a-726f-fb28-285e-b97c6569b6b3:0:0:2.

namespace The namespace of the message, according to the application'sstructurization of its DSL �le.

(message) type A unique identi�er generated from the message's name asspeci�ed in the application's DSL �le.

All of this metadata � with exception of the message destination � is han-dled automatically by Aranea and EMS, a user of Aranea will never encounterthese.

Sending a Message

A user of Aranea initiates the send process by calling


send_message (a_message: A_MESSAGE; a_destination: STRING)

(or another send_foo feature of A_NODE) with the payload messageand desired destination. Aranea's feature to interface with EMS/AMQ issend_to_destination. It will call a_message.new_ems_message, a generalpurpose feature that creates an EMS message. As Aranea currently only usesTEXT_MESSAGEs, the call will be delegated to new_ems_text_message,which creates the EMS message object, serializes and assigns the payloaddata, and sets the message namespace and type. Right after the call returnsto send_to_destination, EMS metadata (the message id) is set, and themessage inserted into the message_producer queue, which thread-safely willcall external C++ code (CMS, ApacheMQ's C++ interface).

Listing 4.1: Feature new_ems_text_message of class A_MESSAGE

1 new_ems_text_message (a_session: SESSION): TEXT_MESSAGE

2 -- Create a map message to transfer this message.

3 local

4 json_string: !STRING

5 do

6 json_string := serialize

78 Result := a_session.create_text_message (json_string)

910 Result.set_string_property (namespace_element ,

namespace)

11 Result.set_string_property (type_element , type)

12 end

Receiving a Message

Aranea nodes need to register upon startup which messages they want toreceive and process. This is done by calling

1 register_message_handler (a_namespace , a_type: STRING;

a_handler: PROCEDURE [ANY , TUPLE [A_MESSAGE ]])

in A_CORE, where the �rst two parameters are the message's namespaceand type respectively, and a_handler, an agent wrapping the feature thatshould be called upon receiving such a message.

The main message-receiving loop resides in handle_mom_messages. Thisfeature sets up a message-consumer (for receiving messages from the externalC++ layer), and continuously reads from it. Once it receives a message, itcreates an Aranea message object from Aranea's Factory class(es). Using


the message's namespace and id values, A_MESSAGE_FACTORY (or inclient applications one of its descendants) creates the proper message: itcalls the creation feature make_from_ems_message and there the feature tohandle text messages, make_from_ems_text_message (as we currently onlyuse TEXT_MESSAGEs). This feature will start the deserialization, i.e. therecreation of the Aranea message from the JSON encoded data.

Listing 4.2: Feature make_from_ems_text_message of class A_MESSAGE

1 make_from_ems_text_message (a_message: TEXT_MESSAGE)

2 -- Create from an ems TEXT_MESSAGE.

3 local

4 json_string: STRING

5 a_tokenizer: A_JSON_TOKENIZER

6 do

7 json_string := a_message.get_text

89 create a_tokenizer.make (json_string)

10 make_from_tokenizer (a_tokenizer)

11 end

The fully recreated message is then passed by handle_mom_messages tothe feature registered upon startup. There, the message is being checked foran available reply-handler, or the proper feature for the message's namespaceand type is fetched from the message-registry. The found message-handlerfeature is then called.

4.2.2 Convert clauses

Users of Aranea in general do not need to concern themselves with internalsof serialization an deserialization, the only point where they touch upon thematter is when creating messages: they are restricted to use Aranea's valueclasses with messages.

Using Ei�el's convert mechanism, we can mitigate this to a large part.Aranea's value classes provide several convert clauses, with the use of whicha user can specify regular Ei�el types when creating messages, or accessingthe values of message �elds.

Below are the convert clauses of two selected classes.

1 frozen class

2 A_INTEGER_VALUE

34 ...


56 convert

7 make ({ INTEGER_64 }),

8 make_from_integer_32 ({ INTEGER_32 }),

9 value: {INTEGER_64}

1011 ...

1 frozen class

2 A_SEQUENCE_VALUE [G -> A_ARANEA_VALUE create

make_from_tokenizer end]

34 ...

56 convert

7 make ({ DS_ARRAYED_LIST [G]}),

8 make_from_array ({ARRAY [G]}),

9 make_from_linear ({ DS_LINEAR [G]}),

10 sequence: {DS_ARRAYED_LIST [G]}

1112 ...

There is one occasion where automatic conversion fails:Creating A_SEQUENCE_VALUEs using manifest arrays that do not con-sist of A_ARANEA_VALUEs.

1 l_field_array_integer: A_SEQUENCE_VALUE[A_INTEGER_VALUE]

2 ...

3 l_field_array_integer := << 23, 42, 81 >>

Automatic type conversion is not transitive, but in a call like above, the IN-TEGER values would �rst have to be converted to A_INTEGER_VALUEs(through the make_from_integer_32 conversion clause), and the manifestarray afterwards to A_SEQUENCE_VALUE (through the conversion clausemake_from_array)3.

Examples of how to use the conversion clauses can be found in section 4.5

3Ei�el's ECMA standard lists the possibility of explicit type conversion (see [14], sec-tion 8.15.13). A call l_field_array_integer := << A_INTEGER_VALUE[23]>> would explicitly convert the integer values, and automatic conversion of a manifestarray of these A_INTEGER_VALUEs to A_SEQUENCE_VALUE should be possible.Alas, as of Ei�elStudio 6.3, this is still �agged as a syntax error


4.3 Message Generation Using

openArchitectureWare

4.3.1 openArchitectureWare

Open Architecture Ware (oAW, [24]), is a modeling framework in Java, im-plemented as a set of Eclipse plug-ins. Aranea uses oAW for de�ning a customlanguage (DSL, Domain Speci�c Language) that � programming languageindependent � describes arbitrary Aranea messages. oAW supports gener-ating multiple Eclipse plug-ins for the language:

• A lexer/parser for the grammar

• A text editor supporting the language, e.g. by providing code comple-tion and run-time grammar checks

• A code-generator producing source code �les

We currently only generate Ei�el code � the message and related classes� but other languages could be added to the code generator (as of thiswriting, a Java port of Aranea is in its early stages [3]).

4.3.2 Domain speci�c language and its grammar

The oAW-Eclipse project araneaMessage.dsl4 describes the grammar, andproduces Lexer/Parser code, as well as an Eclipse editor plug-in. The projectuses oAW's Xtext [40] plug-in, a framework for development of textual do-main speci�c languages.

The grammar for Aranea's DSL is de�ned by AraneaMessage.xtxt5, us-ing Xtext's simple EBNF grammar language. We give here a slightly simpli-�ed version, for clarity reasons.

Listing 4.3: Grammar of message properties an their type

1 Properties P:

2 name=IDENT ":" T (R)* (description=STRING)? ";"

3

4https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl5https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/

AraneaMesssage.xtxt

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/AraneaMesssage.xtxt

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/AraneaMesssage.xtxt

4.3. MESSAGE GENERATION USING OAW 47

4 Datatype T:

5 string

6 integer

7 double

8 boolean

9 binary

1011 sequence <T>

12 map <T>

1314 record(IDENT , [P (; P)+])

1516 Restrictions R:

17 notVoid

18 notEmpty

19 greaterZero

20 greaterEqualZero

Each property - a �eld in the generated Ei�el class - is de�ned as a key-value pair of the �eld's name and data type.

Binary types can be used for transmitting raw, binary data. It is encodedin Base64 external, and transmitted as a regular string. It maps toEi�el binary data using MANAGED_POINTERs.

Sequence types de�ne an ordered list of objects. It maps to the Ei�el classDS_ARRAYED_LIST.

Map types de�ne an unordered key-value pair. The keys are always of typestring, as JSON external only supports this. It maps to the Ei�el classDS_HASH_TABLE.

Record types can be used to inline message structure de�nitions, withoutexplicitly de�ning a message for them. This is currently the only wayto put complex messages into map and sequence containers, as Araneadoes not support referencing other messages as data types (e.g. usingregular messages in composite types like map<MyOtherMessage> is notpossible).

Note that the data types sequence, map and record contain other data typesof arbitrary recursion depth.

To assign properties to a message, and in turn put the message into anamespace within your project, the following framing structure has to beerected:


Listing 4.4: Grammar of Aranea namespace and message

1 Imports:

2 "[ import" model=URI "]";

34 Project:

5 [" project" name=IDENT "prefix ="IDENT (" ignore ")?]

6 {

7 Namespace*

8 }

910 Namespace:

11 [" namespace" name=ID (" ignore ")?]

12 {

13 Message*

14 };

1516 Message:

17 [message" name=ID (" extends ="[ Message ])? (" ignore ")?

(" nested ")?]

18 {

19 (properties=

20 {

21 Property*

22 })?

23 };

• import statements will import de�nitions from another DSL �le, thatyou can extend. You'll probably want to include Aranea's DSL �le, toget access to the session-message.

• Project pre�x will add a pre�x name to all the generated classes ofyour de�nition.

• Optional ignore statements will have di�erent e�ect: ignoring a

project: this will not create any classes for your de�nition at all

namespace: this will only generate a factory class, but no messagesand no constant-class

message: while constants for this message are generated, the message-class itself will not

Ignore-clauses can be used to implement special classes that need to gobeyond the features o�ered by Aranea. Note that you can still reference�ignored� messages, namespaces and projects in other de�nitions. This


gives you the �exibility to de�ne your own special message-class, whilestill being able to integrate it into Aranea's message-framework.

• The extends clause lets you chose an ancestor for the message. Thisenables you to build inheritance between messages: the generated classwill logically extend (in Ei�el: inherit from) the speci�ed message.

Besides processing the DSL description, Xtext provides several other fea-tures to enhance the development process, and to keep the grammar speci�-cation simple.

Extensions

The �le Extensions.ext6 de�nes functions that one can call from the code-generator, that extend the basic model as de�ned by the grammar. It iswritten in a language called Xtend [39].

We use extensions for instance for de�ning the following function to re-trieve all properties of a message - including the ones from extended classes!

Listing 4.5: An extension function for Aranea's DSL's custom editor plug-in

1 List[Property] allProperties(Message m):

2 let list = {} :

3 m.extends == null ? list.addAll(m.properties) :

list.addAll(

allProperties(m.extends)).addAll(m.properties);

• The �rst line creates an empty list. This will be �lled and returned (asit's the last accessed object) on the second line.

• The second line recursively adds ancestor properties to the list. Notethat extends and properties are objects de�ned by the grammar!

In this example you can also see the somewhat special way arguments areused:

• The �rst argument of a function is the object-type that the functioncan be called upon - in this case, on a message.

6https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/

ethz/origo/aranea/message/Extensions.ext

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/ethz/origo/aranea/message/Extensions.ext

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/ethz/origo/aranea/message/Extensions.ext


• You can call the function by supplying it as the �rst argument, butonly if it's the only one: allProperties(m) and m.allProperties()

are equal calls!

• The second argument of the de�nition is the �rst one supplied in aregular function call.

Checks

The �le Checks.ext7 de�nes assertions that the parser as well as the editor(i.e. at runtime) validate. We use these checks in various ways, e.g. to boundthe combinations of data types and restrictions to a valid set (note thatAranea's grammar allows any kind of restriction on any data type). Here'spart of the code for this:

Listing 4.6: A check function for Aranea's DSL's custom editor plug-in

1 context Property ERROR "Restriction \" NotVoid \" not allowed

DataType \" integer \"":

2 !(this.type.name == "integer" &&

this.restrictions.exists(e|e.name == "notVoid "));

Obviously it doesn't make sense to de�ne a notVoid restriction on an integer.

Checks are written using a language called Check [4]: the keyword contextand the following object de�ne the scope of the check, and the keyword this

within the check's code refers to an object of this type. In the example above:the check will be validated on Property objects, this.type.name is the nameof the current property's data type.

4.3.3 Code generation

Aranea's code-generator is de�ned within a separate oAW Eclipse project,araneaMessage.dsl.generator. The generator is executed by user-de�ned�message-projects�, such as Aranea's own project AraneaMessageProject8,and is passed a DSL �le for processing. The main �le implementing the codegenerator is Main.xpt9. It is written in a language called XPand [38].

7https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/

ethz/origo/aranea/message/Checks.chk8https://svn.origo.ethz.ch/aranea/trunk/dev/AraneaMessageproject9https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl.

generator/src/ch/ethz/origo/aranea/message/Main.xpt

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/ethz/origo/aranea/message/Checks.chk

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl/src/ch/ethz/origo/aranea/message/Checks.chk

https://svn.origo.ethz.ch/aranea/trunk/dev/AraneaMessageproject

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl.generator/src/ch/ethz/origo/aranea/message/Main.xpt

https://svn.origo.ethz.ch/aranea/trunk/dev/araneaMessage.dsl.generator/src/ch/ethz/origo/aranea/message/Main.xpt


We present here an overview of Xpand and its use in Aranea's codegenerator.

XPand

Control Statements All of Xpand's control statements (imports, loops,variable de�nitions, etc.) are enclosed in guillemets: � and �.

De�ne Blocks An xpt �le is composed of multiple DEFINE blocks, alsocalled a template. They're always de�ned relative to a metamodel class (i.e.the FOR part)

Listing 4.7: A DEFINE block in XTend/oAW

1 � DEFINE namespaceTemplate FOR Namespace�

2 �EXPAND constantTemplate FOR this�

3 ...

4 �ENDDEFINE�

56 �DEFINE constantTemplate FOR Namespace�

7 ...

8 �ENDDEFINE�

As you can see, both of these blocks are de�ned relative to Namespace, i.e.the keyword this in both blocks refers to a namespace object.

Expand Blocks The EXPAND statement "expands" another DEFINE block(in a separate variable context), inserts its output at the current location andcontinues with the next statement. This is similar in concept to a subroutinecall.

Listing 4.8: A EXPAND block in XTend/oAW

1 �EXPAND definitionName [( parameterList)]

2 [FOR expression | FOREACH expression [SEPARATOR

expression] ]�

A detailed explanation of this syntax can be found in XPand's documenta-tion10.

10http://www.openarchitectureware.org/pub/documentation/4.3.1/html/

contents/core_reference.html

http://www.openarchitectureware.org/pub/documentation/4.3.1/html/contents/core_reference.html

http://www.openarchitectureware.org/pub/documentation/4.3.1/html/contents/core_reference.html


File Blocks File blocks write their contents (sans control-code) into thespeci�ed �le, i.e. everything not enclosed in � �will be written to a �le - verba-tim. Dynamically generated code comes from interspersed control statementsthat produce appropriate strings within their context.

Whitespace Suppressing Normally even the space taken up by controlstatements will be written to a �le verbatim:

1 �FILE "foobar.file"�

2 line 1

3 �REM�Yes , that 's a comment�ENDREM�

4 �REM�Anyone else reminded of good ol ' DOS times?�ENDREM�

5 line 2

6 �ENDFILE�

will produce the �le foobar.file with the contents

1 line 1

234 line 2

Note the extra empty lines.

To suppress this generation of newlines, add a minus sign can be addedbefore the closing guillemets:

Listing 4.9: Suppressing whitespace output in XTend/oAW1 �FILE "foobar.file"�

2 line 1

3 �REM�Yes , that 's a comment�ENDREM -�

4 �REM�Anyone else reminded of good ol ' DOS times?�ENDREM -�

5 % line 2

6 �ENDFILE�

will then generate � as one normally would want � the following output

1 line 1

2 line 2

The code-generator's code

The code-generator, Main.xpt, has to produce several types of classes froma DSL speci�cation:


Factory Class There is only one factory class per project, e.g. Aranea'sA_MESSAGE_FACTORY. It holds a register of all messages' names-pace and type, and an the corresponding feature that creates a message-object of the appropriate type.

Constant Class Each namespace contains a class de�ning string constantsfor the namespace's name, and the types of messages in the namespace.

Message Class The message class, objects of which will be transmitted inbetween nodes.

Message-Handler Class Each message has a corresponding handler class.When inheriting from the handler class, a double-dispatch pattern toprocess incoming messages can be implemented.

Main.xpt processes the parsed DSL �le by iterating over its XPand state-ments. Without detailing all of Main.xpt's statements, we here give twoselected examples.

Processing a namespace Here's the code excerpt that processes names-paces:

Listing 4.10: Namespace template in Main.xpt

1 �DEFINE namespaceTemplate FOR Namespace�

2 �EXPAND constantTemplate FOR this�

3 �FOREACH messages AS message -�

4 �IF !message.ignore -�

5 �EXPAND messageTemplate FOR message -�

6 �EXPAND handlerTemplate FOR message -�

7 �ENDIF -�

8 �ENDFOREACH -�

9 �ENDDEFINE�

The EXPAND statement on line 2 calls another template, generating theconstant class. It is passed this, i.e. the namespace object, as parameter.The loop on line 3 through 8 processes the namespace's messages. Eachis checked for its ignore �ag (line 4), and if appropriate, the template forcreating the message- and handler-class is called, giving the message-objectas parameter (lines 5 and 6 respectively).


Generating the serialization feature This excerpt is part of the messagetemplate. It is part of a FILE block, i.e. statements not enclosed in guillemetswill be written to a �le; in this case the message class �le.

Listing 4.11: Generating feature serialization_list

1 serialization_list: !STRING

2 -- Lists the fields to serialize , and under which

name

3 do

4 �IF extends != null -�

5 Result := precursor

67 if Result.is_empty then

8 create Result.make_empty

9 �IF properties.size > 0-�

10 else

11 Result.append (",")

12 �ENDIF -�

13 end

14 check Result /= Void end

15 �ELSE -�


17 �ENDIF -�

1819 �FOREACH properties AS property -�

20 �IF property.canBeVoid ()-�

21 if (�property.name.toLowerCase ()� /= Void) then

22 Result.append ("%"" +

�property.name.toLowerCase () �_argument +

"%":")

23 Result.append

(�property.name.toLowerCase ()�.serialize)

24 �IF property != properties.last() -�


26 �ENDIF -�

27 end

2829 �ELSE -�

30 Result.append ("%"" + �property.name.toLowerCase ()

�_argument + "%":")

31 Result.append

(�property.name.toLowerCase ()�.serialize)

32 �IF property != properties.last() -�


34 �ENDIF -�

3536 �ENDIF -�

37 �ENDFOREACH -�

4.4. SERIALIZATION AND DESERIALIZATION 55

38 end

The conditional on line 4 through 17 deals with the initialization of the Re-sult object: if the current message extends another one, we generate a callto the Precursor (line 5), otherwise we initialize to an empty string (line16). The else-branch on line 11 adds a comma-character to the JSON re-sult string, as the current message de�nes properties of its own (note that amessage extending another one does not necessarily have to de�ne additionalmessage-�elds).The loop on line 19 through 37 generates the serialization of all propertiesof the current message. The if-branch on line 21 through 28 deals with�elds that lack a notVoid restriction: these �elds shall only be added to theJSON serialization string if they are /= void at runtime, i.e. attached to areference. Otherwise, the code is the same as in the else-branch: add the�eld-name (as string) as the key of a JSON key-value pair, and also the sep-arating colon (line 22 and 30). Afterwards, emit the code to recursively callthe �eld's serialize feature (line 23 and 31). If there are more propertiesof the message to be processed, emit code to append a comma (the JSONseparator of key-value pairs) to Result, as further �elds will be added. Closethe feature's body by emitting the �nal end (line 38).

The code emitted by the code-generator here will be discussed again fromthe viewpoint of serialization itself, see listing 4.15 in section 4.4.2

4.4 Serialization and Deserialization

The key to Aranea's JSON-serialization lies in the provided library for base-and composite types, and the serialization-framework for generated classes.All data to be transmitted must be held in message �elds of the framework'stype A_ARANEA_VALUE or one of its descendants, which provides thetwo (deferred) key features:

• make_from_tokenizer (a_tokenizer: A_JSON_TOKENIZER), and

• serialize: !STRING

A_ARANEA_VALUE is ancestor class to all of Aranea's message-relatedclasses (recall �gure 4.2). This allows for a simple, recursive descendant se-rialization and deserialization mechanism. Using Ei�el's support of databinding and constrained genericity, both serialization and deserialization are


Primitive Types Wrapped Type

A_INTEGER_VALUE INTEGER_64A_BOOLEAN_VALUE BOOLEANA_DOUBLE_VALUE DOUBLEA_STRING_VALUE STRINGA_BINARY_VALUE MANGED_POINTER

Composite types

A_SEQUENCE_VALUE [G -> A_ARANEA_VALUEcreate make_from_tokenizer end]

DS_ARRAYED_LIST

A_MAP_VALUE [G -> A_ARANEA_VALUEcreate make_from_tokenizer end]

DS_HASH_TABLE

Table 4.2: Aranea's value types and corresponding Ei�el types

implemented statically typed: we use neither assignment attempts, nor ob-ject tests.

Diving further into serialization and deserialization, the reader should besomewhat familiar with the JSON format. Appendix C gives an overview ofthe format.

4.4.1 Value classes

Value classes are the basic building blocks of Aranea messages. They wrapfundamental data types in classes suitable for de-/serialization. We dis-tinguish primitive and composite types, table 4.2 lists them and gives thecorresponding Ei�el types.

As JSON has no native support for binary data, Aranea provides theclass A_BINARY_VALUE, which encodes binary data (pointed to by aMANAGED_POINTER) in Base64 and transmits it as a string. Note thatkeys of A_MAP_VALUE are always A_STRING_VALUEs, as JSON onlysupports mappings to string keys.

Composite types make use of Ei�el's strong support for genericity. Bothmap and sequence types use a generic argument, constrained in the type toA_ARANEA_VALUE, and further constrained in the creation feature to


make_from_tokenizer. This ensures conformance to our de-/serializationmechanism: inheriting from the common ancestor class, we ensure that allvalues can be serialized and recreated by use of the mandated creation fea-ture. Dynamic binding allows us to call these features on arbitrary nesteddata-structures of composite types.

4.4.2 Serialization

Value types

Serialization of value types is easy: each type implements feature serialize(inherited from A_ARANEA_VALUE) in such a way as to return the stringof its JSON representation. To give an example, listing 4.12 shows howA_BOOLEAN_VALUE implements serialize.

Listing 4.12: Serialization of A_BOOLEAN_VALUE

1 serialize: !STRING

2 -- Serializes this message to a JSON string

3 do

4 if value then

5 Result := "true"

6 else

7 Result := "false"

8 end

9 ensure then

10 correctly_serialized: value implies Result.is_equal

("true") and (not value) implies Result.is_equal

("false")

11 end

Composite types utilize dynamic binding to serialize the values of thegeneric objects held by them. Listing 4.13 shows the serialization of se-quences: on line 10 the JSON-serialized representation of one of the se-quence's values is added to the serialization string of the sequence itself.

Listing 4.13: Serialization of A_SEQUENCE_VALUE


2 -- Serializes this message to a JSON string

3 -- The values of the sequence are serialized

recursively.

4 do

5 Result := "["

67 from sequence.start


8 until sequence.after

9 loop

10 Result.append

(sequence.item_for_iteration.serialize)

1112 -- Chain elements

13 if not sequence.is_last then

14 Result.append(",")

15 end

1617 -- Increment

18 sequence.forth

19 end

2021 Result.append ("]")

22 end

Messages

We represent messages as JSON objects, i.e. key-value pairs of �eld-names antheir value, surrounded by curly braces. We can use the fact that all messagesare generated from a common DSL speci�cation, for we can generate thecode of the serialization feature. We have to deal with message inheritance,though, but it can be solved by employing Ei�el's Precursor mechanism.

Feature serialize, as de�ned (deferred) in A_ARANEA_VALUE, isimplemented in A_PARSABLE di�erently than in value classes:

Listing 4.14: Serialization feature of A_PARSABLE


2 do


4 Result.append ("{" + serialization_list + "}")

5 end

The curly braces represent the outermost braces of the JSON object thatwill represent our message. Calling the deferred query serialization_list,which is e�ected in each descendant class, lets each message deal only withthe serialization to key-value pairs, without any encapsulation into a JSONobject. This is important for messages extending others: these just add aprecursor-call to get the parent's - already serialized - �elds as JSON string,and just add their own.


Listing 4.15: Simpli�ed serialization feature of a message extending anotherone

1 serialization_list: !STRING

2 -- Lists the fields to serialize , and under which

name

3 do

4 Result := precursor

56 Result.append ("%"" + field_binary_argument + "%":")

7 Result.append (field_binary.serialize)

8 end

Listing 4.15 shows the serialization_list of one of the example-classesprovided by Aranea, A_EXAMPLE_SIMPLE_EXTENDED_MESSAGE.This message inherits four �elds from its parent class. The call to the feature'sprecursor on line 4 returns the JSON representation of the parent class' �elds.

Serialization of regular message-�elds (as shown on lines 6 and 7), i.e.value types as well as nested records, is achieved by building a key-value pair.While the value is retrieved by calling serialize, the key is a simple stringconstant wrapped in double-quotes as mandated by JSON. The constant(as well as the feature serialization_list itself) is generated from themessage's DSL (currently the constant is a string denoting the �eld's name).

4.4.3 Deserialization

Remember: We represent messages as JSON objects, i.e. key-value pairs of�eld-names an their value, surrounded by curly braces. Thus, upon dese-rialization, we just need to read all keys of the JSON-object, and assignthe value to the corresponding �eld in the message class. Similar to se-rialization, we use the creation feature de�ned in A_ARANEA_VALUE,make_from_tokenizer to provide all objects with a way to create itself froma JSON string.

We use A_JSON_TOKENIZER, a simple, handcrafted, recursive de-scent parser of the JSON format used in Aranea. Initialized with a JSONstring upon receiving a message, the parser object is passed in each call tothe creation feature make_from_tokenizer, continually reading the string,creating objects from the JSON token (hence the name), and advancing onthe input.

As with serialization, di�erent parts of the library implement the feature


make_from_tokenizer di�erently. We will look at each of them in turn.

Nested records

A_PARSABLE, ancestor class to both records and regular messages, de�nesthe prototype feature for reading key-value pairs. It is shown in listing 4.16.

Listing 4.16: make_from_tokenizer as implemented in A_PARSABLE

1 make_from_tokenizer (a_tokenizer: A_JSON_TOKENIZER)

2 -- Create this message from its JSON representation

held and pointed to by `a_tokenizer '

3 -- This feature calls deferred procedure `parse_tag '

(which is effected in each descendant class)

4 -- for deserializing & recreating class fields.

5 local

6 l_tag: STRING

7 do

8 a_tokenizer.expect_token

(a_tokenizer.token_open_curly_brace)

910 from

11 until

12 a_tokenizer.has_errors or

13 a_tokenizer.last_token =

a_tokenizer.token_closed_curly_brace

14 loop

15 -- Read tag


(a_tokenizer.token_string)

17 l_tag := a_tokenizer.last_string

1819 -- a colon separates tags from values


(a_tokenizer.token_colon)

2122 parse_tag(l_tag , a_tokenizer)

2324 -- Increment:

25 -- If the scanner is not on a closed_curly_brace ,

consume a comma. Exit loop otherwise.

26 if a_tokenizer.last_token /=

a_tokenizer.token_closed_curly_brace then


(a_tokenizer.token_comma)

28 end

29 end

30


31 -- Expect the object to be finished


(a_tokenizer.token_closed_curly_brace)

33 check not a_tokenizer.has_errors end

34 end

Before entering the loop, the parser's cursor is expected to be on a { token,the start of a JSON object. The loop-body in turn reads key-value pairsfrom the parser: the tag (i.e. key) on lines 16 and 17, the value is read andassigned by the call to parse_tag on line 22 (see below). The set of key-valuepairs is separated by commas, which are consumed by the call on line 27.The loop exists if it encounters a closing curly brace.

The feature parse_tag called on line 22 is the counterpart to featureserialization_list used in serialization: messages do not need to be con-cerned about the JSON object structure. Instead, all they need to provide isthe association between �eld-names (i.e. the tag passed in the call) and the�eld itself.

Listing 4.17: parse_tag: recreating �elds according to the parser's last read�eld-name

1 parse_tag (a_tag: STRING; a_tokenizer: A_JSON_TOKENIZER)

2 do

3 if a_tag.is_equal (field_string_argument) then

4 create field_string.make_from_tokenizer

(a_tokenizer)

5 elseif a_tag.is_equal (field_integer_argument) then

6 create field_integer.make_from_tokenizer

(a_tokenizer)

7 elseif a_tag.is_equal (field_boolean_argument) then

8 create field_boolean.make_from_tokenizer

(a_tokenizer)

9 elseif a_tag.is_equal (field_double_argument) then

10 create field_double.make_from_tokenizer

(a_tokenizer)

11 end

12 end

Listing 4.17 shows the parse_tag feature of one of Aranea's provided exam-ples, A_EXAMPLE_SIMPLE_MESSAGE. The feature's code is generatedfrom the DSL speci�cation, and uses conditional branches to place a �eld'screation call upon processing the corresponding �eld-name.

To deal with messages inheriting message-�elds from other classes, weuse (like upon serialization) Ei�el's Precursor mechanism: we simply call


precursor(a_tag, a_tokenizer) upon entering the feature, and thus recre-ating �elds from parent-classes.

Messages

As shown in listing 4.2, make_from_tokenizer is called when recreating amessage received from AMQ. Its implementation in class A_MESSAGE, theancestor class of all messages transmittable via EMS to AMQ, di�ers onlymarginally from the one inherited from A_PARSABLE. The inherited fea-ture expects the parser's input to reside on a { token, because when process-ing nested records (that can only be used as a message's �elds, but cannotbe transmitted on their own), the cursor-position is advanced when readingthe preceding token.

But as make_from_tokenizer is the entry point that starts deserialization(and thus recreating of the message), the parser has just been initialized, andhasn't read any tokens yet. Thus we have to place a command to read theinitial token. Listing 4.18 shows the implementation of this. Note that tofurther process the message, we use the same make_from_tokenizer featureas with nested records (line 7).

Listing 4.18: make_from_tokenizer from class A_MESSAGE: the entrypoint of deserialization




3 -- This feature calls deferred procedure `parse_tag '

(which is effected in each descendant class)

4 -- for deserializing & recreating class fields.

5 do

6 a_tokenizer.read_token

7 Precursor {A_PARSABLE} (a_tokenizer)

8 a_tokenizer.expect_token (a_tokenizer.token_eof)

9 end

Value types

Creating primitive types is trivial: the type to be expected from the parseris inherent to the value-type, and thus simpli�es the creation procedure.

Listing 4.19: make_from_tokenizer from class A_INTEGER_VALUE





3 do

4 a_tokenizer.expect_token (a_tokenizer.token_integer)

5 value := a_tokenizer.last_integer

6 ensure then

7 value_set_when_no_errors: not a_tokenizer.has_errors

implies value = a_tokenizer.last_integer

8 end

For composite types, the process needs to account for JSON's meta-structure, i.e. we have to consume enclosing braces, and separating commas.As we constrain the generic parameter's creation feature (recall table 4.2) ofthe composite classes, we are fortunate enough that we can use it to recur-sively create the values held by the composite type.

Listing 4.20: make_from_tokenizer from class A_SEQUENCE_VALUE


2 -- Create this message from its JSON

representation held and pointed to by

`a_tokenizer '

3 local

4 l_item: G

5 do

6 create sequence.make_default


(a_tokenizer.token_open_bracket)

89 from

10 until

11 a_tokenizer.has_errors or

12 a_tokenizer.last_token =

a_tokenizer.token_closed_bracket

13 loop

14 create l_item.make_from_tokenizer

(a_tokenizer)

15 check

16 l_item /= Void

17 end

18 sequence.force_last (l_item)

1920 -- increment:

21 -- If the scanner is not on a closed_bracket ,

consume a comma. Exit loop otherwise.

22 if a_tokenizer.last_token /=

a_tokenizer.token_closed_bracket then



(a_tokenizer.token_comma)

24 end

25 end


(a_tokenizer.token_closed_bracket)

28 ensure then

29 created: sequence /= Void

30 end

4.5 Using the Message Framework

Aranea provides extensive examples on how to use all message types of theframework in its examples/messages folder. Below we show some shortenedcode-snippets.

4.5.1 Creating primitive types

As all of the framework's value classes are enhanced with convert clauses(recall section 4.2.2), primitive types can be created directly from their cor-responding Ei�el types.

Listing 4.21: Create primitive Aranea framework types

1 create {A_STRING_VALUE }.make ("some string")

4.5.2 Creating sequence types

Just like primitive types, A_SEQUENCE_VALUES can be created fromcorresponding Ei�el types: ARRAYs or Gobo Libray's DS_LINEAR struc-tures, but also from other Aranea sequences. Aranea usually uses Gobo'sDS_ARRAYED_LIST:

Listing 4.22: Create A_SEQUENCE_VALUEs

1 mylist: A_SEQUENCE_VALUE [A_STRING_VALUE]

2 ds_list: DS_ARRAYED_LIST [A_STRING_VALUE]

34 create ds_list.make (1)

5 ds_list.force_last ("mystring")

4.5. USING THE MESSAGE FRAMEWORK 65

6 mylist.make (ds_list)

4.5.3 Creating messages

The creation feature of a message will be generated (as is the whole class),it thus is crafted to accept Aranea values as parameters that correspond toits �elds. Let's look at a very simple message consisting of four �elds, andthe corresponding constructor:

1 class A_EXAMPLE_SIMPLE_MESSAGE

2 ...

34 field_string: A_STRING_VALUE

5 field_integer: A_INTEGER_VALUE

6 field_boolean: A_BOOLEAN_VALUE

7 field_double: A_DOUBLE_VALUE

89 make (a_field_string: A_STRING_VALUE;

10 a_field_integer: A_INTEGER_VALUE;

11 a_field_boolean: A_BOOLEAN_VALUE;

12 a_field_double: A_DOUBLE_VALUE)

1314 do

15 ...

16 end

17 end

Creating such an object is done by calling its creation feature make withappropriate types:

Listing 4.23: Creating a message object

1 create simple_message.make (

2 "This is a text -string",

3 42,

4 true ,

5 2.71828)

4.5.4 Accessing message values

Accessing values of primitive types is trivial: again, the convert clause willprovide the corresponding Ei�el type.


1 -- `string_field ' of type A_STRING_VALUE

2 my_string := a_msg.string_field

Accessing data of composite types is done by referencing the object'scorresponding storage �eld, foo.sequence or foo.map:

Listing 4.24: Value access of Aranea's composite types

1 ds_list: DS_ARRAYED_LIST [A_INTEGER_VALUE]

2 ds_list := a_msg.array_field.sequence

34 -- accessing sequence values:

5 loop

6 my_int := ds_list.item_for_iteration

7 end

4.6 Summary

Replacing Aranea's serialization format is an important step in openingAranea, as well as Origo, to a wider audience. It lifts the restriction ofEi�el-only nodes, which enables the use of virtually any library, written inany language � as long as a JSON binding exists. And that's hardly a re-striction: JSON's simple format can be implemented quickly if necessary, aswas done in this thesis for Ei�el.

Thanks to several features of the Ei�el language, Aranea's new messagetransport could be implemented in a completely type-safe manner. To achievethis, designated library classes that wrap basic Ei�el types have to be used,but this necessity is mitigated by using Ei�el's automatic type conversionsystem.

Chapter 5

Conclusions

5.1 Impact

The work described in this thesis covered every facet of Origo. Upgradingthe front-end replaced the foundation of the web-interface, the Drupal con-tent management system. This led to substantial refactoring of all its Origomodules, as well as various other front-end related processes on the level ofthe operating system.The exchange of Aranea's serialization protocol, and the extension of its mes-sage transport library, changed the way messages are created and processed.While existing users of Aranea only need minimal refactoring to use the newsystem, it changed all messages used in communicating nodes.

Not only existing Aranea nodes are a�ected by the new message transportprotocol, much bigger is the impact on future development with Aranea. Theuse of JSON, an open, cross-language and very simple serialization formatprovides maximum �exibility when developing new or existing enterprise-level, large-scale applications. Proof of this is the emerging port of Araneato Java [3].

All goals of this thesis could be reached as planned. This enabled toupgrade the main Origo installation at http://origo.ethz.ch: its front-end was upgraded to Drupal 6 after careful planning and testing on May5th, 2009, and its back-end was migrated to Aranea's new message transportprotocol on July 31st, 2009.

67

http://origo.ethz.ch

68 CHAPTER 5. CONCLUSIONS

5.2 Future Work

The objectives set for this thesis were achieved, and are feature-complete.This made the successful deployment of both the new front- as well as theback-end possible. Regardless, there are possibilities to further improve cer-tain facets of the work presented.

5.2.1 Transport of binary data

JSON does not provide any support for binary data. Aranea thus encodesbinary data in Base64, and transmits the output as a JSON string. This solu-tion was chosen because it seamlessly integrates with Aranea's JSON frame-work, at the cost of added processing for de-/encoding the Base64 values.ActiveMQ does support binary payload data, it even provides the possibilityto choose the several di�erent ways of transmission (e.g. transparent trans-mission of large binary �les via ftp instead of ActiveMQ's default OpenWireformat).

Aranea could be extended to bypass its own JSON serialization formatfor binary data, and directly use ActiveMQ's support (i.e use the appropriateEMS message class, BYTES_MESSAGE).

5.2.2 Front-end unit testing

As mentioned in section 3.4, testing any part of Origo's front-end has tobe done manually. When upgrading to Drupal 6.x, the lack of any kind oftesting system was a big handicap. A proper unit-testing suite for Origo issorely needed.Drupal does provide a unit-test system1. It is a non-trivial task to integrateit with Origo, though. Testing the front-end requires interaction with theback-end through XML-RPC, which requires careful design of test-cases.

5.2.3 Other improvements

There are other ways to improve Origo, not related to this thesis' goals.

1SimpleTest, http://drupal.org/project/simpletest, which moved to Drupal Corein 7.x, and is also supported by Drush

http://drupal.org/project/simpletest

5.2. FUTURE WORK 69

Sandbox

There's no easy way of �taking a quick look� at Origo. While opening user-accounts is a small obstacle, creating projects just for testing requires moree�ort on both the user's part, as well as on the administrator's. A sandboxenvironment [32] can improve this situation, and provides an ideal way toshowcase Origo and its possibilities.

5.2.4 User administration

Origo's administrative functions in regards to its users are very limited, partlybecause it can't rely on Drupal's user-management (as we have users asso-ciated with multiple Drupal sites), and partly because it wasn't given muchpriority. At the very least there should be options for spam-management,e.g. deleting all content of a speci�ed user at once.

5.2.5 Administrative functions accessible through web

Origo omits to o�er all of its administrative functions via the web front-end.For example, Origo's Administrators are noti�ed of project-requests solemnlyby emails � and no part of the front-end lists these requests. It is not onlyfor convenience that these functions should be accessible in the front-end:relying exclusively on emails is a dangerous single point of failure.

70 CHAPTER 5. CONCLUSIONS

Appendices

71

Appendix A

Origo Case Study for Drupal.org

This casestudy was written by the author to showcase Origo to the widerDrupal community. It was featured on the frontpage of drupal.org on Au-gust 18th, 2009.

73

drupal.org

Home » Forums » General » Drupal showcase

Origo - Distributed Software Developmentbherlig - August 13, 2009 - 22:16

What is Origo

Origo is an open, modular and extens ible softwaredevelopment, management and distributionplatform. Aimed at software developers , Origoprovides a set of services for combining, integratingand facilitating the development process over anetwork. Origo provides services like source controlmanagement, issue tracking, statis tics , releasehosting, wiki, blog and features for communicatingand networking with other developers . An extens iveAPI can be used to integrate Origo into otherapplications and development processes.

Origo was conceived as a research project of the Software Engineering chair atETH Zürich, and has resulted in several theses. Recently ownership wastransferred into an ETH spinoff, Oriact.

Released in August 2007, the platform as of August 2009 hosts 3,000 projectswith over 6,500 developers worldwide.

Origo's Features

Origo provides all the common features of a source control platform:

Source Control Management & hosting us ing Subvers ion

Release Hosting

Both open- and closed source projects

A wiki for presentation and documentation

Blogs and Forums

Bugtracker ("Issue Tracker")

Community wiki pages

Jabber server

Networking by befriending other users

Process tracking through workitems

Drupal showcase · Drupal 6.x

A user can own or be a member of any number of projects . While all projects areseparate, the user can conveniently track the progress of the projects centrally,as all actions (source commits , wiki-edits , etc.) are tracked by Origo as"workitems". Email notification for each project and each action can beconfigured. The concept of "s imultaneous separation while centrally tracked"projects posed the biggest challenge for this s ite.

Why Drupal?

When looking for a front-end framework that would allow to build Origo, it wasclear from the start that Origo had to be ready for the masses (because hosting aproject on Origo is free), and that stringent security requirements werenecessary. Therefore a solution was needed that

keeps the des ign separate from the content and is thus scalable

allows updating things in one place for all projects hosted on Origo

takes security serious ly and integrates in Origo's development process

The only CMS out there that met all three of these requirements, and additionallyallowed integration of great functionality from other developer's modules (therewas plenty of work to implement Origo anyway, and being able to re-use existingsolutions wherever poss ible was most welcome) was Drupal - thanks to theDrupal community, amazing work!

Architecture & Webpage

Origo uses a two-tieredarchitecture: a back-endhandling the bus inesslogic and most of the data,and a front-end for userinteraction. An APIconnects both layersus ing XML-RPC calls . Partof the API is also open to3rd-party applications,e.g., Origo's Eclipseplug-in. Origo's Webfront-end is implementedusing Drupal, includingcontributed and custom

modules, and set up in a (huge) multi-s ite installation: Each hosted project getsits own Drupal s ite with its own database, but all s ites share the same modules .This enables Origo to completely separate projects from each other whilebuilding upon a common code base for its modules. This also allows for tightcontrolling of each project's data. All project-independent functions, such asuser-related functions, are managed by the back-end and relayed back to thewebsite.

Drupal with a Twist

From a technical viewpoint, integrating multiple Drupal s ites with a back-end isone of the main challenges of Origo. Fortunately, this is where Drupal's modules -and APIs - come into play. Us ing a range of contributed and custom developedmodules, Origo's front-end provides the following main features:

Authentication

User authentication is handled by the back-end and replicated to Drupal. Thecustom module uses XML-RPC calls to authenticate users in the back-end. Theauthentication module works by implementing hook_form_alter, where Drupal'svalidation- and submit functions are swapped against custom ones thatauthenticate with the back-end. Also, functions from Drupal's session.inc areoverwritten with s lightly changed ones that synchronize sess ion information withthe back-end. This interception of Drupal's login mechanism is leveraged toimplement a s ingle-s ign-on system: users that change between projects , i.e.Drupal s ites , are automatically logged in at the new s ite, and the still validback-end authentication recreates a Drupal sess ion at the new s ite-instance.

The module also provides an XML-RPC framework for other Origo-API functions,both authenticated (that require a valid user-sess ion) and anonymous API calls .

Wiki

Origo aims to provide full MediaWiki functionality. It uses multiple modules toaccomplish this :

wikitools to provide wiki nodes in Drupal

Custom module mediawiki_filter wraps MediaWiki's library for pars ing andrendering MediaWiki syntax. The MediaWiki filter is enabled for all of Origo'scontent: issues, blogs, forum posts , etc.

Diff for comparing revis ions or previewing changes

GeSHi Filter for syntax highlighting of code blocks

Image, because they speak a thousand words

Image Ass ist to facilitate image insertion (locally patched to use MediaWikisyntax)

Custom module section_edit which provides the ability to edit individualsections of a wiki page

Custom module developer_pages enables users to restrict access of a wikinode to project members.

All these modules - including the custom ones - interact purely with Origo'sfront-end. Therefore, this wiki setup could be used in other Drupal installations .

Issue tracker

Origo usesits own bugtrackingmodulewhichintegrateswith itsback-end.Issues are acustom nodetype with itsownvocabulary.This way,issues canbe tagged,indexed and searched us ing Drupal's taxonomy module. An issue's metadata(such as open/closed-, ass ignment- or resolution-status) is added to a node'stags, too, and thus seamless ly integrate with other vocabulary data. The IssueTracker's main page provides an interface to this data, and bes ides displaying apaginated lis t of all issues, gives the user the poss ibility to search and filterthese tags. The different filters can be customized in any way, and the resultingcombination can be saved.

issue_tracker implements quite a few Drupal hooks, e.g. hook_view,hook_form_alter, hook_update, hook_insert, hook_nodeapi, etc. This is necessaryto intercept node process ing at various stages, and synchronize issue-nodeswith data from the back-end.

Origo Home: Communicating, tracking, releasing

origo_home is another custom module providing a bulk of Origo's features. Ithandles all user related functions such as profile, settings , networking (userfriendships and communities), as well as workitems (including RSS feed). Itoffers input masks for requesting projects and communities , and displays anaggregated view of issues reported by the user. It also provides the interface toOrigo's release hosting, i.e., displaying the lis t of a project's releases, as well asforms for uploading and categorizing releases. Furthermore, it implements aninterface for the system's administrative functions, such as mass mailings to allusers .

While most of origo_home's functions are straightforward form process ing, itsmain task is bringing the front- and back-end together. On the Drupal s ide thisresults in intercepting all processed nodes: hook_nodeapi is crucial for this . Itchecks which action is performed on what type of node, and registers this actionaccordingly with the back-end.

Restricted content

Origo implements a hierarchical role model. The most important roles are projectowners, project members and authenticated users. These roles - with theirassociated permiss ions - are used throughout Origo and give different usersdifferent poss ibilities of how to use Origo: posting blog entries , moderatingissues, editing wiki pages, etc.

Other modules

Besides the modules mentioned above, Origo uses the following contributedmodules:

Captcha for blocking spambots from user-regis tration

Customerror for blending in errors with Origo's des ign

Google CSE for providing s ite-wide search

Pathauto and Token for generating better readable URL's

Trash for nondestructive deletion of nodes

Views for generating node lis ts : a browsable lis t of content-nodes , filterableby type, and a lis t of taxonomy terms.

Back-end

Integrating, coordinating and tracking all s teps of the software developmentworkflow requires a powerful back-end. Origo uses a decentralized approachusing several process ing units (called nodes) potentially running on separatemachines, which communicate us ing Apache's ActiveMQ message brokersystem. API nodes receive XML-RPC calls , generate a message of an appropriatemessage type and relay them to the relevant nodes, e.g. a storage-node fordatabase access, or mail-node for sending emails .

Challenges

Managing Origo

Managing and maintaining such a large-scale multi-s ite Drupal installation -Origo's web front-end currently combines nearly 3000 Drupal multi-s ite instances- poses some unique problems, as one needs to automate Drupal administrativefunctions. Origo uses a combination of shell-, Python- and PHP-scripts to solvethis : some triggered by cronjobs or the back-end, some manually.

Drupal Installation

A usually s imple action such as adding a new instance to the multi-s iteinstallation is a complex process with Origo: While the creation of the front-end

needs to be coordinated with other services (a subvers ion repos itory is set up,databases created, etc.), there's also the need to tailor the new instance toOrigo's needs. This is achieved through a custom installation profile, whichenables the needed modules, activates the proper theme, creates navigationlinks, and also adds a few default nodes to the new Drupal instance. Origo needsto automate these s teps, i.e., perform command-line installations: for this , amodified vers ion of the Drupal CLI utils are used (which screen-scrape Drupal'sinstallation forms).

Updates

Similar to installations of new projects , Origo orchestrates updates - securityupdates or releases - from the command line. For a few months now, Origo hasused the wonderful Drush module (well, more of a "utility") to access Drupalfunctions via command line. Drush is used extens ively for installing newmodules, security updates, but also for executing custom SQL scripts (e.g. fordeploying bug fixes in custom modules ), emptying caches, etc.

API Synchronization

Origo's open API allows to access most functions from 3rd party applications,e.g., we provide a Mylyn plug-in to manage one's project's issues from withinEclipse. This mandates the propagation of data posted to the back-end via APIcalls to the front-end. The custom module issue_tracker implements a smallXML-RPC interface to this effect, with which a project's issue nodes can becreated or manipulated.

Future Work

In accordance with Drupal's motto, development on Origo never stops. Here's aquick overview of future prospects

Streamlining Drupal DatabasesCurrently all Drupal instances have their own, completely separateddatabases. Some of these could be shared between projects , e.g. acommon user-table for all instances.

SandboxA sandbox project where users can test Origo's functionality. This not onlyconcerns the Web- but also the back-end.

Project time managementExtend Origo with time management, where nodes (especially issues) canbe ass igned to a timeline, and be managed centrally.

Contact

You can contact the Origo team by either emailing [email protected], or posting onthe message board - or by posting comments below.Associated Drupal users: bayt, roetzi, tario and bherlig.

80 APPENDIX A. ORIGO CASE STUDY FOR DRUPAL.ORG

Appendix B

Comparison of Marshalling

Methods

This presentation was given to the Origo team on May 20th 2009 by theauthor.

��

��

��

��

��

��

� �� !

� �∀�#�∃∀��

� ��

� ��

� ��%∀��

�� &

�∀�#�∃∀��

� ��∋(��)��∗

� ��+��,��

� −��%��.��

� ��∀�%�

� ��/0��∀��

� ��)1%%��%�

� 2��,��3��!4��5��

� �∀��0��∀��6��∀��

� ��%∀��62��7��6#��%�8�0/#−��

�� 9

��

� −��%��∀��(�3�:��;∀�<∗

� = ��

� >��?

� 2�%

� ��

� ��#�1)5≅�−

� 8�0/#−�

� ��1−

� Α1�0

81

�� Β

≅%3��

� 2=��

,��∃�5��:��∀�3�6��5��% ��Χ��%�2�% ��Χ��∀��%��∀;��%�� ≅��%��%��;� ��;��+��≅��6��% �%��%��%�� <

� �� ∀% �� ∆��1−Ε

� �� (��Φ6�Γ�(��!∗��ΗΙ��Γ�ϑΚ�� ∗

� ��!

�� Φ

≅%3��

� ��#�1)5≅�−� ��%∀��#�≅� � ��Ι∗

�� Λ

��%��∀��

� −��%��∀��(�3�:��;∀�<∗

� = ��

� >��?

�� Η

−��%��

� >��?

� =�Γ�;��

� ��%��{"menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "CreateNewDoc()"}, {"value": "Open", "onclick": "OpenDoc()"}, {"value": "Close", "onclick": "CloseDoc()"} ] }}}

<menu id="file" value="File"> <popup> <menuitem value="New" onclick="CreateNewDoc()" /> <menuitem value="Open" onclick="OpenDoc()" /> <menuitem value="Close" onclick="CloseDoc()" /> </popup></menu>

��

−��%��

� ��;∀�4� ��

� ��

� ��!��4��%3!��%��;��

� 7��.��

� ≅�/=��

�� Κ�

=��

��;∀�� double, float

� u/s/int 32/64

� s/fixed 32/64

� Bool

� String

� Bytes

� Enum

� Repeated Flag for Fields => Arrays

� Optional/required fields

� Nested types

� Packages

� Services

� �� bool

� byte: 8-bit signed integer

� i16: 16-bit signed integer



� double: 64-bit FP number

� string: UTF-8 encoding

� binary: seq of unenc. bytes

� Struct: strongly typed fields (no inheritance)

� list<type>

� set<type>

� map<type1,type2>

� exception

� services

�� ΚΚ

0��∀��

� ��;∀�Μ��%��

� �ΝΝ6>� �6−��

� ��;∀�Μ�� ;��

� 1%��%��6�6�Ο6��0��62��

� ��3��6>� ��26>�6��%∀��6−��6−�−6#∀;�

� � ��Μ��%��

� �ΝΝ6�6>� �6�Ο

� � ��Μ�� ;��

� 2��6��3��6�;Π�%�� 6�%��6−��6−�−6−�� 6#∀;�6�∃∀��3

�� Κ�

��4��+��;�∀%�;�∀%��

ser_thrift (0.560s) 370009 bytes

ser_pb (4.850s) 171650 bytes

ser_json (0.080s) 680680 bytes

ser_yaml (17.330s) 610680 bytes

ser_thrift_compressed (0.620s) 111326 bytes

ser_pb_compressed (3.980s) 98571 bytes

ser_json_compressed (0.110s) 124919 bytes

serde_thrift (2.130s)

serde_pb (7.550s)

serde_json (0.130s)

10'000 random DNS records:

message DnsRecord {

required fixed32 sip = 1;

required fixed32 dip = 2;

required uint32 sport = 3;

required uint32 dport = 4;

}

�∀��

ser_yaml_compressed (17.160s) 121065 bytes

serde_yaml (56.740s)

�� Κ&

��4��+�

Θ��%∀��Ρ

� −��;∀�� !��% �

option optimize_for = SPEED;

� ��∀��+��+�

� −��;��

�� Κ9

��

��))%��%��)�)� ��/��;∀�/%��)!�3�)��% ��3��

�� ΚΒ

0�3�

Κ� ��))��∀��%��)��Η)�Λ)Κ�)� ��/ �/��

�� ))��)��)� ��/��%

��))��)��)� ��/��%

84 APPENDIX B. COMPARISON OF MARSHALLING METHODS

Appendix C

JSON

This appendix gives an overview of the JSON format. It is taken verbatimfrom http://json.org, and listed here for the reader's convenience.

Introducing JSON

JSON (JavaScript Object Notation) is a lightweight data-interchange format.It is easy for humans to read and write. It is easy for machines to parse andgenerate. It is based on a subset of the JavaScript Programming Language1,Standard ECMA-262 3rd Edition - December 19992. JSON is a text formatthat is completely language independent but uses conventions that are fa-miliar to programmers of the C-family of languages, including C, C++, C#,Java, JavaScript, Perl, Python, and many others. These properties makeJSON an ideal data-interchange language.

JSON is built on two structures:

• A collection of name/value pairs. In various languages, this is real-ized as an object, record, struct, dictionary, hash table, keyed list, orassociative array.

• An ordered list of values. In most languages, this is realized as anarray, vector, list, or sequence.

These are universal data structures. Virtually all modern programminglanguages support them in one form or another. It makes sense that a dataformat that is interchangable with programming languages also be based onthese structures.

1http://javascript.crockford.com2http://www.ecma-international.org/publications/files/ecma-st/ECMA-262.

pdf

85

http://json.org

http://javascript.crockford.com

http://www.ecma-international.org/publications/files/ecma-st/ECMA-262.pdf

http://www.ecma-international.org/publications/files/ecma-st/ECMA-262.pdf

86 APPENDIX C. JSON

In JSON, they take on these forms:

An object is an unordered set of name/value pairs. An object begins with{ (left brace) and ends with } (right brace). Each name is followed by :

(colon) and the name/value pairs are separated by , (comma).

An array is an ordered collection of values. An array begins with [ (leftbracket) and ends with ] (right bracket). Values are separated by , (comma).

A value can be a string in double quotes, or a number, or true or falseor null, or an object or an array. These structures can be nested.

A string is a collection of zero or more Unicode characters, wrapped indouble quotes, using backslash escapes. A character is represented as a singlecharacter string. A string is very much like a C or Java string.

A number is very much like a C or Java number, except that the octaland hexadecimal formats are not used.

87

Whitespace can be inserted between any pair of tokens. Excepting a fewencoding details, that completely describes the language.

88 APPENDIX C. JSON

Appendix D

Messages Provided by Aranea

Aranea provides several messages that can be sent between nodes, or can beused by extending them. Note that some of these are handcrafted, and notgenerated from a DSL speci�cation.

Non-Generated Messages

• Namespace "Session"

� A_SESSION_MESSAGEThis is a deferred class that provides a single string session. Itis a convenience message that can be inherited from to implementmessages that need authentication.Fields:

∗ session: string

• Namespace "General"

� A_GENERAL_STATUS_MESSAGEA class for simple acknowledgements, may also transport error-messages.Fields:

∗ is_success: boolean

∗ message: string

� A_GENERAL_STRING_MESSAGEA convenience class for sending a single string.Fields:

∗ string: string

89

90 APPENDIX D. MESSAGES PROVIDED BY ARANEA

� A_GENERIC_MESSAGEThis special message let's you assign your own namespace andtype. Note that productive use of this requires you to manuallyedit the constant- and factory-classes, which is why it is markedas obsolete. Furthermore it provides a map to hold strings.Fields:

∗ (namespace: string)

∗ (type: string)

∗ arguments: map<string>

� A_GENERAL_STATUS_MESSAGEA class for simple acknowledgements, may also transport error-messages.Fields:

∗ is_success: boolean

∗ message: string

Generated Messages

• Namespace "Con�g"

� A_CONFIG_COMMAND_MESSAGEMessage to execute a command on the host's operating system.Fields:

∗ command: string

� A_CONFIG_COMMAND_LIST_MESSAGESame as above, but contains multiple commands to execute.Fields:

∗ commands: sequence<string>

� A_CONFIG_FILE_WRITE_MESSAGEMessage to write a single �le on the host's �lesystem.Fields:

∗ name: string

∗ content: string

� A_CONFIG_FILE_WRITE_LIST_MESSAGESame as above, but contains multiple �les to write to.Fields:

∗ files: sequence< [name: string, content: string]>

91

• Namespace "Mail"

� A_MAIL_SEND_MESSAGEMessage for communicating with Aranea's mail node, for sendingan email to recipient.Fields:

∗ recipient: string

∗ sender: string

∗ subject: string

∗ text: string

• Namespace "System"

� A_SYSTEM_STATUS_REQUEST_MESSAGEA message to request nodes to return their status. Note thatAranea registers a handler for this message, i.e. all nodes are ableto receive and process it.Fields:

∗ none

92 APPENDIX D. MESSAGES PROVIDED BY ARANEA

Appendix E

Aranea's New Message Transport

This presentation was given to the Origo team on July 14th 2009 by theauthor.

��

��

��

��

!∀��#∃%��&� �� !� %∋!(%

��)��

�∗��∗��+

� � � ��,��

� ��+��∃��

� ��−��

� ��

��)�� .

� � ��,��

� /� 0��%��1�22��3

� 14(��∗��1�22��

� ��∋� ��)

� ��)STRING#INTEGER#!!!

� 5��4��)

� HASH_TABLE [ANY, STRING]

� 1�22��∋� �� ∃� ��

� �� (%��, � ��

��)�� 6

� � ��,��

93

��)�� 7

� � ��,��

��)�� 8

� � ��,��

��)�� 9

� � ��,��

� :%�(%�� ;

� ��+��+� %�� 4�� 1�22��∃��;

� �� ∗�� %��;

� ��4�∃�� #��<�� 2��==

��+��∃��

��)��

��+��∃��

� %�>��)��∋� ��

� ?��#≅��#%��2 #<�� ∃�2#��

� ��∗�!��∋�

� 5��

� ��Α�((�� (�

� 1��2��

� 1�22�� ):��Β2��+Χ5��−��

� ∆�(��

� %�:��)��)�� Ε�

��+��∃��

� ∆��Α��∋� ��2��∃��Φ��

� �� 2��0�(�� (� ��

� 5�� ∃��Γ��)Ε9��Η

� � ��

� ��(�5��)Η

��)�� ΕΕ

��+��∃��

��)�� Ε�

��+��∃��

��)�� Ε.

��+��∃��

��)�� Ε6

��+��∃��

��)�� Ε7

��+��∃��

��)�� Ε8

��+��∃��

Γ��2��Η−��)

Message: (Field)*

Field -> I : TI -> identifier

T -> integer -> double -> boolean -> string -> binary

-> map [T] -> sequence [T]

-> record [ I1 : T

1, I

2 : T

2, … ]

��)�� Ε9

��+��∃��

� 14��

[message Simple]{ properties= { field_integer: integer field_boolean: boolean field_double: double field_string: string }}

��)�� ΕΙ

��+��∃��

� 14��


��)�� Ε�

��+��∃��

� 14��


��)��

��+��∃��

� 14��


��)�� Ε

��+��∃��

� 14��


��)��

��+��∃��

� 14��

��)�� .

��−��

� −��)%�(��2�� (��%��∃�� !

� ��:#+� %

� ��0�(�� Γ∆��#∆��(�2�(��Η

� ��0�(�� 1(��1��

� ��0�(�� 5��−��

��)�� 6

��−��

:��Β2��+

Γ ��1(��Η

Ε!:�� 2�� ∆��2��

�!ϑ��:�:��Β2��+ �� (��

.!5�� 2�� Κ�( !

��)�� 7

��−��

Ε!:�� 2�� ∆��2��

∆� � ��

��)

stringintegerdoublebooleanbinary

sequence<…>map<…>

record(name, […; …])

��

Property:name=ID ":" DataType (Restriction)* (description=STRING)+;

notVoidnotEmptygreaterZerogreaterEqualZero

��)�� 8

��−��

ϑ�(��) record(name, [i1: T

1; i

2: T

2; …])

� ∆�2��

� Γ�� Ηϑ��+��∃��

� ��%�� A_PARSABLE#∃� �� A_MESSAGE

� ��2��#∃� ��

� ϑ��(��2TUPLE�

revisions: custom [" ARRAYED_LIST [ TUPLE [ creation_time: INTEGER; user: STRING; text: STRING; tags: STRING]]"]

revisions: sequence < record(revision, [ creation_time: integer; user: string; text: string; tags: string]) >

��)�� 9

��−��

�(��4��2��

[message ListReply]{ properties= { release_list: map<record(release, [ ... files: sequence< record(list_file, [ name: string; platform: string ]) > ])> notVoid "Release list" }}

��)�� Ι

��−��

ϑ�� ( ��

� 1�(%�� %�∗�� ( ��

� �� 1�22��(�� ( �

� ��2�� 2��(��=

revisions: sequence< record(revision, [ creation_time: integer greaterEqualZero; user: string notEmpty; text: string notEmpty; tags: string notVoid ]) >

��)��

��−��

<� 2��

� Φ�� ∗�� (%�(Β�

� � �� (��

� >��+� ��)

� ��Λ��2�� 2�� 2��(��

� ��(��

� ϑ�(��

� ��%�� (�

� :%� ��Α+��2��;

��

��)�� .Ε

��

Ε!��

�!5��

.!��4��)Η

��)�� .�

��

<��

� ��

� Γϑ��Η ��:��

∆��

� longstringΜ��binary

� Longinteger integer � Γ��INTEGER_64Η

� stringlist#integerlist#longintegerlistΜ��sequence<…>

� customΜ��record#sequence��map

��)�� ..

��

1�22��(�� )��∃��

� A_GENERIC_LIST_MESSAGE [T]Μ�� +� %�2��A_SEQUENCE_VALUE [T]

� A_GENERIC_SESSION_MESSAGEΜ�� %�� 2��A_SESSION_MESSAGE+� %�� 2��

� ∀�� Γ�� Η)��(��2A_GENERIC_MESSAGE+� %�� Γ��∗�� )��2�(� �� (��Η

��)�� .6

��

1�22��(��

� ��4��(��

� ��0�� ∗� ��

Before:make (a_session_string)

Now:make (

create {A_STRING_VALUE}.make (a_session_string))

��)�� .7

��

1�22��(��

� ��4��(��

� 5�� (��4 ��2�� (��

Before: create mylist.make; mylist.add (mystring)Now:ds_list: DS_ARRAYED_LIST [A_STRING_VALUE]

create ds_list.make (1)ds_list.force_last (

create {A_STRING_VALUE}.make (mystring))

mylist.make (create {A_SEQUENCE_VALUE[A_STRING_VALUE]}.make (ds_list))

��)�� .8

��

1�22��(��

� ��4��∗��((��

Before: my_string := a_msg.string_fieldmy_array ?= a_generic_msg.arguments (“my_array”)

Now:my_string := a_msg.string_field.value

ds_list: DS_ARRAYED_LIST [A_INTEGER_VALUE]ds_list := a_msg.array_field.sequence

loopmy_int := ds_list.item_for_iteration.value

end

��)�� .9

��

� <�� +��Β� ��(��

� −��Γ�!�!��2��?��ϑ<5Η��+��Β �� 2�� 2�� ?��

� HASH_TABLE [ANY, STRING]

� %�� %��∃�(%�(Β��∃� %��

� ϑ�� (��

;��

��Β�

� ��)% �)ΑΑ��!��!� %∋!(%

� ��)% �)ΑΑ��!��!� %∋!(%Α+�Β�Α��Ν��

� ��)% �)ΑΑ+++!��!� %∋!(%

� ��:)% �)ΑΑ+++!��(%� �( ��+��!��

� ��)% �)ΑΑ+++!Κ��!��

Bibliography

[1] ActiveMQ-CPP. http://activemq.apache.org/cms.

[2] Apache ActiveMQ. http://activemq.apache.org.

[3] Aranea Java. https://svn.origo.ethz.ch/aranea/trunk/java.

[4] Check Language, as used in openArchitectureWare. http:

//www.openarchitectureware.org/pub/documentation/4.3.1/

html/contents/core_reference.html#Check_language.

[5] Coder. http://drupal.org/project/coder.

[6] CORBA / GIOP. http://www.omg.org/technology/documents/

formal/components.htm.

[7] Drupal 6.x menu system. http://drupal.org/node/102338.

[8] Drupal, an open source content management platform. http://www.

drupal.org.

[9] Drupal CLI utils. http://acquia.com/blog/drupal-cli-utils.

[10] Drupal Multi-site. http://drupal.org/getting-started/6/install/multi-site.

[11] Drupal Taxonomy module. http://drupal.org/handbook/modules/

taxonomy.

[12] Drupal's Coding Standards. http://drupal.org/node/318.

[13] Drush. http://drupal.org/project/drush.

[14] Ei�el: ECMA-367 standard. http://www.ecma-international.org/

publications/standards/Ecma-367.htm.

[15] Ei�el Messaging Service. http://ems.origo.ethz.ch.

99

http://activemq.apache.org/cms

http://activemq.apache.org

https://svn.origo.ethz.ch/aranea/trunk/java

http://www.openarchitectureware.org/pub/documentation/4.3.1/html/contents/core_reference.html#Check_language



http://drupal.org/project/coder

http://www.omg.org/technology/documents/formal/components.htm

http://www.omg.org/technology/documents/formal/components.htm


http://www.drupal.org

http://www.drupal.org

http://acquia.com/blog/drupal-cli-utils

http://drupal.org/getting-started/6/install/multi-site

http://drupal.org/getting-started/6/install/multi-site

http://drupal.org/handbook/modules/taxonomy

http://drupal.org/handbook/modules/taxonomy


http://drupal.org/project/drush

http://www.ecma-international.org/publications/standards/Ecma-367.htm

http://www.ecma-international.org/publications/standards/Ecma-367.htm

http://ems.origo.ethz.ch

100 BIBLIOGRAPHY

[16] ETCH. http://incubator.apache.org/projects/etch.html.

[17] Facebook. http://www.facebook.com.

[18] Hessian. http://hessian.caucho.com.

[19] JSON. http://www.json.org.

[20] JXTA Project. http://www.jxta.org.

[21] MediaWiki. http://www.mediawiki.org.

[22] Mylyn connector for Origo. http://mylyn.origo.ethz.ch.

[23] MySQL. http://www.mysql.com.

[24] openArchitectureWare. http://www.openarchitectureware.org.

[25] Origo. http://www.origo.ethz.ch.

[26] Origo API. http://www.origo.ethz.ch/wiki/Origo_API.

[27] Origo case-study on drupal.org. http://drupal.org/node/548672.

[28] PHP Development Tools. http://www.eclipse.org/pdt.

[29] Project Goanna, The Ei�el Web Application Framework. http://

goanna.origo.ethz.ch.

[30] Protocol Bu�ers. http://code.google.com/p/protobuf.

[31] RRDtool. http://oss.oetiker.ch/rrdtool.

[32] Sandbox. http://en.wikipedia.org/wiki/Sandbox_(software_

development).

[33] Subversion source control system. http://subversion.tigris.org.

[34] Thrift. http://incubator.apache.org/thrift.

[35] Wikipedia: Message-oriented middleware. http://en.wikipedia.org/wiki/Message-oriented_middleware.

[36] Wikipedia: Service Oriented Architectures. http://en.wikipedia.

org/wiki/Service-oriented_architecture.

[37] XDebug. http://www.xdebug.org.

http://incubator.apache.org/projects/etch.html

http://www.facebook.com

http://hessian.caucho.com

http://www.json.org

http://www.jxta.org

http://www.mediawiki.org

http://mylyn.origo.ethz.ch

http://www.mysql.com

http://www.openarchitectureware.org

http://www.origo.ethz.ch

http://www.origo.ethz.ch/wiki/Origo_API


http://www.eclipse.org/pdt

http://goanna.origo.ethz.ch

http://goanna.origo.ethz.ch

http://code.google.com/p/protobuf

http://oss.oetiker.ch/rrdtool

http://en.wikipedia.org/wiki/Sandbox_(software_development)

http://en.wikipedia.org/wiki/Sandbox_(software_development)

http://subversion.tigris.org

http://incubator.apache.org/thrift

http://en.wikipedia.org/wiki/Message-oriented_middleware

http://en.wikipedia.org/wiki/Message-oriented_middleware

http://en.wikipedia.org/wiki/Service-oriented_architecture

http://en.wikipedia.org/wiki/Service-oriented_architecture

http://www.xdebug.org

BIBLIOGRAPHY 101

[38] Xpand Language, as used in openArchitectureWare. http://www.

openarchitectureware.org/pub/documentation/4.3.1/html/

contents/core_reference.html#xpand_reference_introduction.

[39] Xtend Language, as used in openArchitectureWare. http:

//www.openarchitectureware.org/pub/documentation/4.3.1/

html/contents/core_reference.html#Xtend_language.

[40] Xtext Language, as used in openArchitectureWare. http://www.

eclipse.org/Xtext.

[41] Till G. Bay. Hosting distributed software projects: concepts, framework,and the Origo experience. PhD thesis, ETH Zürich, 2008.

[42] E. Lewis. The Role of Wildcards in the Domain Name System. RFC4592 (Proposed Standard), July 2006.

[43] Bertrand Meyer. Ei�el: the language. Object-Oriented Series. PrenticeHall, New York, NY, 1992.

[44] Dennis Rietmann. Origo plugins - extension and maintenance. Master'sthesis, ETH Zürich, March 2008.

[45] Patrick Ruckstuhl. Origo core: Middleware and controller for origo.Master's thesis, ETH Zurich, July 2007.

[46] Beat Strasser. Vampeer, jxta implementation for ei�el. Master's thesis,ETH Zurich, March 2007. http://vampeer.origo.ethz.ch.

[47] Dave Winer. XMLRPC speci�cation, October 1999. http://www.

xmlrpc.org/spec.

[48] Peter Wyss. Origo home: Web interface design and work for origo.Master's thesis, ETH Zurich, September 2007.

http://www.openarchitectureware.org/pub/documentation/4.3.1/html/contents/core_reference.html#xpand_reference_introduction



http://www.openarchitectureware.org/pub/documentation/4.3.1/html/contents/core_reference.html#Xtend_language



http://www.eclipse.org/Xtext

http://www.eclipse.org/Xtext

http://vampeer.origo.ethz.ch

http://www.xmlrpc.org/spec

http://www.xmlrpc.org/spec

rights / license: research collection in copyright - non ...1424/eth... · listings 3.1 menu de...

Documents