integration of software components in an heterogeneous distributed environment

School of Computer ScienceMaster’s dissertation

Integration of softwarecomponents in an heterogeneous

distributed environment

Author: Edouard SwiacNovember 10, 2011

Contents

Introduction 2

I Distributed computing 3

1 Foundations 41.1 Why? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Design and challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2.1 Heterogeneity . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.2 Openness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.2.3 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.2.4 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.2.5 Failure handling . . . . . . . . . . . . . . . . . . . . . . . . . 111.2.6 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.2.7 Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . 121.2.8 Quality of service . . . . . . . . . . . . . . . . . . . . . . . . 13

1.3 Fallacies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2 Trends and future 162.1 The rise of cloud computing . . . . . . . . . . . . . . . . . . . . . . 162.2 How Google uses distributed systems . . . . . . . . . . . . . . . . . 17

1

CONTENTS 2

II 2600hz, an heterogeneous distributed platform 19

3 Presentation of the solution 203.1 Whistle, the cornerstone of cloud-enabled VoIP . . . . . . . . . . . 20

3.1.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 2600hz, a powerful distributed communication stack . . . . . . . . . 223.3 Solving vertical scaling limitations with distributed systems . . . . . 24

4 Programming distributed systems 264.1 Erlang, concurrent programming for distributed applications . . . . 28

4.1.1 Open Telecom Platform . . . . . . . . . . . . . . . . . . . . 284.1.2 Strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.1.3 Weaknesses . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.1.4 Erlang in industry-level communication systems . . . . . . . 31

5 Architecture and integration 325.1 Functional overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.2 Messaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3 Redundancy and fault tolerance . . . . . . . . . . . . . . . . . . . . 365.4 Distribution of data . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.4.1 CAP Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 395.4.2 Eventual consistency . . . . . . . . . . . . . . . . . . . . . . 40

5.5 Unlimited concurrency . . . . . . . . . . . . . . . . . . . . . . . . . 415.6 Directed events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.7 Schema flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445.8 Strong supervision . . . . . . . . . . . . . . . . . . . . . . . . . . . 465.9 Speed for adding features . . . . . . . . . . . . . . . . . . . . . . . . 475.10 Fast server provisioning . . . . . . . . . . . . . . . . . . . . . . . . . 475.11 Avoid downtimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

CONTENTS 3

6 Functional and technical specification of a module 506.1 What is hot desking? . . . . . . . . . . . . . . . . . . . . . . . . . . 506.2 Integration in 2600hz . . . . . . . . . . . . . . . . . . . . . . . . . . 51

III Actors of today’s cloud-enabled VoIP 55

7 2600hz, a disruptive startup company 567.1 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Conclusions 57

IV Personal insight 597.2 San Francisco, startups and entrepreneurship . . . . . . . . . . . . . 607.3 Erlang and distributed systems, from theory to practice . . . . . . . 62

List of Figures

1.1 Constant, linear, logarithmic and quadratic running times . . . . . 10

3.1 Whistle components . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 Overview of 2600hz . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.1 Shared-memory and message-passing models . . . . . . . . . . . . . 274.2 Erlang scalability on multi-core CPUs . . . . . . . . . . . . . . . . . 30

5.1 Functional view of 2600hz . . . . . . . . . . . . . . . . . . . . . . . 335.2 Data partitioning in BigCouch . . . . . . . . . . . . . . . . . . . . . 385.3 Visual guide to NoSQL systems . . . . . . . . . . . . . . . . . . . . 405.4 AMQP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435.5 Directed Events using AMQP . . . . . . . . . . . . . . . . . . . . . 445.6 Supervision of Erlang processes . . . . . . . . . . . . . . . . . . . . 465.7 Hot code swapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.1 Hot desk functional specification: Login . . . . . . . . . . . . . . . . 526.2 Hot desk functional specification: Logout . . . . . . . . . . . . . . . 536.3 Hot desk call flow (simplified) . . . . . . . . . . . . . . . . . . . . . 54

7.1 Startup building process . . . . . . . . . . . . . . . . . . . . . . . . 61

4

Abstract

Most Voice over IP service providers create home-grown tools to manage theirsystems. As their needs and customer demands grow, their toolset falls behindand becomes a pain point. A system built to grow with millions of users in mindmust be conceived to be scalable from start, and that scalability can be achieved byrelying on a distributed architecture. Moreover, distributed systems benefit fromthe cloud: automatic scaling with on-demand servers and distribution of serviceacross the globe.Distributed systems are hard to achieve, because they demand a strong attentionto several principles to ensure communication between heterogeneous components,scalability and fault-tolerance. The architecture and integration of components isvery important, and engineers will benefit from the reuse of technologies across thecomponents. For example, using the same programming language to write differentcomponents or using software built on open standards.Building such a system frequently requires different components, and each must bechosen carefully so as to not become a bottleneck. Inspecting each component froma distributed perspective will help to detect if the component is appropriate, or ifit presents weaknesses that can threaten the system in the future.

LIST OF FIGURES 1

Acknowledgments

I joined 2600hz, a software startup, as a software engineer, thanks to CEO DarrenSchreiber and COO Patrick Sullivan, founders of the company.I would like to thank my two mentors, James Aimonetti and Karl Anderson, Seniorsoftware engineers at 2600hz for their support, teaching, team spirit and vision.Thanks to all my peers at 2600hz, it was a wonderful experience!

Introduction

To overcome the limitations of client-server architecture (scalability, performances),we may look at distributed architecture. This type of architecture may help organi-zations to solve scalability problems by using horizontal scaling instead of verticalscaling. But the design of distributed systems has to be done carefully and thor-oughly, as well as the choice of its components: distribution capacities, scalingfactor, level of integration.In the first part, the attention of the reader will be drawn to the advantages, prin-ciples and challenges offered by distributed systems, as well as the fallacies andweaknesses of this architecture. Distributed architectures also benefit from theelastic nature and the ubiquity of the cloud as it’s becoming an infrastructure ofchoice to deploy distributed applications.A thorough analysis of a distributed architecture implementation in a scalable com-munication platform will be done in the second part, as well as the decisions madefrom an architect’s point of view on the choice of the components: scalability per-spective, integration in a distributed environment, cloud compatibility.Lastly we will focus the reader’s attention on the cloud VoIP market. As 2600hz,the distributed platform being discussed here, is aimed at VoIP and unified com-munications, a closer look at its creators and how competitors do cloud VoIP mayhelp the reader to understand where 2600Hz, the company, is headed.

2

Part I

Distributed computing

3

Chapter 1

Foundations

1.1 Why?

A distributed system is one in which autonomous components located at networkedcomputers communicate and coordinate their actions only by passing messages.Software runs in concurrent processes on different computers on different processors.There are multiple points of controls, and multiple points of failure. This definitionleads to the following significant characteristics of distributed systems: concurrencyof components, lack of a global lock and independent failures of components.

Concurrency In a network of computers, concurrent program execution is thenorm. The capacity of the system to handle shared resources can be increasedby adding more resources (for example: computers) to the network.

Absence of global lock When programs need to cooperate they coordinate theiractions by exchanging messages.

Independent failures All computer systems can fail, and it is the responsibility ofsystem designers to plan for the consequences of possible failures. Distributedsystems can fail in new ways.

This approach is different from centralized systems, where only one componentwith non-autonomous parts is running on a single machine. This single componentis shared by users all the time, runs in a single process. It is ruled by a single pointof control, thus representing a single point of failure.

4

CHAPTER 1. FOUNDATIONS 5

The challenges arising from the construction of distributed systems are the hetero-geneity of their components, openness (which allows components to be added orreplaced), security, scalability – the ability to work well when the load or the num-ber of users increases – failure handling, concurrency of components, transparencyand providing quality of service.

1.2 Design and challenges

While designing distributed systems, significant challenges are encountered. Weshould notice that not all those challenges can be faced. The broader the scope andthe scale of the distributed system is, the more we are likely to encounter all thosechallenges at once.

1.2.1 Heterogeneity

The Internet enables users to access services and run applications over a heteroge-neous collection of computers and networks. Heterogeneity (variety and difference)applies to all of the following:

• networks

• computer hardware

• operating systems

• programming languages

• implementations by different developers

Although the Internet consists of many different sorts of network, their differencesare masked by the fact that all of the computers attached to it use the Internetprotocols (like TCP) to communicate with one another.Data types such as integers may be represented in different ways on different sorts ofhardware – for example, there are two alternatives for the byte ordering of integers(endianness). These differences in representation must be dealt with if messages areto be exchanged between programs running on different hardware.


Although the operating systems of all computers on the Internet need to includean implementation of the Internet protocols, they do not necessarily all provide thesame application programming interface to these protocols. For example, the callsfor exchanging messages in UNIX are different from the calls in Windows.Different programming languages use different representations for characters anddata structures such as arrays and records. These differences must be addressedif programs written in different languages are to be able to communicate with oneanother.Programs written by different developers cannot communicate with one anotherunless they use common standards, for example, for network communication andthe representation of primitive data items and data structures in messages. Forthis to happen, standards need to be agreed and adopted – as have the Internetprotocols.

Middleware

The term middleware applies to a software layer that provides a programming ab-straction as well as masking the heterogeneity of the underlying networks, hardware,operating systems and programming languages. The Common Object Request Bro-ker (CORBA), is an example. Some middleware, such as Java Remote MethodInvocation (RMI), supports only a single programming language. Most middlewareis implemented over the Internet protocols, which themselves mask the differencesof the underlying networks, but all middleware deals with the differences in operat-ing systems and hardware. Middleware communication is discussed further in theAMQP section of this paper.In addition to solving the problems of heterogeneity, middleware provides a uniformcomputational model for use by the programmers of servers and distributed applica-tions. Possible models include remote object invocation, remote event notification,remote SQL access and distributed transaction processing. Its implementation hidesthe fact that messages are passed over a network in order to send the invocationrequest and its reply.The virtual machine approach provides a way of making code executable on a varietyof host computers: the compiler for a particular language generates code for avirtual machine instead of a particular hardware order code. For example, theErlang compiler produces code for a Erlang virtual machine, which executes it by


interpretation. The Erlang virtual machine needs to be implemented once for eachtype of computer to enable Erlang programs to run.

1.2.2 Openness

The openness of a computer system is the characteristic that determines whetherthe system can be extended and reimplemented in various ways. The openness ofdistributed systems is determined primarily by the degree to which new resource-sharing services can be added and be made available for use by a variety of clientprograms.Openness cannot be achieved unless the specification and documentation of the keysoftware interfaces of the components of a system are made available to softwaredevelopers, like APIs (public or not). In a word, the key interfaces to commu-nicate with the system are published. However, the publication of interfaces isonly the starting point for adding and extending services in a distributed system.The challenge is to tackle the complexity of distributed systems consisting of manycomponents engineered by different people.Systems that are designed to support resource sharing in this way are termed opendistributed systems to emphasize the fact that they are extensible. They may beextended at the hardware level by the addition of computers to the network and atthe software level by the introduction of new services and the reimplementation ofold ones, enabling application programs to share resources. A further benefit thatis often cited for open systems is their independence from individual vendors.To summarize:

• Open systems are characterized by the fact that their key interfaces (or API)are published.

• Open distributed systems are based on the provision of a uniform communi-cation mechanism and published interfaces for access to shared resources.

• Open distributed systems can be constructed from heterogeneous hardwareand software, possibly from different vendors. But the conformance of eachcomponent to the published standard must be carefully tested and verified ifthe system is to work correctly.


1.2.3 Security

Many of the information resources that are made available and maintained in dis-tributed systems have a high value to their users. Their security is therefore ofconsiderable importance. Security for information resources has three components:confidentiality (protection against disclosure to unauthorized individuals), integrity(protection against alteration or corruption), and availability (protection againstinterference with the means to access the resources).In a distributed system, clients send requests to access data managed by servers,which involves sending information in messages over a network. For example:

• A doctor might request access to hospital patient data or send additions tothat data.

• In electronic commerce and banking, users send their credit card numbersacross the Internet.

In both examples, the challenge is to send sensitive information in a message over anetwork in a secure manner. But security is not just a matter of hiding the contentsof messages from the exterior – it involves also knowing for sure the identity of theuser on whose behalf a message was sent. In the first example, the server needs toknow that the user is really a doctor, and in the second example, the user needsto be sure of the identity of the shop or bank with which they are dealing. Thesecond challenge here is to identify a remote user or other agent correctly. Both ofthese challenges can be met by the use of encryption techniques developed for thispurpose.

1.2.4 Scalability

Distributed systems operate effectively and efficiently at many different scales, rang-ing from a small intranet to the Internet. A system is described as scalable if itwill remain effective when there is a significant increase in the number of resourcesand the number of users. The number of computers and servers in the Internet hasincreased dramatically these last years.The design of scalable distributed systems presents the following challenges:


Controlling the cost of physical resources As the demand for a resource grows,it should be possible to extend the system, at reasonable cost, to meet it. Itmust be possible to add server computers to avoid the performance bottleneckthat would arise if a single server had to handle all requests. In general, for asystem with n users to be scalable, the quantity of physical resources requiredto support them should be at most O(n)1 – that is, proportional to n. Forexample, if a single server can support 20 users, then two such servers shouldbe able to support 40 users. Although that sounds an obvious goal, it is notnecessarily easy to achieve in practice.

Controlling the performance loss For example, consider a set of data whosesize is proportional to the number of users in the system. Algorithms thatuse hierarchic structures (like trees, heaps) scale better than those that uselinear structures (lists). But even with hierarchic structures an increase in sizewill result in some loss in performance: the time taken to access hierarchicallystructured data is O(log n), where n is the size of the set of data. For a systemto be scalable, the maximum performance loss should be no worse than this.

Preventing software resources running out An example of lack of scalabilityis shown by the numbers used as Internet (IP) addresses (computer addressesin the Internet). In late 1970s, it was decided to use 32 bits for this purpose,and now the supply of available Internet addresses is running out. For thisreason, a new version of the protocol with 128-bit Internet addresses is beingadopted, and this will require modifications to many software components. Itis difficult to predict the demand that will be put on a system years ahead.

Some shared resources are accessed very frequently; for example, many users mayaccess the same resource causing a decline in performance. Caching and replicationmay be used to improve the performance of resources that are very heavily used.Ideally, the system and application software should not need to change when thescale of the system increases, but this is difficult to achieve. The issue of scale is adominant theme in the development of distributed systems. The use of replicateddata, the use of caching and the deployment of multiple servers to handle commonlyperformed tasks, enabling several similar tasks to be performed concurrently.

1http://en.wikipedia.org/wiki/Big_O_notation


Figure 1.1: Comparison of constant O(1), linear O(n), logarithmic O(log n) andquadratic O(n2) running times


1.2.5 Failure handling

Computer systems fail aways. Computer systems fail! Not sometimes, but always.Whether its a bug in the software, a bug in the underlying system software, andhardware failure, someone pulls the plug on the server, power goes out, naturaldisaster wipes out the Eastern seaboard, etc, something will take a system down.Failures in a distributed system are partial – that is, some components fail whileothers continue to function. Therefore the handling of failures is particularly diffi-cult.

Detecting failures Some failures can be detected. For example, checksums canbe used to detect corrupted data in a message or a file. It is difficult or evenimpossible to detect some other failures, such as a remote crashed server in theInternet. The challenge is to manage in the presence of failures that cannotbe detected but may be suspected.

Tolerating failures It would not be practical for most of the services of a dis-tributed system to attempt to detect and hide all of the failures that mightoccur. Their clients can be designed to tolerate failures, which generally in-volves the users tolerating them as well.

Recovery from failures Recovery involves the design of software so that the stateof permanent data can be recovered or "rolled back" after a server has crashed.

Masking failures Some failures that have been detected can be hidden or madeless severe. Just dropping a message that is corrupted is an example of makinga fault less severe – it could be retransmitted. Two examples of hiding failures:

1. Messages can be retransmitted when they fail to arrive (as the retrans-mission in data networks)2

2. File data can be written to a pair of disks so that if one is corrupted, theother may still be correct (redundancy).

Redundancy Services can be created to tolerate failures by the use of redundantcomponents.

Distributed systems are amongst the systems that provide a high degree of avail-ability in the face of hardware faults. The availability of a system is a measure of

2http://en.wikipedia.org/wiki/Retransmission_(data_networks)


the proportion of time that it is available for use. When one of the componentsin a distributed system fails, only the work that was using the failed component isaffected.

1.2.6 Concurrency

Both services and applications provide resources that can be shared by clients in adistributed system. Therefore there is a possibility that several clients will attemptto access a shared resource at the same time.The process that manages a shared resource could take one client request at a time.But that approach limits throughput. Therefore services and applications generallyallow multiple client requests to be processed concurrently. Any shared resource ina distributed system must be responsible for ensuring that it operates correctly ina concurrent environment.

1.2.7 Transparency

Transparency is defined as hiding from the user and the programmer the separationof components in a distributed system, so that the system is perceived as a wholerather than as a collection of independent components (as stated in our definitionof a distributed system).We can identify eight forms of transparency, some being more meaningful thanothers depending on the nature of the system:

Access transparency enables local and remote resources to be accessed usingidentical operations.

Location transparency enables resources to be accessed without knowledge oftheir physical or network location (for example, which building or IP address).

Concurrency transparency enables several processes to operate concurrently us-ing shared resources without interfering between them.

Replication transparency enables multiple instances of resources to be used toincrease reliability and performance without knowledge of the replicas by usersor application programmers.


Failure transparency hides faults, allowing users and application to completetheir tasks despite of the failure of hardware or software components.

Mobility transparency allows the movement of resources and clients within asystem without affecting the operation of users or programs.

Performance transparency allows the system to be reconfigured to improve per-formance as loads vary.

Scaling transparency allows the system and applications to expand in scale with-out any change to the system structure or the application algorithms.

The two most important transparencies that characterize distributed systems areaccess and location transparency; their presence or absence most strongly affectsthe utilization of distributed resources. They are sometimes referred to together asnetwork transparency.Transparency hides and make anonymous the resources that are not of direct rele-vance to the task in hand for users and application programmers.

1.2.8 Quality of service

The main nonfunctional properties of systems that affect the quality of the serviceexperienced by clients and users are reliability, security and performance. Adapt-ability to meet changing system configurations and resource availability has beenrecognized as a further important aspect of service quality.Reliability and security issues are critical in the design of most computer systems.The performance aspect of quality of service was originally defined in terms ofresponsiveness and computational throughput, but it has been redefined in termsof ability to meet guarantees in time.Some applications, including multimedia applications like telephony systems, handletime-critical data – streams of data that are required to be processed or transferredfrom one process to another at a fixed rate. For example, a phone service mightconsist of a client program that is retrieving an audio flux from the server represent-ing the voice of the other person. For a satisfactory result the successive audio dataneed to be streamed to the user in real-time and without hashing or interferencesfor the conversation to be audible and of good quality.


In fact, the abbreviation QoS has effectively been commandeered to refer to theability of systems to meet such deadlines. Its achievement depends upon the avail-ability of the necessary computing and network resources at the appropriate times.This implies a requirement for the system to provide guaranteed computing andcommunication resources that are sufficient to enable applications to complete eachtask on time.The networks commonly used today have high performance but when networks areheavily loaded their performance can deteriorate, and no guarantees are provided.QoS applies to operating systems as well as networks. Each critical resource must bereserved by the applications that require QoS, and there must be resource managersthat provide guarantees. like additional available servers to handle spikes of load.

1.3 Fallacies

Inexperienced programmers and external observers tend to make wrong assumptionsabout distributed systems and their usage of the network.

1. The network is reliable.

2. Latency is zero.

3. Bandwidth is infinite.

4. The network is secure.

5. Topology doesn’t change.

6. There is one administrator.

7. Transport cost is zero.

8. The network is homogeneous.

The network is not reliable: it may be congested at peak hours resulting in a slowtraffic, latencies or packet loss. It may be shut down by an operator without no-tice. The path for a packet goes through different network equipments and varioustopologies that can alter or block the message being sent: proxies, firewalls etc. In-correct usage of message retransmission while packets are lost due to a misuse of the


network may lead to bandwidth waste and bottlenecks, resulting in a price increaseof the bandwidth usage. Network security is an entire art in itself, as the networkis an open battlefield. Malicious users and programs continually adapt to mewsecurity measures to eavesdrop on network traffic using various techniques: man-in-the-middle attack, spoofing etc. Multiple administrators may institute conflictingpolicies of which senders of network traffic must be aware in order to complete theirdesired paths.

Chapter 2

Trends and future

2.1 The rise of cloud computing

Distributed systems are undergoing a period of significant change and this can befound in a number of influential trends:

• the emergence of ubiquitous computing1

• the increasing demand for multimedia services

• the view of distributed systems as a utility, or cloud computing

With the increasing maturity of distributed systems infrastructure, a number ofcompanies are promoting the view of distributed resources as a commodity or util-ity, drawing the analogy between distributed resources and other utilities such aswater or electricity. With this model, resources are provided by appropriate servicesuppliers and effectively rented rather than owned by the end user. This modelapplies to both physical resources and more logical services:

Physical resources such as storage and processing can be made available to net-worked computers, removing the need to own such resources on their own. Atone end of the spectrum, a user may opt for a remote storage facility for filestorage requirements (for example, for multimedia data such as photographs,

1http://www.ubiq.com/ubicomp/

16

CHAPTER 2. TRENDS AND FUTURE 17

music or video) and/or for backups. Similarly, this approach would enablea user to rent one or more computational nodes, either to meet their basiccomputing needs or indeed to perform distributed computation. At the otherend of the spectrum, users can access sophisticated data centers (networkedfacilities offering access to repositories of often large volumes of data to usersor organizations) or indeed computational infrastructure using the sort of ser-vices now provided by companies such as Amazon and Google. Operatingsystem virtualization is a key enabling technology for this approach, implyingthat users may actually be provided with services by a virtual rather than aphysical node. This offers greater flexibility to the service supplier in termsof resource management and operational costs.

Software services can also be made available across the Internet using this ap-proach. Indeed, many companies now offer a comprehensive range of servicesfor effective rental, including services such as email and distributed calendars.Google, for example, bundles a range of business services under the bannerGoogle Apps2.

The term cloud computing is used to capture this vision of computing as a utility.A cloud is defined as a set of Internet-based application, storage and computingservices sufficient to support most users’ needs, thus enabling them to largely ortotally dispense with local data storage and application software. The term alsopromotes a view of everything as a service, from physical or virtual infrastructurethrough to software, often paid for on a per-usage basis rather than purchased.Note that cloud computing reduces requirements on users’ devices, allowing verysimple desktop or portable devices to access a potentially wide range of resourcesand services.Clouds are generally implemented on cluster computers to provide the necessaryscale and performance required by such services. A cluster computer is a set ofinterconnected computers that cooperate closely to provide a single, integrated high-performance computing capability.

2.2 How Google uses distributed systems

Google, the market leader in web search technology, has put significant effort intothe design of a sophisticated distributed system infrastructure to support search

2http://www.google.com/a

CHAPTER 2. TRENDS AND FUTURE 18

(and indeed other Google applications and services such as Google Earth). Thisrepresents one of the largest and most complex distributed systems installations ofthe history of computing and hence demands close examination that are unfortu-nately out of scope of this paper. The reader can find below the highlights of thisdistributed infrastructure:

• An underlying physical infrastructure consisting of very large numbers of net-worked computers located at data centers all around the world3

• A distributed file system designed to support very large files and heavily opti-mized for the style of usage required by search and other Google applications(especially reading from files at high and sustained rates)4

• An associated structured distributed storage system that offers fast access tovery large datasets5

• A lock service that offers distributed system functions such as distributedlocking and agreement6

• A programming model that supports the management of very large paralleland distributed computations across the underlying physical infrastructure7

3http://www.datacenterknowledge.com/archives/2011/08/01/report-google-uses-about-900000-servers/

4http://labs.google.com/papers/gfs.html5http://labs.google.com/papers/bigtable.html6http://labs.google.com/papers/chubby.html7http://labs.google.com/papers/mapreduce.html

Part II

2600hz, an heterogeneousdistributed platform

19

Chapter 3

Presentation of the solution

3.1 Whistle, the cornerstone of cloud-enabled VoIP

I worked as a software engineer on Whistle, a scalable, distributed, cloud-basedsoftware used to build powerful telephony applications with a rich set of APIs. Itis the cornerstone of cloud-enabled VoIP networks, and can be described as a logiclayer for telephony soft switches. As a component, Whistle is part of a bigger,heterogeneous system: 2600hz (which is also the name of the company that provideWhistle).2600hz allows VoIP integrators and resellers to build Class 51 grade switching ap-plications quickly and easily on any infrastructure. This platform is the vision ofexperienced VoIP engineers who came up with a response to the scalability prob-lems faced by VoIP providers and integrators. Most VoIP service providers createhome-grown tools to manage their systems. As their needs and customer demandsgrow, their toolset falls behind and becomes a pain point.Whistle, a component of 2600hz I developed modules for, is composed of two com-ponents: Ecallmanager and Applications (or WhApps).

• WhApps are the logic blocks, self-contained units of functionality of Whistle.Each WhApp is in charge of a particular task in the system: trunking, users,call flows, devices, voicemail boxes etc.

1http://en.wikipedia.org/wiki/Class_5_telephone_switch

20

CHAPTER 3. PRESENTATION OF THE SOLUTION 21

• Ecallmanager is a low coupling2, translation layer between Applications andthe underlying soft switch. Each soft switch (YATE, FreeSWITCH, Asterisk)needs its own translation layer to be able to communicate with WhApps.

Whistle is written in Erlang, a concurrent programming language designed to buildmassively scalable, distributed applications. It is an open-source, API enabled plat-form, meaning anyone can consume Whistle’s APIs and integrate its own WhAppin the system to solve his problem and add value to the product.

Figure 3.1: Whistle is made of two components: Ecallmanager and Applications(WhApps). WhApps are the logic blocks of Whistle.

The source code of Whistle is available on GitHub3.

3.1.1 Applications

The following is an excerpt of some WhApps bundled with Whistle4, and what theirrole consist in:

Callflow Control mechanisms for handling calls.2http://en.wikipedia.org/wiki/Coupling_(computer_science)3https://github.com/2600hz/whistle4https://github.com/2600hz/whistle/tree/master/whistle_apps/apps


CDR Call Detail Records are generated by FreeSWITCH at the end of a call. Theycontains information like who called who, what number, duration of the calletc. Those records are useful for billing.

Conference The Conference WhApp is used to create, modify or delete conferencerooms.

Crossbar Crossbar is a general purpose interface layer for Whistle, exposing Whis-tle APIs as an HTP REST API.

DTH Integration with a VoIP billing software provider5

Registrar Stores and caches the registrations of SIP devices in the network.

Trunkstore Brings the ability to start a trunking platform with ease.

3.2 2600hz, a powerful distributed communica-tion stack

As stated previously, Whistle is a layer, and needs to be connected to other com-ponents to form an heterogeneous distributed system: 2600hz. The componentsWhistle needs are:

• SIP servers

• Telephony soft switches

• Messaging bus

• Data stores

I qualify 2600hz as an heterogeneous because it contains many different kinds ofhardware and software working together in a cooperative fashion to solve problems.6.The components used in the reference implementation of 2600hz are the following:

5http://dthvoipbilling.com/6http://infolab.stanford.edu/~burback/dadl/node95.html


SIP server OpenSIPS acts as a load balancer for the SIP requests. The goal isto minimize the number of public network interfaces needed to inform clientsand carriers of by pointing them to the load balancers (usually two for redun-dancy). Adding capacity becomes as easy as informing openSIPS of the newswitch to balance on.

Switch FreeSWITCH is used in this layer because of the tight integration obtainedbetween Whistle and FreeSWITCH via the mod_erlang_event7 module. AsFreeSWITCH is already a carrier-grade switch on its own, bringing the clus-tering features of Whistle on top result in a high-quality cluster of switcheson which to build the platform.

Control layer Whistle provides an abstraction layer to the underlying switch-ing layer. Application developers can program their applications against theWhistle APIs and know that Whistle will take care of the details. Applicationdevelopers also benefit from Whistle’s ability to distribute processing amongstthe servers in the switching layer. To the application developer, Whistle isone logical switch. Whistle is composed of Ecallmanager and WhApps.

Message bus Conversations between servers are primarily conducted using a stan-dard protocol named AMQP. RabbitMQ8 was chosen as an AMQP implemen-tation because it is a full implementation of the protocol, and it’s written inErlang. Thus it allows to keep everything in native Erlang data types, passthings around quickly, and cluster Whistle and WhApps servers easily. It alsoimplements all the needed brokers out-of-the-box9.

Data store BigCouch10 is a CouchDB (a document-oriented database) cluster. Itis a premium choice as data store because of the distribution and redundancycapacities offered natively by BigCouch. With proper configuration, data isautomatically partitioned, replicated across the different nodes.

7http://wiki.freeswitch.org/wiki/Mod_erlang_event#Introduction8http://www.rabbitmq.com/9Brokers are discussed in section 5.6

10https://github.com/cloudant/bigcouch


Figure 3.2: Whistle’s components, Ecallmanager and WhApps, never communicatedirectly: they exchange messages using an AMQP bus. Several type of communi-cation are taking place in the system: AMQP messages, Erlang messages and othertraffic like TCP.

3.3 Solving vertical scaling limitations with dis-tributed systems

Scalability is the ability of a system to handle growing amounts of work in a gracefulmanner, or its ability to be enlarged to accommodate that growth.11. 2600hz was

11Definition from wikipedia http://en.wikipedia.org/wiki/Scalability


built as a solution to scalability problems12 in telephony applications. Usually,vertical scalability is the way to scale an application running on a single-server:replace current servers or hardware by more powerful servers and hardware to handlethe growth. But this way of scaling has few expansion possibilities, since hardwarehas physical limitations.Horizontal scalability involve the addition of more computational power (or nodes)in the system to manage the additional quantity. As an example, if we have 3FreeSWITCH server, able to each handle 25.000 concurrent calls, and the wholesystem concurrent calls grow to to 80.000 on average, we should be able to deploy afourth FreeSWITCH server, increasing the system’s capacities to 100.000 concurrentcalls.Distributed systems achieve horizontal scalability better than client-server architec-ture by nature (nodes, message-passing), which explains why 2600hz ’s architectureis distributed.

12Refer to section 1.2.4

Chapter 4

Programming distributed systems

A massive distributed infrastructure that must also be scalable is not easy to con-ceive. It must be envisioned from start to be distributed and scalable, it’s notsomething that can be thought of on the way. In the case of Whistle, scalabilitymust not be a blocker, that being the reason the selected architecture is distributedinstead of client-server.As a distributed architecture brings challenges in the design of the application (as Ishowed in section 1.2) the choice of the main programming language for the devel-opment of the platform is critical. Few programming languages are truly built fordistributed programming. Any language capable of communicating over the networkcan be used to build a distributed program, but some languages are more appro-priate because they include features aimed at distributed systems (messaging, faulttolerance, portability, resilience, asynchronous communication, node discovery).The most used languages for industry-level back-end systems (telecom, banking,aeronautics) are C++ and Java. These languages were not built for distributed pro-gramming because their concurrency model and communication model were basedon shared memory. The features they were missing, especially messaging, wereadded later either at the language level or as an implementation of a standardizedcommunication protocol:

• RMI1 and Jini2 are APIs that allow distributed Java programming, but areavailable only for Java programs.

1http://en.wikipedia.org/wiki/Java_remote_method_invocation2http://en.wikipedia.org/wiki/Jini

26

CHAPTER 4. PROGRAMMING DISTRIBUTED SYSTEMS 27

• CORBA (Common Object Request Broker Architecture)3 is a standard thatallow different software components written in different programming languageto communicate. CORBA does not meet all the requirements of DistributedComputing like redundancy.

• Message Passing Interface (MPI)4 is a communication protocol based on message-passing for the purpose of parallel programming, implemented as extensionsin C++, C#, Java, Python etc.

I would like to point out the difference between parallel computing and distributedcomputing.

• In parallel computing, all processors have access to a shared memory. Sharedmemory can be used to exchange information between processors.

• In distributed computing, each processor has its own private memory (dis-tributed memory). Information is exchanged by passing messages betweenthe processors.

Figure 4.1: In a shared memory environment (a), all CPUs read and write data fromthe same memory unit, while in a message-passing environment (b) each CPU isbound to a local memory and never read or alter others’ and communicate sendingmessages to a message bus.

It is unlikely that parallel programming can be done in a distributed environment,since physical memory is not shared among all the nodes of the system.

3http://en.wikipedia.org/wiki/Corba4http://en.wikipedia.org/wiki/Message_Passing_Interface


4.1 Erlang, concurrent programming for distributedapplications

Another programming language than C++ or Java has been chosen for the develop-ment of Whistle: Erlang. Erlang is a general-purpose concurrent, garbage-collectedprogramming language and runtime system. The sequential subset of Erlang is afunctional language, with strict evaluation, single assignment, and dynamic typing.It was designed by Ericsson to support distributed, fault-tolerant, soft-real-time,non-stop applications. It supports hot swapping, so that code can be changed with-out stopping a system. Hot swapping is particularly suitable for high availabilitysystems like telephony systems.It was developed to solve the "time-to-market" requirements of distributed, fault-tolerant5, massively concurrent6, soft real-time systems, and was designed with theaim of improving the development of telephony applications.Erlang presents several advantages for the development of distributed, concurrentapplications and is portable. The low-level nature of C++ does not offer this levelof portability, and developing Whistle in C++ would have required as many com-pilations as the computer architecture and operating systems Whistle is supposedto run in. A distributed system can be heterogeneous at hardware and operatingsystem levels, and portable source code is highly desirable so efforts can be focusedon development and not solving portability problems.

4.1.1 Open Telecom Platform

OTP is simultaneously a framework, a set of libraries, and a methodology for struc-turing applications; it’s an extension of Erlang. These are some of the main advan-tages of OTP:

Productivity Using OTP makes it possible to produce production-quality systemsin a very short time using behaviors.

Stability Code written on top of OTP can focus on the logic and avoid error-proneimplementations of the typical things that every real-world system needs: pro-cess management, servers, state machines, etc.

5See section 1.2.561.2.5


Supervision The application structure provided by the framework makes it simpleto supervise and control the running systems, both automatically and throughgraphical user interfaces.

Upgradability The framework provides patterns for handling code upgrades in asystematic way.

Reliable code base The code for the OTP framework is rock solid and has beenthoroughly battle tested.

OTP behaviors are a formalization of process design patterns. They are imple-mented in library modules that are provided with the standard Erlang distribution.These library modules do all of the generic process work and error handling. Thespecific code is written by the programmer.

4.1.2 Strengths

One of the greatest strength of Erlang for scalable, soft real-time communicationsystems is its concurrency model. Erlang concurrency is fast and scalable, and canhandle high loads with no degradation in throughput, even during sustained peaks.Telephony systems must not suffer from call peaks and be reliable.Rather than providing threads that share memory, each Erlang process executes inits own memory space and owns its own heap and stack. Processes can’t interferewith each other. There are no locks, no synchronized methods, and no possibilityof shared memory corruption, since there is no shared memory, which ensure a highstability of the platform, hence lowered downtimes.Erlang programs can be made from thousands to millions of extremely lightweightprocesses that can run on a single processor, can run on a multicore processor, orcan run on a network of processors. Unlike Java or C++, concurrency in Erlangbelongs to the programming language and not the operating system, which makesErlang portable over different platforms (UNIX/Linux, Windows, Mac OS) andarchitectures (x64, x86).I have to insist on the importance of multicores CPUs of Erlang and the scalabilityadvantage it gives to Whistle. Erlang scales accordingly to the number of coresavailable on the machine it’s running on. Nowadays, CPU vendors have reachedthe frequency limit a single-core CPU can reach. Instead of having a single, highfrequency core. CPU vendors are creating CPUs with 4 or more cores of lesser


frequency. Some servers on the market have up to 32 cores. Being able to scaleon number of core is unheard of other programming languages, and programmersare now starting to learn now how to program multi-core systems to anticipate themarket of multi-core processors.7.

Figure 4.2: Because Erlang uses concurrency at language at level and not the OSlevel, processing distribution over several CPU cores is achieved without efforts.X-axis is the number of connected clients in an Erlang application, Y-axis is theaverage execution time.

Lastly, functional programming forbids code with side effects, which Erlang imple-ments as referential transparency. Referential transparency requires the same resultsfor a given set of inputs at any point in time thus a referentially transparent ex-pression is therefore deterministic by definition. Bugs and errors in a deterministicsystem are easier to reproduce than in a non-deterministic one.

4.1.3 Weaknesses

But Erlang has also weaknesses, although most seems subjectives from a developer’spoint of view.

7http://www.intel.com/intelpress/sum_mcp.htm


Syntax Erlang’s syntax is based on Prolog, which can be daunting at first andhave nothing in common with syntaxes of C inspired programming languageslike C++, Java, PHP, Python ...

Strings Erlang has no built-in type for strings. Strings are represented as a list ofintegers, which can degrade performance on large data sets and makes Erlangslower at intensive data processing. C++ is a better usage for this type ofcomputation.

Code organization Erlang’s namespace is flat, which can lead to naming conflictsor messy code base.

Documentation Documentation is complete but lacks example. A person wantingto learn Erlang will have to buy books, because the documentation only coverthe essentials and is not learning friendly.

4.1.4 Erlang in industry-level communication systems

• T-Mobile uses Erlang in its SMS and authentication systems.

• Motorola is using Erlang in call processing products in the public-safety in-dustry.

• Ericsson uses Erlang in its support nodes, used in GPRS and 3G mobilenetworks worldwide.

• Facebook is using Erlang in the back-end of its chat system.

The fact that web services, retail and commercial banking, computer telephony,messaging systems, and enterprise integration, to mention but a few, happen toshare the same requirements as telecom systems explains why Erlang is gainingheadway in these sectors.

Chapter 5

Architecture and integration

2600hz is distributed and connected. Such a system requires a thorough designprocess and must be architected from a high-level point of view to view to ensurethe correct implementation of distributed principles, especially those who add valuefrom a business perspective1:

• Scalability.

• Openness and extensibility.

• Supervision and monitoring

Software projects have a long history of failure2, and naturally, the more complexthe system is, higher are the rates of failure, especially after the software has beenbrought to market and problems arise while in production: impossibility to scalebeyond a certain point, hard to maintain, to extend etc.

1which features will be sold to customers2http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.

php

32

CHAPTER 5. ARCHITECTURE AND INTEGRATION 33

5.1 Functional overview

Figure 5.1: Once Whistle (included in the designated cloud engine) is integratedin 2600hz, and 2600hz is fully operational, the platform acts as the basis of acommunication platform

2600hz is designed to build a VoIP Cloud infrastructure easily and to scale easily,with a capacity of 1 billion calls per months. This vision of a distributed VoIPsystem came from its creators, who are experimented VoIP engineers, as the bet-ter architecture to confront scalability problems. While distributed systems havetheir own requirements and principles, a resellable communication platform has itsbusiness requirements and features too, because this is what customers are going to


build their businesses on, and buy training and support for. Architectural choiceare made accordingly to the business objectives to fulfill and the requirements tomeet in order to build such a system: high scalability, reliability, massive concur-rence on the first hand, and unified communications, app store capacities, reducedoperational costs in the second handFrom an architect’s point of view, a scalable, distributed platform is characterizedby:

Messaging A way for programs across different servers to talk to each other and/orknow what other servers are doing

Redundancy The ability to have copies of everything (data, software, etc.) allworking together at the same time, with copies coming online/offline at anygiven time

Distribution of data The ability to break information into pieces and spread itacross multiple computers, allowing for adding/removing computers as de-mand requires

Unlimited concurrency There should be no limit to how much is happening onthe platform overall

The above are common characteristics. But telecommunication systems have addi-tional requirements that will be implemented as features in the platform:

Directed Events The ability for message queues across servers to be set up andtear down quickly and to act as a tunnel between different explicit serviceswithout disrupting other nodes

Schema Flexibility The ability to frequently upgrade data and variable structureswithin the entire system without bringing down clusters.

Strong Supervision The ability to detect failures very quickly and re-spawn nodesand processes just as fast. In web servers, delays and failures of 50ms or moreare acceptable - in voice applications, they’re ultimately not

Speed for Adding Features Telecom is growing extremely rapidly. The abil-ity to expose new features quickly, in a reliable, scalable, distributed way isparamount to a successful platform


Fast Server Provisioning The ability to handle spikes in the cloud by procuringand provisioning additional resources (from servers to circuits to DIDs) in aninstant

Avoid downtimes Downtimes are critical in live communication systems. Noteven upgrades to software should result in downtime.

As Whistle has to integrate seamlessly with the other components of a 2600hz, someof the problems we have to solve with this distributed architecture are

• How will 2600hz use messaging to communicate with other components ?

• How will 2600hz handle failure ?

• How will 2600hz scale ?

• How will 2600hz achieve high concurrency ?

5.2 Messaging

A way for programs across different servers to talk to each other and/orknow what other servers are doing.

Message passing is the form of interprocess communication that is the most suitablein distributed systems. In traditional concurrent application built in C++ or Java,the interprocess communication model is largely based on shared-memory, which isthe simultaneous access in the same data in memory by different process.Even without specifically referring to distributed systems, the shared-memory modelis known for its complexity and problems when facing high concurrency. Althoughsolutions exists (mutexes, semaphores) to control access to the same information andwho can modify it, problems quickly arise, like dead-locks and race conditions ormemory corruption. In certain case, the existence of a global lock eases concurrentprogramming by removing synchronization and locking problems, but also severelydegrade performance of the system, especially on multicore architecture. As anexample the GIL3 in the C implementation of Python degrades performance ofPython on a multicore platform.

3http://en.wikipedia.org/wiki/Global_Interpreter_Lock


Processes using shared-memory for communication are very hard to scale, since theinformation is located in a unique point in the network: scaling the processes meansscaling the memory at the same time, leading to other problems : state distribution,coherence of the replicated data, consistency, failure of replication.Message passing can be used in two modes: synchronous and asynchronous.

• Synchronous messages are sent from process A to B. Process A will wait (orblock) for a response from B to continue. Synchronous exchange is used fortransactions, confirmation or cooperation between processes.

• Asynchronous messages are sent from process A to B. As soon as the messageis sent from A, A continues its execution: it doesn’t block. B receives themessage and does not have to acknowledge the reception of the message. Itcan be seen as a fire and forget process.

Although Erlang has language-level primitives for exchanging messages, using anopen standard messaging-oriented protocol, AMQP, ensures the platform stays opento third-party developers, and not let Erlang being a blocker to the extension ofthe system. In a distributed environment like 2600hz, message-passing is clearlythe way for processes to exchange and communicate. For example, Ecallmanagerand Applications may run on different servers. They will communicate exchangingmessages

5.3 Redundancy and fault tolerance

The ability to have copies of everything (data, software, etc.) all workingtogether at the same time, with copies coming online/offline at any giventime.

Distributed systems are hard to design, especially because of the risk of failurethat can happen on any node of the system. As discussed in the design section ofdistributed systems, reliability is a key point and is highly important. Users expectstheir phone to be usable at any time of the day, and independently of the numberof users already connected to the system and the load on it that could slow globaltraffic.


Whistle uses redundancy at physical and software level. Physical redundancy is theeasiest to achieve, especially with loosely coupled components like Ecallmanager andWhApps. Ecallmanager can be installed on different physical servers, and setup tocommunicate with the same of different FreeSWITCH. Which Ecallmanager willrespond in based on a first arrived, first served: the first Ecallmanager to pickup arequest will respond.A good example of software redundancy usage in Whistle is the use of BigCouch.BigCouch is a cluster of CouchDB4 database that are distributed over an arbitrarynumber of servers. Each node of BigCouch is capable of replication with othernodes, which ensures data redundancy as nodes goes up and down.

5.4 Distribution of data

The ability to break information into pieces and spread it across multiplecomputers, allowing for adding/removing computers as demand requiresand failures appear.

When a system scales, each of its components must scale, otherwise the scalabilityof the whole system will be the limited by the scalability of its less scalable com-ponent (Weakest link theory). Distribution of data is hard to achieve by nature.Distribution of data requires synchronization and replication amongst data stores.The choice of the data store is thus critical.Distribution of data is bundled in BigCouch, the data store used by Whistle. Eachnode can be setup on a different server, and BigCouch will automatically spreaddata across those node based on consistent hashing algorithms which will ensuredata is evenly spread and quickly accessible it’s sharding, or horizontal partitioningof databases.BigCouch uses 3 constants to allow fine-grained configuration of data distributionfor performance, consistency and durability.

• N - replication constant. N copies of each document will be written to thedata store, on N different nodes.

4CouchDB is a document oriented database


Figure 5.2: BigCouch spread data across its nodes using consistent hashing.

• R - read quorum constant. N writes have occurred for each document, as notedabove. When reads are requested, N reads are sent to the N nodes that storethe particular document. The system will return to the requesting client withthe document when R successful reads have returned, and agree on versioning.Lower R values often result in faster reads at the expense of consistency.Higher R values usually result in slower reads, but more consistent, or agreed-upon data values returning.

• W - write quorum constant. When writing the N copies, the data store willrespond to the write client after W successful writes have completed. Theremaining N-W writes are still being attempted in the background, but theclient receives a 201 Created status and can resume execution. Lower Wvalues mean more write throughput, and higher W values mean more data


durability.

Horizontal scaling’s simplicity becomes clear: the more node there are, the greaterthe storage capacity expands without impacting lookup times. This kind of distri-bution does not exist in traditional databases like MySQL or SQL Server, or haveto be implemented as sharding. Achieving horizontal scalability using those datastores in hard since the system is not built for data distribution. Sharding anddistribution are achieved using replication (master-slave replication) with typicalproblems or client-server architectures: locks, replication speed etc.The two following theorems allowed us to make assumption and trade-offs on ourdata distribution and consistency over the long term.

5.4.1 CAP Theorem

Eric Brewer stated5 2000 that it is impossible for a distributed system to provideat the same the three following guarantees

Consistency All nodes see the same data at the same time

Availability Every request receive a response, whether it was successful or failed

Partition Tolerance The system continues to operate despite arbitrary messageloss

According to the theorem, a distributed system can satisfy any two of these guar-antees at the same time, but not all three. The use of BigCouch and the choicethis particular NoSQL solution is for its abilities to scale horizontally because of itsdata-partitioning schema. This leaves either Consistency or Availability as a secondguarantee. BigCouch is partition-tolerant by nature (replication), and trades con-sistency for availability. Since data must be replicated to nodes and the clients arenot waiting for updates to be able to read from the database, the system is alwaysavailable while not being consistent. If the clients had to wait for a consistent stateto query the database, it would be a trade-off of availability for consistency. ButBigCouch relies on eventual consistency to ultimately be consistent.

5http://www.cs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf


Figure 5.3: Visual guide to NoSQL systems from the perspective of the CAPtheorem. A distributed system can achieve only two of the three following:availability, consistency, partition-tolerance. Source: http://blog.nahurst.com/visual-guide-to-nosql-systems

5.4.2 Eventual consistency

I would like to point here that there is a way consistency can be attained in anAP (available, partition-tolerant) system like BigCouch. Given a sufficiently longperiod of time over which no changes are sent, all updates can be expected topropagate eventually through the system and all the replicas will be consistent.CouchDB achieves eventual consistency6 between multiple databases by using in-

6http://guide.couchdb.org/draft/consistency.html


cremental replication. Incremental replication is a process where document changesare periodically copied between servers.

5.5 Unlimited concurrency

There should be no limit to how much is happening on the platformoverall

Using Erlang allows Whistle to handle massive concurrency. As Erlang concurrencyis based on lightweight processes (300 words at minimum), spawning a huge numberof processes will not impact system’s performance, and can increase performance ona high factor when running on a multi core processor. Erlang does not make use ofthe underlying OS’s threads and processes for managing its own process pool andthus does not suffer from these limitations.In Erlang:

• Creating and destroying processes is very fast

• Sending messages between processes is very fast

• Processes behave the same way on all operating systems

• There can be a very large numbers of processes

Due to the immutability of Erlang data structures, there is no need to lock andit makes easier to parallelize tasks. While executing tasks in Whistle (processingcalls, data processing), the ability of spawning several process increase drasticallythe execution time, and avoiding blocking is critical in a soft real-time system.Ulf Wiger, CTO of Erlang Solutions, a consulting firm for Erlang, stated7:

In order for concurrency to be useful as a fundamental modeling paradigm,the programmer must feel that processes are cheap enough you can effi-ciently create as many processes as the problem calls for. If there is anysingle defining characteristic of Erlang-style concurrency, it is that youshould be able to model your application after the natural concurrency

7http://ulf.wiger.net/weblog/2008/02/06/what-is-erlang-style-concurrency/


patterns present in your problem. If creating a process is perceived asexpensive, programmers will reuse existing processes instead; if messagepassing is perceived to be expensive, various techniques will be inventedto avoid sending messages.

As I indicated in the chart pointing out the scalability of Erlang on multi-corearchitecture, unlimited concurrency can be achieved by using multicore CPUs inthe system’s servers. The benefits from multicore CPUs will be noticeable since twomajor components of the system are written in Erlang: Whistle, the logic layer andBigCouch, the data store.

5.6 Directed events

The ability for message queues across boxes to be spun up and downquickly and to act as a tunnel between different explicit services withoutdisrupting other nodes.

Directed events in Whistle are based on the messaging capacities of the system.AMQP (Advanced Messaging Queue Protocol) does message orientation, queueing,and routing (point to point or publish/subscribe), in a reliable and secure way.Several AMQP implementation are available, and RabbitMQ is an implementation Irecommend because it is open-source and written in Erlang. This avoid the manage-ment of components in our system written in different languages, and benefit fromthe expertise of Erlang of the members of the team to solve problems that may arisewith RabbitMQ. Other implementations are either commercial (StormMQ) or donot fully implement the AMQP specification (ZeroMQ).As messaging is used across organizations and inside systems, different patterns havesurfaced to structure the way message exchange like design patterns for softwaredesign. Messaging patterns are a reusable solutions to common problems, and werelisted first in the Enterprise Integration Patterns8 book.The AMQP model describes four different entities

• Broker8http://eaipatterns.com/Messaging.html


• Queues

• Exchanges

• Messages

The broker is the server, which manages queues and exchanges. A queue acts likea mailbox and holds messages that are sent by producers. Exchanges are logicswitches which define the routing and filtering for messages. Messages are sentfrom producers to consumers over the network.

Figure 5.4: AMQP describes different entities that compose the system

I used 3 different exchange type while developing modules for Whistle and interact-ing with Ecallmanager:

Fanout 1:N delivery involving no routing.


Direct 1:1 straight delivery

Topic Route messages using a key to consumer based on their subscription to thatkey

A particularly interesting example of how AMQP is used is between Ecallmanagerand WhApps. Different WhApps needs different data, while at the same timedifferent WhApps needs to be notified of the same event. When a new voicemail isleft by the caller, FreeSWITCH will transfer this message to Ecallmanager. DifferentWhApps, like the ones on charge of storing the message in the data store or sendinga text transcription of that message to the callee, will be bound to the exchangewhere Ecallmanager published voicemail. Using a routing key to subscribe only tonew messages, WhApps will receive automatically the new message.

Figure 5.5: Events can be routed to the right consumer using routing keys

5.7 Schema flexibility

The ability to frequently upgrade data and variable structures withinthe entire system without bringing down clusters.

The choice of a NoSQL solution like BigCouch (a CouchDB cluster) was based ondata storage types and by and our needs. Contrary to traditional SQL solutions


like MySQL, PostgreSQL or Oracle Database, CouchDB stores data in document asJSON structures, and those structures can be retrieved using the unique documentID generated at insertion time. As users needs evolves rapidly, being tied to aparticular data schema could be a drawback because of low flexibility. Data thatwas not needed today could be so tomorrow or next year.I considered and compared different NoSQL solutions, and as for programminglanguages, there is the right solution for the right need, and neither NoSQL or SQLare silver bullets. Whistle’s data is loosely coupled: a phone is associated with auser, a call flow is associated to a device, a voicemail message is associated to avoicemail box ... we need basic relationship, such as Foreign Keys in SQL systems.A fully relational system brings a lot of features not needed (joins, keys). As ofall this complexity is not needed in Whistle, the choice of a lighter solution, morescalable, is appropriate.I would like to point out the fact that CouchDB (and BigCouch by extension) alsocommits to the ACID9 properties, which brings some of the benefits of SQL intothe NoSQL world from the transaction perspective.

9Properties of transactions in traditional SQL systems: Atomicity, Consistency, Isolation, Dura-bility


5.8 Strong supervision

The ability to detect failures *very* quickly and re-spawn nodes andprocesses just as fast. In web servers, delays and failures of 50ms ormore are acceptable - in voice applications, they’re not.

Erlang, the language used to build Whistle, brings supervision from its OTP pack-age. Erlang has a philosophy of "Let it crash", meaning that with correct processsupervision, crashed process will be restarted. Process in Erlang can be of two type:Worker or Supervisor. Supervisor can supervise other supervisors and workers. Idiscussed in chapter 1.2.5 how failure tolerance is important, and Erlang is perfectis this role.

Figure 5.6: Crashed Erlang processes can be restarted by supervisors according toa restart strategy

Different restart strategies exist, but two are important: one for one, one for alland . In a one for one strategy, if a child process terminates, only that process isrestarted. In a one for all strategy, if a child process stops, all other child under thissupervisor are terminated and then all children are restarted. A restart frequencycan also be defined, so a child that constantly crashes because of an unmanageableproblem will be put aside and reportedWhy letting crash instead of trying to handle all possible exceptions and manuallyrestart ? First, a programmer cannot foresee all the problems that may arise in asystem. When a process crashes, the source of the error is logged and due to theimmutability of data structures, the error can be replayed and quickly fixed.


5.9 Speed for adding features

Telecom is growing extremely rapidly. The ability to expose new fea-tures quickly, in a reliable, scalable, distributed way is paramount to asuccessful platform.

This is a direct implementation of the principle of Openness as I showed in 1.2.2.As the market and user needs grows, the platform must stay available for extension.Customers’ need and expectations are important, and they will check out at com-petitors if what they want if not quickly implemented. A distributed system mustgrow in a manageable fashion, because development and features addition neverstops. As time passes, the technical debt must be limited by being careful with thecode added to the product: unit testing, integration testing.The modularity of Whistle is a key principle that allows quick additions of features,and exposure of those features through public API. Whistle APIs are exposed viaa REST API because of the ubiquity of HTTP capable programming languages.REST (REpresentational State Transfer) has gained significant popularity theserecent years10, mainly for the simplicity offered and the very high range of pro-gramming language able to implement this type of API. Any client capable of doingHTTP request is capable of doing REST calls. SOAP clients needs an implemen-tation of the SOAP specification, which can limit the capacities of the client: outof date implementation, missing parts of the specifications, incompatibility betweendifferent versions.The WhApp in charge exposing public API is called Crossbar. Crossbar acts as ageneral purpose layer between Whistle APIs and the outside world. Clients maychose to implement the API as they want: creating a UI interface for a browser adesktop based application, from a script program, as a phone application etc.

5.10 Fast server provisioning

The ability to handle spikes “in the cloud” by procuring and provisioningadditional resources in an instant

10http://blog.programmableweb.com/2010/08/13/api-anti-patterns-how-to-avoid-common-rest-mistakes/


As distributed systems are more and more seen as an utility, real-time provisioningis an important. Whistle deployment for clients is easy: they can deploy moreservers for the components of the systems that are under high charge (Ecallmanager?BigCouch? WhApps? FreeSWITCH?), and rearrange their provisioning needs asthey see it.2600hz is deployable on major cloud-hosting providers, like Amazon EC211 or Rackspace12,using the deployment tool developed by 2600hz13.One of the advantage of deploying applications in the Cloud instead of maintaininga custom infrastructure is cost reduction, to the order of 20%14. Moreover, it isimportant to note that the client does not need to know how to setup a server,how to manage it, how to scale it. He only needs to deploy his software using adeployment tool, and the cloud hosting provider will take care of the rest. Hiringan operation team can be costly, especially for small startups and companies whichIT is not the primary business.

5.11 Avoid downtimes

The ability to avoid all downtimes. Not even upgrades to software shouldresult in downtime.

Time has shown that distributed systems built in Erlang are very resilient andreliable. The language has been battle tested in real large scale industrial products,like the AXD301 switch in 1998, containing over a million lines of Erlang, andreported to achieve a reliability of 99.999% (5.26 minutes downtime / year). Whistlehas not yet been deployed on large scale systems and I do not have numbers toreport, but performances other companies achieved with Erlang looks promising forWhistle.Outages are costly on all plans: the system is not available to customers, customersget a bad image of the product and the company, and money stops flowing in. Moredangerous is when other companies are relying on your service to provide theirs,

11http://aws.amazon.com/ec2/12http://www.rackspace.com/13Available at http://apps.2600hz.com/14http://www.informationweek.com/news/cloud-computing/software/229218825


as for hosting providers. Amazon cloud-hosting service EC2 had a downtime for 3hours last year, costing Amazon $55.000 per minute of outage.15.Usually upgrading a system requires a downtime: disconnect users, setup the up-date, shut down, restart, ensure everything is going well and the update was suc-cessfully deployed. These downtimes are also costly and can threaten the system ifthe deployed update has problems and has not been thoroughly tested in a testingenvironment before deployment.Erlang features hot code swapping. Hot code swapping allows, in theory, any Erlangdistributed system to run indefinitely. Updated code can be deployed in a runningWhistle system without interrupting it. Note that code swapping is only possiblefor Erlang module using the same interface. Behind the scene, a redirection of theprocess using the old module is done to the new, thanks to the immutability ofErlang data structures.

Figure 5.7: Erlang modules can be swapped during runtime, allowing uninterruptedupgrades

15http://www.socaltech.com/catchpoint__amazon_down_for_3_hours/s-0029569.html

Chapter 6

Functional and technicalspecification of a module

6.1 What is hot desking?

One of my development was the implementation of a hot desking module. Hotdesking is a module allowing itinerant users, like sales people, to register on anyphone on the network. All calls made to that person will be routed to this phone,and his voicemail box will be accessible from this phone. How can this achieve thisusing the openness and component distribution of 2600hz ?The user dials a feature code on the phone, like *91. He is prompted ID and PINnumber, and is logged in if credentials matches his profile in database. When he isdone, the user dials another feature code, *92. He is logged out of this phone.The implementation of this feature must be seen as how it will integrate with thedifferent component of the system (phone switches, data stores, messaging bus,Whistle).

• It needs to read the numbers dialed by the users: ID and PIN

• It needs to send sounds to the user’s phone: "Login incorrect", "You have beenlogged in"

• It must be able to locate the user in the data store and read his profile:compare stored ID and PIN with user input

50

CHAPTER 6. FUNCTIONAL AND TECHNICAL SPECIFICATION OF A MODULE51

• It needs to redirect calls to the phone the user registered into.

6.2 Integration in 2600hz

The hot desk profile (like ID, PIN, roaming) for a user will be configured usingthe Crossbar WhApp. Crossbar exposes Whistle APIs through a REST interface,accessible from any client HTTP-enabled.Hot desk profile needs to be persisted and associated to the user profile information.BigCouch is in charge of storing this data, and will store this information with theUser record.Any action that involves dialing a number and sending actions (Press 1 for this,press 2 for that) is represented as a module of the Call flow WhApp. A Call flowmodule acts like a state-machine and describes the succession of event that will beproposed to the user. A number to dial has to be associated to a callow to ring it.Any call flow module can retrieve user’s actions on the dial pad and stream soundsto the phone like prompts thanks to Ecallmanager.Call flow modules communicates with Ecallmanager by sending AMQP messagesthrough the messaging bus, and listens for messages sent by FreeSWITCH to Ecall-manager (which contains users’ action for example).


Figure 6.1: Flow diagram of actions required to log in a hot desk


Figure 6.2: Flow diagram of actions required to log out a hot desk


Figure 6.3: Functional diagram of hot desking integration. Once a user has regis-tered its hot desk profile using the hot desk crossbar module, he can be reached onany phone he logged into using his hot desk credentials. A lookup on the device heis logged in is done from the call flow number

Part III

Actors of today’s cloud-enabledVoIP

55

Chapter 7

2600hz, a disruptive startupcompany

7.1 Presentation2600hz is focused on massively scalable cloud telecom infrastructures.

Geared towards integrators and carrier resellers, they provide the mostresilient, secure and cost-effective alternative to expensive and limitedtraditional phone systems. They also provide an open source commu-nications software that allows any company to become its own reliabletelecom carrier while slashing operational costs in half.

2600hz is a startup company headquartered in San Francisco. Founded in 2009 byCEO Darren Schreiber and COO Patrick Sullivan, the company aims at providingopen-source communication softwares and telephony consulting.

Private content removed

56

Conclusions

The integration of software components in an heterogeneous distributed environ-ment is challenging and requires a well thought design and architecture. From theswitch to the HTTP APIs, 2600hz successfully managed to integrate components ofvarious nature (phone switches, data stores, messaging-bus) while ensuring it canscale and no component could potentially become a bottleneck as the platform andthe number of users grow. No business wants to step down because it cannot handlethe massive influx of clients due to a huge success of its product.The core principles of distributed systems, failure tolerance, concurrency and mes-saging, were architected from start. Erlang proved to be an excellent choice becauseof the distribution oriented features it offers at language level it offers: concurrency,supervision. AMQP, as an open standard used by banks and majors institutions, isa de facto reference for messaging in distributed environments. It ensures 2600hzopenness and extension for the future, especially from the third-party developed ap-plications, which can bring useful features to the platform and create an app storeecosystem. Failure is well-tolerated by Whistle, due to redundancy at all levels:from switch to data store nodes.My learnings and decisions in the design and integration of module in Whistle weresuccessful, because my research and the solutions I proposed were innovative (likethe JSON schema validator). As I joined the Erlang team while the project wasalready well-advanced, I supported this effort by adding more features while beingaware and applying the distributed design principles and integration I demonstratedto the work I did, because each developer has to embrace the same vision for thesystem to continue to grow in the same direction. Knowing how to architectingdistributed systems is a highly sought skill, because this is not only about VoIP,but any software solution that could benefit from a distributed design if it facesscalability problems.Whistle is still young, yet its future is bright because the VoIP market segment it is

57

CHAPTER 7. 2600HZ, A DISRUPTIVE STARTUP COMPANY 58

on has few direct competitors, and the innovative solutions (distributed architecture,openness, marketplace) it brought apart from scalability are highly demanded byresellers and integrator.

Part IV

Personal insight

59

60

7.2 San Francisco, startups and entrepreneurship

Working in San Francisco, in the tech industry and in a startup is an enriching andunique experience that can’t be experimented elsewhere. San Francisco and theSilicon Valley are major centers of innovation in the US, and these ecosystems andthe way they work inspired me for the future. It raised my interests in startupsand entrepreneurship. Everything goes so fast within a startup, and what will bebuilt in 10 years in a big company can be build in less than half this time in astartup because of the absence of bureaucracy and individual initiatives. I find thisway more appealing and exciting. During my learnings on startups, I read PaulGraham’s essays1 about the entrepreneurial spirit, and how to grow it, and learnedso much from it.In the short term, I want to continue to work in San Francisco or the Silicon Valley. Istill have things to learn, on the programming side as well as on the startup side, andhow to transform ideas in reality. We had entrepreneurship courses at SUPINFO,from which I learned a lot on the theory, but I need real experience from now. Ihave to meet with tech entrepreneurs, and listen to their story and their advices toget inspiration. And in a few years, when I will be able to bring my ideas to realityand identify a problem to solve, I will probably launch my own start up, whetherit’s in the US or in France.What encourages me to continue is this way is that you can change the world andthe way people think and do. I would be very proud to have this kind of impact,and glad to help other people with their problems.With $10000 you can launch your own company. The tech industry is lucky, becauseyou don’t need to acquire a lot of physical assets to launch your business, only anidea with good programmers using free and open-source technologies. The majorityof tech companies in San Francisco and the Silicon Valley runs on open-sourcesoftware, which drastically lowers the costs. As I am interested in launching myown company in the future, being confronted to the success and problems a startupcan meet in such a dynamic environment is a priceless experience.

1http://www.paulgraham.com/articles.html

61

Figure 7.1: Startups are in a continuous learning process: from a minimum viableproduct, they receive user feedback and measure users needs to refine the productin the next iteration

I discovered that failure and risk taking are major parts of the entrepreneurial spiritof the Silicon Valley, contrary to France where failure is seen as a breakdown andseverely compromise your chance for launching your next project as an entrepreneur.Startups companies are very agile and adapt constantly, and do not hesitate topivot if they realize that their initial vision needs refinement, or is not adaptedto customer’s need. The unit of progress for entrepreneurs here is learning, notexecution. As a continuous learner, I recognize myself in this profile.As Eric Ries, the author of The Lean Startup said:

Most technology start-ups fail not because the technology doesn’t work,but because they’re making something that there is no market for

Before, I tended to think that technology and feature-building was playing a greatpart of the start-up success, and if it was just an updated version on an existingproduct but with a new technology and/or more features, it would work, but I waswrong. You will be able to sell your product or service only if customers need yourproduct and if you solve a problem they have and they are willing to pay for.

62

7.3 Erlang and distributed systems, from theoryto practice

Before joining 2600hz, I knew some theory about distributed systems, few fromSUPINFO a few from personal learnings. When I jumped in the project, I hadlittle to no experience developing in Erlang. People at 2600hz trusted me and mymotivation, and I’ve been able to contribute to this project. I learned Erlang to apoint that I can implement my own modules. I learned Erlang/OTP and how toarchitect a software solution in the way Whistle has been conceived: distributedand connected.As more and more applications are going to be designed in a distributed way tobenefit from all the advantages I exposed, the skills I acquired are invaluable andwill serve as a foundation for my future, either as a programmer or as a startupfounder.I faced limitations though: core VoIP knowledge. I am not very strong at VoIP,and sometimes, when implementing VoIP features, I felt frustrated by the VoIP re-quirements and a lack of advanced skills needed to implement the feature in Erlang.VoIP was more a limiting factor than Erlang.As my internship was in an american company, I improved my english skills, writtenor spoken. I learned to communicate in the most efficiently possible way and to beclear in my thoughts so I am understood.

Bibliography

[1] 2600hz company’s blog. http://blog.2600hz.com/.

[2] 2600hz company’s wiki. http://wiki.2600hz.com.

[3] Joe Armstrong. Making reliable distributed systems in the presence of softwareerrors. PhD thesis, Swedish Institute of Computer Science, 2003.

[4] Francesco Cesarini and Simon Thompson. Erlang Programming. O’Reilly Media,Inc., 2009.

[5] George Coulouris, Jean Dollimore, Tim Kindberg, and Gordon Blair. DistributedSystems, Concepts and Design. Addison-Wesley, 5th edition, 2011.

[6] Google Code University. Introduction to distributed system design,http://code.google.com/edu/parallel/dsd-tutorial.html.

63

integration of software components in an heterogeneous distributed environment

Documents