author's personal copy · author's personal copy the related issues. in[10,24,25], the...

This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institution

and sharing with colleagues.

Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to personal, institutional or third party

websites are prohibited.

In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website orinstitutional repository. Authors requiring further information

regarding Elsevier’s archiving and manuscript policies areencouraged to visit:

http://www.elsevier.com/copyright

http://www.elsevier.com/copyright

Author's personal copy

Resource planning and packet forwarding in multi-radio,multi-mode, multi-channel, multi-rate (M4) wireless mesh networks q

Ting-Yu Lin a,*, Wai-Hong Tam b, Kang-Lun Fan a, Yu-Chee Tseng b

a Department of Communication Engineering, National Chiao-Tung University, 1001 University Road, Hsinchu 300, Taiwanb Department of Computer Science, National Chiao-Tung University, 1001 University Road, Hsinchu 300, Taiwan

Available online 8 February 2008

Abstract

Most earlier works in the area of wireless mesh network assume a single interface being equipped in each node. In this paper, we considerthe next-generation wireless mesh networks in which each node may be equipped with multiple radio interfaces, each capable of running inone of several modes (IEEE 802.11b/g 2.4 GHz or 802.11a 5 GHz mode), one of several channels, and each capable of supporting multiplemodulations. We call such a network an M4 (multi-radio, multi-mode, multi-channel, multi-rate) wireless mesh network. For example, fromoff-the-shelf components, one can easily construct a mesh node with multiple IEEE 802.11a/b/g radio interfaces. Our goal is to address theresource planning and packet forwarding issues in such an environment. The proposed methodology is based on linear programming withnetwork flow principles and radio channel access/interference models. Given a network topology, traffic requirements, and gateway capac-ities, we show how to allocate network interface cards and their channels to fully utilize channel bandwidths. The results can be utilized by awireless Internet service provider to plan their networks under a hardware constraint so as to maximize their profits. To the best of our knowl-edge, this is the first work addressing resource planning in a wireless mesh network. Our numerical results show significant improvement interms of aggregate network throughput with moderate network-layer fairness. The importance of network planning is further corroboratedby the simulative comparisons with other multi-radio systems assuming a known and fixed number of interfaces at each mesh router.� 2008 Elsevier B.V. All rights reserved.

Keywords: Resource planning; Routing; Channel assignment; Linear programming; Wireless ad hoc network; Wireless mesh network

1. Introduction

The wireless mesh network (WMN) is a promising solu-tion to the last-mile wireless Internet access problem. It caneffectively complement the limitation of WLAN coverage.Applications of WMN include enterprise wireless back-bones and community networks [15]. In [5], two mesh hier-archies are defined: infrastructure mesh and client mesh,where the former has much less mobility than the latter.Ref. [14] points out that a WMN may suffer from the sca-lability problem as the network grows due to the conten-tion and interference among hosts. To mitigate thescalability problem, one may explore advanced transmis-sion technologies (such as smart or MIMO antennas

[11,17,22]) or layer-2 or layer-3 solutions based on com-modity radio modules [3,6,8,12,9,16,18,19,21]. Severalworks show how to increase WMN capacity by adaptivelyadjusting the data rates [4,7,13,20].

In this work, we adopt the latter approach based oncommodity components. We explore the possibility ofmulti-interface, multi-channel model. For example, IEEE802.11a/b/g has 12/3/3 non-overlapping channels avail-able. One can easily make a multi-interface mesh node byoff-the-shelf components.1 Several works have addressed

0140-3664/$ - see front matter � 2008 Elsevier B.V. All rights reserved.

doi:10.1016/j.comcom.2008.01.059

q The research of Ting-Yu Lin is sponsored by the NSC of Taiwanunder Grant No. NSC96-2221-E-009-017.

* Corresponding author. Tel.: +886 921 591603.E-mail address: [email protected] (T.-Y. Lin).

1 With the advance of communication hardware technology, and cost-reduced networking modules, nowadays computing devices are oftencapable of operating/communicating on/through different radio frequen-cies (e.g., WiFi/Bluetooth/WCDMA possibly readily available at a singlelaptop, which may be installed with another WiFi card via the PCMCIAinterface). Hence, equipping multiple wireless interfaces at a single host isgetting affordable and its popularity can be expected in next-generationwireless-enabled computers.

www.elsevier.com/locate/comcom

Available online at www.sciencedirect.com

Computer Communications 31 (2008) 1329–1342


the related issues. In [10,24,25], the authors propose to usea dedicated interface running on a control channel to nego-tiate the data channels to be used by other interfaces. Refs.[3,6,8,12,16,18,19] propose to treat interfaces equally andsome channel assignment techniques are used to exploitspatial reuse.

The above works all assume that the number of inter-faces in each mesh node is given. In this paper, we addressthe resource planning problem in a multi-radio multi-modemulti-channel multi-rate (M4) wireless mesh network. Ourapproach is based on linear programming. Based on thewell-known IEEE 802.11 channel contention model, wecompute the near-optimal number of radio modules thatshould be equipped in each node and the channel thatshould be bound with each interface. We present tworesource management and channel assignment algorithms:Decremental Interface Management (DIM) and Incremen-tal Interface Management (IIM).

Our ultimate goal is to maximize the traffic volume in/out of Internet gateways of the mesh network, under therestrictions of network topology (connectivity status),available resources, and user’s traffic needs. We summarizeour contributions as follows:

� Instead of considering only a single factor, our approachaddresses all practical characteristics of wireless commu-nications, including the available non-overlapping radiochannels and the interference factors among neighbor-ing mesh nodes.� Resources are allocated to mesh nodes based on user’s

traffic requirements, available hardware/radio modules,and gateway capacities. We allow nodes to have differ-ent numbers of radio interfaces. Not only addressingthe related multi-channel issues, we also provide a guide-line to wisely distribute the deployment costs consider-ing an optimized network system. To the best of ourknowledge, this is the first work addressing resourceplanning in wireless mesh networks.� In order to enable simultaneous traffic incoming/outgo-

ing through different radio modules of the same meshhost, we propose to perform multi-path packet forward-ing (data flow splitting) to further exploit the benefits ofhaving multiple transceivers. This idea will be elabo-rated in more detail in Section 3.4.

The remaining paper is organized as below. Section 2reviews past related work in M4 wireless mesh networks.In Section 3, we introduce the M4 network architecture,

our linear programming model for network optimization,two resource management and channel assignment algo-rithms, and our packet forwarding strategy. Section 4 pre-sents the numerical and simulation comparison results.Finally Section 5 draws our conclusions and future plans.

2. Related work

The design of multi-channel WMN has been investi-gated in several works [3,8,9,12,16,18,19,21]. These workstreat all channels equally based on the IEEE 802.11MAC mechanisms and have a goal of minimizing the con-tention among wireless links. A single-transceiver model isassumed in [9,21], while a multi-transceiver model isadopted in [3,6,8,12,16,18,19]. For a single-transceiver sys-tem, the radio interface in each node needs to switchamong channels. This will results in the multi-channel hid-

den-terminal problem [21]. So the authors in [21] proposedto embed a negotiation phase in the ATIM (Ad Hoc TrafficIndication Map) window that is periodically sent under thePower Save Mode (PSM). Every node has to go to a pre-defined control channel when entering the ATIM window.The negotiation phase is to determine data channel to beused after the ATIM window finishes. The Slotted SeededChannel Hopping (SSCH) mechanism [9] divides the timeaxis into virtual channels. Each virtual channel’s hoppingsequence is determined by a (channel, seed) pair. Whenevera sender wishes to communicate with a neighbor, it changesits hopping schedule to the receiver’s in the correspondingvirtual channel. SSCH requires a looser time synchroniza-tion than [21], but its channel switching overhead is high.

Refs. [3,8] pointed out the advantage of equipping mul-tiple radio interfaces on a single mesh node. Such systemcan alleviate both inter-route and inter-hop contentions.As shown in Fig. 1, two data paths A-B-D and A-C-E bothoperating on channel 1 cannot be active at the same time,due to inter-route contention. The two data flows can bemade active simultaneously by adding one more interfacebinding to channel 2 on node A and switching nodes C,E to operate on channel 2, removing the inter-route conten-tion (Fig. 1(a)). However, since wireless links A-B and B-Duse the same channel 1 along the A-B-D route, the conten-tion between consecutive hops remains. By adding anotherradio modules on nodes B, C and switching nodes D, E tochannels 3, 4, respectively, the inter-hop contention prob-lem can be further eliminated, so that links A-B, B-D, A-C, and C-E now become interference-disjoint and can bemade active simultaneously (Fig. 1(b)). Note that designing

11 1

1

A E

D

C

B

12 4

3

A E

D

C

B

12 2

1

A E

D

C

B

a b

Fig. 1. Illustration of the multi-radio benefits (the number associated with each edge indicates the channel number used): (a) enabling simultaneoustransmissions between routes (inter-route contention removed); (b) further enabling simultaneous transmissions between consecutive hops along a route(inter-hop contention eliminated).

1330 T.-Y. Lin et al. / Computer Communications 31 (2008) 1329–1342


multi-radio multi-channel protocols is non-trivial, requir-ing not only methods to assign channels intelligently, butalso cautions to take care of network connectivity. Further-more, in case that interface switching between channels isnecessary in any proposed protocols, the multi-channelhidden-terminal problem, as characterized in [9], shouldbe treated carefully to avoid degrading performance dueto packet collisions.

Several works have addressed multi-channel MAC/rout-ing protocols based on IEEE 802.11 [6,12,16,18,19]. In [16],the authors suggested to associate a radio interface to afixed channel and use the remaining interfaces to switchamong other channels. The fixed channel is determinedby each mesh node so as to evenly distribute all availablechannels in a neighborhood. Whenever a sender wishes tocommunicate, it tunes one of its switchable interfaces tooperate on the receiver’s fixed channel. In addition to inter-face assignment strategy, Ref. [16] also proposed a multi-channel routing protocol (MCR) considering the costs ofinterface switching and channel diversity when selecting aroute. Ideally, the MCR protocol assigns the most cost-effi-cient route to each communication flow, so that multiplewireless links operating on different channels along theselected route can be active simultaneously (inter-hop con-tention problem considered). However, since routes aredetermined independently in MCR, the inter-route conten-tion problem, as explained in Fig. 1, remains unaddressed.

In [12], a Multi-Radio Link-Quality Source Routing(MR-LQSR) protocol is proposed. The goal of this proto-col is to discover a high-throughput route between commu-nicating entities in a multi-radio multi-hop wireless meshnetwork. The authors defined a Weighted CumulativeExpected Transmission Time (WCETT) metric for pathselection. Using WCETT, one may predict the transmissionefficiency considering packet data rates and channel diver-sity along a single route. As a result, the inter-hop conten-tion problem as characterized in Fig. 1 is handled by theproposed routing protocol. However, since routes are eval-uated individually, contention between communicationflows (inter-route contention) is also ignored in this work,lacking a global optimization for the inherently coopera-tive wireless network system.

In [18,19], also targeting on a multi-radio multi-channelwireless mesh environment, the authors proposed a set ofcentralized [19] and distributed [18] channel assignmentand routing algorithms. Since channel assignment androuting tend to affect each other, the proposed protocolrepeats the two processes periodically to check if the cur-rent settings, including channel binding and route selection,meet the requirements of traffic loads and inherent wirelesslink capacities. The load-aware channel assignment algo-rithm does not require any specific routing algorithm.Whenever no feasible route can be found by the routingprotocol, the procedure will go back to re-perform thechannel assignment algorithm, so as to find a reasonableroute for the data flow that fits in all wireless link capacitiesalong the path. This work considered both the inter-route

and inter-hop contention problems. However, there is noguarantee of finding an optimal channel setting for arbi-trary routing protocols, especially in real environmentswhere asymmetric links are common, transmission ratesvary a lot, and link quality is often different from channelto channel. In addition, like the previous two works[12,16], proposals in [18,19] also ignore the problem ofoptimizing the number of radio modules assigned to amesh host.

In a recent work [6], the authors proposed a joint chan-nel assignment and routing protocol to optimize the net-work throughput subject to fairness constraints. Theoptimization methodology is also based on linear program-ming. Given the interference model, number of availablechannels, aggregate user traffic demands, and number ofradios at each mesh router, [6] addresses the interference-free link scheduling, routing, and channel assignment prob-lems (RCL algorithm). We share the similar idea of trafficsplitting over multiple routing paths to achieve load bal-ancing by performing traffic engineering techniques as pro-posed by [6]. In RCL, all wireless channels are assumed tohave equally maximal data rate and gateways assumed tohave unlimited capacity, which are different from ourassumptions in this work. Additionally, [6] also uses anequal and given number of interfaces at each mesh routeras most previous works do, ignoring the problem of opti-mizing radio numbers for heterogeneous (forwarding) traf-fic requirements at mesh routers. We will report and discussthe performance comparison results through simulationexperiments in Section 4.3.

3. Resource planning in an M4 wireless mesh network

This section first defines the architecture of our M4 wire-less mesh network. Then we propose a linear programmingmodel to allocate radio interfaces to mesh nodes and bindchannels to these radio interfaces. Two schemes called Dec-remental Interface Management (DIM) and IncrementalInterface Management (IIM) are proposed. In Section3.4, we re-visit the contention problems as depicted inFig. 1, and propose a multi-path packet delivery function(mPDF) to further exploit the advantage of having multipleradios and channels.

3.1. Network architecture

We consider an M4 network as shown in Fig. 2. Eachmesh node is equipped with one or multiple wireless inter-faces. Each interface can operate in one of several modes.In this work, we consider IEEE 802.11a/b/g. Each antennacan be either omni-directional or directional. Also, aninterface can support multiple modulations with differenttransmission rates. It is assumed that an interface is capa-ble of selecting the best modulation depending on the chan-nel quality. We consider link asymmetry, in the sense thatthe transmission rate in one direction of a link could be dif-ferent from that of the other. The mesh network may have

T.-Y. Lin et al. / Computer Communications 31 (2008) 1329–1342 1331


multiple heterogeneous Internet gateways with differentbandwidths.

3.2. Linear programming model

To construct a cost-efficient M4 WMN, we need to allo-cate interfaces to nodes, assign channels to them, and bal-ance traffic loads among gateways. The network is modeledby a directed graph G ¼ ðV ;EÞ, where V is the set of meshnodes and E the set of wireless links. Note that E is deter-mined by how we allocate interfaces. We make the follow-ing assumptions and define several notations:

� There are totally N interfaces available.� The maximal number of non-interfering channels is C.� All user traffic is destined to the Internet. We assume

that each mesh node vi is associated with an uplink loadupper bound uu

i , a downlink load upper bound udi , an

uplink load lower bound lui , and a downlink load lower

bound ldi .

� A subset V g � V of mesh nodes are designated as Inter-net gateways and the remaining subset V h are designatedas hosts, that is, V ¼ V h [ V g. We assume that onlyhosts in V h generate traffic. In case gateways in V g havesome traffic demand, we can re-define the node set V.For example, we can transform the original networkarchitecture as Fig. 3 into a new graph as Fig. 4. We cre-ate two virtual gateway nodes, v10 and v11, which dealwith traffic relaying without generating traffic and haveunlimited bandwidth to/from neighboring hosts, v3

and v5. Note that the two figures have the same networkarchitecture. In addition, for each vm 2 V g, we use Bu

m

and Bdm to denote its uplink and downlink bandwidths,

respectively, to the Internet.� For each pair of neighboring hosts vi and vj, the best bit

rates from vi to vj and from vj to vi on channel k,k ¼ 1 . . . C, are denoted by fij½k� and fji½k�, respectively.2

Note that the existence of such wireless links between vi

and vj depends on how we allocate interfaces to vi andvj. If any of vi and vj does not have an interface on chan-nel k, we simply let fij½k� ¼ fji½k� ¼ 0. The best rates maydepend on factors such as signal quality, transmissiondistance, etc. For link asymmetry, it is not necessarythat fij½k� ¼ fji½k�.� Depending on how interfaces are allocated, we define

the set of wireless links operating on channel k asEk ¼ feijjfij½k� > 0g. As a result, the set of all wirelesslinks is E ¼ [C

k¼1Ek.� In order to represent how interfaces are allocated and

how channels are bound, we define a channel vector ci

for each host vi. For each element k in ci, k ¼ 1 . . . C:

ci½k� ¼1 if vi has an interface operating on channel k

0 otherwise:

�

Note that it makes no sense to bind multiple interfacesof a host to the same channel. So the number of inter-faces owned by vi is the cardinality of channel vectorci. In Section 3.3, we will discuss how to determine thisvector for each mesh node. Then we can define the con-

nectivity vector cij as an indication of connection statusbetween vi and vj. For each element k in cij, k ¼ 1 . . . C:

cij½k� ¼ci½k� � cj½k� if eij 2 Ek

0 otherwise:

(

� To formulate the channel contention behavior, we defineIEk

ij to be the set of links in the interfering range of linkeij that also use channel k: IEk

ij ¼ fepqjepq 2 Ek and oneof vp and vq is in the interfering range of vi or vjg. Forexample, one simple definition of interfering range isto include all vi’s and vj’s two-hop neighbors.3

� Now, we introduce some unknown variables in our lin-ear programming model. We define ku

i as the actualuplink traffic load delivered from node vi, and similarlykd

i as the actual downlink traffic load destined to node vi.

InternetInternet

T1 GPRS

DSL/Cable modemInternet

Fig. 2. An M4 mesh network with heterogeneous Internet gateways.

2 On measuring the best achievable bit rate from one mesh node to theother, one may utilize probing-based quality evaluation by testing allavailable physical channel rates and tracking the respective packet lossratios. In this work, we adopt the methodology proposed by [4] to obtainthe best bit rate for a wireless link.

3 Alternatively, some separate algorithm may be devised to establish theinterfering link set for every node pair.



� Next, we define xuij½s;k� as the actual uplink traffic gener-

ated by source node vs over wireless link eij using chan-nel k, and similarly xd

ij½d;k� as the downlink trafficforwarded to destination node vd over wireless link eij

using channel k. Moreover, we define xij½0;k� as the aggre-gate traffic load on wireless link eij using channel k,where xij½0;k� ¼

Pvs2V ðxu

ij½s;k� � cij½k�ÞþP

vd2V ðxdij½d;k� � cij½k�Þ.

� For each gateway host vm 2 V g, we define the aggregateuplink/downlink traffic via vm to be gout

m /ginm , where:

goutm ¼

Xvs2V

gouts;m; gin

m ¼Xvd2V

gind;m:

A summary of notations is given in Table 1.

Our ultimate goal is to maximize the mesh networkcapacity such that the traffic flowing in/out of the set ofgateways is the largest, without violating the traffic require-ment (upper and lower bounds) of each mesh node. Ourapproach is based on linear programming. The objectivefunction can be written as

MaximizeX

vm2V g

ðgoutm þ gin

mÞ;

subject to the following constraints:

(1) general constraints:

kui P lu

i ;kui 6 uu

i ;kdi P ld

i ;kdi 6 ud

i ;xuij½s;k�P 0; xd

ij½d;k�P 0;Xku

i ¼X

vm2V g

goutm ;

Xkd

i ¼X

vm2V g

ginm ;

(2) gateway constraint:

goutm þ gin

m 6 Bm if uplink and downlink share

the bandwidth

goutm 6 Bu

m; ginm 6 Bd

m otherwise:

8><>:

Due to the fact that radio channel bandwidth is sharedby all wireless links within the interfering range of edgeeij, we add one more constraint to reflect the channel modelbased on IEEE 802.11 DCF contention protocol:Xepq2IEk

ij

ðxpq½0;k�=fpq½k�Þ 6 1:

Finally, the linear programming constraints and flowconservation equations are summarized in Fig. 5(a) and(b), respectively.

3.3. Resource allocation and channel assignment techniques

In this section, we present two algorithms to distributeavailable radio modules and perform channel arrangement:Decremental Interface Management (DIM) and Incremental

v

V

e12

e21

e23

e32

e24

e42

e35

e53

e56

e65

e 74e 47

e48e

84

e25

e52

e69e

96

e78

e87

Gateway node

e45

e54

v v

v

v

v

v

B

B

V

Fig. 3. An example wireless mesh network architecture in graphrepresentation.

v1 v2 v5

v3

v4

v7 v8

v6

v9

v10

v11

Fig. 4. A simplified example graph with separate mesh hosts and Internetgateways.

Table 1Summary of notations: (a) parameters that are given and (b) parametersthat are to be determined

(a)uu

i Maximum uplink traffic load allowed at node vi (upper bound)lu

i Minimum uplink traffic load required at node vi (lower bound)ud

i Maximum downlink traffic load allowed at node vi (upperbound)

ldi Minimum downlink traffic load required at node vi (lower

bound)fij½k� Capacity of directional wireless link eij over channel k

Bm Capacity of (Ethernet/T1/T3) gateway node vm

Bum Uplink capacity of (xDSL/cable modem) gateway node vm

Bdm Downlink capacity of (xDSL/cable modem) gateway node vm

C Number of available non-interfering channelsCi½k� Channel boolean vector of node vi (ci½k� ¼ f0; 1gÞN Number of available sets of communication equipment

(b)lambdau

i Actual uplink traffic load delivered from node vi

lambdadi Actual downlink traffic load destined to node vi

X uij½sk� Uplink traffic flow generated by source node vs over wireless

link eij using channel k

X dij½sk� Downlink traffic flow forwarded to destination node vd over

wireless link eij using channel k

goutm Traffic flow out of gateway node vm gout

ginm Traffic flow back into gateway node vm gin



Interface Management (IIM). Our goal is to derive the chan-nel vector ci½k�, 8vi 2 V h, and feed it back into our linear pro-gramming (LP) model introduced in Section 3.2 to maximizenetwork throughput. Based on the two strategies, wedecrease/increase network interfaces step by step until allavailable modules are used up, solving the linear modelrepetitively. At the end of these algorithms, we can obtainni, the required number of IEEE 802.11a/b/g radios associ-ated with host vi (under the N limitation), in the followingway:

XC

k¼1

ci½k� ¼ ni;

whereP8vi2V h ni ¼ N .

Before we describe the two algorithms in more detail,Fig. 6 summarizes the inputs, outputs, and variables usedin the proposed DIM and IIM mechanisms.

The first proposed technique is Decremental InterfaceManagement (DIM), which starts from equipping eachmesh host with the maximal number of radio interfaces,i.e., C NICs, since C is the total number of non-overlap-ping channels. Assume that the number of available radiomodules N is insufficient to support C NICs on each meshhost. In addition, as we will observe in Section 4, it is notnecessary to use all the C interfaces equipped on each hostin order to achieve the maximal network throughput.Instead, several interfaces can be removed without degrad-ing the system, for there exist several wireless links overcertain channels with zero traffic flows based on our LP cal-

Fig. 5. Re-formatted linear programming (a) constraints and (b) flow conservation equations.

Fig. 6. Summary of inputs, outputs, and variables used in both the DIM and IIM procedures.



culation. In the proposed DIM algorithm, we first removethose useless interfaces and check if the total number ofNICs used satisfies the N limitation. If so, the algorithmterminates and returns the channel vector ci½k� along withcorresponding traffic distribution patterns for our packetdelivery function (mPDF), which will be presented laterin Section 3.4. Otherwise, we need to evaluate each NICand find out a least useful interface for removal from thesystem. This process is repeated until the total number ofused NICs meets the N requirement.

Now we present the interface evaluation strategyadopted by DIM. For each NIC operating on channel k

equipped on mesh host vi, we calculate the aggregate traffic(both uplink and downlink) an

i½k� handled by the interface asfollows:

ani½k� ¼

Xi6¼j;vj2V ;vs0 2V h

ðxuij½s0;k� þ xu

ji½s0;k�ÞþX

i 6¼j;vj2V ;vd0 2V h

ðxdij½d 0;k� þxd

ji½d 0;k�Þ;

ð1Þ

8vi 2 V h; 1 6 k 6 C. We hope to remove the NIC with thesmallest an

i½k�. However, to avoid removing the only inter-face that a mesh host has, we calculate the aggregate trafficah

i experienced by vi via all interfaces equipped on the hostin the following equation:

ahi ¼

XC

k¼1

ani½k�;

8vi 2 V h, and define wi½k� ¼ ani½k�=ah

i . Only those NICs withwi½k� < 1 will be considered for removal. Among those can-didate NICs, we remove the interface which yields the min-imum value of an

i½k� � wi½k�. All interfaces are evaluated andremoved one by one until the number of total used NICsbecomes equal to N. Fig. 7 provides a pseudo-code forthe DIM algorithm.

Next, we introduce the Incremental Interface Manage-ment (IIM) strategy. Initially, we deploy one NIC on eachmesh host, and bind the interface to operate on the best-condition channel. In other words, we test on all availablechannels, and choose the best channel, which produces themaximal network capacity based on our LP calculation.We then use the selected channel to construct a single-channel wireless mesh backbone as the initial phase inour IIM algorithm to avoid any performance bias due tobad initial channel selection.

Assume that N is larger than the network size jV hj, so asto realize a multi-radio system. Once the initial single-radiomesh has been optimized by our LP model, we start to addinterfaces one by one based on the LP results. This processwill be repeated until all N available NICs have been dis-tributed out.

Note that during the process of adding interfaces, wemay be unable to find a feasible LP solution due to insuf-ficient number of deployed NICs for supporting requireduser traffics. In this case, we repetitively reduce the trafficlower bounds (li) for both uplink and downlink at each

Fig. 7. Decremental Interface Management (DIM) algorithm pseudo-code.



mesh host vi in a exponential way (li ! li=2! li=4! � � �)until a feasible LP solution is discovered. The lower boundsare restored to obtain a new LP solution, after wirelesslinks are evaluated and more interfaces are added in.

Now we present the criteria for adding interfaces. Wehope to characterize the most congested wireless link soas to add interfaces binding to another channel for trafficrelief. For more accurate judgement, we recall the set IEk

ij

of interfering links for edge eij using channel k, and definenij½k� ¼ jIEk

ijj. We choose the edge eij with the maximumvalue of ðxij½0;k�=fij½k�Þ � nij½k� (refer to Section 3.2 for the def-initions of xij½0;k� and fij½k�) for adding interfaces on commu-nicating hosts vi and vj.

Once the most congested link is decided, we intend toselect a channel with the lightest traffic load within theneighborhood of selected edge eij. Obviously, we want toavoid choosing the channel that both hosts vi and vj

already have. As a result, for each candidate channel, wecalculate the aggregate link traffic ax

ij½k� for all links in IEkij,

where

axij½k� ¼

Xepq2IEk

ij

xpq½0;k�;

and the aggregate link capacity afij½k� of all links in IEk

ij asfollows:

afij½k� ¼

Xepq2IEk

ij

fpq½k�:

The IIM algorithm chooses the channel with the minimumvalue of ax

ij½k�=afij½k�, and add interfaces on hosts vi and vj

binding to the selected channel accordingly. A detailedpseudo-code for IIM algorithm is illustrated in Fig. 8.

3.4. Multi-path packet delivery function (mPDF)

As one may notice that, in the proposed linear program-ming model, we maximize the network throughput byenabling simultaneous transmissions/receiving over non-interfering channels. As explained previously in Fig. 1,the adoption of multiple radio modules on mesh hostscan effectively mitigate the inter-route and inter-hop con-tention problems. In this section, we re-visit the conceptof simultaneous communication actions, and point out thatour proposed methodology can further exploit the advan-tage of having multiple radios and channels to achieve anoptimized M4 WMN infrastructure.

In traditional single-radio single-channel WMNs, multi-path packet forwarding is not favorable since multipleinterference-disjoint paths are difficult to discover due tothe single-channel inter-route contention problem. As char-acterized in Fig. 1, by utilizing multiple radio modules onmesh hosts, the inter-route contention problem can be alle-viated, making the multi-path packet forwarding becomefeasible. As a result, in addition to enabling simultaneouscommunications between two flows, we propose to furthersplit traffic loads over multiple paths for a single flow.

Fig. 9 (a) shows the resulting radio and channel configura-tion. Suppose that route A-C-E is the original single path.We observe that, by adding one more radio on each ofnodes D and E binding to channel 5, we can enable twonon-interfering forwarding paths for simultaneous trans-missions for a single flow, as illustrated in Fig. 9(b). Therouting solution provided by our proposed LP model isactually a multi-path forwarding mechanism, to which werefer as the multi-path packet delivery function (mPDF).

Note that the multi-path problem is a subset of the inter-route contention problem. Multiple routes whether belong-ing to multiple flows or a single flow are possible to bemade active simultaneously in a multi-radio multi-channelenvironment. Though several upper-layer challenges,including packet re-ordering problem, still remain ques-tionable, in the proposed M4 WMN architecture, weobserve the potential of multi-path packet forwardingmechanism. The benefit of utilizing multiple routing pathsin multi-channel environments has been verified in our pre-vious work presented in [23]. We plan to investigate moreon the feasibility of implementing our mPDF protocol byperforming traffic engineering techniques in a real testbed.

4. Performance evaluation

This section provides performance results derived fromperforming our proposed resource allocation, channelarrangement algorithms, and multi-path packet deliveryfunction (mPDF) in an M4 WMN. We describe the net-work environment settings in Section 4.1, followed bynumerical and simulation comparison results reported inSections 4.2 and 4.3, respectively.

4.1. Network environment settings

We generate an M4 WMN in grid topology as illustratedin Fig. 10. All mesh nodes are assumed to be stationary andspaced 200 m apart from each other. We assume that thetransmission range is 250 m and the interference range is550 m in our network. The IEEE 802.11 MAC protocolwith RTS/CTS four-way handshaking mechanism isadopted in our channel contention model.

4.2. Numerical results

This subsection presents the numerical results. We adopta mixed integer linear programming (MIP) solving tool [1]to perform the LP calculation. In the following presenta-tion, we vary several critical parameters, including avail-able number of channels and radio interfaces, networksizes and configurations, gateway capacities, and effectivelink data rates to observe the feasibility of our proposedmethodology.

4.2.1. Varying number of available channels and interfacesIn this subsection, we first experiment on a 4� 4 grid

mesh with two Internet gateways located at the upper-left



and bottom-right corners separately. The IEEE 802.11benvironment with three orthogonal (non-interfering) chan-nels is considered. Assume that all mesh hosts have thesame traffic requirement for both uplink and downlinkdata flows. Denoted as U and L, the traffic upper boundand lower bound are set to be 5 and 0.2 Mbps, respectively.In addition, suppose that symmetric gateways are used,each with bandwidth capacity B equal to 100 Mbps, and

that all wireless links have the same bit rate F equal to5.5 Mbps. Fig. 11 shows the results for the DIM and IIMstrategies. As we can see from this figure, the aggregate net-work throughput grows as N and C increase. An interestingobservation is that, when three orthogonal channels arebeing used (C ¼ 3), both DIM and IIM yield four timesthe throughput of a single-channel system (C ¼ 1) by add-ing only 10 more (16 + 10 = 26 in total) network interfaces

Fig. 8. Incremental Interface Management (IIM) algorithm pseudo-code.



(i.e., 1.625 NICs per mesh host in average). In other words,to achieve the maximal network capacity with three chan-nels available, it is not necessary to equip each mesh hostwith three NICs for utilizing all available radio band-widths. Furthermore, as shown in Fig. 11, once thethroughput saturates at its maximum point, adding

network interfaces contribute little to the performance,since the bottleneck now lies in the number of orthogonalchannels C.

We conduct another experiment considering the IEEE802.11a environment, also in a 4� 4 grid topology havingtwo Internet gateways. Though with 12 non-overlappingchannels, IEEE 802.11a is conceived to have maximal eightorthogonal (non-interfering) channels available in manyareas around the world. As a result, we adopt this assump-tion and define related parameters U and L to be 20 and0.2 Mbps, respectively, B and F to be 500 and 24 Mbps,respectively. Fig. 12 plots the results. In this figure, weobserve that by using 54 network interfaces in total, aver-agely 3.4 NICs per mesh host, we can maximize the net-work throughput and fully utilize the total radiobandwidths that eight orthogonal channels can provide.This result is encouraging for we do not need to deploy alarge number of eight NICs on each mesh host, in orderto take advantage of all channel bandwidths. Note that dif-ferent network configurations and parameter settings willproduce various values of required N. In real WMN sys-tems, given user traffic requirements, network connectivityfunction, gateway capacities, and wireless link data rates,an optimal value for N should exist to achieve a reasonabledeployment cost.

4.2.2. Varying network configurations

Next, we investigate the impacts of different networkconfigurations on aggregate throughput. We vary the net-work configuration by changing F function, network size,

12 4

3

A E

D

C

B

12 4

3

A E

D

C

B

5

Fig. 9. The idea of traffic splitting for communication flow from sender Ato receiver E in the proposed multi-path packet delivery function (mPDF):(a) original single-path and (b) multi-path delivery by adding one moreradio module on each of nodes D and E binding to channel 5.

. . .

. . .

. . .

. . .

. . .

. . .

200m

. . .

. . .

200m

. . .

. . .

. . .

. . .

. . .

. . .

200m200m

. . .

. . .

200m

200m

Fig. 10. The mesh grid with Internet gateways located at the upper-leftand bottom-right corners.

4*4 grid, U=5, L=0.2, B=100, F=5.5, DIM

0

10

20

30

16 20 24 28 32Number of Available NICs (N)

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

C=1C=2C=3

4*4 grid, U=5, L=0.2, B=100, F=5.5, IIM

0

10

20

30

16 20 24 28 32Number of Available NICs (N)

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s) C=1C=2C=3

a b

Fig. 11. Aggregate network throughput vs. number of available radio interfaces for maximal three orthogonal channels in the IEEE 802.11b environmentusing (a) DIM and (b) IIM algorithms.

4*4 grid, U=20, L=0.2, B=500, F=24, DIM

0

50100150200250300350

16 24 32 40 48 56 64 72 80Number of Available NICs (N)

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

C=1C=2C=3C=4C=5C=6C=7C=8

4*4 grid, U=20, L=0.2, B=500, F=24, IIM

050

100150200250

300350


Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

C=1C=2C=3C=4C=5C=6C=7C=8

a b

Fig. 12. Aggregate network throughput vs. number of available radio interfaces for maximal eight orthogonal channels in the IEEE 802.11a environmentusing (a) DIM and (b) IIM algorithms.



and gateway bandwidth capacities. Below we report theresults in order.

Since in real environments, data rate differs from link tolink due to distinguished surroundings and channel condi-tions, we now remove the constant link capacity assump-tion, and let F uniformly distribute over the range of(0,24] Mbps. The rest of parameter settings is the same asin Fig. 12. Fig. 13 illustrates the results. Though with lowernetwork throughput due to imperfect link data rates,Fig. 13 shows similar trends and phenomena as weobserved from Fig. 12. As we can see from the figure,DIM outperforms IIM as C increases by keeping N at alower number, which suggests that the proposed DIMstrategy is more adaptive than IIM when dealing with vary-ing link bit rates.

Now we focus on the DIM algorithm, and vary networksize from 3� 3 to 7� 7 to verify the scalability of our pro-posed strategy. We experiment on the IEEE 802.11g systemwith three orthogonal channels. The rest of parameter set-tings is the same as the previous experiment. Fig. 14 illus-trates the derived network interface deployment andchannel bindings for different network sizes. As we observefrom the figure, hosts close to gateways (including gatewayitself) are usually equipped with more radio interfaces,since Internet access is the main purpose of our data pack-ets. Because the two gateways have identical bandwidth,the number of radio modules deployed at the two gatewaysis almost the same. In addition, the network throughputsare kept above 100 Mbps whether it is a small (3� 3) orlarge (7� 7) grid, suggesting that the proposed strategy is

4*4 grid, U=20, L=0.2, B=500, F=(0,24], DIM

0

50

100

150

200

250

300


Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

C=1C=2C=3C=4C=5C=6C=7C=8

4*4 grid, U=20, L=0.2, B=500, F=(0,24], IIM

0

50

100

150

200

250

300


Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

C=1C=2C=3C=4C=5C=6C=7C=8

a b

Fig. 13. Aggregate network throughput vs. number of available radio interfaces in the IEEE 802.11a environment with varying link bit rates (uniformlydistributed over (0,24] Mbps) using (a) DIM and (b) IIM algorithms.

(a) 3*3 grid, N=14, Throughput=107.8

channel 1

channel 2

channel 3

gateway

host

channel 1

channel 2

channel 3

channel 1channel 1

channel 2channel 2

channel 3channel 3

gateway

host

gateway

host

(b) 4*4 grid, N=24, Throughput=139.5 (c) 5*5 grid, N=38, Throughput=133.2

(d) 6*6 grid, N=54, Throughput=128.4 (e) 7*7 grid, N=74, Throughput=126.2

Fig. 14. The interface distributions and channel configurations for different network sizes in the IEEE 802.11g environment using the proposed DIMalgorithm.



scalable. Scalability property is critical for WMNs indesigning an easy-to-deploy high-performance wirelessmesh backbone without paying much unnecessary atten-tion to the network size and routing path length.

Also focusing on the DIM strategy, in the next experi-ment, we enable heterogeneous gateways by setting theupper-left gateway capacity to be 5 Mbps and the bot-tom-right one to have 500 Mbps bandwidth capacity. Therest of parameter settings is the same as the previous exper-iment. We test on the 4� 4 and 5� 5 grids. Fig. 15 depictsthe resulting network configurations. Due to distinguishedgateway capacities, most traffic is directed toward the bot-tom-right gateway for load balancing. As a result, moreinterfaces will be assigned to the bottom-right gateway.Furthermore, Fig. 15(a) and (b) show similar throughputperformance despite their different network sizes, whichonce again validates the scalability of our proposedmethodology.

4.2.3. Single-radio vs. multi-radio systems

In the final experiment, we go back to the 4� 4 grid, andstudy the performance improvement provided by multi-radio multi-channel systems. We denote the Single-Inter-face strategy as SI, which is adopted in the single-radio sys-tem. For single-radio networks with varying link capacities,SI performs our LP calculations for all available channels,and selects the best channel producing the maximalthroughput as our comparison base. For multi-radio net-works, we perform the proposed DIM and IIM algorithmsto manage available NICs and arrange channel bindings.

Fig. 16 shows the throughput comparisons between sin-gle-radio and multi-radio systems. The setting of N func-tion is based on the observations from our previousexperiments in the 4� 4 grid, making N to increase by fourevery time one more channel is available to the network. Aswe can see from Fig. 16, the advantage of using multipleradio interfaces on mesh hosts is obvious, as the through-put performance can be easily boosted up to five times thatof the single-radio systems by equipping reasonable num-ber of NICs (<3) on each mesh host.

4.3. Simulation results and comparison

In this section, we report the simulative performancecomparison with the RCL algorithm proposed in [6]. Thesimulator used for experiments is ns-2 [2] with multi-radioextension. Two-ray ground model is adopted for the radiopropagation path loss. Note that to the best of our knowl-edge, our work is the first to address the optimization ofthe number of equipped radios at each mesh router. Asdescribed in Section 2, we have a different problem scopefrom the RCL algorithm. To provide a fair comparison,let us make the following assumptions in the simulationsfor both the RCL and our algorithms. First, up-link anddown-link traffics are assumed to be symmetric, thoughour algorithm handles asymmetric up- and down-link traf-fics. Second, we also adopt the protocol model of interfer-ence, and assume the interference sources consist of two-hop neighbors of both the sender and receiver (withRTS/CTS enabled). Third, equal data rate (capacity) forall channels and links is assumed. Fourth, gateway capacityis limited. Fifth, given a fixed total number of availableradios, RCL will allocate equal number to each mesh rou-ter, while our proposed algorithm will assign heteroge-neous numbers to nodes in order to balance the loads.

To demonstrate the importance of network planning, weuse the same parameter settings as in Fig. 11 with totalavailable number of orthogonal channels C ¼ 2. Experi-ments are performed using three different grid topologies:3� 3, 4� 4, and 5� 5. Table 2 summarizes the aggregatenetwork throughput yielded by the three algorithms underdifferent network sizes. Here, N denotes the total numberof required radio interfaces by each algorithm. As we cansee from this table, to achieve comparable network

4*4 grid, U=20, L=0.2, B=500, F=24,N =16*[1+( C -1)*0.25]

0

100

200

300

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8Number of Available Channels (C)

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s) SIDIMIIM

4*4 grid, U=20, L=0.2, B=500, F=(0,24],N =16*[1+( C -1)*0.25]

0

100

200

300

Number of Available Channels (C)

Agg

rega

te N

etw

ork

Thr

ough

put (

Mbp

s)

SIDIMIIM

a b

Fig. 16. Throughput comparisons between single-radio and multi-radio systems in the IEEE 802.11a environment with (a) constant and (b) varying linkbit rates.

(a) 4*4 grid, N=24, Throughput=75.9 (b) 5*5 grid, N=38, Throughput=74.6

Fig. 15. The interface distributions and channel configurations for (a)4� 4 and (b) 5� 5 grids in the IEEE 802.11g environment withunbalanced gateway capacities using DIM algorithm.



throughput, our proposed DIM and IIM approachesalways result in a smaller total number of radio interfacesneeded. Under the 5� 5 topology, our IIM algorithm evenrequires only half as many as the number of radio inter-faces used by RCL (N ¼ 27 vs. N ¼ 50) to achieve similarthroughput performance, thus saving deployment costs.Consequently, network planning by distributing availableradios based on different (forwarding) traffic requirementsat mesh routers has been effectively exercised by the pro-posed DIM and IIM algorithms.

5. Conclusions

In this work, we propose an M4 wireless mesh architec-ture and design related resource allocation and channelassignment mechanisms to maximize the possible networkcapacity at the deployment stage. The numerical resultsshow encouraging potential in terms of network through-put improvement. We plan to investigate on the optimalarrangement by letting the channel vector ci½k� becomeunknown and solving the nonlinear programming modelin the near future, so that we can observe how close ourproposed linear methodology is to the optimal nonlinearsolution. On the other hand, due to the relatively high com-putational complexity incurred by the linear programmingcalculations, we only perform this optimization task at theWMN deployment stage as an initialization setup. Oncemesh nodes are well configured, the LP modeling will bere-evaluated periodically in an infrequent basis. Based onthe current insights observed from this work, we plan toexplore a sub-optimal tree-induced flow designation strat-egy, which requires less computational complexity. Theseresults and possible improvements will be reported in ourfuture paper. In addition, we are interested in the fairnessproblem in WMNs. In this work, we realize the network-level fairness by setting reasonable user traffic bounds (ui

and li) in our linear programming model and performingflow control in the packet forwarding function. However,there is still short of a link-level technique to prevent band-width occupancy from favoring those users closer to Inter-net gateways. This MAC-layer fairness issue will also bedirected into our future work.

References

[1] lp-solve: a Mixed Integer Programming (MIP) solver. Available from:<http://www.geocities.com/lpsolve>.

[2] ns-2: The Network Simulator. Available from: <http://www.isi.edu/nsnam/ns/>.

[3] A. Adya, P. Bahl, J. Padhye, A. Wolman, L. Zhou, A multi-radiounification protocol for IEEE 802.11 wireless networks, in: Proceed-ings of the IEEE BroadNets, October 2004, pp. 25–29.

[4] D. Aguayo, J. Bicket, S. Biswas, G. Judd, R. Morris, Link-levelmeasurements from an 802.11b mesh network, in: Proceedings of theACM SIGCOMM, August 2004, pp. 121–131.

[5] I.F. Akyildiz, X. Wang, W. Wang, Wireless mesh networks: a survey,Elsevier Computer Networks (2005).

[6] M. Alicherry, R. Bhatia, L. Li, Joint channel assignment and routingfor throughput optimization in multi-radio wireless mesh networks,in: Proceedings of the ACM International Conference on MobileComputing and Networking (MobiCom), September 2005.

[7] B. Awerbuch, D.H.H. Rubens, High throughput route selection inmulti-rate ad hoc wireless networks, in: Proceedings of the FirstWorking Conference on Wireless On-demand Network Systems(WONS), January 2004.

[8] P. Bahl, A. Adya, J. Padhye, A. Wolman, Reconsidering wirelesssystems with multiple radios, ACM SIGCOMM Computer Commu-nications Review (CCR) 34 (5) (2004) 39–46.

[9] P. Bahl, R. Chandra, J. Dunagan, SSCH: slotted seeded channelhopping for capacity improvement in IEEE 802.11 ad-hoc wirelessnetworks, in: Proceedings of the ACM International Conference onMobile Computing and Networking (MobiCom), 2004, pp. 216–230.

[10] C.-Y. Chang, P.-C. Huang, C.-T. Chang, Y.-S. Chen, Dynamicchannel assignment and reassignment for exploiting channel reuseopportunities in mobile ad hoc wireless networks, in: IEICETransaction on Communicaton, April 2003, pp. 1234–1246.

[11] R.R. Choudhury, X. Yang, R. Ramanathan, N.H. Vaidya, Usingdirectional antennas for medium access control in ad hoc networks,in: Proceedings of the ACM International Conference on MobileComputing and Networking (MobiCom), September 2002, pp. 59–69.

[12] R. Draves, J. Padhye, B. Zill, Routing in multi-radio multi-hopwireless mesh networks, in: Proceedings of the ACM InternationalConference on Mobile Computing and Networking (MobiCom),2004, pp. 114–128.

[13] G. Holland, N. Vaidya, P. Bahl, A rate-adaptive MAC protocol formulti-hop wireless networks, in: Proceedings of the ACM Interna-tional Conference on Mobile Computing and Networking (Mobi-Com), July 2001, pp. 236–251.

[14] J. Jun, M.L. Sichitiu, The nominal capacity of wireless meshnetworks, IEEE Wireless Communications 10 (5) (2003) 8–14.

[15] R. Karrer, A. Sabharwal, E.W. Knightly, Enabling large-scalewireless broadband: the case for TAPs, in: Proceedings of the SecondWorkshop on Hot Topics in Networks (HotNets-II), 2003.

[16] P. Kyasanur, N.H. Vaidya, Routing and interface assignment inmulti-channel multi-interface wireless networks, in: Proceedings ofthe IEEE Wireless Communications and Networking Conference(WCNC), 2005.

[17] R. Ramanathan, J. Redi, C. Santivanez, D. Wiggins, S. Polit, Ad hocnetworking with directional antennas: a complete system solution,IEEE Journal on Selected Areas in Communications (JSAC) (2005)496–504.

[18] A. Raniwala, T.-C. Chiueh, Architecture and algorithms for an IEEE802.11-based multi-channel wireless mesh network, in: Proceedings ofthe IEEE INFOCOM, 2005.

[19] A. Raniwala, K. Gopalan, T.-C. Chiueh, Centralized channelassignment and routing algorithms for multi-channel wireless meshnetworks, in: Mobile Computing and Communications Review, April2004, pp. 50–65.

[20] S.-T. Sheu, Y. Tsai, J. Chen, MR2RP: the multi-rate and multi-rangerouting protocol for IEEE 802.11 ad hoc wireless networks, ACM/Kluwer Wireless Networks 9 (2) (2003) 165–177.

[21] J. So, N. Vaidya, Multi-channel MAC for ad hoc networks: handlingmulti-channel hidden terminals using a single transceiver, in: Pro-ceedings of the ACM MobiHoc, May 2004, pp. 222–233.

[22] K. Sundaresan, R. Sivakumar, M.A. Ingram, T.-Y. Chang, A fairmedium access control protocol for ad-hoc networks with MIMOlinks, in: Proceedings of the IEEE INFOCOM, March 2004.

Table 2Comparison of the RCL algorithm equipping two radio interfaces at eachmesh router with our proposed approaches having the capability ofdistributing radio interfaces based on load-sensitivity

3 � 3 4 � 4 5 � 5

RCL (Mbps) 18.0341 N = 18 17.2148 N = 32 15.7682 N = 50DIM (Mbps) 18.0341 N = 15 17.1631 N = 21 15.5473 N = 31IIM (Mbps) 18.0593 N = 13 17.1902 N = 20 15.7504 N = 27



[23] W.-H. Tam, Y.-C. Tseng, Joint multi-channel link layer and multi-path routing design for wireless mesh networks, in: Proceedings of theIEEE INFOCOM, 2007.

[24] S.-L. Wu, C.-Y. Lin, Y.-C. Tseng, J.-P. Sheu, A newmulti-channel MAC protocol with on-demand channel assign-ment for multi-hop mobile ad hoc networks, in: Proceedings

of the International Symposium on Parallel ArchitecturesAlgorithms and Networks (I-SPAN), December 2000, pp.232–237.

[25] S.-L. Wu, Y.-C. Tseng, C.-Y. Lin, J.-P. Sheu, A multi-channel MACprotocol with power control for multi-hop mobile ad hoc networks,The Computer Journal 45 (1) (2002) 101–110.


author's personal copy · author's personal copy the related issues. in[10,24,25], the...

Documents