SFC: A Simple Flow Control Protocol for EnablingReliable Embedded Network Systems Reprogramming

Biswajit Mazumder, Jason O. HallstromSchool of ComputingClemson UniversityClemson, SC 29634

{bmazumd, jasonoh}@cs.clemson.edu

ABSTRACTTo support reprogramming in embedded network systems(ENSs), we present SFC, a simple flow control protocol forcommunication between serial devices, which enables fastand reliable data transmission. The reprogramming strategyuses a boot loader, responsible for receiving the applicationprogram over a serial communication channel, storing it inthe program memory space, and then initiating execution.SFC is used as the communication protocol for the bootloader. SFC uses a finite state automata to control the sendand receive mechanisms, while introducing minimal proto-col overhead. The automata utilizes synchronization andacknowledgment mechanisms to achieve high reliability. Wefirst describe the boot loader design and the reprogrammingstrategy. We then describe the design and implementationof the SFC protocol and present experimental results todemonstrate its performance and efficacy.

KeywordsEmbedded network systems, flow control protocols, bootloaders, flash programming

1. INTRODUCTIONMost ENS applications are “burned into” the embedded

hardware before they are deployed. The development en-vironments provided by typical desktop systems are usedto develop the applications, and the compiled program iswritten to the program memory of the embedded devicewith the use of an in-system-programmer (ISP). TypicalENS deployments for disaster management, structural heathmonitoring, and volcanic activity monitoring [2, 10, 12] re-quire a large number of these devices to be programmedand deployed. Due to the high cost of ISP programmers (≈$30–100) as compared to an individual embedded device likethe TelosB/Tmote Sky, MicaZ, or SHIMMER, often called“motes” (≈ $50–100) [9], the ratio of the number of motes tothe number of ISP programmers is usually large. Thus, theprogramming time required to set up such a deployment is

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ACM SE ’12Copyright 2012 ACM X-XXXXX-XX-XX/XX/XX ...$10.00.

again large. Moreover, adding new functionality or makingbug fixes to existing applications can also be time-consuming— more so if it is required after a ENS deployment has beencompleted. Reprogramming solutions that avoid the use ofISP programmers address these problems.

ISP-free reprogramming strategies usually require installinga boot loader on the target device, which receives the ap-plication program. The data transfer may rely on wired orwireless communication, terminating in a serial (e.g. UART)or parallel communication device. The transferred data is ei-ther stored in external memory, such as EEPROM, or storeddirectly in the application flash memory space.

The transmitting side in this process is typically a comput-ing system with large amounts of RAM (and buffer) space,able to send the program data at a continuous rate. Oneproblem with this strategy is that the application programis usually much larger in size than the mote’s available RAM.Thus, the entire application program cannot be sent to theboot loader all at once; this would overwhelm the device.Thus, the boot loader must be able to receive a smallerblock of the application program, process the data, store itto flash (or external storage), and then proceed to the nextblock of the application program. The rate at which the bootloader receives data is typically much faster than the speedat which it can write the data to flash; the write to flashoperation is a bottleneck. This necessitates a dedicated flowcontrol mechanism to manage the inflow of program dataand to allocate RAM memory with fine-grained control.

Existing hardware flow control solutions, e.g. RTS/CTSand DTR/DSR, usually require out-of-band signaling anddedicated control lines. In the context of ENSs, dedicatedcontrol lines require additional microcontroller pins – of-ten a scarce resource. Software flow control solutions, e.g.,XON/XOFF , do not consume hardware resources, but pro-vide no reliability against bit errors. Supplemental paritybits help in error detection, but provide no reliability unlessan additional retransmission strategy is also implemented.

Our primary objective in this paper is to describe SFC, asimple flow control protocol, in the context of a boot loaderwe have developed to support serial device reprogramming.SFC provides fast and reliable transmission with minimalprotocol overhead. The reliability of SFC is attributed tothe fact that it can recover from bit errors.

2. RELATED WORKThe study of flow control protocols has been ongoing for

decades. In software flow control, also known asXON/XOFFflow control, byte transmission occurs using standard com-

munication lines [6]. Two bytes from the ASCII characterset, called XON and XOFF , are predefined to be used forflow control purposes. Sending the character XON indi-cates the start of data transmission, while XOFF is sent topause it. Since this flow control strategy is stateless, loss ofeither the XON or XOFF byte due to unreliable transmis-sion lines may cause the system to become unrecoverable.Software flow control also does not provide data reliability.Hardware flow control [6], also referred to as RTS/CTS

flow control, uses additional dedicated control lines and soft-ware for achieving controlled data transmission. The sendersets the RTS (request to send) line to indicate that it hasdata to send. If the receiver intends to receive, it sets theCTS (clear to send) line to initiate data transfer. AdditionalDSR (data set ready) and DTR (data terminal ready) linesmay also be used to control data transmissions in null mo-dem conections. Hardware flow control is faster than soft-ware flow control as it does not introduce control characteroverhead; but it also suffers from the lack of data reliability.Binary Synchronous Communication (Bisync) [11], a pop-

ular communication protocol developed by IBM, uses multi-ple byte control sequences, from either the ASCII or EBCDICcharacter sets. Error checking is achieved using either acombination of vertical redundancy checks (VRCs) and lon-gitudinal redundancy checks (LRCs), or 16-bit cyclic redun-dancy checks (CRCs). Bisync uses frames without numbers,and numbered acknowledgments (ACKs) to achieve reliabil-ity [13]. The sender transmits data and waits for an ACK;on a timeout, it sends an ENQ. The receiver accepts datawith a correct CRC, or sends a negative acknowledgement(NAK) on the receipt of a damaged packet. On receiving anENQ, the receiver replies with the last ACK value it sent.For a simple data communication protocol, Bisync is need-lessly complex; adhering to the protocol is difficult [13].Sliding window protocols [13] like stop-and-wait, go-back-

N, receiver buffering, and selective reject (i.e., receiver buffer-ing with out-of-order retransmissions) are data transmissionprotocols widely used for reliable in-order packet delivery indata link layers and TCP. Sliding window protocols use se-quence numbers and ACKs in packets (or frames) to achievereliable, full duplex data transmission. These protocols areable to identify lost packets and initiate retransmissions.Flow control is achieved by slowing down ACK delivery;overall ACK delivery rate governs the sending rate.Flow control in embedded systems is significantly differ-

ent, presenting unique challenges to the protocol designerdue to the limited availability of computational resources.For example, the MoteStack, a state-of-the-art in-situ sens-ing platform, uses an AVR Atmel microcontroller (MCU) [1].It has 4KB of RAM, 64KB of in-system, self-programmableflash program memory, and operates at 10 MHz at 3.3V.While existing software and hardware flow control protocolsare easy to implement, they lack reliability. Additionally,hardware flow control mechanisms need MCU pins. Themore sophisticated protocols lack simplicity of design. Slid-ing window protocols are robust, but require large amountsof processing and memory resources — typically lacking inembedded systems.The serial data communication protocol in TinyOS [3] is

loosely based on PPP in HDLC-like framing (RFC-1662). Ituses the frame structure as in PPP and supports error de-tection with the help of 16-bit CRCs. The protocol also usesan acknowledgment strategy. TinyOS2 [4] includes a frame-

based protocol which also uses the HDLC encoding strategy.The protocol in TinyOS2 consists of a top-most dispatcher,responsible for packet formatting. Below the dispatcher, theprotocol layer handles acknowledgments, error verificationwith CRCs, and windowing. The encoder/framer layer be-low the protocol layer (and over the raw UART layer) isresponsible for translating raw bytes into frame and controlinformation.

3. DESIGNHere we examine the design of the boot loader to better

understand how SFC interacts with the various componentsand facilitates fast and reliable data transfer.

3.1 Boot LoaderThe boot loader enables ISP-free reprogramming (but must

be installed using an ISP initially). In MCUs which supportself-programming [1], the program flash memory space (e.g.,on-chip NAND Flash) is typically divided into two sections,the application program section and the boot program sec-tion, as shown in Figure 1. The boot loader is programmedinto the boot program section using an ISP. By program-ming the BOOTRST fuse1, the MCU can be configured tomake it jump to the boot loader address at reset.

Figure 1: Program Flash Memory Layout2

The various components of the boot loader are shown inFigure 2. The boot loader receives the application programfrom the development system, using either a Wi-Fi module(over the air) or a RS232/UART module (over the wire). Inthis paper, we focus only on modules which terminate in aserial device. The serial communication layer is responsiblefor interacting with the Wi-Fi or the RS232/UART module.The application program is sent by a C program executingon the development system, running Linux, using either theTCP/IP socket library (Wi-Fi) or the Termios terminal I/Olibrary (RS232/UART) for data transmission.

The application data sent to the boot loader is in the16-bit Intel HEX format [8], as shown in Figure 3. Whena block of application data is received on the mote, the hexfile parser is responsible for parsing/translating the receiveddata. The central I/O module is responsible for storing thedata in a temporary buffer of limited size. Since flash writes

1BOOTRST is specific to the AVR family; other processorfamilies also provide similar configuration features.2Reproduced from [1]

Figure 2: Boot Loader Components

Figure 3: Intel Hex File Layout

are performed a page at a time – typically 256B in embeddeddevices – the temporary buffer size is set to match it. Whenthe temporary buffer becomes full, the central I/O moduleproceeds to write the data to on-chip flash. Alteratively, thedata can be stored temporarily in the on-chip EEPROM, oran external NAND flash module. In this case, on completionof the data transfer, the on-chip flash can be programmedfrom the externally stored data.The central I/O module must block data flow after it has

recorded enough data to fill its buffer, until it finishes writingthe last page to flash. Again, after the write completes, themodule must restart the paused data transmission. For theseflow control requirements, the module uses SFC.

3.2 A Simple Flow Control ProtocolThe SFC protocol provides the ability for any program

(boot loader in this case) to initiate and terminate data flowwhen required, while providing high baud rates and reliabil-ity against bit errors. Data transmission using SFC involvesa connection setup phase, followed by a transfer phase, andfinally, a connection teardown phase, as shown in Figure 4.In the event of an unsuccessful transfer, control moves backto the connection setup phase.SFC does not use the concept of frames; the granularity

of data transfer is one byte. The right half of Figure 4 il-lustrates a typical sender–receiver interaction. SFC uses a

Figure 4: Operative Phases of SFC

simplistic, TCP-like, three-way handshake strategy duringthe connection setup and teardown phases. However, un-like TCP, where control messages consume an entire frame,SFC control messages (SYN, SYN-ACK, ACK, FIN, andFIN-ACK) are byte length ASCII codes. SFC uses a vari-ation of the stop-and-wait flow control model for the actualdata transfer, where the transfer granularity is set by thereceiver. SFC does not allow control of the transmissiondata rate; the rate of data transfer is implicitly controlledby the receive buffer size. It allows the receiver to specify theamount of data that it is capable of receiving, and then ini-tiates a transfer for the specified amount of data. Each suchtransfer is treated as an independent transfer. SFC ensuresreliability for each transfer with the help of error-detectingcodes (CRCs) and retransmissions.

In the connection setup phase, the receiver initiates a con-nection by informing the sender when it is ready to acceptdata. Subsequently, the receiver and sender negotiate thenumber of bytes of data to be transferred. The receiver ad-vertises the size of its buffer, while the sender advertises thesize of the data it can send; the minimum of these two val-ues is chosen as the data transfer size. In the data transferphase, the sender sends the determined amount of data tothe receiver. It also sends a CRC value, computed over thedata block, to be used for data verification on the receiverend. On completion of successful data verification, both thereceiver and sender enter the connection teardown phase.In the event the verification fails at the receiver, both thereceiver and sender restart connection setup. The connec-tion teardown phase consists of the receiver indicating tothe sender that data verification was successful, the senderacknowledging the receipt of the message, and the receiverfinally terminating the (logical) connection.

4. IMPLEMENTATIONThe SFC implementation consists of a receiver residing

on the mote, and a sender residing on the development sys-tem. The code for the connection setup, data transfer, andteardown phases resides on both the receiver and sender.

The SFC programming interface is shown in Listing 1.sfc init() is used to initialize the serial connection. Theuart config t struct, contains the serial configuration data.This includes the baud rate, parity selection, number of databits, and number of stop bits. sfc open() is responsible for

waking the UART from power reduction mode; the UARTwill be re-initialized based on the configuration data passedin the last call to sfc init(). sfc close() is responsible forputting the UART back into power reduction mode.

1 void sfc_init(uart_config_t *config );23 void sfc_open ();4 void sfc_close ();56 uint8_t sfc_send(uint8_t *buf , uint8_t size);7 uint8_t sfc_receive(uint8_t *buf , uint8_t size);

Listing 1: SFC Programming Interface

Figure 5: SFC (Receive) State Machine

Figures 5 and 6 illustrate the receiver and sender state ma-chines used by SFC, respectively. The states in the receivermachine are labeled R1 to R8; those in the sender state ma-chine are labeled S1 to S10. The numbers shown below theSend and Timed Receive annotations denote byte count, thenumber of bytes to be sent or received.

4.1 Connection Setup PhaseThe receiver initiates a connection by calling sfc receive().

A corresponding call to sfc send() is made on the sender’sside. The receiver provides sfc receive() the address andsize of the target receive buffer (RCV WIN). The sender

provides sfc send() the address and size of the source buffer(SND WIN). To setup the connection, the receiver starts at

Figure 6: SFC (Send) State Machine

R1 and sends SYN and RCV WIN back-to-back. It thenwaits for a given amount of time at R2 to receive SYN-ACKand SND WIN, failing which, it moves back to R1. It thenre-advertises SYN and RCV WIN.

The sender starts at S1 and waits to receive the SYN andRCV WIN sequence. When received, it records the receivebuffer size and moves to S2, where it waits for another set ofSYN and RCV WIN messages. It again records the receivebuffer size received, and then compares the two RCV WINvalues received in S3; on a mismatch, it moves back to S1and begins again. On a match, the sender sends SYN-ACKand SND WIN in S4 and waits for a fixed amount of timein S5 to receive an ACK from the receiver. On the receiverside, on successful reception of SYN-ACK and SND WIN instate R2, an ACK is sent back to the sender in R3, indicatinga successful connection negotiation.

4.2 Data Transfer PhaseAfter the ACK transmission in R3, the receiver enters a

timed receive state for n+ 1 bytes of data in state R4, wheren is the smaller of RCV WIN and SND WIN; one byte isadded for a CRC. On receiving the ACK in S5, the senderstarts sending n bytes of data in state S6, where n againstands for the smaller of RCV WIN and SND WIN. Onceall the data has been sent, the sender transmits an 8-bit CRCcomputed over the data block just transferred. If n bytes ofdata are not received within the alloted time, the receivermoves back to the start of the connection setup phase in R1.On receiving all n bytes of data, the receiver transitions toR5, where it computes the CRC over the received data blockand compares it to the CRC value received. On a mismatch,it moves back to the start of the connection setup phase inR1. On a successful match, it moves on to R6, the start ofthe connection teardown phase.

4.3 Connection Teardown PhaseThe connection teardown phase consists of the receiver

sending a FIN in state R6, if it has successfully received andverified the data from the sender. The receiver then moveson to the R7 state, where it waits for a limited amount of timefor a FIN-ACK. The sender keeps waiting in S7 for a FIN.On receiving the FIN in S7, the sender moves to the stateS8, where it sends a FIN-ACK. If the sender receives a SYNin state S7, it goes back to S1 and proceeds to retransmit thecurrent data block.If the receiver receives the FIN-ACK in R7, it proceeds to

the end of reception state in R8 and terminates the connec-tion. If the receiver does not receive the FIN-ACK within astipulated time, it resends the FIN from R6 and waits for theFIN-ACK in R7 again. After sending the FIN-ACK in S8,the sender enters a timed receive state in S9 (a multiple ofthe timed receive period in R7). If the sender receives a FINin S9, it assumes the FIN-ACK was lost, and hence movesback to state S8 and resends the FIN-ACK. If the senderdoes not receive a FIN in a fixed amount of time in S9, itassumes that the FIN-ACK transmission was successful andhence moves on to the end of transmission state in S10.

5. EVALUATIONOur design goal was to develop a simple flow control pro-

tocol that enables fast and reliable data transmission. Toevaluate how SFC performs in view of these goals, we reporton the results of several experiments.

5.1 Experimental SetupWe implemented the boot loader and SFC protocol for

the MoteStack, a state-of-the-art in-situ sensing platform,using C. We tested our implementation using the FTDIRS232/UART module [5] so that all TCP/IP induced timedelays could be eliminated. The FTDI RS232/UART mod-ule was operated at variable baud rates from 9600 to 115200,driven by the Termios terminal I/O library for data trans-mission on a Linux development system. The serial config-uration used was 8 data bits, no parity bits, and 1 stop bit(8-N-1). For every 8 data bits, 2 framing bits (1 start and1 stop bit) are sent; hence the achievable data rate wouldbe 80% of the baud rate. Baud rate errors introduced dueto the MCU’s 10MHz clock rate must also be accounted for.The actual baud rates, data rates, and associated error per-centages used in our experiments are shown in Table 1.

Table 1: Serial Data Transfer CharacteristicsBaud Rate Data Rate Error

(bits/second) (bits/second) (%)

9600 7680 -0.1619200 15360 -1.7438400 30720 -1.7757600 46080 +1.52115200 92160 -9.60

In each trial, we send a 32KB file containing random datafrom a development system to the mote. The file size re-flects half the size of the flash program memory. We varythe receiver buffer size from 16B up to 256B, in combina-tion with the different baud rates noted in Table 1. Weconduct each experiment three times, measure the averagetime required to send the complete file, and calculate the ef-fective data transmission rates achieved. All measurementsare conducted on the sending side. We also consider proto-col overhead and channel utilization. To ensure reliability,we compute a CRC over the entire file transferred and com-pare the calculated CRC with the CRC value received bythe mote.

5.2 PerformanceWe now examine the throughput achieved using SFC.

Figure 7(a) summarizes the achieved throughput for vari-able receiver buffer sizes. Figure 7(b) summarizes the im-pact of baud rate on throughput, for fixed buffer sizes. Thefigures show that the throughput increases with increasingbuffer size and increasing baud rate. The peak data transferrate achieved is around 15 Kbps, for a buffer size of 256Band a baud rate of 115200. The rate of change in observedthroughput as a function of the buffer size in Figure 7(a) isalmost logarithmic; it does not scale as expected from Table1. The throughput change is pronounced for higher baudrates, since transmissions in the lower baud rates achievelower peak values. Figure 7(b) shows that for smaller buffersizes, increasing the baud rate does not cause a significantincrease in throughput.

Figures 7(c) and 7(d) summarize the channel utilizationfor various buffer sizes and baud rates, respectively. Channelutilization is calculated as the percentage of the achievablethroughput (for baud rates shown in Table 1) that SFCwas able to obtain. Channel utilization reaches a minimumat 115200 bps, while the throughput reaches a maximum.The highest network utilization is around 69%, when usinga 256B buffer at 9600 bps. Since channel utilization increaseswith an increase in buffer size, the buffer sizes used for higherbaud rates should be larger for attaining higher channel uti-lization. However, for system reprogramming purposes, weare limited to a maximum buffer size of 256B due to memoryconstraints. When a transmission succeeds without retrans-missions, the protocol overhead introduced due to SFC is 8bytes. This translates to 96.97% efficiency for a buffer sizeof 256B, and 66.67% efficiency for a buffer of 16B.

5.3 ReliabilityWe ran a total of 75 experiments. During each trial, we

recorded the number of transmission retries encountered dueto bit errors. Surprisingly, there were no such retries. Weposit that this is due to the high quality crystal used inour development system. At high speeds, on embedded de-vices that use inexpensive crystals (subject to humidity and

0

2000

4000

6000

8000

10000

12000

14000

16000

0 50 100 150 200 250 300

Thro

ughp

ut O

btai

ned

(bps

)

Receive Buffer Size (B)

9600 bps19200 bps38400 bps57600 bps

115200 bps

(a) Throughput vs Buffer Size

0

2000

4000

6000

8000

10000

12000

14000

16000

0 20000 40000 60000 80000 100000 120000

Thro

ughp

ut O

btai

ned

(bps

)

Baud Rate (bps)

16B32B64B

128B256B

(b) Throughput vs Baud Rate

0

10

20

30

40

50

60

70

0 50 100 150 200 250 300

Chan

nel U

tiliza

tion

(%)

Receive Buffer Size (B)

9600 bps19200 bps38400 bps57600 bps

115200 bps

(c) Channel Utilization vs Buffer Size

0

10

20

30

40

50

60

70

0 20000 40000 60000 80000 100000 120000

Chan

nel U

tiliza

tion

(%)

Baud Rate (bps)

16B32B64B

128B256B

(d) Channel Utilization vs Baud Rate

Figure 7: Results

temperature-induced variation), we expect the number to behigher. However, we did observe a number of synchroniza-tion retries since the sender and receiver were not synchro-nized. These were completely masked by our implementa-tion. This is essentialy the same mechanism used to dealwith bit errors, should they occur. Consequently, the relia-bility results appear favorable in our experimental trials.

6. CONCLUSIONSFC can be further improved to support network repro-

gramming. The current SFC implementation is used be-tween a development system and an embedded network sys-tem, because SFC can handle only bit-level errors. In sce-narios where a full byte is lost, the current SFC state ma-chine may not recover from the error. Under such circum-stances, the only workaround is to restart the transmission.If this were remedied to support recovery after byte loss,

SFC could provide end-to-end reliability and be used as thebase for a network reprogramming strategy. By incorporat-ing SFC within a persistent, epidemic data disseminationprotocol (like Deluge [7]), and adding differential updates,we could design a fast, robust, and effective network repro-gramming solution.

In this paper, we presented SFC, a simple flow controlprotocol in the context of an embedded network system re-programming strategy. We designed, implemented, and de-ployed a boot loader which utilizes SFC. We evaluated theSFC protocol and showed that it is capable of fast datatransfer rates, while introducing minimal protocol overheadand providing reliability against bit-errors. Thus we havedemonstrated a flow control mechanism which can form thebasis of a reliable and scalable reprogramming strategy.Acknowledgments. This work was supported in part bythe National Science Foundation (CNS-0745846).

7. REFERENCES[1] AVR. Atmega datasheet. www.atmel.com/dyn

/resources/prod documents/doc2593.pdf, 2010.

[2] Rone Ilıdio da Silva et al. Spatial query processing inwireless sensor network for disaster management. InProceedings of the 2nd IFIP conference on Wirelessdays, WD’09, pages 194–198, Piscataway, NJ, USA,2009. IEEE Press.

[3] Philip Levis et al. Tinyos: An operating system forsensor networks. In Ambient Intelligence. SpringerVerlag, 2004.

[4] Philip Levis et al. T2: A second generation os forembedded sensor networks. Technical report, 2005.

[5] FTDI. Ft232r datasheet. www.ftdichip.com/Support/Documents/DataSheets/ICs/DS FT232R.pdf, 2010.

[6] EE Herald. Online course on embedded systems.http://www.eeherald.com/section/design−guide/esmod8.html, 2006.

[7] Jonathan W. Hui and David Culler. The dynamicbehavior of a data dissemination protocol for networkprogramming at scale. In In Proceedings of the 2ndinternational, pages 81–94. ACM Press, 2004.

[8] Intel. Intel hexadecimal object file format specification.www.microsym.com/editor/assets/intelhex.pdf, 1988.

[9] M. Johnson et al. A comparative review of wirelesssensor network mote technologies. In Sensors, 2009IEEE, pages 1439 –1442, oct. 2009.

[10] Sukun Kim et al. Wireless sensor networks forstructural health monitoring. In Proceedings of the 4thinternational conference on Embedded networkedsensor systems, SenSys ’06, pages 427–428, New York,NY, USA, 2006. ACM.

[11] Telecom Corner Technical Reference Site. Datacommunications protocols. http://telecom.tbi.net/protocol.htm, 2004.

[12] Geoff Werner-Allen et al. Fidelity and yield in avolcano monitoring sensor network. In Proceedings ofthe 7th symposium on Operating systems design andimplementation, OSDI ’06, pages 381–396, Berkeley,CA, USA, 2006. USENIX Association.

[13] James M. Westall. Class notes from cpsc 851.http://www.cs.clemson.edu/∼westall/851/notes/link-layer.pdf, 2010.