signature free buffer overflow attack detection in peer to

International Journal of Electronics, Communication & Soft Computing Science and EngineeringISSN: 2277-9477 Volume 2, Issue 2

28

Signature Free Buffer Overflow Attack Detection in Peerto Peer Network.

Mr.J.P.Mehare, Prof.V.S.Gulhane

Abstract— SigFree - online signature-free out-of-the-boxapplication-layer method for blocking code-injection bufferoverflow attack messages targeting at various Internet servicessuch as web service. Motivated by the observation that bufferoverflow attacks typically contain executables whereaslegitimate client requests never contain executables in mostInternet services, SigFree blocks attacks by detecting thepresence of code. SigFree is signature free, thus it can blocknew and unknown buffer overflow attacks. We show that theattack packets tested in our experiments with very few falsepositives.

Keywords- Intrusion detection, Buffer overflow attacks,code injecting attacks

I. INTRODUCTION

Throughout the history of cyber security, bufferoverflow is one of the most serious vulnerabilities incomputer systems. Although tons of research has been doneto tackle buffer overflow attacks, existing defenses are stillquite limited in meeting four highly desired requirements:(R1) simplicity in maintenance; (R2) transparency toexisting (legacy) server OS, application software, andhardware; (R3) resiliency to obfuscation; (R4) economicalInternet-wide deployment. As a result, although several verysecure solutions have been proposed, they are notpervasively deployed, and a considerable number of bufferoverflow attacks continue to succeed on a daily basis.

Existing defenses are limited in meeting these fourrequirements. Existing buffer overflow defenses arecategorized into six classes. (1) Finding bugs in source code.(2) Compiler extensions. (3) OS modifications. (4)Hardware modifications. (5) Defense-side obfuscation.(6)Capturing code running symptoms of buffer overflowattacks.

To overcome the above limitations, in this paper, wepropose SigFree, an online buffer overflow attack blocker,to protect Internet services. The idea of SigFree is motivatedby an important observation that “the nature ofcommunication to and from network services ispredominantly or exclusively data and not executable code”.In particular, as summarized 1) on Windows platforms, mostweb servers (port 80) accept data only; remote accessservices (ports 111, 137, 138, 139) accept data only;Microsoft SQL Servers (port 1434), which are used tomonitor Microsoft SQL Databases, accept data only. 2) OnLinux platforms, most Apache web servers (port 80) acceptdata only; BIND (port 53) accepts data only; SNMP (port161) accepts data only; most Mail Transport (port 25)

accepts data only; Database servers (Oracle, MySQL,PostgreSQL) at ports 1521, 3306, and 5432 accept data only

Fig. 1 SigFree is an application layer blocker between theprotected server and the corresponding firewall

Accordingly, SigFree (Fig.1) works as follows: SigFreeis an application layer blocker that typically stays between aservice and the corresponding firewall. When a servicerequesting message arrives at SigFree, SigFree first uses anew OðNÞ algorithm, where N is the byte length of themessage, to disassemble and distill all possible instructionsequences from the message’s payload, where every byte inthe payload is considered as a possible starting point of thecode embedded (if any). However, in this phase, some data

bytes may be mistakenly decoded as instructions

In phase 2, SigFree uses a novel technique called codeabstraction. Code abstraction first uses data flow anomaly toprune useless instructions in an instruction sequence, thencompares the number of useful instructions or dependencedegree to a threshold to determine if this instructionsequence contains code. Unlike the existing code detectionalgorithms that are based on signatures, rules, or controlflow detection, SigFree is generic and hard for exploit codeto evade.

II. PROPOSED WORKThe proposed work consists concept of prevention and

detection of buffer overflows. The work proposes SigFree, areal-time, signature-free, out-of the-box, application layerblocker for preventing buffer overflow attacks, one of themost serious cyber security threats. SigFree can filter outcode-injection buffer overflow attack messages targeting atvarious Internet services such as web service. Motivated bythe observation that buffer overflow attacks typicallycontain executables whereas legitimate client requests nevercontain executables in most Internet services, SigFreeblocks attacks by detecting the presence of code. SigFree


29

first blindly dissembles and extracts instruction sequencesfrom a request. It then applies a novel technique called codeabstraction, which uses data flow anomaly to prune uselessinstructions in an instruction sequence. Finally it comparesthe number of useful instructions to a threshold to determineif this instruction sequence contains code. SigFree issignature free, thus it can block new and unknown bufferoverflow attacks; SigFree is also immunized from mostattack-side code obfuscation methods. Since SigFree istransparent to the servers being protected, it is good foreconomical Internet wide deployment with very lowdeployment and maintenance cost.Detection of Data Flow Anomalies: There are static ordynamic methods to detect data flow anomalies in thesoftware reliability and testing field. Static methods are notsuitable in our case due to its slow speed; dynamic methodsare not suitable either due to the need for real execution of aprogram with some inputs.

Their scheme is rule-based, whereas SigFree is ageneric approach which does not require any pre-knownpatterns. Then, it uses the found patterns and a data flowanalysis technique called program slicing to analyze thepacket’s payload to see if the packet really contains codeFour rules (or cases) are discussed in their project: Case 1not only assumes the occurrence of the call/jmp instructions,but also expects the push instruction appears before thebranch; Case 2 relies on the interrupt instruction; Case 3relies on instruction ret; Case 4 exploits hidden branchinstructions. Besides, they used a special rule to detectpolymorphic exploit code which contains a loop. Althoughthey mentioned that the above rules are initial sets and mayrequire updating with time, it is always possible forattackers to bypass those pre-known rules. Moreover, morerules mean more overhead and longer latency in filteringpackets. In contrast, SigFree exploits a different data flowanalysis technique, which is much harder for exploit code toevade.

III. IMPLEMENTATION

There are various tools available for simulating differentnetwork models. Ns2/ns3, OPNET and NetSim are some ofthe tools that can be used for the simulation of the variousnetwork architectures and models. The ns-2 simulator is adiscrete-event network simulator targeted primarily forresearch and educational use. The ns-2 is written in C++. ns-2 is open-source, and the project strives to maintain an openenvironment for researchers to contribute and share theirsoftware.Ns-2 is scripted in OTcl and results of simulations can bevisualized using the Network Animator nam. It is notpossible to run a simulation in ns-2 purely from C++ (i.e., asa main() program without any OTcl). Moreover, some

components of ns-2 are written in C++ and others in OTcl.Considering these features of ns-2 ns-allinone-2.34 is usedfor the implementation of the proposed dissertation work.Initially ns-allinone-2.34 is downloaded from [10]. NS-2 isdesigned to run from on most UNIX based operatingsystems. It is possible to run NS-2 on Windows machinesusing Cygwin. If you don't have a UNIX install, you canalso use a virtual linux machine and run that underWindows. In the dissertation work the Fedora core 13operating system is used for installation and configuration ofthe ns-2.34. The ns-2.34 is configured on the path/home/project/Desktop/project/. For configuring the ns thefollowing commands are executed in the terminal. Beforeconfiguration we should make sure that we have standarddevelopment packages like 'make' and 'gcc'.

tar -xzf ns-allinone-2.34.tar.gz

cd ns-allinone-2.34

./install

After the execution of the above commands on terminal ifeverything is fine without any errors then we will getfollowing messages on the terminal.

Ns-allinone package has been installed successfully.

Here are the installation places:tcl8.4.11:/home/jayant/Desktop/project/ns-allinone-

2.34/{bin,include ,lib}tk8.4.11:/home/jayant/Desktop/project/ns-allinone-2.34/{bin, include, lib}otcl: /home/jayant/Desktop/project/ns-allinone-2.34/otcl-1.11tclcl: /home/jayant/Desktop/project/ns-allinone-2.34/tclcl-1.17ns: /home/jayant/Desktop/project/ns-allinone-2.34/ns-2.29/nsnam: /home/jayant/Desktop/project/ns-allinone-2.34/nam-1.11/namxgraph: /home/project/Desktop/project /ns-allinone-2.34/xgraph-12.1gt-itm: /home/jayant/Desktop/project/ns-allinone-2.34/itm,edriver, sgb2alt, sgb2ns, sgb2comns, sgb2hierns

----------------------------------------------------------------------------------Please put /home/jayant/Desktop/project/ns-allinone-2.34/bin:/home/jayant/Desktop/project/ns-allinone-2.34/tcl8.4.18 /unix:/home/jayant/Desktop/project/ns-allinone-2.34/tk8.4.18/unix


30

into your PATH environment; so that you'll be able to runitm/tclsh/wish/xgraph.

IMPORTANT NOTICES:

(1) You MUST put /home/jayant/Desktop/project/ns-allinone-2.34/otcl-1.13:/home/jayant/Desktop/project/ns-allinone-2.34/lib,

into your LD_LIBRARY_PATH environment variable.If it complains about X libraries, add path to your X

librariesinto LD_LIBRARY_PATH.If you are using csh, you can set it like:

setenv LD_LIBRARY_PATH <paths>If you are using sh, you can set it like:

export LD_LIBRARY_PATH=<paths>

(2) You MUST put /home/jayant/Desktop/project/ns-allinone-2.34/tcl8.4.18/library into your TCL_LIBRARYenvironmental

variable. Otherwise ns/nam will complain during startup.

(3) [OPTIONAL] To save disk space, you can now deletedirectories tcl8.4.11

and tk8.4.11. They are now installed under/home/jayant/Desktop/project/ns-allinone-2.34/{bin,include,lib}

After these steps, you can now run the ns validation suitewithcd ns-2.34; ./validate

For trouble shooting, please first read ns problems pagehttp://www.isi.edu/nsnam/ns/ns-problems.html. Also searchthe ns mailing list archivefor related posts.

For the validation of the installation we can run the nextcommand on terminal as

cd ns-2.34./validate

After successful installation we need to modify some of theenvironment variables namely PATH,LD_LIBRARY_PATH and TCL_LIBRARY. Toaccomplish this the following commands are executed onthe terminal.

export setPATH=$PATH:/home/project/Desktop/project/ns-allinone-

2.34/bin:/home/project/Desktop/project/ns-allinone-2.34/tcl8.4.18/unix :/home/project/Desktop/project/ns-allinone-2.34/tk8.4.18/unix

export setLD_LIBRARY_PATH=/home/project/Desktop/project/ns-allinone-2.34/otcl-1.13:/home/project/Desktop/project/ns-allinone-2.34/lib

export setTCL_LIBRARY=/home/project/Desktop/project/ns-allinone-2.34/tcl8.4.18/library

Fig 2: Network Animator used for viewing the simulationresults

Whenever we want to run any simulation we need toset these variables. We can start ns with the command 'ns<tclscript>' (assuming that we are in the directory with thens executable, or that our path points to that directory),where '<tclscript>' is the name of a Tcl script file whichdefines the simulation scenario (i.e. the topology and theevents). We could also just start ns without any argumentsand enter the Tcl commands in the Tcl shell, but that isdefinitely less comfortableWe can either start nam with the command 'nam <nam-file>'where '<nam-file>' is the name of a nam trace file that wasgenerated by ns, or execute it directly out of the Tclsimulation script for the simulation which we want tovisualize. Below we can see a screenshot of a nam windowwhere the most important functions are being explained.

After the successful configuration of ns environmentfor the implementation of the proposed dissertation workinitially we have created bod (Buffer Overflow Detector)package inside ns-2.34 which defines various properties ofthe data in terms of packet which are transferred to and from

http://www.isi.edu/nsnam/ns/ns-problems.html


31

various node which can be identified as packets which areusing implemented bod protocol for buffer overflow attackdetection in the network. The implemented algorithm whichis written inside bod.cc is situated inside the router throughwhich packets are transferred and filtered. It also includesdifferent parameters which are defined for theimplementation of the bod system which includes differenttypes of packet which is transferred between different nodes.We have generated large number of attack packets andattacks which are categorized as DoS attack and some of thepackets as normal packets. Inside bod we have defined fourfiles bod.cc, bodPacket.cc, bod.h and bodPacket.h. Afteradding bod into ns-2.34 we need to modify some of the filesof ns environment which are ns-2.34/common/packet.h, ns-2.34/tcl/lib/ns-packet.tcl and ns-2.34/Makefile. After thesechanges we need to again execute make command on theterminal to reflect the changes in the ns environment.

IV. RESULT

The result of implemented work is carried out by differentsimulations which are implemented to demonstrate thedifferent ways of buffer overflow attack detection in thedifferent types of network architecture. Initially simnids.tclis implemented to demonstrate network buffer overflowdetection system where there are two attacker nodes, onesender node and one receiving node. For the execution ofthis tcl script initially all the environment variables are setand the following command is executed on the terminal.ns /home/jayant/simbod.tcl

After the execution out.nam file is created inside the currentworking directory and we get the following output on theterminal.

Sending L...Sent in 5.000000 Seconds.

Sending K...Sent in 8.000000 Seconds.

Sending B...Sent in 11.000000 Seconds.


Sending E...Sent in 17.000000 Seconds.



Packet received

Transaction (set D) ==> asdfg

Buffer Overflow Attack Detected

Detected in 4.000000 Seconds.

Packet received

Transaction (set D) ==> jkhgasdfg

DoS attack


Packet received

Transaction (set D) ==> locnnmasdfghf



Packet received

Transaction (set D) ==> oqwdjhkcmd

SET L1={ (a,3) (s,3) (d,5) (f,4) (g,4) (j,2) (k,2) (h,3) (o,2)(c,2) (n,2) (m,2) }

Ln={ (sda,0) (sds,0) (sdd,0) (sdf,0) (sdg,0) (sdj,0) (sdk,0)(sdh,0) (sdo,0) (sdc,0) (sdn,0) (sdm,0) (asd,0) (ssd,0) (dsd,0)(fsd,0) (gsd,0) (jsd,0) (ksd,0) (hsd,0) (osd,0) (csd,0) (nsd,0)(msd,0) }

Cn={ (sdf,3) (asd,3) }

Cn={ (sdfg,3) (asdf,3) }

Cn={ (asdfg,3) }

It can be observed from the output buffer overflow and DoSattacks are detected in the network. The DoS attack isdetected when there is packet loss that means the packetdoesn’t receive to the destination node. We can run thesimulation by executing the following command on theterminal.

nam out.nam

After execution the output is generated inside the networkanimator. Figure shows the network with two attackers’ onesource node and one destination node which receive thepackets from these attacker nodes. The intermediate node isthe router which detects whether a packet is attack packet orsimple packet.


32

Fig 3 Network with One Source Node, One DestinationNode and Three Attacker Nodes.

After running the modified data simulation we get thefollowing output on the terminal.ns /home/jayant/simbodnew.tcl




Sending C...Sent in 14.000000 Seconds.



Packet received

Transaction (set D) ==> asdfg





Sending B...Sent in 17.000000 Seconds

Packet received

Transaction (set D) ==> jkhgasdfg




Sending P...Sent in 14.000000 Seconds.



Sending P...Sent in 23.000000 SecondsPacket received

Transaction (set D) ==> locnnmasdfghf



Packet received

Transaction (set D) ==> oqwdjhkcmd

SET L1={ (a,3) (s,3) (d,5) (f,4) (g,4) (j,2) (k,2) (h,3) (o,2)(c,2) (n,2) (m,2) }

Ln={ (sda,0) (sds,0) (sdd,0) (sdf,0) (sdg,0) (sdj,0) (sdk,0)(sdh,0) (sdo,0) (sdc,0) (sdn,0) (sdm,0) (asd,0) (ssd,0)(dsd,0) (fsd,0) (gsd,0) (jsd,0) (ksd,0) (hsd,0) (osd,0)(csd,0) (nsd,0) (msd,0) }

Cn={ (sdf,3) (asd,3) }

Cn={ (sdfg,3) (asdf,3) }

Cn={ (asdfg,3) }


33

The following table indicates various types of attacks thatare generated in the network.

Generated in

Simulation of

Type of Attack

Normal Buffer

Overflow

DoS

ns simnids.tcl 7 2 1

Table 1: Attacks detected by the implemented bufferoverflow attack detection approach.

The values of precision and recall for the implementedbuffer overflow detection system are shown in the table.

The precision and recall is calculated by the followingformulae.

Precision

Recall

CONCLUSION

As the most serious attacks like code injectionattacks, DoS attacks etc are most vulnerable to thenetworks, the SigFree can block the attacks withouthaving the signature of the attack, thus it is free frommost of the code obfuscation methods. And also be mostuseful for good economical Internet wide development.Also the maintenance cost of the signature free bufferoverflow blocker is very less and the negligiblethroughput.

Hence ,we can conclude that the defined goal of ourImplementation of buffer overflow attack blocker havebeen achieved by introducing cost effective and withnegligible throughput for networking solutions that allowto control networks.

REFERENCES

[1] A. Pasupulati, J. Coit, K. Levitt, S.F.Wu, S.H. Li,R.C. Kuo, and K.P. Fan, “Buttercup: On Network-basedDetection of Polymorphic Buffer OverflowVulnerabilities,” Network Operations and ManagementSymposium 2004(NOMS 2004).

[2] A. Smirnov and T. cker Chiueh, “Dira: AutomaticDetection, Identification, and Repair of Control-Hijacking Attacks,” Proc. 12th Ann. Network andDistributed System Security Symp. (NDSS), 2005.

[3] C. Cowan. Software security for open-source systems.IEEE Security & Privacy, 1(1):38–45, 2003.

[4] C. Kruegel, T. Toth, and E. Kirda, “Service SpecificAnomaly Detection for Network Intrusion Detection,”InSymposium on Applied Computing (SAC), Spain,March 2002.

[5] D. Evans and D. Larochelle, “Improving SecurityUsing Extensible Lightweight Static Analysis,” IEEESoftware, vol. 19, no. 1, 2002.

[6] D. Wagner, J.S. Foster, E.A. Brewer, and A. Aiken,“A First Step towards Automated Detection of BufferOverrun Vulnerabilities,” Proc. Seventh Ann. Networkand Distributed System Security Symp. (NDSS ’00), Feb.2000.

[7] D. Wagner, J. S. Foster, E. A. Brewer, and A. Aiken.A first step towards automated detection of bufferoverrun vulnerabilities. In Network and DistributedSystem Security Symposium, pages 3–17, San Diego,CA, Feb. 2000.

[8] Fu-Hau Hsu and Tzi-cker Chiueh, “CTCP: ATransparent Centralized TCP/IP Architecture forNetwork Security,” Annual Computer SecurityApplication Conference (ACSAC 2004), Tucson,Arizona, Dec., 2004.

[9] GCC Extension for Protecting Applications fromStack-Smashing Attacks,http://www.research.ibm.com/trl/projects/security/ ssp,2007.

[10] G. Holzmann. Static source code checking for user-defined properties. In Proc. IDPT 2002, Pasadena, CA,USA, June 2002.[11] H. Chen, D. Dean, and D. Wagner, “ModelChecking One Million Lines of C Code,” Proc. 11th Ann.Network and Distributed System Security Symp. (NDSS),2004.

[12] J. Condit, M. Harren, S. McPeak, G. C. Necula, andW. Weimer. CCured in the real world. In Proceedings ofthe ACM SIGPLAN 2003 conference on Programminglanguage design and implementation, pages 232–244.

http://www.research.ibm.com/trl/projects/security/


34

ACM Press, 2003.

[13] Jempiscodes—A Polymorphic Shellcode Generator,http://www.shellcode.com.ar/ en/proyectos.html, 2007

[14] J. Newsome and D. Song, “Dynamic Taint Analysisfor Automatic Detection, Analysis, and SignatureGeneration of Exploits on Commodity Software,” Proc.12th Ann. Network and Distributed System SecuritySymp. (NDSS), 2005.

[15] J. Wilander and M. Kamkar. A comparison ofpublicly available tools for dynamic buffer overflowprevention. In Proceedings of the 10th Network andDistributed System Security Symposium, pages 149–162,Feb. 2003.

[16] Ke Wang and S. J. Stolfo, “Anomalous Payload-based Network Intrusion Detection,” Recent Advance inIntrusion Detection (RAID), Sept. 2005.

[17] M. Zitser, R. Lippmann, and T. Leek. Testing staticanalysis tools using exploitable buffer overflows fromopen source code. SIGSOFT Softw. Eng. Notes,29(6):97–106, 2004.

[18] M. Zitser. Securing software: An evaluation of staticsource code analyzers. Master’s thesis, MassachusettsInstitute of Technology, Department of ElectricalEngineering and Computer Science, Aug. 2003.

[19] Manish Prasad, Tzi-cker Chiueh, “A BinaryRewriting Defense against Stack based Buffer OverflowAttacks,” Usenix Annual Technical Conference, GeneralTrack, San Antonio, TX, June 2003

[20] N. Delgado, A. Q. Gates, and S. Roach. A taxonomyand catalog of runtime software-fault monitoring tools.IEEE Transactions on Software Engineering, 30(12):859–872, Dec. 2004.

[21] S. Macaulay, Admmutate: Polymorphic ShellcodeEngine, http:// www.ktwo.ca/security.html, 2007.

http://www.shellcode.com.ar/

www.ktwo.ca/security.html

signature free buffer overflow attack detection in peer to

Documents