use and misuse of test files - final - amtso.org · antivirus companies, it replaced not only the...

TheUseandMisuseofTestFilesinAnti-MalwareTesting

Copyright©2016Anti-MalwareTestingStandardsOrganization,Inc.Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyform,inanelectronicretrievalsystemorotherwise,withouttheprior

writtenconsentofthepublisher.

2

NoticeandDisclaimerofLiabilityConcerningtheUseofAMTSODocuments

ThisdocumentispublishedwiththeunderstandingthatAMTSOmembersaresupplyingthisinformationforgeneraleducationalpurposesonly.Noprofessionalengineeringoranyotherprofessionalservicesoradvice isbeingofferedhereby. Therefore,youmustuseyourownskillandjudgmentwhenreviewingthisdocumentandnotsolelyrelyontheinformationprovidedherein.

AMTSObelievesthattheinformationinthisdocumentisaccurateasofthedateofpublicationalthoughithasnotverifieditsaccuracyordeterminedifthereareanyerrors.Further,suchinformationissubjecttochangewithoutnoticeandAMTSOisundernoobligationtoprovideanyupdatesorcorrections.

Youunderstandandagreethat thisdocument isprovidedtoyouexclusivelyonanas-isbasiswithoutanyrepresentationsorwarrantiesofanykindwhetherexpress, impliedorstatutory. Without limitingthe foregoing, AMTSO expressly disclaims all warranties of merchantability, non-infringement,continuousoperation,completeness,quality,accuracyandfitnessforaparticularpurpose.

InnoeventshallAMTSObeliableforanydamagesorlossesofanykind(including,withoutlimitation,any lost profits, lost data or business interruption) arising directly or indirectly out of any use of thisdocument including, without limitation, any direct, indirect, special, incidental, consequential,exemplary and punitive damages regardless of whether any person or entity was advised of thepossibilityofsuchdamages.

Thisdocument isprotectedbyAMTSO’s intellectualpropertyrightsandmaybeadditionallyprotectedbytheintellectualpropertyrightsofothers.



3

TheUseandMisuseofTestFilesinAnti-MalwareTesting

Introduction

Traditionally, when asked for samples with which to “test” an anti-virus program, anti-malwareresearchers have advocated [1] the use of alternatives to the sharing of truly malicious samples toanyone who doesn’t have access to malware through mainstream, trusted channels, as a way ofsimulatingmalware behaviourwithout the attendant risks of using genuinelymalicious programs [2].The most commonly suggested and used alternative to real malware is the EICAR test file, a utilityintendedforcheckingthatsecuritysoftwarehasbeeninstalledandisactive,namedafterandunderthesponsorshipofEICAR.ButtheEICARfileisnotandcannotbesuitablefortheentirerangeofscenariosinwhichwesometimesseeitused[3].

Whilethecreationofsimulatedmalwaremayseemlesscontentiousthanthecreationofactualmalware[4],generationormodificationofotherwisenon-malicious test filesmaypresentmoreproblemsthanthe aspiring tester may realize. This document derives from the realization that there has beenincreasedandunanticipateduseoftheEICARfileintestingcontextsforwhichitwasnotdesigned-orappropriate.Whileitmightberegardedasaclassicsurvivorofthestrictestformofsignaturedetection,itsuseeven in static testing is very limited indeed. Furthermore,while almost all traditional antivirusprogramsdetecttheEICARfile,thatdetectionisneithermandatorynoruniversal.

While this paper’s main focus is on the much misused EICAR test file, discussing the separation infunctionalitybetween itsuseasan installationcheckand itsuseasa limitedtest tool, italso looksatsomeothertestfilessometimesusedincomparativetesting.

VirusSimulators

SarahGordon[5]classifiedsimulatorsasfollows:

SimulatorsforEducation–demonstrationprogramssuchastheVirusSimulationSuite,Virlab,andAVP. These, she suggested, raise awareness, but mislead by setting up false expectations as tobehaviour.Thisobservationseemsmoreaccuratethanever inanerawheremostmalware,beingintendedtomakeprofitratherthanavisualdemonstrationofsomeamateurcoder’sprowess,runsasinconspicuouslyaspossibletoreducethelikelihoodofdetectionandremoval.

TestsUsing Simulators – inparticular,Rosenthal’sVirus Simulator.While the simulatormarketedformanyyearsbyDorenRosenthalisrarely,ifever,seenincontemporarytests,itremainsaprimeexampleofatestingmodelthatisbothethicallyandtechnicallyflawed[6].



4

o Itwasbasedonthefalsepremisethataproductthatdetectsarealvirusshouldalsodetectasimulationofthatvirus.[7]

o That premise reinforces the equally false premise that a virus signature is some uniquefootprintandthatallsecuritysoftwareusesthesamesignature.[6,7]Thereis,ofcourse,noreasonwhyrandomvirusfragmentsshouldbedetectedasiftheywerearealvirus,andevenlesslikelihoodthatseveralvendorswillusethesamesignature(inthebroadestsenseoftheterm).

o Theregisteredversionoftheutilitiesalsoincludedareal–ifrelativelyinnocuous–virus[4].

o The inappropriate use of Rosenthal’s programs by some testers in comparative testing [5]forcedtheAVindustrytoadddetectionofyetanothernon-virustotheimpressiveselectionofgarbagefiles,“intendeds”(objectsintendedtobemaliciousthatcannot,however,executeandthereforefailtofulfilthatintention)andotherinappropriateobjectsitneedstodetectifaproductisn’ttobepenalizedfornotdetectingwhatitshouldn’tneedtodetect[8].

Even if we ignore the ethical issues [4] regarded by some critics of the antivirus industry (and, byassociation,oftheAnti-MalwareTestingStandardsOrganization)asapurelyself-serving/self-protectiveobjection[9],thetechnicalobjectionsraisedbybothGordon[5]andSambucci[10]inthe1990sarenolesstruetoday.

ThepurposeoftheEICARTestFile,assummarizedbyGordon[5]andothers[2],istoestablish:

• WhetherAVisinstalled“correctly”

• WhathappenswhentheAVfindsavirus

• Whichmessagesaredisplayed

• How ithandles “customwarnings”,batch files,andnotifications to the systemadministratoroverthenetwork.

Gordonstatesthat“theexistenceoftheEICARtestfile,wefeel,obviatestheneedforsimulatedvirusesusedfortestingpurposes”.Shewaswritingatatime(1995)whenthemalwarethreatscapewaslargelydominated by a few comparatively slow –spreading boot sector viruses and classic file infectors likeJerusalem [11]. There is even less use or need for simulated malware (or non-simulated custom-generatedmalware,forthatmatter[9])inanageofmalwareglut,whenviruslabsareseeingtensandevenhundredsofthousandsofuniquemaliciousbinariesperday.However,itseemsthatexpectationsofwhattheEICARfilecanreasonablybeusedforhavebeensettoohigh.[1]

TheEICARTestFile

TheEICARtestfilewasneverspecificallyorovertlyintendedasatoolfortestinginthesenseofeithercomparativedetectiontestingortestingforcertificationpurposes[12],thoughcorrectdetectionofadefactoindustry-standardtestfileiscertainlyalegitimate(ifverylimited)testtarget.Apartfromthat,it’s



5

hardtoseeanyusefulpurposeforitinatestintendedtocompareorevaluatedetectiontesting,excepttoconfirmthatthescannerisactive,asdescribedbelow.

Rather, it’s intendedasan installationcheck (or test).Perhaps theuseof the term“EICARtest file” isunfortunate in that “by default”, people now think of testing in terms of comparative testing orcertification.However,gettingtheworldtorefertoitasthe“EICARinstallationcheckfile”is“probablyashopelessasrestoringtheoriginalandnon-pejorativeuseofthetermhacking”[1].

When the EICAR file became almost universally supported for installation checking by mainstreamantiviruscompanies,itreplacednotonlythewidely-deprecatedRosenthalsimulator,butalsofileswithsimilar functionality created by individual vendors for use with their own products. Clearly, it waspreferable to use a test object detected by all mainstream vendors in approximately the same wayacrossplatforms.

Technically, however, it lacks all the characteristics that we normally associate with self-replicatingmalware.Mostobviously,itdoesn’treplicate,thoughitcanbeandhasbeendescribedassimulatinganoverwritingvirus [10]. Ithasnomaliciouspayload,andcan’tevenbedescribedaccuratelyasaTrojanhorse, since all it does when executed is display a message identifying itself. In other words, itsfunctionality is entirely dependent on the precise identification of the byte sequence of which itconsists.Namely,thefollowingstringofcharacters–actually,thespecificationallowsfora(verylittle)bitmorethanthat,butwe’llcometothat.

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

If allowed to execute (though an active on-access antivirus scanner should normally prevent it fromexecuting),itprintsthefollowingsubstringtothescreen.

EICAR-STANDARD-ANTIVIRUS-TEST-FILE!

NotethatwhiledetectionandflaggingoftheEICARtestfileduringbothon-demandandon-accessscanscanbedescribed as a de facto industry standard, it isn’tmandatory for anti-virus to conform to thatstandard, so recognition or non-recognition of the EICAR string is not, strictly speaking, a suitablecriterionforevaluatingitseffectiveness.

TheEICARfiledoesn’ttellyoumuchapartfromsayingwhatitis,ofcourse:itsonlyfunctionistocheckthataconformantanti-malwareprogramisinstalledand“awake”.Itdoesn’t, initself,provethatAVisconfiguredproperly.Itdoesn’tevenproveitdetectsanyrealviruses.[2]

Unlikemost of the other “simulations” listed by Gordon, the EICAR file was never intended to be arealistic malware simulation. In fact, it’s a matter for debate whether simulated malware can everbehavelike“real”malwareinanysensethatmakesitusefulindetectiontestingwithoutactuallybeingmalware.However,aslongastheEICARfilemeetsEICAR’sownspecificationcorrectly[1],itexhibitsnomalicious behaviour of its own. It simply aims to generate a response from security software thatapproximatestotheresponsethatrealmalwarewouldgenerate.



6

While most mainstream products will identify it, and usually won’t allow it to execute (oftenquarantining itor limitingaccess inotherways), theway inwhichascannerresponds israrelyexactlythe sameas thatelicitedby realmalware. Evendisplayedmessagesare likely todiffer fromstandardmessages, reflecting the fact that most security researchers consider it inappropriate to flag a non-maliciousfileasmalicious.Hencesuchmessagesas“EICARtestfilenot-a-virusdetected”.(That’snotarealexample,butit’snotfar-fetchedeither.)

StringsAttached

TheEICARtestfile isatypicalofrealmalware inanotherway.Whereastheanti-malware industryhasbecomehighlyfocusedongenericandproactivetechnologies(underthevariousbutrelatedumbrellasofheuristicanalysis,behaviouranalysis,sandboxing,emulationandsoon),theEICARtestfileisaclassicsurvivorofthestrictestformofsignaturedetection,sometimesreferredtoasexactidentification.

ExactIdentificationcanbedefinedastherecognitionofaviruswheneverysectionofthenonmodifiablepartsofthevirusbodyisuniquelyidentified:inprinciple,thesameappliestonon-viralmalware.Inthiscase,identificationisevenmoreatypicalinthat:

1. Theentire“sample”isanASCIItextstring,eventhough(apartfromthesubstringwhichisactuallydisplayed on execution) it’s a rather goofy-looking string. This is deliberate: the file was alwaysintendedtobestrictlyASCIIcharacterssothatitcouldbecreatedbytypingthestringintoatextfileusingasimpletexteditor.[12]

2. More crucially, the entire “sample” (appendedwhitespace characters apart) is the nonmodifiablecode.

Tounderstandexactlywhythisisimportant,wefirsthavetoknowalittleofthehistoryoftheEICARfile.

EvolutionoftheEICARFile

TheEICARtestfilewascreated[12]bymembersofCAROforEICARintheearly1990s.CARO(ComputerAntiVirusResearcher'sOrganization)isaninformalgroupofindividualswhohaveworkedtogethersincearound1990acrosscorporateandacademicborderstostudythewholerangeofcomputermalware.[2,13]WhereasCAROwasalways a technical group, EICARalsohadadistinct legal andgeneral securityfocus.Nowadays, the twogroupsoperate largely independentlyof eachother, but there is of courseconsiderableoverlapinmembershipandobjectives.

TheEICAR test filewasahistoric jointproject, createdbyCAROmembersandpublishedbyEICAR, inordertomeetaperceivedneedforasimplemeansbywhichahelpdeskoperatorcouldreaditoverthetelephonetoanenduser,toallowacheckonwhetherhisorherantiviruswasworking.[12]Theoriginaldefinition,inshort,wasasfollows[2]:

“The file is a legitimate DOS program, and produces sensible results when run (it prints themessage"EICAR-STANDARD-ANTIVIRUS-TEST-FILE").



7

Itisalsoshortandsimple-infactitconsistsentirelyofprintableASCIIcharacters,sothatitcaneasilybecreatedwitha regular texteditor.Anyanti-virusproduct that supports the test fileshoulddetectitinanyfileprovidingthatthefilestartswiththefollowing68characters:


Tokeepthingssimple, thefileusesonlyuppercase letters,digitsandpunctuationmarks,anddoesnotincludespaces.Theonlythingtowatchoutforwhentypinginthetestfileisthatthethirdcharacteristhecapitalletter"O",notthedigitzero.”

However,eventsduringtheyearsbetweentheoriginalspecificationand2003necessitatedarethink.Avirus commonly known as Bat/Bwg.a@MM made use of the EICAR file as a stealthed approach toinfection.Bat/Bwg.a@MMstartswiththeEICARstring:whenthewormisrun, itgeneratesa"Filenotfound"errorbuttheexecutioncontinues.ManyAVproducts incorrectlydetectedthismalwareastheEICAR test file when it first appeared. While this was the first widelyknown instance of malwareimpersonatingtheEICARfile,itwasn’tactuallythefirstproblemderivingfromaslightloosenessinthisinitialdefinition[6].Whilethedefinitionisentirelyaccurateasadefinitionofwhatthefileactuallyis(orwas at that time), it assumed a commonsense approachon the part of all AV vendors and thereforedidn’tgointodetailonwhatthefilecouldnotorshouldnotbe.ThisledtosuchanomaliesasdetectionoffilesbeingdetectedastheEICARfilebecausetheycontainedtheEICARstring,evenwhere itdidn’tconstitutetheveryfirstcharactersofthefile.

EICARthereforewantedtohelptheAVindustryanditscustomersbyintroducingaslightchangetotheformaldefinition so thata fully-regularized, correct, safedefinitionwasavailable thatwould leavenodoubtinthemindsofeithervendorsorusersastowhatafullyconformantEICARtestfileshouldlooklike.

ThedefinitionagreedbetweenEICARandtheAVindustry[14]nowlookslikethis:

“The file is a legitimate DOS program, and produces sensible results when run (it prints themessage"EICAR-STANDARD-ANTIVIRUS-TEST-FILE").

Itisalsoshortandsimple-infact,itconsistsentirelyofprintableASCIIcharacters,sothatitcaneasilybecreatedwitha regular texteditor.Anyanti-virusproduct that supports the test fileshoulddetectitinanyfileprovidingthatthefilestartswiththefollowing68characters:


Thefirst68charactersistheknownstring.Itmaybeoptionallyappendedbyanycombinationofwhitespace characters with the total file length not exceeding 128 characters. The onlywhitespacecharactersallowedarethespacecharacter,tab,LF,CR,CTRL-Z.”

Thus, while the file was always essentially a string of 68 characters that have not changed betweendefinitions, theenhanceddefinition, as longas it is scrupulously followed, comes close toeradicating



8

riskoffalsenegatives(malwaremanagingtoexecutebecauseitspresenceismaskedbythepresenceoftheteststring). Italsolessenstheriskofaspecialcaseoffalsepositive:that is,whentheEICARfile isdetectedwhenitshouldn’tbe(forexample,whenembeddedinadocumentasabove)[6].Whitespaceapart,thefilewasalwaysintendedtobeanexactsequenceofbytes.

AnumberofingeniouswayshavebeenusedtoextendthefunctionalityoftheEICARfile,someofthemgenuinely useful [15], but some misleading and inappropriate because they were based on amisunderstandingofthespecification,orignoreditcompletely[16].

Mainstream testers haveprivileged access to sample repositories and sample exchange as a result ofvalidated, trustworthy contact with each other and the anti-virus industry, including contractualagreementsandjointmembershipofforumsandgroupssuchasAMTSO

Attractivethoughitisforaspiringtesterswithoutthosecontactstobeabletotestsafely[17],mixingupEICAR detection withmalware detection createsmore potential problems than it solves. In extremecases,we’veseeninstances[18]whereasingletesthasattemptedtoassess:

Recognition of the EICAR test file (not necessarily an invalid objective, but that depends onwhatconclusionsaredrawn).

Recognitionofpresumed(butunvalidated)IntheWild(ItW)malware.

Recognitionofpresumed(unvalidated)malwarenotknowntobeItW

Recognitionofpresumed(unvalidated)malwarenotexpectedtobeknowntothescanner

Attheveryleastthisindicatesamuddledand,therefore,undependablemethodology.

However, therehavebeenanumberofattempts toassessandcomparetheeffectivenessofmultipleantivirusproductsusingsomeformofmodificationofthetestfile itself. Inaseriesofcrosspostingstoalt.comp.virus,bugtraqetal[16],“keepitsecret”suggestedthatthetestfilecouldbeusedto“learnhowAVs do their job...watch how heuristics work with a code in principle detected by its signature(somehow,awaytoassessthelimitsofthismethod)...”

This,likeanumberofsimilarattemptstodrawconclusionsabouthowAVproductswork,wasdoomedtofailurefortworeasons.

Firstly,itfailedtotakeintoaccountthenarrownessofthespecification.Ingeneral,sixdifferentscannersmay identify a known malicious program equally effectively, yet get there by very different routes.EICAR’s strict definition, however, introduces limitations. Whatever the mechanism, the underlyingalgorithmisstillalongthelinesof:

IF

{filestartswiththeunmodified68-byteEICARstring}



9

AND

{filelengthislessthanorequalto128bytes}

AND

{appendedcharactersareallincludedintheset[space,tab,LF,CR,^Z}

THEN

ECHO“ThisistheEICARtestfile”

Secondly,thestrictnessofthedefinitionrelievesthedeveloperofanymandatetodoanythingwithanobject thatdoesn’tmeet the specificationunless ithappens to setoff adifferentdetectionalgorithmthatidentifiesitasmalware(or“mightbe”malware,or“possiblyunwanted”,orsomeotherlabelthatbrings it back into the range of programs that should be detected). After all, it doesn’t do anythingmalicious,sothere’snoreasontodetect itwhenthedisplayedcharacterstring isno longerthestringdefinedinthespecification.

Clearly, a version that displays "EICAR-STANDING-ANTIVIRUS-TEST-FILE!" isn’t the test file, isn’tmalicious,andneedn’tbedetected.Somevendorshavechosen,atvarioustimes,todisplayamessagesuch as “EICAR_Test ( modified )”, but it’s debatable whether this is a good idea, since it causesconfusion. Itmayormaynotbe“appropriate”detectionbehaviour,but thatdoesn’tmake ignoringamodified version of the EICAR file “inappropriate”. (Detection of or failure to detect a maliciouslymodifiedversionisadifferentissue.)IdentifyingamodifiedEICARdisplaystringastheEICARfileisjustplainwrong.

Othermodificationsthathavebeensuggestedinthepastinclude:

• PrependingaNOPorJMP

• UsingXORandORencryption,orothercryptographictechniques

• Usingpolymorphiccode

• Addingreplicativeorfile-searchingcode

• Splittingthefile

• Paddingwithorappendingadditionalcharacters

While in general products will disregard triviallymodified versions, as they bear less resemblance toEICARandmore toapossiblymaliciousprogram,more scannerswill start todetect themgenerically.However, suchchangesconstitutea seriouslyunreliableguide todetectionperformance.Evenworse,changesthat“resemble”amaliciousprogramareofnovaluefortestingpurposesunlesstheyreallyareinsomemeaningfulsensemalicious,butcreationofmaliciousprograms(irrespectiveofanyrelationshiptotheEICARfile)issubjecttoadifferentandverysignificantsetofproblems.[4]



10

CloudCar

CloudCarcanbedefinedinshortasatestfilethatmaybeprovidedbyanyvendorthatprovidesacloudsolution.

Manysecurityvendorsareenablingcloudbasedsolutions.Thesesolutionsrelyonspecificandreliablenetwork communication for proper functioning. Testing labs construct elaborate systems to simulatethe real world in order to conduct their tests and monitor results safely. It is possible that whileconstructing these systems the network rules imposed may interfere with a product’s ability tocommunicatecorrectlywithitsbackendservers.ThisiswhereCloudCarcomesin.CloudCarisafilethatwillonlybedetectedbyaproduct’scloudservice.ScanningCloudCarwiththenetworkdisconnected(orinoperable)mustnotresultinadetection.Scanningwiththeproductconnectedproperlytotheinternetmustresultinadetection.WithCloudCaratestercanverifythatasetupforagivenproductallowsittocommunicateproperlywithitscloudservice.

EachvendorwillhaveitsownCloudCarandthesefilesshouldneverhaveatraditionaldetectionwrittenfor them because they are not malicious. The CloudCar executable itself need have no specificfunctionality.Itsimplymusthaveauniquehash.

Spycar

Spycar (http://www.spycar.org/) came out of a research project at Intelguardians Labs, as a result ofcollaboration between Ed Skoudis, Tom Liston and Mike Poor. It was intended to test anti-spywareprograms by observing their response to certain behaviours commonly associated with Windowsspyware,usingtoolsthatwerenotmalicious.LikeRosenthal’svirus,theymadenopermanentchangesto the system. Not everyone was convinced by the methodology, though the idea of checkingbehaviouraldetectionratherthansignaturedetectionwasbynomeanstotallyinvalid.However,whilethe self-conscious “homage” to the EICAR name might lead you to expect a kind of EICAR file forspyware,theSpycarteamnailedboththedifferencebetweentheintentbehindthetwoapproachesandthefunctionalityoftheEICARfileratherwell.

“TheEICARfilecanbeusedtoverifythatyouranti-virustool isaliveandrunning.Spycartestsbehavior-basedalertingandblocking....You’vegotasmokedetector,andyouwanttoseeifitisworking. The EICAR file is like the big red test button on the smoke detector ... the smokedetector beeps, telling you that the battery is charged and everything seems to be workingproperly...Spycar...mimicsthebehaviorofarealfire(again,inabenignfashion)toseeifyoursmokedetectorisprotectingyou.”

Well,wemightquestion“everythingseemstobeworkingproperly”.Toquote thealt.comp.virusFAQ[2]:

“It's a (limited) checkonwhether theprogram is installed, but I'mnot sure it's ameasureofwhetherit'sinstalledcorrectly.Forinstance,thefactthatascannerreportscorrectlythatafilecalled EICAR.COM contains the EICAR string, doesn't tell you whether it will detect macro



11

viruses,forexample.Infact,ifIwantedtobereallypicky,I'dhavetosaythatitdoesn'tactuallytell you anything except that the scanner detects the EICAR string in files with a particularextension.“

Still,theSpycardefinitionsillustratequiteclearlythedifferencebetweenaninstallationcheckfileandanattempttocreateatypeoftoolthatcould,inprinciple,beofpotentialvalueinevaluatingproducts.

In practice, however, Spycar has long since been of little practical use as an evaluation tool becausevendors can (and do!)write detections that are based on static signatures aswell as behaviour. [19]While it’s probable that allmainstreamvendors are capableof detecting the (presumed spywarelike)behavioursSpycarusedasatestoftheirefficiency,theydon’thaveto.Infact,Spycarhasjoinedalonglistof tests thatattempttomeasureeffectivenessbycriteriawithoutrecognizingthatmainstreamAVusesmultiple detection techniques thatmay be effective in the real world even though they ignorethosespecificcriteria.

Conclusion

TheEICARtest filehasonlythemost limitedapplicationtoandconnectionwithtestingasthetermisnormally understood (i.e. comparative and certification testing) byAMTSO [20], security vendors andtesters,andmostmembersofthepublic,anditwouldprobablybemoreappropriateandlessconfusingtorefertoitastheEICARinstallationcheckfile,orsomethingsimilar,thoughthisisunlikelytohappen.

• It’s intended as an installation check, not for detection testing. It doesn’t tell you whether it’sinstalledoptimally,orhowitdetectsrealmalware,andhasnoplaceinatestintendedtoevaluatedetectionofrealmalware.

• It’susefulasaninstallationcheckinthatmostscannersdetectit,evenonplatformswhere,asaDOSexecutable,itcan’texecutenatively(forexampleunderMacOSwherenoDOS/Windowsemulationis inplace).However, theway inwhicha scanner responds isnot standardized:neither theexactdetectionmethod,nor theway inwhichdetection is flaggedandprocessed, are laiddown in theEICARspecification.ItcannotandshouldnotbeassumedthatthewayinwhichascannerbehaveswhenitdetectstheEICARtestfileisidenticaltothewayinwhichitwillbehavewhenitdetectsrealmalware.

• Becauseoftheverytightspecification,modificationtothecoreexecutablewillnormallyinvalidatetesting,thoughtheuseofsomesortofwrappermaybeacceptableandevenuseful (forexample,storingitinanarchivefile[1,15]),aslongasthefileitselfisnotmodified.Ascanningproductmaybe designed to be flexible enough to recognize the EICAR file when harmless modifications aremade,butthat’sadesigndecisionthatisn’treallyasuitabletargetfortesting,asit’snotasignificantindicatorofeffectiveness indetection.Aproduct that fully supports theEICARspecificationbut isnotflexibleenoughtorecognizemodificationstothecorecodeisnotbehavingincorrectly,andmaybemore“correct”thanascannerthatismoreflexible.AscannerthatflagsEICARandEICARvariantsdifferentlycouldbesaidtobebehavingappropriatelyaslongasitdoesn’tpresentopportunitiesformalwareto“piggyback”thetestfile.However,it’shardtoseewhatpracticaluseatesttodeterminethosecharacteristicswouldbeintherealworld.



12

• It’spossibletousetheEICARtestfiletotestcharacteristicsandissuesthatarerelatedtodetectionbutdon’trequiretheuseofrealmalwaresamples.[25].However,itwouldtakeagreatdealofcareandexperiencetousesuchtechniquesaccuratelyinthecontextofacomparativetest:atesterwhohad that skill and experience would probably also have the facilities to enable a more directapproachusingrealmalware.

AMTSO has nowish to disparage a utility that has proved useful inmany contexts as an installationcheckover theyears.However,asa tool forcomparative testing,given the limitations imposedby itsformaldefinition,useoftheEICARfileinitspresentformisrarely(ifever)appropriate:noristhereanyobviousapplicationoftheSpycarorCloudCarfacilitiesindetectiontesting.Infact,CloudCarisaspecialcaseofaproduct-specificutilitytotestconfigurationsettings.Whilethisapplicationofthe“isitactive?”principle that underlies the EICAR test file might be usefully extended to other tools testing otherfunctionalities – for instance,whether product-specific detection of “possibly unwanted” programs isenabled – it’s unlikely that a single generic binarywould be acknowledged and detected bymultiplevendors in the same way as the EICAR test file, if only because of the risk that it would be usedinappropriatelyincomparativetesting,ashashappenedwiththeEICARfile.It’spossiblethatatoolkitoffunctionalitytestingprogramsmayhavewideapplication,butitsuseintestingwouldbelimitedatbestsincechangesintechnologycouldrendertoolsobsoleteinmuchthesamewaythatSpycarhasbeen.

ItiscertainlymisleadingtodescribetheEICARfileassimulatedmalware,atleastinitsprescribedform[1].Isthereaplaceatallfor“real”simulationindetectiontesting?Philosophicallyspeaking,thereisanobviouscontradiction:ifanobjectisn’tintendedtodoharm(dependingonwhatyouunderstandbytheterm“harm”),canitbedescribedasmalicious?Ifnot,shoulditbedetectedbysecurityproductsasifitwere? Generally, the answer from the security industry has been an emphatic “no”, even whereindividual companies have taken the pragmatic decision to add a detection that they considerinappropriateratherthanbepenalizedforfailinginteststhatuseit.[7]

Recently, an online banking security test has attracted some media attention by taking a cohort ofsecurityproducts(includingbutnotconfinedtointernetsecurityproductswithanAVcomponent)andtrying them against its own simulator. Of course, the idea of a test that evades the necessity ofoperatingwithinarealorvirtualbotnetenvironmenthasitsattractions.Therehavealsobeenanumberofsomewhatsimilarteststhatsimulateattacksusingexploitstoevaluateperformanceoffirewallsandinternet security products. However, these simulations are prey to the same generic objections asdescribed in the preceding paragraph, even if the audience is prepared to accept the company’sassurancesthatthesimulationissufficientlysimilartorealmalwaretobeanappropriatetestsubject.Inthe case of simulatedmalware, it’s also amatter ofwhether it’s safe to assume that the number ofinstancesof theexploit isstatisticallyadequate.Andthequestionalways lingers in thebackground: isthe tester’s understanding of the way that a security product works sufficient to ensure that it’sreasonabletoexpectittodetectthesimulation,orisitcolouredbyamistakenconvictionastohowaproduct“should”work?



13

Citations

[1] http://eicar.org/anti_virus_test_file.htm

[2] DavidHarley,alt.comp.virusFAQ,http://www.faqs.org/faqs/computer-virus/alt-faq/part4/

[3] DavidHarley,EddyWillems,andLysaMyers,“TestFilesandProductEvaluation:theCaseforandagainstMalwareSimulationinTesting

[4] AMTSO,“IssuesInvolvedintheCreationofSamplesforTesting”www.amtso.org

[5] SarahGordon,“AreGoodVirusSimulatorsStillaBadIdea?”,NetworkSecurity,Volume1996,Issue9,September1996,Pages7-13.Availableonlineathttp://www.sciencedirect.com/

[6] DavidHarley,RobertSladeandUrsGattiker,“VirusesRevealed”,Osborne2001

[7] JoeWellsetal.,OpenLetter2000,CybersoftWhitepaper

[8] PeterKosinár,JurajMalcho,RichardMarko,andDavidHarley,“TestingExposed”,http://go.eset.com/us/resources/white-papers/Kosinar-etal-VB2010.pdf

[9] DavidHarley,“l’AMTSOMysteriosoandMalwareCreation,”AMTSOMemberBlog

[10] LucaSambucci,VirusSimulatorTest.ItalianComputerAntivirusResearchOrganization.1994.

[11] November1995WildList,http://www.wildlist.org/WildList/199511.htm

[12] PadgettPeterson,personalcommunication

[13] KennethBechtelandDavidHarleyin“TheAVIENMalwareDefenseGuide”(ed.Harley),Syngress2007

[14] EddyWillems,“TheWindsofChange:UpdatestotheEICARTestFile”,VirusBulletinJune2003

[15] RandyAbrams,“GivingtheEICARtestfilesometeeth”,VirusBulletin1999ConferenceProceedings

[16] “Let’sHaveFunwithEICARtestfile”:http://archive.cert.unihttp://archive.cert.uni-stuttgart.de/bugtraq/2003/06/msg00251.htmlstuttgart.de/bugtraq/2003/06/msg00251.html;http://marc.info/?l=ntbugtraq&m=105733144930014&w=2]

[17] TheFutureofAV-Testing:EICARpositionpaper,http://www.eicar.org/information/infomaterial/eicar_positionpaper_web.pdf[6]DavidHarleyandAndrewLee,“CalloftheWildList:LastOrdersforWildcore-BasedTesting?”,VirusBulletin2010.

[18] DavidHarley,“UntanglingtheWheatfromtheChaffinComparativeAntiVirusReviews”,Eset.comResourcePaper

[19] EdSkoudis,“WhatisSpycar?”,http://searchsecurity.techtarget.com/expert/KnowledgebaseAnswer/0,289625,sid14_gci1286802,00.html

[20] Anti-MalwareTestingStandardsOrganization,http://www.amtso.org



14

___________________________________________________________

ThisdocumentwasadoptedbyAMTSOonFebruary24,2012