penetration testing demystified

Penetration testing demystifiedPenetration testing demystified

James A. WhittakerSecurity ArchitectMicrosoft

Types of TestingTypes of Testing

There are lots of different types of testing:There are lots of different types of testing: PerformancePerformance ReliabilityReliability AvailabilityAvailability Upgrade testingUpgrade testing Internationalization testingInternationalization testing … …

– The one thing each of these has in common is The one thing each of these has in common is that they involve that they involve testing under realistic user testing under realistic user scenariosscenarios

Security testing is the one glaring exceptionSecurity testing is the one glaring exception– Problem: where do you draw testing cases from?Problem: where do you draw testing cases from?

Information SourcesInformation Sources

Functional testingFunctional testing– The specThe spec– User documentationUser documentation

– ““Use cases”Use cases”– What you know What you know

about how the about how the system is supposed system is supposed to workto work

Security testingSecurity testing– The interfaceThe interface– Error messages/ Error messages/

outputsoutputs– ““Abuse cases”Abuse cases”– What you know What you know

about how the about how the system is NOT system is NOT supposed to worksupposed to work

Information Sources (1)Information Sources (1)

The interfaceThe interface– The information on the screen is The information on the screen is

directed at legitimate usersdirected at legitimate users– Hackers are users too!Hackers are users too!

And they can learn a lot from what they And they can learn a lot from what they seesee

Consider the following screenshot … can Consider the following screenshot … can you spot the hack opportunity?you spot the hack opportunity?


Error messagesError messages– Are a special concern, they deal with Are a special concern, they deal with

bad input databad input data– Usability v. information disclosureUsability v. information disclosure

Being helpful to users is fine, but don’t Being helpful to users is fine, but don’t give hackers something they can usegive hackers something they can use

Review all error message for their Review all error message for their strategic value to users and adversaries strategic value to users and adversaries

Another Example: Message Another Example: Message ComparisonComparison

Compare this …Compare this …

……to…to…

……this!this!

// TODO - Test this functionFunction.prototype.implementsInterface = function(interfaceType) { var interfaces = this._interfaces; if (interfaces) { if (interfaces.contains(interfaceType)) { return true; } } else if (this.bases) { for (var i=0; i < this.bases.length; i++) { if(this.bases[i].implementsInterface(interfaceType)) { return true; } } } return false;}


Abuse casesAbuse cases– Can be gleaned from historical Can be gleaned from historical

informationinformation MSRC cases against similar productsMSRC cases against similar products Bugtraq and CERT advisoriesBugtraq and CERT advisories The black hat literatureThe black hat literature

Threat modelsThreat models– Enumerate entry points and describe Enumerate entry points and describe

threatsthreats


What you know about how the What you know about how the system is system is notnot supposed to work supposed to work– Inputs map to outputs … so what is Inputs map to outputs … so what is

the universe of dangerous inputs and the universe of dangerous inputs and unacceptable outputs?unacceptable outputs?

– It helps to understand more about It helps to understand more about inputs and outputsinputs and outputs

Which Inputs?Which Inputs?

Think clearly Think clearly about where an about where an input is coming input is coming fromfrom

app

OS/RTE UI

file systemexternalresource


Think clearly Think clearly about where an about where an input is coming input is coming fromfrom– What is the What is the

expected behavior expected behavior when these inputs when these inputs arrive?arrive?

– How might things How might things go wrong?go wrong?

app

OS/RTE UI



Think clearly Think clearly about where an about where an input is going toinput is going to– Can it do any Can it do any

damage at its damage at its destination?destination?

– How might the How might the app try to app try to mitigate this mitigate this damage?damage?

app

OS/RTE UI


SELECT * FROM <table> WHERE Account = ‘usersupplied’ AND Pin = ‘usersupplied’

SELECT * FROM <table> WHERE Account = ‘sam’ --

SELECT * FROM <table> WHERE Account = ‘usersupplied’ AND Pin = ‘usersupplied’

Categories of Dangerous Categories of Dangerous InputsInputs**

– Long stringsLong strings– Format stringsFormat strings– Numeric Numeric

boundariesboundaries– ScriptsScripts– CodeCode– OS commandsOS commands– ‘‘Control’ charsControl’ chars– Error codesError codes– Return valuesReturn values

app

OS/RTE UI


* This list subject to change without notice

Automating Security TestsAutomating Security Tests

Input generation and applicationInput generation and application– Lists, lists and more listsLists, lists and more lists

Not often shared since they are business assetsNot often shared since they are business assets Not often cross-applicable anywayNot often cross-applicable anyway Plan to create your ownPlan to create your own Applying inputs is easy, verifying results is the hard Applying inputs is easy, verifying results is the hard

partpart

– Some interfaces lend themselves well to Some interfaces lend themselves well to automationautomation

Network and file system specificallyNetwork and file system specifically Commercial web testing tools get historically bad Commercial web testing tools get historically bad

reviewsreviews Automation can and should focus on the above listsAutomation can and should focus on the above lists Monitor and modify: fault injection and fuzz testing are Monitor and modify: fault injection and fuzz testing are

low cost, high payoff techniqueslow cost, high payoff techniques

Defect SeverityDefect Severity

Ever heard “a user would never do Ever heard “a user would never do that”?that”?– A user might not but an adversary A user might not but an adversary willwill– The team who refuses a bug should be The team who refuses a bug should be

able to say (with a straight face): able to say (with a straight face): ““Over the lifetime of this product (which may Over the lifetime of this product (which may

be ten years), I predict that no adversary will be ten years), I predict that no adversary will be clever enough to find a way to exploit that be clever enough to find a way to exploit that bug”bug”

““And this is true for all environments in which And this is true for all environments in which the code will run now and in the future”the code will run now and in the future”

Where bad things come fromWhere bad things come from

The Source of the ProblemThe Source of the Problem

The Origin of Bad ThingsThe Origin of Bad Things

They are in the environment in They are in the environment in which our application executeswhich our application executes

They arrive as inputs to our They arrive as inputs to our applicationapplication

They are embedded within the They are embedded within the logic our applicationlogic our application

The Origin of Bad ThingsThe Origin of Bad Things

They are in the They are in the environmentenvironment in in which our application executeswhich our application executes

They arrive as They arrive as inputsinputs to our to our applicationapplication

They are embedded within the They are embedded within the logiclogic our application our application

Environment IssuesEnvironment Issues

Software does not execute in isolationSoftware does not execute in isolation– It relies on binaries and other code-It relies on binaries and other code-

equivalent (e.g., script) files equivalent (e.g., script) files – It uses configuration files and the registryIt uses configuration files and the registry– It uses databases and other local or remote It uses databases and other local or remote

servicesservices These entities may provide an attacker a These entities may provide an attacker a

route to exploit our app!route to exploit our app!– A system is only as secure as its weakest A system is only as secure as its weakest

linklink– The issue is: how much can we trust our The issue is: how much can we trust our

environment?environment?

Environment IssuesEnvironment Issues

DLLs that are faulty or have been DLLs that are faulty or have been replaced or intentionally modifiedreplaced or intentionally modified

Binaries/files that are incorrectly Binaries/files that are incorrectly ACL’dACL’d

Other apps that access shared dataOther apps that access shared data Sensitive data stored in the registry Sensitive data stored in the registry

or in temporary filesor in temporary files System stressSystem stress

– Slow network, low memory, etc. Slow network, low memory, etc.

Example: Update ExpertExample: Update Expert

Input IssuesInput Issues

Inputs originate from outside the system Inputs originate from outside the system in which our app runs and arrive via in which our app runs and arrive via entry pointsentry points– Communication pathsCommunication paths

Communication protocols, socketsCommunication protocols, sockets Exposed remote functionality (DCOM, RPC, Web Exposed remote functionality (DCOM, RPC, Web

Services) Services)

– FilesFiles ““Control” files (scripts, HTML, XML, controls …)Control” files (scripts, HTML, XML, controls …) Data files, temp files, configuration filesData files, temp files, configuration files

– UIUI Logon screens, web front-end, … Logon screens, web front-end, …

Input IssuesInput Issues

Dangerous input entered in UI Dangerous input entered in UI controlscontrols– Special characters, encoded input, Special characters, encoded input,

script fragments, format strings, script fragments, format strings, escape sequences escape sequences

Long strings embedded in filesLong strings embedded in files Corrupt packets in protocol streamsCorrupt packets in protocol streams Localization of defenses (input Localization of defenses (input

filters) against bad inputfilters) against bad input

Data and Logic IssuesData and Logic Issues

Some faults are embedded in an Some faults are embedded in an application’s internals (data and logic): application’s internals (data and logic): – Code path variation, not all code paths are Code path variation, not all code paths are

securesecure– Information disclosure that gives advantage Information disclosure that gives advantage

to an attackerto an attacker– Hard coded accounts or test APIs not Hard coded accounts or test APIs not

removed when the product shipsremoved when the product ships– Authenticating on easily faked credentialsAuthenticating on easily faked credentials

Most of these failures can be revealed Most of these failures can be revealed with legitimate input without rigging the with legitimate input without rigging the environment environment

THE ENDTHE ENDTakeaways are next, then questionsTakeaways are next, then questions

TakeawaysTakeaways

As security testers, we:As security testers, we:– Depend on a different set of Depend on a different set of

documents and requirementsdocuments and requirements ThreatsThreats Interfaces (inputs and outputs)Interfaces (inputs and outputs)

It’s about thinking on your feet and It’s about thinking on your feet and thinking like an attackerthinking like an attacker

TakeawaysTakeaways

As security testers, we:As security testers, we:– Depend on a different set attacks Depend on a different set attacks

pointspoints It’s not about the UIIt’s not about the UI It’s the invisible interfaces that we must It’s the invisible interfaces that we must

worry aboutworry about

It’s about getting used to looking for It’s about getting used to looking for the invisiblethe invisible

TakeawaysTakeaways

As security testers, we:As security testers, we:– Must do our homeworkMust do our homework

The past must be part of our presentThe past must be part of our present– MSRC, CERT, bugtraqMSRC, CERT, bugtraq– Keep current…what are the bad guys in your Keep current…what are the bad guys in your

application space doing lately?application space doing lately?– Extrapolate from this…what’s next?Extrapolate from this…what’s next?

It’s about being prepared and expecting It’s about being prepared and expecting the expected and unexpectedthe expected and unexpected

TakeawaysTakeaways

As security testers, we:As security testers, we:– Must build an arsenalMust build an arsenal

Attack toolsAttack tools Lists of dangerous inputsLists of dangerous inputs Attack database/wiki/SharePoint site …Attack database/wiki/SharePoint site …

It’s about staying at the front of the It’s about staying at the front of the arms racearms race

TakeawaysTakeaways

As security testers, we:As security testers, we:– Must understand that the threat is Must understand that the threat is

constantly changingconstantly changing We cannot rely on yesterday’s exploitsWe cannot rely on yesterday’s exploits We must try to stay a few steps aheadWe must try to stay a few steps ahead

It’s about what happened yesterday, It’s about what happened yesterday, today today andand what might happen what might happen tomorrowtomorrow

Questions?Questions?

Recommended companion coursesRecommended companion courses– Threat modelingThreat modeling– Fuzz testingFuzz testing

Recommended readingRecommended reading

penetration testing demystified

Documents