DistributedSystemsSecurity

Topics

• Byzan7nefaultresistance

• BitCoin

• CourseWrapUp

FaultTolerance

• Wehavesofarassumed“fail-stop”failures(e.g.,powerfailuresorsystemcrashes)

• Inotherwords,iftheserverisup,itfollowstheprotocol

• Hardenough:

• difficulttodis7nguishbetweencrashvs.networkdown

• difficulttodealwithnetworkpar77on

LargerClassofFailures

• Canonehandlealargerclassoffailures?

• Buggyserversthatcomputeincorrectlyratherthanstopping

• Serversthatdonotfollowtheprotocol

• ServersthathavebeenmodifiedbyanaQacker

• ReferredtoasByzan7nefaults

Model

• Provideareplicatedstatemachineabstrac7on

• Assume2f+1of3f+1nodesarenon-faulty

• Inotherwords,oneneeds3f+1replicastohandleffaults

• Asynchronoussystem,unreliablechannels

• Usecryptography(bothpublic-keyandsecret-keycrypto)

GeneralIdea

• Primary-backupplusquorumsystem

• Execu7onsaresequencesofviews

• Clientssendsignedcommandstoprimaryofcurrentview

• Primaryassignssequencenumbertoclient’scommand

• Primarywritessequencenumbertothe“register”implementedbythequorumsystemdefinedbyalltheservers

AQacker’sPowers

• Worstcase:asingleaQackercontrolstheffaultyreplicas

• Suppliesthecodethatfaultyreplicasrun

• Knowsthecodethenon-faultyreplicasarerunning

• Knowsthefaultyreplicas’cryptokeys

• Canreadnetworkmessages

• CantemporarilyforcemessagestobedelayedviaDoS

Whatfaultscannothappen?

• Nomorethanfoutof3f+1replicascanbefaulty

• Noclientfailure--clientscanneverdoanythingbad(orrathersuchbehaviorcanbedetectedusingstandardtechniques)

• Noguessingofcryptokeysorbreakingofcryptography

• Ques7on:inaPaxosRSMsebng,whatcouldtheaQackersorbyzan7nenodesdo?

Whatcouldgowrong?

• Primarycouldbefaulty!

• Couldignorecommands;assignsamesequencenumbertodifferentrequests;skipsequencenumbers;etc.

• Backupscouldbefaulty!

• Couldincorrectlystorecommandsforwardedbyacorrectprimary

• Faultyreplicascouldincorrectlyrespondtotheclient!

ExampleUseScenario

• Arvind:

echoA>grade

echoB>grade

tellPaul"thegradefileisready"

• Paul:

catgrade

Design1

• client,nservers

• clientsendsrequesttoallofthem

• waitsforallntoreply

• onlyproceedsifallnagree

• whatiswrongwiththisdesign?

Design2

• letushavereplicasvote

• 2f+1servers,assumenomorethanfarefaulty

• clientwaitsforf+1matchingreplies

• ifonlyfarefaulty,andnetworkworkseventually,mustgetthem!

• whatiswrongwithdesign2?

IssueswithDesign2

• f+1matchingrepliesmightbefbadnodes&1good

• somaybeonlyonegoodnodegottheopera7on!

• nextopera7onalsowaitsforf+1

• mightnotincludethatonegoodnodethatsawop1

• example:S1S2S3(S1isbad)

• everyonehearsandrepliestowrite("A")

• S1andS2replytowrite("B"),butS3missesit

• clientcan'twaitforS3sinceitmaybetheonefaultyserver

• S1andS3replytoread(),butS2missesit;read()yields"A"

• result:clienttrickedintoaccep7ngout-of-datestate

Design3

• 3f+1servers,ofwhichatmostfarefaulty

• clientwaitsfor2f+1matchingreplies

• fbadnodesplusamajorityofthegoodnodes

• soallsetsof2f+1overlapinatleastonegoodnode

• doesdesign3haveeverythingweneed?

RefinedApproach

• letushaveaprimarytopickorderforconcurrentclientrequests

• useaquorumof2f+1outof3f+1nodes

• haveamechanismtodealwithfaultyprimary

• replicassendresultsdirecttoclient

• replicasexchangeinfoaboutopssentbyprimary

• clientsno7fyreplicasofeachopera7on,aswellasprimary;ifnoprogress,forcechangeofprimary

PBFT:Overview

• Normalopera7on:howtheprotocolworksintheabsenceoffailures;hopefully,thecommoncase

• Viewchanges:howtodeposeafaultyprimaryandelectanewone

• Garbagecollec7on:howtoreclaimthestorageusedtokeepvariouscer7ficates

• Recovery:howtomakeafaultyreplicabehavecorrectlyagain

NormalOpera7on

• Threephases:

• Pre-prepare:assignssequencenumbertorequest

• Prepare:ensuresfault-tolerantconsistentorderingofrequestswithinviews

• Commit:ensuresfault-tolerantconsistentorderingofrequestsacrossviews

• Eachreplicamaintainsthefollowingstate:

• Servicestate

• Messagelogwithallmessagessent/received

• Integerrepresen7ngthecurrentviewnumber

Clientissuesrequest

• o:statemachineopera7on

• t:7mestamp

• c:clientid

Pre-prepare

• v:view

• n:sequencenumber

• d:digestofm

• m:client’srequest

Pre-prepare

Pre-prepare

Prepare

Prepare

PrepareCer7ficate

• P-cer7ficatesensuretotalorderwithinviews

• ReplicaproducesP-cer7ficate(m,v,n)iffitslogholds:

• Therequestm

• APRE-PREPAREforminviewvwithsequencenumbern

• 2fPREPAREfromdifferentbackupsthatmatchthepre-prepare

• AP-cer7ficate(m,v,n)meansthataquorumagreeswithassigningsequencenumberntominviewv

• Notwonon-faultyreplicaswithP-cer7ficate(m1,v,n)andP-cer7ficate(m2,v,n)

P-cer7ficatesarenotenough

• AP-cer7ficateprovesthatamajorityofcorrectreplicashasagreedonasequencenumberforaclient’srequest

• Yetthatordercouldbemodifiedbyanewleaderelectedinaviewchange

Commit

CommitCer7ficate

• C-cer7ficatesensuretotalorderacrossviews

• can’tmissP-cer7ficateduringaviewchange

• AreplicahasaC-cer7ficate(m,v,n)if:

• ithadaP-cer7ficate(m,v,n)

• logcontains2f+1matchingCOMMITfromdifferentreplicas(includingitself)

• ReplicaexecutesarequestaoeritgetsaC-cer7ficateforit,andhasclearedallrequestswithsmallersequencenumbers

Reply

BackupsDisplacePrimary

• Adisgruntledbackupmu7nies:

• stopsaccep7ngmessages(butforVIEW-CHANGE&NEW-VIEW)

• mul7casts<VIEW-CHANGE,v+1,P>

• PcontainsallP-Cer7ficatesknowntoreplicai

• Abackupjoinsmu7nyaoerseeingf+1dis7nctVIEW-CHANGEmessages

• Mu7nysucceedsifnewprimarycollectsanew-viewcer+ficateV,indica7ngsupportfrom2f+1dis7nctreplicas(includingitself)

ViewChange:NewPrimary

• The“primaryelect”p’(replicav+1modN)extractsfromthenew-viewcer7ficateV:

• thehighestsequencenumberhofanymessageforwhichVcontainsaP-cer7ficate

• twosetsOandN:

• ifthereisaP-cer7ficateforn,minV,n≤h

• O=O∪<PRE-PREPARE,v+1,n,m>

• Otherwise,ifn≤hbutnoP-cer7ficate:

• N=N∪<PRE-PREPARE,v+1,n,null>

• p’mul7casts<NEW-VIEW,v+1,V,O,N>

ViewChange:Backup

• BackupacceptsNEW-VIEWmessageforv+1if

• itissignedproperly

• itcontainsinVavalidVIEW-CHANGEmessagesforv+1

• itcanverifylocallythatOiscorrect(repea7ngtheprimary’scomputa7on)

• AddsallentriesinOtoitslog(sodidp’)

• Mul7castsaPREPAREforeachmessageinO

• AddsallPREPAREtologandentersnewview

GarbageCollec7on

• Forsafety,acorrectreplicakeepsinlogmessagesaboutrequestoun7lit

• ohasbeenexecutedbyamajorityofcorrectreplicas,and

• thisfactcanprovenduringaviewchange

• TruncatelogwithStableCer7ficate

• Eachreplicaiperiodically(aoerprocessingkrequests)checkpointsstateandmul7casts<CHECKPOINT,n,d,i>

• 2f+1CHECKPOINTmessagesareaproofofthecheckpoint’scorrectness

BFTDiscussion

• IsPBFTprac7cal?

• Doesitaddresstheconcernsthatenterpriseuserswouldliketobeaddressed?

Topics

• Byzan7nefaultresistance

• BitCoin

Bitcoin

• adigitalcurrency

• apublicledgertopreventdouble-spending

• nocentralizedtrustormechanism<--thisishard!

Whydigitalcurrency?

• mightmakeonlinepaymentseasier

• creditcardshaveworkedwellbutaren'tperfect

• insecure->fraud->fees,restric7ons,reversals

• recordofallyourpurchases

Whatishardtechnically?

• forgery

• doublespending

• theo

What’shardsocially/economically?

• whydoBitcoinshavevalue?

• howtopayforinfrastructure?

• monetarypolicy(inten7onalinfla7on)

• laws(taxes,laundering,drugs,terrorists)

Idea

• Signedsequenceoftransac7ons

• thereareabunchofcoins,eachownedbysomeone

• everycoinhasasequenceoftransac7onrecords

• oneforeach7methiscoinwastransferredaspayment

• acoin'slatesttransac7onindicateswhoownsitnow

Transac7onRecord

• pub(user1):publickeyofnewowner

• hash(prev):hashofthiscoin'sprevioustransac7onrecord

• sig(user2):signatureovertransac7onbypreviousowner'sprivatekey

• BitCoinhasmorecomplexity:amount(frac7onal),mul7plein/out,...

Transac7onExample

1. Yownsacoin,previouslygiventoitbyX:

• T7:pub(Y),hash(T6),sig(X)

2. YbuysahamburgerfromZandpayswiththiscoin

• ZsendspublickeytoY

• Ycreatesanewtransac7onandsignsit

• T8:pub(Z),hash(T7),sig(Y)

3. Ysendstransac7onrecordtoZ

4. Zverifies:T8'ssig()correspondstoT7'spub()

5. ZgiveshamburgertoY

DoubleSpending

• Ycreatestwotransac7onsforsamecoin:Y->Z,Y->Q

• bothwithhash(T7)

• Yshowsdifferenttransac7onstoZandQ

• bothtransac7onslookgood,includingsignaturesandhash

• nowbothZandQwillgivehamburgerstoY

Defense

• publishlogofalltransac7onstoeveryone,insameorder

• soQknowsaboutY->Z,andwillrejectY->Q

• a"publicledger"

• ensureYcan'tun-publishatransac7on

StrawmanSolu7on

• Assumeap2pnetwork

• Peersfloodnewtransac7onsover“overlay”

• Transac7onisacceptableonlyifmajorityofpeersthinkitisvalid

• Whataretheissueswiththisscheme?

BitCoinBlockChain

• theblockchaincontainstransac7onsonallcoins

• manypeers,eachwithacompletecopyofthechain

• proposedtransac7onsfloodedtoallpeers

• newblocksfloodedtoallpeers

• eachblock:hash(prevblock),setoftransac7ons,nonce,currentwallclock7mestamp

• newblockevery10minutescontainingnewxac7ons

• payeedoesn'tverifyun7lxac7onisintheblockchain

“Mining”Blocks

• requirement:hash(block)hasNleadingzeros

• eachpeertriesnoncevaluesun7lthisworksout

• tryingonenonceisfast,butmostnonceswon'twork

• miningablocknotaspecificfixedamountofwork

• onenodecantakemonthstocreateoneblock

• butthousandsofpeersareworkingonit

• suchthatexpected7metofirsttofindisabout10minutes

• thewinnerfloodsthenewblocktoallpeers

• thereisanincen7vetomineablock—12.5bc

Timing

• start:allpeersknow7llB5

• andareworkingonB6(tryingdifferentnonces)

• YsendsY->Ztransac7ontopeers,whichfloodit

• peersbufferthetransac7onun7lB6iscomputed

• peersthatheardY->Zincludeitinnextblock

• soeventuallyblockchainis:B5,B6,B7,whereB7includesY->Z

DoubleSpending

• whatifYsendsoutY->ZandY->Qatthesame7me?

• nocorrectpeerwillacceptboth

• ablockwillhaveonebutnotboth

• buttherecouldbeafork:B6<-BZandB6<-BQ

ForkedChain

• eachpeerbelieveswhicheverofBZ/BQitsawfirst

• triestocreateasuccessor

• ifmanymoresawBZthanBQ,morewillmineforBZ

• soBZsuccessorlikelytobecreatedfirst

• evenotherwiseonewillbeextendedfirstgivensignificantvarianceinminingsuccess7me

• peersalwaysswitchtominingthelongestfork,reinforcingagreement

DoubleSpendingDefense

• waitforenoughblockstobeminted

• ifafewblockshavebeenminted,unlikelythatadifferentforkwillwin

• ifsellingahigh-valueitem,thenwaitforafewblocksbeforeshipping

• couldaQackerstartaforkfromanoldblock?

• yes,butforkmustbelongerforotherstobelieve

• yes--butforkmustbelongerinorderforpeerstoacceptit

• iftheaQackerhas1000sofCPUs--morethanallthehonestbitcoinpeers--thentheaQackercancreatethelongestfork

• systemworksonlyifnoen7tycontrolsamajorityofnodes

BitCoinSummary

• Keyidea:blockchain

• Publicledgerisagreatidea

• Decentraliza7onmightbegood

• MiningisacleverwaytoavoidsybilaQacks

• WillBitCoinscalewell?

ClassSummary

• Implemen7ngdistributedsystems:systemandprotocoldesign

• Corealgorithms:clocks,snapshots,transac7ons,2PC,Paxos

• Realsystems:VM-FT,DSM,GFS,BigTable,MegaStore,Spanner,Chord,Dynamo

• Abstrac7onsforbigdataanaly7cs

• Buildingsecuresystemsfromuntrustedcomponents

Trends

• Transac7onsovergeo-distributed,replicateddata

• COPS(Princeton),Tapir(UW),RIFL/RamCloud/Rao(Stanford)

• Accelera7ngdistributedsystemsusinghardwaresupport

• Catapult(Microsoo),Annapurna(Amazon),Cavium,Mellanox

• Bigdataanaly7csforDNNs

• MXNet/TVM(UW),Torch,Theano,Dawn(Stanford),Rise(Berkeley)