Lecture3Big-Onotation,morerecurrences!!

Announcements!

• HW1isposted!(DueFriday)

• SeePiazzaforalistofHWclarifications

• Firstrecitationsectionwasthismorning,there’sanothertomorrow(samematerial). (Theseareoptional,it’sachanceforTAstogoovermoreexamplesthanwecangettoinclass).

FAQ• HowrigorousdoIneedtobeonmyhomework?• SeeourexampleHWsolutiononline

• Ingeneral,weareshootingfor:

Youshouldbeabletogiveafriendyoursolution

andtheyshouldbeabletoturnitintoarigorousproofwithoutmuchthought.

• Thisisadelicatelinetowalk,andthere’snoeasyanswer.Thinkofitlikemorelikewritingagoodessaythan“correctly”solvingamathproblem.

• What’swiththearrayboundsinpseudocode?• SORRY!I’mtryingtomatchCLRSandthiscausesmetomakemistakessometimes. Inthisclass,I’mtryingtodo:• Arraysare1-indexed

• A[1..n]isallentriesbetween1andn,inclusive

• IwillalsouseA[1:n](pythonnotation)tomeanthesamething(notpythonnotation).

• PleasecallmeoutwhenImessup.

Lasttime….

• Sorting:InsertionSort andMergeSort

• Analyzingcorrectnessofiterative+recursivealgs

• Via“loopinvariant”andinduction

• Analyzingrunningtimeofrecursivealgorithms

• Bywritingoutatreeandaddingupalltheworkdone.

Today

• Howdowemeasuretheruntimeofanalgorithm?

• Worst-caseanalysis

• AsymptoticAnalysis

• Recurrencerelations:

• IntegerMultiplicationandMergeSort again

• The“MasterMethod”forsolvingrecurrences.

Recallfromlasttime…

• WeanalyzedINSERTIONSORTandMERGESORT.

• Theywerebothcorrect!

• INSERTIONSORTtooktimeaboutn2

• MERGESORT tooktimeaboutnlog(n).

0

200000

400000

600000

800000

1000000

0 200 400 600 800 1000

nlog(n) n^2

nlog(n)iswaybetter!!!

Afewreasonstobegrumpy

• Sorting

shouldtakezerosteps…whynlog(n)??

• What’swiththisT(MERGE)<2+4n<=6n?

1 2 3 4 5 6 7 8

Analysis

T(n)=T(n/2)+T(n/2)+T(MERGE)

Let’ssay

T(MERGEofsizen/2)≤ 6n

operations

Wewillseelaterhowtoanalyzerecurrencerelationslikethese

automagically…buttodaywe’lldoitfromfirstprinciples.

T(n)=timetorun

MERGESORTona

listofsizen

T(MERGEtwolistsofsizen/2)

isthetimetodo:

• 3 variableassignments(counters←1)

• ncomparisons

• nmoreassignments

• 2ncounterincrements

Sothat’s

2T(assign)+nT(compare)+

nT(assign)+2nT(increment)

or4n+2operations

Pluckythe

pedanticpenguin Luckythe

lackadaisicallemur

=2T(n/2)+6n

Thisiscalledarecurrencerelation:it

describestherunningtimeofaproblem

ofsizenintermsoftherunningtimeof

smallerproblems.

Or 4n + 3…

Afewreasonstobegrumpy

• Sorting

shouldtakezerosteps…whynlog(n)??

• What’swiththisT(MERGE)<2+4n<=6n?

• The“2+4n”operationsthingdoesn’tevenmakesense.Differentoperationstakedifferentamountsoftime!

• Webounded2+4n<=6n.Iguessthat’strue,butthatseemsprettydumb.

1 2 3 4 5 6 7 8

Howwewilldealwithgrumpiness

• Takeadeepbreath…

• Worstcaseanalysis

• Asymptoticnotation

Worst-caseanalysis

• Inthisclass,wewillfocusonworst-caseanalysis

• Pros:verystrongguarantee

• Cons:verystrongguarantee

1 2 3 4 5 6 7 8

Sortingasortedlist

shouldbefast!!

Algorithm

designer

Algorithm:

Do the thing

Do the stuff

Return the answer

Here is my algorithm!Here is an

input!

Big-Onotation

• Whatdowemeanwhenwemeasureruntime?

• Weprobablycareaboutwalltime: howlongdoesittaketosolvetheproblem,insecondsorminutesorhours?

• Thisisheavilydependentontheprogramminglanguage,architecture,etc.

• Thesethingsareveryimportant,butarenotthepointofthisclass.

• Wewantawaytotalkabouttherunningtimeofanalgorithm,independentoftheseconsiderations.

Howlongdoesan

operationtake?Whyare

webeingsosloppyabout

that“6”?

Rememberthisslide?

n nlog(n) n^2

8 24 64

16 64 256

32 160 1024

64 384 4096

128 896 16384

256 2048 65536

512 4608 262144

1024 10240 1048576 0

200000

400000

600000

800000

1000000

0 200 400 600 800 1000 1200

nlog(n) n^2

Changenlog(n)to5nlog(n)….

n 5nlog(n) n^2

8 120 64

16 320 256

32 800 1024

64 1920 4096

128 4480 16384

256 10240 65536

512 23040 262144

1024 51200 1048576 0

200000

400000

600000

800000

1000000

0 200 400 600 800 1000 1200

5nlog(n) n^2

AsymptoticAnalysisHowdoestherunningtimescaleasngetslarge?

• Abstractsawayfromhardware- andlanguage-specificissues.

• Makesalgorithmanalysismuchmoretractable.

• Onlymakessenseifnis

large(comparedtothe

constantfactors).

Pros:Cons:

Onealgorithmis“faster”thananotherifits

runtimegrowsmore“slowly”asngetslarge.

2100000000000000 n

is“better”thann2 ?!?!

Thisisespeciallyrelevant

now,asdatagetbiggerand

biggerandbigger…

Thiswillprovideaformalwayofsaying

thatn2 is“worse”than100nlog(n).

• Quickreminders:

• ∃: “Thereexists”• ∀:”Forall”• Example: ∀ studentsinCS161,∃ analgorithmsproblemthatreallyexcitesthestudent.

• Muchstrongerstatement: ∃ analgorithmsproblemsothat,∀ studentsinCS161,thestudentisexcitedbytheproblem.

• We’regoingtoformallydefineanupperbound:

• “T(n)growsnofasterthanf(n)”

Nowforsomedefinitions…

O(…)meansanupperbound

• LetT(n),f(n)befunctionsofpositiveintegers.• ThinkofT(n)asbeingaruntime:positiveandincreasinginn.

• Wesay“T(n)isO(f(n))”iff(n)growsatleastasfastasT(n)asngetslarge.

• Formally,

� � = � � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � � ≤ � ⋅ �(�)

pronounced“big-ohof…”orsometimes“ohof…”

Parsingthat…

� � = � � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � � ≤ � ⋅ �(�)

T(n)

f(n)

cf(n)

n0n

T(n)=O(f(n))means:

Eventually,(forlargeenoughn)

somethingthatgrowslikef(n)

isalwaysbiggerthanT(n).

Example1

• T(n)=n,f(n)=n2.

• T(n)=O(f(n))f(n)

n

T(n)

c=1

n0 =1

� � = � � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � � ≤ � ⋅ �(�)

(formalproof

onboard)

cf(n)=

Examples2and3

• AlldegreekpolynomialswithpositiveleadingcoefficientsareO(nk).

• Foranyk≥ 1,nk isnot O(nk-1).

(Ontheboard)

Take-awayfromexamples

• ToproveT(n)=O(f(n)),youhavetocomeupwithcandn0sothatthedefinitionissatisfied.

• ToproveT(n)isNOT O(f(n)),onewayisbycontradiction:

• Supposethatsomeonegivesyouacandann0 sothatthedefinitionissatisfied.

• Showthatthissomeonemustbylyingtoyoubyderivingacontradiction.

O(…)meansanupperbound,and

Ω(…)meansalowerbound

• Wesay“T(n)isΩ(f(n))”iff(n)growsatmostasfastasT(n)asngetslarge.

• Formally,

� � = Ω � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � ⋅ � � ≤ � �

Switchedthese!!

Parsingthat…

� � = Ω � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � ⋅ � � ≤ � �

T(n)

cf(n)

f(n)

n0n

Θ(…)meansboth!

•Wesay“T(n)isΘ(f(n))”if:

T(n)=O(f(n))

-AND-

T(n)=Ω(f(n))

Yetmoreexamples

• n3 – n2+3n=O(n3)

• n3 – n2+3n=Ω(n3)

• n3 – n2+3n=Θ(n3)

• 3n isnot O(2n)

• nlog(n)=Ω(n)

• nlog(n)isnot Θ(n). Funexercise:

checkallofthese

carefully!!

We’llbeusinglotsofasymptoticnotationfromhereonout

• ThismakesbothPluckyandLuckyhappy.• PluckythePedanticPenguinishappybecausethere

isaprecisedefinition.

• LuckytheLackadaisicalLemurishappybecausewe

don’thavetopaycloseattentiontoallthosepesky

constantfactorslike“4”or“6”.

• Butweshouldalwaysbecarefulnottoabuseit.

• Inthecourse,(almost)everyalgorithmwesee

willbeactuallypractical,withoutneedingto

take� ≥ �. = 2:........

Thisismy

happyface!

Questionsaboutasymptoticnotation?

Backtorecurrencerelations

We’veseenthreerecursivealgorithmssofar.

• Needlesslyrecursiveintegermultiplication

• T(n)=4T(n/2)+O(n)

• T(n)=O(n2)

• Karatsubaintegermultiplication

• T(n)=3T(n/2)+O(n)

• T(n)=O(nlog_2(3) ≈ n1.6)

• MergeSort

• T(n)=2T(n/2)+O(n)

• T(n)=O(nlog(n))

T(n)=timetosolveaproblemofsizen.

What’sthepattern?!?!?!?!

(Remindersonboard)

Themastertheorem

• Aformula thatsolvesrecurrenceswhenallofthesub-problemsarethesamesize.

• (We’llseeanexampleWednesdaywhennotallproblemsarethesamesize).

JedimasterYoda

A useful

formula it is.

Know why it works

you should.

Themastertheorem

• Suppose� � = � ⋅ � => + � �@ .Then

Many symbols

those are….

Threeparameters:

a:numberofsubproblems

b:factorbywhichinputsizeshrinks

d:needtodond worktocreateallthe

subproblems andcombinetheirsolutions.

Examples(detailsonboard)

• Needlesslyrecursiveintegermult.

• T(n)=4T(n/2)+O(n)

• T(n)=O(n2)

• Karatsubaintegermultiplication

• T(n)=3T(n/2)+O(n)

• T(n)=O(nlog_2(3) ≈ n1.6)

• MergeSort

• T(n)=2T(n/2)+O(n)

• T(n)=O(nlog(n))

� � = � ⋅ � => + � �@ .

a=4

b=2

d=1

a=3

b=2

d=1

a=2

b=2

d=1

a>bd

a>bd

a=bd

✓

✓

✓

Proofofthemastertheorem

• We’lldothesamerecursiontreethingwedidforMergeSort,butbemorecareful.

• Supposethat � � = � ⋅ � => + � ⋅ �@.

Pluckythe

PedanticPenguin

Hangon!ThehypothesisoftheMasterTheoremwas

thetheextraworkateachlevelwasO(nd).That’sNOT

thesameaswork<=cn forsomeconstantc.

Luckythe

lackadaisicallemur

That’strue… we’llactuallyproveaweaker

statementthatusesthishypothesisinsteadof

thehypothesisthat� � = � ⋅ � => + � �@ .

It’sagoodexercisetrytomakethisproofwork

rigorouslywiththeO()notation.

Recursiontree

Sizen

n/bn/b

(Size1)

…

n/b2

n/btn/btn/btn/btn/btn/bt

…

Level

Amountof

workatthis

level

0

#

problems

1

2

t

logb(n)

1

a

a2

at

alog_b(n)

Sizeof

each

problem

n

n/b

n/b2

n/bt

1

…

n/b

n/b2

n/b2n/b2

n/b2

n/b2

n/b2

� � = � ⋅ � �� + � ⋅ �@

� ⋅ ��

�C� ��C

@

�� ��@

�D� ��D

@

�EFG_>(=)�

Recursiontree

Sizen

n/bn/b

(Size1)

…

n/b2

n/btn/btn/btn/btn/btn/bt

…

Level

Amountof

workatthis

level

0

#

problems

1

2

t

logb(n)

1

a

a2

at

alog_b(n)

Sizeof

each

problem

n

n/b

n/b2

n/bt

1

…

n/b

n/b2

n/b2n/b2

n/b2

n/b2

n/b2

� � = � ⋅ � �� + � ⋅ �@

� ⋅ ��

�C� ��C

@

�� ��@

�D� ��D

@

�EFG_>(=)�

Totalwork(derivationonboard)isatmost:

� ⋅ �@ ⋅ I ��@

DJKLM(=)

DN.

Nowlet’scheckallthecases(onboard)

Evenmoregenerally,forT(n)=aT(n/b)+f(n)…

Recap

• O()notationmakesourliveseasier.

• The”MasterMethod”alsomakeourliveseasier.

Nexttime:

• Whatifthesub-

problemsare

differentsizes?

• Andwhenmight

thathappen?

Extraslides…

Somebrainteasers

• Aretherefunctionsf,gsothatNEITHER f=O(g)norf=Ω(g)?

• Aretherenon-decreasing functionsf,gsothattheaboveistrue?

• Definethen’th fibonacci numberbyF(0)=1,F(1)=1,F(n)=F(n-1)+F(n-2)forn>2.

• 1,1,2,3,5,8,13,21,34,55,…

Trueorfalse:

• F(n)=O(2n)

• F(n)=Ω(2n)

AfewmoreO()examples

ExampleA

• g(n)=2,f(n)=1.

• g(n)=O(f(n))(andalsof(n)=O(g(n)))

� � = � � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � � ≤ � ⋅ �(�)

f(n)

n0=1n

g(n)

2.1⋅ f(n)

1

2

2.1

ExampleB

• f(n)=1,g(n)asbelow.

• g(n)=O(f(n))(andalsof(n)=O(g(n)))

� � = � � � ⟺

∃�, �. > 0�. �. ∀� ≥ �.,0 ≤ � � ≤ � ⋅ �(�)

f(n)

n

g(n)

2.1⋅ f(n)

1

2

2.1

n0