introduction to probability and...
TRANSCRIPT
![Page 1: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/1.jpg)
Introduction to probability and statistics
Alireza Fotuhi Siahpirani & Brittany [email protected]
Computational Network BiologyBiostatistics & Medical Informatics 826
https://compnetbiocourse.discovery.wisc.edu
Sep 13th 2016
Some of the material covered in this lecture is adapted from BMI 576
![Page 2: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/2.jpg)
Goals for today
• Probability primer• Introduction to linear regression
![Page 3: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/3.jpg)
A few key concepts
• Sample spaces• Random variables• Discrete and continuous continuous
distributions• Joint, conditional and marginal distributions• Statistical independence
![Page 4: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/4.jpg)
Definition of probability
• Intuitively, we use “probability” to refer to our degree of confidence in an event of an uncertain nature.
• Always a number in the interval [0,1]0 means “never occurs”1 means “always occurs”
![Page 5: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/5.jpg)
Sample space
• Sample space: a set of possible outcomes for some experiment• Examples– Flight to Chicago: {on time, late}– Lottery: {ticket 1 wins, ticket 2 wins,…,ticket n wins}– Weather tomorrow:
{rain, not rain} or{sun, rain, snow} or{sun, clouds, rain, snow, sleet}
– Roll of a die: {1,2,3,4,5,6}– Coin toss: {Heads, Tail}
![Page 6: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/6.jpg)
Random variables
• Random variable: A variable that represents the outcome of a uncertain experiment
• A random variable can be – Discrete/Categorical: Outcomes take a fixed set of
values• Roll of die, flight to Chicago, weather tomorrow
– Continuous: Outcomes take continuous values• Height, weight
![Page 7: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/7.jpg)
Notation• Uppercase letters and words denote random variables– X, Y
• Lowercase letters and words denote values– x, y
• Probability that X takes value x
• We will also use the shorthand form
• For Boolean random variables, we will use the shorthand€
P(X = x)
P(x) for P(X=x)
P(fever) for P(Fever = true)P(¬fever) for P(Fever = false)
![Page 8: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/8.jpg)
Discrete probability distributions
• A probability distribution is a mathematical function that specifies the probability of each possible outcome of a random variable
• We denote this as P(X) for random variable X• It specifies the probability of each possible value of X, x• Requirements:
€
P(x) =1x∑
sun
cloud
sra
insn
ow sleet
0.2
0.3
0.1
€
P(x) ≥ 0 for every x
![Page 9: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/9.jpg)
Joint probability distributions• Joint probability distribution: the function given by P(X =
x, Y = y)• Read as “X equals x and Y equals y”• Example
x, y P(X = x, Y = y)sun, on-time 0.20
rain, on-time 0.20
snow, on-time 0.05
sun, late 0.10
rain, late 0.30
snow, late 0.15
probability that it’s sunny and my flight is on time
![Page 10: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/10.jpg)
Marginal probability distributions
• The marginal distribution of X is defined by
“the distribution of X ignoring other variables”
• This definition generalizes to more than two variables, e.g.€
P(x) = P(x,y)y∑
€
P(x) = P(x,y,z)z∑
y∑
![Page 11: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/11.jpg)
Marginal distribution example
x, y P(X = x, Y = y)sun, on-time 0.20
rain, on-time 0.20
snow, on-time 0.05
sun, late 0.10
rain, late 0.30
snow, late 0.15
x P(X = x)sun 0.3
rain 0.5
snow 0.2
joint distribution marginal distribution for X
![Page 12: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/12.jpg)
Conditional distributions
• The conditional distribution of X given Y is defined as:
• Or in short
• The distribution of X given that we know the value of Y
• Intuitively, how much does knowing Y tell us about X?
€
P(X = x |Y = y) =P(X = x,Y = y)
P(Y = y)
P(X |Y ) = P(X,Y )P(Y )
![Page 13: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/13.jpg)
Conditional distribution example
x, y P(X = x, Y = y)sun, on-time 0.20
rain, on-time 0.20
snow, on-time 0.05
sun, late 0.10
rain, late 0.30
snow, late 0.15
x P(X = x|Y=on-time)sun 0.20/0.45 = 0.444
rain 0.20/0.45 = 0.444
snow 0.05/0.45 = 0.111
joint distributionconditional distribution for Xgiven Y=on-time
![Page 14: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/14.jpg)
Independence
• Two random variables, X and Y, are independent if
• Another way to think about this is knowing X does not tell us anything about Y
€
P(x,y) = P(x) × P(y) for all x and y
![Page 15: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/15.jpg)
Independence example #1
x, y P(X = x, Y = y)
sun, on-time 0.20
rain, on-time 0.20
snow, on-time 0.05
sun, late 0.10
rain, late 0.30
snow, late 0.15
x P(X = x)sun 0.3
rain 0.5
snow 0.2
joint distribution marginal distributions
y P(Y = y)on-time 0.45
late 0.55
Are X and Y independent here? NO.
![Page 16: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/16.jpg)
Independence example #2
x, y P(X = x, Y = y)
sun, fly-United 0.27
rain, fly-United 0.45
snow, fly-United 0.18
sun, fly-Northwest 0.03
rain, fly-Northwest 0.05
snow, fly-Northwest 0.02
x P(X = x)sun 0.3
rain 0.5
snow 0.2
joint distribution marginal distributions
y P(Y = y)fly-United 0.9
fly-Northwest 0.1
Are X and Y independent here? YES.
![Page 17: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/17.jpg)
• Two random variables X and Y are conditionally independentgiven Z if
“once you know the value of Z, knowing Y doesn’t tell you anything about X ”
• Alternatively
Conditional independence
€
P(X |Y,Z) = P(X | Z)
€
P(x,y | z) = P(x | z) × P(y | z) for all x,y,z
![Page 18: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/18.jpg)
Conditional independence exampleFlu Fever Headache Ptrue true true 0.04true true false 0.04true false true 0.01true false false 0.01false true true 0.009false true false 0.081false false true 0.081false false false 0.729
€
e.g. P( fever,headache) ≠ P( fever) × P(headache)
Are Fever and Headache independent? NO.
![Page 19: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/19.jpg)
Conditional independence exampleFlu Fever Headache Ptrue true true 0.04true true false 0.04true false true 0.01true false false 0.01false true true 0.009false true false 0.081false false true 0.081false false false 0.729
Are Fever and Headache conditionally independent given Flu:
€
P( fever,headache | flu) = P( fever | flu) × P(headache | flu)P( fever,headache |¬flu) = P( fever |¬flu) × P(headache |¬flu)etc.
YES.
![Page 20: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/20.jpg)
Chain rule of probability• For two variables
• For three variables
etc.• To see that this is true, note that
€
P(X,Y ) = P(X |Y ) × P(Y )
€
P(X,Y,Z) = P(X |Y,Z) × P(Y | Z) × P(Z)
€
P(X,Y,Z) =P(X,Y,Z)P(Y,Z)
×P(Y,Z)P(Z)
× P(Z)
![Page 21: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/21.jpg)
Example discrete distributions
• Binomial distribution
• Multinomial distribution
![Page 22: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/22.jpg)
• Two outcomes per trial of an experiment• Distribution over the number of successes in a fixed number n of
independent trials (with same probability of success p in each)
• e.g. the probability of x heads in n coin flips
The binomial distribution
€
P(x) =nx"
# $ %
& ' px (1− p)n−x
P(X=
x)
p=0.5 p=0.1
x x
P(X=
x)
![Page 23: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/23.jpg)
The multinomial distribution• A generalization of the binomial distribution to more than two outcomes• Provides a distribution of the number of times any of the outcomes
happen.• For example consider rolling of a die n = 100 times. Each time we can have
one of k = 6 outcomes, 1, . . , 6• &' is the variable representing the number of times the die landed on the
ith face, ( ∈ {1, . . , 6}• ,' is the probability of the die landing on the ith face
![Page 24: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/24.jpg)
Continuous random variables• When our outcome is a continuous number we need
a continuous random variable• Examples: Weight, Height• We specify a density function for random variable X
as
• Probabilities are specified over an interval• Probability of taking on a single value is 0.
f(x) � 0Z 1
�1f(x)dx = 1
![Page 25: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/25.jpg)
Continuous random variables contd
• To define a probability distribution for a continuous variable, we need to integrate f(x)
P (X a) =Z a
�1f(x)dx
P (b X a) =Z a
bf(x)dx
![Page 26: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/26.jpg)
Example continuous distributions
• Uniform distribution
• Gaussian distribution
• Exponential distribution
![Page 27: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/27.jpg)
Uniform distribution
• A variable X is said to have a uniform distribution, between [", $], where, " < $, if
f(x) =
(1
b�a , if x 2 [a, b]
0, otherwise<latexit sha1_base64="Jvcx6cryp9SgJ605xe3rG3BRQMk=">AAACRXicbVBBSxtBGJ2NttXY1lWPXgaDJYU07IpQL4LoxaOC0UB2CbOTb5PB2dll5ts2Ydk/56X33vwHXjxYiledjbG0SR8MPN77vjczL8qkMOh5t05tafnN23crq/W19x8+rrsbm5cmzTWHDk9lqrsRMyCFgg4KlNDNNLAkknAVXZ9U/tU30Eak6gInGYQJGyoRC87QSn03iJvjz4dBBEOhCm6DTFkPYs144ZdF9IWVrU80QBhjIWJa0nEgFO2xVhQGQd3746U4Av1dGLDLoAavQX234bW9Kegi8WekQWY467s/g0HK8wQUcsmM6flehmHBNAouq/DcQMb4NRtCz1LFEjBhMW2hpLtWGdA41fYopFP1742CJcZMkshOJgxHZt6rxP95vRzjg7AQKssRFH+5KM4lxZRWldKB0MBRTixhXAv7VspHzHaItviqBH/+y4vkcq/te23/fL9xdDyrY4Vskx3SJD75So7IKTkjHcLJDbkjD+SX88O5d347jy+jNWe2s0X+gfP0DLsKsTs=</latexit><latexit sha1_base64="Jvcx6cryp9SgJ605xe3rG3BRQMk=">AAACRXicbVBBSxtBGJ2NttXY1lWPXgaDJYU07IpQL4LoxaOC0UB2CbOTb5PB2dll5ts2Ydk/56X33vwHXjxYiledjbG0SR8MPN77vjczL8qkMOh5t05tafnN23crq/W19x8+rrsbm5cmzTWHDk9lqrsRMyCFgg4KlNDNNLAkknAVXZ9U/tU30Eak6gInGYQJGyoRC87QSn03iJvjz4dBBEOhCm6DTFkPYs144ZdF9IWVrU80QBhjIWJa0nEgFO2xVhQGQd3746U4Av1dGLDLoAavQX234bW9Kegi8WekQWY467s/g0HK8wQUcsmM6flehmHBNAouq/DcQMb4NRtCz1LFEjBhMW2hpLtWGdA41fYopFP1742CJcZMkshOJgxHZt6rxP95vRzjg7AQKssRFH+5KM4lxZRWldKB0MBRTixhXAv7VspHzHaItviqBH/+y4vkcq/te23/fL9xdDyrY4Vskx3SJD75So7IKTkjHcLJDbkjD+SX88O5d347jy+jNWe2s0X+gfP0DLsKsTs=</latexit><latexit sha1_base64="Jvcx6cryp9SgJ605xe3rG3BRQMk=">AAACRXicbVBBSxtBGJ2NttXY1lWPXgaDJYU07IpQL4LoxaOC0UB2CbOTb5PB2dll5ts2Ydk/56X33vwHXjxYiledjbG0SR8MPN77vjczL8qkMOh5t05tafnN23crq/W19x8+rrsbm5cmzTWHDk9lqrsRMyCFgg4KlNDNNLAkknAVXZ9U/tU30Eak6gInGYQJGyoRC87QSn03iJvjz4dBBEOhCm6DTFkPYs144ZdF9IWVrU80QBhjIWJa0nEgFO2xVhQGQd3746U4Av1dGLDLoAavQX234bW9Kegi8WekQWY467s/g0HK8wQUcsmM6flehmHBNAouq/DcQMb4NRtCz1LFEjBhMW2hpLtWGdA41fYopFP1742CJcZMkshOJgxHZt6rxP95vRzjg7AQKssRFH+5KM4lxZRWldKB0MBRTixhXAv7VspHzHaItviqBH/+y4vkcq/te23/fL9xdDyrY4Vskx3SJD75So7IKTkjHcLJDbkjD+SX88O5d347jy+jNWe2s0X+gfP0DLsKsTs=</latexit><latexit sha1_base64="Jvcx6cryp9SgJ605xe3rG3BRQMk=">AAACRXicbVBBSxtBGJ2NttXY1lWPXgaDJYU07IpQL4LoxaOC0UB2CbOTb5PB2dll5ts2Ydk/56X33vwHXjxYiledjbG0SR8MPN77vjczL8qkMOh5t05tafnN23crq/W19x8+rrsbm5cmzTWHDk9lqrsRMyCFgg4KlNDNNLAkknAVXZ9U/tU30Eak6gInGYQJGyoRC87QSn03iJvjz4dBBEOhCm6DTFkPYs144ZdF9IWVrU80QBhjIWJa0nEgFO2xVhQGQd3746U4Av1dGLDLoAavQX234bW9Kegi8WekQWY467s/g0HK8wQUcsmM6flehmHBNAouq/DcQMb4NRtCz1LFEjBhMW2hpLtWGdA41fYopFP1742CJcZMkshOJgxHZt6rxP95vRzjg7AQKssRFH+5KM4lxZRWldKB0MBRTixhXAv7VspHzHaItviqBH/+y4vkcq/te23/fL9xdDyrY4Vskx3SJD75So7IKTkjHcLJDbkjD+SX88O5d347jy+jNWe2s0X+gfP0DLsKsTs=</latexit>
a b
1$ − "
x
)(+)
Adapted from Wikipedia
![Page 28: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/28.jpg)
Gaussian Distribution• The univariate Gaussian distribution is defined by
two parameters, Mean: ! and Standard deviation: "
f(x) =1p2⇡�2
exp(� (x� µ)2
2�2)
From Wikipedia: Normal distribution, https://en.wikipedia.org/wiki/Normal_distribution
![Page 29: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/29.jpg)
Exponential Distribution• Exponential distribution for a random variable X is defined as
f(x) = �e��x<latexit sha1_base64="QySutEi+S4Vq5S2yuYLFmD+O4MY=">AAACBXicbVC7SgNBFL0bXzG+Vi21GAxCLAy7ImgjBG0sI5gHJGuYnZ1Nhsw+mJmVhCWNjb9iY6GIrf9g5984m0TQxAMDh3Pu4c49bsyZVJb1ZeQWFpeWV/KrhbX1jc0tc3unLqNEEFojEY9E08WSchbSmmKK02YsKA5cThtu/yrzG/dUSBaFt2oYUyfA3ZD5jGClpY6575cGRxdtrhMeRvQuPf7hg1GhYxatsjUGmif2lBRhimrH/Gx7EUkCGirCsZQt24qVk2KhGOF0VGgnksaY9HGXtjQNcUClk46vGKFDrXjIj4R+oUJj9XcixYGUw8DVkwFWPTnrZeJ/XitR/rmTsjBOFA3JZJGfcKQilFWCPCYoUXyoCSaC6b8i0sMCE6WLy0qwZ0+eJ/WTsm2V7ZvTYuVyWkce9uAASmDDGVTgGqpQAwIP8AQv8Go8Gs/Gm/E+Gc0Z08wu/IHx8Q2Qg5dT</latexit><latexit sha1_base64="QySutEi+S4Vq5S2yuYLFmD+O4MY=">AAACBXicbVC7SgNBFL0bXzG+Vi21GAxCLAy7ImgjBG0sI5gHJGuYnZ1Nhsw+mJmVhCWNjb9iY6GIrf9g5984m0TQxAMDh3Pu4c49bsyZVJb1ZeQWFpeWV/KrhbX1jc0tc3unLqNEEFojEY9E08WSchbSmmKK02YsKA5cThtu/yrzG/dUSBaFt2oYUyfA3ZD5jGClpY6575cGRxdtrhMeRvQuPf7hg1GhYxatsjUGmif2lBRhimrH/Gx7EUkCGirCsZQt24qVk2KhGOF0VGgnksaY9HGXtjQNcUClk46vGKFDrXjIj4R+oUJj9XcixYGUw8DVkwFWPTnrZeJ/XitR/rmTsjBOFA3JZJGfcKQilFWCPCYoUXyoCSaC6b8i0sMCE6WLy0qwZ0+eJ/WTsm2V7ZvTYuVyWkce9uAASmDDGVTgGqpQAwIP8AQv8Go8Gs/Gm/E+Gc0Z08wu/IHx8Q2Qg5dT</latexit><latexit sha1_base64="QySutEi+S4Vq5S2yuYLFmD+O4MY=">AAACBXicbVC7SgNBFL0bXzG+Vi21GAxCLAy7ImgjBG0sI5gHJGuYnZ1Nhsw+mJmVhCWNjb9iY6GIrf9g5984m0TQxAMDh3Pu4c49bsyZVJb1ZeQWFpeWV/KrhbX1jc0tc3unLqNEEFojEY9E08WSchbSmmKK02YsKA5cThtu/yrzG/dUSBaFt2oYUyfA3ZD5jGClpY6575cGRxdtrhMeRvQuPf7hg1GhYxatsjUGmif2lBRhimrH/Gx7EUkCGirCsZQt24qVk2KhGOF0VGgnksaY9HGXtjQNcUClk46vGKFDrXjIj4R+oUJj9XcixYGUw8DVkwFWPTnrZeJ/XitR/rmTsjBOFA3JZJGfcKQilFWCPCYoUXyoCSaC6b8i0sMCE6WLy0qwZ0+eJ/WTsm2V7ZvTYuVyWkce9uAASmDDGVTgGqpQAwIP8AQv8Go8Gs/Gm/E+Gc0Z08wu/IHx8Q2Qg5dT</latexit><latexit sha1_base64="QySutEi+S4Vq5S2yuYLFmD+O4MY=">AAACBXicbVC7SgNBFL0bXzG+Vi21GAxCLAy7ImgjBG0sI5gHJGuYnZ1Nhsw+mJmVhCWNjb9iY6GIrf9g5984m0TQxAMDh3Pu4c49bsyZVJb1ZeQWFpeWV/KrhbX1jc0tc3unLqNEEFojEY9E08WSchbSmmKK02YsKA5cThtu/yrzG/dUSBaFt2oYUyfA3ZD5jGClpY6575cGRxdtrhMeRvQuPf7hg1GhYxatsjUGmif2lBRhimrH/Gx7EUkCGirCsZQt24qVk2KhGOF0VGgnksaY9HGXtjQNcUClk46vGKFDrXjIj4R+oUJj9XcixYGUw8DVkwFWPTnrZeJ/XitR/rmTsjBOFA3JZJGfcKQilFWCPCYoUXyoCSaC6b8i0sMCE6WLy0qwZ0+eJ/WTsm2V7ZvTYuVyWkce9uAASmDDGVTgGqpQAwIP8AQv8Go8Gs/Gm/E+Gc0Z08wu/IHx8Q2Qg5dT</latexit>
From Wikipedia: Exponential distribution, https://en.wikipedia.org/wiki/Exponential_distribution
![Page 30: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/30.jpg)
Goals for today
• Probability primer• Introduction to linear regression
![Page 31: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/31.jpg)
Linear regression
y
x
Linear regression assumes that output (y) is a linear function of the input (x)
Slope Intercept
Given: Data=
Estimate:
![Page 32: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/32.jpg)
Residual Sum of Squares (RSS)
Residualy
x
To find the !, we need to minimize the Residual Sum of Squares
![Page 33: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/33.jpg)
Minimizing RSS
Residualy
x
![Page 34: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/34.jpg)
Linear regression with p inputs
• Y: output• Inputs:
intercept Parameters/coefficients
Given: Data=
Estimate:
{X1, · · · , Xp}<latexit sha1_base64="ceVT8pwCMNFxj0PN1cUH9c9M7Zs=">AAAB+3icbVBNS8NAEN34WetXrEcvi0XwUEoigh6LXjxWsG2gCWGz2bRLN7thdyOWkL/ixYMiXv0j3vw3btsctPXBwOO9GWbmRRmjSjvOt7W2vrG5tV3bqe/u7R8c2keNvhK5xKSHBRPSi5AijHLS01Qz4mWSoDRiZBBNbmf+4JFIRQV/0NOMBCkacZpQjLSRQrvhF17otnwcC61aXpj5ZWg3nbYzB1wlbkWaoEI3tL/8WOA8JVxjhpQauk6mgwJJTTEjZd3PFckQnqARGRrKUUpUUMxvL+GZUWKYCGmKazhXf08UKFVqmkamM0V6rJa9mfifN8x1ch0UlGe5JhwvFiU5g1rAWRAwppJgzaaGICypuRXiMZIIaxNX3YTgLr+8SvoXbddpu/eXzc5NFUcNnIBTcA5ccAU64A50QQ9g8ASewSt4s0rrxXq3Phata1Y1cwz+wPr8AUWWk+8=</latexit><latexit sha1_base64="ceVT8pwCMNFxj0PN1cUH9c9M7Zs=">AAAB+3icbVBNS8NAEN34WetXrEcvi0XwUEoigh6LXjxWsG2gCWGz2bRLN7thdyOWkL/ixYMiXv0j3vw3btsctPXBwOO9GWbmRRmjSjvOt7W2vrG5tV3bqe/u7R8c2keNvhK5xKSHBRPSi5AijHLS01Qz4mWSoDRiZBBNbmf+4JFIRQV/0NOMBCkacZpQjLSRQrvhF17otnwcC61aXpj5ZWg3nbYzB1wlbkWaoEI3tL/8WOA8JVxjhpQauk6mgwJJTTEjZd3PFckQnqARGRrKUUpUUMxvL+GZUWKYCGmKazhXf08UKFVqmkamM0V6rJa9mfifN8x1ch0UlGe5JhwvFiU5g1rAWRAwppJgzaaGICypuRXiMZIIaxNX3YTgLr+8SvoXbddpu/eXzc5NFUcNnIBTcA5ccAU64A50QQ9g8ASewSt4s0rrxXq3Phata1Y1cwz+wPr8AUWWk+8=</latexit><latexit sha1_base64="ceVT8pwCMNFxj0PN1cUH9c9M7Zs=">AAAB+3icbVBNS8NAEN34WetXrEcvi0XwUEoigh6LXjxWsG2gCWGz2bRLN7thdyOWkL/ixYMiXv0j3vw3btsctPXBwOO9GWbmRRmjSjvOt7W2vrG5tV3bqe/u7R8c2keNvhK5xKSHBRPSi5AijHLS01Qz4mWSoDRiZBBNbmf+4JFIRQV/0NOMBCkacZpQjLSRQrvhF17otnwcC61aXpj5ZWg3nbYzB1wlbkWaoEI3tL/8WOA8JVxjhpQauk6mgwJJTTEjZd3PFckQnqARGRrKUUpUUMxvL+GZUWKYCGmKazhXf08UKFVqmkamM0V6rJa9mfifN8x1ch0UlGe5JhwvFiU5g1rAWRAwppJgzaaGICypuRXiMZIIaxNX3YTgLr+8SvoXbddpu/eXzc5NFUcNnIBTcA5ccAU64A50QQ9g8ASewSt4s0rrxXq3Phata1Y1cwz+wPr8AUWWk+8=</latexit><latexit sha1_base64="ceVT8pwCMNFxj0PN1cUH9c9M7Zs=">AAAB+3icbVBNS8NAEN34WetXrEcvi0XwUEoigh6LXjxWsG2gCWGz2bRLN7thdyOWkL/ixYMiXv0j3vw3btsctPXBwOO9GWbmRRmjSjvOt7W2vrG5tV3bqe/u7R8c2keNvhK5xKSHBRPSi5AijHLS01Qz4mWSoDRiZBBNbmf+4JFIRQV/0NOMBCkacZpQjLSRQrvhF17otnwcC61aXpj5ZWg3nbYzB1wlbkWaoEI3tL/8WOA8JVxjhpQauk6mgwJJTTEjZd3PFckQnqARGRrKUUpUUMxvL+GZUWKYCGmKazhXf08UKFVqmkamM0V6rJa9mfifN8x1ch0UlGe5JhwvFiU5g1rAWRAwppJgzaaGICypuRXiMZIIaxNX3YTgLr+8SvoXbddpu/eXzc5NFUcNnIBTcA5ccAU64A50QQ9g8ASewSt4s0rrxXq3Phata1Y1cwz+wPr8AUWWk+8=</latexit>
y = f(x) = �0 +pX
j=1
xj�j
<latexit sha1_base64="k016TN1kcA4frqF1/0KPZOnn+RM=">AAACGnicbVDLSsNAFJ34rPUVdelmsAgVoSQi6KZQdOOygn1AE8NkOmmnnTyYmUhDyHe48VfcuFDEnbjxb5y0WWjrgYHDOfdy5xw3YlRIw/jWlpZXVtfWSxvlza3tnV19b78twphj0sIhC3nXRYIwGpCWpJKRbsQJ8l1GOu74Ovc7D4QLGgZ3MomI7aNBQD2KkVSSo5tJ3ataPpJD10sn2UndcolEjgFPLRH7Tjqqm9l9GmUTZzRzRo5eMWrGFHCRmAWpgAJNR/+0+iGOfRJIzJAQPdOIpJ0iLilmJCtbsSARwmM0ID1FA+QTYafTaBk8VkofeiFXL5Bwqv7eSJEvROK7ajIPIea9XPzP68XSu7RTGkSxJAGeHfJiBmUI855gn3KCJUsUQZhT9VeIh4gjLFWbZVWCOR95kbTPaqZRM2/PK42roo4SOARHoApMcAEa4AY0QQtg8AiewSt40560F+1d+5iNLmnFzgH4A+3rBy2loPc=</latexit><latexit sha1_base64="k016TN1kcA4frqF1/0KPZOnn+RM=">AAACGnicbVDLSsNAFJ34rPUVdelmsAgVoSQi6KZQdOOygn1AE8NkOmmnnTyYmUhDyHe48VfcuFDEnbjxb5y0WWjrgYHDOfdy5xw3YlRIw/jWlpZXVtfWSxvlza3tnV19b78twphj0sIhC3nXRYIwGpCWpJKRbsQJ8l1GOu74Ovc7D4QLGgZ3MomI7aNBQD2KkVSSo5tJ3ataPpJD10sn2UndcolEjgFPLRH7Tjqqm9l9GmUTZzRzRo5eMWrGFHCRmAWpgAJNR/+0+iGOfRJIzJAQPdOIpJ0iLilmJCtbsSARwmM0ID1FA+QTYafTaBk8VkofeiFXL5Bwqv7eSJEvROK7ajIPIea9XPzP68XSu7RTGkSxJAGeHfJiBmUI855gn3KCJUsUQZhT9VeIh4gjLFWbZVWCOR95kbTPaqZRM2/PK42roo4SOARHoApMcAEa4AY0QQtg8AiewSt40560F+1d+5iNLmnFzgH4A+3rBy2loPc=</latexit><latexit sha1_base64="k016TN1kcA4frqF1/0KPZOnn+RM=">AAACGnicbVDLSsNAFJ34rPUVdelmsAgVoSQi6KZQdOOygn1AE8NkOmmnnTyYmUhDyHe48VfcuFDEnbjxb5y0WWjrgYHDOfdy5xw3YlRIw/jWlpZXVtfWSxvlza3tnV19b78twphj0sIhC3nXRYIwGpCWpJKRbsQJ8l1GOu74Ovc7D4QLGgZ3MomI7aNBQD2KkVSSo5tJ3ataPpJD10sn2UndcolEjgFPLRH7Tjqqm9l9GmUTZzRzRo5eMWrGFHCRmAWpgAJNR/+0+iGOfRJIzJAQPdOIpJ0iLilmJCtbsSARwmM0ID1FA+QTYafTaBk8VkofeiFXL5Bwqv7eSJEvROK7ajIPIea9XPzP68XSu7RTGkSxJAGeHfJiBmUI855gn3KCJUsUQZhT9VeIh4gjLFWbZVWCOR95kbTPaqZRM2/PK42roo4SOARHoApMcAEa4AY0QQtg8AiewSt40560F+1d+5iNLmnFzgH4A+3rBy2loPc=</latexit><latexit sha1_base64="k016TN1kcA4frqF1/0KPZOnn+RM=">AAACGnicbVDLSsNAFJ34rPUVdelmsAgVoSQi6KZQdOOygn1AE8NkOmmnnTyYmUhDyHe48VfcuFDEnbjxb5y0WWjrgYHDOfdy5xw3YlRIw/jWlpZXVtfWSxvlza3tnV19b78twphj0sIhC3nXRYIwGpCWpJKRbsQJ8l1GOu74Ovc7D4QLGgZ3MomI7aNBQD2KkVSSo5tJ3ataPpJD10sn2UndcolEjgFPLRH7Tjqqm9l9GmUTZzRzRo5eMWrGFHCRmAWpgAJNR/+0+iGOfRJIzJAQPdOIpJ0iLilmJCtbsSARwmM0ID1FA+QTYafTaBk8VkofeiFXL5Bwqv7eSJEvROK7ajIPIea9XPzP68XSu7RTGkSxJAGeHfJiBmUI855gn3KCJUsUQZhT9VeIh4gjLFWbZVWCOR95kbTPaqZRM2/PK42roo4SOARHoApMcAEa4AY0QQtg8AiewSt40560F+1d+5iNLmnFzgH4A+3rBy2loPc=</latexit>
![Page 35: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/35.jpg)
Ordinary least squares for estimating !
• Pick the " that minimizes the residual sum of squares RSS
![Page 36: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/36.jpg)
How to minimize RSS?
• Easier to think in matrix form
This is the square of y-Xb in matrix world
RSS(�) = (Y �X�)T (Y �X�)<latexit sha1_base64="l05YhGRTVExrd4thxQIjsb3mKlA=">AAACHnicbZDLSsNAFIYnXmu9RV26GSxCXVgSUXQjFN24rPYqTSyT6aQdOpmEmYlQQp7Eja/ixoUigit9GydtF9r2h4Gf75zDnPN7EaNSWdaPsbC4tLyymlvLr29sbm2bO7sNGcYCkzoOWShaHpKEUU7qiipGWpEgKPAYaXqD66zefCRC0pDX1DAiboB6nPoUI6VRxzy7q1aLjkcUOros3h87AVJ9z09a6Zg9JLV0Du6YBatkjQRnjT0xBTBRpWN+Od0QxwHhCjMkZdu2IuUmSCiKGUnzTixJhPAA9UhbW44CIt1kdF4KDzXpQj8U+nEFR/TvRIICKYeBpzuzPeV0LYPzau1Y+RduQnkUK8Lx+CM/ZlCFMMsKdqkgWLGhNggLqneFuI8Ewkonmtch2NMnz5rGScm2SvbtaaF8NYkjB/bBASgCG5yDMrgBFVAHGDyBF/AG3o1n49X4MD7HrQvGZGYP/JPx/Qvkg6G/</latexit><latexit sha1_base64="l05YhGRTVExrd4thxQIjsb3mKlA=">AAACHnicbZDLSsNAFIYnXmu9RV26GSxCXVgSUXQjFN24rPYqTSyT6aQdOpmEmYlQQp7Eja/ixoUigit9GydtF9r2h4Gf75zDnPN7EaNSWdaPsbC4tLyymlvLr29sbm2bO7sNGcYCkzoOWShaHpKEUU7qiipGWpEgKPAYaXqD66zefCRC0pDX1DAiboB6nPoUI6VRxzy7q1aLjkcUOros3h87AVJ9z09a6Zg9JLV0Du6YBatkjQRnjT0xBTBRpWN+Od0QxwHhCjMkZdu2IuUmSCiKGUnzTixJhPAA9UhbW44CIt1kdF4KDzXpQj8U+nEFR/TvRIICKYeBpzuzPeV0LYPzau1Y+RduQnkUK8Lx+CM/ZlCFMMsKdqkgWLGhNggLqneFuI8Ewkonmtch2NMnz5rGScm2SvbtaaF8NYkjB/bBASgCG5yDMrgBFVAHGDyBF/AG3o1n49X4MD7HrQvGZGYP/JPx/Qvkg6G/</latexit><latexit sha1_base64="l05YhGRTVExrd4thxQIjsb3mKlA=">AAACHnicbZDLSsNAFIYnXmu9RV26GSxCXVgSUXQjFN24rPYqTSyT6aQdOpmEmYlQQp7Eja/ixoUigit9GydtF9r2h4Gf75zDnPN7EaNSWdaPsbC4tLyymlvLr29sbm2bO7sNGcYCkzoOWShaHpKEUU7qiipGWpEgKPAYaXqD66zefCRC0pDX1DAiboB6nPoUI6VRxzy7q1aLjkcUOros3h87AVJ9z09a6Zg9JLV0Du6YBatkjQRnjT0xBTBRpWN+Od0QxwHhCjMkZdu2IuUmSCiKGUnzTixJhPAA9UhbW44CIt1kdF4KDzXpQj8U+nEFR/TvRIICKYeBpzuzPeV0LYPzau1Y+RduQnkUK8Lx+CM/ZlCFMMsKdqkgWLGhNggLqneFuI8Ewkonmtch2NMnz5rGScm2SvbtaaF8NYkjB/bBASgCG5yDMrgBFVAHGDyBF/AG3o1n49X4MD7HrQvGZGYP/JPx/Qvkg6G/</latexit><latexit sha1_base64="l05YhGRTVExrd4thxQIjsb3mKlA=">AAACHnicbZDLSsNAFIYnXmu9RV26GSxCXVgSUXQjFN24rPYqTSyT6aQdOpmEmYlQQp7Eja/ixoUigit9GydtF9r2h4Gf75zDnPN7EaNSWdaPsbC4tLyymlvLr29sbm2bO7sNGcYCkzoOWShaHpKEUU7qiipGWpEgKPAYaXqD66zefCRC0pDX1DAiboB6nPoUI6VRxzy7q1aLjkcUOros3h87AVJ9z09a6Zg9JLV0Du6YBatkjQRnjT0xBTBRpWN+Od0QxwHhCjMkZdu2IuUmSCiKGUnzTixJhPAA9UhbW44CIt1kdF4KDzXpQj8U+nEFR/TvRIICKYeBpzuzPeV0LYPzau1Y+RduQnkUK8Lx+CM/ZlCFMMsKdqkgWLGhNggLqneFuI8Ewkonmtch2NMnz5rGScm2SvbtaaF8NYkjB/bBASgCG5yDMrgBFVAHGDyBF/AG3o1n49X4MD7HrQvGZGYP/JPx/Qvkg6G/</latexit>
![Page 37: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/37.jpg)
Simple matrix calculus
The Matrix Cookbookhttps://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
1.
2.
3.
![Page 38: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/38.jpg)
Estimating ! by minimizing RSS
Works well when (XTX)-1 is invertible. But often this is not true. Need to regularize or add a prior
![Page 39: Introduction to probability and statisticspages.discovery.wisc.edu/~sroy/teaching/network_biology/... · 2018-09-14 · Introduction to probability and statistics Alireza FotuhiSiahpirani&](https://reader033.vdocuments.net/reader033/viewer/2022042909/5f3aff0e3875d829ea06848d/html5/thumbnails/39.jpg)
References• Chapter 2, Probabilistic Graphical Models. Principles and Techniques.
Friedman & Koller.• Slides adapted from Prof. Mark Craven’s Introduction to Bioinformatics
lectures.• All of Statistics, Larry Wasserman.• Chapter 3, The Elements of Statistical Learning, Hastie, Tibshirani,
Friedman.