![Page 1: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/1.jpg)
![Page 2: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/2.jpg)
Courant Institute of Mathematical Sciences&
Department of EconomicsNew York University
Sumit Chopra Trivikraman ThampyJohn Leahy Andrew Caplin Yann LeCun
Discovering Hidden Structure of House Prices
with Relational Latent Manifold Model
![Page 3: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/3.jpg)
Relational Learning• Traditional Learning
• Samples are i.i.d. from an unknown distribution D
• Relational Learning
• Samples are no longer i.i.d
• Related to each other in complex ways: values of unknown variables depends on each other
• Dependencies could be direct or indirect (hidden)
• Examples
• Web page classification
• Scientific document classification
• Need a form of collective prediction
![Page 4: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/4.jpg)
Predicting House Price• “Location Location Location”
![Page 5: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/5.jpg)
Predicting House Price• “Location Location Location”
![Page 6: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/6.jpg)
Predicting House Price• “Location Location Location”
![Page 7: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/7.jpg)
Predicting House Price
• Price is a function of quality/desirability of neighborhood
• Which in turn is a function of quality/desirability of other houses
• This is the relational aspect of the price
• “Location Location Location”
![Page 8: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/8.jpg)
Predicting House Price• However there is more to it• The Non-Relational aspect of price
![Page 9: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/9.jpg)
1 bedroom, 1 bathroom
Predicting House Price• However there is more to it • The Non-Relational aspect of price
![Page 10: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/10.jpg)
1 bedroom, 1 bathroom 4 bedroom, 3 bathroom
Predicting House Price• However there is more to it • The Non-Relational aspect of price
![Page 11: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/11.jpg)
1 bedroom, 1 bathroom 4 bedroom, 3 bathroom
• One can view this as the “intrinsic price”
Predicting House Price• However there is more to it• The Non-Relational aspect of price
![Page 12: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/12.jpg)
The Idea• Price of a house is modeled as
price = (intrinsic price) ! (desirability of its location)
*
GW H
Xh Xnb
Parametric
Model
Non-Parametric
Relational Model
Pr = Ip ! D
Ip D
House
Characteristics
(# bedrooms, # bathrooms)
Neighborhood
Characteristics
(census tract, ...)
desirabilities
of neighboring
houses
![Page 13: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/13.jpg)
The Idea• In fact we compute the log of the price
+
GW H
Xh Xnb
Parametric
Model
Non-Parametric
Relational Model
Ip D
House
Characteristics
(# bedrooms, # bathrooms)
Neighborhood
Characteristics
(census tract, ...)
desirabilities
of neighboring
houses
Pr = Ip + D
![Page 14: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/14.jpg)
Our Contribution
• A novel technique for relational regression problems
• Allows relationships via the hidden variables
• Allows non-linear likelihood functions
• Propose efficient training and inference algorithm
• Apply it to the house price prediction problem
![Page 15: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/15.jpg)
Estimating Desirabilities
![Page 16: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/16.jpg)
Estimating Desirabilities
Z1
Z2
Z3
Zi
ZN
![Page 17: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/17.jpg)
Estimating Desirabilities
Z1
Z2
Z3
Zi
ZN
![Page 18: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/18.jpg)
Estimating Desirabilities
Z1
Z2
Z3
Zi
ZN
![Page 19: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/19.jpg)
• Any form of smooth interpolation is good
• Kernel Interpolation
• Local Weighted Linear Regression
(!!ZNi,"!ZNi
) = argmin!,"
!
j"N i
(Zj ! (! + "Xj))2Ker(Xinb, X
jnb)
H(ZN i , Xinb) = !!
ZNi+ "!
ZNiXi
=!
k
U ikZk
H(ZN i , Xinb) =
!
j!N i
Ker(Xinb, X
jnb)Z
j
Estimating Desirabilities
![Page 20: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/20.jpg)
The Inference Algorithm
+
Xh Xnb ZN
G(W,Xh) H(ZN , Xnb)
Ip D
Pr = Ip + D
![Page 21: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/21.jpg)
+
Xh Xnb ZN
G(W,Xh) H(ZN , Xnb)
Ip D
Y
Pr = Ip + D
E(W,ZN , Y,X)
(Y ! Pr)2
The Learning Algorithm• Done by maximizing the
likelihood of the data
• Achieved by minimizing the negative log-likelihood function wrt
• Boils down to minimizing energy loss
W, Z
L(W,Z) =12
N!
i=1
(Y i ! (G(W,Xih) + H(ZN i , Xi
nb))2 + R(Z)
![Page 22: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/22.jpg)
The Learning Algorithm
• Two phase: generalized EM type
• Phase 1: • Keep fixed and optimize with respect to
• The above loss reduces to
• Sparse linear system: was solved using conjugate gradient
W Z
L(W,Z) =12
N!
i=1
(Y i ! (G(W,Xih) + H(ZN i , Xi
nb))2 +
r
2||Z||2)
L(Z) =r
2||Z||2 +
12
N!
i=1
(Y i ! (G(W,Xih) + U i · Z))2
![Page 23: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/23.jpg)
The Learning Algorithm
• Phase II: • Keep fixed and optimize with respect to
• The parameters are shared among samples
• Neural network was optimized using simple gradient decent
L(W,Z) =12
N!
i=1
(Y i ! (G(W,Xih) + H(ZN i , Xi
nb))2 + R(Z)
WZ
![Page 24: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/24.jpg)
X1Y 1
Z1
X2Y 2
Z2 Z3
X3Y 3d1 d2 d3
E1xyz E2
xyz E3xyz
E1yy E2
yy E3yy
E3zzE2
zzE1zz
The General Framework
E(W,Z,Y,X) =N!
i=1
Eixyz(Wxyz, X
i, Y i, Di) + Eiyy(Wyy, Y i,Y) + Ei
zz(Wzz, Di,Z)
![Page 25: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/25.jpg)
Experiments
• Dataset• Houses from Los Angeles Country transacted only in 2004• They span 1754 census tracts and 28 school district• A total of around 70,000 samples • We used a total of 19 features in
• living area, year built, # bedrooms, # bathrooms, pool, prior sale price, parking spaces, parking types, lot acerage, land value, improvement value, % improvement, new construction, foundation, roof type, heat type, site influence, and gps coordinates
• We used 6 features as part of • median house hold income, average time of commute to work,
proportion of units owner occupied, and academic performance index
Xh
Xnb
![Page 26: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/26.jpg)
Experiments
• Dataset• All variables containing any form of price/area/income
information were mapped into log space• Non-numeric discrete variables were coded using a
1-of-K coding scheme• Only Single Family Residences were estimated• A total of 42025 complete labeled samples• Training set
• 37822 (90%)
• Test set• 4203 (10%)
![Page 27: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/27.jpg)
Baseline Methods
• Nearest Neighbor
• Linear Regression
• Locally Weighted Linear Regression
• Fully Connected Neural Network
![Page 28: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/28.jpg)
Results
Model Class Model < 5% < 10% < 15%
Non-Parametric K - Nearest Neighbor 25.41 47.44 64.72
Parametric Linear Regression 26.58 48.11 65.12
Non-Parametric Locally Weighted Regression 32.98 58.46 75.21
Parametric Fully Connected Neural Network
33.79 60.55 76.47
Hybrid Relational Factor Graph 39.47 65.76 81.04
• Absolute Relative Forecasting error is computed
errori =|Pri !Ai|
Ai
![Page 29: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/29.jpg)
Results
Model Class Model < 5% < 10% < 15%
Non-Parametric K - Nearest Neighbor 25.41 47.44 64.72
Parametric Linear Regression 26.58 48.11 65.12
Non-Parametric Locally Weighted Regression 32.98 58.46 75.21
Parametric Fully Connected Neural Network
33.79 60.55 76.47
Hybrid Relational Factor Graph 39.47 65.76 81.04
![Page 30: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/30.jpg)
Results
Model Class Model < 5% < 10% < 15%
Non-Parametric K - Nearest Neighbor 25.41 47.44 64.72
Parametric Linear Regression 26.58 48.11 65.12
Non-Parametric Locally Weighted Regression 32.98 58.46 75.21
Parametric Fully Connected Neural Network
33.79 60.55 76.47
Hybrid Relational Factor Graph 39.47 65.76 81.04
![Page 31: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/31.jpg)
Results
Model Class Model < 5% < 10% < 15%
Non-Parametric K - Nearest Neighbor 25.41 47.44 64.72
Parametric Linear Regression 26.58 48.11 65.12
Non-Parametric Locally Weighted Regression 32.98 58.46 75.21
Parametric Fully Connected Neural Network
33.79 60.55 76.47
Hybrid Relational Factor Graph 39.47 65.76 81.04
![Page 32: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/32.jpg)
Results
Model Class Model < 5% < 10% < 15%
Non-Parametric K - Nearest Neighbor 25.41 47.44 64.72
Parametric Linear Regression 26.58 48.11 65.12
Non-Parametric Locally Weighted Regression 32.98 58.46 75.21
Parametric Fully Connected Neural Network
33.79 60.55 76.47
Hybrid Relational Factor Graph 39.47 65.76 81.04
![Page 33: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/33.jpg)
Learnt Desirability Manifold
![Page 34: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/34.jpg)
Thank You Very Much!!!
![Page 35: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/35.jpg)
• Direct dependencies between is not captured
• First factor
• Non-relational: captures dependencies between individual variables and the price
• Second factor
• Relational: captures the influence on the price of a house from other (related houses) via the hidden variables
• non-parametric estimate of desirability of the location of the house, obtained from related houses
Eixyz(Y
i, Xi, Di) = (Y i ! (G(Wxyz, Xih) + Di))2
Eizz(D
i,H(Xinb, ZN i)) = (Di !H(Xi
nb, ZN i))2
Y
H(Xinb, ZN i)
Real Estate Price Prediction
![Page 36: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/36.jpg)
• could take any smooth form
• Kernel Interpolation
• Local Weighted Linear Regression
H(Xinb, ZN i)
H(Xinb, ZN i) =
!
j!N i
Ker(Xnb, Xjnb)Z
j
(!!,"!) = arg min!,"
!
j"N i
(Zj ! (! + "Xj))2Ker(Xi, Xj)
H(Xinb, ZN i) = !! + "!X
=!
j"N i
ajZj
Real Estate Price Prediction
![Page 37: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/37.jpg)
• The total energy associated with a single sample is
Eixyz(Y
i, Xi, Di) + Eizz(D
i,H(Xinb, ZN i) = (Y i ! (G(Wxyz, X
ih) + Di))2 + (Di !H(Xi
nb, ZN i))2
dm
p
F(Y, p)
+
YXh Xnb ZN
H(Xnb, ZN )
E(W,ZN , Y,X)
G(Wxyz, Xh)Ei(Y i, Xi) = (Y i ! (G(Wxyz, X
ih) + H(Xi
nb, ZN i))2
Real Estate Price Prediction
![Page 38: Discovering Hidden with Relational Latentamnih/cifar/talks/chopra_talk.pdf · Predicting House Price • Price is a function of quality/desirability of neighborhood • Which in turn](https://reader033.vdocuments.net/reader033/viewer/2022060422/5f19233fe6ed625d5d60c75b/html5/thumbnails/38.jpg)
X1hX1
nb
Y 1
Z1
X2hX2
nb
Y 2
Z2
X3h X3
nb
Y 3
Z3
Z4
Y 4
X4nbX4
h
XihXi
nb
Y i
Zi
Ei
E1
E2
E3
E4
• The factor graph is
Real Estate Price Prediction