16. censoring, tobit and two part models

16. Censoring, Tobit and Two Part Models

Censoring and Corner Solution Models• Censoring model: y = T(y*) = 0 if y* < 0 y = T(y*) = y* if y* > 0. • Corner solution: y = 0 if some exogenous condition is met; y = g(x)+e if the condition is not met. We then model P(y=0) and E[y|x,y>0]. • Hurdle Model: y = 0 with P(y=0|z) Model E[y|x,y > 0]

The Tobit Modely* = x’+ε, y = Max(0,y*), ε ~ N[0,2]

Variation: Nonzero lower limit Upper limit Both tails censored

Easy to accommodate. (Already done in major software.)

Log likelihood and Estimation. See Appendix.

(Tobin: “Estimation of Relationships for Limited Dependent Variables,” Econometrica, 1958. Tobin’s probit?)

Conditional Mean Functions

2y* , ~N[0, ], y Max(0,y*)E[y* | ]E[y| ]=Prob[y=0| ]×0+Prob[y>0| ]E[y|y>0, ] =Prob[y*>0| ] E[y*|y*>0, ]

( ) = ( ) = ( )

x βx x

x x x xx x

x β x β/x β+ x β/x β/ x

( )( ) "Inverse Mills ratio"( )

( )E[y| ,y>0]= ( )

β + x β/x β/x β/

x β/x x β+ x β/

Conditional Means

Predictions and Residuals• What variable do we want to predict?

• y*? Probably not – not relevant• y? Randomly drawn observation from the population• y | y>0? Maybe. Depends on the desired function

• What is the residual?• y – prediction? Probably not. What do you do with

the zeros?• Anything - x? Probably not. x is not the mean.

• What are the partial effects? Which conditional mean?

OLS is Inconsistent - Attenuation E[y| ] = ( / ) ( / )

Nonlinear function of . What is estimated by OLS regression of y on ?Slopes of the linear projection are approximatelyequal to the derivatives of the conditional me

x x β x β + x βx

x

anevaluated at the means of the data.E[y| ] ( / )

Note the ; 0 < ( / ) < 1attenuation

x x β βxx β

Partial Effects in Censored Regressions

For the latent regression:E[y*| ]E[y*| ]= ;

E[y| ]E[y| ]= ;

E[y| 0]= /

(a)

xx x'β βx

x'β x'β x x'βx x'β+ βx

x'β x'βx,y> x'β+

x'βx'β+ x'β+

E[y| 0] [1- (a)(a (a))]x,y> βx

Application: Fair’s DataFair’s (1977) Extramarital Affairs Data, 601 observations. Psychology Today. Source: Fair (1977) and http://fairmodel.econ.yale.edu/rayfair/pdf/1978ADAT.ZIP. Several variables not used are denoted X1, ..., X5.y = Number of affairs in the past year, (0,1,2,3,4-10=7, more=12, mean = 1.46. (Frequencies 451, 34, 17, 19, 42, 38)z1 = Sex, 0=female; mean=.476z2 = Age, mean=32.5z3 = Number of years married, mean=8.18z4 = Children, 0=no; mean=.715z5 = Religiousness, 1=anti, …,5=very. Mean=3.12z6 = Education, years, 9, 12, 16, 17, 18, 20; mean=16.2z7 = Occupation, Hollingshead scale, 1,…,7; mean=4.19z8 = Self rating of marriage. 1=very unhappy; 5=very happy

Fair, R., “A Theory of Extramarital Affairs,” Journal of Political Economy, 1978.Fair, R., “A Note on Estimation of the Tobit Model,” Econometrica, 1977.

Fair’s Study• Corner solution model• Discovered the EM method in the Econometrica

paper• Used the tobit instead of the Poisson (or some

other) count model• Did not account for the censoring at the high

end of the data

Estimated Tobit Model

Discarding the Limit Data

2

22

2

22

y*= +y=y* if y* > 0, y is unobserved if y* 0.f(y| )=f(y*| ,y*>0)= Truncated (at zero) normal

1 (y ) 1f(y*| ,y*>0)= exp Prob[y*>0|x]221 (y ) 1 = exp (22

x β

x xx βx

x βx β / )

ML is not OLS

Regression with the Truncated Distribution

i i

i

1n

ii 1

( / )E[y| ,y>0]= + ( / ) = +OLS will be inconsistent:

1Plim = plim plimn nA left out variable problem.Approximately: plim plim(1 - a

ii i

i

i

i

x βx xβ xβx β

X'Xb β+ x

b β

2

ni 1

)1a= , (a)n

General result: Attenuationix β/

Estimated Tobit Model

Two Part Specifications

ni=1

ni=1

Truncated Regression Probit

Tobit Log LikelihoodlogL= dlogf(y| y 0) (1 d)logP(y 0) = dlogf(y| y 0) dlogP(y 0) d logP(y>0) + (1 d)logP(y 0) logL + logL = logL for the model for y when y > 0 + logL for the model for whether y is = 0 or > 0.A two part model: Treat the model for quantity differentlyfrom the model for whether quantity is positive or not.==> A probit model and a separate truncated regression.

Doctor Visits (Censored at 10)

Two Part Hurdle Model

Critical chi squared [7] = 14.1. The tobit model is rejected.

Panel Data Application• Pooling: Standard results, incuding

“cluster” estimator(s) for asymptotic covariance matrices

• Random effects• Butler and Moffitt – same as for probit• Mundlak/Wooldridge extension – group means• Extension to random parameters and latent

class models• Fixed effects: Some surprises (Greene,

Econometric Reviews, 2005)

Neglected Heterogeneity

it it i it i it

itit it 2 2

c2 2

c

2 2 itc 2 2

c

y * (c ) assuming c

Prob[y * 0| ]

MLE estimates / ( ) Attenuated for

Partial effects are / ( )

Consistently estimated by ML eve

x β xx βx

β βx ββ

n though is not(The standard result.)

β .

Fixed Effects MLE for Tobit

No bias in slopes. Large bias in estimator of

APPENDIX: TOBIT MATH

Estimating the Tobit Model

n ii ii=1

i i i

Log likelihood for the tobit model for estimation of and :y1logL= (1-d) log d log

d 1 if y 0, 0 if y = 0. Derivatives are very complicated,Hessian is

i i

βx β x β

ni i ii=1

2i i ii=1

nightmarish. Consider the Olsen transformation*:=1/ , =- / . (One to one; =1/ ,

logL= (1-d) log d log y

(1-d)log d(log (1/ 2)log2 (1/ 2) y )i i

i i

β β=-x x

x x

n

ni i ii 1

ni i ii 1

logL (1-d) de

logL 1d ey

*Note on the Uniqueness of the MLE in the Tobit Model," Econometrica, 1978.

ii

i

x xx

Hessian for Tobit Model

n 2i i ii=1

n ni i i i i ii 1 i 1

2 nii 1

logL= log (1-d)log d(log (1/ 2)log2 (1/ 2) y )

logL logL 1(1-d) de d ey

logL (1-d) (

i i

ii

i

i ii

i i

x x

x xx

x xx x x

2

i

2 nii 1

2 ni i2i 1

d

logL d y

logL 1d d y y

i i

i i

i i

x x

x

Simplified Hessian

22 ni ii 1

2 nii 1

2 ni 2i 1

logL (1-d) ( d

logL d y

logL 1d

i ii i i

i i

i i

x xx x xx x

x

i

2 n i i i i2 2i 1

i i i

i i i i i i i i

d y y

((1 d) d) d ylogLdy d(1/ y )

a (a) / (a), (a )

i i

i i i i

i i

i

x x xx

x

Recovering Structural Parameters

2

2

2

1( , )( , ) 1

ˆ ˆ( , )ˆUse the delta method to estimate Asy.Var ˆ( , )ˆˆ

( , ) 1 1( , ) 1 ( , )1 1

ˆ ˆ( ,ˆEst.Asy.Var

β

β

βI I G

0'0'

β

ˆ) ˆ ˆ( , ) Est.Asy.Var ( , )ˆ ˆˆˆ( , )ˆˆ

G G '