types of variables, treatment of missing data

Types of variables, treatment of missing data

20.10. 2009

Types of variables (number of values)

continuous variables (any value within a range)

discrete variables (finite number of values)– some discrete variables can be treated as

continuous (counts)

Types of variables (measurment)

Type of the data Measurment property

Nominal We can only differentiate between the values

Ordinal We can order them

Cardinal data We use measurment unit

data$country<-as. factor(data$country)

data$qf1<-as.ordered(data$qf1)

data$vd8<-as.numeric (data$vd8)

Data type definition can be• usefull = some of the procedures work differently for different types of the data• an obstacle (sometimes you want to do something that is, strictly speaking, not allowed

Missing data

What is the nature of missing data? How can we deal with misssing data

depends of their mechanism, i.e. mechanism that produced them...

No easy answers... All decisions about how to deal with missing

data have impacts the results...

Types of missing data

Missing completely at random (MCAR)– probability that an observation Xi is missing is unrelated to the value of X or any

other veriable– weather conditions, failure of measuring equipment etc.

Missing at random (MAR)– probability of missing value does not depend on the value of X after controling for

another variable– e.g. depressed people less likely to report income, when controlling for

depression, missing is random

Functional missing– prob. of missing = function of a variable

missingness is ignorable = do not have to model missingness property

missingness has to be dealt with completely

How to deal with missings

1. get rid of them - deletion (listwise, casewise)

2. replace them by mean etc.

3. who are they? - regression analysis, ANOVA

4. what they did not tell us?– imputation

Missing data in R

reserved value “NA” important issue (build-in function have ussually no default setup for

dealing with NA’s = you have to specify– is it an error?– should it be ignored (deleted)? – should it be replaced by some value (mean)?

na.rm=T (remove all cases with missing values)

> a<-c(1,2,3,4,5,NA,NA) > a[1] 1 2 3 4 5 NA NA> mean(a)[1] NA #Why? Because you have not specified how to deal with NA’s.

#So try the following: > mean(a, na.rm=TRUE) [1] 3

How many NA’s?

> table(a)a1 2 3 4 5 1 1 1 1 1

> table(a, exclude=NULL)a 1 2 3 4 5 <NA> 1 1 1 1 1 2

# to get relative counts> prop.table(table(a, exclude=NULL))a 1 2 3 4 5 <NA> 0.1428571 0.1428571 0.1428571 0.1428571 0.1428571 0.2857143

How to find NA’s?

#find complete cases (no NA’s):> complete.cases(a) [1] TRUE TRUE TRUE TRUE TRUE FALSE FALSE

> b<-c(NA,NA, 1,2,3,4,5)> d<-cbind(a,b) #nyní máme matici 7x2> d

a b[1,] 1 NA[2,] 2 NA[3,] 3 1[4,] 4 2[5,] 5 3[6,] NA 4[7,] NA 5

> complete.cases(d) [1] FALSE FALSE TRUE TRUE TRUE FALSE FALSE

This result can be used as an argument in another funcion that would filter out rows with missing

cases.

How to remove NA’s?

> da b

[1,] 1 NA[2,] 2 NA[3,] 3 1[4,] 4 2[5,] 5 3[6,] NA 4[7,] NA 5

> e<-d[complete.cases(d),]> e a b[1,] 3 1[2,] 4 2[3,] 5 3

> e<-d[complete.cases(d)]> e[1] 3 4 5 1 2 3

Vrátí kompletní pozorování (v řádku) jako matici

Vrátí kompletní pozorování (v řádku) jako numerický vektor

!!!

By the way...

this a good trick how to filter the dataset

For example, you want to get Czech data only

czech.data<-data[data$country==17,]

Get your Eurobarometer data...

source<-file.choose()

data<-read.csv(source, header = T, sep = "\t")

How to recode certain values as NA‘s?

- missings in the data- system missing (no information at all)- user-defined missing (can be thought of as missing) – “don’t know”, “undecided” – but it always depends....

qf12“Please tell me whether you totally agree, tend to agree, tend to disagree or totally disagree with the following statement: You

are ready to buy environmentally friendly products even if they cost a little bit more.”

- totally agree- tend to agree- tend to disagree- totally disagree- don’t know

> table(data$qf12, exclude=NULL)

1 2 3 4 5 7442 12947 3636 1292 1413

data$qf12<-ifelse(data$qf12==5, NA, data$qf12) 1 2 3 4 <NA> 7442 12947 3636 1292 1413

for testing of equality, the equal sign has to be doubled (otherwise it is attribution of a value)

Look at you variables

what type of variables are you interested in?– measurment?– continuous or descrete?– how can you describe them?

are there any missings in those variables?– what type (system or user-defined)?– how many?– is that a problem?

Questions? Comments?

Thank you for your attention!