2011 - current research

Outline MD Classification MDSSL Application to SA Conclusions Current Work

Semi-supervised Learning of Multi-dimensionalClass Bayesian Network Classifiers

Jonathan Ortigosa-Hernandez

January 28th, 2011


Semi-supervised Learning of Multi-dimensional Class Bayesian Network Classifiers


Multi-dimensional Supervised Classification

Multi-dimensional Semi-supervised Learning

Application to Sentiment Analysis

Conclusions

Current Topics of Research




Supervised Classification

I It consists of building a classifier Ψ from a given labelledtraining dataset D, by using an induction algorithm A(A(D) = Ψ),

X1 X2 ... Xn C

x(1)1 x

(1)2 ... x

(1)n c (1)

x(2)1 x

(2)2 ... x

(2)n c (2)

... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n c (N)

I in order to predict the value of a class variable C for any newunlabelled instance x (Ψ(x) = c).




Uni-dimensional and Multi-dimensional Classification

I Uni-dimensional classification tries to predict a single classvariable based on a dataset composed of a set of labelledexamples.

I (Uni-dimensional Class) Bayesian Network Classifiers(Larranaga et al, 2005).

I Multi-dimensional classification is the generalisation of thesingle-class classification task to the simultaneous predictionof a set of class variables.

I Multi-dimensional Class Bayesian Network Classifiers (v.d.Gaag and d. Waal, 2006).

I Do not confuse with multi-class and multi-label classification.




Multi-dimensional Supervised Learning

A typical supervised training dataset

X1 X2 ... Xn C1 C2 ... Cm

x(1)1 x

(1)2 ... x

(1)n c

(1)1 c

(1)2 ... c

(1)m

x(2)1 x

(2)2 ... x

(2)n c

(2)1 c

(2)2 ... c

(2)m

... ... ... ... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n c

(N)1 c

(N)2 ... c

(N)m

Each instance of the dataset contains both the values of theattributes and m labels which characterise the attributes.




Bayesian Network Classifiers

X1 X2 X3 X4 X5 X6

C

Figure: A (uni-dimensional) naive Bayes structure.




Multi-dimensional Class Bayesian Network Classifiers(MDBNC)

X1 X2 X3 X4 X5 X6

C1 C2 C3

Figure: A multi-dimensional naive Bayes structure.




MDBNC Structure

X1 X2 X3 X4 X5

C1 C2 C3

X1 X2 X3 X4 X5

C1 C2 C3

X1 X2 X3 X4 X5

(a) Complete graph

(c) Class subgraph

(b) Feature selection subgraph

(d) Feature subgraph

C1 C2 C3

Figure: A MDNBC structure and its division.




Sub-families of MDBNC

(a) Multi-dimensional naive Bayes

(c) Multi-dimensional J/K dependence Bayesian (2/3)

(b) Multi-dimensional tree-augmented network

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$

!" !# !$ !% !&

'" '#

Figure: Different subfamilies of MDBNC.




Sub-families of MDBNC - MDnB

Multi-dimensional naive Bayes (MDnB)

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$

The class and featuresubgraphs are empty.

Each class variable isparent of all thefeatures.





Multi-dimensional naive Bayes (MDnB)

!" !# !$ !% !& !' !( !) !* !"+

," ,# ,$

It has a fixed structure.

Thus, it has nostructural learning (v.d.Gaag and d. Waal,2006).





Multi-dimensional tree-augmented network classifier (MDTAN)

!" !# !$ !% !&

'" '#

The class and featuresubgraphs are trees.





Multi-dimensional tree-augmented network classifier (MDTAN)

!" !# !$ !% !&

'" '#

A wrapper structurallearning algorithm isproposed in (v.d. Gaagand d. Waal, 2006).

[NEW] We have recentlyproposed a filterapproach to learnMDTAN structures.





Multi-dimensional J/K dependence Bayesian classifier (MD J/K )

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

The class subgraph is aJ-dependence graph.

The feature subgraph isa K -dependence graph.





Multi-dimensional J/K dependence Bayesian classifier (MD J/K )

!"

!#

!$

!%

!& !'

!(

!)

*# *$ *%

!+ !",

*"

There was not a specificstructural learningalgorithm.

So, we proposed alearning algorithm in(Ortigosa-Hernandez etal, 2010).




A supervised method to learn a MD J/K structure(Ortigosa-Hernandez et al, 2010)

Step 0 - Initialisation

Establish the maximumnumber of parents inboth class and featuresubgraphs, i.e. J = 2and K = 2.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Step 1 - Learn the structure between the class variables (Ac)

Calculate the mutualinformation MI (Ci ,Cj)for each pair of classvariables.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






Calculate the p-values(significance of eachmutual information)using independence test.

C1 C2 C3

C4 0.36 0.57 0.01C3 0.27 0.63C2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






Remove the p-valuesgreater than thethreshold α = 0.1.

C1 C2 C3

C4 0.36 0.57 0.01C3 0.27 0.63C2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






From the lowest value,add arcs to the graphfulfilling the conditionsof no cycles and no morethan J-parents per classvariable.

C1 C2 C3

C4 x x 0.01C3 x xC2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






From the lowest value,add arcs to the graphfulfilling the conditionsof no cycles and no morethan J-parents per classvariable.

C1 C2 C3

C4 x x xC3 x xC2 0.06

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Step 2 - Learn the structure between the class variables and thefeatures (ACF )

Calculate the mutualinformation MI (Ci ,Xj)for each pair Ci and Xj .

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






Calculate the p-value ofthe mutual informations.

C1 C2 C3 C4

X1 0.64 0.00 0.77 0.98X2 0.82 0.03 0.11 0.37X3 0.00 0.06 0.00 0.01X4 0.68 0.09 0.00 0.55X5 0.81 0.12 0.81 0.65X6 0.57 0.24 0.00 0.00X7 0.25 0.26 0.00 0.00X8 0.32 0.15 0.00 0.44

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






Remove the p-valuesgreater than α = 0.1.

C1 C2 C3 C4

X1 0.64 0.00 0.77 0.98X2 0.82 0.03 0.11 0.37X3 0.00 0.06 0.00 0.01X4 0.68 0.09 0.00 0.55X5 0.81 0.12 0.81 0.65X6 0.57 0.24 0.00 0.00X7 0.25 0.26 0.00 0.00X8 0.32 0.15 0.00 0.44

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4






Add all the arcs to thestructure.

C1 C2 C3 C4

X1 x 0.00 x xX2 x 0.03 x xX3 0.00 0.06 0.00 0.01X4 x 0.09 0.00 xX5 x x x xX6 x x 0.00 0.00X7 x x 0.00 0.00X8 x x 0.00 x

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Step 3 - Learn the structure between the features(AF )

I Calculate theconditional mutualinformationMI (Xi ,Xj ||Pac(Xj)).

I Calculate thep-values.

I Remove thep-values greaterthan the thresholdα = 0.1.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4





Step 3 - Learn the structure between the features (AF )

Add arcs between thefeatures fulfilling theconditions of no cyclesbetween the featuresand no more thanK -parents per feature.

X1 X2 X3 X4 X5 X6 X7 X8

C1 C2 C3 C4




A supervised method to learn a MDTAN structure(MDTANfi)

It is similar to the method to learn MD J/K structures, but treesare learnt in the AC and AF by means of a maximum spanning treealgorithm




Major Problem of Supervised Learning

I However, in many real world problems, obtaining data isrelatively easy, while labelling is difficult, expensive or laborintensive (usually done by an external mechanism, e.g. humanbeings).

I This problem is accentuated when using multiple targetvariables.

I DESIRE: Learning algorithms able to incorporate a largenumber of unlabelled data with a small number of labeleddata when learning competitive classifiers.




Multi-dimensional Semi-supervised Learning

A typical semi-supervised training dataset

X1 X2 ... Xn C1 C2 ... Cm

x(1)1 x

(1)2 ... x

(1)n c

(1)1 c

(1)2 ... c

(1)m

x(2)1 x

(2)2 ... x

(2)n c

(2)1 c

(2)2 ... c

(2)m

... ... ... ... ... ... ... ...

x(L)1 x

(L)2 ... x

(L)n c

(L)1 c

(L)2 ... c

(L)m

x(L+1)1 x

(L+1)2 ... x

(L+1)n ? ? ... ?

x(L+2)1 x

(L+2)2 ... x

(L+2)n ? ? ... ?

... ... ... ... ... ... ... ...

x(N)1 x

(N)2 ... x

(N)n ? ? ... ?

Semi-supervised Learning fulfils this desire.




The Expectation-Maximisation Algorithm

The EM algorithm (Dempster et al, 1977)

Learn an initial model.Repeat until convergence:(a) Expectation step: Using the current model, estimate themissing values of the data.(b) Maximisation step: Using the whole data and the previousestimations, learn a new current model.

Any MDBNC learning algorithm can be used as model in thisalgorithm.




Artificial Experimentation

Study the behaviour of the proposed algorithms along several axesof variability:

1. Complexity of the problem (generative structure)√

2. Number of variables (features and class variables)

3. Balance of the labels in the generative structure (values of thehyperparamenters for the Dirichlet)

4. Size of the labelled sample

5. Ratio of labelled-unlabelled data




Artificial Experimentation

A preliminary experimentation onthe complexity of the problem canbe found in:

http://www.sc.ehu.es/ccwbayes/members/

jonathan/home/News_and_Notables/Entries/

2010/11/30_IMACS_2011.html

2DB

3DB

MDnB

123456789

nB

MDTAN

TAN

MD 1/1

MD 2/2

MD 2/3

Figure: Accuracy ranking fordifferent algorithms on 20artificial datasets, α = 0.05.



http://www.sc.ehu.es/ccwbayes/members/jonathan/home/News_and_Notables/Entries/2010/11/30_IMACS_2011.html




Application to Sentiment Analysis

I Sentiment Analysis (AKA Opinion Mining) is thecomputational study of opinions, sentiments and emotionsexpressed in text (Liu, 2010).

I When treating Sentiment Analysis as a classification problem,several different (but related) problems appear. For example:

1. Subjectivity Classification. Its aim is to classify a text assubjective or objective.

2. Sentiment Classification. It classifies an opinionated text asexpressing a positive, neutral, or negative opinion.




Motivation for Using Semi-supervised Learning ofMulti-dimensional Classifiers

1. Up to now, these subproblems have been studied in isolationdespite of being closely related. So, probably it would behelpful to use multi-dimensional classifiers.

2. Obtaining enough labeled examples for a classifier may becostly and time consuming. This motivates us to deal withunlabelled examples in a semi-supervised framework.




Hypothesis Formulated

First Hypothesis

The explicit use of the relationships between different classvariables can be beneficial to improve their recognition rates.

Second Hypothesis

Multi-dimensional techniques can work with unlabelled data inorder to improve the classification rates in Sentiment Analysis.




Properties of the Dataset

I Collected by Socialware Company S.A., from the ASOMOservice of mobilised opinion analysis.

I It consists of 2, 542 Spanish reviews extracted from a blog:I 150 documents have been labeled in isolation by an expert.I 2, 392 posts are left unlabelled.

Figure: The ASOMO corpus.Jonathan Ortigosa-Hernandez



Properties of Each Document I

Each document is represented as 14 features:I Obtained by using an open source morphological analyser

(Carreras et al, 2006).I Each feature provide different information related to

part-of-speech (PoS).

Feature Description Example1 First Persons Number of verbs in the fist person. Contrate ... .2 Second Persons Number of verbs in the second per-

son.Tienes ...

3 Third Persons Number of verbs in the third per-son.

Sabe ... .

4 Relational Forms Number of phatic expressions, i.e.expressions whose only function isto perform a social task.

(1) Hola.(2) Gracias de antemano.

5 Agreement Expres-sions

Number of expressions that showagreement or disagreement.

(1) Estoy de acuerdo contigo.(2) No tienes razon.

6 Request Number of sentences that expressa certain degree of request.

(1) Me gustarıa saber ...(2) Alguien podrıa ...

Table: Subset of features related to the implication of the author withother customers.




Properties of Each Document II

Each document has 3 class variables:

I Will to Influence: {declarative sentence, soft WI, medium WI,strong WI}

I Sentiment: {very negative, negative, neutral, positive, verypositive}

I Subjectivity: {Yes (subjective), No (objective)}




Experiment 1 - Set Up I

First Hypothesis


I The ASOMO corpus has been used to learn:I 3 (uni-dimensional) naive Bayes classifiers, one per each class

variable.I A (uni-dimensional) naive Bayes classifier with a compound

class variable.I A multi-dimensional naive Bayes classifier.




Experiment 1 - Set Up II

First Hypothesis


I Features from ASOMO dataset are discretised into 3 valuesusing equal frequency.

I In addition to the ASOMO feature set, three state-of-the-artfeature sets are used:

I UnigramsI Unigrams + BigramsI PoS tagging

I Results averaged over 20 × 5 fold cross validation.




Experiment 1- JOINT Accuracies

Figure: JOINT accuracies on ASOMO corpus using three differentfeature sets in both uni and multi-dimensional scenarios (20× 5cv)




Experiment 2 - Set Up

Second Hypothesis

Multi-dimensional techniques can work with unlabelled data inorder to improve the classification rates in Sentiment Analysis.

I The ASOMO dataset has been used to learn:I 3 (uni-dimensional) Bayesian network classifiers: nB, TAN and

2DB.I 5 MDBNC: MDnB, MDTAN, MD 2/2, MD 2/3 and MD 2/4.

I In both Supervised and Semi-supervised (EM algorithm)learning frameworks.

I Features from ASOMO dataset are discretised into 3 valuesusing equal frequency.

I Results averaged over 20 × 5 fold cross validation.




Experiment 2 - JOINT Accuracy

Figure: JOINT accuracies on ASOMO dataset in the supervised andsemi-supervised learning frameworks.




Experiment 2 - Will to Influence

Figure: Accuracies for the Will to Influence class variable on ASOMOdataset in the supervised and semi-supervised learning frameworks.




Experiment 2 - Sentiment Polarity

Figure: Accuracies for the Sentiment Polarity class variable on ASOMOdataset in the supervised and semi-supervised learning frameworks.




Experiment 2 - Subjectivity

Figure: Accuracies for the Subjectivity class variable on ASOMO datasetin the supervised and semi-supervised learning frameworks.




Conclusions I - Methodology

I Multi-dimensional classification and semi-supervised learningare two different branches of machine learning.

I With this research, we have established a bridge betweenthem showing that:

I Uni-dimensional approaches cannot capture the real nature ofmulti-dimensional problems.

I More accurate classifiers can be found using themulti-dimensional learning approaches.

I The use of large amounts of unlabelled data can be beneficialto improve recognition rates.




Conclusions II - Application

I With respect to the Sentiment Analysis application, we haveproposed a novel perspective to solve the problem.Experimental results demonstrate that the use ofmulti-dimensional classification, as well as the use ofunlabelled data, can lead us to more accurate classifiers.




Feature Selection

Title: Semi-supervised Feature Selection in Multi-dimensionalProblems

Description: Develop a methodology able to identify irrelevant and redundantfeatures in multi-dimensional problems for dimension reduction ina semi-supervised framework.

Motivation

I Feature selection try to avoid problems related to overfitting,computation burden, etc.

I Up to now, there is no feature selection technique which is able to dealwith multiple class variables.

I Few work has been done in semi-supervised feature selection (Cai et al,2011).




Semi-supervised Feature Selection

X1 XnC1 Cm

Supervised feature selection Unsupervised feature selection

I Feature relevance is usuallyevaluated by their correlation withthe class label.

I The labelled sample is generallytoo small and insufficient for thispurpose.

I Evaluated by their capability ofkeeping certain properties of thedata, e.g. variance or separability.

I Ignoring label information cancause downgrades in theperformance.




Semi-supervised Feature Selection via Spectral Analysis I

Based on the cluster assumption - Unsupervised feature selection(Zhao et al, 2007)

f f'

I Unsupervised perspective:Both solutions are OK.

I Supervised point of view: fis better than f ′.




Semi-supervised Feature Selection via Spectral Analysis II

Proposal: Use the clustering assumption to identify the relevantfeatures, but giving more relevance to the features which clearlyseparate the labels.

I Drawback: This algorithm does not take into account theredundant detection.




Semi-supervised Feature Selection via BASSUM

Calculate the Markov blanket of a class variable by using G 2

conditional independence tests with both labelled and unlabelleddata. It detects redundant features (Cai et al, 2011).

Markov blanket of a classvariable A is the set of all parents,children and spouses of A in theBayesian network.




BASSUM Example

F3

F1

F2

I F3 is the class variable

I F2 is in the Markov Blanket S(F3), i.e.S(F3) = {F2}

I We are checking if F1 is also in S(F3)

I So, want to determineF1 ⊥ F3|S(F3) = F1 ⊥ F3|F2

F1 F2 F3

DEFINITIONS

cijk ≡ number ofinstances that satisfyF1 = f i

1 ,F2 = f j2 ,F3 =

f k3

Marginal sumsc+jk =

Pi cijk ,

ci+k =P

j cijk ,cij+ =

Pk cijk .

G 2 conditional independence test

G 2 = 2Xijk

cijk lncijkc++k

ci+kc+jk∼ χ2

df = (|F1| − 1)(|F2| − 1)|F3|

Labelled data Unlabelled data




Important Ideas

I Redundancy and Relevancy

I Spectral Analysis → A definition of the Markov Blankets inmulti-dimensional Bayesian networks is needed.

C*

X2X4

X1X6

X5 X3

X2X4

X1X6

X5 X3

C1

C2 C3

I BASSUM approach → Modify a classical feature selectiontechnique (Saeys et al, 2007) to be able to deal withmulti-dimensional problems in a semi-supervised framework.




Affect Analysis

Title: Application to Affect AnalysisDescription: Use the methodology proposed in this presentation to dealwith the problem of Affect Analysis.Collaboration: Socialware S.A.

Motivation (Abbasi et al., 2008)

I Affect Analysis is concerned with the analysis of text containing emotionsand it tries to extract a large number of potential emotions, e.g.happiness, sadness, anger, hate, violence, excitement, etc.




Plutchik’s Affect Model

We want to take a step forward in this problem taking advantageof the potential possibilities of the MDBNC to model complexrelationships between the class variables.

4 class variables with threepossible values: {−1, 0, 1}.

I LOVE-REMORSE(Aceptacion-Disgusto)

I CONTEMPT-SUBMISSION(Anticipacion-Sorpresa)

I AGGRESSIVENESS-AWE(Ira-Miedo)

I OPTIMISM-DISAPPROVAL(Alegrıa-Tristeza)




Questions

[email protected]



2011 - current research

Documents