Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1754–1763,Sofia, Bulgaria, August 4-9 2013. c©2013 Association for Computational Linguistics

Syntactic Patterns versus Word Alignment: Extracting Opinion Targetsfrom Online Reviews

Kang Liu, Liheng Xu and Jun ZhaoNational Laboratory of Pattern Recognition

Institute of Automation, Chinese Academy of Sciences{kliu, lhxu, jzhao}@nlpr.ia.ac.cn

AbstractMining opinion targets is a fundamen-tal and important task for opinion min-ing from online reviews. To this end,there are usually two kinds of methods:syntax based and alignment based meth-ods. Syntax based methods usually ex-ploited syntactic patterns to extract opin-ion targets, which were however prone tosuffer from parsing errors when dealingwith online informal texts. In contrast,alignment based methods used word align-ment model to fulfill this task, which couldavoid parsing errors without using pars-ing. However, there is no research fo-cusing on which kind of method is morebetter when given a certain amount of re-views. To fill this gap, this paper empiri-cally studies how the performance of thesetwo kinds of methods vary when chang-ing the size, domain and language of thecorpus. We further combine syntactic pat-terns with alignment model by using a par-tially supervised framework and investi-gate whether this combination is useful ornot. In our experiments, we verify thatour combination is effective on the corpuswith small and medium size.

1 Introduction

With the rapid development of Web 2.0, hugeamount of user reviews are springing up on theWeb. Mining opinions from these reviews be-come more and more urgent since that customersexpect to obtain fine-grained information of prod-ucts and manufacturers need to obtain immediatefeedbacks from customers. In opinion mining, ex-tracting opinion targets is a basic subtask. It isto extract a list of the objects which users expresstheir opinions on and can provide the prior infor-mation of targets for opinion mining. So this task

has attracted many attentions. To extract opin-ion targets, pervious approaches usually relied onopinion words which are the words used to ex-press the opinions (Hu and Liu, 2004a; Popescuand Etzioni, 2005; Liu et al., 2005; Wang andWang, 2008; Qiu et al., 2011; Liu et al., 2012). In-tuitively, opinion words often appear around andmodify opinion targets, and there are opinion re-lations and associations between them. If we haveknown some words to be opinion words, the wordswhich those opinion words modify will have highprobability to be opinion targets.

Therefore, identifying the aforementioned opin-ion relations between words is important for ex-tracting opinion targets from reviews. To fulfillthis aim, previous methods exploited the wordsco-occurrence information to indicate them (Huand Liu, 2004a; Hu and Liu, 2004b). Obviously,these methods cannot obtain precise extraction be-cause of the diverse expressions by reviewers, likelong-span modified relations between words, etc.To handle this problem, several methods exploitedsyntactic information, where several heuristic pat-terns based on syntactic parsing were designed(Popescu and Etzioni, 2005; Qiu et al., 2009; Qiuet al., 2011). However, the sentences in onlinereviews usually have informal writing styles in-cluding grammar mistakes, typos, improper punc-tuation etc., which make parsing prone to gener-ate mistakes. As a result, the syntax-based meth-ods which heavily depended on the parsing per-formance would suffer from parsing errors (Zhanget al., 2010). To improve the extraction perfor-mance, we can only employ some exquisite high-precision patterns. But this strategy is likely tomiss many opinion targets and has lower recallwith the increase of corpus size. To resolve theseproblems, Liu et al. (2012) formulated identifyingopinion relations between words as an monolin-gual alignment process. A word can find its cor-responding modifiers by using a word alignment

1754

Figure 1: Mining Opinion Relations between Words using Partially Supervised Alignment Model

model (WAM). Without using syntactic parsing,the noises from parsing errors can be effectivelyavoided. Nevertheless, we notice that the align-ment model is a statistical model which needs suf-ficient data to estimate parameters. When the datais insufficient, it would suffer from data sparsenessand may make the performance decline.

Thus, from the above analysis, we can observethat the size of the corpus has impacts on thesetwo kinds of methods, which arises some impor-tant questions: how can we make selection be-tween syntax based methods and alignment basedmethod for opinion target extraction when givena certain amount of reviews? And which kind ofmethods can obtain better extraction performancewith the variation of the size of the dataset? Al-though (Liu et al., 2012) had proved the effective-ness of WAM, they mainly performed experimentson the dataset with medium size. We are still curi-ous about that when the size of dataset is largeror smaller, can we obtain the same conclusion?To our best knowledge, these problems have notbeen studied before. Moreover, opinions may beexpressed in different ways with the variation ofthe domain and language of the corpus. When thedomain or language of the corpus is changed, whatconclusions can we obtain? To answer these ques-tions, in this paper, we adopt a unified frameworkto extract opinion targets from reviews, in the keycomponent of which we vary the methods betweensyntactic patterns and alignment model. Then werun the whole framework on the corpus with dif-ferent size (from #500 to #1, 000, 000), domain(three domains) and language (Chinese and En-glish) to empirically assess the performance varia-tions and discuss which method is more effective.

Furthermore, this paper naturally addresses an-other question: is it useful for opinion targets ex-traction when we combine syntactic patterns andword alignment model into a unified model? To

this end, we employ a partially supervised align-ment model (PSWAM) like (Gao et al., 2010; Liuet al., 2013). Based on the exquisitely designedhigh-precision syntactic patterns, we can obtainsome precisely modified relations between wordsin sentences, which provide a portion of links ofthe full alignments. Then, these partial alignmentlinks can be regarded as the constrains for a stan-dard unsupervised word alignment model. Andeach target candidate would find its modifier un-der the partial supervision. In this way, the er-rors generated in standard unsupervised WAM canbe corrected. For example in Figure 1, “kindly”and “courteous” are incorrectly regarded as themodifiers for “foods” if the WAM is performedin an whole unsupervised framework. However,by using some high-precision syntactic patterns,we can assert “courteous” should be aligned to“services”, and “delicious” should be aligned to“foods”. Through combination under partial su-pervision, we can see “kindly” and “courteous”are correctly linked to “services”. Thus, it’s rea-sonable to expect to yield better performance thantraditional methods. As mentioned in (Liu et al.,2013), using PSWAM can not only inherit theadvantages of WAM: effectively avoiding noisesfrom syntactic parsing errors when dealing withinformal texts, but also can improve the miningperformance by using partial supervision. How-ever, is this kind of combination always useful foropinion target extraction? To access this problem,we also make comparison between PSWAM basedmethod and the aforementioned methods in thesame corpora with different size, language and do-main. The experimental results show the combina-tion by using PSWAM can be effective on datasetwith small and medium size.

1755

2 Related Work

Opinion target extraction isn’t a new task for opin-ion mining. There are much work focusing onthis task, such as (Hu and Liu, 2004b; Ding et al.,2008; Li et al., 2010; Popescu and Etzioni, 2005;Wu et al., 2009). Totally, previous studies can bedivided into two main categories: supervised andunsupervised methods.

In supervised approaches, the opinion target ex-traction task was usually regarded as a sequencelabeling problem (Jin and Huang, 2009; Li et al.,2010; Ma and Wan, 2010; Wu et al., 2009; Zhanget al., 2009). It’s not only to extract a lexicon or listof opinion targets, but also to find out each opin-ion target mentions in reviews. Thus, the contex-tual words are usually selected as the features toindicate opinion targets in sentences. And classi-cal sequence labeling models are used to train theextractor, such as CRFs (Li et al., 2010), HMM(Jin and Huang, 2009) etc.. Jin et al. (2009) pro-posed a lexicalized HMM model to perform opin-ion mining. Both Li et al. (2010) and Ma et al.(2010) used CRFs model to extract opinion tar-gets in reviews. Specially, Li et al. proposed aSkip-Tree CRF model for opinion target extrac-tion, which exploited three structures includinglinear-chain structure, syntactic structure, and con-junction structure. However, the main limitationof these supervised methods is the need of labeledtraining data. If the labeled training data is insuf-ficient, the trained model would have unsatisfiedextraction performance. Labeling sufficient train-ing data is time and labor consuming. And for dif-ferent domains, we need label data independently,which is obviously impracticable.

Thus, many researches focused on unsupervisedmethods, which are mainly to extract a list of opin-ion targets from reviews. Similar to ours, most ap-proaches regarded opinion words as the indicatorfor opinion targets. (Hu and Liu, 2004a) regardedthe nearest adjective to an noun/noun phrase asits modifier. Then it exploited an associationrule mining algorithm to mine the associations be-tween them. Finally, the frequent explicit prod-uct features can be extracted in a bootstrappingprocess by further combining item’s frequency indataset. Only using nearest neighbor rule to minethe modifier for each candidate cannot obtain pre-cise results. Thus, (Popescu and Etzioni, 2005)used syntax information to extract opinion targets,which designed some syntactic patterns to capture

the modified relations between words. The experi-mental results showed that their method had betterperformance than (Hu and Liu, 2004a). Moreover,(Qiu et al., 2011) proposed a Double Propagationmethod to expand sentiment words and opiniontargets iteratively, where they also exploited syn-tactic relations between words. Specially, (Qiuet al., 2011) didn’t only design syntactic patternsfor capturing modified relations, but also designedpatterns for capturing relations among opinion tar-gets and relations among opinion words. How-ever, the main limitation of Qiu’s method is thatthe patterns based on dependency parsing tree maymiss many targets for the large corpora. There-fore, Zhang et al. (2010) extended Qiu’s method.Besides the patterns used in Qiu’s method, theyadopted some other special designed patterns toincrease recall. In addition they used the HITS(Kleinberg, 1999) algorithm to compute opiniontarget confidences to improve the precision. (Liuet al., 2012) formulated identifying opinion re-lations between words as an alignment process.They used a completely unsupervised WAM tocapture opinion relations in sentences. Then theopinion targets were extracted in a standard ran-dom walk framework where two factors were con-sidered: opinion relevance and target importance.Their experimental results have shown that WAMwas more effective than traditional syntax-basedmethods for this task. (Liu et al., 2013) extendLiu’s method, which is similar to our method andalso used a partially supervised alignment modelto extract opinion targets from reviews. We noticethese two methods ((Liu et al., 2012) and (Liu etal., 2013)) only performed experiments on the cor-pora with a medium size. Although both of themproved that WAM model is better than the meth-ods based on syntactic patterns, they didn’t dis-cuss the performance variation when dealing withthe corpora with different sizes, especially whenthe size of the corpus is less than 1,000 and morethan 10,000. Based on their conclusions, we stilldon’t know which kind of methods should be se-lected for opinion target extraction when given acertain amount of reviews.

3 Opinion Target ExtractionMethodology

To extract opinion targets from reviews, we adoptthe framework proposed by (Liu et al., 2012),which is a graph-based extraction framework and

1756

has two main components as follows.1) The first component is to capture opinion

relations in sentences and estimate associationsbetween opinion target candidates and potentialopinion words. In this paper, we assume opiniontargets to be nouns or noun phrases, and opinionwords may be adjectives or verbs, which are usu-ally adopted by (Hu and Liu, 2004a; Qiu et al.,2011; Wang and Wang, 2008; Liu et al., 2012).And a potential opinion relation is comprised ofan opinion target candidate and its correspondingmodified word.

2) The second component is to estimate theconfidence of each candidate. The candidates withhigher confidence scores than a threshold will beextracted as opinion targets. In this procedure, weformulate the associations between opinion targetcandidates and potential opinion words in a bipar-tite graph. A random walk based algorithm is em-ployed on this graph to estimate the confidence ofeach target candidate.

In this paper, we fix the method in the sec-ond component and vary the algorithms in thefirst component. In the first component, we re-spectively use syntactic patterns and unsupervisedword alignment model (WAM) to capture opinionrelations. In addition, we employ a partially super-vised word alignment model (PSWAM) to incor-porate syntactic information into WAM. In exper-iments, we run the whole framework on the differ-ent corpora to discuss which method is more effec-tive. In the following subsections, we will presentthem in detail.

3.1 The First Component: CapturingOpinion Relations and EstimatingAssociations between Words

3.1.1 Syntactic PatternsTo capture opinion relations in sentences by usingsyntactic patterns, we employ the manual designedsyntactic patterns proposed by (Qiu et al., 2011).Similar to Qiu, only the syntactic patterns basedon the direct dependency are employed to guar-antee the extraction qualities. The direct depen-dency has two types. The first type indicates thatone word depends on the other word without anyadditional words in their dependency path. Thesecond type denotes that two words both dependon a third word directly. Specifically, we employMinipar1 to parse sentences. To further make syn-

1http://webdocs.cs.ualberta.ca/lindek/minipar.htm

tactic patterns precisely, we only use a few depen-dency relation labels outputted by Minipar, suchas mod, pnmod, subj, desc etc. To make a clearexplanation, we give out some syntactic patternexamples in Table 1. In these patterns, OC is apotential opinion word which is an adjective or averb. TC is an opinion target candidate which isa noun or noun phrase. The item on the arrowsmeans the dependency relation type. The item inparenthesis denotes the part-of-speech of the otherword. In these examples, the first three patternsare based on the first direct dependency type andthe last two patterns are based on the second directdependency type.

Pattern#1: <OC> mod−−−→<TC>Example: This phone has an amazing design

Pattern#2: <TC>obj−−→<OC>

Example: I like this phone very much

Pattern#3: <OC>pnmod−−−−→<TC>

Example: the buttons easier to use

Pattern#4: <OC> mod−−−→(NN)subj←−−−<TC>

Example: IPhone is a revolutionary smart phone

Pattern#5: <OC>pred−−−→(VBE)

subj←−−−<TC>Example: The quality of LCD is good

Table 1: Some Examples of Used Syntactic Pat-terns

3.1.2 Unsupervised Word Alignment ModelIn this subsection, we present our method for cap-turing opinion relations using unsupervised wordalignment model. Similar to (Liu et al., 2012),every sentence in reviews is replicated to gener-ate a parallel sentence pair, and the word align-ment algorithm is applied to the monolingual sce-nario to align a noun/noun phase with its modi-fiers. We select IBM-3 model (Brown et al., 1993)as the alignment model. Formally, given a sen-tence S = {w1, w2, ..., wn}, we have

Pibm3(A|S)

∝N∏

i=1

n(φi|wi)N∏

j=1

t(wj |waj )d(j|aj , N)(1)

where t(wj |waj ) models the co-occurrence infor-mation of two words in dataset. d(j|aj , n) mod-els word position information, which describes theprobability of a word in position aj aligned with aword in position j. And n(φi|wi) describes theability of a word for modifying (being modifiedby) several words. φi denotes the number of words

1757

that are aligned with wi. In our experiments, weset φi = 2.

Since we only have interests on capturing opin-ion relations between words, we only pay at-tentions on the alignments between opinion tar-get candidates (nouns/noun phrases) and potentialopinion words (adjectives/verbs). If we directlyuse the alignment model, a noun (noun phrase)may align with other unrelated words, like prepo-sitions or conjunctions and so on. Thus, we setconstrains on the model: 1) Alignment links mustbe assigned among nouns/noun phrases, adjec-tives/verbs and null words. Aligning to null wordsmeans that this word has no modifier or modifiesnothing; 2) Other unrelated words can only alignwith themselves.

3.1.3 Combining Syntax-based Method withAlignment-based Method

In this subsection, we try to combine syntactic in-formation with word alignment model. As men-tioned in the first section, we adopt a partiallysupervised alignment model to make this com-bination. Here, the opinion relations obtainedthrough the high-precision syntactic patterns (Sec-tion 3.1.1) are regarded as the ground truth andcan only provide a part of full alignments in sen-tences. They are treated as the constrains for theword alignment model. Given some partial align-ment links A = {(k, ak)|k ∈ [1, n], ak ∈ [1, n]},the optimal word alignment A∗ = {(i, ai)|i ∈[1, n], ai ∈ [1, n]} can be obtained as A∗ =argmax

AP (A|S, A), where (i, ai) means that a

noun (noun phrase) at position i is aligned withits modifier at position ai.

Since the labeled data provided by syntactic pat-terns is not a full alignment, we adopt a EM-basedalgorithm, named as constrained hill-climbing al-gorithm(Gao et al., 2010), to estimate the parame-ters in the model. In the training process, the con-strained hill-climbing algorithm can ensure thatthe final model is marginalized on the partial align-ment links. Particularly, in the E step, their methodaims to find out the alignments which are consis-tent to the alignment links provided by syntacticpatterns, where there are main two steps involved.

1) Optimize towards the constraints. This stepaims to generate an initial alignments for align-ment model (IBM-3 model in our method), whichcan be close to the constraints. First, a simplealignment model (IBM-1, IBM-2, HMM etc.) is

trained. Then, the evidence being inconsistentto the partial alignment links will be got rid ofby using the move operator operator mi,j whichchanges aj = i and the swap operator sj1,j2 whichexchanges aj1 and aj2 . The alignment is updatediteratively until no additional inconsistent linkscan be removed.

2) Towards the optimal alignment under theconstraints. This step aims to optimize towardsthe optimal alignment under the constraints whichstarts from the aforementioned initial alignments.Gao et.al. (2010) set the corresponding cost valueof the invalid move or swap operation in M andS to be negative, where M and S are respec-tively called Moving Matrix and Swapping Ma-trix, which record all possible move and swapcosts between two different alignments. In thisway, the invalid operators will never be pickedwhich can guarantee that the final alignment linksto have high probability to be consistent with thepartial alignment links provided by high-precisionsyntactic patterns.

Then in M-step, evidences from the neighbor offinal alignments are collected so that we can pro-duce the estimation of parameters for the next iter-ation. In the process, those statistics which comefrom inconsistent alignment links aren’t be pickedup. Thus, we have

P (wi|wai , A)

=

{λ, otherwise

P (wi|wai) + λ, inconsistent with A(2)

where λ means that we make soft constraints onthe alignment model. As a result, we expect someerrors generated through high-precision patterns(Section 3.1.1) may be revised in the alignmentprocess.

3.2 Estimating Associations between WordsAfter capturing opinion relations in sentences, wecan obtain a lot of word pairs, each of which iscomprised of an opinion target candidate and itscorresponding modified word. Then the condi-tional probabilities between potential opinion tar-get wt and potential opinion word wo can be es-timated by using maximum likelihood estimation.Thus, we have P (wt|wo) = Count(wt,wo)

Count(wo), where

Count(·) means the item’s frequency informa-tion. P (wt|wo) means the conditional probabili-ties between two words. At the same time, we canobtain conditional probability P (wo|wt). Then,

1758

similar to (Liu et al., 2012), the association be-tween an opinion target candidate and its modifieris estimated as follows. Association(wt, wo) =(α× P (wt|wo) + (1− α)× P (wo|wt))

−1, whereα is the harmonic factor. We set α = 0.5 in ourexperiments.

3.3 The Second Component: EstimatingCandidate Confidence

In the second component, we adopt a graph-basedalgorithm used in (Liu et al., 2012) to computethe confidence of each opinion target candidate,and the candidates with higher confidence than thethreshold will be extracted as the opinion targets.Here, opinion words are regarded as the impor-tant indicators. We assume that two target candi-dates are likely to belong to the similar category, ifthey are modified by similar opinion words. Thus,we can propagate the opinion target confidencesthrough opinion words.

To model the mined associations betweenwords, a bipartite graph is constructed, whichis defined as a weighted undirected graph G =(V,E,W ). It contains two kinds of vertex: opin-ion target candidates and potential opinion words,respectively denoted as vt ∈ V and vo ∈ V .As shown in Figure 2, the white vertices repre-sent opinion target candidates and the gray ver-tices represent potential opinion words. An edgeevt,vo ∈ E between vertices represents that there isan opinion relation, and the weight w on the edgerepresents the association between two words.

Figure 2: Modeling Opinion Relations betweenWords in a Bipartite Graph

To estimate the confidence of each opinion tar-get candidate, we employ a random walk algo-rithm on our graph, which iteratively computesthe weighted average of opinion target confidencesfrom neighboring vertices. Thus we have

Ci+1 = (1− β)×M ×MT × Ci + β × I (3)

where Ci+1 and Ci respectively represent theopinion target confidence vector in the (i + 1)th

and ith iteration. M is the matrix of word asso-ciations, where Mi,j denotes the association be-tween the opinion target candidate i and the po-tential opinion word j. And I is defined as theprior confidence of each candidate for opinion tar-get. Similar to (Liu et al., 2012), we set each itemin Iv = tf(v)idf(v)∑

v tf(v)idf(v) , where tf(v) is the term fre-quency of v in the corpus, and df(v) is computedby using the Google n-gram corpus2. β ∈ [0, 1]represents the impact of candidate prior knowl-edge on the final estimation results. In experi-ments, we set β = 0.4. The algorithm run un-til convergence which is achieved when the confi-dence on each node ceases to change in a tolerancevalue.

4 Experiments

4.1 Datasets and Evaluation Metrics

In this section, to answer the questions men-tioned in the first section, we collect a largecollection named as LARGE, which includes re-views from three different domains and differ-ent languages. This collection was also usedin (Liu et al., 2012). In the experiments, re-views are first segmented into sentences accord-ing to punctuation. The detailed statistical in-formation of the used collection is shown in Ta-ble 2, where Restaurant is crawled from the Chi-nese Web site: www.dianping.com. The Hotel andMP3 are used in (Wang et al., 2011), which are re-spectively crawled from www.tripadvisor.com andwww.amazon.com. For each dataset, we performrandom sampling to generate testing set with dif-ferent sizes, where we use sampled subsets with#sentences = 5× 102, 103, 5× 103, 104, 5×104, 105 and 106 sentences respectively. Each

Domain Language Sentence ReviewsRestaurant Chinese 1,683,129 395,124Hotel English 1,855,351 185,829MP3 English 289,931 30,837

Table 2: Experimental Dataset

sentence is tokenized, part-of-speech tagged byusing Stanford NLP tool3, and parsed by usingMinipar toolkit. And the method of (Zhu et al.,2009) is used to identify noun phrases.

2http://books.google.com/ngrams/datasets3http://nlp.stanford.edu/software/tagger.shtml

1759

We select precision and recall as the metrics.Specifically, to obtain the ground truth, we man-ually label all opinion targets for each subset. Inthis process, three annotators are involved. First,every noun/noun phrase and its contexts in reviewsentences are extracted. Then two annotators wererequired to judge whether every noun/noun phraseis opinion target or not. If a conflict happens, athird annotator will make judgment for final re-sults. The average inter-agreements is 0.74. Wealso perform a significant test, i.e., a t-test with adefault significant level of 0.05.

4.2 Compared Methods

We select three methods for comparison as fol-lows.

• Syntax: It uses syntactic patterns mentionedin Section 3.1.1 in the first component tocapture opinion relations in reviews. Thenthe associations between words are estimatedand the graph based algorithm proposed inthe second component (Section 3.3) is per-formed to extract opinion targets.

• WAM: It is similar to Syntax, where the onlydifference is that WAM uses unsupervisedWAM (Section 3.1.2) to capture opinion re-lations.

• PSWAM is similar to Syntax and WAM,where the difference is that PSWAM uses themethod mentioned in Section 3.1.3 to captureopinion relations, which incorporates syntac-tic information into word alignment model byusing partially supervised framework.

The experimental results on different domains arerespectively shown in Figure 3, 4 and 5.

4.3 Syntax based Methods vs. Alignmentbased Methods

Comparing Syntax with WAM and PSWAM, wecan obtain the following observations:

Figure 3: Experimental results on Restaurant

Figure 4: Experimental results on Hotel

Figure 5: Experimental results on MP3

1) When the size of the corpus is small, Syntaxhas better precision than alignment based meth-ods (WAM and PSWAM). We believe the reasonis that the high-precision syntactic patterns em-ployed in Syntax can effectively capture opinionrelations in a small amount of texts. In contrast,the methods based on word alignment model maysuffer from data sparseness for parameter estima-tion, so the precision is lower.

2) However, when the size of the corpus in-creases, the precision of Syntax decreases, evenworse than alignment based methods. We believeit’s because more noises were introduced fromparsing errors with the increase of the size of thecorpus , which will have more negative impacts onextraction results. In contrast, for estimating theparameters of alignment based methods, the datais more sufficient, so the precision is better com-pared with syntax based method.

3) We also observe that recall of Syntax isworse than other two methods. It’s because thehuman expressions of opinions are diverse and themanual designed syntactic patterns are limited tocapture all opinion relations in sentences, whichmay miss an amount of correct opinion targets.

4) It’s interesting that the performance gap be-tween these three methods is smaller with the in-crease of the size of the corpus (more than 50,000).We guess the reason is that when the data is suffi-cient enough, we can obtain sufficient statistics foreach opinion target. In such situation, the graph-based ranking algorithm in the second componentwill be apt to be affected by the frequency infor-mation, so the final performance could not be sen-sitive to the performance of opinion relations iden-

1760

tification in the first component. Thus, in this situ-ation, we can get conclusion that there is no obvi-ously difference on performance between syntax-based approach and alignment-based approach.

5) From the results on dataset with different lan-guages and different domains, we can obtain thesimilar observations. It indicates that choosing ei-ther syntactic patterns or word alignment modelfor extracting opinion targets can take a few con-sideration on the language and domain of the cor-pus.

Thus, based on the above observations, we candraw the following conclusions: making choosesbetween different methods is only related to thesize of the corpus. The method based on syn-tactic patterns is more suitable for small cor-pus (#sentences < 5 × 103 shown in ourexperiments). And word alignment model ismore suitable for medium corpus (5 × 103 <#sentences < 5 × 104). Moreover, when thesize of the corpus is big enough, the performanceof two kinds of methods tend to become the same(#sentences ≥ 105 shown in our experiments).

4.4 Is It Useful Combining Syntactic Patternswith Word Alignment Model

In this subsection, we try to see whether combin-ing syntactic information with alignment model byusing PSWAM is effective or not for opinion tar-get extraction. From the results in Figure 3, 4 and5, we can see that PSWAM has the similar recallcompared with WAM in all datasets. PSWAMoutperforms WAM on precision in all dataset. Butthe precision gap between PSWAM and WAMdecreases when the size of the corpus increases.When the size is larger than 5 × 104, the perfor-mance of these two methods is almost the same.We guess the reason is that more noises from pars-ing errors will be introduced by syntactic patternswith the increase of the size of corpus , which havenegative impacts on alignment performance. Atthe same time, as mentioned above, a great deal ofreviews will bring sufficient statistics for estimat-ing parameters in alignment model, so the rolesof partial supervision from syntactic informationwill be covered by frequency information used inour graph based ranking algorithm.

Compared with State-of-the-art Methods.However, it’s not say that this combination isnot useful. From the results, we still see thatPSWAM outperforms WAM in all datasets on

precision when size of corpus is smaller than5 × 104. To further prove the effectiveness ofour combination, we compare PSWAM with somestate-of-the-art methods, including Hu (Hu andLiu, 2004a), which extracted frequent opinion tar-get words based on association mining rules, DP(Qiu et al., 2011), which extracted opinion tar-gets through syntactic patterns, and LIU (Liu etal., 2012), which fulfilled this task by using un-supervised WAM. The parameter settings in thesebaselines are the same as the settings in the orig-inal papers. Because of the space limitation, weonly show the results on Restaurant and Hotel, asshown in Figure 6 and 7.

Figure 6: Compared with the State-of-the-artMethods on Restaurant

Figure 7: Compared with the State-of-the-artMethods on Hotel

From the experimental results, we can obtainthe following observations. PSWAM outperformsother methods in most datasets. This indicatesthat our method based on PSWAM is effectivefor opinion target extraction. Especially comparedPSWAM with LIU, both of which are based onword alignment model, we can see PSWAM iden-tifies opinion relations by performing WAM underpartial supervision, which can effectively improvethe precision when dealing with small and mediumcorpus. However, these improvements are limitedwhen the size of the corpus increases, which hasthe similar observations obtained above.

The Impact of Syntactic Information onWord Alignment Model. Although we haveprove the effectiveness of PSWAM in the corpuswith small and medium size, we are still curiousabout how the performance varies when we incor-

1761

porate different amount of syntactic informationinto WAM. In this experiment, we rank the usedsyntactic patterns mentioned in Section 3.1.1 ac-cording to the quantities of the extracted alignmentlinks by these patterns. Then, to capture opin-ion relations, we respectively use top N syntacticpatterns according to frequency mentioned aboveto generate partial alignment links for PSWAM insection 3.1.3. We respectively define N=[1,7]. Thelarger is N , the more syntactic information is in-corporated. Because of the space limitation, onlythe average performance of all dataset is shown inFigure 8.

Figure 8: The Impacts of Different Syntactic In-formation on Word Alignment Model

In Figure 8, we can observe that the syntactic in-formation mainly have effect on precision. Whenthe size of the corpus is small, the opinion rela-tions mined by high-precision syntactic patternsare usually correct, so incorporating more syntac-tic information can improve the precision of wordalignment model more. However, when the size ofthe corpus increases, incorporating more syntacticinformation has little impact on precision.

5 Conclusions and Future Work

This paper discusses the performance variation ofsyntax based methods and alignment based meth-ods on opinion target extraction task for the datasetwith different sizes, different languages and dif-ferent domains. Through experimental results, wecan see that choosing which method is not related

with corpus domain and language, but stronglyassociated with the size of the corpus . We canconclude that syntax-based method is likely to bemore effective when the size of the corpus is small,and alignment-based methods are more useful forthe medium size corpus. We further verify that in-corporating syntactic information into word align-ment model by using PSWAM is effective whendealing with the corpora with small or mediumsize. When the size of the corpus is larger andlarger, the performance gap between syntax based,WAM and PSWAM will decrease.

In future work, we will extract opinion targetsbased on not only opinion relations. Other seman-tic relations, such as the topical associations be-tween opinion targets (or opinion words) shouldalso be employed. We believe that consideringmultiple semantic associations will help to im-prove the performance. In this way, how to modelheterogenous relations in a unified model for opin-ion targets extraction is worthy to be studied.

Acknowledgement

This work was supported by the National Natu-ral Science Foundation of China (No. 61070106,No. 61272332 and No. 61202329), the Na-tional High Technology Development 863 Pro-gram of China (No. 2012AA011102), the Na-tional Basic Research Program of China (No.2012CB316300), Tsinghua National Laboratoryfor Information Science and Technology (TNList)Cross-discipline Foundation and the OpeningProject of Beijing Key Laboratory of Inter-net Culture and Digital Dissemination Research(ICDD201201).

ReferencesPeter F. Brown, Vincent J. Della Pietra, Stephen

A. Della Pietra, and Robert L. Mercer. 1993. Themathematics of statistical machine translation: pa-rameter estimation. Comput. Linguist., 19(2):263–311, June.

Xiaowen Ding, Bing Liu, and Philip S. Yu. 2008. Aholistic lexicon-based approach to opinion mining.In Proceedings of the Conference on Web Search andWeb Data Mining (WSDM).

Qin Gao, Nguyen Bach, and Stephan Vogel. 2010. Asemi-supervised word alignment algorithm with par-tial manual alignments. In Proceedings of the JointFifth Workshop on Statistical Machine Translationand MetricsMATR, pages 1–10, Uppsala, Sweden,July. Association for Computational Linguistics.

1762

Mingqin Hu and Bing Liu. 2004a. Mining opinion fea-tures in customer reviews. In Proceedings of Con-ference on Artificial Intelligence (AAAI).

Minqing Hu and Bing Liu. 2004b. Mining and summa-rizing customer reviews. In Proceedings of the tenthACM SIGKDD international conference on Knowl-edge discovery and data mining, KDD ’04, pages168–177, New York, NY, USA. ACM.

Wei Jin and Hay Ho Huang. 2009. A novel lexical-ized hmm-based learning framework for web opin-ion mining. In Proceedings of International Confer-ence on Machine Learning (ICML).

Jon M. Kleinberg. 1999. Authoritative sources in ahyperlinked environment. J. ACM, 46(5):604–632,September.

Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu,Yingju Xia, Shu Zhang, and Hao Yu. 2010.Structure-aware review mining and summarization.In Chu-Ren Huang and Dan Jurafsky, editors, COL-ING, pages 653–661. Tsinghua University Press.

Bing Liu, Minqing Hu, and Junsheng Cheng. 2005.Opinion observer: analyzing and comparing opin-ions on the web. In Allan Ellis and Tatsuya Hagino,editors, WWW, pages 342–351. ACM.

Kang Liu, Liheng Xu, and Jun Zhao. 2012. Opin-ion target extraction using word-based translationmodel. In Proceedings of the 2012 Joint Confer-ence on Empirical Methods in Natural LanguageProcessing and Computational Natural LanguageLearning, pages 1346–1356, Jeju Island, Korea,July. Association for Computational Linguistics.

Kang Liu, Liheng Xu, Yang Liu, and Jun Zhao. 2013.Opinion target extraction using partially supervisedword alignment model.

Tengfei Ma and Xiaojun Wan. 2010. Opinion tar-get extraction in chinese news comments. In Chu-Ren Huang and Dan Jurafsky, editors, COLING(Posters), pages 782–790. Chinese Information Pro-cessing Society of China.

Ana-Maria Popescu and Oren Etzioni. 2005. Ex-tracting product features and opinions from reviews.In Proceedings of the conference on Human Lan-guage Technology and Empirical Methods in Natu-ral Language Processing, HLT ’05, pages 339–346,Stroudsburg, PA, USA. Association for Computa-tional Linguistics.

Guang Qiu, Bing Liu, Jiajun Bu, and Chun Che. 2009.Expanding domain sentiment lexicon through dou-ble propagation.

Guang Qiu, Bing Liu 0001, Jiajun Bu, and Chun Chen.2011. Opinion word expansion and target extractionthrough double propagation. Computational Lin-guistics, 37(1):9–27.

Bo Wang and Houfeng Wang. 2008. Bootstrappingboth product features and opinion words from chi-nese customer reviews with cross-inducing.

Hongning Wang, Yue Lu, and ChengXiang Zhai. 2011.Latent aspect rating analysis without aspect key-word supervision. In Chid Apt, Joydeep Ghosh,and Padhraic Smyth, editors, KDD, pages 618–626.ACM.

Yuanbin Wu, Qi Zhang, Xuanjing Huang, and Lide Wu.2009. Phrase dependency parsing for opinion min-ing. In EMNLP, pages 1533–1541. ACL.

Qi Zhang, Yuanbin Wu, Tao Li, Mitsunori Ogihara,Joseph Johnson, and Xuanjing Huang. 2009. Min-ing product reviews based on shallow dependencyparsing. In Proceedings of the 32nd internationalACM SIGIR conference on Research and develop-ment in information retrieval, SIGIR ’09, pages726–727, New York, NY, USA. ACM.

Lei Zhang, Bing Liu, Suk Hwan Lim, and EamonnO’Brien-Strain. 2010. Extracting and rankingproduct features in opinion documents. In Chu-Ren Huang and Dan Jurafsky, editors, COLING(Posters), pages 1462–1470. Chinese InformationProcessing Society of China.

Jingbo Zhu, Huizhen Wang, Benjamin K. Tsou, andMuhua Zhu. 2009. Multi-aspect opinion pollingfrom textual reviews. In David Wai-Lok Cheung,Il-Yeol Song, Wesley W. Chu, Xiaohua Hu, andJimmy J. Lin, editors, CIKM, pages 1799–1802.ACM.

1763