determining trust in media-rich websites using semantic similarity

Multimed Tools Appl (2012) 60:69–96DOI 10.1007/s11042-011-0798-x

Determining trust in media-rich websitesusing semantic similarity

Pradeep K. Atrey · Hicham Ibrahim ·M. Anwar Hossain · Sheela Ramanna ·Abdulmotaleb El Saddik

Published online: 28 April 2011© Springer Science+Business Media, LLC 2011

Abstract Significant growth of multimedia content on the World Wide Web (orsimply ‘Web’) has made it an essential part of peoples lives. The web providesenormous amount of information, however, it is very important for the users tobe able to gauge the trustworthiness of web information. Users normally accesscontent from the first few links provided to them by search engines such as Googleor Yahoo!. This is assuming that these search engines provide factual information,which may be popular due to criteria such as page rank but may not always betrustworthy from the factual aspects. This paper presents a mechanism to determinetrust of websites based on the semantic similarity of their multimedia content withalready established and trusted websites. The proposed method allows for dynamiccomputation of the trust level of websites of different domains and hence overcomesthe dependency on traditional user feedback methods for determining trust. In fact,our method attempts to emulate the evolving process of trust that takes place in a

Authors sincerely thank the Natural Science and Engineering Research Council of Canadaand the King Saud University Visiting Professors Program for supporting this research.

P. K. Atrey (B) · S. RamannaDepartment of Applied Computer Science, University of Winnipeg, Winnipeg, Canadae-mail: [email protected]

S. Ramannae-mail: [email protected]

H. Ibrahim · A. El SaddikMultimedia Communications Research Laboratory, University of Ottawa, Ottawa, Canadae-mail: [email protected]

A. El Saddike-mail: [email protected]

M. Anwar HossainSoftware Engineering Department, CCIS, King Saud University, Riyadh, Saudi Arabiae-mail: [email protected]

70 Multimed Tools Appl (2012) 60:69–96

user’s mind. The experimental results have been provided to demonstrate the utilityand practicality of the proposed method.

Keywords Multimedia · Web · Trust · Semantic similarity

1 Introduction

In the past decade, the World Wide Web has become extraordinarily popular dueto its ability to provide an enormous amount of information to users. In general,there are several websites that can provide information for a particular interest. Forexample, the websites such as www.cnn.com, www.bbc.com, www.cbc.ca, etc. providenews belonging to politics. These websites also provide general news in severalother domains such as health, sports, entertainment, business, etc. Furthermore, onecan visit specified websites that concentrate in particular domains such as health(e.g. familDoctor.org, kidsHealth.org, HealthFinder.gov, etc.) or sports (e.g. tsn.ca,espn.go.com, skySports.com, etc.).

Due to the tremendous growth in the Internet’s infrastructure, websites today areusually media-rich [37]. Apart from textual content, information is also presented inother different forms such as image, audio and video. For instance, most of the newswebsites also present an image or a video related to the news. Multimedia websitesare generally more informative and appealing to the users.

The availability of a large number of multimedia websites belonging to a particulardomain raises the issue of which website should be trusted, since not all websitesalways provide factual and trustworthy information. For example, the websitewww.cbc.ca may be trustworthy for political news, but any arbitrary website may notbe trustworthy, or vice versa. Moreover, the existing search engines do not performtrust-based searching, but they tend to reward highly popular sites by listing themon top of the search results. Popularity of a website is usually determined based onvarious criteria such as page rank [13] which does not necessarily reflect its trust level.For instance, in existing approaches a website with a very low page rank will have alow trust level than a website with a very high page rank, although the actual contentof the former is quite similar to a trustworthy website than the latter.

The rewarding strategy such as page rank and browsing history adopted by theexisting search mechanisms makes it quite difficult to prevent these websites fromdominating the global exchange of ideas on the Internet. Due to this, several impor-tant trustable websites go unnoticed and the users tend to limit their exploration toa few websites such as MySpace, Yahoo, eBay, for instance. This could definitelyraise more issues with search engines that seem to amplify this tendency by rankingsearch results partially based on a particular site’s current popularity. Thus, highlypopular sites could become even more popular, whereas less popular sites couldfade even more in the World Wide Web. This could lead one to detect a bias orinequitable element introduced by the more popular search engines which seem todiscriminate against these so called smaller and less popular, yet trustable, websites.These discrepancies with the existing trust computation mechanisms motivate us toresearch alternative techniques that would take the factual aspect into considerationin determining the trust level of a website.

Multimed Tools Appl (2012) 60:69–96 71

This paper considers the trust based on the factual aspects and proposes a methodto dynamically compute the trust level of a website of a particular domain basedon how similar its content is with a trusted website of the same domain. Earlierwe introduced the idea of determining trust based on semantic similarity betweentwo websites in [24]. In that work, semantic similarity was computed based on onlytext, however in this paper we extend it to multimedia content and present detailedresults and analysis. The proposed approach could be illustrated by a simple example.Let us assume that one consults a certain website for the sports news, and thusbuilds a certain trust in this website over a certain period of time. Then, uponconsulting another website for the sport news simultaneously and observing thesimilarity/dissimilarity between the two, one will eventually develop a certain trustin the second website if the content of both websites are found to be similar over thistime frame. Thus, the computation of this trust level will be based on the past historyof similarity measured between this website and a so called “well-trusted” website inthat domain. The similarity measured between the two websites is computed basedon how similar the content of the two websites has been in the past. The so called“well-trusted” website within a particular domain could be found by using traditionalmethods such as user surveys.

The main contribution of this paper is summarized as follows. We propose amethod to dynamically compute the trust level of a website based on the similarity ofits multimedia content with the other already well-trusted websites. The proposedmethod determines the trust that is based on factual aspects of the website andthe trust value can be incorporated in the current search engine’s page rankingmechanism.

The remainder of this paper is organized into five sections. In Section 2, we coverthe related work. Section 3 describes the proposed method and Section 4 presentsthe experimental results. Finally, in Section 5, we conclude the paper by discussingfuture work.

2 Related work

Before describing the past works on trust, we look at the various definitions oftrust. Oxford English Dictionary defines trust as confidence in or reliance on somequality or attribute of a person or thing, or the truth of a statement [4]. Furthermore,an analysis of the term by Nissenbaum [32] led to this following statement “Trustis an extraordinary rich concept, covering a variety of relationships, conjoining avariety of objects. One can trust (or distrust) persons, institutions, governments,information, deities, physical things, systems, and more” [32]. Even basic definitionsof the word have caused several disagreements in the research world [23]. In fact,we can attribute this literal divergence among scholars to the multiple definitionsthat the word presents. On one hand, trust can be viewed as an abstract notion,where its use can be interchanged with words such as confidence, reliability orcredibility. Therefore, simply making a clear and comprehensible distinction betweentrust and its related notions shows to be quite an obstacle for researchers across theworld [19]. On the other hand, trust can be viewed as “a multi-faced concept thatincorporates cognitive, emotional, and behavioral dimensions” [19]. Several studies


emerging from different disciplines have been made regarding the concept of trust,while respectively presenting a distinctive understanding of the concept.

In the context of websites, Schneider [36] has presented a five-point system todetermine which website one can trust when a search engine displays hundredsof websites corresponding to the desired topic. This five-point system includes thefollowing principals: availability, credibility, authorship, external links, and legality.The author also presented five additional questions to consider when choosinga trustworthy website for a particular topic, when a selection of websites aredisplayed by a search engine [36]. These are: (1) Is the origin of the content (i.e.information source) provided by the author? (2) Does the site clearly distinguishbetween opinions and facts when the former is given? (3) Is the content of thesite relevant to the targeted audience? (4) Does the site’s perception of the topicor information an objective one, providing certain equilibrium to the reader? and(5) Is this site similar or comparable to another site that presents information onthe same topic? By examining these principles and questions, we can state that theyare indeed general guidelines and are not based on any form of factual analysis of thewebsite information. The only principle that could be relevant to the trustworthiness,reliability, integrity or fidelity of the facts, is the credibility principle, where thecredibility or reputation of the author can be an indication of how trustworthy thefacts provided really are. However, the last question in these five additional questionssuggested by the author is directly related to the proposed work, where factualinformation is compared between sites covering similar subjects. Furthermore, issuesrelated to questions 1, 2 and 4 are inherently addressed in the proposed work since wepropose to use a trustworthy website for the comparison purpose. In a trustworthywebsite, we can assume to have a trustworthy source of information, and factualand objective information. Question 3 discussed above requires us to consider thecontext while choosing a trustworthy website. For example, people living in Canadamay be biased towards cbc news website. In this work, we have not specificallyintegrated the contextual information, rather our work actually aims at measuringthe comparison between the trusted and not-so-trusted websites and eventuallytranslating this measurement into a quantitative trust value.

Most work relevant to “online trust” do not deal with the factual aspect ofthe information, as stated previously, but address security and privacy issues. Forinstance, the works [8, 11, 19, 21], and [20] identify trust issues by consumers when itcomes to engaging in e-commerce. This essentially involves submission of financialand personal information to merchants through the Internet. Neilsen [31] providedtechniques in order to efficiently communicate trust in web design, once again relatedto e-commerce web applications. Emurian and Wang [19] have taken it a step furtherby giving an overview of several concepts of trust from different disciplines, we willexplore this in the next few paragraphs. They also present “trust-inducing frameworkinterface design features” that could be applied to e-commerce websites in order tooptimize trust levels from the customers’ perspective. We observed that these workspresent essential trust elements or issues, and techniques to handle these issues inorder to enhance online trust from the consumer’s point of view. However, they didnot provide any computational model to determine the trust. Furthermore, they alsodid not consider any factual aspect while determining the trust of a particular website.

Golbeck and Hendler [22] have presented a method to determine the trust rela-tionship between individuals in Web-based social networks. The method computesthe binary trust based on the interaction of individuals and is more suitable in the


context of social networks. In contrast, our method computes the trust of a websitebased on the similarity of its content with the well trusted website. In another work,Bizer and Oldakowski [12] proposed the usage of context and content in computingthe trust for semantic web applications, which are inherent features of the methodproposed in this paper. In addition, the proposed method allows for the dynamiccomputation of trust value of the web content.

In the context of trust computation in multimedia systems, there are a fewresearches such as the works by Torres et al. [39] and by Li and Raimund [29].However, in these works, the researchers have considered trust more from thesecurity perspective rather than the factual contents as has been regarded in thispaper.

3 Proposed method

In the proposed method, the two factors that influence the trust level of a not-so-trusted website are - 1) the semantic similarity between the content of the not-so-trusted and well-trusted websites, and 2) the trust level of the well-trusted websitewith which this similarity is considered. The trust level of a not-so-trusted websiteis considered directly proportional to these two factors. The semantic similaritybetween the content of two websites is computed using the techniques suitable fordifferent media types, e.g. the Latent Semantic Analysis (LSA) technique [28] fordetermining the semantic similarity between two text documents.

In what follows, we first formulate the targeted problem in Section 3.1. Next,Section 3.2 describes how semantic similarity between multimedia content of twowebsites is computed (specifically, text similarity in Section 3.2.1 and image similarityin Section 3.2.2). Finally, Section 3.3 covers the proposed trust computation method.In particular, in Section 3.3.1, we describe the trust computation of a website for aspecific domain while Section 3.3.2 will describe the overall trust computation of thewebsite. In this section, we show the architecture of such a typical web system.

3.1 Problem formulation

We formulate below the problem of determining the trust level of a website.

– Let W = {W1, W2, . . . , Wn} represent a set of n websites that we are exploring.Each of these websites can contain documents of various domains and eachdocument can have information in multiple forms. For instance, a websitewww.cnn.com may contain documents related to politics, entertainment andsports domains, and a document on this website can have information in textand image forms.

– For 1 ≤ i ≤ n, let 0 < φk,qi, j < 1 be the similarity value between the qth media con-

tent of two websites Wi, W j ∈ W for domain k up to a given time instant. Here,q ∈ {text, image, video, audio, . . . }; and k ∈ (1, m), m being the total number ofdomains. φ

k,qi, j is determined by using different media similarity computation

techniques as described in Section 3.2. Note that when the k and q superscriptsare dropped from φ

k,qi, j , the resultant symbol φi, j represents the similarity value

between websites Wi and W j for all domains and all media types.


– For 1 ≤ i ≤ n, 1 ≤ k ≤ m, let 0 < Tki < 1 be the user’s trust in the ith website for

the kth domain up to a given time instant. We assume that, for each domain,we have at least one website which we call “trusted”. A trusted website is theone that has a trust level greater than a threshold Tthreshold. This trust can beestablished using the traditional user feedback method. Similar to φ

k,qi, j , in the

case of Tki , when the k superscript is dropped, the resultant symbol Ti represents

the trust value of website Wi for all domains.– Since the similarity value and the trust level evolve over time, we represent

their updated values with a hat ( ˆ ) symbol. For example, φk,qi, j is denoted as the

similarity value up to the previous time instant, and its updated value up to thecurrent time instant is represented as φ̂

k,qi, j .

Our objective here is to determine the overall trust level, T̂i, of a new “not-so-trusted” website Wi up to the current time instant based on, (1) its trust level up tothe previous time instant Ti, and (2) its similarity with the “trusted” website Wj upto the current time instant φi, j, assuming that the initial trust level of this “not-so-trusted” website is a number close to a positive infinitesimal for any domain.

3.2 Determining semantic similarity

The semantic similarity between the qth media content of two websites Wi, W j ∈ Wfor a domain k up to a given time instant, φ̂

k,qi, j , is computed recursively as the

weighted sum of the similarity value at the current time instant, denoted as ϕk,qi, j ,

and the similarity value up to the previous time instant, φk,qi, j . Precisely, φ̂

k,qi, j is calcu-

lated as:

φ̂k,qi, j = α × ϕ

k,qi, j + (1 − α) × φ

k,qi, j (1)

Here, α ∈ [0, 1] and 1 − α are the weights assigned to the current and past similarityvalues, respectively. Note that, in the absence of any prior information in (1), theinitial value of φ

k,qi, j is considered equal to some positive infinitesimal, ε.

The semantic similarity between the two websites Wi and W j for a do-main k, φk

i, j, is computed by combining the φk,qi, j values for all media types q ∈

{text, image, video, audio, . . . }, as:

φki, j =

∑

q∈{text,image,video,audio,... }wq × φ

k,qi, j (2)

where wq is the weight assigned to a particular type of media. For instance, the weightfor the text could be higher than the weight for an image on a website.

The semantic similarity value between different media content can be computedusing various methods. In the following subsections, we describe the methods tocompute semantic similarities φ

k,texti, j and φ

k,imagei, j between two types of media, text

and image, respectively. Note that although in this paper we use only two types of


media, the proposed method is generic and can also be applied for other types ofmedia.

3.2.1 Text similarity

We use the LSA technique to determine the semantic similarity between the textcontent of two websites. This technique has been widely used to compute thesimilarity between text documents [28]. The LSA technique has been shown to beconsistent with the human performance in judging semantic similarity between textdocuments [26].

LSA starts by forming a matrix C = [cx,y], where cx,y is the frequency of the xthword in the yth document in the corpus. According to this document representationschema, LSA uses three local weighting functions to measure similarity. The func-tions essentially reflect the importance of a word within a document, the frequency ofa word throughout the whole corpus of documents, and “the number of dimensionsretained during the singular value decomposition, which makes assumptions aboutthe complexity of the underlying semantic regularities expressed by the corpus” [28].

These functions are then used to generate a weighted corpus representation W ′ =[w′

x,y]. This schema is then subjected to singular value decomposition. Afterwards, avariant of the Cosine Model is used to measure the similarity.

Note that we used LSA as one of the similarity measure approach. However dueto its limitations LSA may fail in some cases. For instance, website 1 contains the text“the attack to [country] is justified” whereas website 2 contains the text “the attack to[country] is not justified”. These are two completely different opinions, but the LSAalgorithm would compute a high similarity value. In these cases, other techniquessuch as opinion mining [27] can be used.

3.2.2 Image similarity

In literature, there are various works for computing the semantic similarity betweentwo images (e.g. Choi [17], Chen and Kiyoki [16], Andreatta [7], and Chen [15]).Some of them, e.g. Batko et al. [10], are used for web-based searching. There aretwo basic approaches to compare images: content-based and metadata-based [7].The content-based approach uses low level features of the images such as color,texture, shape, and spatial location [38]; however, the metadata-based approach aimsto classify the images into broad categories (e.g. airport, sea beach) [30]. We adoptthe latter (i.e. metadata-based approach) because it leads more towards semanticsimilarity rather than content-similarity. In particular, we process the image captionsto extract the concepts. These captions are generally available below the images onthe websites.

An example of images from two websites providing similar news is shown in Fig. 1.The picture on the left (Fig. 1a) shows the caption “Heathrow has suffered two majorsecurity breaches in recent weeks” below the image. Similarly, the image captionin the picture on the right (Fig. 1b) is “The runway was closed during a controlledexplosion of the bag”. The semantic similarity between these two images is that bothimages are related to the common concepts “airport” and “security”. This is foundbased on the extracted tags.

The similarity between the tag sets obtained from two image captions is computedas follows. Let Gi be the set of tags contained in the image caption of website Wi, and


(a) (b)

Fig. 1 An example of images in two websites providing similar news: a www.cnn.com;b www.bbc.com

G j be the set of tags contained in the image caption of website W j. Let both websitesbelong to a domain k. The similarity of two image captions is determined as:

φk,imagei, j = |Gi ∩ G j|

|Gi ∪ G j| (3)

Precisely, the similarity value is the intersection of the set of tags from Gi and G j

over the union [14].In the example shown in Fig. 1 the tags are obtained as follows. From the image

caption of the picture on left we obtain the tags “Heathrow” and “security”; whilefrom the image caption of the picture on right, “runway”, “explosion” and “bag”tags are obtained. We can refine these tags using some predefined knowledge todetermine that the “Heathrow” and “runway” tags belong to the common concept“airport” and the tags “explosion” belongs to the concept “security”. With thisobservation, we determined Gleft = {airport, security} and Gright = {airport, security,bag}. Using these two sets of tags, we obtained φ

k,imagei, j = 2

3 = 0.66. Note that one canalso use ontology to refine these tags. In the case of ontology, one can either createa new ontology or use an existing ontology in the relevant domain, for example,LSCOM [1] and WordNet [3].

There could be situations when Wi and W j articles have different number ofimages. For instance, the web article on one of these websites to be compared maynot have an image at all while the similar article on the other website may have animage. These scenarios are illustrated in Table 1. As shown in the table, if there areno images in the web articles, we use only text for the comparison purpose. In caseone of the web article has an image but other does not have, we again compare onlytext. In these two cases, we use w1 = 1 and w2 = 0 as the relative weights assigned totext and image content of the websites, i.e. only text is considered and image contentis given a zero weightage. However, if both the web articles have images, we use bothtype of content for the comparison purpose with w1 = 0.70 and w2 = 0.30. Here, a

http://www.cnn.com

http://www.bbc.com


Table 1 The values of different parameters used

Number of images in Overall similarity is computed based on

Wi (trusted) W j (not-so-trusted)

= 0 = 0 Only text (w1 = 1 and w2 = 0)= 0 ≥ 1 Only text (w1 = 1 and w2 = 0)≥ 1 = 0 Only text (w1 = 1 and w2 = 0)≥ 1 ≥ 1 Text as well as image (w1 = 0.70 and w2 = 0.30)

Two image captions are compared using (3)For combinations of tag sets of Wi and W j,

the one having the maximum φk,imagei, j is chosen.

higher weight is given to the text content because text is usually the most prominentaspect of any website. Furthermore, if the web articles have more than one image, allpairs of image content are compared and the one having maximum similarity valueis chosen.

Note that although in this paper we have used image captions metadata to com-pute the image similarity, the proposed method does not have any such restriction.If image captions or other metadata are not available, a content-based approach thatprovides semantic similarity by comparing the low-level image features can also beused [34, 38].

Furthermore, in this paper we restrict our focus on the use of the text and imagecontent of the websites for determining their trust, however other modalities such asaudio and video can also be explored. For these modalities, corresponding semanticsimilarity computation techniques (e.g. [9, 40] for audio and [25] for video) canbe used.

3.3 Similarity based trust computation

In the following two subsections, we describe how the proposed method computesthe trust level of a not-so-trusted website for a particular domain (in Section 3.3.1)and for all the domains (in Section 3.3.2).

3.3.1 Domain-level trust computation

For a particular domain k, the trust level of a not-so-trusted website W j updated upto a given time instant, T̂k

j , is given by a function g, which consists of the followingtwo variables:

1. Tkj : The trust level of website W j for a particular domain k up to the previous

time instant;2. φk

i, j: The semantic similarity between the well-trusted website Wi, and the not-so-trusted website W j, up to the current time instant.

Precisely, T̂kj , which is the trust level of website W j for a particular domain k up

to the current time instant, is updated as:

T̂kj = g

(Tk

j , φki, j

)(4)


We use an exponential model to develop the function g. The choice of this functionis based on the heuristics that the trust evolves exponentially [5, 18, 35], which isjustified by resorting to results proven in [33], where the authors showed that thequality of a heuristic algorithm is determined by the accuracy of the heuristic functionit uses. Furthermore, the exponential model used in this paper will be validated withthe experiments in Section 4.

The (4) is rewritten as:

T̂kj = Tk

j × eβ

N(5)

where, β represents the growth/decay factor for the trust level Tkj of website W j for

domain k at a given time instant (see (6)) and N is used as a normalization factor(see (7)).

β = Tki × �φk

i, j × μ (6)

In (6), the growth/decay factor β is a product of Tki , �φk

i, j and μ. Tki is essentially

the trust level of a so called well-trusted website up to a given time instant, as definedearlier; �φk

i, j is the difference (φ̂ki, j − φk

i, j) in the similarity values between the websitesWi and W j for domain k at the two consecutive time instances; and μ is used tocontrol the rate of growth or decay. The growth and decay depends on the fact thatthe change in the degree of similarity, �φk

i, j, is either positive or negative. Hence,μ can hold two different values, μgrowth and μdecay depending on the value of �φk

i, j.Precisely, if �φk

i, j > 0, the μ is considered as the rate of growth μgrowth; otherwise itis regarded as the decay rate μdecay.

The normalization factor N limits the trust level value between 0 and 1, and iscomputed as:

N = Tkj × eβ + (1 − Tk

j ) × e−β (7)

3.3.2 Overall trust computation

The overall trust T j for a particular website W j up to a given time instant is calculatedby averaging the trust levels computed for all the domains (shown in (8)).

T j = 1

m

∑m

k=1Tk

j (8)

where m is the total number of domains.

4 Experiments and results

In this section, we describe the experiments and present the results in order todemonstrate the utility of the proposed method. The experiments have been de-signed with the following two objectives - first, to demonstrate that the proposedmethod works well; and second, to establish that it performs as good as any othertraditional approach of computing trust with an additional advantage of overcomingthe dependency on user feedback.

We organize this section as follows. In Section 4.1, we will describe the data setused in the experiments. Section 4.2 will provide an overview of the user survey


method used in order to determine the “trusted” website for each domain. InSection 4.3, we will describe the empirical user study employed for the purposeof further evaluating and hence verifying the results obtained by the proposedmethod. Furthermore, in this subsection we also present five different experimentsand detailed results in order to demonstrate the utility of the proposed method.

4.1 Data set

The data set used in the experiments consists of five websites corresponding (W1,W2, W3, W4, W5) to three different domains (k1, k2, k3) (see Table 2). These fivewebsites have been evaluated for the above mentioned three domains. However, it isimportant to note that Aljazeera.net is an exception to this, for its evaluation was onlydone for the first domain due to its lack of information about the last two domains.

For each domain, we have gathered 72 web news articles from each of the fivewebsites (except in the case of k2 and k3 where W4 was excluded as mentionedearlier). These articles were gathered over a period of six months, where threearticles per week were actually collected. The length of the articles varied from 50to 200 meaningful words. Out of 72 articles, almost one third of them had imagesalong with the text content.

The evolution of trust has been performed based on data collected over a period ofsix months. Note that some domains such as International Politics had more articlesto compare due to the fact that several topics, such as the war in Iraq, are coveredextensively by all websites reporting the news.

The values of different parameters used in the experiments are shown in Table 2.α is chosen as 0.50 to assign equal weights to the past and current similarity values.We have chosen Tthreshold = 0.70, which is reasonably higher than 0.50, the thresholdto designate a website as trustworthy.

4.2 Determining the initial trusted websites using a user feedback method

A user survey was performed in order to establish the so called “trusted” website.This survey allowed us to determine a trust level for a particular website within aspecific domain, making this website a “trusted” one for that domain. This trust valueis indeed an integral one in the proposed method and is used to populate the trustdatabase.

Table 2 The values ofdifferent parametersused in the experiments

Parameter Value

W1 www.CBC.caW2 www.CNN.comW3 www.FoxNews.comW4 www.Aljazeera.netW5 www.BBC.comq {Text, Image}k1 International politicsk2 Sportsk3 Health sciencesα 0.50Tthreshold 0.70

http://www.CBC.ca

http://www.CNN.com

http://www.FoxNews.com

http://www.Aljazeera.net

http://www.BBC.com


The survey was conducted with 100 users (volunteers) from different disciplines.The users were asked to rank each website from 0 to 5 (zero being the lowest trustlevel—“distrust” and five being the highest trust level—“full trust”). A typical sce-nario illustrating this would be the fact that one may trust CBC.com for InternationalPolitical news with a rank of three out of five. The rank values collected from 100users were averaged in order to compute the trust level of the “trusted” website foreach of these three domains. A summary of the results of this user survey has beenprovided in Table 3. These results conclude that BBC.com was found to be the most“trusted” website for all three domains. More specifically, as shown in Table 3, thiswebsite was found to be trusted 82% when it comes to International Political news,76% for Sports related news, and 70% in Health Sciences news. Note that these trustvalues were determined by calculating the average of the rankings obtained from 100users.

Note that a user survey is adequate in such a case where news websites areevaluated, since most of these websites are familiar to the users and their reputationis well known in general. However, when it comes to determining the trustedwebsite(s) within a set of specialized websites that concentrate on one particulardomain (e.g. medicine, sports, information technology, etc.), typical users may nothave an appropriate understanding of the domain in order to make an accurateranking of the trust level or general assessment based on the factual informationpresented in the website. Therefore, a typical user survey could be unsuitable, andcould be replaced by an expert user survey where experts in the domain are consultedbased on their knowledge and past experience in that particular domain. We can alsoconsult a renowned trusted source in a particular domain such as the Medical LibraryAssociation website [2], as mentioned previously, in order to establish the “trusted”websites in a particular domain. Nonetheless, this type of reliable source could bedifficult to find for other and somewhat less crucial domains than health or medicine.Also, as mentioned previously, one has to be vigilant in his findings of such sources.In other words, one has to ask himself whether or not this source is actually credibleitself, before trusting the information sources it endorses.

4.3 Empirical user study

The second type of user study, which we have employed, also involves the feedbackof users. We call this study the “Empirical User Study”. In contrast to the userfeedback method which is primarily needed for the establishment of the “trusted”websites, this study is essential for a more in-depth evaluation of the proposedmethod in order to reinforce the obtained results. In this way, the comparisonbetween the results of the proposed method and those obtained by empirical userstudy allows us to assess the validity of the proposed method.

Table 3 Trust levelsdetermined by the user surveymethod for each domain

Website International politics Sports Health sciences

W1 0.79 0.73 0.63W2 0.72 0.68 0.62W3 0.53 0.56 0.54W4 0.67 – –W5 0.82 0.76 0.70


The empirical user study was essentially designed in order to view the evolutionof trust, and all its related psychological and social factors, with regards to factualinformation on the internet from a real world perspective. The evaluation consistedof ten subjects (users), all being graduate students chosen from different disciplinesincluding administration, engineering, science, and social science. The assortmentof disciplines were chosen purposely in order to cover a wider range of perspectives.Thus, this diversity in schools of thought would lead to less biased overall assessment,and hence more accurate results that depict a better picture of the diversity in theworld.

Essentially, the users were given the same articles (See Section 4.1) and they wereasked to compare each article per domain with the trusted article (i.e. from BBCwebsite): first, the users had to judge the similarity of these documents on a 10-point scale (zero being “completely unrelated” and ten being “completely related”);second, they had to give a trust level value also on a 10-point scale (zero being thelowest trust level—“distrust” and ten being the highest trust level—“full trust”).

In this empirical study, the articles were presented to the users at differenttimes. Also, the not-so-trusted websites were anonymously provided to the users, i.e.without giving the name of the website but only a number (e.g. the first article camefrom W3, the second article came from source W2, etc). The users were asked tokeep track of these source websites, referred to by number only, so that anonymitycould be maintained to eliminate the bias of the users towards particular websites.Furthermore, the subjects were asked to give trust levels based not only on thecurrent similarity value between the trusted and not-so-trusted articles, but basedon the previous similarity value given by the user, and most importantly, based onthe previous trust level given by the user to that particular anonymous source.

4.4 Evaluation of proposed method

We have designed the following four experiments:

Experiment #1 This experiment would show how the growth of the trust level of the“not-so-trusted” websites is affected by different values of μgrowth,and hence justify the rationale of using such a particular value overanother in the latter experiments. Note that for this experiment weused 1/3 of the whole data set as a training data. The rest 2/3 ofthe whole data is used as test data for in the following experiments(Experiments #2–#5).

Experiment #2 This experiment is designed to show the dependency and correla-tion between the trust level of a not-so-trusted website with respectto its similarity with the trusted website over a period of 90 days.Note that the choosing of the μgrowth value is subjective. In thisexperiment, we have chosen this value to be 3.5. The rationalebehind this decision is explained in Experiment #1.

Experiment #3 The main purpose of this extensive experiment is to examine andanalyze the results (e.g. similarity values and trust levels) obtainedfrom the empirical user study by comparing them to those obtainedfrom the proposed method, in order to verify it’s practicality andeffectiveness. Moreover, this analysis will allow us to assess certainstatements made with regards to the notion of “trust” in general,


by looking at psychological and social facets that could explainthe results obtained from the empirical user study. Once again,the choice of the growth value for the proposed method is 3.5, adecision which is justified in Experiment #1.

Experiment #4 In this experiment we show the impact of considering images (visualcontent) in determining the trust of the websites.

Experiment #5 This experiment is an extension of the second experiment. It willdepict a consolidated version of the results already covered in Ex-periment #3 in order to further solidify the strength of the proposedmethod. Also, additional analysis will be given with respect to theresults obtained by the user feedback method to help us understandthe proposed method’s limitations and further enhancements.

We describe these five experiments in the subsequent sections.

4.4.1 Experiment #1—determining the ideal growth value (μgrowth)

In this experiment, we aim to determine the appropriate or ideal growth value,μgrowth, which is an important element of the growth factor function β (6). Essentially,this subjective value mainly dictates the rate of growth in the trust level. It alsoindicates the rate of decay or decrease (denoted by μdecay), for this value is inverselyproportional to the growth value.

The ideal growth value is one where a not-so-trusted website would be able toachieve a trust level close to what it obtained using the user feedback based method.We define a term Dk

i as the mean square difference between the trust level ofa website Wi for domain k when the proposed method and the traditional userfeedback method are used. In short, we refer to this term, Dk

i , as the individualdifference (‘individual’ implying one domain within one website) in trust levelsbetween the proposed method and the user feedback method. The term Dk

i iscomputed as follows:

Dki = 1

l

√∑

l

|(Tki )2 − (�k

i )2| (9)

Note that in (9), �ki is the trust level of the ith website for the kth domain up to a give

time instant using the user feedback method. Also, the term l represents the totallength of time that the summation of squared differences between trust levels (insidethe squared root) is carried out. In our case, based on the empirical user study, thissummation is carried out over a period of 90 days (i.e. l = 90).

Equation (10) gives the average difference, Di, for all domains of a particularwebsite Wi. It is computed as the summation of all individual differences, Dk

i , for aparticular domain, divided by the total number of domains, m:

Di = 1

m

m∑

k=1

Dki (10)

Finally, the overall average difference Dall for all the domains in all the websites,between the proposed method and the feedback based method is computed as the


summation of all average differences, Di, divided by the total number of websites, n.The mathematical equation is as follows:

Dall = 1

n

m∑

i=n

Di (11)

The overall average difference, Dall (for 1 ≤ i ≤ n), is plotted for different growthvalues (1 ≤ μgrowth ≤ 7) in Fig. 2. This is done in order to examine the value ofμgrowth at which the value of Dall is minimal. Note that the value of μdecay used tocompute the trust levels for each domain (and hence compute Dk

i followed by Di), is20% of the growth value (μdecay = 0.2 × μgrowth) in this experiment. We have chosenthis value of μdecay based on its suitability with our data set, however, it could bedifferent for different set of websites. Nevertheless, this decay value, being inverselyproportional to the growth value, has an opposite yet equally important role as thelatter: it dictates the rate of decrease when the difference in similarity measures, �φ,(i.e. �φ being the difference in similarity measures between two consecutive days) isnegative.

As can be seen in Fig. 2, the overall average difference Dall decreases as the growthvalue increases. This is observed until a certain point where the lowest difference isreached when the growth value (μgrowth) has an approximate value of 3.5. At thispoint in time, the difference is minimal. This implies that the trust levels dynamicallycomputed according to the proposed model are ‘very similar’, in numerical value,to those given by the users in the empirical user study. Note that this value is infact the one used in the first and second experiment. This general tendency could beconfirmed for each domain in each website, for the average individual differencesamong these domains and websites is indeed the value represented and reflectedby this overall average difference in Fig. 2. In other words, the curve illustrated inFig. 2 clearly reflects the overall behavior of the curves representing each individualdifference Dk

i versus different growth values (i.e. ten graphs of Dki vs. μgrowth) if these

graphs were to be plotted. In turn, this could imply that the ideal value of the growthfactor, where the dynamically computed trust level values are closest to the users’

Fig. 2 The overall averagedifference in trust levels,between the proposedmethod and the userfeedback method,when different growthvalues are used

0 1 2 3 4 5 6 70

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2


trust values, is approximately 3.5 for this particular evaluation period of 90 days. Putin a practical sense, we would have to use a growth factor of 3.5 in the proposedmethod in order to reach similar trust level values established by the users in theempirical study, given the similarity measures obtained for this 90 day period.

After this growth value of approximately 3.5 is reached, the value of the differencebetween the methods increases sharply, as expected. This growth is at a faster ratethan the decrease initially observed until the growth value reaches 3.5. Eventually,after a certain growth value, this difference will follow a more gradual increase.

To conclude this experiment, we can state that subjective decision making of thegrowth value could be seen as another overhead to the proposed method. However,we have shown that the ideal growth value could be tailored to make the trust levelsprovided by the proposed method more realistic and closer to the values establishedby such techniques as empirical user studies. Within a period of three months, theresults obtained seemed quite reasonable and practical. This allowed us to adapt theproposed method in such a way to reflect how factual trust is actually built in a user’smind.

4.4.2 Experiment #2—trust level vs. similarity value

In this experiment, we study the dependencies among important variables in theproposed trust computation model. In particular, we will examine the relationbetween the trust level of a particular “not-so- trusted” website, and the similarityvalue between this website and the “trusted” website.

Figure 3 depicts a graph which shows the behavior of the evolved trust level ina website W2 (CNN) for the International politics domain, in terms of its similaritywith the trusted website W5 (BBC) in that same domain. The first curve observed,denoted by T1

2 , represents the trust level computed by the proposed method of anot-so-trusted website W2; whereas the second, denoted by φ1

2,5, shows the semanticsimilarity value between the former website and the trusted website over a periodof time.

Fig. 3 Trust level (T12 )

versus similarity value (φ12,5)

over a period of time

0 10 20 30 40 50 60 70 80 900

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of days


From Fig. 3, it is clear that an increase in the similarity value between the twowebsites, leads in turn to an increase in the trust level. Conversely, a decrease in trustlevel is caused principally by a decrease in the similarity value. This is an apparentimplication of dependency and correlation between both variables, where the trustlevel is in effect proportional to the similarity value.

Let us illustrate the importance of this proportionality by observing both curvesin Fig. 3 within specific time frames. We observe a sharp increase between days0–10 and days 55–59, in the similarity value, which in turn leads to an increase inthe trust level within the same period of time. This is a result of high semanticsimilarity obtained between the compared news articles (from the “not-so-trusted”and “trusted” websites) over these periods of time. In contrast, a sharp decrease insimilarity on days 52–53, causes a corresponding decay in the trust level at that time.

4.4.3 Experiment #3—the proposed method versus empirical study: an example

From Experiment #2, it can be seen that trust is indeed an evolving process built overa certain period of time. This leads us to Experiment #3 which is designed to performa comparison of the two methods: the proposed method and the empirical user study.

In Fig. 4a, we have plots ψ11,5 and φ1

1,5, representing the similarity values betweenthe first website and the trusted website, using the user feedback method andthe proposed method respectively. Note that ψ1

1,5 represents the average semanticsimilarity values by the ten users. Figure 4b shows the curves �1

1 and T11 , which

represent the average trust levels of the users based on the first case of the empiricalstudy and the trust level values based on the proposed method, respectively. Bothsets of curves are plotted using data collected over a period of 90 days.

Note that these curves represent similarity and trust values for the first newsdomain (International Politics) of the first website W1 (CBC.ca). Furthermore, thetwo remaining news domains (Sports and Health) where not represented graphicallyin order to prevent redundancy, since the result trends were very similar to thoseof the first domain. This redundancy of trend also explains the reason why thedetailed analysis for the remaining ‘not-so-trusted’ websites (W2, W3, W4) has alsobeen omitted. In order to confirm this highly related tendency among domains andwebsites, the similarity in the results will be presented further in this section, using abriefer format.

We can see from the graph shown in Fig. 4b that both curves behave in a similarmanner, where a progressive evolution of the trust levels is observed. In addition,the trust level values increase/decrease at a similar rate over the period of time, andboth curves have some intersection points (e.g. day 58, 70, 81, etc.), where trust levelvalues between both methods are identical or very similar. For instance, betweendays 6–10, there is a sharp increase in trust levels, and between days 29–34, a similarincrease is observed due to a higher observed similarity between W1 and W5 duringthe same period of time (shown in Fig. 4a). In contrast, between days 35–37 or days53–54 for example, a decrease is perceived, which could be explained by similar yetopposite reasons as stated previously: there was a low similarity value observed bythe users and computed by the LSA model at that point in time, which consequentlyaffected the value of trust.

Moreover, by examining the graph shown in Fig. 4b, we see that the user’s trustlevel is slightly lower than the trust computed by the proposed method for most of thetime, until approximately day 57, where the trust value based on the empirical study


0 10 20 30 40 50 60 70 80 900.4

0.5

0.6

0.7

0.8

Number of days

0 10 20 30 40 50 60 70 80 900

0.2

0.4

0.6

0.8

Number of days

(a)

(b)

Fig. 4 A comparison of the proposed method with the user feedback method: a Similarity valuesbetween W1 (CBC—not-so-trusted website) and W5 (BBC—trusted website)—ψ computed usingthe user feedback method versus φ computed using the proposed method, b Trust values of W1—�computed using the user feedback method versus T computed using the proposed method

was very similar to the trust level dynamically computed by the proposed method.Afterwards, around the 75th day, the users’ trust surpassed that of the proposedmethod. What follows is a discussion of reasons for this trend.

From a social and psychological perspective, we can assume that these educated,and fairly more skeptical or less vulnerable users, were somewhat reluctant to trustthese unknown websites. This unwillingness to fully engage remained persistent for aperiod of approximately two months, even when documents were found to be quitesimilar to those originating from the trusted website. However, after this periodof time, the users found themselves more familiar with the sources. Consequently,their skepticism was somewhat lowered, and their readiness to engage eventuallyaugmented, resulting in an increase in their trust.

For example, if a user gave a trust level of ‘2’ on a certain day, all the othertrust level values given after that day were mostly more than ‘2’, even if the userfound lower similarity between the trusted and ‘not-so-trusted’ websites. Instead,users gave this trust level value repeatedly for a longer period of time, leading toa much slower increase in the trust level. This behavior was particularly observedafter several weeks within the empirical study, where lower similarity values didnot directly affect the trust level values as much, since users had already built acertain trust level with the unknown sources. For instance, an average 10% decrease


in similarity value did not necessarily lead to a decrease in the average users’ trustlevel, whereas a decrease in the same manner according to the proposed method,actually lead in turn to a decrease in trust level. This can be seen in Fig. 4 betweendays 40–41.

On the other hand, in cases where trust level values were scored dramaticallylower by a particular user based on lower similarity measures observed, most usersreturned, after only a short period of time to the initial trust level given as soon asthe current similarity measure observed was considered higher than the value thatcaused the decrease in the trust level. This occurred even if this current similaritymeasure was not necessarily amongst the highest observed by the user. Users hadto observe lower similarities over several consecutive days in order to cause them tosignificantly lower their previously indicated trust value. However, when the usersreturned to their initial trust level value before the sharp decrease, this value waskept for a “longer than expected” period of time, which in turn led to an overallslower increase of users’ trust. It is essential to note that this noticeable decreasein trust values at certain times (e.g. on days 35–37, 53–54, 72–76) by users was notdirectly proportional to the decrease observed by the proposed method since thescales used for each method was not identical, limiting the users’ assessment. Forexample, when a decrease in trust by a user went from 4 to 3.5 at a given time instant,the proposed method saw a decrease from 4 to 3.9 at the same time; a difference of0.4 between both methods. This type of difference is almost inevitable when there isuse of empirical studies involving users, and it would seem to be quite unreasonableto ask users to start using a tenth of a point on a ten-point scale to make either trustor similarity assessments.

The trends discussed so far could also be interpreted from another psychologicalpoint of view, where most users were not able to assess or attribute an exactnumerical value that one should add or deduct to the previous trust level, basedon the similarity measure. Otherwise stated, the mental relation between similarityand trust was difficult to establish in terms of the limited ten-point scale used for theevaluation.

From a different viewpoint, differences between trust level values from the twomethods could have been affected from the fact that several pairs of news documentswere found to have a slightly higher/lower semantic similarity by the users thanthe similarity measure calculated by the LSA model (i.e. depicted in Fig. 4a). Forinstance, several articles covered the main idea for the most part, but drifted off ina different direction for the remainder of the article, while the main message wasstill conveyed in a similar manner, with some varying aspects. This facet was notcaptured effectively by the LSA model, who gave such pair wise comparisons a lowersimilarity value than the users, and human judgment was needed to make a moreadequate detection of semantic similarity. On the other hand, some articles reportedsimilar details, but the message or main theme of the article was not essentially thesame. Also, users might have seen certain bias elements that could have influencedthe articles’ credibility even when giving factual information in coherence with thetrusted website. Here is a real-life scenario to illustrate this point: let us say thatone reads an entire article conveying several known facts consistent with thosefrom a credible source, and then finds one important and subtle fact, not found orcontradicting the well-known source. This one element could indeed jeopardize theentire articles’ credibility in the reader’s eyes, even if the remainder of the article isactually very similar to the credible source.


In sum, despite the slender discrepancy among the similarity between the twomethods and the limitations mentioned above, we can assert, according to Fig. 4a,that the overall trend between the methods is quite similar. Hence, this analysisallows us to deduce that both methods are quite comparable, justifying the practi-cality and effectiveness of the proposed method when used in the real world. Thisstatement will be further validated in Section 4.4.5, where the quantitative differencebetween the two methods will be presented for all the websites in different domains.

4.4.4 Experiment #4—impact of using images

This experiment shows the impact of using images in determining the trust value ofthe websites. In 72 articles, there were 24 instances when both trusted and not-so-trusted websites had images. Similarity between the text of two web articles wascomputed using LSA technique (as described in Section 3.2.1 and the similaritybetween the images of two web articles was computed by comparing image captions(as described in Section 3.2.2).

Figure 5 shows two graphs. The top graph (Fig. 5a) shows how similarity valuevaried over the 24 instances when images were found on the websites. The threeplots in this graph show the similarity values based on text only (φ1,text

1,5 ), image only

(φ1,image1,5 ), and both text and image (φ1,{text,image}

1,5 ). The similarity value φ1,{text,image}1,5 is

computed using (2) with w1 = 0.70 and w2 = 0.30.

0 5 10 15 20 250

0.2

0.4

0.6

0.8

1

Instances when images were available on the websites

Sim

ilarit

y va

lue

0 5 10 15 20 250

0.1

0.2

0.3

0.4

0.5

Instances when images were available on the websites

Evo

lvin

g tr

ust v

alue

(a)

(b)

Fig. 5 Effect of considering images in determing the trust value of websites


The graph in Fig. 5b shows the evolution of trust value using the proposed methodbased on text only (T1,text

1 ), and both text and image (T1,{text,image}1 ). As can be seen in

the figure, usage of images in addition to text improves the trust of the websites. Inthis graph, the trust computed using user feedback method (�1,text

1 and �1,{text,image}1 )

was also plotted against the trust computed using the proposed method (T1,text1 and

T1,{text,image}1 ) and both were found quite comparable. It was observed that although

in the beginning users did not give much weight to images, after some time (e.g. from14th instance onwards) they started to be more inclined to give higher trust level tothe websites which had images.

4.4.5 Experiment #5—overall result verif ication

This experiment was designed to verify the overall results of the proposed method.Here, we present the consolidated mean square difference, Dk

i , for each domainin each not-so-trusted website in Fig. 6. As we can see, the differences betweenthe proposed method and the user feedback method for all the websites in all thedomains are quite small.

For instance, we observe noticeable lower differences (approximately 2%) in thesports domain (as shown in Fig. 6b), for all the websites, than any other domain. Thisis due to the fact that general sports information is mainly based on well-known factsdisplayed across most news sites. In fact, most sports articles chosen for this studyreflected this general and commonly found pool of sports information, where gamereports revealing scores, player injuries, summaries of press conferences, etc. werethe main topics covered. However, there is in effect more complex sports informationcomprised of intricate facts that could be questionable (e.g. player trade rumors, salesof sports teams, sports scandals, etc.) found in sport specific websites. Perhaps thedifference between the two methods would have been higher if articles were takenfrom such sites.

Conversely, a relatively higher difference (approximately 7–9%) among the healthdomains (Fig. 6c), is observed. This tendency could be explained by a certaindifficulty in the finding of relevant articles (i.e. articles relevant to each other) forcomparisons in the study due to the limited health information provided by generalwebsites, and the breadth of the topic. Therefore, the information found in healtharticles for this study did not present as high of similarity measures as the sportsdomain or even the entertainment domain, as topics covered in several articles weresomewhat similar, but were presented from different perspectives with a variancein details. More importantly, this variation led to a wider difference in semanticsimilarity assessments between users and media similarity computation techniques,which in turn lead to a more noticeable difference in trust levels. Nonetheless,for the use of health or medically oriented websites, experts in the domain wouldhave allowed us to choose more specific and relevant topics, and perhaps get bettersimilarity results among articles, leading to higher trust levels. But then again, inthe cases of extreme specificity, certain subjects are covered by a few sources only,or there are several disagreements or contradictions with regards to this particularsubject, making it difficult to not only make adequate comparisons due to the limitedinformation, but to find a trusted source in this very specific sub-domain. This couldbe seen as another, less conspicuous, limitation to consider within the proposedmethod; a limitation derived from the previously mentioned restraint (in Section


0

1

2

3

4

5

6

7

8

9

Websites Websites Websites

Mea

n S

quar

e D

iffer

ence

(in

%)

Mea

n S

quar

e D

iffer

ence

(in

%)

Mea

n S

quar

e D

iffer

ence

(in

%)

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

(a) (b) (c)

W1 W

2 W3

W4

W1

W2

W3

W3W

1W

2

Fig. 6 Mean square difference, Dki , between the proposed method and the user feedback method

for each domain: a Entertainment (D1); b Sports (D2); c Health (D3)

4.4.3) of finding appropriate and sufficient techniques and/or resources to determinethe trusted website in multifaceted domains with various sub-domains.

From a different perspective, we can see from Fig. 6 that some websites havehigher differences than others. This could be explained by some bias factors, evenif the website was kept anonymous. Even if users did not know the source ofinformation where articles were written, after a certain period of time, a vigilantuser could be able to deduce what source of information this article might havecame from. This could have been done by several ways, one of them being the waythe articles coming from a particular source were written, indicating whether or notthe source was a western one or a middle-eastern one for example. In other words,users may have felt certain bias from authors indicating their political inclination forexample, and hence deducing what source this article could come from. This is goodfor the study, because the purpose is to give untrusted and maybe biased sources achance to be trusted and read, users might always have a certain prejudice becauseof the ownership of the source, the author, the references, etc.

In sum, we can affirm that the difference is very small for all domains in all ofthe websites, therefore, we can conclude that the proposed method is comparableto the user feedback method in the empirical study. This being said, a typical andpractical scenario would be the use of this growth factor in order to build the trustlevel of several websites starting from 0 when the proposed system is launched,and eventually evolve the trust level of several websites with respect to numerousdomains, in order to fill the trust database with several trust values and similaritymeasures. This would be done as an initial phase for a period of approximately threemonths, which would give an acceptable trust level as a starting point. Afterwards,as users view these web pages, the trust level of these pages would either increaseor decrease from the pre-established value. Finally, we envision that the proposedmethod when incorporated with the search engines would allow the search results toobtain based not only on hit counts, but trust level also.


5 Conclusions and future work

This paper has presented a method to dynamically compute and evolve the trustlevel of a “not-so-trusted” website of a particular domain (e.g. politics, health,sports, economy, information technology, etc.) based on how similar its contentis with a trusted website of the same domain. This ‘trust level’ in informativewebsites allows us to determine the trustworthiness and credibility of the factualinformation presented in a not-so-trusted website, in contrast to the more familiarconcept of ‘internet/online trust’, which involves security and cryptography issues ine-commerce sites.

As shown in the experimental results, the proposed method allows us to surmountthe users’ feedback dependency, which is typically used as an essential elementin the computation or depiction of online trust of a website. This statement isreinforced after evaluating the proposed method with respect to the user feedbackmethod/empirical user study—a study allowing us to examine from a psychologicaland social perspective the progressive and evolving aspects of trust in users’ minds.Based on the results reported in this paper, both methods are quite comparable andwe can assert that the proposed model to dynamically compute the factual trust of awebsite is indeed a practical and effective one. Although the user survey is somewhatessential to determine the so called “trusted” website, the proposed method has anadded advantage of allowing us to compute the trust level of other websites whosetrustworthiness is not yet available.

In future, the proposed method can be extended in many ways. For instance, im-partial experts in each domain could be consulted in order to obtain their trust levelin a particular website based on their personal experiences combined with collecteddata from this research or other researches. From a different, yet related approach,the proposed method could be fine tuned to accommodate certain ethnicities orcommunities. In this case, the trusted website would be determined according tothe specific views, standpoints, and general cultural background of this particulargroup of people. Furthermore, our method can be extended to use ontology-drivenmetadata [6]. We would also like to investigate the suitability of combining trust withother popularity based mechanism in future.

References

1. LSCOM: large scale ontology for multimedia. URL: http://lastlaugh.inf.cs.cmu.edu/lscom/. Lastaccessed 22 March 2011

2. Medical Library Association. URL: http://www.mlanet.org/resources/medspeak/topten.html.Last accessed 22 March 2011

3. WordNet. URL: http://www.w3.org/2001/sw/BestPractices/WNET/wordnet-sw-20040713.html.Last accessed 22 March 2011

4. Oxford english dictionary: the compact edition. Oxford University Press, New York (1971)5. Al-Ani B, Redmiles D (2008) Forces that influence trust in technology in the Middle East:

culture, politics and history. In: ACM workshop on HCI for community and internationaldevelopment

6. Albertoni R, Martino MD (2008) Asymmetric and context-dependent semantic similarity amongontology instances. Springer journal on data semantics X. LNCS 4900:1–30

http://lastlaugh.inf.cs.cmu.edu/lscom/

http://www.mlanet.org/resources/medspeak/topten.html

http://www.w3.org/2001/sw/BestPractices/WNET/wordnet-sw-20040713.html


7. Andreatta C (2008) Ontology based viso-semantic similarity for image retrieval. In: Internationalworkshop on content-based multimedia indexing, pp 70–76. London, UK

8. Ang L, Dubelaar C, Lee BC (2001) To trust or not to trust? A model of internet trust from thecustomer’s point of view. In: The 14th Bled electronic commerce conference, pp 40–52. Bled,Slovenia

9. Barrington L, Chan A, Turnbull D, Lanckriet G (2007) Audio information retrieval using seman-tic similarity. In: The IEEE international conference on acoustics, speech and signal processing,vol 2, pp II-725–II-728

10. Batko M, Falchi F, Lucchese C, Novak D, Perego R, Rabitti F, Sedmidubsky J, Zezula P (2010)Building a web-scale image similarity search system. Multimed Tools Appl 47:599–629

11. Belanger F, Hiller JS, Smith WJ (2002) Trustworthiness in electronic commerce: the role ofprivacy, security, and site attributes. J Strateg Inf Syst 11:245–270

12. Bizer C, Oldakowski R (2004) Using context- and content-based trust policies on the semanticweb. In: The 13th international world wide web conference on alternate track papers and posters,pp 228–229

13. Brin S, Page L (1998) The anatomy of a large-scale hyper textual web search engine. ComputNetw ISDN Syst 30(1–7):107–117

14. Buttler D (2004) A short survey of document structure similarity algorithms. In: The 5th inter-national conference on internet computing

15. Chen W (2010) Content-based image detection of semantic similarity. In: The 2nd internationalworkshop on education technology and computer science, vol 2, pp 452–455. Wuhan, China

16. Chen X, Kiyoki Y (2007) A visual and semantic image retrieval method based on similaritycomputing with query-context recognition. In: The 18th European–Japanese conference oninformation modelling and knowledge bases, pp 245–252. Yyteri, Pori, Finland

17. Choi J, Cho M, Park SH, Kim P (2003) Concept-based image retrieval using the new semanticsimilarity measurement. In: International conference on computational science and its applica-tions, vol 2667, pp 967–968. Montreal, Canada

18. Daniel B (2002) Building social capital in virtual learning communities. URL: http://www.usask.ca/education/coursework/802papers/daniel/index.htm. Last accessed 22 March 2011

19. Emurian HH, Wang DY (2005) An overview of online trust. Comput Hum Behav 21:105–12520. Fox S (2000) Trust and privacy online: why Americans want to rewrite the rules. The Pew In-

ternet and American Life Project, Washington, DC. URL: http://www.pewinternet.org/Reports/2000/Trust-and-Privacy-Online/Summary/Findings.aspx. Last accessed 22 March 2011

21. Friedman B, Howe D, Kahn P (2000) Trust online. Commun ACM 43(12):34–4022. Golbeck J, Hendler J (2006) Inferring binary trust relationships in web-based social networks.

ACM Trans Internet Technol 6(4):497–52923. Husted B (1998) The ethical limits of trust in business relations. Bus Ethics Q 8(2):233–24224. Ibrahim H, Atrey PK, El Saddik A (2007) Semantic similarity based trust computation in

websites. In: The ACM international workshop on the many faces of multimedia semantics,pp 65–72. Augsburg, Germany

25. Jung MY, Park SH (2009) Semantic similarity based video retrieval. In: Studies in computa-tional intelligence—new directions in intelligent interactive multimedia systems and services—2,vol 226, pp 381–390

26. Kaur I, Hornof AJ (2005) A comparison of LSA, WordNet and PMI-IR for predicting user clickbehavior. In: The SIGCHI conference on human factors in computing systems, pp 51–60

27. Kobayashi N, Inui K, Matsumoto Y (2007) Opinion mining from web documents: extraction andstructurization. Inf Media Technol 2(1):326–337

28. Lee MD, Pincombe B, Welsh M (2005) An empirical evaluation of models of text documentsimilarity. In: The 27th annual meeting of the Cognitive Science Society, pp 1254–1259. Austin,USA

29. Li Y, Raimund E (2008) Integrating trust management into usage control in p2p multimediadelivery. In: The 20th international conference on software engineering and knowledge engi-neering. San Francisco Bay, USA

30. Liu Y, Zhang D, Lu G, Ma W-Y (2007) A survey of content-based image retrieval with high-levelsemantics. Pattern Recogn 40:262–282

31. Neilsen J (1999) Trust or bust: communicating trustworthiness in web design. Jacob Nielsen’sAlertbox. URL: http://www.useit.com/alertbox/990307.html. Last accessed 22 March 2011

32. Nissenbaum H (2001) Securing trust online: wisdom or oxymoron? Boston Univ Law Rev81:101–131

33. Oommen JB, Rueda L (2005) A formal analysis of why heuristic functions work. Artif Intell164:1–22

http://www.usask.ca/education/coursework/802papers/daniel/index.htm

http://www.usask.ca/education/coursework/802papers/daniel/index.htm

http://www.pewinternet.org/Reports/2000/Trust-and-Privacy-Online/Summary/Findings.aspx

http://www.pewinternet.org/Reports/2000/Trust-and-Privacy-Online/Summary/Findings.aspx

http://www.useit.com/alertbox/990307.html


34. Ramanna S (2010) Discovering image similarities: tolerance near set approach. In: Pal S, Peters J(eds) Rough fuzzy image analysis. Foundations and methodologies, CRC Press, Taylor & FrancisGroup

35. Ryan MJ (2004) Trusting yourself: how to stop feeling overwhelmed and live more happily withless effort. Crown Archetype

36. Schneider GK (2008) Beyond algorithms: a librarian’s guide to finding web sites you can trust.URL: http://www.google.com/librariancenter/articles/0601_02.html. Last accessed 22 March 2011

37. Singh R, Bhattarai BD (2010) Dynamic content-page identification for media-rich websites.Multimed Tools Appl 50(3):491–507

38. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrievalat the end of the early-years. IEEE Trans Pattern Anal Mach Intell 22:1349–1380

39. Torres V, Rodríguez E, Llorente S, Delgado J (2005) Trust and rights in multimedia contentmanagement systems. In: The IASTED conference on web technologies, applications, and ser-vices. Calgary, Canada

40. Wei W, Bin Y, Bang-sheng C (2010) Audio scene semantic similarity computing approach.In: The 2nd IEEE international conference on information management and engineering,pp 85–89

Pradeep K. Atrey is an assistant professor at the University of Winnipeg, Canada. He receivedhis Ph.D. in computer science from National University of Singapore, B.Tech. in computer scienceand engineering and M.S. in software systems from India. He was a postdoctoral researcher atthe Multimedia Communications Research Laboratory, University of Ottawa, Canada. His currentresearch interests are in the area of multimedia computing with a focus on multimedia surveillanceand privacy, image/video security, and web. He has authored/co-authored over 50 research articlesat reputed ACM, IEEE, and Springer journals and conferences. Dr. Atrey is on the editorial boardof ETRI Journal and Journal of Convergence (Web and Multimedia), and is the guest editor forSpringer Multimedia Systems Journal. He is actively involved in his research community and he hasbeen associated with over 20 international conferences in various roles such as general chair, programchair, publicity chair, web chair, and TPC member. Dr. Atrey was recipient of the ETRI Journal BestReviewer Award (2009) and the University of Winnipeg President’s Merit Award for ExceptionalPerformance (2010).

http://www.google.com/librariancenter/articles/0601_02.html


Hicham Ibrahim received his M.Sc. in computer science from University of Ottawa, Canada. He hasbeen a member of Multimedia Communications Research Laboratory at the University of Ottawa.His research interests include multimedia and web.

M. Anwar Hossain is an assistant professor in the Software Engineering Department, College ofComputer and Information Sciences (CCIS) at King Saud University, Saudi Arabia. He receivedthe B.Eng. degree in computer science and engineering from Khulna University, Bangladesh, andthe M.C.S. degree in computer science from the University of Ottawa, Canada, in 2005. Later, heobtained Ph.D. in electrical and computer engineering from the University of Ottawa, Canada,in 2010. At this university, he was associated with the Multimedia Communications ResearchLaboratory (MCRLab), School of Information Technology and Engineering. His research interestsinclude human-computer and human-environment interaction, multi-sensor systems, multimodalsurveillance, service oriented architecture and ambient intelligence. He has authored and co-authored more than 40 publications including refereed journals, conference papers, and bookchapters.


Sheela Ramanna is a full professor and head of the Applied Computer Science Department at theUniversity of Winnipeg. She received a Ph.D. in computer science from Kansas State University,USA and a B.E. in electrical engineering and M.Tech. in computer science and engineering fromOsmania University, India. She is the managing editor for Springer Transactions on Rough SetsJournal (TRS). She serves on the editorial board of TRS, Journal of Intelligent Decision Technology(IOS Press), International Journal of Advanced Intelligent Paradigms (Inderscience), Journal ofAgents and Multi-Agent Systems, Journal of Fuzzy and Rough Systems (IJFSRS). She served asprogram co-chair for RSKT2011, RSCTC2010 and JRS2007. She is currently a co-editor of a Springerbook on Emerging Paradigms in Machine Learning. Her paper on rough control co-authored withJames F. Peters received the IFAC Best Paper Award in 1998. Her research interests include theoryand applications of computational intelligence techniques (rough sets, near sets and fuzzy sets) andperceptual systems.

Abdulmotaleb El Saddik (F’IEEE-09) is university research chair and professor, SITE, University ofOttawa and recipient of the Professional of the Year Award (2008), the Friedrich Wilhelm Bessel Re-search Award from Germany’s Alexander von Humboldt Foundation (2007), the Premier’s ResearchExcellence Award (PREA 2004), and the National Capital Institute of Telecommunications (NCIT)New Professorship Incentive Award (2004). He is the director of the Multimedia CommunicationsResearch Laboratory (MCRLab). He is a theme co-leader in the LORNET NSERC Research Net-work. He is associate editor of the ACM Transactions on Multimedia Computing, Communicationsand Applications (ACM TOMCCAP) and IEEE Transactions on Computational Intelligence andAI in Games (IEEE TCIAIG) and Guest Editor for several IEEE Transactions and Journals.


Dr. El Saddik has been serving on several technical program committees of numerous IEEE andACM events. He has been the General Chair and/or Technical Program Chair of more than 20 in-ternational conferences, symposia and workshops on collaborative haptoaudio-visual environments,multimedia communications and instrumentation and measurement. He was the General Co-Chairof ACM MM 2008. He is leading researcher in haptics, service-oriented architectures, collaborativeenvironments and ambient interactive media and communications. He has authored and co-authoredtwo books and more than 200 publications. He has received research grants and contracts totalingmore than $14 million and has supervised more than 90 researchers. His research has been selectedfor the BEST Paper Award three times. Dr. El Saddik is a Senior Member of ACM, and is an IEEEDistinguished Lecturer.

determining trust in media-rich websites using semantic similarity

Documents